Posted on Leave a comment

A quick introduction to Toolbox on Fedora

Toolbox allows you to sort and manage your development environments in containers without requiring root privileges or manually attaching volumes. It creates a container where you can install your own CLI tools, without installing them on the base system itself. You can also utilize it when you do not have root access or cannot install programs directly. This article gives you an introduction to toolbox and what it does.

Installing Toolbox

Silverblue includes Toolbox by default. For the Workstation and Server editions, you can grab it from the default repositories using dnf install toolbox.

Creating Toolboxes

Open your terminal and run toolbox enter. The utility will automatically request permission to download the latest image, create your first container, and place your shell inside this container.

$ toolbox enter
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download registry.fedoraproject.org/f30/fedora-toolbox:30 (500MB)? [y/N]: y

Currently there is no difference between the toolbox and your base system. Your filesystems and packages appear unchanged. Here is an example using a repository that contains documentation source for a resume under a ~/src/resume folder. The resume is built using the pandoc tool.

$ pwd /home/rwaltr $ cd src/resume/ $ head -n 5 Makefile all: pdf html rtf text docx pdf: init pandoc -s -o BUILDS/resume.pdf markdown/* $ make pdf
bash: make: command not found
$ pandoc -v
bash: pandoc: command not found

This toolbox does not have the programs required to build the resume. You can remedy this by installing the tools with dnf. You will not be prompted for the root password, because you are running in a container.

$ sudo dnf groupinstall "Authoring and Publishing" -y && sudo dnf install pandoc make -y
... $ make all #Successful builds
mkdir -p BUILDS
pandoc -s -o BUILDS/resume.pdf markdown/*
pandoc -s -o BUILDS/resume.html markdown/*
pandoc -s -o BUILDS/resume.rtf markdown/*
pandoc -s -o BUILDS/resume.txt markdown/*
pandoc -s -o BUILDS/resume.docx markdown/*
$ ls BUILDS/
resume.docx resume.html resume.pdf resume.rtf resume.txt

Run exit at any time to exit the toolbox.

$ cd BUILDS/
$ pandoc --version || ls
pandoc 2.2.1
Compiled with pandoc-types 1.17.5.4, texmath 0.11.1.2, skylighting 0.7.5
...
for a particular purpose.
resume.docx resume.html resume.pdf resume.rtf resume.txt
$ exit logout
$ pandoc --version || ls
bash: pandoc: command not found...
resume.docx resume.html resume.pdf resume.rtf resume.txt

You retain the files created by your toolbox in your home directory. None of the programs installed in your toolbox will be available outside of it.

Tips and tricks

This introduction to toolbox only scratches the surface. Here are some additional tips, but you can also check out the official documentation.

  • Toolbox –help will show you the man page for Toolbox
  • You can have multiple toolboxes at once. Use toolbox create -c Toolboxname and toolbox enter -c Toolboxname
  • Toolbox uses Podman to do the heavy lifting. Use toolbox list to find the IDs of the containers Toolbox creates. Podman can use these IDs to perform actions such as rm and stop. (You can also read more about Podman in this Magazine article.)

Photo courtesy of Florian Richter from Flickr.

Posted on Leave a comment

Using SSH port forwarding on Fedora

You may already be familiar with using the ssh command to access a remote system. The protocol behind ssh allows terminal input and output to flow through a secure channel. But did you know that you can also use ssh to send and receive other data securely as well? One way is to use port forwarding, which allows you to connect network ports securely while conducting your ssh session. This article shows you how it works.

About ports

A standard Linux system has a set of network ports already assigned, from 0-65535. Your system reserves ports up to 1023 for system use. In many systems you can’t elect to use one of these low-numbered ports. Quite a few ports are commonly expected to run specific services. You can find these defined in your system’s /etc/services file.

You can think of a network port like a physical port or jack to which you can connect a cable. That port may connect to some sort of service on the system, like wiring behind that physical jack. An example is the Apache web server (also known as httpd). The web server usually claims port 80 on the host system for HTTP non-secure connections, and 443 for HTTPS secure connections.

When you connect to a remote system, such as with a web browser, you are also “wiring” your browser to a port on your host. This is usually a random high port number, such as 54001. The port on your host connects to the port on the remote host, such as 443 to reach its secure web server.

So why use port forwarding when you have so many ports available? Here are a couple common cases in the life of a web developer.

Local port forwarding

Imagine that you are doing web development on a remote system called remote.example.com. You usually reach this system via ssh but it’s behind a firewall that allows very little additional access, and blocks most other ports. To try out your web app, it’s helpful to be able to use your web browser to point to the remote system. But you can’t reach it via the normal method of typing the URL in your browser, thanks to that pesky firewall.

Local forwarding allows you to tunnel a port available via the remote system through your ssh connection. The port appears as a local port on your system (thus “local forwarding.”)

Let’s say your web app is running on port 8000 on the remote.example.com box. To locally forward that system’s port 8000 to your system’s port 8000, use the -L option with ssh when you start your session:

$ ssh -L 8000:localhost:8000 remote.example.com

Wait, why did we use localhost as the target for forwarding? It’s because from the perspective of remote.example.com, you’re asking the host to use its own port 8000. (Recall that any host usually can refer to itself as localhost to connect to itself via a network connection.) That port now connects to your system’s port 8000. Once the ssh session is ready, keep it open, and you can type http://localhost:8000 in your browser to see your web app. The traffic between systems now travels securely over an ssh tunnel!

If you have a sharp eye, you may have noticed something. What if we used a different hostname than localhost for the remote.example.com to forward? If it can reach a port on another system on its network, it usually can forward that port just as easily. For example, say you wanted to reach a MariaDB or MySQL service on the db.example.com box also on the remote network. This service typically runs on port 3306. So you could forward it with this command, even if you can’t ssh to the actual db.example.com host:

$ ssh -L 3306:db.example.com:3306 remote.example.com

Now you can run MariaDB commands against your localhost and you’re actually using the db.example.com box.

Remote port forwarding

Remote forwarding lets you do things the opposite way. Imagine you’re designing a web app for a friend at the office, and want to show them your work. Unfortunately, though, you’re working in a coffee shop, and because of the network setup, they can’t reach your laptop via a network connection. However, you both use the remote.example.com system at the office and you can still log in there. Your web app seems to be running well on port 5000 locally.

Remote port forwarding lets you tunnel a port from your local system through your ssh connection, and make it available on the remote system. Just use the -R option when you start your ssh session:

$ ssh -R 6000:localhost:5000 remote.example.com

Now when your friend inside the corporate firewall runs their browser, they can point it at http://remote.example.com:6000 and see your work. And as in the local port forwarding example, the communications travel securely over your ssh session.

By default the sshd daemon running on a host is set so that only that host can connect to its remote forwarded ports. Let’s say your friend wanted to be able to let people on other example.com corporate hosts see your work, and they weren’t on remote.example.com itself. You’d need the owner of the remote.example.com host to add one of these options to /etc/ssh/sshd_config on that box:

GatewayPorts yes # OR
GatewayPorts clientspecified

The first option means remote forwarded ports are available on all the network interfaces on remote.example.com. The second means that the client who sets up the tunnel gets to choose the address. This option is set to no by default.

With this option, you as the ssh client must still specify the interfaces on which the forwarded port on your side can be shared. Do this by adding a network specification before the local port. There are several ways to do this, including the following:

$ ssh -R *:6000:localhost:5000 # all networks
$ ssh -R 0.0.0.0:6000:localhost:5000 # all networks
$ ssh -R 192.168.1.15:6000:localhost:5000 # single network
$ ssh -R remote.example.com:6000:localhost:5000 # single network

Other notes

Notice that the port numbers need not be the same on local and remote systems. In fact, at times you may not even be able to use the same port. For instance, normal users may not to forward onto a system port in a default setup.

In addition, it’s possible to restrict forwarding on a host. This might be important to you if you need tighter security on a network-connected host. The PermitOpen option for the sshd daemon controls whether, and which, ports are available for TCP forwarding. The default setting is any, which allows all the examples above to work. To disallow any port fowarding, choose none, or choose only a specific host:port setting to permit. For more information, search for PermitOpen in the manual page for sshd daemon configuration:

$ man sshd_config

Finally, remember port forwarding only happens as long as the controlling ssh session is open. If you need to keep the forwarding active for a long period, try running the session in the background using the -N option. Make sure your console is locked to prevent tampering while you’re away from it.

Posted on Leave a comment

Make your Python code look good with Black on Fedora

The Python programing language is often praised for its simple syntax. In fact the language recognizes that code is read much more often than it is written. Black is a tool that automatically formats your Python source code making it uniform and compliant to the PEP-8 style guide.

How to install Black on Fedora

Installing Black on Fedora is quite simple. Black is maintained in the official repositories.

$ sudo dnf install python3-black

Black is a command line tool and therefore it is run from the terminal.

$ black --help

Format your Python code with Black

Using Black to format a Python code base is straight forward.

$ black myfile.py
All done! ✨ 🍰 ✨ 1 file left unchanged.
$ black path_to_my_python_project/
All done! ✨ 🍰 ✨
165 files reformatted, 24 files left unchanged.

By default Black allows 88 characters per line, meaning that the code will be reformatted to fit within 88 characters per line. It is possible to change this to a custom value, for example :

$ black --line-length 100 my_python_file.py

This will set the line length to allow 100 characters.

Run Black as part of a CI pipeline

Black really shines when it is integrated with other tools, like a continuous integration pipeline.

The –check option allows to verify if any files need to be reformatted. This is useful to run as a CI test to ensure all your code is formatted in consistent manner.

$ black --check myfile.py
would reformat myfile.py
All done! 💥 💔 💥
1 file would be reformatted.

Integrate Black with your code editor

Running Black during the continuous integration tests is a great way to keep the code base correctly formatted. But developers really wants to forget about formatting and have the tool managing it for them.

Most of the popular code editors support Black. It allows developers to run the format tool every time a file is saved. The official documentation details the configuration needed for each editor.

Black is a must-have tool in the Python developer toolbox and is easily available on Fedora.

Posted on Leave a comment

Fedora projects for Hacktoberfest

It’s October! That means its time for the annual Hacktoberfest presented by DigitalOcean and DEV. Hacktoberfest is a month-long event that encourages contributions to open source software projects. Participants who register and submit at least four pull requests to GitHub-hosted repositories during the month of October will receive a free t-shirt.

In a recent Fedora Magazine article, I listed some areas where would-be contributors could get started contributing to Fedora. In this article, I highlight some specific projects that provide an opportunity to help Fedora while you participate in Hacktoberfest.

Fedora infrastructure

  • Bodhi — When a package maintainer builds a new version of a software package to fix bugs or add new features, it doesn’t go out to users right away. First it spends time in the updates-testing repository where in can receive some real-world usage. Bodhi manages the flow of updates from the testing repository into the updates repository and provides a web interface for testers to provide feedback.
  • the-new-hotness — This project listens to release-monitoring.org (which is also on GitHub) and opens a Bugzilla issue when a new upstream release is published. This allows package maintainers to be quickly informed of new upstream releases.
  • koschei — koschei enables continuous integration for Fedora packages. It is software for running a service for scratch-rebuilding RPM packages in Koji instance when their build-dependencies change or after some time elapses.
  • MirrorManager2 — Distributing Fedora packages to a global user base requires a lot of bandwidth. Just like developing Fedora, distributing Fedora is a collaborative effort. MirrorManager2 tracks the hundreds of public and private mirrors and routes each user to the “best” one.
  • fedora-messaging — Actions within the Fedora community—from source code commits to participating in IRC meetings to…lots of things—generate messages that can be used to perform automated tasks or send notifications. fedora-messaging is the tool set that makes sending and receiving these messages possible.
  • fedocal — When is that meeting? Which IRC channel was it in again? Fedocal is the calendar system used by teams in the Fedora community to coordinate meetings. Not only is it a good Hacktoberfest project, it’s also looking for a new maintainer to adopt it.

In addition to the projects above, the Fedora Infrastructure team has highlighted good Hacktoberfest issues across all of their GitHub projects.

Community projects

  • bodhi-rs — This project provides Rust bindings for Bodhi.
  • koji-rs — Koji is the system used to build Fedora packages. Koji-rs provides bindings for Rust applications.
  • fedora-rs — This project provides a Rust library for interacting with Fedora services like other languages like Python have.
  • feedback-pipeline — One of the current Fedora Council objectives is minimization: work to reduce the installation and patching footprint of Fedora releases. feedback-pipeline is a tool developed by this team to generate reports of RPM sizes and dependencies.

And many more

The projects above are only a small sample focused on software used to build Fedora. Many Fedora packages have upstreams hosted on GitHub—too many to list here. The best place to start is with a project that’s important to you. Any contributions you make help improve the entire open source ecosystem. If you’re looking for something in particular, the Join Special Interest Group can help. Happy hacking!

Posted on Leave a comment

Fedora and CentOS Stream

From the desk of the Fedora Project Leader:

Hi everyone! You may have seen the announcement about changes over at the CentOS Project. (If not, please go ahead and take a few minutes and read it — I’ll wait!) And now you may be wondering: if CentOS is now upstream of RHEL, what happens to Fedora? Isn’t that Fedora’s role in the Red Hat ecosystem?

First, don’t worry. There are changes to the big picture, but they’re all for the better.

If you’ve been following the conference talks from Red Hat Enterprise Linux leadership about the relationship between Fedora, CentOS, and RHEL, you have heard about “the Penrose Triangle”. That’s a shape like something from an M. C. Escher drawing: it’s impossible in real life!

We’ve been thinking for a while that maybe impossible geometry is not actually the best model. 

For one thing, the imagined flow where contributions at the end would flow back into Fedora and grow in a “virtuous cycle” never actually worked that way. That’s a shame, because there’s a huge, awesome CentOS community and many great people working on it — and there’s a lot of overlap with the Fedora community too. We’re missing out.

But that gap isn’t the only one: there’s not really been a consistent flow between the projects and product at all. So far, the process has gone like this: 

  1. Some time after the previous RHEL release, Red Hat would suddenly turn more attention to Fedora than usual.
  2. A few months later, Red Hat would split off a new RHEL version, developed internally.
  3. After some months, that’d be put into the world, including all of the source — from which CentOS is built. 
  4. Source drops continue for updates, and sometimes those updates include patches that were in Fedora — but there’s no visible connection.

Each step here has its problems: intermittent attention, closed-door development, blind drops, and little ongoing transparency. But now Red Hat and CentOS Project are fixing that, and that’s good news for Fedora, too.

Fedora will remain the first upstream of RHEL. It’s where every RHEL came from, and is where RHEL 9 will come from, too. But after RHEL branches off, CentOS will be upstream for ongoing work on those RHEL versions. I like to call it “the midstream”, but the marketing folks somehow don’t, so that’s going to be called “CentOS Stream”.

We — Fedora, CentOS, and Red Hat — still need to work out all of the technical details, but the idea is that these branches will live in the same package source repository. (The current plan is to make a “src.centos.org” with a  parallel view of the same data as src.fedoraproject.org). This change gives public visibility into ongoing work on released RHEL, and a place for developers and Red Hat’s partners to collaborate at that level.

CentOS SIGs — the special interest groups for virtualization, storage, config management and so on — will do their work in shared space right next to Fedora branches. This will allow much easier collaboration and sharing between the projects, and I’m hoping we’ll even be able to merge some of our similar SIGs to work together directly. Fixes from Fedora packages can be cherry-picked into the CentOS “midstream” ones — and where useful, vice versa.

Ultimately, Fedora, CentOS, and RHEL are part of the same big project family. This new, more natural flow opens possibilities for collaboration which were locked behind artificial (and extra-dimensional!) barriers. I’m very excited for what we can now do together!

— Matthew Miller, Fedora Project Leader

Posted on Leave a comment

CodeReady Containers: complex solutions on OpenShift + Fedora

Want to experiment with (complex) solutions on OpenShift 4.1+? CodeReady Containers (CRC) on a physical Fedora server is a great choice. It lets you:

  • Configure the RAM available to CRC / OpenShift (this is key as we’ll deploy Machine Learning, Change Data Capture, Process Automation and other solutions with significant memory requirements)
  • Avoid installing anything on your laptop
  • Standardize (on Fedora 30) so that you get the same results every time

Start by installing CRC and Ansible Agnostic Deployer (AgnosticD) on a Fedora 30 physical server. Then, you’ll use AgnosticD to deploy Open Data Hub on the OpenShift 4.1 environment created by CRC. Let’s get started!

Set up CodeReady Containers

$ dnf config-manager --set-enabled fedora
$ su -c 'dnf -y install git wget tar qemu-kvm libvirt NetworkManager jq libselinux-python'
$ sudo systemctl enable --now libvirtd

Let’s also add a user.

$ sudo adduser demouser
$ sudo passwd demouser
$ sudo usermod -aG wheel demouser

Download and extract CodeReady Containers:

$ su demouser
$ cd /home/demouser
$ wget https://mirror.openshift.com/pub/openshift-v4/clients/crc/1.0.0-beta.3/crc-linux-amd64.tar.xz
$ tar -xvf crc-linux-amd64.tar.xz
$ cd crc-linux-1.0.0-beta.3-amd64/
$ sudo cp ./crc /usr/bin

Set the memory available to CRC according to what you have on your physical server. For example, on a physical server with around 100GB you can allocate 80G to CRC as follows:

$ crc config set memory 81920
$ crc setup

You’ll need your pull secret from https://cloud.redhat.com/openshift/install/metal/user-provisioned.

$ crc start

That’s it — you can now login to your OpenShift environment:

eval $(crc oc-env) && oc login -u kubeadmin -p <password> https://api.crc.testing:6443

Set up Ansible Agnostic Deployer

github.com/redhat-cop/agnosticd is a fully automated two-phase deployer. Let’s deploy it!

$ su demouser
$ cd /home/demouser
$ git clone https://github.com/redhat-cop/agnosticd.git
$ cd agnosticd/ansible
$ python -m pip install --upgrade --trusted-host files.pythonhosted.org -r requirements.txt
$ python3 -m pip install --upgrade --trusted-host files.pythonhosted.org -r requirements.txt
$ pip3 install kubernetes
$ pip3 install openshift
$ pip install kubernetes
$ pip install openshift

Set up Open Data Hub on Code Ready Containers

Open Data Hub is a machine-learning-as-a-service platform built on OpenShift and Kafka/Strimzi. It integrates a collection of open source projects.

First, create an Ansible inventory file with the following content.

$ cat inventory
$ 127.0.0.1 ansible_connection=local

Set up the WORKLOAD environment variable so that Ansible Agnostic Deployer knows that we want to deploy Open Data Hub.

$ export WORKLOAD="ocp4-workload-open-data-hub"
$ sudo cp /usr/local/bin/ansible-playbook /usr/bin/ansible-playbook

We are only deploying one Open Data Hub project, so set user_count to 1. You can set up workshops for many students by setting user_count.

An OpenShift project (with Open Data Hub in our case) will be created for each student.

$ eval $(crc oc-env) && oc login -u kubeadmin -p <password> https://api.crc.testing:6443
$ ansible-playbook -i inventory ./configs/ocp-workloads/ocp-workload.yml -e"ocp_workload=${WORKLOAD}" -e"ACTION=create" -e"user_count=1" -e"ocp_username=kubeadmin" -e"ansible_become_pass=<password>" -e"silent=False"
$ oc project open-data-hub-user1
$ oc get route
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
jupyterhub jupyterhub-open-data-hub-user1.apps-crc.testing jupyterhub 8080-tcp edge/Redirect None

On your laptop, add jupyterhub-open-data-hub-user1.apps-crc.testing to your /etc/hosts file. For example:

127.0.0.1 localhost fedora30 console-openshift-console.apps-crc.testing oauth-openshift.apps-crc.testing mapit-app-management.apps-crc.testing mapit-spring-pipeline-demo.apps-crc.testing jupyterhub-open-data-hub-user1.apps-crc.testing jupyterhub-open-data-hub-user1.apps-crc.testing

On your laptop:

$ sudo ssh marc@fedora30 -L 443:jupyterhub-open-data-hub-user1.apps-crc.testing:443

You can now browse to https://jupyterhub-open-data-hub-user1.apps-crc.testing.

Now that we have Open Data Hub ready, you could deploy something interesting on it. For example, you could deploy IBM’s Qiskit open source framework for quantum computing. For more information, refer to Video no. 9 at this YouTube playlist, and the Github repo here.

You could also deploy plenty of other useful tools for Process Automation, Change Data Capture, Camel Integration, and 3scale API Management. You don’t have to wait for articles on these, though. Step-by-step short videos are already available on YouTube.

The corresponding step-by-step instructions are also on YouTube. You can also follow along with this article using the GitHub repo.


Photo by Marta Markes on Unsplash.

Posted on Leave a comment

Announcing the release of Fedora 31 Beta

The Fedora Project is pleased to announce the immediate availability of Fedora 31 Beta, the next step towards our planned Fedora 31 release at the end of October.

Download the prerelease from our Get Fedora site:

<!–

Or, check out one of our popular variants, including KDE Plasma, Xfce, and other desktop environments, as well as images for ARM devices like the Raspberry Pi 2 and 3:

–>

Beta Release Highlights

GNOME 3.34 (almost)

The newest release of the GNOME desktop environment is full of performance enhancements and improvements. The beta ships with a prerelease, and the full 3.34 release will be available as an update. For a full list of GNOME 3.34 highlights, see the release notes.

Fedora IoT Edition

Fedora Editions address specific use-cases the Fedora Council has identified as significant in growing our userbase and community. We have Workstation, Server, and CoreOS — and now we’re adding Fedora IoT. This will be available from the main “Get Fedora” site when the final release of F31 is ready, but for now, get it from iot.fedoraproject.org.

Read more about Fedora IoT in our Getting Started docs.

Fedora CoreOS

Fedora CoreOS remains in a preview state, with a planned generally-available release planned for early next year. CoreOS is a rolling release which rebases periodically to a new underlying Fedora OS version. Right now, that version is Fedora 30, but soon there will be a “next” stream which will track Fedora 31 until that’s ready to become the “stable” stream.

Other updates

Fedora 31 Beta includes updated versions of many popular packages like Node.js, the Go language, Python, and Perl. We also have the customary updates to underlying infrastructure software, like the GNU C Library and the RPM package manager. For a full list, see the Change set on the Fedora Wiki.

Farewell to bootable i686

We’re no longer producing full media or repositories for 32-bit Intel-architecture systems. We recognize that this means newer Fedora releases will no longer work on some older hardware, but the fact is there just hasn’t been enough contributor interest in maintaining i686, and we can provide greater benefit for the majority of our users by focusing on modern architectures. (The majority of Fedora systems have been 64-bit x86_64 since 2013, and at this point that’s the vast majority.)

Please note that we’re still making userspace packages for compatibility when running 32-bit software on a 64-bit systems — we don’t see the need for that going away anytime soon.

Testing needed

Since this is a Beta release, we expect that you may encounter bugs or missing features. To report issues encountered during testing, contact the Fedora QA team via the mailing list or in #fedora-qa on Freenode. As testing progresses, common issues are tracked on the Common F31 Bugs page.

For tips on reporting a bug effectively, read how to file a bug.

What is the Beta Release?

A Beta release is code-complete and bears a very strong resemblance to the final release. If you take the time to download and try out the Beta, you can check and make sure the things that are important to you are working. Every bug you find and report doesn’t just help you, it improves the experience of millions of Fedora users worldwide! Together, we can make Fedora rock-solid. We have a culture of coordinating new features and pushing fixes upstream as much as we can. Your feedback improves not only Fedora, but Linux and free software as a whole.

More information

For more detailed information about what’s new on Fedora 31 Beta release, you can consult the Fedora 31 Change set. It contains more technical information about the new packages and improvements shipped with this release.

Posted on Leave a comment

GNOME 3.34 released — coming soon in Fedora 31

Today the GNOME project announced the release of GNOME 3.34. This latest release of GNOME will be the default desktop environment in Fedora 31 Workstation. The Beta release of Fedora 31 is currently expected in the next week or two, with the Final release scheduled for late October.

GNOME 3.34 includes a number of new features and improvements. Congratulations and thank you to the whole GNOME community for the work that went into this release! Read on for more details.

GNOME 3.34 desktop environment at work

Notable features

The desktop itself has been refreshed with a pleasing new background. You can also compare your background images to see what they’ll look like on the desktop.

There’s a new custom application folder feature in the GNOME Shell Overview. It lets you combine applications in a group to make it easier to find the apps you use.

You already know that Boxes lets you easily download an OS and create virtual machines for testing, development, or even daily use. Now you can find sources for your virtual machines more easily, as well as boot from CD or DVD (ISO) images more easily. There is also an Express Install feature available that now supports Windows versions.

Now that you can save states when using GNOME Games, gaming is more fun. You can snapshot your progress without getting in the way of the fun. You can even move snapshots to other devices running GNOME.

More details

These are not the only features of the new and improved GNOME 3.34. For an overview, visit the official release announcement. For even more details, check out the GNOME 3.34 release notes.

The Fedora 31 Workstation Beta release is right around the corner. Fedora 31 will feature GNOME 3.34 and you’ll be able to experience it in the Beta release.

Posted on Leave a comment

How RPM packages are made: the spec file

In the previous article on RPM package building, you saw that source RPMS include the source code of the software, along with a “spec” file. This post digs into the spec file, which contains instructions on how to build the RPM. Again, this article uses fpaste as an example.

Understanding the source code

Before you can start writing a spec file, you need to have some idea of the software that you’re looking to package. Here, you’re looking at fpaste, a very simple piece of software. It is written in Python, and is a one file script. When a new version is released, it’s provided here on Pagure: https://pagure.io/releases/fpaste/fpaste-0.3.9.2.tar.gz

The current version, as the archive shows, is 0.3.9.2. Download it so you can see what’s in the archive:

$ wget https://pagure.io/releases/fpaste/fpaste-0.3.9.2.tar.gz
$ tar -tvf fpaste-0.3.9.2.tar.gz
drwxrwxr-x root/root 0 2018-07-25 02:58 fpaste-0.3.9.2/
-rw-rw-r-- root/root 25 2018-07-25 02:58 fpaste-0.3.9.2/.gitignore
-rw-rw-r-- root/root 3672 2018-07-25 02:58 fpaste-0.3.9.2/CHANGELOG
-rw-rw-r-- root/root 35147 2018-07-25 02:58 fpaste-0.3.9.2/COPYING
-rw-rw-r-- root/root 444 2018-07-25 02:58 fpaste-0.3.9.2/Makefile
-rw-rw-r-- root/root 1656 2018-07-25 02:58 fpaste-0.3.9.2/README.rst
-rw-rw-r-- root/root 658 2018-07-25 02:58 fpaste-0.3.9.2/TODO
drwxrwxr-x root/root 0 2018-07-25 02:58 fpaste-0.3.9.2/docs/
drwxrwxr-x root/root 0 2018-07-25 02:58 fpaste-0.3.9.2/docs/man/
drwxrwxr-x root/root 0 2018-07-25 02:58 fpaste-0.3.9.2/docs/man/en/
-rw-rw-r-- root/root 3867 2018-07-25 02:58 fpaste-0.3.9.2/docs/man/en/fpaste.1
-rwxrwxr-x root/root 24884 2018-07-25 02:58 fpaste-0.3.9.2/fpaste
lrwxrwxrwx root/root 0 2018-07-25 02:58 fpaste-0.3.9.2/fpaste.py -> fpaste

The files you want to install are:

  • fpaste.py: which should go be installed to /usr/bin/.
  • docs/man/en/fpaste.1: the manual, which should go to /usr/share/man/man1/.
  • COPYING: the license text, which should go to /usr/share/license/fpaste/.
  • README.rst, TODO: miscellaneous documentation that goes to /usr/share/doc/fpaste.

Where these files are installed depends on the Filesystem Hierarchy Standard. To learn more about it, you can either read here: http://www.pathname.com/fhs/ or look at the man page on your Fedora system:

$ man hier

Part 1: What are we building?

Now that we know what files we have in the source, and where they are to go, let’s look at the spec file. You can see the full file here: https://src.fedoraproject.org/rpms/fpaste/blob/master/f/fpaste.spec

Here is the first part of the spec file:

Name: fpaste
Version: 0.3.9.2
Release: 3%{?dist}
Summary: A simple tool for pasting info onto sticky notes instances
BuildArch: noarch
License: GPLv3+
URL: https://pagure.io/fpaste
Source0: https://pagure.io/releases/fpaste/fpaste-0.3.9.2.tar.gz Requires: python3 %description
It is often useful to be able to easily paste text to the Fedora
Pastebin at http://paste.fedoraproject.org and this simple script
will do that and return the resulting URL so that people may
examine the output. This can hopefully help folks who are for
some reason stuck without X, working remotely, or any other
reason they may be unable to paste something into the pastebin

Name, Version, and so on are called tags, and are defined in RPM. This means you can’t just make up tags. RPM won’t understand them if you do! The tags to keep an eye out for are:

  • Source0: tells RPM where the source archive for this software is located.
  • Requires: lists run-time dependencies for the software. RPM can automatically detect quite a few of these, but in some cases they must be mentioned manually. A run-time dependency is a capability (often a package) that must be on the system for this package to function. This is how dnf detects whether it needs to pull in other packages when you install this package.
  • BuildRequires: lists the build-time dependencies for this software. These must generally be determined manually and added to the spec file.
  • BuildArch: the computer architectures that this software is being built for. If this tag is left out, the software will be built for all supported architectures. The value noarch means the software is architecture independent (like fpaste, which is written purely in Python).

This section provides general information about fpaste: what it is, which version is being made into an RPM, its license, and so on. If you have fpaste installed, and look at its metadata, you can see this information included in the RPM:

$ sudo dnf install fpaste
$ rpm -qi fpaste
Name : fpaste
Version : 0.3.9.2
Release : 2.fc30
...

RPM adds a few extra tags automatically that represent things that it knows.

At this point, we have the general information about the software that we’re building an RPM for. Next, we start telling RPM what to do.

Part 2: Preparing for the build

The next part of the spec is the preparation section, denoted by %prep:

%prep
%autosetup

For fpaste, the only command here is %autosetup. This simply extracts the tar archive into a new folder and keeps it ready for the next section where we build it. You can do more here, like apply patches, modify files for different purposes, and so on. If you did look at the contents of the source rpm for Python, you would have seen lots of patches there. These are all applied in this section.

Typically anything in a spec file with the % prefix is a macro or label that RPM interprets in a special way. Often these will appear with curly braces, such as %{example}.

Part 3: Building the software

The next section is where the software is built, denoted by “%build”. Now, since fpaste is a simple, pure Python script, it doesn’t need to be built. So, here we get:

%build
#nothing required

Generally, though, you’d have build commands here, like:

configure; make

The build section is often the hardest section of the spec, because this is where the software is being built from source. This requires you to know what build system the tool is using, which could be one of many: Autotools, CMake, Meson, Setuptools (for Python) and so on. Each has its own commands and style. You need to know these well enough to get the software to build correctly.

Part 4: Installing the files

Once the software is built, it needs to be installed in the %install section:

%install
mkdir -p %{buildroot}%{_bindir}
make install BINDIR=%{buildroot}%{_bindir} MANDIR=%{buildroot}%{_mandir}

RPM doesn’t tinker with your system files when building RPMs. It’s far too risky to add, remove, or modify files to a working installation. What if something breaks? So, instead RPM creates an artificial file system and works there. This is referred to as the buildroot. So, here in the buildroot, we create /usr/bin, represented by the macro %{_bindir}, and then install the files to it using the provided Makefile.

At this point, we have a built version of fpaste installed in our artificial buildroot.

Part 5: Listing all files to be included in the RPM

The last section of the spec file is the files section, %files. This is where we tell RPM what files to include in the archive it creates from this spec file. The fpaste file section is quite simple:

%files
%{_bindir}/%{name}
%doc README.rst TODO
%{_mandir}/man1/%{name}.1.gz
%license COPYING

Notice how, here, we do not specify the buildroot. All of these paths are relative to it. The %doc and %license commands simply do a little more—they create the required folders and remember that these files must go there.

RPM is quite smart. If you’ve installed files in the %install section, but not listed them, it’ll tell you this, for example.

Part 6: Document all changes in the change log

Fedora is a community based project. Lots of contributors maintain and co-maintain packages. So it is imperative that there’s no confusion about what changes have been made to a package. To ensure this, the spec file contains the last section, the Changelog, %changelog:

%changelog
* Thu Jul 25 2019 Fedora Release Engineering  - 0.3.9.2-3
- Rebuilt for https://fedoraproject.org/wiki/Fedora_31_Mass_Rebuild * Thu Jan 31 2019 Fedora Release Engineering  - 0.3.9.2-2
- Rebuilt for https://fedoraproject.org/wiki/Fedora_30_Mass_Rebuild * Tue Jul 24 2018 Ankur Sinha  - 0.3.9.2-1
- Update to 0.3.9.2 * Fri Jul 13 2018 Fedora Release Engineering  - 0.3.9.1-4
- Rebuilt for https://fedoraproject.org/wiki/Fedora_29_Mass_Rebuild * Wed Feb 07 2018 Fedora Release Engineering  - 0.3.9.1-3
- Rebuilt for https://fedoraproject.org/wiki/Fedora_28_Mass_Rebuild * Sun Sep 10 2017 Vasiliy N. Glazov  - 0.3.9.1-2
- Cleanup spec * Fri Sep 08 2017 Ankur Sinha  - 0.3.9.1-1
- Update to latest release
- fixes rhbz 1489605
...
....

There must be a changelog entry for every change to the spec file. As you see here, while I’ve updated the spec as the maintainer, others have too. Having the changes documented clearly helps everyone know what the current status of the spec is. For all packages installed on your system, you can use rpm to see their changelogs:

$ rpm -q --changelog fpaste

Building the RPM

Now we are ready to build the RPM. If you want to follow along and run the commands below, please ensure that you followed the steps in the previous post to set your system up for building RPMs.

We place the fpaste spec file in ~/rpmbuild/SPECS, the source code archive in ~/rpmbuild/SOURCES/ and can now create the source RPM:

$ cd ~/rpmbuild/SPECS
$ wget https://src.fedoraproject.org/rpms/fpaste/raw/master/f/fpaste.spec $ cd ~/rpmbuild/SOURCES
$ wget https://pagure.io/fpaste/archive/0.3.9.2/fpaste-0.3.9.2.tar.gz $ cd ~/rpmbuild/SOURCES
$ rpmbuild -bs fpaste.spec
Wrote: /home/asinha/rpmbuild/SRPMS/fpaste-0.3.9.2-3.fc30.src.rpm

Let’s have a look at the results:

$ ls ~/rpmbuild/SRPMS/fpaste*
/home/asinha/rpmbuild/SRPMS/fpaste-0.3.9.2-3.fc30.src.rpm $ rpm -qpl ~/rpmbuild/SRPMS/fpaste-0.3.9.2-3.fc30.src.rpm
fpaste-0.3.9.2.tar.gz
fpaste.spec

There we are — the source rpm has been built. Let’s build both the source and binary rpm together:

$ cd ~/rpmbuild/SPECS
$ rpmbuild -ba fpaste.spec
..
..
..

RPM will show you the complete build output, with details on what it is doing in each section that we saw before. This “build log” is extremely important. When builds do not go as expected, we packagers spend lots of time going through them, tracing the complete build path to see what went wrong.

That’s it really! Your ready-to-install RPMs are where they should be:

$ ls ~/rpmbuild/RPMS/noarch/
fpaste-0.3.9.2-3.fc30.noarch.rpm

Recap

We’ve covered the basics of how RPMs are built from a spec file. This is by no means an exhaustive document. In fact, it isn’t documentation at all, really. It only tries to explain how things work under the hood. Here’s a short recap:

  • RPMs are of two types: source and binary.
  • Binary RPMs contain the files to be installed to use the software.
  • Source RPMs contain the information needed to build the binary RPMs: the complete source code, and the instructions on how to build the RPM in the spec file.
  • The spec file has various sections, each with its own purpose.

Here, we’ve built RPMs locally, on our Fedora installations. While this is the basic process, the RPMs we get from repositories are built on dedicated servers with strict configurations and methods to ensure correctness and security. This Fedora packaging pipeline will be discussed in a future post.

Would you like to get started with building packages, and help the Fedora community maintain the massive amount of software we provide? You can start here by joining the package collection maintainers.

For any queries, post to the Fedora developers mailing list—we’re always happy to help!

References

Here are some useful references to building RPMs:


Posted on Leave a comment

Command line quick tips: Using pipes to connect tools

One of the most powerful concepts of Linux is carried on from its predecessor, UNIX. Your Fedora system has a bunch of useful, single-purpose utilities available for all sorts of simple operations. Like building blocks, you can attach them in creative and complex ways. Pipes are key to this concept.

Before you hear about pipes, though, it’s helpful to know the basic concept of input and output. Many utilities in your Fedora system can operate against files. But they can often take input not stored on a disk. You can think of input flowing freely into a process such as a utility as its standard input (also sometimes called stdin).

Similarly, a tool or process can display information to the screen by default. This is often because its default output is connected to the terminal. You can think of the free-flowing output of a process as its standard output (or stdout — go figure!).

Examples of standard input and output

Often when you run a tool, it outputs to the terminal. Take for instance this simple sequence command using the seq tool:

$ seq 1 6
1
2
3
4
5
6

The output, which is simply to count integers up from 1 to 6, one number per line, comes to the screen. But you could also send it to a file using the > character. The shell interpreter uses this character to mean “redirect standard output to a file whose name follows.” So as you can guess, this command puts the output into a file called six.txt:

$ seq 1 6 > six.txt

Notice nothing comes to the screen. You’ve sent the ouptut into a file instead. If you run the command cat six.txt you can verify that.

You probably remember the simple use of the grep command from a previous article. You could ask grep to search for a pattern in a file by simply declaring the file name. But that’s simply a convenience feature in grep. Technically it’s built to take standard input, and search that.

The shell uses the < character similarly to mean “redirect standard input from a file whose name follows.” So you could just as well search for the number 4 in the file six.txt this way:

$ grep 4 < six.txt
4

Of course the output here is, by default, the content of any line with a match. So grep finds the digit 4 in the file and outputs that line to standard output.

Introducing pipes

Now imagine: what if you took the standard output of one tool, and instead of sending it to the terminal, you sent it into another tool’s standard input? This is the essence of the pipe.

Your shell uses the vertical bar character | to represent a pipe between two commands. You can find it on most keyboard above the backslash \ character. It’s used like this:

$ command1 | command2

For most simple utilities, you wouldn’t use an output filename option on command1, nor an input file option on command2. (You might use other options, though.) Instead of using files, you’re sending the output of command1 directly into command2. You can use as many pipes in a row as needed, creating complex pipelines of several commands in a row.

This (relatively useless) example combines the commands above:

$ seq 1 6 | grep 4
4

What happened here? The seq command outputs the integers 1 through 6, one line at a time. The grep command processes that output line by line, searching for a match on the digit 4, and outputs any matching line.

Here’s a slightly more useful example. Let’s say you want to find out if TCP port 22, the ssh port, is open on your system. You could find this out using the ss command* by looking through its copious output. Or you could figure out its filter language and use that. Or you could use pipes. For example, pipe it through grep looking for the ssh port label:

$ ss -tl | grep ssh
LISTEN 0 128 0.0.0.0:ssh 0.0.0.0:* LISTEN 0 128 [::]:ssh [::]:*

* Those readers familiar with the venerable netstat command may note it is mostly obsolete, as stated in its man page.

That’s a lot easier than reading through many lines of output. And of course, you can combine redirectors and pipes, for instance:

$ ss -tl | grep ssh > ssh-listening.txt

This is barely scratching the surface of pipes. Let your imagination run wild. Have fun piping!