RPM – Sick Gaming

Posted on April 14, 2023 by — Leave a comment

Use rpmautospec in Fedora Linux

Use the %autorelease and %autochangelog tags simplify package maintenance and make it easier to contribute packages to the Fedora Project. These rpmautospec tags cause no noticeable difference in the packages from the end user’s perspective. As of the Fedora Linux 38 release, package maintainers should use these new tags.

Information about package history

Every package in a distribution carries identifying information. For example, the latest version of Firefox is available as firefox-110.0-3.fc38.x86_64. This can be unpacked as:

a name (firefox),
a version (110.0),
a “release tag”, consisting of a packaging release version + a distribution marker + an architecture tag (3.fc38.x86_64).

In modern practice, the name and the version are supplied directly by the project upstream and unambiguously identify what was built. The release tag describes the downstream distro build (where, which distro, build count). This may sound natural, but in the past packagers would split parts of the upstream version into the release tag according to some rather complicated rules.

A package also contains useful information in its changelog.

Continuing with the Firefox example:

$ rpm -q --changelog firefox-110.0-3.fc38.x86_64 | head -n5
* Tue Feb 14 2023 Martin Stránský <[email protected]>- 110.0-3
- Updated to 110.0 build 3 * Mon Feb 13 2023 Martin Stránský <[email protected]>- 110.0-2
- Added fix for orca

The changelog is created by package maintainers. It describes changes to the package that are relevant to a user. New software versions, modified file paths, and important bugfixs are examples of things that would be mentioned in the changelog. Whitespace changes in packaging scripts and other cleanups are examples of things that would not be mentioned in the changelog. When things go well, users generally do not look at the changelog. However, the changelog is useful when a bug is found and people need to track down what changed, when, and why.

All this changelog information must be provided by the maintainer. When the maintainer builds a rpm package, they must provide this information in the appropriate fields of the package’s spec file.

For example, the firefox.spec might look like this:

Name: firefox
Version: 110.0
Release: 3%{dist}
...
%changelog
* Tue Feb 14 2023 Martin Stránský <[email protected]>- 110.0-3
- Updated to 110.0 build 3 * Mon Feb 13 2023 Martin Stránský <[email protected]>- 110.0-2
- Added fix for orca
...

This is the traditional way. Every time the maintainer makes a new build, they update the number in the Release field and add a corresponding entry in the %changelog section. For example, for the 110.0-3.fc38 build, Martin would have changed Release: 2%{dist} → Release: 3%{dist} and added the first paragraph under %changelog.

Packages in Fedora are maintained using git. This means that after making changes to the package, and adding some text to the changelog, the maintainer would also write a description of the changes in the git commit message. Often this is exactly the same text as the changelog. For example, for the 110.0-3.fc38 build, Martin wrote “Updated to latest 110.0 upstream build” in the git commit message. It should be noted that every commit in git also contains a name, the email address of the author, and a timestamp for the change. If this sounds a bit repetitive, that’s because it is. Thankfully, some of this can now be automated.

rpmautospec

The rpmautospec method takes advantage of the fact that the spec file is maintained in a git repository:

Name: firefox
Version: 110.0
Release: %autorelease
...
%changelog
%autochangelog

The purpose of the Release field is to identify the distro build number for a specific upstream Version. The Release field should be set to %autorelease and never changed again. The %autorelease macro provides a count of commits since the last commit that changed the Version field. This is nifty. Every time the packager changes the spec file, they have to make a commit to “save” the changes and do a build and the number in %autorelease will be incremented. When the Version is changed, the autorelease number is reset to 1.

The purpose of the git commit message is to summarize changes to the contents of the repository. The purpose of the %changelog section is to summarize changes to the package. The %autochangelog macro takes a git commit message, the author name, and the commit timestamp, and formats them in a way that is suitable for the %changelog section.

If Martin were to do another build, let’s say with a patch added, he would adjust the spec file adding a Patch line, and create a commit with:

$ git commit -a -m 'Add patch to fix rhbz#1000002'

The %autorelease field would be automatically increased by one, and the %autochangelog text would now start with:

* Thu Mar 23 2023 Martin Stránský <[email protected]>- 110.0-4
- Add patch to fix rhbz#1000002

What are the effects of the new workflow?

It is easiest to consider the effect for the users: no change. %autorelease and %autochangelog get replaced by “real” content before the package is built, and the binary package downloaded by users looks exactly the same.

For the packager, there is less busywork. The Release field is constant and the git commit text is reused for the changelog. I’ve glossed over the details here, but the git commit text can contain parts which are not included in the changelog text. It can even have commits that are completely elided from the changelog.

This automation also makes some mistakes less likely. For example:

The maintainer makes changes, but forgets to bump Release, and the build fails because a previous build with the same version-release already exists.
The maintainer makes changes, but forgets to describe them in %changelog, and users don’t know what changed.
The maintainer makes a changelog entry, but writes Tue instead of Thu, and rpm complains about an invalid date.

On the other hand, the packager has to be more disciplined. The text in the git commit message ends up visible to users. So it must be formatted accordingly. Every commit in git bumps the number in the release tag. The changelog is now formatted in the same specific style in all packages. Arguably, those are not huge limitations, but some adjustment of packager habits is required.

Using rpmautospec has a positive effect for external contributors. In Fedora, anyone who wants to contribute a change to the package is encouraged to open a pull request.

Unfortunately, for changes that touch the spec file, with traditional Release and %changelog, we have a conundrum. If the contributor does not update those in their commit, the maintainer has to do this before the build, and effectively the contribution is incomplete. If the contributor does update those in their commit, and the pull request is not merged immediately, it is likely that by the time it is merged
the Release number will be out of date, the date in the %changelog will be in the past, and the spec file may even already have entries with later dates, and git will always show a merge conflict in the %changelog section.

With rpmautospec, all these problems go away. The release number is counted automatically. The date in the changelog is derived from the timestamp of when the patch was merged. And the changelog is generated from the stream of commits so there is no conflict to be had.

A specific variant of this contributor workflow occurs when the maintainer wants to copy (cherry-pick in git parlance) a commit to another branch. For example, because the important bugfix that was necessary in F38 also needs to be applied in F37, it is likely that, with rpmautospec, the commit can be applied without any changes to a different packaging branch.

Wrap-up

%autorelease and %autochangelog have been available for a while, but have now reached a level where they work nicely for common maintenance patterns and a great majority of packages; even if some some complicated corner cases are not yet supported. With Fedora 38, rpmautospec is now the recommended method. Hopefully, we will have happier maintainers and contributors with no negative changes noted by the users.

For more information, see: Changes/Rpmautospec_by_Default and Rpmautospec documentation. For an example package that uses rpmautospec, see python-sphinxcontrib-programoutput.

Posted on August 5, 2020 by — Leave a comment

Matthew Arnold: Why I switched to Fedora

To a veteran user of other distributions, Fedora can be a challenge. Many things are not where you expect them to be. The default LVM volume allocations are a bit tricky. And packages including the kernel are frequently upgraded. So why switch after years of using other distributions?

In my case, for a variety of technical and political reasons, Fedora was the best option if I wanted to continue using Linux as my daily driver. If you are making the transition from another distribution, here are some observations and tips to get you started.

Firm foundations

In Fedora you will find a community just as fiercely dedicated to its users and free software as Debian, as fanatical about polish and design as anyone in Ubuntu, and as passionate about learning and discovery as users of Arch or Slackware. Flowing under it all you will find a welcoming community dedicated to technical excellence. The form may change, but underneath all the trappings of systemd, dnf, rpm, and other differences, you will find a thriving healthy and growing community of people who have gathered together to make something awesome. Welcome to Fedora, and I hope you stay awhile.

The best way to get to know the Fedora community is to explore it for yourself. I hope a future article will highlight some of the more interesting aspects of Fedora for newcomers. Below are a few tips that I have put together to help you find your way around a new Fedora installation.

Install and explore

Installation proceeds as you would expect but be aware that you might want to adjust the LVM volume allocations in the install process or shortly afterwards or you might run low on space in a key place unexpectedly! Btrfs is also a supported option that is worth a look if you have lots of small disks.

Freedom matters

As stated above Fedora has a software freedom commitment similar in spirit to that of Debian. This means that you should be able to give Fedora to anyone, anywhere without violating intellectual property laws. Any software which is either not licensed in a way that Fedora finds acceptable or that bares US patent encumbrances can be found in the rpmfusion.org repository.

After the install your next concern is undoubtedly configuring things and installing new packages. Fedora’s command-line package manager is dnf. It works as you would expect.

Note also that since rpm uses file-based dependency tracking instead of package-based dependency tracking, as almost all others do, there are very few traditional metapackages. There are, however, package groups. To get a list of package groups, the command is:

$ dnf group list

To get a list of all installed packages on the system, the command is:

$ rpm -qa

All rpm commands are easily filterable using traditional Unix tools. So you should have no trouble adapting your workflow to the new environment. All the information gathered with the below commands can also be gathered through the dnf command. For information gathering, I prefer to use the rpm command because it presents information in a way that is easily parseable by commands like grep. But if you are making changes to the system, it is easier and safer to use dnf.

To get a package’s version, description, and other metainformation the command is:

$ rpm -qi <packagename>

To list the contents of an installed package the command is:

$ rpm -ql <packagename>

One way in which rpm is easier to use then dpkg or the slack package tools is that rpm stores change log information for each package in the package manager database itself so it is very easy to diagnose whether an update might have broken or changed something unexpectedly. This command is:

$ rpm -q --changes <packagname>

On the kernel

Perhaps one of the most exciting differences between Fedora and other projects, for newcomers at least, is Fedora’s policy on the kernel. Fedora’s policy is to align the distribution’s kernel package life cycle with the upstream mainline kernel life cycle. This means that every Fedora release will have multiple major kernel versions during its lifetime.

This offers several advantages for both users and developers. Primarily, Fedora users are among the first to receive all of the latest drivers, security fixes, new features, etc.

If you do not have an installation that uses out-of-tree modules or custom patches this should not be much of concern to you. However, if you rely on a kernel module like zfs, for example. Rebuilding the filesystem module every 2-3 months can get tedious and error prone after a while. This problem only compounds if you depend upon custom patches for your system to work correctly. There is good news and bad news on this issue.

The good news is that Fedora’s process for building a custom kernel is well documented

The bad news is, as with all things kernel related in all projects, going the custom route means you’re on your own in terms of support. The 2-3 month lifecycle means you’ll be building modules and kernels far more often then you are used to. This may be a deal breaker for some. But even this offers an advantage to the discerning or adventuress user. You will find that being encouraged to rebase your custom kernel setup every two to three months will give you far greater insight into what is going on upstream in mainline Linux and the various out of tree projects you rely on.

Conclusion

Hopefully these tips will get you started exploring and configuring your new Fedora system. Once you have done that. I urge you to explore the community. Like any other free software product of Fedora’s age and size, there are a plethora of communication channels available. You should read the code of conduct and then head over to the communication page on the wiki to get started. As with the distribution itself, for all the differences in culture you will find that much remains the same.

Posted on September 2, 2019 by — Leave a comment

How RPM packages are made: the spec file

In the previous article on RPM package building, you saw that source RPMS include the source code of the software, along with a “spec” file. This post digs into the spec file, which contains instructions on how to build the RPM. Again, this article uses fpaste as an example.

Understanding the source code

Before you can start writing a spec file, you need to have some idea of the software that you’re looking to package. Here, you’re looking at fpaste, a very simple piece of software. It is written in Python, and is a one file script. When a new version is released, it’s provided here on Pagure: https://pagure.io/releases/fpaste/fpaste-0.3.9.2.tar.gz

The current version, as the archive shows, is 0.3.9.2. Download it so you can see what’s in the archive:

$ wget https://pagure.io/releases/fpaste/fpaste-0.3.9.2.tar.gz
$ tar -tvf fpaste-0.3.9.2.tar.gz
drwxrwxr-x root/root 0 2018-07-25 02:58 fpaste-0.3.9.2/
-rw-rw-r-- root/root 25 2018-07-25 02:58 fpaste-0.3.9.2/.gitignore
-rw-rw-r-- root/root 3672 2018-07-25 02:58 fpaste-0.3.9.2/CHANGELOG
-rw-rw-r-- root/root 35147 2018-07-25 02:58 fpaste-0.3.9.2/COPYING
-rw-rw-r-- root/root 444 2018-07-25 02:58 fpaste-0.3.9.2/Makefile
-rw-rw-r-- root/root 1656 2018-07-25 02:58 fpaste-0.3.9.2/README.rst
-rw-rw-r-- root/root 658 2018-07-25 02:58 fpaste-0.3.9.2/TODO
drwxrwxr-x root/root 0 2018-07-25 02:58 fpaste-0.3.9.2/docs/
drwxrwxr-x root/root 0 2018-07-25 02:58 fpaste-0.3.9.2/docs/man/
drwxrwxr-x root/root 0 2018-07-25 02:58 fpaste-0.3.9.2/docs/man/en/
-rw-rw-r-- root/root 3867 2018-07-25 02:58 fpaste-0.3.9.2/docs/man/en/fpaste.1
-rwxrwxr-x root/root 24884 2018-07-25 02:58 fpaste-0.3.9.2/fpaste
lrwxrwxrwx root/root 0 2018-07-25 02:58 fpaste-0.3.9.2/fpaste.py -> fpaste

The files you want to install are:

fpaste.py: which should go be installed to /usr/bin/.
docs/man/en/fpaste.1: the manual, which should go to /usr/share/man/man1/.
COPYING: the license text, which should go to /usr/share/license/fpaste/.
README.rst, TODO: miscellaneous documentation that goes to /usr/share/doc/fpaste.

Where these files are installed depends on the Filesystem Hierarchy Standard. To learn more about it, you can either read here: http://www.pathname.com/fhs/ or look at the man page on your Fedora system:

$ man hier

Part 1: What are we building?

Now that we know what files we have in the source, and where they are to go, let’s look at the spec file. You can see the full file here: https://src.fedoraproject.org/rpms/fpaste/blob/master/f/fpaste.spec

Here is the first part of the spec file:

Name: fpaste
Version: 0.3.9.2
Release: 3%{?dist}
Summary: A simple tool for pasting info onto sticky notes instances
BuildArch: noarch
License: GPLv3+
URL: https://pagure.io/fpaste
Source0: https://pagure.io/releases/fpaste/fpaste-0.3.9.2.tar.gz Requires: python3 %description
It is often useful to be able to easily paste text to the Fedora
Pastebin at http://paste.fedoraproject.org and this simple script
will do that and return the resulting URL so that people may
examine the output. This can hopefully help folks who are for
some reason stuck without X, working remotely, or any other
reason they may be unable to paste something into the pastebin

Name, Version, and so on are called tags, and are defined in RPM. This means you can’t just make up tags. RPM won’t understand them if you do! The tags to keep an eye out for are:

Source0: tells RPM where the source archive for this software is located.
Requires: lists run-time dependencies for the software. RPM can automatically detect quite a few of these, but in some cases they must be mentioned manually. A run-time dependency is a capability (often a package) that must be on the system for this package to function. This is how dnf detects whether it needs to pull in other packages when you install this package.
BuildRequires: lists the build-time dependencies for this software. These must generally be determined manually and added to the spec file.
BuildArch: the computer architectures that this software is being built for. If this tag is left out, the software will be built for all supported architectures. The value noarch means the software is architecture independent (like fpaste, which is written purely in Python).

This section provides general information about fpaste: what it is, which version is being made into an RPM, its license, and so on. If you have fpaste installed, and look at its metadata, you can see this information included in the RPM:

$ sudo dnf install fpaste
$ rpm -qi fpaste
Name : fpaste
Version : 0.3.9.2
Release : 2.fc30
...

RPM adds a few extra tags automatically that represent things that it knows.

At this point, we have the general information about the software that we’re building an RPM for. Next, we start telling RPM what to do.

Part 2: Preparing for the build

The next part of the spec is the preparation section, denoted by %prep:

%prep
%autosetup

For fpaste, the only command here is %autosetup. This simply extracts the tar archive into a new folder and keeps it ready for the next section where we build it. You can do more here, like apply patches, modify files for different purposes, and so on. If you did look at the contents of the source rpm for Python, you would have seen lots of patches there. These are all applied in this section.

Typically anything in a spec file with the % prefix is a macro or label that RPM interprets in a special way. Often these will appear with curly braces, such as %{example}.

Part 3: Building the software

The next section is where the software is built, denoted by “%build”. Now, since fpaste is a simple, pure Python script, it doesn’t need to be built. So, here we get:

%build
#nothing required

Generally, though, you’d have build commands here, like:

configure; make

The build section is often the hardest section of the spec, because this is where the software is being built from source. This requires you to know what build system the tool is using, which could be one of many: Autotools, CMake, Meson, Setuptools (for Python) and so on. Each has its own commands and style. You need to know these well enough to get the software to build correctly.

Part 4: Installing the files

Once the software is built, it needs to be installed in the %install section:

%install
mkdir -p %{buildroot}%{_bindir}
make install BINDIR=%{buildroot}%{_bindir} MANDIR=%{buildroot}%{_mandir}

RPM doesn’t tinker with your system files when building RPMs. It’s far too risky to add, remove, or modify files to a working installation. What if something breaks? So, instead RPM creates an artificial file system and works there. This is referred to as the buildroot. So, here in the buildroot, we create /usr/bin, represented by the macro %{_bindir}, and then install the files to it using the provided Makefile.

At this point, we have a built version of fpaste installed in our artificial buildroot.

Part 5: Listing all files to be included in the RPM

The last section of the spec file is the files section, %files. This is where we tell RPM what files to include in the archive it creates from this spec file. The fpaste file section is quite simple:

%files
%{_bindir}/%{name}
%doc README.rst TODO
%{_mandir}/man1/%{name}.1.gz
%license COPYING

Notice how, here, we do not specify the buildroot. All of these paths are relative to it. The %doc and %license commands simply do a little more—they create the required folders and remember that these files must go there.

RPM is quite smart. If you’ve installed files in the %install section, but not listed them, it’ll tell you this, for example.

Part 6: Document all changes in the change log

Fedora is a community based project. Lots of contributors maintain and co-maintain packages. So it is imperative that there’s no confusion about what changes have been made to a package. To ensure this, the spec file contains the last section, the Changelog, %changelog:

%changelog
* Thu Jul 25 2019 Fedora Release Engineering  - 0.3.9.2-3
- Rebuilt for https://fedoraproject.org/wiki/Fedora_31_Mass_Rebuild * Thu Jan 31 2019 Fedora Release Engineering  - 0.3.9.2-2
- Rebuilt for https://fedoraproject.org/wiki/Fedora_30_Mass_Rebuild * Tue Jul 24 2018 Ankur Sinha  - 0.3.9.2-1
- Update to 0.3.9.2 * Fri Jul 13 2018 Fedora Release Engineering  - 0.3.9.1-4
- Rebuilt for https://fedoraproject.org/wiki/Fedora_29_Mass_Rebuild * Wed Feb 07 2018 Fedora Release Engineering  - 0.3.9.1-3
- Rebuilt for https://fedoraproject.org/wiki/Fedora_28_Mass_Rebuild * Sun Sep 10 2017 Vasiliy N. Glazov  - 0.3.9.1-2
- Cleanup spec * Fri Sep 08 2017 Ankur Sinha  - 0.3.9.1-1
- Update to latest release
- fixes rhbz 1489605
...
....

There must be a changelog entry for every change to the spec file. As you see here, while I’ve updated the spec as the maintainer, others have too. Having the changes documented clearly helps everyone know what the current status of the spec is. For all packages installed on your system, you can use rpm to see their changelogs:

$ rpm -q --changelog fpaste

Building the RPM

Now we are ready to build the RPM. If you want to follow along and run the commands below, please ensure that you followed the steps in the previous post to set your system up for building RPMs.

We place the fpaste spec file in ~/rpmbuild/SPECS, the source code archive in ~/rpmbuild/SOURCES/ and can now create the source RPM:

$ cd ~/rpmbuild/SPECS
$ wget https://src.fedoraproject.org/rpms/fpaste/raw/master/f/fpaste.spec $ cd ~/rpmbuild/SOURCES
$ wget https://pagure.io/fpaste/archive/0.3.9.2/fpaste-0.3.9.2.tar.gz $ cd ~/rpmbuild/SOURCES
$ rpmbuild -bs fpaste.spec
Wrote: /home/asinha/rpmbuild/SRPMS/fpaste-0.3.9.2-3.fc30.src.rpm

Let’s have a look at the results:

$ ls ~/rpmbuild/SRPMS/fpaste*
/home/asinha/rpmbuild/SRPMS/fpaste-0.3.9.2-3.fc30.src.rpm $ rpm -qpl ~/rpmbuild/SRPMS/fpaste-0.3.9.2-3.fc30.src.rpm
fpaste-0.3.9.2.tar.gz
fpaste.spec

There we are — the source rpm has been built. Let’s build both the source and binary rpm together:

$ cd ~/rpmbuild/SPECS
$ rpmbuild -ba fpaste.spec
..
..
..

RPM will show you the complete build output, with details on what it is doing in each section that we saw before. This “build log” is extremely important. When builds do not go as expected, we packagers spend lots of time going through them, tracing the complete build path to see what went wrong.

That’s it really! Your ready-to-install RPMs are where they should be:

$ ls ~/rpmbuild/RPMS/noarch/
fpaste-0.3.9.2-3.fc30.noarch.rpm

Recap

We’ve covered the basics of how RPMs are built from a spec file. This is by no means an exhaustive document. In fact, it isn’t documentation at all, really. It only tries to explain how things work under the hood. Here’s a short recap:

RPMs are of two types: source and binary.
Binary RPMs contain the files to be installed to use the software.
Source RPMs contain the information needed to build the binary RPMs: the complete source code, and the instructions on how to build the RPM in the spec file.
The spec file has various sections, each with its own purpose.

Here, we’ve built RPMs locally, on our Fedora installations. While this is the basic process, the RPMs we get from repositories are built on dedicated servers with strict configurations and methods to ensure correctness and security. This Fedora packaging pipeline will be discussed in a future post.

Would you like to get started with building packages, and help the Fedora community maintain the massive amount of software we provide? You can start here by joining the package collection maintainers.

For any queries, post to the Fedora developers mailing list—we’re always happy to help!

References

Here are some useful references to building RPMs:

Posted on August 26, 2019 by — Leave a comment

How RPM packages are made: the source RPM

In a previous post, we looked at what RPM packages are. They are archives that contain files and metadata. This metadata tells RPM where to create or remove files from when an RPM is installed or uninstalled. The metadata also contains information on “dependencies”, which you will remember from the previous post, can either be “runtime” or “build time”.

As an example, we will look at fpaste. You can download the RPM using dnf. This will download the latest version of fpaste that is available in the Fedora repositories. On Fedora 30, this is currently 0.3.9.2:

$ dnf download fpaste ...
fpaste-0.3.9.2-2.fc30.noarch.rpm

Since this is the built RPM, it contains only files needed to use fpaste:

$ rpm -qpl ./fpaste-0.3.9.2-2.fc30.noarch.rpm
/usr/bin/fpaste
/usr/share/doc/fpaste
/usr/share/doc/fpaste/README.rst
/usr/share/doc/fpaste/TODO
/usr/share/licenses/fpaste
/usr/share/licenses/fpaste/COPYING
/usr/share/man/man1/fpaste.1.gz

Source RPMs

The next link in the chain is the source RPM. All software in Fedora must be built from its source code. We do not include pre-built binaries. So, for an RPM file to be made, RPM (the tool) needs to be:

given the files that have to be installed,
told how to generate these files, if they are to be compiled, for example,
told where these files must be installed,
what other dependencies this particular software needs to work properly.

The source RPM holds all of this information. Source RPMs are similar archives to RPM, but as the name suggests, instead of holding the built binary files, they contain the source files for a piece of software. Let’s download the source RPM for fpaste:

$ dnf download fpaste --source
...
fpaste-0.3.9.2-2.fc30.src.rpm

Notice how the file ends with “src.rpm”. All RPMs are built from source RPMs. You can easily check what source RPM a “binary” RPM comes from using dnf too:

$ dnf repoquery --qf "%{SOURCERPM}" fpaste
fpaste-0.3.9.2-2.fc30.src.rpm

Also, since this is the source RPM, it does not contain built files. Instead, it contains the sources and instructions on how to build the RPM from them:

$ rpm -qpl ./fpaste-0.3.9.2-2.fc30.src.rpm
fpaste-0.3.9.2.tar.gz
fpaste.spec

Here, the first file is simply the source code for fpaste. The second is the “spec” file. The spec file is the recipe that tells RPM (the tool) how to create the RPM (the archive) using the sources contained in the source RPM—all the information that RPM (the tool) needs to build RPMs (the archives) are contained in spec files. When we package maintainers add software to Fedora, most of our time is spent writing and perfecting the individual spec files. When a software package needs an update, we go back and tweak the spec file. You can see the spec files for ALL packages in Fedora at our source repository at https://src.fedoraproject.org/browse/projects/

Note that one source RPM may contain the instructions to build multiple RPMs. fpaste is a very simple piece of software, where one source RPM generates one “binary” RPM. Python, on the other hand is more complex. While there is only one source RPM, it generates multiple binary RPMs:

$ sudo dnf repoquery --qf "%{SOURCERPM}" python3
python3-3.7.3-1.fc30.src.rpm
python3-3.7.4-1.fc30.src.rpm $ sudo dnf repoquery --qf "%{SOURCERPM}" python3-devel
python3-3.7.3-1.fc30.src.rpm
python3-3.7.4-1.fc30.src.rpm $ sudo dnf repoquery --qf "%{SOURCERPM}" python3-libs
python3-3.7.3-1.fc30.src.rpm
python3-3.7.4-1.fc30.src.rpm $ sudo dnf repoquery --qf "%{SOURCERPM}" python3-idle
python3-3.7.3-1.fc30.src.rpm
python3-3.7.4-1.fc30.src.rpm $ sudo dnf repoquery --qf "%{SOURCERPM}" python3-tkinter
python3-3.7.3-1.fc30.src.rpm
python3-3.7.4-1.fc30.src.rpm

In RPM jargon, “python3” is the “main package”, and so the spec file will be called “python3.spec”. All the other packages are “sub-packages”. You can download the source RPM for python3 and see what’s in it too. (Hint: patches are also part of the source code):

$ dnf download --source python3
python3-3.7.4-1.fc30.src.rpm $ rpm -qpl ./python3-3.7.4-1.fc30.src.rpm
00001-rpath.patch
00102-lib64.patch
00111-no-static-lib.patch
00155-avoid-ctypes-thunks.patch
00170-gc-assertions.patch
00178-dont-duplicate-flags-in-sysconfig.patch
00189-use-rpm-wheels.patch
00205-make-libpl-respect-lib64.patch
00251-change-user-install-location.patch
00274-fix-arch-names.patch
00316-mark-bdist_wininst-unsupported.patch
Python-3.7.4.tar.xz
check-pyc-timestamps.py
idle3.appdata.xml
idle3.desktop
python3.spec

Building an RPM from a source RPM

Now that we have the source RPM, and know what’s in it, we can rebuild our RPM from it. Before we do so, though, we should set our system up to build RPMs. First, we install the required tools:

$ sudo dnf install fedora-packager

This will install the rpmbuild tool. rpmbuild requires a default layout so that it knows where each required component of the source rpm is. Let’s see what they are:

# Where should the spec file go?
$ rpm -E %{_specdir}
/home/asinha/rpmbuild/SPECS # Where should the sources go?
$ rpm -E %{_sourcedir}
/home/asinha/rpmbuild/SOURCES # Where is temporary build directory?
$ rpm -E %{_builddir}
/home/asinha/rpmbuild/BUILD # Where is the buildroot?
$ rpm -E %{_buildrootdir}
/home/asinha/rpmbuild/BUILDROOT # Where will the source rpms be?
$ rpm -E %{_srcrpmdir}
/home/asinha/rpmbuild/SRPMS # Where will the built rpms be?
$ rpm -E %{_rpmdir}
/home/asinha/rpmbuild/RPMS

I have all of this set up on my system already:

$ cd
$ tree -L 1 rpmbuild/
rpmbuild/
├── BUILD
├── BUILDROOT
├── RPMS
├── SOURCES
├── SPECS
└── SRPMS 6 directories, 0 files

RPM provides a tool that sets it all up for you too:

$ rpmdev-setuptree

Then we ensure that we have all the build dependencies for fpaste installed:

sudo dnf builddep fpaste-0.3.9.2-3.fc30.src.rpm

For fpaste you only need Python and that must already be installed on your system (dnf uses Python too). The builddep command can also be given a spec file instead of an source RPM. Read more in the man page:

$ man dnf.plugin.builddep

Now that we have all that we need, building an RPM from a source RPM is as simple as:

$ rpmbuild --rebuild fpaste-0.3.9.2-3.fc30.src.rpm
..
.. $ tree ~/rpmbuild/RPMS/noarch/
/home/asinha/rpmbuild/RPMS/noarch/
└── fpaste-0.3.9.2-3.fc30.noarch.rpm 0 directories, 1 file

rpmbuild will install the source RPM and build your RPM from it. You can now install the RPM to use it as you do–using dnf. Of course, as said before, if you want to change anything in the RPM, you must modify the spec file—we’ll cover spec files in next post.

Summary

To summarise this post in two short points:

the RPMs we generally install to use software are “binary” RPMs that contain built versions of the software
these are built from source RPMs that include the source code and the spec file that are needed to generate the binary RPMs.

If you’d like to get started with building RPMs, and help the Fedora community maintain the massive amount of software we provide, you can start here: https://fedoraproject.org/wiki/Join_the_package_collection_maintainers

For any queries, post to the Fedora developers mailing list—we’re always happy to help!

Posted on June 27, 2019 by — Leave a comment

RPM packages explained

Perhaps the best known way the Fedora community pursues its mission of promoting free and open source software and content is by developing the Fedora software distribution. So it’s not a surprise at all that a very large proportion of our community resources are spent on this task. This post summarizes how this software is “packaged” and the underlying tools such as rpm that make it all possible.

RPM: the smallest unit of software

The editions and flavors (spins/labs/silverblue) that users get to choose from are all very similar. They’re all composed of various software that is mixed and matched to work well together. What differs between them is the exact list of tools that goes into each. That choice depends on the use case that they target. The basic unit of all of these is an RPM package file.

RPM files are archives that are similar to ZIP files or tarballs. In fact, they uses compression to reduce the size of the archive. However, along with files, RPM archives also contain metadata about the package. This can be queried using the rpm tool:

 $ rpm -q fpaste

fpaste-0.3.9.2-2.fc30.noarch



$ rpm -qi fpaste

Name        : fpaste

Version     : 0.3.9.2

Release     : 2.fc30

Architecture: noarch

Install Date: Tue 26 Mar 2019 08:49:10 GMT

Group       : Unspecified

Size        : 64144

License     : GPLv3+

Signature   : RSA/SHA256, Thu 07 Feb 2019 15:46:11 GMT, Key ID ef3c111fcfc659b9

Source RPM  : fpaste-0.3.9.2-2.fc30.src.rpm

Build Date  : Thu 31 Jan 2019 20:06:01 GMT

Build Host  : buildhw-07.phx2.fedoraproject.org

Relocations : (not relocatable)

Packager    : Fedora Project

Vendor      : Fedora Project

URL         : https://pagure.io/fpaste

Bug URL     : https://bugz.fedoraproject.org/fpaste

Summary     : A simple tool for pasting info onto sticky notes instances

Description :

It is often useful to be able to easily paste text to the Fedora

Pastebin at http://paste.fedoraproject.org and this simple script

will do that and return the resulting URL so that people may

examine the output. This can hopefully help folks who are for

some reason stuck without X, working remotely, or any other

reason they may be unable to paste something into the pastebin



$ rpm -ql fpaste

/usr/bin/fpaste

/usr/share/doc/fpaste

/usr/share/doc/fpaste/README.rst

/usr/share/doc/fpaste/TODO

/usr/share/licenses/fpaste

/usr/share/licenses/fpaste/COPYING

/usr/share/man/man1/fpaste.1.gz

When an RPM package is installed, the rpm tools know exactly what files were added to the system. So, removing a package also removes these files, and leaves the system in a consistent state. This is why installing software using rpm is preferred over installing software from source whenever possible.

Dependencies

Nowadays, it is quite rare for software to be completely self-contained. Even fpaste, a simple one file Python script, requires that the Python interpreter be installed. So, if the system does not have Python installed (highly unlikely, but possible), fpaste cannot be used. In packager jargon, we say that “Python is a run-time dependency of fpaste“.

When RPM packages are built (the process of building RPMs is not discussed in this post), the generated archive includes all of this metadata. That way, the tools interacting with the RPM package archive know what else must must be installed so that fpaste works correctly:

 $ rpm -q --requires fpaste

/usr/bin/python3

python3

rpmlib(CompressedFileNames) &lt;= 3.0.4-1

rpmlib(FileDigests) &lt;= 4.6.0-1

rpmlib(PayloadFilesHavePrefix) &lt;= 4.0-1

rpmlib(PayloadIsXz) &lt;= 5.2-1



$ rpm -q --provides fpaste

fpaste = 0.3.9.2-2.fc30



$ rpm -qi python3

Name        : python3

Version     : 3.7.3

Release     : 3.fc30

Architecture: x86_64

Install Date: Thu 16 May 2019 18:51:41 BST

Group       : Unspecified

Size        : 46139

License     : Python

Signature   : RSA/SHA256, Sat 11 May 2019 17:02:44 BST, Key ID ef3c111fcfc659b9

Source RPM  : python3-3.7.3-3.fc30.src.rpm

Build Date  : Sat 11 May 2019 01:47:35 BST

Build Host  : buildhw-05.phx2.fedoraproject.org

Relocations : (not relocatable)

Packager    : Fedora Project

Vendor      : Fedora Project

URL         : https://www.python.org/

Bug URL     : https://bugz.fedoraproject.org/python3

Summary     : Interpreter of the Python programming language

Description :

Python is an accessible, high-level, dynamically typed, interpreted programming

language, designed with an emphasis on code readability.

It includes an extensive standard library, and has a vast ecosystem of

third-party libraries.



The python3 package provides the "python3" executable: the reference

interpreter for the Python language, version 3.

The majority of its standard library is provided in the python3-libs package,

which should be installed automatically along with python3.

The remaining parts of the Python standard library are broken out into the

python3-tkinter and python3-test packages, which may need to be installed

separately.



Documentation for Python is provided in the python3-docs package.



Packages containing additional libraries for Python are generally named with

the "python3-" prefix.



$ rpm -q --provides python3

python(abi) = 3.7

python3 = 3.7.3-3.fc30

python3(x86-64) = 3.7.3-3.fc30

python3.7 = 3.7.3-3.fc30

python37 = 3.7.3-3.fc30

Resolving RPM dependencies

While rpm knows the required dependencies for each archive, it does not know where to find them. This is by design: rpm only works on local files and must be told exactly where they are. So, if you try to install a single RPM package, you get an error if rpm cannot find the package’s run-time dependencies. This example tries to install a package downloaded from the Fedora package set:

 $ ls

python3-elephant-0.6.2-3.fc30.noarch.rpm



$ rpm -qpi python3-elephant-0.6.2-3.fc30.noarch.rpm

Name        : python3-elephant

Version     : 0.6.2

Release     : 3.fc30

Architecture: noarch

Install Date: (not installed)

Group       : Unspecified

Size        : 2574456

License     : BSD

Signature   : (none)

Source RPM  : python-elephant-0.6.2-3.fc30.src.rpm

Build Date  : Fri 14 Jun 2019 17:23:48 BST

Build Host  : buildhw-02.phx2.fedoraproject.org

Relocations : (not relocatable)

Packager    : Fedora Project

Vendor      : Fedora Project

URL         : http://neuralensemble.org/elephant

Bug URL     : https://bugz.fedoraproject.org/python-elephant

Summary     : Elephant is a package for analysis of electrophysiology data in Python

Description :

Elephant - Electrophysiology Analysis Toolkit Elephant is a package for the

analysis of neurophysiology data, based on Neo.



$ rpm -qp --requires python3-elephant-0.6.2-3.fc30.noarch.rpm

python(abi) = 3.7

python3.7dist(neo) >= 0.7.1

python3.7dist(numpy) >= 1.8.2

python3.7dist(quantities) >= 0.10.1

python3.7dist(scipy) >= 0.14.0

python3.7dist(six) >= 1.10.0

rpmlib(CompressedFileNames) &lt;= 3.0.4-1

rpmlib(FileDigests) &lt;= 4.6.0-1

rpmlib(PartialHardlinkSets) &lt;= 4.0.4-1

rpmlib(PayloadFilesHavePrefix) &lt;= 4.0-1

rpmlib(PayloadIsXz) &lt;= 5.2-1



$ sudo rpm -i ./python3-elephant-0.6.2-3.fc30.noarch.rpm

error: Failed dependencies:

        python3.7dist(neo) >= 0.7.1 is needed by python3-elephant-0.6.2-3.fc30.noarch

        python3.7dist(quantities) >= 0.10.1 is needed by python3-elephant-0.6.2-3.fc30.noarch

In theory, one could download all the packages that are required for python3-elephant, and tell rpm where they all are, but that isn’t convenient. What if python3-neo and python3-quantities have other run-time requirements and so on? Very quickly, the dependency chain can get quite complicated.

Repositories

Luckily, dnf and friends exist to help with this issue. Unlike rpm, dnf is aware of repositories. Repositories are collections of packages, with metadata that tells dnf what these repositories contain. All Fedora systems come with the default Fedora repositories enabled by default:

 $ sudo dnf repolist

repo id              repo name                             status

fedora               Fedora 30 - x86_64                    56,582

fedora-modular       Fedora Modular 30 - x86_64               135

updates              Fedora 30 - x86_64 - Updates           8,573

updates-modular      Fedora Modular 30 - x86_64 - Updates     138

updates-testing      Fedora 30 - x86_64 - Test Updates      8,458

There’s more information on these repositories, and how they can be managed on the Fedora quick docs.

dnf can be used to query repositories for information on the packages they contain. It can also search them for software, or install/uninstall/upgrade packages from them:

 $ sudo dnf search elephant

Last metadata expiration check: 0:05:21 ago on Sun 23 Jun 2019 14:33:38 BST.

============================================================================== Name &amp; Summary Matched: elephant ==============================================================================

python3-elephant.noarch : Elephant is a package for analysis of electrophysiology data in Python

python3-elephant.noarch : Elephant is a package for analysis of electrophysiology data in Python



$ sudo dnf list \*elephant\*

Last metadata expiration check: 0:05:26 ago on Sun 23 Jun 2019 14:33:38 BST.

Available Packages

python3-elephant.noarch      0.6.2-3.fc30      updates-testing

python3-elephant.noarch      0.6.2-3.fc30              updates

Installing dependencies

When installing the package using dnf now, it resolves all the required dependencies, then calls rpm to carry out the transaction:

 $ sudo dnf install python3-elephant

Last metadata expiration check: 0:06:17 ago on Sun 23 Jun 2019 14:33:38 BST.

Dependencies resolved.

==============================================================================================================================================================================================

 Package                                      Architecture                     Version                                                        Repository                                 Size

==============================================================================================================================================================================================

Installing:

 python3-elephant                             noarch                           0.6.2-3.fc30                                                   updates-testing                           456 k

Installing dependencies:

 python3-neo                                  noarch                           0.8.0-0.1.20190215git49b6041.fc30                              fedora                                    753 k

 python3-quantities                           noarch                           0.12.2-4.fc30                                                  fedora                                    163 k

Installing weak dependencies:

 python3-igor                                 noarch                           0.3-5.20150408git2c2a79d.fc30                                  fedora                                     63 k



Transaction Summary

==============================================================================================================================================================================================

Install  4 Packages



Total download size: 1.4 M

Installed size: 7.0 M

Is this ok [y/N]: y

Downloading Packages:

(1/4): python3-igor-0.3-5.20150408git2c2a79d.fc30.noarch.rpm                                                                                                  222 kB/s |  63 kB     00:00

(2/4): python3-elephant-0.6.2-3.fc30.noarch.rpm                                                                                                               681 kB/s | 456 kB     00:00

(3/4): python3-quantities-0.12.2-4.fc30.noarch.rpm                                                                                                            421 kB/s | 163 kB     00:00

(4/4): python3-neo-0.8.0-0.1.20190215git49b6041.fc30.noarch.rpm                                                                                               840 kB/s | 753 kB     00:00

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Total                                                                                                                                                         884 kB/s | 1.4 MB     00:01

Running transaction check

Transaction check succeeded.

Running transaction test

Transaction test succeeded.

Running transaction

  Preparing        :                                                                                                                                                                      1/1

  Installing       : python3-quantities-0.12.2-4.fc30.noarch                                                                                                                              1/4

  Installing       : python3-igor-0.3-5.20150408git2c2a79d.fc30.noarch                                                                                                                    2/4

  Installing       : python3-neo-0.8.0-0.1.20190215git49b6041.fc30.noarch                                                                                                                 3/4

  Installing       : python3-elephant-0.6.2-3.fc30.noarch                                                                                                                                 4/4

  Running scriptlet: python3-elephant-0.6.2-3.fc30.noarch                                                                                                                                 4/4

  Verifying        : python3-elephant-0.6.2-3.fc30.noarch                                                                                                                                 1/4

  Verifying        : python3-igor-0.3-5.20150408git2c2a79d.fc30.noarch                                                                                                                    2/4

  Verifying        : python3-neo-0.8.0-0.1.20190215git49b6041.fc30.noarch                                                                                                                 3/4

  Verifying        : python3-quantities-0.12.2-4.fc30.noarch                                                                                                                              4/4



Installed:

  python3-elephant-0.6.2-3.fc30.noarch   python3-igor-0.3-5.20150408git2c2a79d.fc30.noarch   python3-neo-0.8.0-0.1.20190215git49b6041.fc30.noarch   python3-quantities-0.12.2-4.fc30.noarch



Complete!

Notice how dnf even installed python3-igor, which isn’t a direct dependency of python3-elephant.

DnfDragora: a graphical interface to DNF

While technical users may find dnf straightforward to use, it isn’t for everyone. Dnfdragora addresses this issue by providing a graphical front end to dnf.

dnfdragora (version 1.1.1-2 on Fedora 30) listing all the packages installed on a system.

From a quick look, dnfdragora appears to provide all of dnf‘s main functions.

There are other tools in Fedora that also manage packages. GNOME Software, and Discover are two examples. GNOME Software is focused on graphical applications only. You can’t use the graphical front end to install command line or terminal tools such as htop or weechat. However, GNOME Software does support the installation of Flatpaks and Snap applications which dnf does not. So, they are different tools with different target audiences, and so provide different functions.

This post only touches the tip of the iceberg that is the life cycle of software in Fedora. This article explained what RPM packages are, and the main differences between using rpm and using dnf.

In future posts, we’ll speak more about:

The processes that are needed to create these packages
How the community tests them to ensure that they are built correctly
The infrastructure that the community uses to get them to community users in future posts.