Managing RAID arrays with mdadm

Mdadm stands for Multiple Disk and Device Administration. It is a command line tool that can be used to manage software RAID arrays on your Linux PC. This article outlines the basics you need to get started with it.

The following five commands allow you to make use of mdadm’s most basic features:

  1. Create a RAID array:
    # mdadm –create /dev/md/test –homehost=any –metadata=1.0 –level=1 –raid-devices=2 /dev/sda1 /dev/sdb1
  2. Assemble (and start) a RAID array:
    # mdadm –assemble /dev/md/test /dev/sda1 /dev/sdb1
  3. Stop a RAID array:
    # mdadm –stop /dev/md/test
  4. Delete a RAID array:
    # mdadm –zero-superblock /dev/sda1 /dev/sdb1
  5. Check the status of all assembled RAID arrays:
    # cat /proc/mdstat

Notes on features

mdadm --create

The create command shown above includes the following four parameters in addition to the create parameter itself and the device names:

  1. –homehost:
    By default, mdadm stores your computer’s name as an attribute of the RAID array. If your computer name does not match the stored name, the array will not automatically assemble. This feature is useful in server clusters that share hard drives because file system corruption usually occurs if multiple servers attempt to access the same drive at the same time. The name any is reserved and disables the homehost restriction.
  2. –metadata:
    mdadm reserves a small portion of each RAID device to store information about the RAID array itself. The metadata parameter specifies the format and location of the information. The value 1.0 indicates to use version-1 formatting and store the metadata at the end of the device.
  3. –level:
    The level parameter specifies how the data should be distributed among the underlying devices. Level 1 indicates each device should contain a complete copy of all the data. This level is also known as disk mirroring.
  4. –raid-devices:
    The raid-devices parameter specifies the number of devices that will be used to create the RAID array.

By using level=1 (mirroring) in combination with metadata=1.0 (store the metadata at the end of the device), you create a RAID1 array whose underlying devices appear normal if accessed without the aid of the mdadm driver. This is useful in the case of disaster recovery, because you can access the device even if the new system doesn’t support mdadm arrays. It’s also useful in case a program needs read-only access to the underlying device before mdadm is available. For example, the UEFI firmware in a computer may need to read the bootloader from the ESP before mdadm is started.

mdadm --assemble

The assemble command above fails if a member device is missing or corrupt. To force the RAID array to assemble and start when one of its members is missing, use the following command:

# mdadm --assemble --run /dev/md/test /dev/sda1

Other important notes

Avoid writing directly to any devices that underlay a mdadm RAID1 array. That causes the devices to become out-of-sync and mdadm won’t know that they are out-of-sync. If you access a RAID1 array with a device that’s been modified out-of-band, you can cause file system corruption. If you modify a RAID1 device out-of-band and need to force the array to re-synchronize, delete the mdadm metadata from the device to be overwritten and then re-add it to the array as demonstrated below:

# mdadm --zero-superblock /dev/sdb1
# mdadm --assemble --run /dev/md/test /dev/sda1
# mdadm /dev/md/test --add /dev/sdb1

These commands completely overwrite the contents of sdb1 with the contents of sda1.

To specify any RAID arrays to automatically activate when your computer starts, create an /etc/mdadm.conf configuration file.

For the most up-to-date and detailed information, check the man pages:

$ man mdadm 
$ man mdadm.conf

The next article of this series will show a step-by-step guide on how to convert an existing single-disk Linux installation to a mirrored-disk installation, that will continue running even if one of its hard drives suddenly stops working!

Kubernetes on Fedora IoT with k3s

Fedora IoT is an upcoming Fedora edition targeted at the Internet of Things. It was introduced last year on Fedora Magazine in the article How to turn on an LED with Fedora IoT. Since then, it has continued to improve together with Fedora Silverblue to provide an immutable base operating system aimed at container-focused workflows.

Kubernetes is an immensely popular container orchestration system. It is perhaps most commonly used on powerful hardware handling huge workloads. However, it can also be used on lightweight devices such as the Raspberry Pi 3. Read on to find out how.

Why Kubernetes?

While Kubernetes is all the rage in the cloud, it may not be immediately obvious to run it on a small single board computer. But there are certainly reasons for doing it. First of all it is a great way to learn and get familiar with Kubernetes without the need for expensive hardware. Second, because of its popularity, there are tons of applications that comes pre-packaged for running in Kubernetes clusters. Not to mention the large community to provide help if you ever get stuck.

Last but not least, container orchestration may actually make things easier, even at the small scale in a home lab. This may not be apparent when tackling the the learning curve, but these skills will help when dealing with any cluster in the future. It doesn’t matter if it’s a single node Raspberry Pi cluster or a large scale machine learning farm.

K3s – a lightweight Kubernetes

A “normal” installation of Kubernetes (if such a thing can be said to exist) is a bit on the heavy side for IoT. The recommendation is a minimum of 2 GB RAM per machine! However, there are plenty of alternatives, and one of the newcomers is k3s – a lightweight Kubernetes distribution.

K3s is quite special in that it has replaced etcd with SQLite for its key-value storage needs. Another thing to note is that k3s ships as a single binary instead of one per component. This diminishes the memory footprint and simplifies the installation. Thanks to the above, k3s should be able to run k3s with just 512 MB of RAM, perfect for a small single board computer!

What you will need

  1. Fedora IoT in a virtual machine or on a physical device. See the excellent getting started guide here. One machine is enough but two will allow you to test adding more nodes to the cluster.
  2. Configure the firewall to allow traffic on ports 6443 and 8472. Or simply disable it for this experiment by running “systemctl stop firewalld”.

Install k3s

Installing k3s is very easy. Simply run the installation script:

curl -sfL | sh -

This will download, install and start up k3s. After installation, get a list of nodes from the server by running the following command:

kubectl get nodes

Note that there are several options that can be passed to the installation script through environment variables. These can be found in the documentation. And of course, there is nothing stopping you from installing k3s manually by downloading the binary directly.

While great for experimenting and learning, a single node cluster is not much of a cluster. Luckily, adding another node is no harder than setting up the first one. Just pass two environment variables to the installation script to make it find the first node and avoid running the server part of k3s

curl -sfL | K3S_URL=https://example-url:6443 \

The example-url above should be replaced by the IP address or fully qualified domain name of the first node. On that node the token (represented by XXX) is found in the file /var/lib/rancher/k3s/server/node-token.

Deploy some containers

Now that we have a Kubernetes cluster, what can we actually do with it? Let’s start by deploying a simple web server.

kubectl create deployment my-server --image nginx

This will create a Deployment named “my-server” from the container image “nginx” (defaulting to docker hub as registry and the latest tag). You can see the Pod created by running the following command.

kubectl get pods

In order to access the nginx server running in the pod, first expose the Deployment through a Service. The following command will create a Service with the same name as the deployment.

kubectl expose deployment my-server --port 80

The Service works as a kind of load balancer and DNS record for the Pods. For instance, when running a second Pod, we will be able to curl the nginx server just by specifying my-server (the name of the Service). See the example below for how to do this.

# Start a pod and run bash interactively in it
kubectl run debug --generator=run-pod/v1 --image=fedora -it -- bash
# Wait for the bash prompt to appear
curl my-server
# You should get the "Welcome to nginx!" page as output

Ingress controller and external IP

By default, a Service only get a ClusterIP (only accessible inside the cluster), but you can also request an external IP for the service by setting its type to LoadBalancer. However, not all applications require their own IP address. Instead, it is often possible to share one IP address among many services by routing requests based on the host header or path. You can accomplish this in Kubernetes with an Ingress, and this is what we will do. Ingresses also provide additional features such as TLS encryption of the traffic without having to modify your application.

Kubernetes needs an ingress controller to make the Ingress resources work and k3s includes Traefik for this purpose. It also includes a simple service load balancer that makes it possible to get an external IP for a Service in the cluster. The documentation describes the service like this:

k3s includes a basic service load balancer that uses available host ports. If you try to create a load balancer that listens on port 80, for example, it will try to find a free host in the cluster for port 80. If no port is available the load balancer will stay in Pending.


The ingress controller is already exposed with this load balancer service. You can find the IP address that it is using with the following command.

$ kubectl get svc --all-namespaces
default kubernetes ClusterIP 443/TCP 33d
default my-server ClusterIP 80/TCP 30m
kube-system kube-dns ClusterIP 53/UDP,53/TCP,9153/TCP 33d
kube-system traefik LoadBalancer 80:31596/TCP,443:31539/TCP 33d

Look for the Service named traefik. In the above example the IP we are interested in is

Route incoming requests

Let’s create an Ingress that routes requests to our web server based on the host header. This example uses to avoid having to set up DNS records. It works by including the IP adress as a subdomain, to use any subdomain of to reach the IP In other words, is used to reach the ingress controller in the cluster. You can try this right now (with your own IP instead of Without an ingress in place you should reach the “default backend” which is just a page showing “404 page not found”.

We can tell the ingress controller to route requests to our web server Service with the following Ingress.

apiVersion: extensions/v1beta1
kind: Ingress
name: my-server
- host:
- path: /
serviceName: my-server
servicePort: 80

Save the above snippet in a file named my-ingress.yaml and add it to the cluster by running this command:

kubectl apply -f my-ingress.yaml

You should now be able to reach the default nginx welcoming page on the fully qualified domain name you chose. In my example this would be The ingress controller is routing the requests based on the information in the Ingress. A request to will be routed to the Service and port defined as backend in the Ingress (my-server and 80 in this case).

What about IoT then?

Imagine the following scenario. You have dozens of devices spread out around your home or farm. It is a heterogeneous collection of IoT devices with various hardware capabilities, sensors and actuators. Maybe some of them have cameras, weather or light sensors. Others may be hooked up to control the ventilation, lights, blinds or blink LEDs.

In this scenario, you want to gather data from all the sensors, maybe process and analyze it before you finally use it to make decisions and control the actuators. In addition to this, you may want to visualize what’s going on by setting up a dashboard. So how can Kubernetes help us manage something like this? How can we make sure that Pods run on suitable devices?

The simple answer is labels. You can label the nodes according to capabilities, like this:

kubectl label nodes <node-name> <label-key>=<label-value>
# Example
kubectl label nodes node2 camera=available

Once they are labeled, it is easy to select suitable nodes for your workload with nodeSelectors. The final piece to the puzzle, if you want to run your Pods on all suitable nodes is to use DaemonSets instead of Deployments. In other words, create one DaemonSet for each data collecting application that uses some unique sensor and use nodeSelectors to make sure they only run on nodes with the proper hardware.

The service discovery feature that allows Pods to find each other simply by Service name makes it quite easy to handle these kinds of distributed systems. You don’t need to know or configure IP addresses or custom ports for the applications. Instead, they can easily find each other through named Services in the cluster.

Utilize spare resources

With the cluster up and running, collecting data and controlling your lights and climate control you may feel that you are finished. However, there are still plenty of compute resources in the cluster that could be used for other projects. This is where Kubernetes really shines.

You shouldn’t have to worry about where exactly those resources are or calculate if there is enough memory to fit an extra application here or there. This is exactly what orchestration solves! You can easily deploy more applications in the cluster and let Kubernetes figure out where (or if) they will fit.

Why not run your own NextCloud instance? Or maybe gitea? You could also set up a CI/CD pipeline for all those IoT containers. After all, why would you build and cross compile them on your main computer if you can do it natively in the cluster?

The point here is that Kubernetes makes it easier to make use of the “hidden” resources that you often end up with otherwise. Kubernetes handles scheduling of Pods in the cluster based on available resources and fault tolerance so that you don’t have to. However, in order to help Kubernetes make reasonable decisions you should definitely add resource requests to your workloads.


While Kubernetes, or container orchestration in general, may not usually be associated with IoT, it certainly makes a lot of sense to have an orchestrator when you are dealing with distributed systems. Not only does is allow you to handle a diverse and heterogeneous fleet of devices in a unified way, but it also simplifies communication between them. In addition, Kubernetes makes it easier to utilize spare resources.

Container technology made it possible to build applications that could “run anywhere”. Now Kubernetes makes it easier to manage the “anywhere” part. And as an immutable base to build it all on, we have Fedora IoT.


Joe Doss: How Do You Fedora?

We recently interviewed Joe Doss on how he uses Fedora. This is part of a series on the Fedora Magazine. The series profiles Fedora users and how they use Fedora to get things done. Contact us on the feedback form to express your interest in becoming a interviewee.

Who is Joe Doss?

Joe Doss lives in Chicago, Illinois USA and his favorite food is pizza. He is the Director of Engineering Operations and Kenna Security, Inc. Doss describes his employer this way: “Kenna uses data science to help enterprises combine their infrastructure and application vulnerability data with exploit intelligence to measure risk, predict attacks and prioritize remediation.”

His first Linux distribution was Red Hat Linux 5. A friend of his showed him a computer that wasn’t running Windows. Doss thought it was just a program to install on Windows when his friend gave him a Red Hat Linux 5 install disk. “I proceeded to install this Linux ‘program’ on my Father’s PC,” he says. Luckily for Doss, his father supported his interest in computers. “I ended up totally wiping out the Windows 95 install as a result and this was how I got my first computer.”

At Kenna, Doss’ group makes use of Fedora and Ansible: “We run Fedora Cloud in multiple VPC deployments in AWS and Google Compute with over 200 virtual machines. We use Ansible to automate everything we do with Fedora.”

Doss brews beer at home and contributes to open source in my free time. He also has a cat named Tibby. “I rescued Tibby off the street the Hyde Park neighborhood of Chicago when she was 7 months old. She is not very smart, but she makes up for that with cuteness.” His favorite place to visit is his childhood home of Michigan, but Doss says, “anywhere with a warm beach, a cool drink, and the ocean is pretty nice too.”

Tibby the cute cat!

The Fedora community

Doss became involved with Fedora and the Fedora community through his job at Kenna Security. When he first joined the company they were using Ubuntu and Chef in production. There was a desire to make the infrastructure more reproducible and reliable, and he says, “I was able to greenfield our deployments with Fedora Cloud and Ansible.” This project got him involved in the Fedora Cloud release.

When asked about his first impression of the Fedora community, Doss said, “Overwhelming to be honest. There is so much going on and it is hard to figure out who are the stakeholders of each part of Fedora.” Once he figured out who he needed to talk to he found the community very welcoming and super supportive.

One of the ideas he had to improve the community was to unite the various projects and team under on bug tracking tool and community resource. “Pagure, Bugzilla, Github, Fedora Forums, Discourse Forums, Mailing lists… it is all over the place and hard to navigate at first.” Despite the initial complexity of becoming familiar with the Fedora Project, Doss feels it is amazingly rewarding to be involved. “It feels awesome it to be apart of a Linux distro that impacts so many people in very positive ways. You can make a difference.”

Doss called out Dusty Mabe at Red Hat for helping him become involved, saying Dusty “has been an amazing mentor and resource for enabling me to contribute back to Fedora.”

Doss has an interesting way of explaining to non-technical friends what he does. “Imagine changing the tires on a very large bus while it is going down the highway at 70 MPH and sometimes you need to get involved with the tire manufacturer to help make this process work well.” This metaphor helps people understand what replacing 200-plus VMs across more than five production VPCs in AWS and Google Compute with every Fedora release.

Doss drew my attention to one specific incident with Fedora 29 and Vagrant. “Recently we encountered an issue where Vagrant wouldn’t set the hostname on a Fresh Fedora 29 Beta VM. This was due to Fedora 29 Cloud no longer shipping the network service stub in favor of NetworkManager. This led to me working with a colleague at Kenna Security to send a patch upstream to the Vagrant project to help their developers produce a fix for Fedora 29. Vagrant usage with Fedora is a very large part of our development cycle at Kenna, and having this broken before the Fedora 29 release would have impacted us a lot.” As Doss said, “Sometimes you need to help make the tires before they go on the bus.”

Doss is the COPR Fedora, RHEL, and CentOS package maintainer for WireGuard VPN. “The CentOS repo just went over 60 thousand downloads last month which is pretty awesome.”

What Hardware?

Doss uses Fedora 29 cloud in the over five VPC deployments in AWS and Google computer. At home he has a SuperMicro SYS-5019A-FTN4 1U Server that runs Fedora 29 Server with Openshift OKD installed on it. His laptops are all Lenovo. “For Laptops I use a ThinkPad T460s for work and a ThinkPad 25 at home. Both have Fedora 29 installed. ThinkPads are the best with Fedora.”

What Software?

Doss used GNOME 3 as his preferred desktop on Fedora Workstation. “I use Sublime Text 3 for my text editor on the desktop or vim on servers.” For development and testing he uses Vagrant. “Ansible is what I use for any kind of automation with Fedora. I maintain an Ansible playbook for setting up my workstation.”


I asked Doss if he had advice for people trying to learn Ansible.

“Start small. Automate the stuff that makes your life easier, but don’t over complicate it. Ansible Galaxy is a great resource to get things done quickly, but if you truly want to learn how to use Ansible, writing your own roles and playbooks the path I would take.

“I have helped a lot of my coworkers that have joined my Operations team at Kenna get up to speed on using Ansible by buying them a copy of Ansible for Devops by Jeff Geerling. This book will give anyone new to Ansible the foundation they need to start using it everyday. #ansible on Freenode is a great resource as well along with the official Ansible docs.”

Doss also said, “Knowing what to automate is most likely the most difficult thing to master without over complicating things. Debugging complex playbooks and roles is a close second.”

Home lab

He recommended setting up a home lab. “At Kenna and at home I use Vagrant with the Vagrant-libvirt plugin for developing Ansible roles and playbooks. You can iterate quickly to build your roles and playbooks on your laptop with your favorite editor and run vagrant provision to run your playbook. Quick feedback loop and the ability to burn down your Vagrant VM and start over quickly is an amazing workflow. Below is a sample Vagrant file that I keep handy to spin up a Fedora VM to test my playbooks.”

-- mode: ruby --
vi: set ft=ruby :
Vagrant.configure(2) do |config|
config.vm.provision "shell", inline: "dnf install nfs-utils rpcbind @development-tools @ansible-node redhat-rpm-config gcc-c++ -y"
config.ssh.forward_agent = true
config.vm.define "f29", autostart: false do |f29| = "fedora/29-cloud-base"
f29.vm.hostname = ""
f29.vm.provider "libvirt" do |vm|
vm.memory = 2048
vm.cpus = 2
vm.driver = "kvm"
vm.nic_model_type = "e1000"
config.vm.synced_folder '.', '/vagrant', disabled: true

config.vm.provision "ansible" do |ansible|
ansible.groups = {
ansible.playbook = "playbooks/main.yml"
ansible.inventory_path = "inventory/development"
ansible.extra_vars = {
ansible_python_interpreter: "/usr/bin/python3"
# ansible.verbose = 'vvv' end

Managing Partitions with sgdisk

Roderick W. Smith‘s sgdisk command can be used to manage the partitioning of your hard disk drive from the command line. The basics that you need to get started with it are demonstrated below.

The following six parameters are all that you need to know to make use of sgdisk’s most basic features:

  1. -p
    Print the partition table:
    # sgdisk -p /dev/sda
  2. -d x
    Delete partition x:
    # sgdisk -d 1 /dev/sda
  3. -n x:y:z
    Create a new partition numbered x, starting at y and ending at z:
    # sgdisk -n 1:1MiB:2MiB /dev/sda
  4. -c x:y
    Change the name of partition x to y:
    # sgdisk -c 1:grub /dev/sda
  5. -t x:y
    Change the type of partition x to y:
    # sgdisk -t 1:ef02 /dev/sda
  6. –list-types
    List the partition type codes:
    # sgdisk –list-types

The SGDisk Command

As you can see in the above examples, most of the commands require that the device file name of the hard disk drive to operate on be specified as the last parameter.

The parameters shown above can be combined so that you can completely define a partition with a single run of the sgdisk command:

# sgdisk -n 1:1MiB:2MiB -t 1:ef02 -c 1:grub /dev/sda

Relative values can be specified for some fields by prefixing the value with a + or symbol. If you use a relative value, sgdisk will do the math for you. For example, the above example could be written as:

# sgdisk -n 1:1MiB:+1MiB -t 1:ef02 -c 1:grub /dev/sda

The value 0 has a special-case meaning for several of the fields:

  • In the partition number field, 0 indicates that the next available number should be used (numbering starts at 1).
  • In the starting address field, 0 indicates that the start of the largest available block of free space should be used. Some space at the start of the hard drive is always reserved for the partition table itself.
  • In the ending address field, 0 indicates that the end of the largest available block of free space should be used.

By using 0 and relative values in the appropriate fields, you can create a series of partitions without having to pre-calculate any absolute values. For example, the following sequence of sgdisk commands would create all the basic partitions that are needed for a typical Linux installation if in run sequence against a blank hard drive:

# sgdisk -n 0:0:+1MiB -t 0:ef02 -c 0:grub /dev/sda
# sgdisk -n 0:0:+1GiB -t 0:ea00 -c 0:boot /dev/sda
# sgdisk -n 0:0:+4GiB -t 0:8200 -c 0:swap /dev/sda
# sgdisk -n 0:0:0 -t 0:8300 -c 0:root /dev/sda

The above example shows how to partition a hard disk for a BIOS-based computer. The grub partition is not needed on a UEFI-based computer. Because sgdisk is calculating all the absolute values for you in the above example, you can just skip running the first command on a UEFI-based computer and the remaining commands can be run without modification. Likewise, you could skip creating the swap partition and the remaining commands would not need to be modified.

There is also a short-cut for deleting all the partitions from a hard disk with a single command:

# sgdisk –zap-all /dev/sda

For the most up-to-date and detailed information, check the man page:

$ man sgdisk

InitRAMFS, Dracut, and the Dracut Emergency Shell

The Linux startup process goes through several stages before reaching the final graphical or multi-user target. The initramfs stage occurs just before the root file system is mounted. Dracut is a tool that is used to manage the initramfs. The dracut emergency shell is an interactive mode that can be initiated while the initramfs is loaded.

This article will show how to use the dracut command to modify the initramfs. Some basic troubleshooting commands that can be run from the dracut emergency shell will also be demonstrated.


Initramfs stands for Initial Random-Access Memory File System. On modern Linux systems, it is typically stored in a file under the /boot directory. The kernel version for which it was built will be included in the file name. A new initramfs is generated every time a new kernel is installed.

A Linux Boot Directory

By default, Fedora keeps the previous two versions of the kernel and its associated initramfs. This default can be changed by modifying the value of the installonly_limit setting the /etc/dnf/dnf.conf file.

You can use the lsinitrd command to list the contents of your initramfs archive:

The LsInitRD Command

The above screenshot shows that my initramfs archive contains the nouveau GPU driver. The modinfo command tells me that the nouveau driver supports several models of NVIDIA video cards. The lspci command shows that there is an NVIDIA GeForce video card in my computer’s PCI slot. There are also several basic Unix commands included in the archive such as cat and cp.

By default, the initramfs archive only includes the drivers that are needed for your specific computer. This allows the archive to be smaller and decreases the time that it takes for your computer to boot.

The Dracut Command

The dracut command can be used to modify the contents of your initramfs. For example, if you are going to move your hard drive to a new computer, you might want to temporarily include all drivers in the initramfs to be sure that the operating system can load on the new computer. To do so, you would run the following command:

# dracut –force –no-hostonly

The force parameter tells dracut that it is OK to overwrite the existing initramfs archive. The no-hostonly parameter overrides the default behavior of including only drivers that are germane to the currently-running computer and causes dracut to instead include all drivers in the initramfs.

By default dracut operates on the initramfs for the currently-running kernel. You can use the uname command to display which version of the Linux kernel you are currently running:

$ uname -r

Once you have your hard drive installed and running in your new computer, you can re-run the dracut command to regenerate the initramfs with only the drivers that are needed for the new computer:

# dracut –force

There are also parameters to add arbitrary drivers, dracut modules, and files to the initramfs archive. You can also create configuration files for dracut and save them under the /etc/dracut.conf.d directory so that your customizations will be automatically applied to all new initramfs archives that are generated when new kernels are installed. As always, check the man page for the details that are specific to the version of dracut you have installed on your computer:

$ man dracut

The Dracut Emergency Shell

The Dracut Emergency Shell

Sometimes something goes wrong during the initramfs stage of your computer’s boot process. When this happens, you will see “Entering emergency mode” printed to the screen followed by a shell prompt. This gives you a chance to try and fix things up manually and continue the boot process.

As a somewhat contrived example, let’s suppose that I accidentally deleted an important kernel parameter in my boot loader configuration:

# sed -i ‘s/\/root / /’ /boot/grub2/grub.cfg

The next time I reboot my computer, it will seem to hang for several minutes while it is trying to find the root partition and eventually give up and drop to an emergency shell.

From the emergency shell, I can enter journalctl and then use the Space key to page down though the startup logs. Near the end of the log I see a warning that reads “/dev/mapper/fedora-root does not exist”. I can then use the ls command to find out what does exist:

# ls /dev/mapper
control  fedora-swap

Hmm, the fedora-root LVM volume appears to be missing. Let’s see what I can find with the lvm command:

# lvm lvscan
  ACTIVE            ‘/dev/fedora/swap’ [3.85 GiB] inherit
  inactive          ‘/dev/fedora/home’ [22.85 GiB] inherit
  inactive          ‘/dev/fedora/root’ [46.80 GiB] inherit

Ah ha! There’s my root partition. It’s just inactive. All I need to do is activate it and exit the emergency shell to continue the boot process:

# lvm lvchange -a y fedora/root
# exit
The Fedora Login Screen

The above example only demonstrates the basic concept. You can check the troubleshooting section of the dracut guide for a few more examples.

It is possible to access the dracut emergency shell manually by adding the rd.break parameter to your kernel command line. This can be useful if you need to access your files before any system services have been started.

Check the dracut.kernel man page for details about what kernel options your version of dracut supports:

$ man dracut.kernel

Command line quick tips: Cutting content out of files

The Fedora distribution is a full featured operating system with an excellent graphical desktop environment. A user can point and click their way through just about any typical task easily. All of this wonderful ease of use masks the details of a powerful command line under the hood. This article is part of a series that shows you some common command line utilities. So let’s drop into the shell, and have a look at cut.

Often when you work in the command line, you are working with text files. Sometimes these files may be quite long. Reading them in their entirety, while feasible, can be time consuming and prone to errors. In this installment you’ll learn how to extract content from text files, and get the information you want from them.

It’s important to recognize that there are many ways to accomplish similar command line tasks in Fedora. The Fedora repositories include entire language systems for parsing and working with text, as an example. Also, there are multiple command line utilities available for just about any purpose conceivable in the shell. This article will only focus on using a few of those utility choices, to extract some information from a file and present it in a readable format.

Making the cut

To illustrate this example use a standard sizable file on the system like /etc/passwd. As seen in a prior article in this series, you can execute the cat command to view an entire file:

$ cat /etc/passwd

This file contains information on all accounts present on the system. It has a specific format:


Imagine that you want to simply have a list of all the account names on the system. If you could only cut out the name value from each line. This is where the cut command comes in handy! This command treats any input one line at a time, and extracts a specific part of the line.

The cut command provides options for selecting parts of a line differently, and in this example two of them are needed, -d which is an option to specify a delimiter type to use, and -f which is an option to specify which field of the line to cut. The -d option lets you declare the delimiter that separates values in a line. In this case a colon (:) is used to separate values. The -f option lets you choose which field value or values to extract. So for this example the command entered would be:

$ cut -d: -f1 /etc/passwd

That’s great, it worked! But you get the printout to the standard output, which in a terminal session at least means the screen. What if you needed the information for another task to be done later? It would be really nice if there was a way to put the output of the cut command into a text file to save it. There is an easy builtin shell function for such a task, the redirect function (>).

$ cut -d: -f1 /etc/passwd > names.txt

This will place the output of cut into a file called names.txt and you can check the contents with cat:

$ cat names.txt

With two commands and one shell function, it was easy to identify using cat, extract using cut, and redirect the extracted information from one file, saving it to another file for later use.

Photo by Joel Mbugua on Unsplash.


How to rebase to Fedora 30 Beta on Silverblue

Silverblue is an operating system for your desktop built on Fedora. It’s excellent for daily use, development, and container-based workflows. It offers numerous advantages such as being able to roll back in case of any problems. If you want to test Fedora 30 on your Silverblue system, this article tells you how. It not only shows you what to do, but also how to revert back if anything unforeseen happens.

Switching to Fedora 30 branch

Switching to Fedora 30 on Silverblue is easy. First, check if the 30 branch is available, which should be true now:

ostree remote refs fedora-workstation

You should see the following in the output:


Next, import the GPG key for the Fedora 30 branch. Without this step, you won’t be able to rebase.

sudo ostree remote gpg-import fedora-workstation -k /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-30-primary

Next, rebase your system to the Fedora 30 branch.

rpm-ostree rebase fedora-workstation:fedora/30/x86_64/silverblue

Finally, the last thing to do is restart your computer and boot to Fedora 30.

How to revert things back

Remember that Fedora 30’s still in beta testing phase, so there could still be some issues. If anything bad happens — for instance, if you can’t boot to Fedora 30 at all — it’s easy to go back. Just pick the previous entry in GRUB, and your system will start in its previous state before switching to Fedora 30. To make this change permanent, use the following command:

rpm-ostree rollback

That’s it. Now you know how to rebase to Fedora 30 and back. So why not test it today? 🙂

Announcing the release of Fedora 30 Beta

The Fedora Project is pleased to announce the immediate availability of Fedora 30 Beta, the next big step on our journey to the exciting Fedora 30 release.

Download the prerelease from our Get Fedora site:

Or, check out one of our popular variants, including KDE Plasma, Xfce, and other desktop environments, as well as images for ARM devices like the Raspberry Pi 2 and 3:

Beta Release Highlights

New desktop environment options

Fedora 30 Beta includes two new options for desktop environment. DeepinDE and Pantheon Desktop join GNOME, KDE Plasma, Xfce, and others as options for users to customize their Fedora experience.

DNF performance improvements

All dnf repository metadata for Fedora 30 Beta is compressed with the zchunk format in addition to xz or gzip. zchunk is a new compression format designed to allow for highly efficient deltas. When Fedora’s metadata is compressed using zchunk, dnf will download only the differences between any earlier copies of the metadata and the current version.

GNOME 3.32

Fedora 30 Workstation Beta includes GNOME 3.32, the latest version of the popular desktop environment. GNOME 3.32 features updated visual style, including the user interface, the icons, and the desktop itself. For a full list of GNOME 3.32 highlights, see the release notes.

Other updates

Fedora 30 Beta also includes updated versions of many popular packages like Golang, the Bash shell, the GNU C Library, Python, and Perl. For a full list, see the Change set on the Fedora Wiki. In addition, many Python 2 packages are removed in preparation for Python 2 end-of-life on 2020-01-01.

Testing needed

Since this is a Beta release, we expect that you may encounter bugs or missing features. To report issues encountered during testing, contact the Fedora QA team via the mailing list or in #fedora-qa on Freenode. As testing progresses, common issues are tracked on the Common F30 Bugs page.

For tips on reporting a bug effectively, read how to file a bug.

What is the Beta Release?

A Beta release is code-complete and bears a very strong resemblance to the final release. If you take the time to download and try out the Beta, you can check and make sure the things that are important to you are working. Every bug you find and report doesn’t just help you, it improves the experience of millions of Fedora users worldwide! Together, we can make Fedora rock-solid. We have a culture of coordinating new features and pushing fixes upstream as much as we can. Your feedback improves not only Fedora, but Linux and free software as a whole.

More information

For more detailed information about what’s new on Fedora 30 Beta release, you can consult the Fedora 30 Change set. It contains more technical information about the new packages and improvements shipped with this release.

3 cool text-based email clients

Writing and receiving email is a big part of everyone’s daily routine and choosing an email client is usually a major decision. The Fedora OS provides a large choice of email clients and among these are text-based email applications.


Mutt is probably one of the most popular text-based email clients. It supports all the common features that one would expect from an email client. Color coding, mail threading, POP3, and IMAP are all supported by Mutt. But one of its best features is it’s highly configurable. Indeed, the user can easily change the keybindings, and create macros to adapt the tool to a particular workflow.

To give Mutt a try, install it using sudo and dnf:

$ sudo dnf install mutt

To help newcomers get started, Mutt has a very comprehensive wiki full of macro examples and configuration tricks.


Alpine is also among the most popular text-based email clients. It’s more beginner friendly than Mutt, and you can configure most of Alpine via the application itself — no need to edit a configuration file. One powerful feature of Alpine is the ability to score emails. This is particularly interesting for users that are registered to a high volume mailing list like Fedora’s devel list. Using scores, Alpine can sort the email based on the user’s interests, showing emails with a high score first.

Alpine is also available to install from Fedora’s repository using dnf.

 $ sudo dnf install alpine

While using Alpine, you can easily access the documentation by pressing the Ctrl+G key combination.


nmh (new Mail Handling) follows the UNIX tools philosophy. It provides a collection of single purpose programs to send, receive, save, retrieve, and manipulate e-mail messages. This lets you swap the nmh command with other programs, or create scripts around nmh to create more customized tools. For example, you can use Mutt with nmh.

nmh can be easily installed using dnf.

$ sudo dnf install nmh

To learn more about nmh and mail handling in general you can read this GPL licenced book.

Setting kernel command line arguments with Fedora 30

Adding options to the kernel command line is a common task when debugging or experimenting with the kernel. The upcoming Fedora 30 release made a change to use Bootloader Spec (BLS). Depending on how you are used to modifying kernel command line options, your workflow may now change. Read on for more information.

To determine if your system is running with BLS or the older layout, look in the file


If you see


in there, you are running with the BLS setup and you may need to change how you set kernel command line arguments.

If you only want to modify a single kernel entry (for example, to temporarily work around a display problem) you can use a grubby command

$ grubby --update-kernel /boot/vmlinuz-5.0.1-300.fc30.x86_64 --args="amdgpu.dc=0"

To remove a kernel argument, you can use the


argument to grubby

$ grubby --update-kernel /boot/vmlinuz-5.0.1-300.fc30.x86_64 --remove-args="amdgpu.dc=0"

If there is an option that should be added to every kernel command line (for example, you always want to disable the use of the rdrand instruction for random number generation) you can run a grubby command:

$ grubby --update-kernel=ALL --args="nordrand"

This will update the command line of all kernel entries and save the option to the saved kernel command line for future entries.

If you later want to remove the option from all kernels, you can again use



$ grubby --update-kernel=ALL --remove-args="nordrand"