Login

Trace code in Fedora with bpftrace

<div>bpftrace is <a href="https://github.com/iovisor/bpftrace">a new eBPF-based tracing tool</a> that was first included in Fedora 28. It was developed by Brendan Gregg, Alastair Robertson and Matheus Marchini with the help of a loosely-knit team of hackers across the Net. A tracing tool lets you analyze what a system is doing behind the curtain. It tells you which functions in code are being called, with which arguments, how many times, and so on.
This article covers some basics about bpftrace, and how it works. Read on for more information and some useful examples.
 
<h4>eBPF (extended Berkeley Packet Filter)</h4>
<a href="https://lwn.net/Articles/740157/">eBPF</a> is a tiny virtual machine, or a virtual CPU to be more precise, in the Linux Kernel. The eBPF can load and run small programs in a safe and controlled way in kernel space. This makes it safer to use, even in production systems. This virtual machine has its own instruction set architecture (<a href="https://github.com/iovisor/bpf-docs/blob/master/eBPF.md">ISA</a>) resembling a subset of modern processor architectures. The ISA makes it easy to translate those programs to the real hardware. The kernel performs just-in-time translation to native code for main architectures to improve the performance. 
The eBPF virtual machine allows the kernel to be extended programmatically. Nowadays several kernel subsystems take advantage of this new powerful Linux Kernel capability. Examples include networking, seccomp, tracing, and more. The main idea is to attach eBPF programs into specific code points, and thereby extend the original kernel behavior.
eBPF machine language is very powerful. But writing code directly in it is extremely painful, because it’s a low level language. This is where bpftrace comes in. It provides a high-level language to write eBPF tracing scripts. The tool then translates these scripts to eBPF with the help of clang/LLVM libraries, and then attached to the specified code points.
<h2>Installation and quick start</h2>
To install bpftrace, run the following command in a terminal <a href="https://fedoramagazine.org/howto-use-sudo/">using </a><a href="https://fedoramagazine.org/howto-use-sudo/">sudo</a>:
<pre class="wp-block-preformatted">$ sudo dnf install bpftrace</pre>
Try it out with a “hello world” example:
<pre class="wp-block-preformatted">$ sudo bpftrace -e 'BEGIN { printf("hello world\n"); }'</pre>
Note that you must run bpftrace as root due to the privileges required. Use the -e option to specify a program, and to construct the so-called “one-liners.” This example only prints hello world, and then waits for you to press Ctrl+C.
BEGIN is a special probe name that fires only once at the beginning of execution. Every action inside the curly braces { } fires whenever the probe is hit — in this case, it’s just a printf.
Let’s jump now to a more useful example:
<pre class="wp-block-preformatted">$ sudo bpftrace -e 't Confused

yscalls

ys_enter_execve { printf("%s called %s\n", comm, str(args->filename)); }'</pre>
This example prints the parent process name (comm) and the name of every new process being created in the system. t Confused

yscalls

ys_enter_execve is a kernel tracepoint. It’s a shorthand for tracepoint Confused

yscalls

ys_enter_execve, but both forms can be used. The next section shows you how to list all available tracepoints.
comm is a bpftrace builtin that represents the process name. filename is a field of the t Confused

yscalls

ys_enter_execve tracepoint. You can access these fields through the args builtin.
All available fields of the tracepoint can be listed with this command:
<pre class="wp-block-preformatted">bpftrace -lv "t Confused

yscalls

ys_enter_execve"</pre>
<h2>Example usage</h2>
<h3>Listing probes</h3>
A central concept for bpftrace are probe points. Probe points are instrumentation points in code (kernel or userspace) where eBPF programs can be attached. They fit into the following categories:
<ul>
<li>kprobe – kernel function start</li>
<li>kretprobe – kernel function return</li>
<li>uprobe – user-level function start</li>
<li>uretprobe – user-level function return</li>
<li>tracepoint – kernel static tracepoints</li>
<li>usdt – user-level static tracepoints</li>
<li>profile – timed sampling</li>
<li>interval – timed output</li>
<li>software – kernel software events</li>
<li>hardware – processor-level events</li>
</ul>
All available kprobe/kretprobe, tracepoints, software and hardware probes can be listed with this command:
<pre class="wp-block-preformatted">$ sudo bpftrace -l</pre>
The uprobe/uretprobe and usdt probes are userspace probes specific to a given executable. To use them, use the special syntax shown later in this article.
The profile and interval probes fire at fixed time intervals. Fixed time intervals are not covered in this article.
<h3>Counting system calls</h3>
Maps are special BPF data types that store counts, statistics, and histograms. You can use maps to summarize how many times each syscall is being called:
<pre class="wp-block-preformatted">$ sudo bpftrace -e 't Confused

yscalls

ys_enter_* { @[probe] = count(); }'</pre>
Some probe types allow wildcards to match multiple probes. You can also specify multiple attach points for an action block using a comma separated list. In this example, the action block attaches to all tracepoints whose name starts with t Confused

yscalls

ys_enter_, which means all available syscalls.
The bpftrace builtin function count() counts the number of times this function is called. @[] represents a map (an associative array). The key of this map is probe, which is another bpftrace builtin that represents the full probe name.
Here, the same action block is attached to every syscall. Then, each time a syscall is called the map will be updated, and the entry is incremented in the map relative to this same syscall. When the program terminates, it automatically prints out all declared maps.
This example counts the syscalls called globally, it’s also possible to filter for a specific process by PID using the bpftrace filter syntax:
<pre class="wp-block-preformatted">$ sudo bpftrace -e 't Confused

yscalls

ys_enter_* / pid == 1234 / { @[probe] = count(); }'</pre>
<h3>Write bytes by process</h3>
Using these concepts, let’s analyze how many bytes each process is writing:
<pre class="wp-block-preformatted">$ sudo bpftrace -e 't Confused

yscalls

ys_exit_write /args->ret > 0/ { @[comm] = sum(args->ret); }'</pre>
bpftrace attaches the action block to the write syscall return probe (t Confused

yscalls

ys_exit_write). Then, it uses a filter to discard the negative values, which are error codes (/args->ret > 0/).
The map key comm represents the process name that called the syscall. The sum() builtin function accumulates the number of bytes written for each map entry or process. args is a bpftrace builtin to access tracepoint’s arguments and return values. Finally, if successful, the write syscall returns the number of written bytes. args->ret provides access to the bytes.
<h3>Read size distribution by process (histogram):</h3>
bpftrace supports the creation of histograms. Let’s analyze one example that creates a histogram of the read size distribution by process:
<pre class="wp-block-preformatted">$ sudo bpftrace -e 't Confused

yscalls

ys_exit_read { @[comm] = hist(args->ret); }'</pre>
Histograms are BPF maps, so they must always be attributed to a map (@). In this example, the map key is comm. 
The example makes bpftrace generate one histogram for every process that calls the read syscall. To generate just one global histogram, attribute the hist() function just to ‘@’ (without any key).
bpftrace automatically prints out declared histograms when the program terminates. The value used as base for the histogram creation is the number of read bytes, found through args->ret.
<h3>Tracing userspace programs</h3>
You can also trace userspace programs with uprobes/uretprobes and USDT (User-level Statically Defined Tracing). The next example uses a uretprobe, which probes to the end of a user-level function. It gets the command lines issued in every bash running in the system:
<pre class="wp-block-preformatted">$ sudo bpftrace -e 'uretprobe:/bin/bash:readline { printf("readline: \"%s\"\n", str(retval)); }'</pre>
To list all available uprobes/uretprobes of the bash executable, run this command:
<pre class="wp-block-preformatted">$ sudo bpftrace -l "uprobe:/bin/bash"</pre>
uprobe instruments the beginning of a user-level function’s execution, and uretprobe instruments the end (its return). readline() is a function of /bin/bash, and it returns the typed command line. retval is the return value for the instrumented function, and can only be accessed on uretprobe.
When using uprobes, you can access arguments with arg0..argN. A str() call is necessary to turn the char * pointer to a string.
<h2>Shipped Scripts</h2>
There are many useful scripts shipped with bpftrace package. You can find them in the /usr/share/bpftrace/tools/ directory.
Among them, you can find:
<ul>
<li>killsnoop.bt – Trace signals issued by the kill() syscall.</li>
<li>tcpconnect.bt – Trace all TCP network connections.</li>
<li>pidpersec.bt – Count new procesess (via fork) per second.</li>
<li>opensnoop.bt – Trace open() syscalls.</li>
<li>vfsstat.bt – Count some VFS calls, with per-second summaries.</li>
</ul>
You can directly use the scripts. For example:
<pre class="wp-block-preformatted">$ sudo /usr/share/bpftrace/tools/killsnoop.bt</pre>
You can also study these scripts as you create new tools.
<h2>Links</h2>
<ul>
<li>bpftrace reference guide – <a href="https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md">https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md</a></li>
<li>bpftrace (DTrace 2.0) for Linux 2018 – <a href="http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html">http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html</a></li>
<li>BPF: the universal in-kernel virtual machine – <a href="https://lwn.net/Articles/599755/">https://lwn.net/Articles/599755/</a></li>
<li>Linux Extended BPF (eBPF) Tracing Tools – <a href="http://www.brendangregg.com/ebpf.html">http://www.brendangregg.com/ebpf.html</a></li>
<li>Dive into BPF: a list of reading material – <a href="https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/">https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf</a></li>
</ul>
<hr class="wp-block-separator" />
Photo by <a href="https://unsplash.com/@wehavemegapixels?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Roman Romashov</a> on <a href="https://unsplash.com/search/photos/trace?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>.
</div>

xSicKxBot