Did you know…?

LWN.net is a subscriber-supported publication; we rely on subscribers
to keep the entire operation going. Please help out by buying a subscription and keeping LWN on the

By Jake Edge
February 26, 2020

BPF has exploded within the Linux world
over the last few years, growing
from its networking roots into the go-to tool for running custom
in-kernel programs. Its role seems to expand with every kernel release
into diverse areas such as security and device control. But none of that
is the focus of a relatively new book from Brendan Gregg, BPF
Performance Tools
; it looks, instead, at how BPF provides visibility into
the guts of the kernel. Finding performance bottlenecks of
various sorts on (generally large) production systems is an area where BPF
and the tool set that has grown up around it can excel; Gregg’s book
describes that landscape in great depth.

The book is meant to be both a way to learn about what BPF can do to
improve the observability of Linux systems and applications and a reference guide
to a large body of tools that Gregg and others have built up to peer inside
the running system. Interestingly, it does not actually cover the
underlying BPF virtual-machine
instructions all that much, except in an appendix; the focus is on how to
use BPF at a higher level. Even then, learning to actually write tools using the
high-level environments (BCC
and bpftrace) is not
truly the intent either, though
code samples for bpftrace abound. The book is definitely geared toward
finding problems at multiple levels on Linux systems running in

It begins by introducing BPF, noting its origin as the Berkeley
Packet Filter and its eventual upgrade to extended BPF (eBPF), before
giving a quick overview of the tracing and sampling techniques available on
Linux. It then gives a taste of what the BPF Compiler Collection (better
known as BCC) can actually do by using canned tools to examine system-wide
calls and block I/O latency. The different levels of
tracing available in a Linux system, from applications through system
libraries and the system-call interface down to internal kernel tracepoints
and hardware counters, are briefly described with an eye toward a few
bpftrace “one-liners” to examine open() and
system calls. Examples of bpftrace one-liners (and more) can be
found in Gregg’s LWN article from
July 2019 and a report on his talk at
the 2019 Linux Storage, Filesystem, and
Memory-Management Summit

That first chapter would be useful to anyone who is curious what
the BPF fuss is all about. The concepts introduced in the first
chapter (and more) are
spelled out in greater detail in the rest of Part I (“Technologies”). The book is meant
to be read straight through, if desired, or simply used as a reference of
the tools and techniques that can be used to track down problems in a system. That
leads to a bit of repetitiveness here and there throughout, so that readers
popping into a particular place will not be completely lost. It can be a
bit irritating at times for those just reading through it, but it is probably
unavoidable in a dual-purpose book like this.

BPF itself is a complicated beast, which hooks into a wide variety of
facilities for gathering tracing information. That includes both static
options (kernel tracepoints and user-level statically defined tracing
(USDT) markers) and ways to insert dynamic instrumentation into
the kernel (kprobes) or user-space programs (uprobes). BPF programs can be
used to collect information from those sources (and others like hardware
performance monitoring counters (PMCs) and perf_events), summarize it in-kernel, and
display the results in a variety of forms. Chapter 2 describes all of
them in some detail

One of the key advantages of BPF over other tracing techniques is that it does
its work efficiently in the kernel and can simply present its results; many
other tools require storing lots of information in memory or log files and
then post-processing it to actually pull out the data of
interest. Some also require adding code to the kernel, either by
rebuilding it with a different configuration or by adding a kernel module;
BPF dispenses with all of that. In addition, BPF has data structures and
helper functions to collect the kinds of information that might be of interest
(e.g. stack traces); descriptions of all of that is gathered up in
Chapter 2 as well.

While using BPF is the focus of the book, Gregg does
not ignore the other tools available for diagnosing problems. The chapter
on the process of analyzing a system starts with a look at the goals and
methodologies that can be used to narrow things down. There are two separate
checklists that are presented as starting points. The first uses standard
Linux tools (e.g. vmstat,
and sar) in
a “60-second checklist”, before moving into a checklist of BCC tools
(e.g. execsnoop,
and tcpaccept).
Each of the entries on the
checklists is described along with how the output can be useful in
pinpointing where problems might be; the BCC tool descriptions also
reference other parts of the book where they are described in even more detail.

Rounding out Part I are a chapter each for BCC and bpftrace covering
their installation, internal operation, and how they can be used; each
chapter has multiple examples of them in action. These
days, many Linux distributions provide packages for both of these
interfaces, including the tools developed using them. There is also a
large set of tools that Gregg developed specifically for the book, which
can be seen in the diagram below in red; the existing tools are shown in
black. All of the new tools can be found in his GitHub repository

[Tools diagram]

While the first part of the book gives a lot of useful context and a large,
tasty bite of what BPF can do, the meat of the book is contained in
Part II (“Using BPF Tools”). There are 11 separate chapters, each
looking at a different area of the system with an eye toward how to use the
tools and bpftrace one-liners to dig into the operation of that area. For example,
there are chapters covering the CPU, memory, I/O, networking, security,
applications, languages (e.g. Java), containers, and hypervisors.

Each chapter gives some background information to help understand the role
of the area covered in the chapter; it also describes aspects of it
that might lead to performance or other problems. The traditional tools
for investigating problems are introduced with examples given of the kinds
of information they can provide. The chapters then move into BPF tools and
bpftrace programs (or one-liners) that can be used for troubleshooting and
pinpointing problem areas. Many of the chapters have an “Optional
Exercises” section with ideas for ways to extend the existing tools or
write new ones either using BCC or bpftrace; the ones marked “advanced, unsolved” are,
of course, particularly challenging.

The remaining parts of the book are supplemental material at some level. Part III
(“Additional Topics”) has a chapter on other BPF-based performance-analysis
tools and one on “Tips, Tricks, and Common Problems”. The final part is
appendixes, including a list of all the one-liners used in Part II, a
bpftrace cheat sheet, information on developing BCC-based and C-based BPF
programs, and a reference on the BPF instruction set. That is followed by
a glossary, bibliography, and an index.

I have a couple of nits to point out with the book, but overall it is an
excellent book with comprehensive coverage of BPF-based tools and how to
use them for investigating performance and other problems. The book can be
a bit overwhelming at times, but that is really due to the subject matter
at hand; there are lots of parts and pieces in the BPF landscape, so trying
to keep them all straight can be a challenge.

I got a review
copy of the EPUB
version of the book from the publisher that I read in two different ways:
on a tablet using Lithium
and on my desktop with calibre;
I did not try it on my Kobo E Ink
reader as the layout of the book did not seem conducive to a small,
monochrome screen. There were some rendering problems I encountered on
Lithium, which I have used successfully with other technically oriented
books; examples and tool output that spanned page boundaries on the screen would not
display the piece on the next page. But there were links that would take you
to a full-page rendering of the item, which could then be tapped to return
to the right place.
Calibre did not have that flaw and presumably other EPUB readers would not
either, but it was obviously not annoying enough for me to go search out
another reader.

The book has quite a number of in-line footnotes, which are useful; they
often highlight the history and developer behind a particular tool. But the
use of square-bracket-style links in the text left something to be
desired. Clicking (or tapping) those would take you to an entry in the
list after the bibliography, but each listed item was itself simply a link
to a web URL. Some way to directly go to the linked-to item would have
been a bit easier to navigate. Obviously, a dead-tree version of the book
would not suffer from that lack, but paging to the list might be a bit
painful as well. Perhaps newer editions could simply use regular footnotes
for the links as well, making them directly selectable in electronic copies
and saving the paging on paper copies.

While the book focuses on performance problems on “big iron”—many of the
examples show output from 48-CPU systems—the techniques and tools will be
useful for a wide variety of other environments. Tracking down bugs on a
desktop system or gaining familiarity with the internals of the kernel are
just two of the possibilities that the book helps unlock. Nearly anyone
running Linux will find a bpftrace one-liner (or three) that
will pique their curiosity.
BPF Performance Tools is definitely
worth a look for anyone curious about the workings of their Linux systems.

(Log in to post comments)

Source link