patch – Blog

What's new in zcov - March 2026

Mon, 23 Mar 2026 00:00:00 +0000

Graph improvements

I wanted zcov to quickly and naturally show the coverage status of a piece of code. Most of the time we’re interested in missing coverage as that is where we need to do work, either adding a test, improving the spec, or figuring out if there is a mismatch between the code as-is and then goals we’re trying to achieve. zcov can now colour code the blocks to show this, using a common red/yellow/green system. This works quite well; the red blocks immediately stand out. In fact, differently coloured blocks stand out, so if a function is mostly covered, missing coverage stands out, and vice versa.

It comes with two modes, block and edge. The block mode is the most basic one - is the block visited or not, a direct improvement of the traditional line coverage red/green highlight. Here’s an example from GNU coreutils:

The edge mode is slightly more sophisticated and will colour blocks green when all outgoing edges are taken, yellow when some are taken, and red when none are taken. This is the same function as before, but in edge mode, which colours block 2 yellow.

What’s missing is a way to neatly show which edges are covered and which are missing. I tried colouring the edges using the same colour schema, but they actually became a lot harder to read that way.

Colouring for MC/DC is not yet implemented, but shouldn’t be too far off. The colours used for missing/covered/partial will be configurable, too, both to mesh well with colour schemes, and to improve accessibility (colour blindness)

I have also added labels to edges of conditions so it’s easy to tell which edge (and outcome) is the true and false one. This makes it a lot easier to understand the graph as the distance between the code in blocks makes it a lot harder to recognize which successor is the then and else.

Sequence numbers in prime paths

zcov has been able to highlight a single path for a long time by making the block borders and edges thicker, but it was very difficult to actually see where paths began, especially in tight loops. A prime path is a sequence of blocks, not simply a collection of blocks. Recently zcov will paint the sequence number in the block header when the path is selected. This is a necessity when interrogating paths in loops, as it would otherwise be impossible to tell which of the rotated paths we’re looking at. Here are two examples of a rotated path in the or function in GNU coreutils; the first path goes from block 4 through 4, the other from block 5 through 5.

The list itself has been simplified to show the path number and the first .. last block. The actual sequence is hard to read out of a list, and sequences can be very long, and long sequences are easy to understand in graph form.

Control panel for the function/graph view

There is a new info- and control panel for the function/graph view. This new panel makes it easy to get an overview of which function we’re looking at, presenting its name (both mangled and demangled), the source file it is located in, the coverage as nice progress bars, and a control for selecting paths for highlighting.

By default the uncovered prime paths will be shown as they are the most likely to be interesting. When working towards coverage, this will be the shrinking set, and could even function as a todo list. Two more path filters are currently supported, covered and all, as seen in this picture.

Function filtering

The function list was one of the first features I added to zcov, and I have added a filter-and-search box to it. As shown here it uses a regular expression search/filter to only show functions matching a pattern. This is a requirement for large projects. As a data point, GNU coreutils has more than 2500 functions, and it’s not really all that large.

Closing remarks

These features were made possible by an overhaul and, for the most part, a simplification of the zcov internals. The core engine is a lot more capable now than just a few months ago, and improving still. I plan to quite a few more features and polish over the coming months.

I have also updated the software page with the new screenshots.

Some fixes and improvements in GCC

Sat, 07 Mar 2026 00:00:00 +0000

GCC 16 will probably release in a couple of months, and comes with a couple of my patches. There’s nothing too big this time, but a couple of bug fixes and some quality-of-life changes.

You no longer need to explicitly pass -ftest-coverage for -fcondition-coverage and -fpath-coverage to be useful, it is now implied. The -ftest-coverage flag controls if GCC creates the .gcno files gcov needs to create the report. The coverage support in GCC is built on top of arc profiling which underpins profile guided optimization (PGO), and the PGO doesn’t need the .gcno, only the .gcda (counters). Coverage was a sort of side effect, and MC/DC and prime path coverage was built on that framework. Unlike arc profiling, it doesn’t make much sense to ask for MC/DC and prime path coverage without also wanting to read the reports, so this makes GCC a bit easier to use.

I fixed a bug where gcov-dump printed the wrong offset for condition blocks, and taught it how to print the PATHS tag. gcov-dump is mostly useful for developing gcov itself, but it’s nice that it’s there. There’s a another bugfix in there too, which caused bad counter updates, but this bug was never included in a release.

I have revised my paper on MC/DC, collected some data for it; on how the instrumentation affects compile time, runtime overhead, and object size. What I found was that compile times suffered greatly when analyzing expressions with many conditions joined by a single operator. A phase of the CFG analysis is figuring out which other conditions to mask when we take an edge, and this step evaluated all possible candidates. What I realised is that we don’t need to evaluate all of them, the search starting at the left-adjacent operand (which we must also include) will dominate and always find all the masked conditions. This had a massive impact on compile times.

This is one of those problems that don’t really show up in testing that easily, because under normal circumstances this isn’t a problem. I wrote a small test program which causes the worst behaviour, a single (x && y && ... && z). I tested two cases, 1 is (x && y && ... && z) and 4 is (x1 || ... || x8) && .... These are the compile times before and after the fix:

Those numbers are for all of GCC, including parsing, code generation, and linking. I also measured just the MC/DC analysis pass (finding the masking table and emit the instrumentation code), and get somewhere between 15–20 times speedup, not bad at all. As it turns out, algorithms matter.

    before: 20822.303 ms (41.645 ms per expression)
    after:   1288.548 ms ( 2.577 ms per expression)

As you can see from the graphs, the performance hit really starts to kick in past 16 conditions, which is quite rare in practice. Still, faster is nice. I did find one case with 27 conditions in GNU ls, but that’s very much the exception.

A brief summary of 2025

Fri, 16 Jan 2026 00:00:00 +0000

2025 in a minute

2025 was an eventful year. My prime path coverage was included and released with GCC, along with a few minor patches. I presented prime path coverage and my work on GCC at NASA at the Johnson Space Center in Houston, and published the pre-prints of my papers on MC/DC and prime path coverage in GCC. I wrote and did a course on package management with Guix. I have delivered on contracts on build systems and package management, and data processing in Zig. A client hired me to do an interesting system in Rust, leading to a large-ish contribution to a physical units library. zcov saw steady improvements, too.

Techtown 2025 and 2026

The recordings of the Techtown 2025 talks are out. There were a lot of good talks this year (as every year, really). I can in particular recommend JF Bastien’s Much ado about Noping, Anders Knatten’s Real-time safety - Guaranteed by the Compiler!, Jana Machutová’s Parallel Algorithms: Work Smarter, Not Harder for a good overview of the C++ parallel algorithms library, Xavier Bonaventura’s Dynamic memory allocation challenges in safety critical systems, and of course Robert Seacord’s _ Composite Array Types in C_. This is just a small sample of the full programme. Kevlin’s locknote The Past, Present and Future of Programming Languages was a blast, too.

There’s also my own talk, Coverage, quality, and Goodhart’s Law. We had some sound problems in the room with a rebellious loudspeaker, but the recording is mostly fine.

The NDC Techtown 2026 Call for papers is open. Last year was brilliant and I have no doubt this year will be another fantastic experience. The submission deadline is May 3rd and the event is September 21-24 at Kongsberg.

A git bisect story

Wed, 05 Nov 2025 00:00:00 +0000

There are many good reasons to maintain a proper git history, with deliberately crafted and focused commits, a state that builds, and good messages that provide context, motivation, and reasoning for the change. A neat side effect of a well kept log is that it boosts the usefulness of git bisect. Briefly, git bisect binary searches the history to find the commit which introduced the bug, and it’s an indispensable tool.

I won’t cover how to use bisect in too much detail in this post as there is plenty of material on it out there (there is, however, a section on it in the course on effective git), but I wanted to share this story just to show that bisect is quite useful when solving real problems, and isn’t just some unwieldy bonkers tool that’s useful in theory only.

So this all started with some small work on gcc, when I noticed some new test failures. I suppose I was extra sensitive to the failure because one of the tests failing was mine, one of the prime path coverage tests.

FAIL: gcc.misc-tests/gcov-29.c line 826: expected 0/5 covered paths, was 0/6

I quickly stashed away my work and confirmed that I did not just introduce the problem, and once I had verified that the hunt could start. The thing is, the failure could have been introduced at any point between now and whenever I worked on gcc last, which was this patch in June.

commit 246c33ac8e8e1967c74ae20c07454a24ef02822a
Author: Jørgen Kvalsvik <j@lambda.is>
Date:   Thu Jun 19 20:56:30 2025 +0200

    Free buffer on function exit [PR120634]

    Using auto_vec ensures that the buffer is always free'd when the
    function returns.

            PR gcov-profile/120634

    gcc/ChangeLog:

            * prime-paths.cc (trie::paths): Use auto_vec.

Between that patch and the current HEAD I count 3450 commits, give or take a couple. git diff --shortstat shows a decent changeset, too, which means we’re looking for a specific change among 800k lines of code. It’s a reasonable assumption that the problem is in the gcc/ dir, which limits the changeset to roughly 110k. Smaller, but still quite the haystack.

$ git diff 246c33ac8e..HEAD --shortstat
  8266 files changed, 875667 insertions(+), 249224 deletions(-)

$ git diff 246c33ac8e..HEAD --shortstat -- ':!*test*' 'gcc/*.cc' 'gcc/*.h'
  1125 files changed, 113140 insertions(+), 60739 deletions(-)

The bisect process works by marking a pair of commits good and bad (with the current commit usually being bad). Good could be any commit we remember or at least assume is good, i.e. where the bug is not present. One of the nice properties of the binary search is that it’s ok to overshoot a little, because past a couple of hundred commits overshooting a bit means at most having to check one more commit. For the ~3500 commits since my own we need to do 12 checks to find the commit that introduced the bug.

Confession time. I didn’t even think of using my own commit as the good one, so I just picked a commit at random from about a year ago. Curiously, this was from before the prime path coverage was even merged, so at least the test didn’t fail. There were just north of 9000 commits between then and now, so I increased the workload from 12 to 14 jumps.

Anyway, bisecting. Building gcc takes a while, and manually checking if a test passes or fails is both dull work, and quite slow because the bisect pauses until the current commit is marked as good or bad. I prefer to let the computer do work (which it is, in fact, quite excellent at), so I wrote a small script to configure, build, and test gcc, and let bisect know if the commit is good or bad.

#!/bin/bash

set -euo pipefail

red='\033[0;31m'
green='\033[0;32m'
nc='\033[0m'

bisectdir=build-$(git rev-parse HEAD)

# cache and re-use builds for bisect replays and inspection
if [ ! -d "$bisectdir" ]; then
    mkdir -p $bisectdir
    pushd $bisectdir
    ../configure --enable-languages=c,c++ --disable-libsanitizer --disable-bootstrap --disable-multilib
    make -j$(nproc) > /dev/null
    popd
fi

pushd $bisectdir
set +e
make -k check-gcc RUNTESTFLAGS="gcov.exp" | grep "FAIL: gcc.misc-tests/gcov-29.c"
if [ $? -eq 0 ]; then
    printf "$red$bisectdir is bad$nc\n"
    exit 1
fi

printf "$green$bisectdir is good$nc\n"

Having everything we need, it is time to start bisecting. We tell git what range to search and what commands to run at each commit:

git bisect start HEAD d5cebf7e4491ea0306a072ca3368d771a5ff9455
git bisect run ./test-gcc.sh

I started the bisect and walked away for an hour or so and came back to bisect having found the commit. Bisect narrowed the search down from some 110k lines to 67 insertions(+), 189 deletions(-). I verified that the problem indeed was introduced here, running tests for this commit and the previous one. Now the problem was just figuring out what this change is, and how it would affect the tests. Thankfully, the author left an excellent explanation in the commit message which helped me understand the change, and I quickly figured out a fix which I have proposed to the maintainers.

That’s the story. Knowing when the problem began is half the battle, and git bisect makes that trivial. Finally, here’s the log of this bisect adventure.

$ git bisect log
# bad: [c0132281f671a79da992cec75a65c4f739ad991f] Ada: Fix explicit raise on subtype of lock-free protected type
# good: [d5cebf7e4491ea0306a072ca3368d771a5ff9455] c: Diagnose compound literal for empty array [PR114266]
git bisect start 'c0132281f67' 'd5cebf7e4491ea0306a072ca3368d771a5ff9455'
# good: [21842fe301caa5dbc69a69033cdc17bb29b8c399] c++: CWG2369 workaround and ... [PR120185]
git bisect good 21842fe301caa5dbc69a69033cdc17bb29b8c399
# bad: [b963237a903f73cf22ad024c0ef6098a9ee27ed4] optinfo: use enum class
git bisect bad b963237a903f73cf22ad024c0ef6098a9ee27ed4
# good: [e6406aefd1a25b6dba845a52cfd9484188ff5720] analyzer: fix missing "final override"
git bisect good e6406aefd1a25b6dba845a52cfd9484188ff5720
# bad: [5300e2bda9c74ca458f01c7e8fc3ea035687b900] aarch64: Allow 64-bit vector modes in pattern for BCAX instruction
git bisect bad 5300e2bda9c74ca458f01c7e8fc3ea035687b900
# good: [37bf13adcda564dfdb28c3aa736f2cac71c73d09] tree-optimization/118669 - fixup wrongly aligned loads/stores
git bisect good 37bf13adcda564dfdb28c3aa736f2cac71c73d09
# good: [dc30f404170f538af6bf2457ccff252b08302dec] Widening-Mul: Support unsigned scalar SAT_MUL form 1
git bisect good dc30f404170f538af6bf2457ccff252b08302dec
# good: [7135990e3b07d3f29de46e9ca7ddbb66c094d52e] libstdc++: Added missing members to numeric_limits specializations for integer-class types
git bisect good 7135990e3b07d3f29de46e9ca7ddbb66c094d52e
# good: [b7bd72ce71df5266e7a7039da318e49862389a72] aarch64: PR target/120999: Adjust operands for movprfx alternative of NBSL implementation of NOR
git bisect good b7bd72ce71df5266e7a7039da318e49862389a72
# good: [baaee10123db6cf896283175b345d535b225defb] c++, libstdc++: Implement C++26 P3068R5 - constexpr exceptions [PR117785]
git bisect good baaee10123db6cf896283175b345d535b225defb
# good: [3e7dd06860bf0589bc743e458b16bd1d33d73835] cobol: Fix build on 32-bit Darwin [PR120621]
git bisect good 3e7dd06860bf0589bc743e458b16bd1d33d73835
# good: [52d9c2272f6366cd5b30e9540ce8ef16b482cee5] Fix ICE in speculative devirtualization
git bisect good 52d9c2272f6366cd5b30e9540ce8ef16b482cee5
# bad: [9eea49825ebb607f4b67de48c9cba1f85e005932] testsuite: Add testcase for already fixed PR [PR120954]
git bisect bad 9eea49825ebb607f4b67de48c9cba1f85e005932
# bad: [385d9937f0e23cbf9c62f0b2553a33ff70e56ecf] Rewrite assign_discriminators
git bisect bad 385d9937f0e23cbf9c62f0b2553a33ff70e56ecf
# first bad commit: [385d9937f0e23cbf9c62f0b2553a33ff70e56ecf] Rewrite assign_discriminators

Guix shell for programs with batteries included

Wed, 08 Oct 2025 00:00:00 +0000

One of the aspects of Guix that really sets it apart from other package managers is how there is not much difference in power between the user and the repository. Sure, you can build your own .debs just fine and manage them with dpkg, but it takes a lot more infrastructure to have your own small packages play well with apt (and other users). For sharing packages in Guix, most of the time it is sufficient to share a small pkg.scm or similar.

Sometimes you discover these that such features lead to tricks that are just bonkers, equal parts brilliant and terrifying.

Let’s start with the guix shell. Running guix shell pkgs... drops us in a shell where all pkgs... are made available. This is quite useful for one-off tasks such as when building software or testing with different versions of a program. It is also very useful when writing scripts, because you can “bundle” difficult dependencies with the script itself, and have the script fetch them as needed. I like doing it in makefile targets, too, so that when I run make doc.pdf, make knows how to retrieve my document processor as well as running it.

To show it off, we can start by fixing Python. Running Python programs can be painful because libraries, much like shared libraries in the traditional unix sense, share namespace and different programs have a knack for requiring different versions. Over the years there have been multiple tools that try to solve this; virtual environments (venv), pipenv, lately uv is gaining popularity. Let’s solve it with Guix; here’s hello-numpy.py:

#!/usr/bin/env -S guix shell python python-numpy -- python3
import numpy
print("This is numpy", numpy.version.version)

This script loads numpy and prints the version. This is from my Debian 12 system, which has numpy installed:

$ python3 hello-numpy.py
This is numpy 1.24.2

The file is executable with a shebang, so let’s put it in PATH and run it:

$ env PATH=$(pwd):$PATH hello-numpy.py
This is numpy 2.2.5

If you run this on a fresh system, Guix will first download Python and Numpy before running it.

This is already quite useful because we can now reliably run programs with many and even very specific dependencies. But we can take it further, because Guix is not specific to Python environments, it works for all kinds of software. Here’s an example with python, cmake, and coreutils; note how the versions change if run through bash rather than using the shebang.

$ env PATH=$(pwd):$PATH mix
cat (GNU coreutils) 9.1
Python 3.11.2
cmake version 3.25.1

$ bash mix
cat (GNU coreutils) 9.1
Python 3.11.11
cmake version 4.0.3

Guix supports manifest, small Guile programs that guix shell can run to determine what packages to provide. It turns out that with a clever trick we can embed these manifests inside our scripts. There is a nice post on this at futurile, and the trick was originally suggested on the Guix mailing list.

The program in the futurile post is a shell script which spawns a Guix shell in a container with itself as the manifest. This is already spectacular, but since it is really just a shell script we can combine it with other classic tricks. Let’s transform the hello-numpy to use the same trick (plus a new one):

#!/usr/bin/env bash

exec guix shell --manifest="$0" -- python3 <<EOF
import numpy
print("This is numpy", numpy.version.version)
EOF

!#
(specifications->manifest
 (list "python"
       "python-numpy"))

So this is a shell script that embeds a Python script (in a HEREDOC) that knows how to download and configure the Python dependencies by providing itself as a Guile program input to a command. When I run it on my system it prints “This is numpy 2.2.5”, showing that it’s using the Guix one.

Most of the time, manifests are just lists of package names, but since they’re just Guile programs we can put even more interesting things in there. Let’s say we want write a program that prints a nice header and footer and dumps its inputs in bytes and braille. In the examples so far we’ve only looked at very standard pieces of software, but this is more exotic. Here is my program, cowbraille (there’s a demo at the end of this post):

#!/usr/bin/env bash

if [ -z "${BRAILLE_GUIX_ENV+1}" ]; then
    exec guix shell -m "$0" -- env BRAILLE_GUIX_ENV=1 bash "$0" "$@"
fi;

cowsay "They say curiosity killed the cat, but I'm a cow so what do I know"
for f in "$@"; do
    cowsay "This is $f"
    bd "$f"
done
cowsay "That's all Folks!"

exit

!#
(use-modules (guix packages)
             (guix download)
             (guix gap)
             (guix build-system gnu))

(define brailledump
  (package
    (name "brailledump")
    (version "6e6fc38935054db0534d5af4fb99c6193305b946")
    (source
     (origin
       (method url-fetch)
       (uri (string-append
             "https://raw.githubusercontent.com/jart/cosmopolitan/"
             version "/tool/viz/bd.c"))
       (sha256
        (base32
         "1a6l7xym23fscslgxikhfnc91yfx0jscs77djsm08f213mz4sips"))
       (modules '((guix build utils)))
       (snippet
        '(substitute* "bd.c"
           ;; We need to use standard includes, not the cosmopolitan
           ;; internal ones
           (("\"libc/stdio/stdio.h\"")
            "<stdio.h>\n#include <stddef.h>\n#include<locale.h>")
           ;; Call setlocale to print utf8 on glibc
           (("long o;") "long o; setlocale(LC_ALL, \"\");")))))
    (build-system gnu-build-system)
    (arguments
     (list
      #:phases
      #~(modify-phases %standard-phases
          (delete 'configure)
          (delete 'check)
          (replace 'build
            (lambda _ (invoke "gcc" "-O2" "bd.c" "-o" "bd")))
          (replace 'install
            (lambda* (#:key outputs #:allow-other-keys)
              (let ((bin (string-append (assoc-ref outputs "out")
                                        "/bin")))
                (install-file "bd" bin)))))))
    (synopsis "hexdump -C with braille")
    (description
     "braille dump is a drop in replacement for hexdump -C that uses
unicode braille characters to display hex codes 0x81..0xff thereby
improving the readibility of binary.")
    (home-page "https://justine.lol/braille/")
    (license #f)))

(packages->manifest
 (append
  (list brailledump)
  (specifications->packages
   (list "bash" "coreutils" "cowsay"))))

If you have Guix available you should be able to run this yourself, just copy the contents into a file and make it executable!

There’s a lot to unpack here, and maybe a later post will go through the details carefully, but running this program will:

Download the source for brailledump
Patch it slightly to make it work with glibc
Tweak the build rules to call gcc directly, without going through ./configure && make
Build brailledump
Create a shell where brailledump, cowsay, and bash are available
Run brailledump on each argument, with progress reported by cows

All of this neatly packaged up into a single file.

We need to guard the exec guix shell to avoid infinite recursion, and exit so that the script does not try to execute the Guile (which obviously wouldn’t work).

This should work anywhere, but if you can assume bash, zsh, or any other shell that sets the _ env var to the program currently being run you can use this funny-looking shebang and skip the BRAILLE_GUIX_ENV guard.

#!/usr/bin/env -S guix shell -m "${_}" -- bash "${_}"

env (1) will expand named variables, in this case _, before running the interpreter. For cowbraille, it would expand cowbraille files... to:

exec guix shell -m "cowbraille" -- bash "cowbraille"

This fails in dash, and using bash as the interpreter is probably more robust overall, and the guard should feel way more familiar.

So, should we do this for all our programs? Probably not, but it’s a stellar trick that can occasionally be very useful.

Interested in learning more? Patch offers training and workshops on Guix, and can help you manage software and packages with Guix in your organization.

And as promised, here is a demo of running cowbraille:

cowbraille demo; running this program for the first time will build & install brailledump

Programming for coverage part 6

Mon, 08 Sep 2025 00:00:00 +0000

Drawing insights

We can draw interesting insights from measuring code coverage. This is a binary search that I borrowed from the internet and used in my talk on prime path coverage in GCC at TechTown 2024, very slightly revised. It returns the index of the needle (x) in a sorted array, or not-found (-1) if it could not be found.

int bs(const int A[], int N, int x) {
    int min = 0;
    int max = N - 1;
    int mid = min + ((max - min) / 2);
    while (A[mid] != x && min <= max) {
        mid = min + ((max - min) / 2);
        if (x > A[mid])
            min = mid + 1;
        else
            max = mid - 1;
    }

    return A[mid] == x ? mid : -1;
}

We add tests for the cases we can think of along with the expected results. All good so far, no real problems, and the tests looks solid.

void check() {
    static const int haystack[] = {
        1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144,
        233, 377, 610, 987, 1597, 2584, 4181,
    };

    const int len = sizeof (haystack) / sizeof (*haystack);

    // smaller than smallest
    assert(bs(haystack, len, 0) == -1);
    // hit left
    assert(bs(haystack, len, 2) == 1);
    // miss left
    assert(bs(haystack, len, 4) == -1);

    // larger than largest
    assert(bs(haystack, len, 5000) == -1);
    // hit right
    assert(bs(haystack, len, 1597) == 15);
    // miss right
    assert(bs(haystack, len, 1024) == -1);
}

Measuring coverage

We measure the coverage and are surprised to find that we have 100% block- and edge coverage, but just ~63% prime path coverage, with 15/24 paths covered. If we search for all values between 0 and 5000, which covers the whole haystack with margins on either side we end up covering 16/24 paths, just one more than our hand-written set. We need to understand why coverage is still so poor. This is the CFG from zcov:

Original binary search

Path 2, 3

These are the paths where min > max on the first iteration, which can only happen when N <= 0. N is the array length, so a negative value does not make sense. These are both inputs we simply didn’t test for, but that we surely want to handle correctly by always returning not-found, and making sure we’re not accessing memory out of bounds.

Path 2

Path 3

Path 5, 23

We found the value we’re looking for, but we’re testing if we should return not-found (-1). This is a contradiction; since we broke the loop on A[mid] == x, we can never see A[mid] != x in the ternary.

Path 5

Path 23

Path 7, 10

These are also contradictions. In these paths we break the loop on min > max, which is checked after A[mid] != x => true. Since A[mid] == x and A[mid] != x can’t be true at the same time, we cannot cover these paths.

Path 7

Path 10

Path 19, 20

These paths are hard to distinguish from 7, 10; they start in block 7, which is the comparison min <= max. In these paths, x > A[mid] is true, but the loop immediately breaks on A[mid] != x => false, which implies A[mid] == x => true. This is a contradiction; x > A[mid] and A[mid] == x cannot be true simultaneously, and these paths cannot be covered.

Path 19

Path 20

Rewriting the algorithm

The key insight from measuring the coverage is that we have a lot of redundant checks. Poor coverage aside, this is a common source of bugs. When we read code we often simulate the execution, draw mental data flow diagrams and compose long chains of reasoning. Even when the comparisons are reused they form yet another branching point, just like the control flow graph. We can rewrite the binary search with the new insights in mind and eliminate the redundant comparisons.

We move the A[mid] == x check into the loop. mid is no longer carried between iterations, and the value can be moved into the loop, too. If we find the value we’re looking for we immediately return as there is nothing more to do, which eliminates the ternary in the return statement.

int bs(const int A[], int N, int x) {
    int min = 0;
    int max = N - 1;
    while (min <= max) {
        int mid = min + ((max - min) / 2);
        if (x == A[mid])
            return mid;
        else if (x > A[mid])
            min = mid + 1;
        else
            max = mid - 1;
    }

    return -1;
}

We test it using the same checks and measure the coverage. This version has 16/18 (89%) paths covered, and we can quicky spot the missing test cases with zcov. These are the missing cases, paths 0 and 3.

Path 0

Path 3

These are the cases where the needle is in the exact middle (path 0) and where the loop is never entered (the empty list). Both of these are oversights that should clearly be a part of the functional testing suite. We add these checks to complete our test suite and measure to verify that 18/18 paths are covered.

    // missing tests:
    // empty list
    assert(bs(haystack, 0, 1) == -1);
    // mid hit
    assert(bs(haystack, len, 55) == 8);

The new function violates MISRA (only a single return per function), but I find that perfectly acceptable and better than the alternatives.

Programming for coverage part 5

Thu, 28 Aug 2025 00:00:00 +0000

The last post used custom replacement functions and allocator interfaces to simulate allocation failures for testing purposes. Those approaches work well when you control both sides of the function call, but it won’t work if we want to test handling errors originating in libraries we don’t control.

The malloc might fail because the system is out of memory or when the resource limit is reached. This can happen both when calling malloc directly, or indirectly through functions like strdup or fopen. C does not have replacement functions, but it is not uncommon to substitute the libc malloc with a different allocator like tcmalloc or jemalloc when linking or loading the program. From the jemalloc documentation:

# At load time
LD_PRELOAD=`jemalloc-config --libdir`/libjemalloc.so.`jemalloc-config --revision` app
# Link jemalloc dynamically
cc app.c -o app -L`jemalloc-config --libdir` -Wl,-rpath,`jemalloc-config --libdir` -ljemalloc `jemalloc-config --libs`
# Link jemalloc statically
cc app.c -o app `jemalloc-config --libdir`/libjemalloc.a `jemalloc-config --libs`

This shows we can provide our own custom functions at link time, such as one that simulate failures. Providing the a different malloc at link time this way works without a multiple definitions error because malloc is a weak symbol. It would not work for most third party libraries as it is unusual to declare symbols weak. Still, weak symbols would not address the main hurdle; we want to simulate failures, but some calls should still succeed. Going back to our custom new from part 4:

void* operator new(std::size_t sz) {
    if (!remaining) throw std::bad_alloc{};
    remaining -= 1;
    if (sz == 0)
        ++sz;
    if (void *ptr = std::malloc(sz))
        return ptr;
    throw std::bad_alloc{};
}

In our replacement new we use malloc to get memory from the OS. If we similarly try to replace malloc we would get this, which clearly cannot work as the recursive malloc calls would run infinitely (or probably crash stack overflow and pretty quickly). So how do we make this work without having to implement a complete allocator?

void *malloc(size_t sz) {
    if (!remaining)
        return NULL;
    remaining -= 1;
    return malloc(sz);
}

Some linkers (GNU ld, GNU gold, lld, maybe more) supports the --wrap=<symbol> flag, which replaces calls to symbol with calls to __wrap_symbol while keeping the original definition of symbol available as __real_symbol.

void *__real_malloc(size_t);
void *__wrap_malloc(size_t sz) {
    if (!remaining)
        return NULL;
    remaining -= 1;
    return __real_malloc(sz);
}

Here is an example test. This is a custom strdup and a test that expects strdup to fail the second time it is called. Memory allocation failures are quite rare, so this test is unlikely to pass.

#include <assert.h>
#include <stdlib.h>
#include <string.h>

char *xstrdup(const char *s) {
    size_t len = strlen(s) + 1;
    void *new = malloc(len);
    if (new == NULL)
        return NULL;
    return (char *) memcpy(new, s, len);
}

void check() {
    const char *refstr = "The quick brown fox jumps over the lazy dog";
    assert(xstrdup(refstr) != NULL);
    assert(xstrdup(refstr) == NULL);
}

int main() {
    check();
}

$ gcc xstrdup.c -o xstrdup
$ ./xstrdup
xstrdup: xstrdup.c:16: check: Assertion `xstrdup(refstr) == NULL' failed.

Now that we have confirmed that our test fails as expected we can make it pass. We add a new file alloc.c with our wrapped malloc, and include a print for good measure so it is easier to see what is going on.

#include <stdio.h>

static size_t remaining = (size_t)-1;
void fail_nth_malloc(size_t n) { remaining = n; }

void *__real_malloc(size_t);
void *__wrap_malloc(size_t sz) {
    printf ("malloc(%zu), %zu remaining\n", sz, remaining);
    if (!remaining)
        return NULL;
    remaining -= 1;
    return __real_malloc(sz);
}

We must also instruct our test to fail the next malloc by calling fail_nth_malloc().

void fail_nth_malloc(size_t);
void check() {
    const char *refstr = "The quick brown fox jumps over the lazy dog";
    assert(xstrdup(refstr) != NULL);
    // Fail next malloc
    fail_nth_malloc(0);
    assert(xstrdup(refstr) == NULL);
}

$ gcc xstrdup-2.c alloc.c -o xstrdup-2 -Wl,--wrap=malloc
$ ./xstrdup-2
malloc(44), 18446744073709551615 remaining
malloc(44), 0 remaining

Using function wrapping to simulate failures is less ergonomic than replacement new and dynamic allocators, and should maybe be used as a last resort. The wrapping is done by the linker and has to be wired up outside the programming language and, because of their global nature, require small and focused test binaries to not interfere with other tests. Still, it a quite useful feature that can support complex failure simulation sourced from libraries and we can’t (or won’t) change the design. This post used malloc as an example, but function wrapping works well for arbitrary functions.

More resources:

Programming for coverage part 4

Wed, 27 Aug 2025 00:00:00 +0000

This is a trivial C++ function that does an unchecked access of an array element. It is perfectly fine to use if i is in bounds, and the CFG is just the straight line you would expect. Testing wise, the only thing we really can do is call it with a valid vec and i in bounds, as anything else would be undefined.

int& unsafe_at(int* vec, std::size_t i) {
    return vec[i];
}

unsafe_at()

Here is a similar function with a bounds check. If the access would be out of bounds we throw an exception. This is what happens internally in std::vector::at.

int& checked_at(int* vec, std::size_t size, std::size_t i) {
    if (i >= size)
        throw std::exception();
    return vec[i];
}

checked_at()

Exception paths must exercised for coverage since exceptions are control flow constructs, too. Most C++ testing frameworks have first-class support for checking that the an exception is thrown, like this example using catch2:

TEST_CASE ("checked_at") {
    const std::size_t len = 5;

    // 0 1 2 3 4
    auto* vec = make_sequence(len);

    CHECK (checked_at(vec, len, 0) == 0);
    CHECK (checked_at(vec, len, 4) == 4);
    CHECK_THROWS (checked_at(vec, len, len));
}

The big change exceptions bring are paths that do not terminate in the special end block as the function does not return when an exception is thrown. When an exception is thrown the function terminates in a special throw block, which starts the stack unwinding and exception propagation. If we were to handle the exception it becomes regular control flow, as in this example.

int sum(int acc, int* vec, std::size_t size, std::size_t i) {
    try {
        const int v = checked_at(vec, size, i);
        acc += v;
    } catch (...) {
        acc += 0;
    }
    return acc;
}

Handling exceptions turn them into regular control flow

It is quite important to also verify that exceptions are thrown as expected, and that they are handled as intended. I have written code where I assumed an exception would be thrown when it in fact did not. Exception handling is also a common source of bugs. Error paths can be difficult to test because the conditions leading up to them are hard to synthesize, and sometimes they are missed because the control flow is invisible. Coverage make these paths light up.

The exception thrown by checked_at should be easy to avoid in real programs. If an index comes from untrusted (e.g. user) input we usually validate before using it for access, and indices sourced from program logic tend to be derived from the array and consistent. This does not apply to all errors. Here is an rather innocent improvement to checked_at(), where the exception thrown is the richer std::out_of_range, the same exception thrown by std::vector::at.

int& checked_at(int* vec, std::size_t size, std::size_t i) {
    if (i >= size)
        throw std::out_of_range("i >= size");
    return vec[i];
}

checked_at() throwing out_of_range

Can you guess why there is a second throw block? Here are the signatures for the std::out_of_range constructor:

out_of_range(const std::string& what_arg);
out_of_range(const char* what_arg);
out_of_range(const out_of_range& other);

The std::out_of_range object constructs and copies a std::string for its message. std::string is dynamically sized which means it must allocate, which might fail and throw a std::bad_alloc. For it to be exercised we must try to access the array out-of-bounds and run out of memory during the error handling. This is a a problem for testing (and coverage) because memory is system dependent, global, and implicit resource that is hard to control. So what can we do?

One strategy is to accept missing coverage for exception paths. This is more of a coping strategy than anything, because you absolutely want to exercise some exceptions, or show why they cannot happen.

Since we do not control the allocations happening inside the standard library it is difficult to have them fail at the right time. Yet these failures are important as seen in the previous example because those paths are very much a part of the program. We also really don’t want to pollute with code that exists to support testing either, so adding if (in_test) fail_next_alloc() is not viable. Not all hope is lost, because C++ supports replacement functions which enable us to write a custom operator new.

What I usually do is instruct my allocation function to make the nth allocation fail. It might be possible to use sophisticated techniques such as stack scanning or parsing backtraces to figure out when to fail, but using a counter is usually enough. I add a few extra functions in my test suite:

static std::size_t remaining = std::numeric_limits<std::size_t>::max();

void fail_new_after(std::size_t n) {
    remaining = n;
}

void* operator new(std::size_t sz) {
    if (!remaining) throw std::bad_alloc{};
    remaining -= 1;
    if (sz == 0)
        ++sz;
    if (void *ptr = std::malloc(sz))
        return ptr;
    throw std::bad_alloc{};
}

void* operator new[](std::size_t sz) {
    if (!remaining) throw std::bad_alloc{};
    remaining -= 1;
    if (sz == 0)
        ++sz;
    if (void *ptr = std::malloc(sz))
        return ptr;
    throw std::bad_alloc{};
}

In my test I repeatedly increase the allocation allowance until the function succeeds. This is a standard testing technique John Lakos referred to as orthogonal perturbation at ACCU 2022 (ca 20m30s). It is a heavy-handed approach, but it works well enough. Pretty much nothing in C++ makes any guarantee on when or how allocations happen, so we can’t know in advance how many allocations are required for an operation to succeed. By limiting the iterations we can guarantee that the test will terminate, and if the limit is so low that we don’t observe the allocation failure we want, the coverage report will tell us.

TEST_CASE ("checked_at") {
    const std::size_t len = 5;

    // 0 1 2 3 4
    auto* vec = make_sequence(len);

    CHECK (checked_at(vec, len, 0) == 0);
    CHECK (checked_at(vec, len, 4) == 4);
    CHECK_THROWS (checked_at(vec, len, len));

    // If this call allocates more than 1000 times, consider it failing
    for (std::size_t i = 0; i != 1000; ++i) {
        fail_new_after(i);
        try {
            checked_at(vec, len, len);
        } catch (std::bad_alloc&) {
            continue;
        } catch (std::out_of_range&) {
            // The function succeeds by throwing out_of_range
            break;
        }
        assert (false);
    }
}

What alternatives do we have to replacing the global new? For some functions we can do all allocations up front and never use functions that could throw, but that’s honestly not practical in C++. For example, vector.push_back would be out.

The standard C++ container types have an allocator parameter. From cppref:

template<
    class T,
    class Allocator = std::allocator<T>
> class vector;

It is tempting to think that we could achieve the same thing using the allocator, but it is unfortunately not that simple. The allocator is a static parameter, which means that std::vector<int, std::allocator<int>> is a different type than std::vector<int, failing_allocator<int>>. That means functions expecting a std::vector<int> cannot be called with the failing cousin, and since we won’t be exercising the same code this feature is not helpful for coverage. Here is an example program that shows the allocation parameter causing different overload selection.

#include <cstdio>
#include <vector>

template <typename T>
struct noisy_allocator : public std::allocator<int> {
    template <class U>
    struct rebind { using other = noisy_allocator<U>; };

    using std::allocator<T>::allocator;
    T* allocate(std::size_t n) {
        std::printf("Allocating %zu elements\n", n);
        return std::allocator<int>::allocate(n);
    }
};

int sum(const std::vector<int>& v) {
    int s = 0;
    for (auto x : v)
        s += x;
    return s;
}

int sum(const std::vector<int, noisy_allocator<int>>& v) {
    int s = 0;
    for (auto x : v)
        s += x;
    return s;
}

int main() {
    std::vector<int> quiet;
    std::vector<int, noisy_allocator<int>> noisy;

    for (std::size_t i = 0; i != 100; ++i) {
        quiet.push_back(i);
        noisy.push_back(i);
    }
    std::printf("quiet sum = %d, noisy sum = %d\n", sum(quiet), sum(noisy));
}

The sum() calls will resolve to different function calls. This can be clearly seen in the symbol table.

$ g++ alloc.cc -o alloc && ./alloc
Allocating 1 elements
Allocating 2 elements
Allocating 4 elements
Allocating 8 elements
Allocating 16 elements
Allocating 32 elements
Allocating 64 elements
Allocating 128 elements
quiet sum = 4950, noisy sum = 4950

$ nm -g alloc | grep sum
00000000004015be T _Z3sumRKSt6vectorIi15noisy_allocatorIiEE
0000000000401376 T _Z3sumRKSt6vectorIiSaIiEE

Even if these types had been overload-compatible it would not have solved the problem when code uses the standard containers in functions. For example, if we want to find the most frequent word in a text:

std::string wordcount(const std::vector<std::string>& text) {
    std::map<std::string, std::size_t> wc;
    for (const auto& word : text)
        wc[word] += 1;
    std::string* winner;
    std::size_t top = 0;
    for (auto& [word, count] : wc) {
        if (count > top) {
            top = count;
            winner = &word;
        }
    }
    return *winner;
}

This is quite straight forward C++, and the function variables use the default allocator. Even if we could drop in our failing allocator for the text argument, that would not change the map or strings in the map, which have have error paths involving allocation failure as seen in the CFG as paths ending in block 35-THROW. All exceptions that this function might throw are allocation failures.

wordcount(). The only exception that will be thrown by this code is bad_alloc

What we need is for the program to be more dynamic and configurable. The POSIX open_memstream provides a FILE handle compatible with the stdio API (fread, fwrite, etc.) backed by user-provided memory, which makes it easy to simulate truncated files. SQLite uses similar techniques to synthesize failed writes, failed reads, etc., all dynamically resolved through configurable handles. SQLite applies the same technique to allocations by consistently using its own configurable malloc. Going through a vtable does not invalidate testing or change the structure, unlike changing the type statically.

One way to introduce dynamism is to use the polymorphic allocators everywhere, and seed them with a failing memory resource in test suite. This is a more precise alternative than replacing the new itself, with a much smaller blast radius. Code would have to be specifically written to leverage the pmr types, which means it may interoperate poorly with third-party libraries, but it is worth keeping in mind.

In Zig there is a convention that an allocator is an explicit parameter to any function that may need to allocate. Allocator is an interface, which gives the caller the opportunity to select the allocation strategy at call time, which for testing may include leak monitoring and allocation failure. This is a rough port of the C++ function to Zig; note the explicit allocator argument.

const std = @import("std");
const Allocator = std.mem.Allocator;

fn wordcount(allocator: Allocator, text: []const [] const u8) ![]u8 {
    var wc = std.StringHashMap(usize).init(allocator);
    defer wc.deinit();

    for (text) |word| {
        const v = try wc.getOrPutValue(word, 0);
        v.value_ptr.* += 1;
    }

    var winner: *[]const u8 = undefined;
    var top: usize = 0;
    var it = wc.iterator();
    while (it.next()) |entry| {
        const count = entry.value_ptr.*;
        if (count > top) {
            top = count;
            winner = entry.key_ptr;
        }
    }

    return try allocator.dupe(u8, winner.*);
}

const expect = std.testing.expect;
test "basic test" {
    const text: [12][]const u8 = .{
        "foo", "bar", "baz",
        "fof", "bar", "biz",
        "foo", "bar", "bap",
        "oof", "rab", "zab",
    };
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    const allocator = gpa.allocator();
    const mostfreq = try wordcount(allocator, &text);
    defer allocator.free(mostfreq);
    try expect(std.mem.eql(u8, mostfreq, "bar"));
}

test "failing allocation" {
    const text: [12][]const u8 = .{
        "foo", "bar", "baz",
        "fof", "bar", "biz",
        "foo", "bar", "bap",
        "oof", "rab", "zab",
    };
    const allocator = failing_allocator();
    const mostfreq = try wordcount(allocator, &text);
    defer allocator.free(mostfreq);
    // should fail
    try expect(std.mem.eql(u8, mostfreq, "bar"));
}

In C++ we apply this design by using polymorphic allocator containers consistently and pass std::memory_resource to functions that might need to allocate. The explicit (or embedded, in the case of pmr) dynamic allocator is an example of dependency injection applied to memory.

Programming for coverage part 3

Sun, 24 Aug 2025 00:00:00 +0000

In this post we will work with the basic structure of a switch in a loop, and look at how slightly different decisions and designs affect the complexity and the testing requirements. This construct shows up in many programs, such as state machines, file format parsing parsing, command line argument processing, and event loops. It a consideration became sensitive to when I started measuring prime path coverage, and since I have used it both to significantly simplify my programs and squash some bugs.

This snippet shows the general structure, a loop with a switch body. It has two main sources of complexity; the number of cases (in this case, events), and the code handling each case. In this post we will focus on the number of cases.

while (true) {
    switch (next_event()) {
        case event_a:
            // process a
            break;
        case event_b:
            // process b
            break;
        // more events
        ...
        case event_quit:
            return;
    }
}

The switch and the if

Here is an example we can compile. These two functions are functionally the same, but one is implemented with the the switch, and the other conditionals. In both functions, each case is handled with a single function call, which is about as simple as it can be. This allows us to focus on the control flow implications of the switch-loop structure.

int next();

void dispatch_switch() {
    int e;
    while (e = next()) {
        switch (e) {
            case 1:
                puts("1");
                break;
            case 2:
                puts("2");
                break;
            default:
                puts("unknown");
                break;
        }
    }
}

void dispatch_ifs() {
    int e;
    while (e = next()) {
        if (e == 1)
            puts("1");
        else if (e == 2)
            puts("2");
        else
            puts("unknown");
    }
}

dispatch_switch()

dispatch_ifs()

As seen in the zcov screenshots, these functions have roughly the same number of prime paths (28 and 27), which makes sense since they’re mostly equivalent. In fact, given a few more cases and optimizations enabled, the ifs would likely be transformed to a jump table. The switch provides a useful piece of information to the reader; all comparisons are simple value equality and there are no overlapping values or precedence rules. Note how in the CFG representation, all the switch cases are laid out on the same level, which is how the case dispatch is understood. In this example, all the comparison operators are equality, but if we modify dispatch_ifs() slightly and change the comparison, the structure remains unchanged but the semantics are quite different.

void dispatch_ifs() {
    int e;
    while (e = next()) {
        if (e <= 1) // was ==
            puts("1");
        else if (e <= 2) // was ==
            puts("2");
        else
            puts("unknown");
    }
}

Now e = 0 prints 1 where it before printed unknown, as we have widened the first condition. Because there are only two cases we can tell all comparisons are equality, but this does not work when there are more than a handful, or when there is non-trivial bodies. The switch guarantees an equality comparison and communicates that as clear intent, while the ifs open up for anything. This is a good demonstration of how we can leverage the different properties and power of the constructs we choose in our programs for clarity.

Prime path growth and complexity

dispatch_switch_6() with 6 cases and 73 prime paths

If we double the number of cases in the switch (to six cases) the function gets 73 prime paths, up from 28, which shows that the prime paths grow faster than the cases. I have seen straight forward switch-loop constructs with roughly 15 000 prime paths. Recall that a prime path is a maximal simple path or maximal simple cycle. Each case label will be the first node in the paths that go through the loop header and to every other case, such as the emphasized paths in these screenshots. Both prime paths start at 4, case 1:, and go through the loop, but branch at 3, switch (e). Prime path coverage requires that both these paths are taken, but we can decide from reading the source that the sequence is uninteresting as long as next() can produce every value.

Prime path: 4 5 10 11 3 6 7

Prime path: 4 5 10 11 3 8 9

We also need to cover the path from every case to the loop exit. Depending on the program this may not even make sense at all as some programs are designed to loop until there is a quit or interrupt event.

while (more) {
    switch (next_event()) {
        case event_a:
            // do something
            break;
        // ...
        case event_quit:
            more = false;
    }
}
work();

Paths like event_a; while (!more); work() cannot be covered. That’s not necessarily a problem if we take this into consideration and decide it is acceptable, but it does means a portion of the prime paths will remain uncovered and we lose some sensitivity to changes in coverage. A small rewrite could fix the problem, by weaving the exit condition into the loop structure. The unreachable() is added to silence the warning on unhandled cases, but it could be removed without really changing the program.

for (event e = next_event(); e != event_quit; e = next_event()) {
    switch (e) {
        // ...
        case event_quit:
            unreachable();
    }
}
work();

Why even care about paths from a case, through the loop, and to some other case? The updated state from the previous step is an implicit input to every case in the switch at the current step. This specific function doesn’t really use any of that state, but the structure still suggests it might. Here is an example where it does:

int acc = 1;
for (; *c; ++c) {
    switch (*c) {
        case 1: acc = (acc + 1) + 1; break;
        case 2: acc = (acc + 1) * 2; break;
        case 3: acc = (acc + 1) * 3; break;
        case 4: acc = (acc * 1.5) / acc; break;
        case 5: // fallthrough
        default: acc = 0; break;
    }
}

If the sequence of values were *c >= 5, *c = 4 we would divide-by-zero in case 4, which is undefined in C. Executing this sequence is not required to achieve block coverage, edge coverage, or MC/DC, but it is required by prime path coverage. In fairness, prime path coverage only requires exercising such sequences goin through the looping once (otherwise the loop header repeat), so if a problem would only occur for the block sequence (5, 1, 4) it would still not be required by prime path coverage. The tests might accidentally exercise the path because of the required block sequences (5,1) and (5,4), or you might notice the problem in the process. Either way is clearly a win and in line with our goal of better programs.

Tweaking the structure

We can rewrite the switch-loop so that it both captures the intent and is more practical to test. First we need to understand what constraints we are working with.

Are the cases truly independent and does the order-of-processing matter?
Do we need to share data between cases?
Are the details of handling each case important?

We can split the iteration and processing by rewriting the first example and move the loop body to a separate function. Making a separate function obviously does not change the program semantics, but it greatly affects structure.

void dispatch1_process(int e) {
    switch (e) {
        case 1:
            puts("1");
            break;
        case 2:
            puts("2");
            break;
        default:
            puts("unknown");
            break;
    }
}

void dispatch1() {
    int e;
    while (e = next())
        dispatch1_process(e);
}

dispatch1_process()

dispatch1()

We can see the effect of this change with gcc -fpath-coverage --coverage dispatch.c -o dispatch.o and zcov. We went from a single function with 27 paths to two functions with 3 and 6 paths, respectively. Not only do we need fewer tests, roughly a third, but it is emphasized that each step is processed independently. There is no way (except through globals) for the consecutive calls to communicate or carry dependencies. We can test our processing function by calling it directly, and don’t need to bother setting up a stream and calling next(). All of these properties aid in testing. A test may end up looking like this, testing dispatch1_process() independently of the event generator:

void check_dispatch() {
    assert_prints(dispatch1_process(1), "1");
    assert_prints(dispatch1_process(2), "2");
    assert_prints(dispatch1_process(3), "unknown");

    // Skip the loop
    next_reset();
    dispatch1();

    // Loop twice
    next_reset();
    next_add(1);
    next_add(1);
    dispatch1();
}

Making state explicit

Sometimes we do need to maintain some state between steps. We extend the example and count how many times we have seen each case:

void dispatch2() {
    int e;
    size_t count[3] = {};

    while (e = next()) {
        switch (e) {
            case 1:
                count[0] += 1;
                printf("1: %d\n", count[0]);
                break;
            case 2:
                count[1] += 1;
                printf("2: %d\n", count[1]);
                break;
            default:
                count[2] += 1;
                printf("unknown: %d\n", count[2]);
                break;
        }
    }
}

Count is an implicit parameter to each case. If we were to split processing and iteration again we need to pass the state explicitly. We move the whole switch into a process function and add a size_t* argument. Thanks to inlining, no performance should be lost in optimized builds.

void dispatch2_process(int e, size_t *count) {
    switch (e) {
        case 1:
            count[0] += 1;
            printf("1: %d\n", count[0]);
            break;
        case 2:
            count[1] += 1;
            printf("2: %d\n", count[1]);
            break;
        default:
            count[2] += 1;
            printf("unknown: %d\n", count[2]);
            break;
    }
}

void dispatch2() {
    int e;
    size_t count[3] = {};
    while (e = next())
        dispatch2_process(e, count);
}

This structure is also letting your readers know that the count is indeed shared and updated in the processing step and that is clearly your intention to do so. In this case our state was a single variable and array. For more complex states I like using a bespoke struct for maintaining, but if there are only a couple of variables I usually just pass pointers to them.

Since its responsibility of the process function is to only process a single event, we can afford more sophisticated control flow and more code for each case code for each case. It opens up for using return which no longer terminates the loop, just the step, which may significantly simplify the control flow.

Returning to the example with divide-by-zero. If the switch is moved to a separate function we get this. This redesign is driven by the goal of reducing the number of prime paths as a proxy for complexity, and as a result problems stands out. Programmers are sensitive to division by unchecked inputs, but less so when the operand is a variable that is used in a complex context with a large surface area. This is a nice example of how even small restructurings “open up” for even more significant improvements.

int process(int acc, int c) {
    switch (c) {
        case 1: return (acc + 1) + 1;
        case 2: return (acc + 1) * 2;
        case 3: return (acc + 1) * 3;
        case 4: return (acc * 1.5) / acc;
        case 5: // fallthrough
        default: return 0;
    }
}

// original function
int acc = 1;
for (; *c; ++c)
    acc = process(acc, *c);

A real world example

Here is a real world example from GNU coreutils v9.7 od.c. This is the original:

static void
print_ascii (idx_t fields, idx_t blank, void const *block,
             MAYBE_UNUSED char const *unused_fmt_string, int width,
             idx_t pad)
{
  unsigned char const *p = block;
  idx_t pad_remaining = pad;
  for (idx_t i = fields; blank < i; i--)
    {
      unsigned char c = *p++;
      char const *s;
      char buf[4];

      switch (c)
        {
        case '\0':
          s = "\\0";
          break;

        case '\a':
          s = "\\a";
          break;

        case '\b':
          s = "\\b";
          break;

        case '\f':
          s = "\\f";
          break;

        case '\n':
          s = "\\n";
          break;

        case '\r':
          s = "\\r";
          break;

        case '\t':
          s = "\\t";
          break;

        case '\v':
          s = "\\v";
          break;

        default:
          sprintf (buf, (isprint (c) ? "%c" : "%03o"), c);
          s = buf;
        }

      idx_t next_pad = pad_at (fields, i - 1, pad);
      int adjusted_width = pad_remaining - next_pad + width;
      xprintf ("%*s", adjusted_width, s);
      pad_remaining = next_pad;
    }
}

And rewritten using state explicitly passed to a helper function:

static char const *
as_ascii_octal (unsigned char c, char *buf) {
  switch (c)
    {
    case '\0':
      return "\\0";
    case '\a':
      return "\\a";
    case '\b':
      return "\\b";
    case '\f':
      return "\\f";
    case '\n':
      return "\\n";
    case '\r':
      return "\\r";
    case '\t':
      return "\\t";
    case '\v':
      return "\\v";
    default:
      sprintf (buf, (isprint (c) ? "%c" : "%03o"), c);
      return buf;
    }
}

static void
print_ascii (idx_t fields, idx_t blank, void const *block,
             MAYBE_UNUSED char const *unused_fmt_string, int width,
             idx_t pad)
{
  unsigned char const *p = block;
  idx_t pad_remaining = pad;
  for (idx_t i = fields; blank < i; i--)
    {
      char buf[4];
      char const *s = as_ascii_octal (*p++, buf);
      idx_t next_pad = pad_at (fields, i - 1, pad);
      int adjusted_width = pad_remaining - next_pad + width;
      xprintf ("%*s", adjusted_width, s);
      pad_remaining = next_pad;
    }
}

Because all the cases fit in half a screen and simply assigned a literal to s, the value proposition is not immediately obvious, but I do find the revised version an improvement. If anything, it lets us focus on the other things going on in the loop (the padding and width adjustment) rather than the mapping from a character to its octal representation. If anything, prime path coverage is significantly simplified; all paths through as_ascii_octal() can be tested with a single run of print_ascii with the input "\0\a\b\f\n\r\t\vc\0x01".

Non-trivial cases

I want to include one last example in this post to show what happens when the cases include control flow by updating the previous example to only print on the first and then every 10th occurrence. This adds what I initially would think was moderate complexity, but this change takes us to 54 prime paths. The control flow in each case acts as a multiplier in a way, as the path case 1; if-then; (e); case 2: if-then is different from case 1; if-then; (e); case 2: if-else. These two paths can be seen in the screenshots. We can apply the same tricks are in the other examples to bring the complexity down, and the ability to do so would verify the independence of each case.

void dispatch3() {
    int e;
    size_t count[3] = {};

    while (e = next()) {
        switch (e) {
            case 1:
                count[0] += 1;
                if (count[0] % 10 == 1)
                    printf("1: %d\n", count[0]);
                break;
            case 2:
                count[1] += 1;
                if (count[1] % 10 == 1)
                    printf("2: %d\n", count[1]);
                break;
            default:
                count[2] += 1;
                printf("unknown: %d\n", count[2]);
                break;
        }
    }
}

Closing remarks

Most of the examples in this post are what I would call standard refactoring techniques and can neatly summed up as “move a block into a separate function”. That summary does not capture the process of deciding which blocks to move into separate functions, it describes the what and not the why. The prime paths and coverage is a data driven approach that approximates complexity by associating it with the number of prime paths that, when mixed with good engineering and sensibilities, guides testing and results in better programs (ideally).

The observations of this post actually don’t just apply to the switch-loop, they apply to any complex structure in a loop. The switch is a bit special because it is a simple dispatch based on simple values and often handle a lot of cases that makes loops larger, while each case at the same time has little overlap.

Breaking up sources of complexity also helps clean up the signal from your test suite. Having covered 12 out of 357 prime paths probably means that you cannot use that data to make a qualified decision on whether the structure is well exercised or if a key path is missing. 352/357 prime paths covered, means that you can look at the each of the missing coverage and determine if it is valuable, or even possible, to cover them.

Duff’s device

I want to include a showcase of maybe the most (in)famous switch-loop, Duff’s device. Looking at this construct with the zcov CFG visualization is a great way of demystifying the uncommon technique of putting a loop inside a switch where every case falls through. Complexity wise it’s not all that bad either, as Duff’s device only has 27 prime paths and mostly straight forward control flow.

void send(short *to, short *from, int count) {
    register int n = (count + 7) / 8;
    switch (count % 8) {
        case 0: do { *to = *from++;
        case 7:      *to = *from++;
        case 6:      *to = *from++;
        case 5:      *to = *from++;
        case 4:      *to = *from++;
        case 3:      *to = *from++;
        case 2:      *to = *from++;
        case 1:      *to = *from++;
    } while (--n > 0);
    }
}

Duff's device

Programming for coverage part 2

Mon, 18 Aug 2025 00:00:00 +0000

In this post we will use path coverage as a guide for testing a C function and explore how a light rewrite improves our ability to test it. Most testing is really about observing how the program responds to data, and coverage is mapping that response to the program structure. By changing the program we change the structure the data operates on, even when two programs are functionally equivalent.

We are writing a C program and need to know if today is the leap day, 29th February. This is the first draft and test:

bool leap_day() {
    const time_t now = time(NULL);
    const struct tm *tm = localtime(&now);
    return tm && tm->tm_mon == 1 && tm->tm_mday == 29;
}

void check() {
    assert(leap_day() == false);
}

The test is broken since it will fail on the actual leap day, when the function returns true. Time is an implied input to this function, but we cannot control it other than by doing things like changing the system clock. Bluntly, it has an interface that does not really support testing. Before addressing that, let’s look at the CFG with zcov:

Control flow graph of leap_day()

As expected, it has straight line flow until the conditional expression in the return statement. One neat thing about the CFG visualization is that phases of the function become so visible. In this case, the graph widens at block 4, before re-joining at 9. The 3-4 edge is a bridge, an edge that all paths of the (sub)graph must go through. Bridges are useful markers for natural places to split functions, and where to make strong assertions on state and the pre- and post conditions (so far). The conditional expression has four paths, [1 1 1] [1 1 0] [1 0 *] [0 * *], here shown in CFG form:

Path 1: 1 1 1

Path 2: 1 1 0

Path 3: 1 0 *

Path 4: 0 * *

To achieve path coverage we need to control the value returned by localtime(), which is derived from time_t now. If we lift the responsibility of computing now to the caller, we can control it from our tests:

bool leap_day(const time_t *now) {
    const struct tm *tm = localtime(now);
    return tm && tm->tm_mon == 1 && tm->tm_mday == 29;
}

void check() {
    time_t now;
    struct tm tm = {};
    // 2024 is a leap year
    tm.tm_year = 2024;

    // February, not 29th
    tm.tm_mon = 1;
    tm.tm_mday = 1;
    now = mktime(&tm);
    assert(leap_day(&now) == false);

    // 29th, not February
    tm.tm_mon = 3;
    tm.tm_mday = 29;
    now = mktime(&tm);
    assert(leap_day(&now) == false);

    // February 29th
    tm.tm_mon = 1;
    tm.tm_mday = 29;
    now = mktime(&tm);
    assert(leap_day(&now) == true);
}

These tests cover 3/4 paths as seen in the screenshot. We can add a fourth test for the case where the month is not February and the day not the 29th, but the coverage shows us that the test would not be all that meaningful. Notice in the rightmost pane that we have 100% block coverage and, because block coverage subsumes it, we have 100% line coverage. The path coverage shows us that we are not exercising the case of tm == NULL; 3-7 is the edge from the first check to the outcome.

3/4 paths covered

tm == NULL when localtime() fails. Here is the relevant section from the manual:

Upon successful completion, the localtime() function shall return a pointer to the broken-down time structure. If an error is detected, localtime() shall return a null pointer and set errno to indicate the error.

Ok, so let’s feed localtime() a bad time_t to cover this path. Testing for localtime() failure is important because we no longer control the call to time(). In fact, for testing we don’t even need to call time() at all, which makes the tests independent of the system clock, a nice property as it means they won’t suddenly fail on a different computer. As a fun side note, it was surprisingly hard to make localtime() fail. I tried populating now with bad data; negative day, month way out of range, even just setting it to (time_t)-1, but the call kept succeeding. I peeked at the values used in the overflow check in musl and added a test.

// Garbage time
now = (time_t)((INT_MAX * 31622400LL) + 1);
assert(leap_day(&now) == false);

Great, 4/4 paths covered, but something still feels off. We would be done if the goal is code coverage, but our goal is good software. This function still has a problem that the quest for coverage made us sensitive to; we have covered the path where localtime() fails, but the even when it did, leap_day() succeeded. This would be a hard bug to detect as the problem would only manifest on the leap day, roughly every four years. There is not enough room in a bool to signal the failure, which gives us a few options: either widen the return type (e.g. to an enum) and reserve a value for failure, add an out-parameter for the success/failure, use a global like errno, or apply the same trick again and have the caller call localtime(). I think the neatest solution is to lift out the localtime() call.

bool leap_day(const struct tm *tm) {
    return tm->tm_mon == 1 && tm->tm_mday == 29;
}

Excellent. What these changes boil down to is making all the input data explicit. The function is reduced to its essence, and coverage becomes trivial. The NULL check is gone as it common, and reasonable, in C to require the callers to only pass pointers to valid objects. While we shift the burden to call time() and localtime() onto the caller, errors can be handled where there is context to do so, and the caller can choose to use other functions like localtime_r() or gmtime() when it makes sense. This applies to our testsuite too, as we don’t even need call time() or localtime() at all to test leap_day():

void check() {
    struct tm tm = {};

    // February, not 29th
    tm.tm_mon = 1;
    tm.tm_mday = 1;
    assert(leap_day(&tm) == false);

    // 29th, not February
    tm.tm_mon = 3;
    tm.tm_mday = 29;
    assert(leap_day(&tm) == false);

    // February 29th
    tm.tm_mon = 1;
    tm.tm_mday = 29;
    assert(leap_day(&tm) == true);
}

leap_day() with full path coverage (right pane)

There you have it. The new leap_day() is obviously simpler to the point of being obsolete, but this is unfortunately the curse of examples. Designing interfaces that support testing is common wisdom, and this coverage guided testing effort shows both why it matters and how it improves programs. If we had settled for 100% line coverage and 3/4 paths we might not have detected the localtime() bug. By pushing possible failures up we can make functions pure and predictable, and handle (or ignore!) errors where it makes sense in the application.

A few remarks to close out this post, because in fairness the final leap_day() function does not do exactly what the original did. Let’s add a new function to restore the functionality:

/* -1 if error, 0 if not leap day, 1 if leap day.  If -1, errno describes the
   error.  */
int leap_day_today() {
    const time_t now = time(NULL);
    if (now == (time_t)-1)
        return -1;
    const struct tm *tm = localtime(&now);
    if (!tm)
        return -1;
    return leap_day(tm) ? 1 : 0;
}

void example() {
    const int leap = leap_day_today();
    if (leap < 0)
        perror("failed getting localtime");
    else if (leap > 0)
        puts("Hello, leap day!");
    else
        puts("Hello, not leap day!");
}

Unlike the first leap_day(), this function checks if the time() and localtime() calls succeeded. It widens the interface to return a raw int, but this could just as well be an enum. If knowing which supporting function failed is important for the application, it is probably better time() and localtime() explicitly and pass the result to leap_day(), and now we have that opportunity.

If we want to cover all the paths of leap_day_today() we’re back at square one, because time() and localtime() are not easily controllable as a caller. In that case we need to manipulate the global environment, for example by setting the system clock before running the test. This is still an improvement over the original revision because we don’t have to combine those tests with the testing of the leap_day() logic, which is firewalled off through by the function call.

Programming for coverage

Wed, 13 Aug 2025 00:00:00 +0000

I have been wanting to type up some notes on effective code coverage for a while. I am doing a talk at NDC Techtown 2025 on a few code coverage topics, which is an excellent opportunity to do so. These notes, and the talk, will touch on a few different topics; how coverage can guide testing, on effective metrics, coverage as a concept, and how different programming styles affect the possibility and feasibility of coverage.

Notes on coverage

For me, coverage sits in this odd place where it is both underappreciated and applied so poorly it becomes counterproductive. I think line coverage is partly to blame, which is just a poor metric we should deprecate. Line coverage feels like it is not worth the effort because full line coverage is hopelessly inadequate at finding defects, while at the same time being somewhat expensive to achieve. In my experience, some code is just hard to reach (high cost), such as out-of-memory errors or specific helper functions, and the payoff of just exercising each line is poor (low value). Lines may be perfectly fine in isolation, but be problematic when combined with other code (or data), which is why the context-less line coverage is so underwhelming.

A common “fix” for this is to set an arbitrary coverage target, say 80%, but this plays right into Goodhart’s law, commonly stated as when a measure becomes a target, it ceases to be a good measure. Code coverage is not a goal in itself, functioning and correct code is. Furthermore, not all code is created equal; there are segments of the program that warrants much more scrutiny and testing, but this is not reflected in coverage targets. If the goal is to identify “blind spots” that are (mostly) untouched by tests you don’t really need a target, you need to analyze the coverage report. Meeting the threshold may give a false sense of security; “this is well tested, we have 80% coverage” while important code slips through the cracks with poor test coverage. Code coverage doesn’t find defects, testing and validation does. Code coverage is quite useful for guiding testing.

Before we move on, a note on coverage requirements. Safety critical systems tend to require 100% MC/DC in order to ensure adequate testing. Avionics software has to comply with DO-178B and DO-178C which requires MC/DC for critical (Level A) systems, and the automotive standard ISO 26262 requires MC/DC for safety critical systems (e.g. brakes, not necessarily the music system). MC/DC does not guarantee a defect free system, but it does require a minimum number of effective tests. The 100% coverage requirement means that also the hard-to-reach code need full coverage.

More sophisticated criteria, like MC/DC and (prime) path coverage, require more test cases than simpler metrics like block or edge coverage, but are in return are excellent as guide for writing good test cases. Difficult to reach code tends to be difficult to write the test setup for, but once that is in place it becomes ok to write many test cases and use the stronger coverage metrics.

Should we design for coverage?

Should we write our programs with coverage in mind? I suppose it depends on what exactly “write with coverage in mind” means, and I don’t think there’s an obvious answer to that. Code is never written in a vacuum, and the programmer has to balance all sorts of properties; familiarity, clarity, speed, complexity, testability, etc. Coverage would be yet another such property, or maybe an aspect of testability, and writing testable code is generally a good thing. This in no way means we should prioritise coverage above all else; if “coverability” is at odds with clarity, it could be justified that the clarity is more important.

There are several metrics that try to objectively quantify the complexity (or testability) of a function, such as McCabe’s cyclomatic complexity, Nejmeh’s NPATH, and Bagnara et al. ACPATH. These metrics are closely related to counting the prime paths, as the prime paths are maximal simple paths or simple cycles. The simple path is by definition acyclic with no repeated nodes. By reducing the number of paths we reduce the complexity and the number of tests necessary to cover them.

One of the best way to reduce the cost of testing is to refactor. Put bluntly, adding a test “because coverage said so” is never really a good answer. I do think, however, “to make testing easier” is a very good reply to the question “why did you rewrite/refactor this code?”.

To summarise, the cost of testing and coverage is driven by how is easy it is to exercise a specific code path, and the number of test cases. This can be interpreted as the cost of testing is driven by the complexity of program. The ability to exercise specific code paths is closely related to function size, (global) state dependency, and the complexity of data- and control flow, all of which are good to minimize when possible.

Where does coverage fit in the development loop?

The unit tests are often (usually?) written in tandem with the code they support. This allows for early detection of problems and a sort of scratch pad for testing the interface of the unit, while providing some real-world feedback. Limiting structural complexity makes units easier to test; code is easier to reach, and there are fewer paths to exercise, yet coverage is often postponed or applied as an afterthought. Why?

I have lately experimented with weaving coverage into the write-compile-test cycle. I measure prime path coverage before and after running the test to verify that it actually runs the code I expect it to. I spend some time looking at the control flow graph, and try to figure out if it looks different than it should for the problem I’m solving. It has been a pleasant experience, and given me a lot of “huh, I never thought about it that way” moments, the occasional deep insight, and a fresh way of thinking about what I write.

While tempting, I try not to add test cases just because the coverage says a path is not covered. By first figuring out why the path isn’t already covered by the tests I have written, I can make an informed decision on whether that path is meaningful to test at all. For example, do we bother testing for allocation failures when resizing strings when we just propagate the exception? Maybe, it depends. Maybe there should not be an exception at all, and we should instead preallocate buffers outside the function, or use a different error signaling mechanism. In a way, the feedback from coverage does not just guide testing, it guides programming.

The classic coverage-when-done as a part of a larger validation and QA effort still makes sense, but I have found it quite nice to integrate coverage into the core programming loop.

Wrapping up

This will be the first post in a series on code coverage. The next posts will include more specifics on programming styles, techniques, refactoring, and how it all affects testing.

Construction junction, what's that function?

Fri, 20 Jun 2025 00:00:00 +0000

I saw something puzzling today while using zcov – a function I didn’t write that had no coverage. In my parsing code I have a small struct, something like this:

struct block {
    block();
    int id;
    int call_site;
    int nonlocal_return;
    std::vector<int> preds;
    std::vector<int> succs;
};

The block constructor shows up twice in the report, and only one is covered. Where does this second constructor come from, and why is it not run? Let’s look at it in the zcov CFG visualization:

This is the covered function:

And this is the non-covered function:

Ok, so the covered one is the implemented block constructor, but what about the other one? C++ implicitly defines the copy- and move constructors for us, and since they are implicit they are not anchored to the struct definition. Defining (and defaulting them) would should change this.

struct block {
    block();
    block(const block&) = default;
    block(block&&) = default;
    int id;
    int call_site;
    int nonlocal_return;
    std::vector<int> preds;
    std::vector<int> succs;
};

Looking at the non-covered CFG again immediately shows us that the second constructor is the move constructor:

There is still a few mysteries to solve. The coverage summary still only lists two constructors (the default and the move), but the copy constructor is defined. And why is the move constructor never called?

A piece of the puzzle is that the struct is defined in an unnamed namespace, so the compiler knows that it won’t be visible to the linker. No explicit copy of the struct is made anywhere, so it decides there is no need for the copy constructor. But what about the move? The struct is only used inside a vector, which does impose requirements depending on what operations are used. This is the relevant section since C++11 from cppref:

The requirements that are imposed on the elements depend on the actual operations performed on the container. Generally, it is required that element type is a complete type and meets the requirements of Erasable, but many member functions impose stricter requirements.

Since there are no struct block created outside the vector, I started looking for operations that would cause a copy or move. As it turns out, there is only one:

    fn->blocks.resize(nblocks);
    for (std::size_t i = 0; i != fn->blocks.size(); ++i)
        fn->blocks[i].id = getid();

Since we know ahead of time how many elements we need to construct we do it all in one go, and the blocks vector is only moved (pointer-swapped) after. GCC seems to not figure out that blocks is always empty before the resize call, which is why it has to emit the block move constructor, but since it never has to resize-and-move it is never called. We can confirm this by defining the copy and deleting the move constructor, and what do you know:

struct block {
    block();
    block(const block&) = default;
    block(block&&) = delete;
    int id;
    int call_site;
    int nonlocal_return;
    std::vector<int> preds;
    std::vector<int> succs;
};

Finally, I tried adding the items with vector.push_back, and now both constructors had 100% coverage.

This was a fun puzzle. Using coverage this way actually gave a me couple of interesting insights. We can observe that the use of the vector was efficient, in the sense that each vector instance allocated exactly once. This is one of those cases where missing coverage is a good thing, and shows that coverage is a useful tool for program analysis beyond simple pass/fail on 80% coverage. We can also observe that GCC only uses the move if it can get away with it (or not at all, like in this case), which is nice since the non-move constructor can throw and is more expensive. This mystery also shows that the demangling in zcov needs some work, because it would be nice to quickly see exactly which overload the entry represents.

Phantom conditions

Mon, 05 May 2025 00:00:00 +0000

This post is based on an interesting case NASA brought me; a small test program with a phantom condition. In the example, timed_out() is always true, which means checkout() cannot satisfy MC/DC, but let’s pretend it does something interesting. The function has one decision with two conditions, and four outcomes. Let’s measure MC/DC and confirm:

bool reset;
int count;

bool timed_out() {
     return true;
}

void checkout() {
    if (timed_out() && reset) {
        count = 0;
        reset = false;
    }
}

int main() {
    reset = true;
    count = 0;
    checkout();
    checkout();
}

$ gcc --coverage -fcondition-coverage checkout.cc -o checkout
$ ./checkout
$ gcov --conditions --stdout checkout.gcno
        -:    0:Source:checkout.cc
        -:    0:Graph:checkout.gcno
        -:    0:Data:checkout.gcda
        -:    0:Runs:1
        -:    1:bool reset;
        -:    2:int count;
        -:    3:
        2:    4:bool timed_out() {
        2:    5:    return true;
        -:    6:}
        -:    7:
        2:    8:void checkout() {
        2:    9:    if (timed_out() && reset) {
condition outcomes covered 3/4
condition  0 not covered (false)
condition outcomes covered 2/2
        1:   10:        count = 0;
        1:   11:        reset = false;
        -:   12:    }
        2:   13:}
        -:   14:
        1:   15:int main() {
        1:   16:    reset = true;
        1:   17:    count = 0;
        1:   18:    checkout();
        1:   19:    checkout();
        1:   20:}

Ok? There’s clearly just the one decision (if), but gcov reports two (the condition outcomes covered line). This is a ghost; a conditional jump we can measure with our tools which is absent from the code. The code is fine. The only thing that stands out, really, is the function call, so let’s replace it with a value. Now we get the one decision we would expect.

        2:    8:void checkout() {
        2:    9:    const bool expired = timed_out();
        2:   10:    if (expired && reset) {
condition outcomes covered 3/4
condition  0 not covered (false)
        1:   11:        count = 0;
        1:   12:        reset = false;
        -:   13:    }
        2:   14:}

The function-call-as-term obviously has an effect. Let’s also verify that it isn’t important that the function call is the first term. We swap the terms, and repeat the test, and still observe the mysterious second decision.

        2:    8:void checkout() {
        2:    9:    if (reset && timed_out()) {
condition outcomes covered 3/4
condition  1 not covered (false)
condition outcomes covered 2/2
        1:   10:        count = 0;
        1:   11:        reset = false;
        -:   12:    }
        2:   13:}

Let’s take a step back and revisit our assumptions about what it is we’re measuring and running. GCC does not actually use the source code much when it instruments for MC/DC, it mainly uses the control flow graph. This is very accurate with respect to what is executed¹, but come with some surprises in the gcov report. To be fair, this is a hard problem in gcov because it really only has source annotations to work with. To get a richer look at the report, let’s have a look at the report in zcov:

Quite interesting, indeed. Clearly, according to GCC there are two decisions here; blocks 3 through 5,6 and blocks 7 through 8,9. Blocks 3 through 5,6 has the shape we would expect from the Boolean expression, too. Note that the then/else blocks are successors of the later decision. This is a clue, another piece of the puzzle. It seems like GCC splits the decision in two; it first computes the result of timed_out() && reset, and stores it in temporary value, and jumps to the then/else after testing the temporary. The temporary is not explicit in the source, but the generated code is (rightly) attributed to the Boolean expression which the temporary is used to compute.

To confirm we can either check the assembly or dump the intermediate representation (gimple). Gimple is reasonably C-like, so we opt for that. The gimple confirms this theory; GCC uses the iftmp.1 for the “real” branch, and the mystery is solved. As to why GCC chooses to use a temporary, I simply don’t know.

$ gcc --coverage -fcondition-coverage checkout.cc -fdump-tree-gimple
void checkout ()
{
  bool retval.0;
  bool iftmp.1;

  _1 = timed_out ();
  if (_1 != 0) goto <D.2834>; else goto <D.2832>;
  <D.2834>:
  reset.2_2 = reset;
  if (reset.2_2 != 0) goto <D.2835>; else goto <D.2832>;
  <D.2835>:
  iftmp.1 = 1;
  goto <D.2833>;
  <D.2832>:
  iftmp.1 = 0;
  <D.2833>:
  retval.0 = iftmp.1;
  if (retval.0 != 0) goto <D.2836>; else goto <D.2837>;
  <D.2836>:
  count = 0;
  reset = 0;
  goto <D.2838>;
  <D.2837>:
  <D.2838>:
}

This was an interesting case and a reminder to be well aware of what we measure. At the end of the day, it is not really the source code that is executed, but a set of instructions equivalent to the semantics of the source code after a series of compiler transformations. A consequence of GCC analyzing the control flow graph to instrument for MC/DC is that sometimes other compiler details leak into the coverage, as we saw here. While the use of the temporary here is confusing, it is largely harmless. GCC still accurately measured MC/DC and require the same test vectors for full coverage. The extra one-term decision is a physical manifestation of the decision/jump in the source.

There are more details on measuring MC/DC based on control flow graphs in my paper on MC/DC in GCC. NASA gave me permission to write about this case.

At least at with optimizations disabled (-O0) ↩︎

GCC 15.1 released, and a paper on prime path coverage (in preprint)

Fri, 25 Apr 2025 00:00:00 +0000

GCC 15.1 was released today, and it comes with prime path coverage support! Try it out and build your programs with -fpath-coverage, it truly is a brilliant metric.

I wrote another paper detailing the implementation of the prime path coverage support. You can get a copy here or from arxiv.

Abstract:

We describe the implementation of the prime path coverage support introduced the GNU Compiler Collection 15, a structural coverage metric that focuses on paths of execution through the program. Prime path coverage strikes a good balance between the number of tests and coverage, and requires that loops are taken, taken more than once, and skipped. We show that prime path coverage subsumes modified condition/decision coverage (MC/DC). We improve on the current state-of-the-art algorithms for enumerating prime paths by using a suffix tree for efficient pruning of duplicated and redundant subpaths, reducing it to O(n²m) from O(n²m²), where n is the length of the longest path and m is the number of candidate paths. We can efficiently track candidate paths using a few bitwise operations based on a compact representation of the indices of the ordered prime paths. By analyzing the control flow graph, GCC can observe and instrument paths in a language-agnostic manner, and accurately report what code must be run in what order to achieve coverage.

I would like to emphasize two particular features of the paper; that prime path coverage subsumes MC/DC, and that suffix trees are fabulous improvement when enumerating paths. There are more details in the paper, and maybe I’ll do a separate post. One day.

Happy testing!

Prime path coverage merged into GCC

Thu, 27 Mar 2025 00:00:00 +0000

I am happy to announce that the prime path coverage support has been merged into GCC mainline. This was a continuation of my MC/DC project with NASA. I did most of the programming in the spring through autumn of 2024, followed by a long period of mostly testing and paper writing. NASA have been using GCC builds with this feature for about six months now, and they are quite happy with what they can learn from it. Process wise, it is an opportunity for simplification too – because prime path coverage subsumes most other coverage metrics we can now focus on a single metric.

Prime path coverage has grown on me, really. The common stance is that any sort of path coverage is way too expensive, i.e. requiring too many test cases, to even bother measuring. The more I have worked on this feature the less convinced I am that this is true. While path explosion is real it can be managed with refactoring and simplification, parametrized tests, property testing, and carefully considering input classes. Just as importantly, because we know what the prime paths are we can quite easily detect and understand non-local (that is, beyond a single line/statement/condition) interactions, which increases our sensitivity to potential problems.

So what’s next? I will probably follow up on the immediate issues from users (and buildbots). I also plan to write a few posts showcasing prime path coverage, what problems it can detect, and how to use it to guide refactoring and design.

Speaking at NASA

Tue, 18 Mar 2025 00:00:00 +0000

I gave a talk today at the Johnson Space Center in Houston, on the GCC coverage features I have done with them. The talk covered what prime paths and the motivation for prime path coverage, why they are an important development in dynamic analysis and how leveraging it will improve the quality of the testing we do of our systems. This marks the end of my two projects with them, MC/DC and prime path coverage, and it has truly been an incredible journey. The team at Orion have been absolutely brilliant to work with; they put trust in me when we scoped out the features and requirements, were great sparring parterns in development, and helped me test the features and gathered a lot of useful user feedback.

The teams at NASA have been using the prime path coverage support for over six months now, with good results. They have been able to accurately figure out what tests to write, found new defects, and used it as a guide to refactor code into cleaner and simpler programs. All of this work is in service of the goal of correct software and a safer spacecraft; I believe it will play an increasingly important role going forward in developing high quality and safe systems, quicker.

One day, after the flight, I will be able to look at the moon and think I was a small part of that. And that fills me with great pride and joy.

MC/DC in GCC paper (in preprint)

Wed, 08 Jan 2025 00:00:00 +0000

My paper on Modified Condition/Decision Coverage in GCC is published (in preprint)! You can get a copy from here or from arxiv. It covers the reasning, inner workings, and algorithms behind the -fcondition-coverage flag in GCC.

Abstract:

We describe the implementation of the masking Modified Condition/Decision Coverage (MC/DC) support in GCC 14, a powerful structural coverage metric with wide industry adoption for safety critical applications. By analyzing the structure of Boolean expressions with Binary Decision Diagrams we can observe the key property of MC/DC, the power to independently affect the outcome, and map to the edges of the Control Flow Graph. This mapping can be translated to a few bitwise instructions and enables GCC to instrument programs to efficiently observe and record when conditions have been taken and have an independent effect on the outcome of a decision. By analyzing the BDD rather than the program syntax, GCC can measure MC/DC for almost all of its languages with a single language-agnostic implementation, including support for C, C++, D, and Rust.

The design and programming of MC/DC support was initiated when I was at Woven by Toyota and finalised with funding from NASA.

Deprecating line coverage

Mon, 09 Dec 2024 00:00:00 +0000

Line coverage is typically the first structured metric looked at, and it provides a rough and imprecise measure of how thoroughly the test harness exercises the code. Briefly, line coverage is the answer to the question was code at this line ever run? The most important insight is finding the code that is not at all exercised in testing, but it can also be used as a coarse profiling tool by highlighting what lines are run more often and could warrants more optimization or other improvements. However, the unit of line is flawed in a couple of aspects, and it is time we stop using it and instead favour better metrics.

The line is too coarse

The first problem is the line itself. What does it mean exactly that a line is executed? Consider this C snippet:

double safediv(double a, double b) {
    if (b == 0) return 0.0;
    return a / b;
}

Clearly line 2 is executed regardless of the value of b, but it is not until b both zero and non-zero all of line 2 is executed. This issue would not be if the return 0.0 was on a separate line, but the function as it is written is perfectly fine. Most languages don’t really consider lines a construct of the language anyway, and even fewer as syntactically or semantically significant. For example, you can remove most newlines (and whitespaces) from C programs and it would not change anything for the compiler, which is used to great effect in the The International Obfuscated C Code Contest. The newline problem goes both ways, as seen in the GCC bug report 117415. This minimal example reproduces the problem on GCC 12.2:

int arr[10];
int *foo() {
    return arr;
}
int main(void) {
    int *a = arr;
    *foo()=
        *a;
    &a;
}

$ gcc ex.c --coverage -o ex && ./ex && gcov -t ex
1:    5:int main(void) {
1:    6:    int *a = arr;
2:    7:    *foo()=
1:    8:        *a;
-:    9:    &a;
-:   10:}

The first column is the execution count, and the *foo() = line apparently gets executed twice. This is obviously not the case, which can be demonstrated by sprinkling newlines everywhere:

$ gcc ex2.c --coverage -o ex2 && ./ex2 && gcov -t ex2
1:    5:int main(void) {
1:    6:    int *a = arr;
-:    7:    *
1:    8:    foo()
1:    9:    =
1:   10:    *a;
-:   11:    &a;
-:   12:}

A clear improvement over line coverage would be statement coverage: was this statement ever run? I choose to include expressions in this definition as it does not make too much sense to keep them apart. This would, along with decision coverage (has every if/boolean taken both then and else?), fix the attribution and counting problem, but it brings a few new problems: how to report coverage, the cost of coverage, and signal-to-noise.

The statement is too noisy

Let’s start with the easy one, the cost of coverage. Explicitly measuring every statement (and expression!) would mean that the instrumentation must interleave every operation, possibly with atomics, which would balloon binary size and make programs significantly slower. Some overhead is certainly to be expected, but nobody will turn on coverage if it means tests become too slow to practically run. Note that GCC today does not instrument lines either (which is why it gets the attribution wrong in the example), which will be discussed later.

Presenting the report is another problem. Line coverage is simple because it is sufficient to simply add a count column (or red/green for coverage) when annotating the source file, which is all the structure we need. Since statements may span multiple lines, and a line may have multiple statements, reports must now know how to break up or overlay parts of a line, which breaks the relationship with the original source.

Finally, there’s signal-to-noise. The ratio of covered/uncovered is a useful data point when deciding what module, file, or function to investigate. Measuring statements does not interfere with this, but not all statements are created equal. Some statements carry much more weight in the program, especially those in precondition checks and error handling paths. Measuring statements skews ratios towards long series of straight line code, happy paths, and underemphasise short branches. Consider this program with roughly 20 statements and expressions:

int main() {
    int a = foo1();
    int b = foo2();
    int c = foo3();
    int d = foo4() + foo5();
    int e = a + b*c - d;
    int f = e > b ? foo6() : foo7();
    if (err(f)) exit(EXIT_FAILURE);
}

I would say coverage-wise this has three main points of interest: foo6(), foo7(), and the exit() (which, granted, correspond to decision coverage). Missing either of those would still give 19/20 statements which looks like a fairly well covered function and underemphasises an important part of it. Worse yet, statement coverage can cover all statements and never exercise the case where err(f) is false and the normal end-of-main is reached. Oops.

Blocks all along

Enter the basic block. The research in this area does not really consider line or even statement coverage very much. Rather, the fundamental piece is the basic block. A basic block is defined as an uninterruptible stream of instructions such that when you enter a basic block, you will execute every instruction in order until the end of the basic block. This is how the compiler analyses the program, and how coverage is measured by GCC. Instead of recording every statement, GCC injects code on the entry- and exit of basic blocks. The basic blocks are then mapped back to source lines, which sometimes get wrong like in the example in this post.

I think the basic block is the obvious unit for coverage, and that node coverage (or block coverage) should take over as the most basic metric. Node coverage is the answer to the question has every node been visited? This answers all the problems brought on by statement coverage. By achieving block coverage you also achieve line coverage - every line must be a part of a block, and by visiting all block all lines must have been visited, too.

Per-node instrumentation is already how GCC measures coverage, and the overhead is acceptable. Instructions are only inserted at the beginning or end of a block, and there are far fewer blocks per function than statements. Small functions tend to have less than 10 blocks, and complicated functions may have a few hundred. There is no limit on the number of blocks, but for most practical purposes the number of blocks can be considered small, and by extension, cheap.

Signal-to-noise is also good as the contribution of a block is not dependent on the number of instructions in a block. A long sequence of non-branching code will usually be represented as a single block, where an 4-term conditional usually ends up as 4 blocks.

Finally, there’s reporting and presentation. A table summary is not really affected by the unit (line vs statement vs blocks). The table is good for overview and to figure out where to focus, but the real value of coverage comes from understanding the interactions between code, tests, and data, which builds on relationships that are not always clear from the line-column constraints of the text file. The graph is a powerful aid for this.

On zcov

The preference for blocks over lines is not accidental - it is the fundamental idea in zcov, to abandon the traditional line coverage metrics in favour of block oriented ones, provide useful, intuitive, and powerful visualisations for them, and good querying support.

In fact, the zcov report summary does not even include line coverage at all, it has been replaced with block coverage. Here are some screenshots of the latest developments:

This is the summary of all the files in the loaded report. Relative progress is indicated with progress bars with the coverage count as labels. I plan to add colour support, sorting and filtering, and more, but already it is very useful for quickly interpreting the state of the coverage.

This is a simple graph. I have re-done most of the node rendering and extended the header to show the basic block ID, plus room for other interesting data such as the execution count. This view, too, will get querying and filtering support, plus highlighting of covered/uncovered nodes, paths, etc.

This is a neat demonstration of block-based coverage and divorcing lines from coverage. Block 15 has many more statements than the blocks around it, and you can clearly see the data flow and the effect of taking branches. The exit() is even clearly captured as a leaf. It still maps code to blocks at the line level, so the repeated ifs look a bit odd, something I hope to improve in the future.

zcov is developing rapidly. For inquiries and purchases, reach out to j@patch.no, and see the software page for more details.

Early access: zcov

Mon, 21 Oct 2024 00:00:00 +0000

I am excited to announce my latest project, under the name zcov. It is a program for interpreting and appreciating coverage, with the goal of being the only tool you need for working with compiler instrumented coverage. It is still under development, but I have already been able to use it to understand and explain subtle program interactions, the path coverage view being particularly powerful.

Planned features include MC/DC explainer, a way to explore the structure of a conditional expression and how the terms interact and mask, rich flow graph annotations, function search and filtering, path search and filtering, and HTML reports, plus a long list of quality of life features. You can influence and request features already now by purchasing early builds - see the software page for details, and reach out to j@patch.no.

Bug hunt: Independent functions crashing gcov

Thu, 03 Oct 2024 00:00:00 +0000

NASA reported an odd crash in gcov, and I found it interesting and deep enough to warrant describing the root cause and the path it took me on. The actual crash itself was quickly found with gdb; reading the block.locations.lines when printing the source lines of a path when the array was empty (with size()-1, even!). A fix would be to not do that e.g. by guarding the access, but the question is why the array is empty in the first place.

The crash seems to happen when all of the following are true:

You have two classes with a method returning a static instance (singleton). The method does not need to be static.
The method’s name (symbol) is on the same line in both files.
The methods are both implemented in the header, directly in the class.
The class itself has a non-trivial member for a non-trivial destructor.

If any of these change gcov doesn’t crash anymore, but some more digging seems to confirm that the problem is deeper in gcc/gcov. If I move the implementation to the source file, gcov prints this. Notice the static A.

function B::getInstance called 0 returned 0% blocks executed 0%
paths covered 0 of 3
path 0 not covered:
BB  2:          11:B& B::getInstance(){
BB  2: (true)   12:    static B theInstance;
BB  3:          12:    static B theInstance;
BB  4: (true)   12:    static B theInstance;
BB  5:          12:    static B theInstance;
BB  6:           7:    static A theInstance;
BB  6:          12:    static B theInstance;
BB  7:          12:    static B theInstance;
BB  8:          13:    return theInstance;
BB  9:          13:    return theInstance;

Clearly two functions are mixed up, and the problem was only shown by path coverage, not caused by it. To reproduce:

// fst.h
#include <string>
struct fst {
    fst() {}
    static fst& instance() {
        static fst self;
        return self;
    }
    std::string data;
};

// snd.h
#include <string>
struct snd {
    snd() {}
    static snd& instance() {
        static snd self;
        return self;
    }
    std::string data;
};

// test.cc
#include "fst.h"
#include "snd.h"

int main(int args, char **argv) {
    fst::instance().data = "foo";
    snd::instance().data = "bar";
}

$ gcc --coverage -fpath-coverage test.cc -o test
$ gcov --stdout --prime-paths-source test --include=instance

Interestingly, this seems to only crash in one of them.

$ gcov --prime-paths-source test --stdout --include=fst # works
$ gcov --prime-paths-source test --stdout --include=snd # crashes

This is a result of inclusion order; if fst.h is included before snd.h, it crashes in snd, and vice versa. This is a good hint, so I dump the CFG from gcc. While these are unsurprisingly similar there is no obvious hint here.

At this point I started digging into gcc itself, since it seems gcov does the right thing given its input. I have gcov dump out the line mapping it reads from the .gcno file:

reading tag-line: lineno = 0, block = 2, file-name = snd.h
reading tag-line: lineno = 0, block = 3, file-name = snd.h
reading tag-line: lineno = 0, block = 4, file-name = snd.h
reading tag-line: lineno = 0, block = 5, file-name = snd.h
reading tag-line: lineno = 0, block = 6, file-name = snd.h
reading tag-line: lineno = 0, block = 6, file-name = fst.h
reading tag-line: lineno = 0, block = 7, file-name = snd.h
reading tag-line: lineno = 0, block = 8, file-name = snd.h
reading tag-line: lineno = 0, block = 9, file-name = snd.h
reading tag-line: lineno = 0, block = 2, file-name = snd.h
reading tag-line: lineno = 0, block = 3, file-name = snd.h
reading tag-line: lineno = 0, block = 2, file-name = snd.h
reading tag-line: lineno = 0, block = 3, file-name = snd.h

reading tag-line: lineno = 0, block = 2, file-name = fst.h
reading tag-line: lineno = 0, block = 3, file-name = fst.h
reading tag-line: lineno = 0, block = 4, file-name = fst.h
reading tag-line: lineno = 0, block = 5, file-name = fst.h
reading tag-line: lineno = 0, block = 6, file-name = fst.h
reading tag-line: lineno = 0, block = 7, file-name = fst.h
reading tag-line: lineno = 0, block = 8, file-name = fst.h
reading tag-line: lineno = 0, block = 9, file-name = fst.h
reading tag-line: lineno = 0, block = 2, file-name = fst.h
reading tag-line: lineno = 0, block = 3, file-name = fst.h
reading tag-line: lineno = 0, block = 2, file-name = fst.h
reading tag-line: lineno = 0, block = 3, file-name = fst.h

Ok, so the problem might be in the .gcno file - notice there is a single fst.h in the block of snd.h, but not the other way around. That probably means that means that gcc either records the source plain wrong, implicitly shares implementation (by accident), or implicitly shares implementation (on purpose).

I dump some diagnostics at the the corresponding writes in gcc where this is recorded:

diff --git a/gcc/profile.cc b/gcc/profile.cc
index c3d06c4d387..877d6ca0ac9 100644
--- a/gcc/profile.cc
+++ b/gcc/profile.cc
@@ -68,6 +68,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "file-prefix-map.h"

 #include "profile.h"
+#include "gimple-pretty-print.h"

 struct condcov;
 struct condcov *find_conditions (struct function*);
@@ -1108,6 +1109,9 @@ output_location (hash_set<location_triplet_hash> *streamed_locations,
       prev_file_name = file_name;
       gcov_write_unsigned (0);
       gcov_write_filename (prev_file_name);
+      fprintf (stderr, "\n\nname-differs -- prev = %s, self = %s\n",
                prev_file_name, file_name);
+      gimple_dump_bb (stderr, bb, 0, TDF_NONE);
     }
   if (line_differs)
     {

name-differs -- prev = snd.h, self = snd.h
<bb 6> :
_11 = 1;
__cxxabiv1::__cxa_atexit (__dt_comp , &self, &__dso_handle);

name-differs -- prev = fst.h, self = fst.h
<bb 6> :
_11 = 1;
__cxxabiv1::__cxa_atexit (__dt_comp , &self, &__dso_handle);

[...]

name-differs -- prev = fst.h, self = fst.h
<bb 6> :
_11 = 1;
__cxxabiv1::__cxa_atexit (__dt_comp , &self, &__dso_handle);

So snd::instance gets two __cxa_atexit prints, one from fst.h for some reason, and fst::instance does not. This is the root cause as far as gcov is concerned - gcc either generates a surprising entry, or a bogus one. This leads to an empty block.locations.lines which turns into a bad access that crashes gcov. My best guess is that it duplicates the basic block for static destructors which happens to preserve the first source reference.

for (unsigned i = 0; i != loc.lines.size () - 1; ++i)
  {
    const unsigned line = loc.lines[i];
    fprintf (gcov_file, "BB %2d: %-7s %3d", bb, "", line);
    print_source_line (gcov_file, lines, line);
  }

But this raises a few more questions. The locations array is not empty for the fst block because the __cxa_atexit is associated with it and there is no file change. Effectively, the line is dropped from the coverage report and fst prints like you would expect. This is not strange, since it is compiler generated to guard the static. The question now is, does this happen consistently for other singletons, and why is this even a problem for snd?

I added yet another singleton (lst), which also got two __cxa_atexit entries printed, but neither snd nor lst got a third entry, and fst still only had its one. This indicates that the problem does not change except when going from one to two statics.

The next step was to count the invocations of get_atexit_node(), the function which creates represents the __cxa_atexit calls, which is 1. It is in turn called from register_dtor_fn which is called once per static instance.

Tracking it to where basic blocks are built (tree-cfg.cc/make_blocks_1) confirms this: printing every statement assigned to the block including its location gives these notable statements:

[snd.h]: 4:20 D.35685 = 1;
[fst.h]: 4:20 __cxxabiv1::__cxa_atexit (__dt_comp , &self, &__dso_handle);
[snd.h]: 4:20 __cxa_guard_release (&_ZGVZN3snd8InstanceEvE4self);

...

[fst.h]: 4:20 D.35637 = 1;
[fst.h]: 4:20 __cxxabiv1::__cxa_atexit (__dt_comp , &self, &__dso_handle);
[fst.h]: 4:20 __cxa_guard_release (&_ZGVZN3fst8instanceEvE4self);

This pretty much concludes the hunt for the root cause. The same object (or at least a plain copy) is reused for the __cxa_atexit call, which happens to preserve the source of where it was first generated. This call itself is not anchored too strongly to a line in the source (well, when the static is constructed, but it is still implicit) and the association to an otherwise unrelated file is then misinterpreted by gcov as inlining. The __cxa_atexit call being shared is not really a problem, it is compiler generated after all, which means the right thing for gcov to do is to skip empty location arrays. This was first proposed fix, but now it is righteous.

Trip report: NDC Techtown 2024

Sun, 15 Sep 2024 00:00:00 +0000

This year’s Techtown conference in Kongsberg is through, and I wanted to do a trip report of sorts. There were ~400 attendees, the highest attendance yet, and an all-around fantastic atmosphere. I learnt a lot and had the privilege of striking conversations with many brilliant people, on diverse topics such as language design, compiler development, compiler features, embedded constraints, programming techniques, and I am already excited for next year. The programme was superb and I cannot think of a single slot where which talk to attend was an obvious choice. These are the ones I went to and a few thoughts on them.

It all opened with Anastasia Kazakova’s keynote on addressing language pain points with tooling. I’m happy this year’s keynote was on tooling considering my very own talk this year was on the new prime path coverage feature in GCC, and it certainly set the right tone for the conference as far as I am concerned. It is very impressive what tools can do, even in a language like C++, when given enough understanding of the language and environment. I am not yet personally convinced of the autocomplete family of features, but there is without a doubt real benefits to instant feedback.

I went to see Robert Seacord’s talk on the correct use of integers in safety-critical systems. His talks has changed my programming before, and I generally support his position of almost always preferring standard integer types over always using intN_t. That, sadly, meant missing Hana’s talk on her favourite data structures, but she was kind enough to give me a rundown later plus showing off her brilliant visualisations. I recommend watching the talk just for that. It is a good primer on data structures (as in available library implementations) as well as computer science, too. After a few chats I apparently also missed out by not attenting Piotr Wierciński’s talk Demystfying process address space: Heap, Stack, and Beyond.

My own talk, Path coverage in gcc was up next, where I made a case for measuring prime path coverage, and how to do so using the upcoming GCC feature. It was well received with a nice Q&A, which is really all I can ask for. It also gave me the chance to be approached by many throughout at the hallway track with even more questions and some genuine insights. Good stuff.

Robert Seacord had another talk, this time reflecting on the state of C, C++, and Rust (and other languages, really, but Rust was the big one) for safety critical systems. I found it particularly interesting as it addressed some of the practical challenges when looking to replace C and C++ in the industry. Tooling and tooling experience shows up again, as well as the mountains of existing code, and neither is odd considering C’s 40 year head start. Unfortunately, this meant missing both Martin Ertsås and Björn Fahller, but that’s how it is sometimes. I am happy the talks are recorded.

Darrel Raines of NASA had a fabulous talk on the realities and constraints of embedded development for space flight. Software and computers have been a part of their missions since day one, and in some cases very modest hardware in a hostile environment where failure to perform correctly can be fatal, and replacing and upgrading can be extremely difficult. For one, it makes you appreciate getting things just right, and also appreciate the discipline and process that makes it happen.

Day two opened with JeanHeyd Meneid’s talk on adding operator overloading for C. I think it is an interesting proposal and I am certainly not opposed to operators, and the proposed solutions seem to address the common criticisms. It’s neat, too, because it really just feels like a beefy macro system and not overly magical, but the big question is whether or not the committee will actually accept it. I think my main worry is that if it does get voted in it will never really be used because 1. people will also target older standards and 2. the programs and domains that benefit the most from operator overloading (which C already has, anyway) aren’t usually written in C anymore.

Theodor Thornhill did a talk on performance tracing and application debugging using (processor built-in) tracing rather than sampling. He showcased it both for guiding his Zig program for the 1 billion row challenge, as well as finding a real performance problem in a distributed application. I found the latter particularly interesting. I really wanted to see Tina Ulbrich and Hendrik Niemeyer’s talk on green programming, too, and look forward to the recording.

I rounded up the talks with two talks by young, first-time speakers, Paulo Martinez’ and Joakim Lier’s talks on binary size and compilers respectively. The talks were quite enjoyable and I am sure they will both keep on giving excellent talks in the coming years.

Finally, there was Sean Parent’s locknote on local reasoning in C++. Communication is probably the most important aspect when programming and designing interfaces, and he is very much on point in that just poor signatures alone prohibit local reasoning. I would recommend this talk to anyone, if anything just to freshen up on the implications of various parameter types and qualifiers.

Before the main conference event I did the two-day training on git with an excellent group of students. They were curious, inquisitive, and quick, and we spent two whole days training on how to better leverage git when programming We went over techniques and philosophy when crafting commits, merging, history rewriting, cherry-picking, log querying, transports, and more.

The Oslo C++ User Group upheld the tradition of having a meetup the day before the conference, and this year we got to do a book launch – Anders Knatten’s “C++ Brain Teasers”. There was a fun interview session with him and Olve Maudal, and we were treated with a couple of puzzles. This was interesting and educational as is tradition.

That wraps up the trip report for Techtown 2024, and now beings counting days until next year’s event.

Progress report - prime path coverage in gcc 3

Thu, 08 Aug 2024 00:00:00 +0000

The prime path coverage support is shaping up nicely, so I wanted to write a bit about the last series of improvements to the upcoming prime path coverage feature in gcc.

gcov reports

Path coverage differs from statement, branch, and condition coverage in that coverage ties to the function and not individual lines. This is quite different from how gcov has reported up until now. Furthermore, it is not necessarily easy to look at some code and figure out even what the paths even are, in particular after compiler transformations.

I have written three different reporting formats: summary, dense, and detailed.

Summary

The summary is always included, and is quite simply:

        -:    0:Source:tmp.cpp
        -:    0:Graph:tmp.gcno
        -:    0:Data:-
        -:    0:Runs:0
        -:    1:#include <stdio.h>
        -:    2:
paths covered 0 of 15
    #####:    3:int main ()
        -:    4:{

This is nice when glancing at a file to get an overview of the state of the coverage. The --function-summaries flag has also been taught how to include path information:

Function 'usage'
Lines executed:0.00% of 5
Branches executed:0.00% of 4
Taken at least once:0.00% of 4
Calls executed:0.00% of 3
Prime paths covered:0.00% of 4

Function 'setoutput'
Lines executed:0.00% of 8
Branches executed:0.00% of 6
Taken at least once:0.00% of 6
Calls executed:0.00% of 3
Prime paths covered:0.00% of 4

Function 'main'
Lines executed:0.00% of 358
Branches executed:0.00% of 192
Taken at least once:0.00% of 192
Calls executed:0.00% of 76
No path information

Note the No path information in main. This happens when a function has too many paths and gcc gives up instrumenting it, a mechanism to cope with path explosion to keep compile times low. The main in question has about a million paths which is more-or-less infeasible to cover anyway.

Dense

Moving on to the dense format. This format is line-oriented to work well with grep, while at the same time having enough information to figure out what’s going on.

$ gcov -te --prime-paths-lines tmp
        -:    0:Source:tmp.cpp
        -:    0:Graph:tmp.gcno
        -:    0:Data:-
        -:    0:Runs:0
        -:    1:#include <stdio.h>
        -:    2:
paths covered 0 of 15
path  0 not covered: lines 8 8(true) 9
path  1 not covered: lines 8 8(false) 11(true) 11 13(true) 13(true) 14 17
path  2 not covered: lines 8 8(false) 11(true) 11 13(true) 13(false) 16 17
path  3 not covered: lines 8 8(false) 11(true) 11 13(false) 16 17
path  4 not covered: lines 8 8(false) 11(false) 11 13(true) 13(true) 14 17
path  5 not covered: lines 8 8(false) 11(false) 11 13(true) 13(false) 16 17
path  6 not covered: lines 8 8(false) 11(false) 11 13(false) 16 17
path  7 not covered: lines 9 8(true) 9
path  8 not covered: lines 9 8(false) 11(true) 11 13(true) 13(true) 14 17
path  9 not covered: lines 9 8(false) 11(true) 11 13(true) 13(false) 16 17
path 10 not covered: lines 9 8(false) 11(true) 11 13(false) 16 17
path 11 not covered: lines 9 8(false) 11(false) 11 13(true) 13(true) 14 17
path 12 not covered: lines 9 8(false) 11(false) 11 13(true) 13(false) 16 17
path 13 not covered: lines 9 8(false) 11(false) 11 13(false) 16 17
path 14 not covered: lines 8(true) 9 8
    #####:    3:int main ()
        -:    4:{
        -:    5:  int i, total;
    #####:    6:  total = 0;
        -:    7:
    #####:    8:  for (i = 0; i < 10; i++)
    #####:    9:    total += i;
        -:   10:
    #####:   11:  int v = total > 100 ? 1 : 2;
        -:   12:
    #####:   13:  if (total != 45 && v == 1)
    #####:   14:    printf ("Failure\n");
        -:   15:  else
    #####:   16:    printf ("Success\n");
    #####:   17:  return 0;
        -:   18:}

In this mode gcov will list every uncovered path and the lines in the order you need to execute them to cover the path. If there is a decision on that line, gcov will tell you what outcome it expects for the decision. The decision could be inferred from the next line in the sequence, but I found that hard to follow and unecessary friction. If a transition happens to be a throw rather than a decision, gcov will print that, too. Note that gcov only prints the last line in each basic block, which is why there are no paths thart start with line 4 or 5.

I find this mode very useful when playing with different inputs and following how paths are eliminated. The fact that it plays well with grep means it is easy to quickly determine if an input covered a specific path, without having to manually inspect. Something along the lines of gcov --stdout --prime-paths-lines --include main | grep "path 11". Because it is denser it is easier to map several paths to the source code.

Detailed

Finally, there’s my favourite mode, the detailed --prime-paths-source, which gives me x-ray goggles for paths. This mode is similar to --prime-paths-lines in the sense that it will show you the lines and decisions you need to run to cover a path, but it will show it alongside the source itself. Here is an example for paths 2, 3, and 4 from the dense example:

path 2:
BB  2:           3:int main ()
BB  2:           6:  total = 0;
BB  2:           8:  for (i = 0; i < 10; i++)
BB  4: (false)   8:  for (i = 0; i < 10; i++)
BB  5: (true)   11:  int v = total > 100 ? 1 : 2;
BB  6:          11:  int v = total > 100 ? 1 : 2;
BB  8: (true)   13:  if (total != 45 && v == 1)
BB  9: (false)  13:  if (total != 45 && v == 1)
BB 11:          16:    printf ("Success\n");
BB 12:          17:  return 0;

path 3:
BB  2:           3:int main ()
BB  2:           6:  total = 0;
BB  2:           8:  for (i = 0; i < 10; i++)
BB  4: (false)   8:  for (i = 0; i < 10; i++)
BB  5: (true)   11:  int v = total > 100 ? 1 : 2;
BB  6:          11:  int v = total > 100 ? 1 : 2;
BB  8: (false)  13:  if (total != 45 && v == 1)
BB 11:          16:    printf ("Success\n");
BB 12:          17:  return 0;

path 4:
BB  2:           3:int main ()
BB  2:           6:  total = 0;
BB  2:           8:  for (i = 0; i < 10; i++)
BB  4: (false)   8:  for (i = 0; i < 10; i++)
BB  5: (false)  11:  int v = total > 100 ? 1 : 2;
BB  7:          11:  int v = total > 100 ? 1 : 2;
BB  8: (true)   13:  if (total != 45 && v == 1)
BB  9: (true)   13:  if (total != 45 && v == 1)
BB 10:          14:    printf ("Failure\n");
BB 12:          17:  return 0;

Here we see the basic blocks (BB), decisions, line number, and source code in the order it needs to be run in order to cover. I have played a bit with it and I find it an excellent tool for understanding exactly what the paths are, and find interesting interactions, and I am always surprised at what I learn when I use it. For example, I tested it on a binary search with some obvious inputs: a few hits, misses, lower-than-lowest, higher-than-highest. The test suite seemed ok, but I never actually tried the item in the exact middle, the one with no jumps. With this, I noticed right away.

This output is also great for identifying and understanding code which is made unreachable by earlier decisions. This can be approached both ways, even - if you expect code to be unreachable, run a lot of inputs and verify that it is never actually taken. If it disappears from the coverage report (which means it is now covered), it would be reachable under certain conditions. Likewise, if you run lots of different inputs, but never seem to cover that one path it could be worth investing if it is accidentally path sensitive.

Both the dense and detailed reports are aware of inlining, which is particularly relevant for C++, and would otherwise make the annotated source difficult to follow. This is what it looks like – notice how the basic blocks weave through the inlined function:

BB  2: (true)    4:int notmain(const char *entity)
== inlined from hello.h ==
BB  2: (true)    6:  if (s)
BB  3:           7:    printf ("hello, %s!\n", s);
BB  5:          10:  return 0;
-------------------------
BB  7:           6:  return hello (entity);
BB  8:           6:  return hello (entity);

The inlined code is rendered directly in the caller, which would otherwise print line numbers and decisions which does not actually map to the source line, something like this:

BB  2: (true)    4:int notmain(const char *entity)
BB  3:           4:int notmain(const char *entity)
BB  5:           4:int notmain(const char *entity)
BB  7:           6:  return hello (entity);
BB  8:           6:  return hello (entity);

I find the former easier to follow, personally.

Limits and Robustness

I did a rebuild-the-world experiment this time, too, like I did for MC/DC, all the way through tree, cpio, git, and openjdk with dependencies. This did uncover a few cases which crashed, mostly with setjmp/longjmp. These have been fixed.

Sometimes, functions are so complex with so many paths that the only thing we can do is give up. The problem is that there is no obvious relationship between graph size (number of nodes and edges) and the number of paths, and even small graphs may explode in size. For example, the number of paths in a series of $n$ non-nested ifs is $2^n$.

My solution has been to introduce a limit with -fpath-coverage-limit which currently defaults to 250000. We keep track of an approximate running count of number of paths seen so far, and apply a few heuristics. When the number of paths exceeds the limit, the function is aborted and gcc warns that it gives up coverage. If you are willing to work with the compile time, gcc will let you increase this limit, at the cost of compile time (and memory). This is a blunt instrument as gcc will overestimate the number of paths, but it is meant as a failsafe for exploding compile times, and not as fine-grained control on which functions to instrument or not.

I would say that if you are aiming for path coverage it is likely than anything about a few thousand paths is a no-go anyway, and I hope this will not be a problem in practice. In fact, I would say that aiming for path coverage forces you to write in a very specific style, which should be part of the motivation for shooting for path coverage anyway.

Closing

There has been a some minor performance improvements, bugfixes, and json output. This work will be the topic of my NDC Techtown talk this year, which I am very much looking forward to. The patch has been posted on the gcc mailing list for review, and I hope to get it merged with time to spare for gcc 15.

Progress report - prime path coverage in gcc 2

Tue, 25 Jun 2024 00:00:00 +0000

It’s time for another progress report. The last post covered the phases at a high level – please read that first for an overview.

Observing paths

To observe paths we associate a bitset with each function, where each bit corresponds to a path. Which bit maps to which path is simply its index in the lexicographically sorted set of paths. When an edge is taken, the a mask is applied to the bitset so the paths not taken are masked out: accu[block] = accu[pred] & paths[block]. Finally, paths that start in the block are masked in accu[block] = accu[block] | starts[block]. Because the bitset may be larger than a single host native integer, larger bitsets are broken down into “buckets”, and every block is instrumented with code that is roughly equivalent to this:

for (b : blocks):
  for (x : buckets):
	maybe-flush (accu[b][x])
	accu[b][x] = (accu[b][x] & accu[pred][x]) | starts[b][x]

Doing less work

The first revision of this patch was terribly slow and created massive binaries. Some of this is inevitable - as the number of paths grow large so does the number of instructions we need just to keep track of the paths we are currently on. I focused on reducing unnecessary work which had a big impact The previous version built tree.c in just under 10 minutes and created a 50M tree.o – the current version builds the same file in approx. 2m30s and results in a 32M tree.o. This improvement came from applying these techniques:

Subset aware instructions. Most basic blocks are in just a (much smaller) subset of the paths, which means we don’t need to apply updates to unaffected buckets. This means generating fewer pointless assignments and phis.

Removing ineffective masks. If all paths in the bucket goes through a basic block there is no need to emit AND instructions as it would not mask out any bits. x & ~0 == x. An interesting extension of this is not emitting individual ANDs for the phi when multiple predecessors all apply the same mask. In this case no mask needs to be applied at all as it would not unset or undo any path. There must be an effective discriminating AND later and the masking can be put off until then.

Straight line optimizations for the subpaths without any branching. In this case we only need to apply the mask once for the full chain, as repeated applications are ineffective. In code this would be (x & mask & mask) == (x & mask)

Replacing bitwise operations with constants. This is typically the case when paths start in a basic block, but no other paths in the same bucket go through that basic block. In this case we can replace 0 | mask with just mask.

Experimenting with the optimizer

Running this version through the optimizer with gcc -Os adds about 10 minutes to the build time. Interestingly, the effect on build time is less for the previous version - it only adds about 5 minutes. For both version the final object size is reduced objects to roughly the same size (approx 16M, down from 32M and 50M respectively). This means the current version is at about 2x optimal (in space) which leaves room for improvement but is probably still ok.

Progress report - prime path coverage in gcc

Sat, 08 Jun 2024 00:00:00 +0000

After the success of getting MC/DC support into gcc, NASA decided to keep funding the project and focus on (prime) path coverage. I have been working on this problem for a while now and it is about time to write a post on the progress so far.

Prime paths

The number of paths in a program grows very fast. Somewhat simplified, a path is a sequence of blocks evaluated for a run. Any control flow structure – if, for, while, switch, etc. – will multiply the number of paths up until that point. Additionally, loops are problematic because taking the loop zero times, one time, two times etc. are all different paths.

Let’s take a step back and look at a real program. This program has no loops, and the paths from entry to exit are not too surprising. The entry and exit vertices are omitted.

int fn (int x, int y) {
    int acc = 0;
    if (x)
        acc += x;
    else
        acc = 10;

    if (y)
        acc *= y + x;
    else
        acc -= (y - x);
    return acc;
}

To address loops we can limit the paths we consider to simple paths. A simple path is a path with no repeated vertices unless the last vertex is also the first vertex, that is, we close a loop. Considering only simple paths eliminates the problem of sequences like [2 4 5 6] [2 4 5 6 4 5 6] [2 4 5 6 2 4 5 6 2 4 5 6] ... which is nice, but they are still problematic. Going back to the fn which has 4 paths between entry and exit – simple paths can start at any vertex, and not just the entry. For the sequence 2 3 5 6 8 we have these simple paths:

Note that all of these paths would be covered by a single call where both x and y are non-zero. A prime path is the longest simple path which is not a subpath of any other simple path, and provides a good balance for the number of paths and the quality of coverage. The original 4 paths were all prime paths. That being said, the number of prime paths still grows very fast – this program has 14 prime paths.

while (a) {
    if (b)
        return;
    while (c--)
        a = fn(c);
}

Finding the paths

To find the prime paths of a function we need a good algorithm. A straight forward naïve implementation simply doesn’t finish for even reasonably sized problems. For examplple, I have experimented with tree. The function unix_getfulltree in tree.c has just short of a million paths, and I killed the process after 24 hours.

I found the paper Time and Space-Efficient Compositional Method for Prime and Test Paths Generation by Fazli & Afsharchi which describe a neat algorithm for finding prime paths, and I implemented it in gcc. That algorithm, combined with a suffix tree, brings the runtime down to something managable. I find the approx. two million paths in tree.c in less than four seconds. It is important that finding paths is fast, because the paths are too large to store explicitly, and gcov will have to recompute them.

I am certain my program can still be improved, speed-wise. About half the time is spent adding entries into the suffix tree (which is good), and I think a better algorithm would significantly cut down the search time. At its core there is also a simple recursive function for finding simple paths, which I think could benefit from a more clever algorithm. That being said, you probably won’t bother to try to cover millions of paths, and if path coverage is interesting at all you probably want to refactor anyway. Measuring (and trying to achieve) path coverage effectively forces you into to write programs in a particular style, but this is probably the goal.

Measuring coverage

To measure I applied a technique very similar to the one I used for MC/DC. First, I create a huge bitset where the nth bit corresponds to the nth prime path in the (lexicographically) sorted set of paths. When entering a vertex, a few bitwise operations mask out the bits of paths this vertex is not a part of. Then the paths starting in this vertex are masked in, before finally the paths which end in this vertex are written to the global covered sets. The instrumentation overhead is quite small, but creating so many new nodes in this stage is very punishing for the compiler. For small functions this seems fine, but tree.c takes up to ten minutes or so to build.

The bitset scales with the number of paths, which also means object sizes grow very fast. tree.o ends up being 50M which is just impressive balooning. We might need to come up with something clever to reduce this, but again, you probably want to refactor these types of functions which makes most of the problem go away.

Reporting coverage

So far, I have only sketched out a few alternatives for reporting. The key difference between path coverage and the other coverage reports in gcov is that a path is function wide, and does not have a natural anchor to a single line like statement, branch, and even condition coverage. This also makes line oriented tools like grep less useful.

The solution is probably to report right below the function summaries. I have a few sketches on this, and a few supporting features I will present in a later post. Still, my prototype does simple reporting and I have already found it quite useful to understand what I am measuring.

Going forward

I have posted a message on gcc-patches asking for feedback. There is still a lot of work left to do, but I believe all the key pieces are there. There are a few open questions and a lot of polishing I would like to do over the summer, but I am very hopeful for this feature and I think it will end up being quite useful.

MC/DC merged into gcc

Thu, 18 Apr 2024 00:00:00 +0000

I have to admit, it feels quite nice to finally write this post. My support for MC/DC in gcc was merged April 4th, followed by a couple of bug fixes. This post is just an announcement - I plan to some time later write a more detailed post on how it works and how to use it, and maybe one on MC/DC itself.

It has been a long project. I started working on it around the start of 2022, so it took a bit more than two calendar years from starting the work to the feature being merged, with mixed activity. It is at the time of writing (2024-04-18) not part of a release, so to use it you must build gcc from source.

I hope that gcc+gcov will prove a powerful tool to measure MC/DC and in particular contribute to making safety critical systems even safer.

PS: If using gcc from trunk does not work, I can support you in backporting it to older releases.

Rebuilding the world (with nix)

Sun, 03 Mar 2024 00:00:00 +0000

I ran into an interesting problem when adding support for Modified Condition/Decision Coverage (MC/DC) in gcc (a topic for a later post): how do you large scale test a new compiler feature? In this case, I wanted to figure out what kind of control flow graphs gcc could create, how particular graph shapes would interact with my changes, and if my assumptions actually held. Hand rolled examples took me quite far, but not to the point I was confident I didn’t miss anything.

Trying to answer some of these questions I started pulling random new and old free software packages and building them with my snapshot. It paid off right away - already in the first few projects I came over a few graph shapes that were surprising and found defects. All good, except now I had to rebuild the same projects with the new snapshot. Every new project meant also figuring out exactly how to ./configure && make, which tends to be similar, but not with enough exceptions for it to be troublesome.

I got to about five projects before I gave up - there must be a better way. This is where the big software distributions come in. I would imagine you can achieve something similar with Debian, portage, ports, but would probably involve setting up some infrastructure so that packages pull my snapshot compiler. The simple apt source <pkg> && dpkg-buildpackage is not really sufficient.

So I gave it a go in nix, and ended up with this default.nix:

{ pkgs ? import <nixpkgs> {
    overlays = [
        (self: super:
            let
                # Use a different source tarball for gcc13
                gcc13cc = super.gcc13.cc.overrideAttrs (old: rec {
                    version = "13.1.0-mcdc";
                    src = super.fetchurl {
                        url = "file:///home/src/gcc/gcc-snapshot.tar.gz";
                        sha256 = "sha256:0000000000000000000000000000000000000000000000000000";
                    };
                    nativeBuildInputs = old.nativeBuildInputs ++ [super.flex];
                    patches = builtins.filter (p: builtins.baseNameOf p != "no-sys-dirs-riscv.patch") old.patches;
                });

                # haskell's compose: [f g h] env = h(g(f(env)))
                compose = fns: stdenv: builtins.foldl' (e: fn: fn e) stdenv fns;

                cflags  = super.withCFlags ["--coverage" "-fcondition-coverage"];
                ldflags = super.addAttrsToDerivation { NIX_CFLAGS_LINK = "--coverage"; }; # TODO: -lgcov
                # link perl modules with gcc, which properly links lcov
                plflags = super.addAttrsToDerivation { perlPreHook = "export LD=$CC"; };
                withcc  = cc: stdenv: super.overrideCC stdenv cc;
                gcc13   = super.gcc13.override { cc = gcc13cc; };
            in {
                gcc13Stdenv = compose [(withcc gcc13) cflags plflags] super.gcc13Stdenv;
            }
        )

        (self: super: {
            # skip check phase on libxcrypt as it uses --wrap, which seems to not
            # resolve -lgcov, probably linking order
            libxcrypt = super.libxcrypt.overrideAttrs (_: { doCheck = false; });

            # these packages are necessary to run the tests, so skipping the
            # tests entirely would be an alternative
            gnutls = super.gnutls.overrideAttrs (old: {
                buildInputs = old.buildInputs ++ [
                    super.openssl
                    super.datefudge
                    super.socat
                ];
            });

            libdaemon = super.libdaemon.overrideAttrs (old: {
                patches = old.patches ++ [./libdaemon-typedef-right-order.patch];
            });

            # libhwy's logical_tests never finished linking (?? no idea what this is)
            libhwy = super.libhwy.overrideAttrs (old: {
                doCheck = false;
            });

            zeromq = super.zeromq.overrideAttrs (old: {
                # this is applied in upstream nixpkgs, so for future proofing
                # this must be checked to not be applied twice.
                patches = (old.patches or []) ++ [
                    # Backport gcc-13 fix:
                    #   https://github.com/zeromq/libzmq/pull/4480
                    (self.fetchpatch {
                        name = "gcc-13.patch";
                        url  = "https://github.com/zeromq/libzmq/commit/438d5d88392baffa6c2c5e0737d9de19d6686f0d.patch";
                        hash = "sha256-tSTYSrQzgnfbY/70QhPdOnpEXX05VAYwVYuW8P1LWf0=";
                    })
                ];
            });

            openexr = super.openexr.overrideAttrs (old: {
                patches = old.patches ++ [./0001-openexr-include-cstdint.patch];
            });

            # the elfutils tests cannot find rpm2cpio. rpm depends on rpm2cpio
            # so either the rpm source must be impure, or we must skip the
            # test. The latter is easier for this purpose.
            elfutils = super.elfutils.overrideAttrs (_: {
                doCheck = false;
                doInstallCheck = false;
            });
        })

        # on gcc13 the llvm tree does not build, which is a known problem and fixed with a small patch
        # https://github.com/JuliaLang/llvm-project/commit/ff1681ddb303223973653f7f5f3f3435b48a1983
        #
        # compiler-rt is problematic because -ffreestanding removes
        # declarations like free, malloc, strtoull from stdlib.h. The flag is
        # used to disable a particular optimization in clang but only gcc is
        # used, and the sanitizers (where this happen) are not interesting.
        # 
        # running the tests takes for ever and is pretty uninteresting for this
        # experiment so they are disabled too.
        (self: super:
            let
                patch = pkg: patches: pkg.overrideAttrs (old: {
                    patches = (old.patches or []) ++ patches;
                    doCheck = false;
                });
                llvm-patches        = [./0001-include-cstdint-llvm11.patch];
                compiler-rt-patches = [./0001-compiler-rt-no-ffreestanding.patch];
            in {
                llvmPackages_11 = super.llvmPackages_11 // {
                    libraries = super.llvmPackages_11.libraries.extend(lfinal: lprev: {
                        libcxx      = patch lprev.libcxx    llvm-patches;
                        libcxxabi   = patch lprev.libcxxabi llvm-patches;
                        compiler-rt-libc    = patch lprev.compiler-rt-libc    compiler-rt-patches;
                        compiler-rt-no-libc = patch lprev.compiler-rt-no-libc compiler-rt-patches;
                    });
                    tools = super.llvmPackages_11.tools.extend(lfinal: lprev: {
                        libllvm  = patch lprev.libllvm llvm-patches;
                        libcxxClang = lprev.libcxxClang.overrideAttrs (old: {
                            setupHooks = old.setupHooks ++ [./no-fprofile.sh];
                        });
                        libstdcxxClang = lprev.libstdcxxClang.overrideAttrs (old: {
                            setupHooks = old.setupHooks ++ [./no-fprofile.sh];
                        });
                    });

                    # we also bring in the other attrs because they still
                    # capture the old package. Doing this manually is not great
                    # and there should be a programmatic solution
                    inherit (self.llvmPackages_11.tools)     libllvm clang;
                    inherit (self.llvmPackages_11.libraries) compiler-rt;
                };
            }
        )

        # python's cffi tests building modules and check that only the expected
        # artifacts are produced, which breaks with gcov because the .gcno
        # files are generated too. It also adds extra symbols cffi does not
        # expect, so the no_unknown_exported_symbols test fails.
        (self: super: {
            python310 = super.python310.override {
                packageOverrides = pyself: pysuper: {
                    cffi = pysuper.cffi.overrideAttrs(old: {
                        disabledTests = old.disabledTests ++ [
                            "test_no_unknown_exported_symbols"
                            "test_api_compile_1"
                            "test_api_compile_2"
                            "test_api_compile_3"
                            "test_api_compile_explicit_target_1"
                            "test_api_compile_explicit_target_2"
                            "test_api_compile_explicit_target_3"
                        ];
                    });
                };
            };
        })

        # valgrind, systemd builds with -fprofile-coverage which is all good,
        # but the link fails because they do not link to libc (-nostdlib) which
        # means the fwrite etc. calls in libgcov do not resolve. Since we have
        # the information we need at this point we can just build them without
        # mcdc.
        (self: super: 
            let no-fcondition-coverage = pkg: pkg.overrideDerivation (old: {
                    NIX_CFLAGS_COMPILE = builtins.replaceStrings
                        ["--coverage" "-fcondition-coverage"] ["" ""]
                        old.NIX_CFLAGS_COMPILE;
                });
            in {
                valgrind = no-fcondition-coverage super.valgrind;
                systemd  = no-fcondition-coverage super.systemd;
                gnu-efi  = no-fcondition-coverage super.gnu-efi;
        })
    ];
    config.replaceStdenv = { pkgs, ... }: pkgs.gcc13Stdenv;
}}:

pkgs.mkShell { 
    inputs = with pkgs; [
        tree
        cpio
        git
        icu
        openjdk
    ];
}

This uses mkShell, but it is quite easy to transform it into mkDerivation. mkShell suited my workflow at the time, so I kept it around. Running nix-shell default.nix would build (re)build a gcc snapshot, and build tree, cpio, git, icu, and openjdk and every build- and runtime dependency of these targets with the -fcondition-coverage flag. That is a huge list of software, which includes perl and llvm. In order to test my latest snapshot, all I had to do was replace the hash (gcc13cc.src.sha256). I found a lot of defects that way (openssl was particularly good at creating surprising control flow graphs), and by the time openjdk built I was quite confident that at least the cfg analysis phase of my program worked well. Every defect I found made its way into the test suite. More targets could be added to

Some packages needed a few tweaks in order to build. Nothing too bad, and occasionally lazily fixed with patches. Those problems were just worked out one-by-one as they stopped the build.

Patches and script: