Blog
Deprecating line coverage
Line coverage is typically the first structured metric looked at, and it provides a rough and imprecise measure of how thoroughly the test harness exercises the code. Briefly, line coverage is the answer to the question was code at this line ever run? The most important insight is finding the code that is not at all exercised in testing, but it can also be used as a coarse profiling tool by highlighting what lines are run more often and could warrants more optimization or other improvements. However, the unit of line is flawed in a couple of aspects, and it is time we stop using it and instead favour better metrics.
December 9, 2024
Early access: zcov
I am excited to announce my latest project, under the name zcov. It is a program for interpreting and appreciating coverage, with the goal of being the only tool you need for working with compiler instrumented coverage. It is still under development, but I have already been able to use it to understand and explain subtle program interactions, the path coverage view being particularly powerful.
October 21, 2024
Bug hunt: Independent functions crashing gcov
NASA reported an odd crash in gcov, and I found it interesting and deep enough to warrant describing the root cause and the path it took me on. The actual crash itself was quickly found with gdb; reading the block.locations.lines when printing the source lines of a path when the array was empty (with size()-1, even!). A fix would be to not do that e.g. by guarding the access, but the question is why the array is empty in the first place.
October 3, 2024
Trip report: NDC Techtown 2024
This year’s Techtown conference in Kongsberg is through, and I wanted to do a trip report of sorts. There were ~400 attendees, the highest attendance yet, and an all-around fantastic atmosphere. I learnt a lot and had the privilege of striking conversations with many brilliant people, on diverse topics such as language design, compiler development, compiler features, embedded constraints, programming techniques, and I am already excited for next year. The programme was superb and I cannot think of a single slot where which talk to attend was an obvious choice. These are the ones I went to and a few thoughts on them.
September 15, 2024
Progress report - prime path coverage in gcc 3
The prime path coverage support is shaping up nicely, so I wanted to write a bit about the last series of improvements to the upcoming prime path coverage feature in gcc. gcov reports Path coverage differs from statement, branch, and condition coverage in that coverage ties to the function and not individual lines. This is quite different from how gcov has reported up until now. Furthermore, it is not necessarily easy to look at some code and figure out even what the paths even are, in particular after compiler transformations.
August 8, 2024
Progress report - prime path coverage in gcc 2
It’s time for another progress report. The last post covered the phases at a high level – please read that first for an overview. Observing paths To observe paths we associate a bitset with each function, where each bit corresponds to a path. Which bit maps to which path is simply its index in the lexicographically sorted set of paths. When an edge is taken, the a mask is applied to the bitset so the paths not taken are masked out: accu[block] = accu[pred] & paths[block]. Finally, paths that start in the block are masked in accu[block] = accu[block] | starts[block]. Because the bitset may be larger than a single host native integer, larger bitsets are broken down into “buckets”, and every block is instrumented with code that is roughly equivalent to this:
June 25, 2024
Progress report - prime path coverage in gcc
After the success of getting MC/DC support into gcc, NASA decided to keep funding the project and focus on (prime) path coverage. I have been working on this problem for a while now and it is about time to write a post on the progress so far. Prime paths The number of paths in a program grows very fast. Somewhat simplified, a path is a sequence of blocks evaluated for a run. Any control flow structure – if, for, while, switch, etc. – will multiply the number of paths up until that point. Additionally, loops are problematic because taking the loop zero times, one time, two times etc. are all different paths.
June 8, 2024
MC/DC merged into gcc
I have to admit, it feels quite nice to finally write this post. My support for MC/DC in gcc was merged April 4th, followed by a couple of bug fixes. This post is just an announcement - I plan to some time later write a more detailed post on how it works and how to use it, and maybe one on MC/DC itself. It has been a long project. I started working on it around the start of 2022, so it took a bit more than two calendar years from starting the work to the feature being merged, with mixed activity. It is at the time of writing (2024-04-18) not part of a release, so to use it you must build gcc from source.
April 18, 2024
Rebuilding the world (with nix)
I ran into an interesting problem when adding support for Modified Condition/Decision Coverage (MC/DC) in gcc (a topic for a later post): how do you large scale test a new compiler feature? In this case, I wanted to figure out what kind of control flow graphs gcc could create, how particular graph shapes would interact with my changes, and if my assumptions actually held. Hand rolled examples took me quite far, but not to the point I was confident I didn’t miss anything.
March 3, 2024