Programming for coverage part 2

In this post we will use path coverage as a guide for testing a C function and explore how a light rewrite improves our ability to test it. Most testing is really about observing how the program responds to data, and coverage is mapping that response to the program structure. By changing the program we change the structure the data operates on, even when two programs are functionally equivalent.

We are writing a C program and need to know if today is the leap day, 29th February. This is the first draft and test:

bool leap_day() {
    const time_t now = time(NULL);
    const struct tm *tm = localtime(&now);
    return tm && tm->tm_mon == 1 && tm->tm_mday == 29;
}

void check() {
    assert(leap_day() == false);
}

The test is broken since it will fail on the actual leap day, when the function returns true. Time is an implied input to this function, but we cannot control it other than by doing things like changing the system clock. Bluntly, it has an interface that does not really support testing. Before addressing that, let’s look at the CFG with zcov:

As expected, it has straight line flow until the conditional expression in the return statement. One neat thing about the CFG visualization is that phases of the function become so visible. In this case, the graph widens at block 4, before re-joining at 9. The 3-4 edge is a bridge, an edge that all paths of the (sub)graph must go through. Bridges are useful markers for natural places to split functions, and where to make strong assertions on state and the pre- and post conditions (so far). The conditional expression has four paths, [1 1 1] [1 1 0] [1 0 *] [0 * *], here shown in CFG form:

To achieve path coverage we need to control the value returned by localtime(), which is derived from time_t now. If we lift the responsibility of computing now to the caller, we can control it from our tests:

bool leap_day(const time_t *now) {
    const struct tm *tm = localtime(now);
    return tm && tm->tm_mon == 1 && tm->tm_mday == 29;
}

void check() {
    time_t now;
    struct tm tm = {};
    // 2024 is a leap year
    tm.tm_year = 2024;

    // February, not 29th
    tm.tm_mon = 1;
    tm.tm_mday = 1;
    now = mktime(&tm);
    assert(leap_day(&now) == false);

    // 29th, not February
    tm.tm_mon = 3;
    tm.tm_mday = 29;
    now = mktime(&tm);
    assert(leap_day(&now) == false);

    // February 29th
    tm.tm_mon = 1;
    tm.tm_mday = 29;
    now = mktime(&tm);
    assert(leap_day(&now) == true);
}

These tests cover 3/4 paths as seen in the screenshot. We can add a fourth test for the case where the month is not February and the day not the 29th, but the coverage shows us that the test would not be all that meaningful. Notice in the rightmost pane that we have 100% block coverage and, because block coverage subsumes it, we have 100% line coverage. The path coverage shows us that we are not exercising the case of tm == NULL; 3-7 is the edge from the first check to the outcome.

tm == NULL when localtime() fails. Here is the relevant section from the manual:

Upon successful completion, the localtime() function shall return a pointer to the broken-down time structure. If an error is detected, localtime() shall return a null pointer and set errno to indicate the error.

Ok, so let’s feed localtime() a bad time_t to cover this path. Testing for localtime() failure is important because we no longer control the call to time(). In fact, for testing we don’t even need to call time() at all, which makes the tests independent of the system clock, a nice property as it means they won’t suddenly fail on a different computer. As a fun side note, it was surprisingly hard to make localtime() fail. I tried populating now with bad data; negative day, month way out of range, even just setting it to (time_t)-1, but the call kept succeeding. I peeked at the values used in the overflow check in musl and added a test.

// Garbage time
now = (time_t)((INT_MAX * 31622400LL) + 1);
assert(leap_day(&now) == false);

Great, 4/4 paths covered, but something still feels off. We would be done if the goal is code coverage, but our goal is good software. This function still has a problem that the quest for coverage made us sensitive to; we have covered the path where localtime() fails, but the even when it did, leap_day() succeeded. This would be a hard bug to detect as the problem would only manifest on the leap day, roughly every four years. There is not enough room in a bool to signal the failure, which gives us a few options: either widen the return type (e.g. to an enum) and reserve a value for failure, add an out-parameter for the success/failure, use a global like errno, or apply the same trick again and have the caller call localtime(). I think the neatest solution is to lift out the localtime() call.

bool leap_day(const struct tm *tm) {
    return tm->tm_mon == 1 && tm->tm_mday == 29;
}

Excellent. What these changes boil down to is making all the input data explicit. The function is reduced to its essence, and coverage becomes trivial. The NULL check is gone as it common, and reasonable, in C to require the callers to only pass pointers to valid objects. While we shift the burden to call time() and localtime() onto the caller, errors can be handled where there is context to do so, and the caller can choose to use other functions like localtime_r() or gmtime() when it makes sense. This applies to our testsuite too, as we don’t even need call time() or localtime() at all to test leap_day():

void check() {
    struct tm tm = {};

    // February, not 29th
    tm.tm_mon = 1;
    tm.tm_mday = 1;
    assert(leap_day(&tm) == false);

    // 29th, not February
    tm.tm_mon = 3;
    tm.tm_mday = 29;
    assert(leap_day(&tm) == false);

    // February 29th
    tm.tm_mon = 1;
    tm.tm_mday = 29;
    assert(leap_day(&tm) == true);
}

There you have it. The new leap_day() is obviously simpler to the point of being obsolete, but this is unfortunately the curse of examples. Designing interfaces that support testing is common wisdom, and this coverage guided testing effort shows both why it matters and how it improves programs. If we had settled for 100% line coverage and 3/4 paths we might not have detected the localtime() bug. By pushing possible failures up we can make functions pure and predictable, and handle (or ignore!) errors where it makes sense in the application.

A few remarks to close out this post, because in fairness the final leap_day() function does not do exactly what the original did. Let’s add a new function to restore the functionality:

/* -1 if error, 0 if not leap day, 1 if leap day.  If -1, errno describes the
   error.  */
int leap_day_today() {
    const time_t now = time(NULL);
    if (now == (time_t)-1)
        return -1;
    const struct tm *tm = localtime(&now);
    if (!tm)
        return -1;
    return leap_day(tm) ? 1 : 0;
}

void example() {
    const int leap = leap_day_today();
    if (leap < 0)
        perror("failed getting localtime");
    else if (leap > 0)
        puts("Hello, leap day!");
    else
        puts("Hello, not leap day!");
}

Unlike the first leap_day(), this function checks if the time() and localtime() calls succeeded. It widens the interface to return a raw int, but this could just as well be an enum. If knowing which supporting function failed is important for the application, it is probably better time() and localtime() explicitly and pass the result to leap_day(), and now we have that opportunity.

If we want to cover all the paths of leap_day_today() we’re back at square one, because time() and localtime() are not easily controllable as a caller. In that case we need to manipulate the global environment, for example by setting the system clock before running the test. This is still an improvement over the original revision because we don’t have to combine those tests with the testing of the leap_day() logic, which is firewalled off through by the function call.