The joys of coverage

In the project I’m working on, we have been tracking the test coverage of a Clojure web backend with Cloverage and Codecov. We use Codecov’s ratcheting scheme where every pull request has to have as high form coverage as the master branch.

We’ve had this setup for eight months and we’ve gotten a number of benefits out of it:

The coverage report helps you to check if your tests work correctly. Many times I’ve thought that I’ve written a unit test to run all the code of a function, but the report has revealed that there’s a branch that has not been executed. Either the test is wrong, the code is wrong. In any case I am wrong.
The coverage report can help you to find dead code. If a private function is not covered, it’s not needed for anything.
Sometimes I think that “this code is so simple that it does not need a test”, but then Codecov complains and I write one anyway. These tests have found way more bugs than I’d like to admit.

Because Clojure is a dynamically typed language, even just trying to run your code without checking the results can find bugs.

The downsides

In practice, we sometimes have to merge PRs even though they do not have high enough coverage. Cloverage measures both the line coverage (which lines have code that was executed) and form coverage (which Clojure forms were executed). Getting high line coverage is straightforward, but getting full form coverage is tricky:

Sometimes getting full form coverage is impossible. Cloverage measures the form coverage on macroexpanded code. Many macros expand to code with unreachable branches. For example, consider an assertion:
```
(assert (pos? x))
```
It expands to the following code:

    (if (clojure.core/int? user/x)
     nil
     (do
      (throw
       (new
        java.lang.AssertionError
        (clojure.core/str
         "Assert failed: "
         (clojure.core/pr-str '(clojure.core/int? user/x)))))))

The only way to get full form coverage for this code is to execute the both branches of the if expression. That is, you need to get the assertion to both pass and fail! But since a failing assertion means that there’s a bug in the program, the failure branch should be impossible to reach.

Cloverage is implemented with side-effecting macros, but Clojure is implemented in such a way that a macro call may be evaluated more than once (see CLJ-1407). This causes bugs where e.g. fully covered loop and doseq forms are reported as partially covered.
We have a namespace which is too large to instrument with Cloverage. Since Cloverage works by adding annotations to the code before evaluating it, it can make the code size go over the “Method code too large” threshold.
There are some oddities and outright bugs in Codecov’s reporting, so you can’t blindly trust their reports.

In conclusion

When we started tracking the coverage in February, our coverage was around 50% and it has been floating around 80-85% for the last six months. For select parts we’ve made the effort to maintain it over 90%.

I know some experienced developers believe that 100% coverage is mandatory. I want to believe it but I just don’t know how it should be interpreted in the world of Clojure with all the problems! I’m also hesitant to adopt this practice for open-source projects.

That said, Cloverage and Codecov are straightforward to set up if you already are running your tests via a CI service. If you’re okay with dealing with imperfect coverage measurements, I recommend trying it out.