How do you run your Clojure unit tests? Does it make you happy?
When working on new code, I rely on CIDER’s clojure.test support. It allows you to run all tests in a namespace and then re-run the failed ones after you’ve made changes. This is good, because it makes the feedback loop tight: write code, send it to the REPL, run the tests, repeat. Cursive supports a similar workflow.
When I want to run the full test suite from the command-line, for example in a CI job, none of the available test runners makes me fully happy. For large test suites - especially the ones with integration tests - I’d like to have the following:
- Output catching. If a test prints something to stdout, it should be shown only if the test fails. There’s no point in wading through thousands of lines of logs generated by succesful tests to find that one actual exception.
- Test tagging. Many test runners allow you to filter tests by name. I’d
like to filter them by custom tags. For example, I’d like to say “run
all the tests except those tagged with
:large”.
Here are some less essential but still nice to have features:
- JUnit output in addition to the normal output. CI tools like Jenkins or Circle know how to create nice reports from JUnit output. The reports are helpful for making sense of large test suites. clojure.test knows how to generate JUnit, but it can only do it instead of the normal output. When debugging test failures, I prefer the usual logs to the nice reports.
- Reporting the slowest tests. This is not something I care about all the time, but it’s handy when you wonder why the full test suite takes 45 minutes to run.
I’ve seen or implemented all of the above in custom test runners, but none of
the open-source runners – lein test, boot-test,
eftest, boot-alt-test, … – offers everything in a
coherent package. Basically what I want is pytest for Clojure.
Supporting ClojureScript would be great, too, although you can already use
Karma with karma-cljs-test.
Since I know what I want, I should just go ahead and implement this, right? Right. I wanted to try something new and first write about this to see if anyone else cares. Does anyone else miss these features?
Update 2017-04-27: Check out the update on progress!