Dav Glass and I visited the Yammer office in San Francisco this week to discuss build & test tools we use at YUI.
We showed off Shifter for building YUI, Grover for testing headlessly, Yeti for testing in browsers, and Istanbul for statement-level code coverage. We use Travis for running most of this stuff in public CI. We now require 70% statement coverage before a new YUI module is allowed to land in the core and nobody can commit when the Travis or internal CI build is broken, unless the commit fixes the build.
This is all very impressive. But @mde was quick to notice that we didn’t drop everything to get to this point—before diving in, you first need to prioritize what you work on. I couldn’t agree more.
When you’re starting from scratch, you start to love the metrics. Green dots for passing builds. Green coverage reports when you hit 80% of your statements. The increasing number of passing tests. I’m all for having good code coverage, but before you go crazy, you should be careful that you don’t start writing tests for the wrong part of your code.
Your code is not uniform
Your code has various levels of quality starting at the first commit you make. You will write some code that’ll last for weeks or months, and some code that’ll need a rewrite next week. You need to embrace this kind of change and understand where it happens in your project.
Node.js solves this problem quite well with the notion of a Stability Index.
Throughout the documentation, you will see indications of a section’s stability. The Node.js API is still somewhat changing, and as it matures, certain parts are more reliable than others. Some are so proven, and so relied upon, that they are unlikely to ever change at all. Others are brand new and experimental, or known to be hazardous and in the process of being redesigned. The Stability Index ranges from Deprecated (0) and Experimental (1) for unstable code to Locked (5) for code that only changes for the most serious bugs.
It’s a good idea for any post-1.0 project to assign a Stability Index to the APIs in your own code. Not only is it a clear message to those using your APIs, but it’s also a clear message for your team. It tells you where you should—and shouldn’t—write tests.
More stable, more tests
If you write tests like they cost nothing, you’re going to find yourself writing tests instead of writing code that works.
Kent Beck’s wisdom says it best:
I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris). If I don’t typically make a kind of mistake (like setting the wrong variables in a constructor), I don’t test for it. This answer is good and I’ll take it a step further: You should prioritize writing tests for parts of your code with a higher Stability Index. Especially if you’re just starting on a new project.
If you’re writing tests for code that’s rapidly changing, you’re going to spend more of your time writing tests instead of shipping features. For code that’s brand new, I typically only test-first a small amount of code and wait a while before hitting that green bar on a code coverage report.
Don’t get sucked into the allure of metrics too early. Remember what your job is: writing code that works. Code coverage and good testing tools are very important, but not if they get in the way of building what you’re supposed to build.