Unit Tests Coverage: Less Is More 2
In TDD a unit test has to be very small to isolate failures. This does funny things to code coverage as a metric. Each test should have a very small area of effect, and so each unit test should have a negligible effect on the overall code coverage statistic. Bear with me here, see where I’m missing out.
Say you have an existing (legacy) system with no coverage at all. Zero percent. If you start doing TDD today, the overall coverage percentage should barely change at all. If only the new code is test-driven, then the old code is not gaining coverage except where tests are necessary to ensure that the new code is being called. The low coverage per test is a good thing because it shows that the unit tests have good isolation. In such a situation, code coverage is really telling you the ratio of new code to old code. Again, this is so obvious and logical to me that I must be missing some cool subtleties.
If all the code was test-driven from the beginning, you should have a very high coverage number, and writing a new unit test before you add code should not impact that number. You only write enough code to pass the test, so again you aren’t getting much in the way of uncovered code. A lowering of the ratio might indicate a problem with test-to-production-code ratio. This seems pretty simple and logical, so I’m sure I’m missing some interesting corner cases.
OTOH, system tests and integration tests paths through many components at once, and should have a more significant effect on coverage in a previously-untested system, though their job is to prove function points, not to raise the metrics. These non-unit tests are the thing that boost your coverage of old code. That is also a good effect, because it is the goal of the system test to ensure that the parts work together.
I’m not saying that code coverage should be low, only that as we move incrementally, the unit tests we write should individually have negligible effect on our overall code coverage numbers… a thought that intrigues me.
Of course in a Test Driven Context, the first stage in fixing a bug is writing a test to expose it, so you should end up with some coverage of old code.
Of course. But even then, your test shouldn’t change the total coverage percentage by much. It matters more than it shows, which is my problem with coverage in general.
You can show progress by bumping up the coverage number, but your tests don’t have to be any good. But if you write good tests where they’re needed (as given by Lindsay) then you have the real goal covered, but the prosthetic goal is largely untouched.
Coverage grows very slowly in legacy code.