First, a little history…

The term “Continuous Testing” was originally coined by the Program Analysis Group at MIT. Their research found, among other things, that “Developers using continuous testing were three times more likely to complete the [assigned] task before the deadline than those without” and that “continuous testing reduced wasted time [related to testing] by 92–98%.” These benefits come from a number of interesting effects:

Less Overhead

The first and most obvious effect of continuous testing is that you don’t have to manually run the tests. One less button to push…that’s always a good thing.

Defect Lag Reduced

Just as modern Java IDEs like IntelliJ and Eclipse increase feedback and productivity though continuous compilation, continuous testing reduces the time between the introduction of an error and it’s detection. This leads to more effective regression testing, because it’s not the expected failures that cost you time, it’s the unexpected ones. Continuous testing finds unexpected failures right away.

Promotes Good TDD Habits

If you’re running the tests on every save, it makes it more difficult to wander off the TDD path. Even though many of the participants in the MIT study were not versed in TDD, several of them reported that they “got a small part of [their] code working before moving on to the next section, rather than trying to debug everything at the end.” Another remarked, “It was easier to see my errors when they were only with one method at a time.” Making incremental changes is a core Agile principle, and directly reflected in the practice of TDD. Continuous testing makes the benefits of incremental development immediately apparent.

Tests Stay Fast

Using continuous testing helps keep your tests fast. This is something that was not mentioned in the MIT study, but I can attest to it in my own work. If you write a slow test, you’ll know right away, and you’ll probably have a good idea exactly what is making the test slow (because you just added it). I think the reason that they didn’t mention it in the MIT study is that they weren’t working with legacy code, or external systems, which is where a lot of test slowness comes from.

Today, there are a number of tools that support continuous testing:

• CT-Eclipse (was Continuous Testing Plugin for Eclipse) – Java/Eclipse
• ZenTest::Autotest – Ruby
• Fireworks – Java/IntelliJ
• Infinitest – Java

For full disclosure, I should mention that I’m the admin of the Infinitest project. Our goal is to create a continuous test runner for every development platform. We’ve already released a Java test runner, and we’re nearing completion of a Python runner.

Continuous testing is a practice that I have followed in my own work for most of 2007. There is little doubt in my mind that it is the future of TDD. Looking forward to the days of multicore systems, continuous testing may be an effective way to realize productivity improvements through better hardware. So long as they are independent (as they should be), test runs can easily be distributed across multiple cores, and perhaps even, multiple machines.