Search | Blog | Contact

Continuous Testing Explained 29

Posted by Ben Rady Thu, 20 Sep 2007 21:11:00 GMT

What is continuous testing? It’s turning the knob on Test Driven Development up to 11, by automatically running the tests on every save. This has profound effects on the way that TDD is applied, and is likely to make you a more efficient and productive programmer.

First, a little history…

The term “Continuous Testing” was originally coined by the Program Analysis Group at MIT. Their research found, among other things, that “Developers using continuous testing were three times more likely to complete the [assigned] task before the deadline than those without” and that “continuous testing reduced wasted time [related to testing] by 92–98%.” These benefits come from a number of interesting effects:

Less Overhead
The first and most obvious effect of continuous testing is that you don’t have to manually run the tests. One less button to push…that’s always a good thing.

Defect Lag Reduced
Just as modern Java IDEs like IntelliJ and Eclipse increase feedback and productivity though continuous compilation, continuous testing reduces the time between the introduction of an error and it’s detection. This leads to more effective regression testing, because it’s not the expected failures that cost you time, it’s the unexpected ones. Continuous testing finds unexpected failures right away.

Promotes Good TDD Habits
If you’re running the tests on every save, it makes it more difficult to wander off the TDD path. Even though many of the participants in the MIT study were not versed in TDD, several of them reported that they “got a small part of [their] code working before moving on to the next section, rather than trying to debug everything at the end.” Another remarked, “It was easier to see my errors when they were only with one method at a time.” Making incremental changes is a core Agile principle, and directly reflected in the practice of TDD. Continuous testing makes the benefits of incremental development immediately apparent.

Tests Stay Fast
Using continuous testing helps keep your tests fast. This is something that was not mentioned in the MIT study, but I can attest to it in my own work. If you write a slow test, you’ll know right away, and you’ll probably have a good idea exactly what is making the test slow (because you just added it). I think the reason that they didn’t mention it in the MIT study is that they weren’t working with legacy code, or external systems, which is where a lot of test slowness comes from.

Today, there are a number of tools that support continuous testing:

For full disclosure, I should mention that I’m the admin of the Infinitest project. Our goal is to create a continuous test runner for every development platform. We’ve already released a Java test runner, and we’re nearing completion of a Python runner.

Continuous testing is a practice that I have followed in my own work for most of 2007. There is little doubt in my mind that it is the future of TDD. Looking forward to the days of multicore systems, continuous testing may be an effective way to realize productivity improvements through better hardware. So long as they are independent (as they should be), test runs can easily be distributed across multiple cores, and perhaps even, multiple machines.

Comments

Leave a response

  1. Avatar
    Samuel A. Falvo II about 5 hours later:

    I’ve been doing this for years, but not because I’m smarter than anyone else. It was borne out of pure laziness. Having to edit a file, then exit, then “make check” is pretty quick, but considering how fast development runs can be, it still becomes a burden. So, I use plain shell scripts to automate the task for me.

    If you prefer to keep your editor running as you hack on code, I’m sure a vim- or elisp-script can be written to implement the desired behavior (assuming you’re using either vim or emacs). You do not need to use a special purpose IDE.

  2. Avatar
    Erik about 6 hours later:

    <fun about typo>
    Did you really mean “continuous complication”? I’d vote for “continuous compilation”.

    Probably a Freudian slip – if so, you should talk to your analyst about your relationship with your IDE ;-) Or was it just some brainless editor trying to correct a typo?
    </fun about typo>
    Besides that, I totally agree with you on the topic.

  3. Avatar
    Ben about 17 hours later:

    Erik – Wow, yeah, that takes on a whole new meaning…

    Ha!

    Let me fix that….

  4. Avatar
    Peter Lawrey 1 day later:

    We use continous integration and release so that successful releases are immediately available as snapshots to other projects.

    We also use conditional checkin. What this does is perform a number of builds using the build server with the proposed changes and gives you the option of checking it in iff the builds all pass. This means you can do checkins and go to lunch or go home if it breaks a build, the checkin fails.

  5. Avatar
    Daniele 10 months later:

    ct-eclipse seems to be dead and the only installation instructions page is broken… http://ct-eclipse.tigris.org/install.html

  6. Avatar
    beliebte Kasinos about 1 year later:

    I admit, I have not been on this webpage in a long time… however it was another joy to see It is such an important topic and ignored by so many professionals. I thank you to help making people more aware of possible issues. Great stuff as usual….

  7. Avatar
    top auto insurance companies over 2 years later:

    A great blog post makes you think and you’ve certainly given me a few tid bits to consider.

  8. Avatar
    psn card over 2 years later:

    Hah,that was really funny. I even ordered an essay paper on it.These online essay writers made an excellent essay for me.

  9. Avatar
    face blog over 3 years later:

    This is the best short and sweet explanation of What is continuous testing I found ever. Great post, brilliant.

  10. Avatar
    ShAaNiG over 3 years later:

    Erik – Wow, yeah, that takes on a whole new meaning…

    Prudential West

  11. Avatar
    ShAaNiG over 3 years later:

    We also use conditional checking. What this does is perform a number of builds using the build server with the proposed changes and gives you the option of checking it in if the builds all pass. This means you can do checking and go to lunch or go home if it breaks a build, the checking fails.

    Prudential West

  12. Avatar
    ShAaNiG over 3 years later:

    Erik – Wow, yeah, that takes on a whole new meaning.

    Wholesale Brand Name Clothing

  13. Avatar
    ShAaNiG over 3 years later:

    We also use conditional checking. What this does is perform a number of builds using the build server with the proposed changes and gives you the option of checking it in iff the builds all pass. This means you can do checking and go to lunch or go home if it breaks a build, the checking fails.

    Wholesalers

  14. Avatar
    Frederick over 3 years later:

    Thanks for the information. I love tips on how to become a much-improved programmer. I have not gone through formal programming training so any info from the net is a great help.

    Free Bondage Videos

  15. Avatar
    daniel over 3 years later:

    I agree that one less button to push is a good thing. I would have thought of a different meaning for continuous meaning if you have not explained it clearly. I would have thought it something negative and tedious.

    Heartbreak

  16. Avatar
    Kent over 3 years later:

    Are expert programmers not expert spellers? Does this point matter? I just noticed that you misspelled panicky. Anyway, that is moot in your world but I figure you also have spelling skills in terms of programming code words. I am not picking a fight, just very curious.

    statement writing

  17. Avatar
    sohbet over 3 years later:

    The web client and the window and the java script causes problem for some. Anyway as everything has some sort of disadvantages this software may also have them but the point we have to note here is that how many people

  18. Avatar
    ask ve sevgi over 3 years later:

    Hah,that was really funny. I even ordered an essay paper on it.These online essay writers made an excellent essay for me.

  19. Avatar
    http://www.ostube.net/ over 3 years later:

    The web client and the window and the java script causes problem for some. Anyway as everything has some sort of disadvantages this software may also have them but the point we have to note here is that how many people

  20. Avatar
    galbur over 3 years later:

    Anyway, that is moot in yours world but I figure you also have spelling skills in terms of programming code word. I am not picking a fight, just very curious.

  21. Avatar
    ilan over 3 years later:

    Anyway, that is moot in yours world but I figure you also have spelling skills in terms of programming code word. I am not picking a fight, just very curious.

  22. Avatar
    dvd to ipad mac over 3 years later:

    this ah ha

  23. Avatar
    Youtube converter over 3 years later:

    Thank you for posting this.

  24. Avatar
    memorias usb over 3 years later:

    Anyway, that is moot in yours world but I figure you also have spelling skills in terms of programming code word. I am not picking a fight, just very curious.

  25. Avatar
    cheap vps over 3 years later:

    or full disclosure, I should mention that I’m the admin of the Infinitest project. Our goal is to create a continuous test runner for every development platform. We’ve already released a Java test runner, and we’re nearing completion of a Python runner.

    Continuous testing is a practice that I have followed in my own work for most of 2007. There is little doubt in my mind that it is the future of TDD. Looking forward to the days of multicore systems, continuous testing may be an effective way to realize productivity improvements through better hardware. So long as they are independent (as they should be), test runs can easily be distributed across multiple cores, and perhaps even, multiple machines.cheap VPS

  26. Avatar
    discount tiffany earrings over 3 years later:

    I’ve been doing this for years, but not because I’m smarter than anyone else. It was borne out of pure laziness. Having to edit a file, then exit, then “make check” is pretty quick, but considering how fast development runs can be, it still becomes a burden. So, I use plain shell scripts to automate the task for me.

  27. Avatar
    Listings over 3 years later:

    This is great blog.That is moot in yours world but I figure you also have spelling skills in terms of programming code word.I would have thought of a different meaning for continuous meaning if you have not explained it clearly Listings

  28. Avatar
    craigslist reviews over 3 years later:

    Great article .I totally agree with you we use continous integration and release so that successful releases are immediately available as snapshots to other projects.thanks for making discussion on this topic..keeps going on!

  29. Avatar
    Ray Cruz over 3 years later:

    At start, I’d prefer to give thanks to you for this enlightening article. 2nd, I’d prefer to doubt wheresoever I can find lot more information regarding your article. I just came here via Yahoo and i can not determine any other linked up web sites on this subject. How do I subscribe for your web blog? I had prefer to adhere to your updates as they arrive along! I had a query to interrogate but I forgot what it absolutely was… anyway, thank you very much. Author of how to cook beef tenderloin

    Best wishes, Ray Cruz
Comments