Object Mentor Blog: Continuous Testing Explained

Continuous Testing Explained

Ben Rady — Thu, 20 Sep 2007 16:11:00 -0500

What is continuous testing? It’s turning the knob on Test Driven Development up to 11, by automatically running the tests on every save. This has profound effects on the way that TDD is applied, and is likely to make you a more efficient and productive programmer.

First, a little history…

The term “Continuous Testing” was originally coined by the Program Analysis Group at MIT. Their research found, among other things, that “Developers using continuous testing were three times more likely to complete the [assigned] task before the deadline than those without” and that “continuous testing reduced wasted time [related to testing] by 92â€“98%.” These benefits come from a number of interesting effects:

Less Overhead

The first and most obvious effect of continuous testing is that you don’t have to manually run the tests. One less button to push…that’s always a good thing.

Defect Lag Reduced

Just as modern Java IDEs like IntelliJ and Eclipse increase feedback and productivity though continuous compilation, continuous testing reduces the time between the introduction of an error and it’s detection. This leads to more effective regression testing, because it’s not the expected failures that cost you time, it’s the unexpected ones. Continuous testing finds unexpected failures right away.

Promotes Good TDD Habits

If you’re running the tests on every save, it makes it more difficult to wander off the TDD path. Even though many of the participants in the MIT study were not versed in TDD, several of them reported that they â€œgot a small part of [their] code working before moving on to the next section, rather than trying to debug everything at the end.â€ Another remarked, â€œIt was easier to see my errors when they were only with one method at a time.â€ Making incremental changes is a core Agile principle, and directly reflected in the practice of TDD. Continuous testing makes the benefits of incremental development immediately apparent.

Tests Stay Fast

Using continuous testing helps keep your tests fast. This is something that was not mentioned in the MIT study, but I can attest to it in my own work. If you write a slow test, you’ll know right away, and you’ll probably have a good idea exactly what is making the test slow (because you just added it). I think the reason that they didn’t mention it in the MIT study is that they weren’t working with legacy code, or external systems, which is where a lot of test slowness comes from.

Today, there are a number of tools that support continuous testing:

CT-Eclipse (was Continuous Testing Plugin for Eclipse) – Java/Eclipse
ZenTest::Autotest – Ruby
Fireworks – Java/IntelliJ
Infinitest – Java

For full disclosure, I should mention that I’m the admin of the Infinitest project. Our goal is to create a continuous test runner for every development platform. We’ve already released a Java test runner, and we’re nearing completion of a Python runner.

Continuous testing is a practice that I have followed in my own work for most of 2007. There is little doubt in my mind that it is the future of TDD. Looking forward to the days of multicore systems, continuous testing may be an effective way to realize productivity improvements through better hardware. So long as they are independent (as they should be), test runs can easily be distributed across multiple cores, and perhaps even, multiple machines.

"Continuous Testing Explained" by Youtube converter

Sun, 25 Jul 2010 21:42:34 -0500

Thank you for posting this.

"Continuous Testing Explained" by dvd to ipad mac

Tue, 06 Jul 2010 02:53:27 -0500

this ah ha

"Continuous Testing Explained" by ilan

Mon, 28 Jun 2010 15:03:50 -0500

Anyway, that is moot in yours world but I figure you also have spelling skills in terms of programming code word. I am not picking a fight, just very curious.

"Continuous Testing Explained" by galbur

Mon, 28 Jun 2010 07:51:58 -0500

Anyway, that is moot in yours world but I figure you also have spelling skills in terms of programming code word. I am not picking a fight, just very curious.

"Continuous Testing Explained" by http://www.ostube.net/

Mon, 28 Jun 2010 06:42:47 -0500

The web client and the window and the java script causes problem for some. Anyway as everything has some sort of disadvantages this software may also have them but the point we have to note here is that how many people

"Continuous Testing Explained" by ask ve sevgi

Mon, 28 Jun 2010 06:07:43 -0500

Hah,that was really funny. I even ordered an essay paper on it.These online essay writers made an excellent essay for me.

"Continuous Testing Explained" by sohbet

Wed, 19 May 2010 11:34:12 -0500

"Continuous Testing Explained" by Kent

Wed, 19 May 2010 09:24:43 -0500

Are expert programmers not expert spellers? Does this point matter? I just noticed that you misspelled panicky. Anyway, that is moot in your world but I figure you also have spelling skills in terms of programming code words. I am not picking a fight, just very curious.

statement writing

"Continuous Testing Explained" by daniel

Wed, 19 May 2010 06:04:53 -0500

I agree that one less button to push is a good thing. I would have thought of a different meaning for continuous meaning if you have not explained it clearly. I would have thought it something negative and tedious.

Heartbreak

"Continuous Testing Explained" by Frederick

Tue, 18 May 2010 11:17:10 -0500

Thanks for the information. I love tips on how to become a much-improved programmer. I have not gone through formal programming training so any info from the net is a great help.

Free Bondage Videos

"Continuous Testing Explained" by ShAaNiG

Mon, 12 Apr 2010 05:56:56 -0500

We also use conditional checking. What this does is perform a number of builds using the build server with the proposed changes and gives you the option of checking it in iff the builds all pass. This means you can do checking and go to lunch or go home if it breaks a build, the checking fails.

Wholesalers

"Continuous Testing Explained" by ShAaNiG

Sat, 10 Apr 2010 11:09:04 -0500

Erik – Wow, yeah, that takes on a whole new meaning.

Wholesale Brand Name Clothing

"Continuous Testing Explained" by ShAaNiG

Fri, 02 Apr 2010 05:40:03 -0500

We also use conditional checking. What this does is perform a number of builds using the build server with the proposed changes and gives you the option of checking it in if the builds all pass. This means you can do checking and go to lunch or go home if it breaks a build, the checking fails.

Prudential West

"Continuous Testing Explained" by ShAaNiG

Fri, 02 Apr 2010 05:39:43 -0500

Erik – Wow, yeah, that takes on a whole new meaning…

Prudential West

"Continuous Testing Explained" by face blog

Sat, 27 Mar 2010 00:53:01 -0500

This is the best short and sweet explanation of What is continuous testing I found ever. Great post, brilliant.

"Continuous Testing Explained" by psn card

Tue, 16 Mar 2010 22:09:57 -0500

Hah,that was really funny. I even ordered an essay paper on it.These online essay writers made an excellent essay for me.

"Continuous Testing Explained" by top auto insurance companies

Sat, 05 Dec 2009 10:40:03 -0600

A great blog post makes you think and you’ve certainly given me a few tid bits to consider.

"Continuous Testing Explained" by beliebte Kasinos

Fri, 22 May 2009 03:02:08 -0500

I admit, I have not been on this webpage in a long time… however it was another joy to see It is such an important topic and ignored by so many professionals. I thank you to help making people more aware of possible issues. Great stuff as usual….

"Continuous Testing Explained" by Daniele

Thu, 24 Jul 2008 09:06:55 -0500

ct-eclipse seems to be dead and the only installation instructions page is broken… http://ct-eclipse.tigris.org/install.html

"Continuous Testing Explained" by Peter Lawrey

Sat, 22 Sep 2007 14:32:24 -0500

We use continous integration and release so that successful releases are immediately available as snapshots to other projects.

We also use conditional checkin. What this does is perform a number of builds using the build server with the proposed changes and gives you the option of checking it in iff the builds all pass. This means you can do checkins and go to lunch or go home if it breaks a build, the checkin fails.

"Continuous Testing Explained" by Ben

Fri, 21 Sep 2007 08:48:59 -0500

Erik – Wow, yeah, that takes on a whole new meaning…

Ha!

Let me fix that….

"Continuous Testing Explained" by Erik

Thu, 20 Sep 2007 22:22:04 -0500

<fun about typo>
Did you really mean “continuous complication”? I’d vote for “continuous compilation”.

Probably a Freudian slip – if so, you should talk to your analyst about your relationship with your IDE ;-) Or was it just some brainless editor trying to correct a typo?
</fun about typo>
Besides that, I totally agree with you on the topic.

"Continuous Testing Explained" by Samuel A. Falvo II

Thu, 20 Sep 2007 21:17:06 -0500

I’ve been doing this for years, but not because I’m smarter than anyone else. It was borne out of pure laziness. Having to edit a file, then exit, then “make check” is pretty quick, but considering how fast development runs can be, it still becomes a burden. So, I use plain shell scripts to automate the task for me.

If you prefer to keep your editor running as you hack on code, I’m sure a vim- or elisp-script can be written to implement the desired behavior (assuming you’re using either vim or emacs). You do not need to use a special purpose IDE.