Object Mentor Blog: Continuous Testing Explained

Continuous Testing Explained

Ben Rady — Thu, 20 Sep 2007 16:11:00 -0500

What is continuous testing? It’s turning the knob on Test Driven Development up to 11, by automatically running the tests on every save. This has profound effects on the way that TDD is applied, and is likely to make you a more efficient and productive programmer.

First, a little history…

The term “Continuous Testing” was originally coined by the Program Analysis Group at MIT. Their research found, among other things, that “Developers using continuous testing were three times more likely to complete the [assigned] task before the deadline than those without” and that “continuous testing reduced wasted time [related to testing] by 92â€“98%.” These benefits come from a number of interesting effects:

Less Overhead

The first and most obvious effect of continuous testing is that you don’t have to manually run the tests. One less button to push…that’s always a good thing.

Defect Lag Reduced

Just as modern Java IDEs like IntelliJ and Eclipse increase feedback and productivity though continuous compilation, continuous testing reduces the time between the introduction of an error and it’s detection. This leads to more effective regression testing, because it’s not the expected failures that cost you time, it’s the unexpected ones. Continuous testing finds unexpected failures right away.

Promotes Good TDD Habits

If you’re running the tests on every save, it makes it more difficult to wander off the TDD path. Even though many of the participants in the MIT study were not versed in TDD, several of them reported that they â€œgot a small part of [their] code working before moving on to the next section, rather than trying to debug everything at the end.â€ Another remarked, â€œIt was easier to see my errors when they were only with one method at a time.â€ Making incremental changes is a core Agile principle, and directly reflected in the practice of TDD. Continuous testing makes the benefits of incremental development immediately apparent.

Tests Stay Fast

Using continuous testing helps keep your tests fast. This is something that was not mentioned in the MIT study, but I can attest to it in my own work. If you write a slow test, you’ll know right away, and you’ll probably have a good idea exactly what is making the test slow (because you just added it). I think the reason that they didn’t mention it in the MIT study is that they weren’t working with legacy code, or external systems, which is where a lot of test slowness comes from.

Today, there are a number of tools that support continuous testing:

CT-Eclipse (was Continuous Testing Plugin for Eclipse) – Java/Eclipse
ZenTest::Autotest – Ruby
Fireworks – Java/IntelliJ
Infinitest – Java

For full disclosure, I should mention that I’m the admin of the Infinitest project. Our goal is to create a continuous test runner for every development platform. We’ve already released a Java test runner, and we’re nearing completion of a Python runner.

Continuous testing is a practice that I have followed in my own work for most of 2007. There is little doubt in my mind that it is the future of TDD. Looking forward to the days of multicore systems, continuous testing may be an effective way to realize productivity improvements through better hardware. So long as they are independent (as they should be), test runs can easily be distributed across multiple cores, and perhaps even, multiple machines.

"Continuous Testing Explained" by real simple recipes

Wed, 25 Jan 2012 12:08:03 -0600

Thanks for this bit of history. I have learned something new today.

"Continuous Testing Explained" by Savita

Mon, 23 Jan 2012 01:00:58 -0600

Hi Ben,

I was writing on ‘Continuous Testing’ as I am a tester so lot of questions popped in my mind. I think developer’s do most of the time checking. Like they check the output of some function or sometime if there is any change then the check if it is not changing existing functionality. I think instead of saying ‘Continuous Testing’ it should be ‘Continuous Unit testing’ :-) Because ‘Testing is very vast term. What say?

"Continuous Testing Explained" by Computer repair 89118

Sat, 07 Jan 2012 07:38:31 -0600

"Continuous Testing Explained" by Arizona Internet Marketing

Fri, 16 Dec 2011 09:56:44 -0600

Test, test, test! LOL… Great post, I couldn’t agree more that testing is key to optimizing a great internet marketing plan!

"Continuous Testing Explained" by Arizona Internet Marketing

Fri, 16 Dec 2011 09:54:35 -0600

Test, test, test! LOL… Great post, I couldn’t agree more that testing is key to optimizing a great internet marketing plan!

"Continuous Testing Explained" by indianapolis seo

Tue, 06 Dec 2011 11:16:59 -0600

Can’t test enough! You want increase your bottom line…test.

"Continuous Testing Explained" by outsourced it support

Mon, 21 Nov 2011 13:56:32 -0600

Very interesting stuff, this is an area I need to better educate myself on (coding,etc.). outsourced IT support

"Continuous Testing Explained" by Beats By Dre Canada

Sat, 12 Nov 2011 02:21:29 -0600

is critical.M810 at both ends of the roller headset buttons, you can toggle to adjust.

"Continuous Testing Explained" by http://www.airmaxbwltd.com/

Thu, 10 Nov 2011 20:55:04 -0600

pas cher nike air max Tn , Adidas pour la bataille deuxième et troisième circuits de commercialisation niveaux plus tragiques Chaussures et industrie de l’habillement Magang commentateurs indépendants de la micro-Boti, le nombre de domestiques siège d’articles de sport de plus de 40 000, par tête de Li Ning et Anta a déclaré, en s’appuyant sur la croissance pour améliorer les performances de la fin de canal de l’époque, cependant, air max tn et d’une plans Adidas dans les cinq prochaines années, a déclaré les cinq prochaines années, augmenter la deuxième et troisième ligne de nouveaux magasins, l’introduction de produits au plus près du marché avec des échos deuxième et troisième ligne, destiné aux marché principal de la marque nationale, les deux prochaines canaux du marché de troisième ligne en compétition pour la guerre plus cruelle. Nike Air Jordan 11 marques AJ anniversaire 25 Causer de la fièvre au cours des quatre dernières années, mais nous ne pouvons ignorer si rapidement mai énumérés air max 90s Air Jordan 11 AJ 25 anniversaire de la couleur! En fait, un examen plus attentif trouvera que le talon n’est pas le traditionnel “Air” logo, mais on dit que 25 paires de logo 25 anniversaire imprimés, mais pas encore exposées. Maintenant regardons cette paire j’ai fait spécialement pour Michael Jordan Air Jordan XI 25 anniversaire de la photos publicitaires. Nous pouvons clairement voir l’argent métallique “Air” logo.

"Continuous Testing Explained" by Custom Backpacks

Fri, 04 Nov 2011 08:13:01 -0500

good job, very nice article

"Continuous Testing Explained" by christian louboutin

Sun, 30 Oct 2011 05:18:06 -0500

Good artical,I learn something!

"Continuous Testing Explained" by Ashley Bowling

Sun, 16 Oct 2011 05:34:23 -0500

Accounts receivable—where the company enters money received

"Continuous Testing Explained" by DR OZ african Mango

Tue, 27 Sep 2011 11:23:00 -0500

hmm ,i’m not sure if this is what i’m looking for but anyway this is interresting and could be useful some day,thanks for taking time to write such cool stuff

"Continuous Testing Explained" by moncler-jacket-cheap

Tue, 20 Sep 2011 15:10:32 -0500

Cheap Moncler Jackets are not the same thing with women Buy Moncler jackets. Elements of Didcount Moncler Coats Sale down shop, beautiful bright color, men are not only seemed. The deep color of the grid can also elements of chic Moncler Jackets Men. Perhaps also with decorative straps jackets women down.

"Continuous Testing Explained" by dr beats headphones

Fri, 16 Sep 2011 02:11:54 -0500

i have added in my favorites list

"Continuous Testing Explained" by beats by dre store

Tue, 23 Aug 2011 03:15:27 -0500

so I shared it with my friends. ,,, thanks for sharing your posting.high quality headphones new design headphones

"Continuous Testing Explained" by casio

Wed, 06 Jul 2011 09:07:16 -0500

i like this and nice info thanks

"Continuous Testing Explained" by ????? ????

Wed, 06 Jul 2011 09:03:31 -0500

its nice and usefull information and i have added in my favorites list

"Continuous Testing Explained" by mariyawood@yahoo.co.uk

Sat, 18 Jun 2011 07:13:49 -0500

Hey i like this blog.Your article is very nice.Thanks for shearing great information.This info is really useful. http://www.classicbedsteads.co.uk/contact.php.

"Continuous Testing Explained" by unique article wizard article

Fri, 17 Jun 2011 14:50:50 -0500

very nice article ! quite helpful for a newbie like me !

"Continuous Testing Explained" by epub-conversion

Fri, 10 Jun 2011 01:57:08 -0500

Your article is really good and looking for this information for the more than one year….

"Continuous Testing Explained" by kindle conversion

Fri, 10 Jun 2011 01:55:56 -0500

i like this blog to share about the articles..

"Continuous Testing Explained" by blackhawk holsters

Thu, 09 Jun 2011 00:31:47 -0500

Hello Friend,Whichever style of Fashion Shoes you’re looking for, classical,

blackhawk holsters

"Continuous Testing Explained" by Hancy

Wed, 08 Jun 2011 10:54:11 -0500

Hello Friend,Whichever style of Fashion Shoes you’re looking for, classical, fashionable, lovely or the latest design, you can find your favorite designer shoes in www.dunkpage.com ,several days ago I bought one pair of shoes from there,It’s beautiful and very comfortable!

"Continuous Testing Explained" by Jewellery

Sat, 04 Jun 2011 05:21:45 -0500

Online UK costume and fashion jewellery shop with, aWEWS