Object Mentor Blog: Generated Tests and TDD

Generated Tests and TDD

Uncle Bob — Thu, 10 Jan 2008 13:59:30 -0600

TDD has become quite popular, and many companies are attempting to adopt it. However, some folks worry that it takes a long time to write all those unit tests and are looking to test-generation tools as a way to decrease that burden.

The burden is not insignificant. FitNesse, an application created using TDD, is comprised of 45,000 lines of Java code, 15,000 of which are unit tests. Simple math suggests that TDD increases the coding burden by a full third!

Of course this is a naive analysis. The benefits of using TDD are significant, and far outweigh the burden of writing the extra code. But that 33% still feels “extra” and tempts people to find ways to shrink it without losing any of the benefits.

Test Generators.

Some folks have put their hope in tools that automatically generate tests by inspecting code. These tools are very clever. They generate random calls to methods and remember the results. They can automatically build mocks and stubs to break the dependencies between modules. They use remarkably clever algorithms to choose their random test data. They even provide ways for programmers to write plugins that adjust those algorithms to be a better fit for their applications.

The end result of running such a tool is a set of observations. The tool observes how the instance variables of a class change when calls are made to its methods with certain arguments. It notes the return values, changes to instance variables, and outgoing calls to stubs and mocks. And it presents these observations to the user.

The user must look through these observations and determine which are correct, which are irrelevant, and which are bugs. Once the bugs are fixed, these observations can be checked over and over again by re-running the tests. This is very similar to the record-playback model used by GUI testers. Once you have registered all the correct observations, you can play the tests back and make sure those observations are still being observed.

Some of the tools will even write the observations as JUnit tests, so that you can run them as a standard test suite. Just like TDD, right? Well, not so fast…

Make no mistake, tools like this can be very useful. If you have a wad of untested legacy code, then generating a suite of JUnit tests that verifies some portion of the behavior of that code can be a great boon!

The Periphery Problem

On the other hand, no matter how clever the test generator is, the tests it generates will always be more naive than the tests that a human can write. As a simple example of this, I have tried to generate tests for the bowling game program using two of the better known test generation tools. The interface to the Bowling Game looks like this:

  public class BowlingGame {
    public void roll(int pins) {...}
    public int score() {...}
  }

The idea is that you call roll each time the balls gets rolled, and you call score at the end of the game to get the score for that game.

The test generators could not randomly generate valid games. It’s not hard to see why. A valid game is a sequence of between 12 and 21 rolls, all of which must be integers between 0 and 10. What’s more, within a given frame, the sum of rolls cannot exceed 10. These constraints are just too tight for a random generator to achieve within the current age of the universe.

I could have written a plugin that guided the generator to create valid games; but such an algorithm would embody much of the logic of the BowlingGame itself, so it’s not clear that the economics are advantageous.

To generalize this, the test generators have trouble getting inside algorithms that have any kind of protocol, calling sequence, or state semantics. They can generate tests around the periphery of the classes; but can’t get into the guts without help.

TDD?

The real question is whether or not such generated tests help you with Test Driven Development. TDD is the act of using tests as a way to drive the development of the system. You write unit test code first, and then you write the application code that makes that code pass. Clearly generating tests from existing code violates that simple rule. So in some philosophical sense, using test generators is not TDD. But who cares so long as the tests get written, right? Well, hang on…

One of the reasons that TDD works so well is that it is similar to the accounting practice of dual entry bookkeeping. Accountants make every entry twice; once on the credit side, and once on the debit side. These two entries follow separate mathematical pathways. In the end a magical subtraction yields a zero if all the entries were made correctly.

In TDD, programmers state their intent twice; once in the test code, and again in the production code. These two statements of intent verify each other. The tests, test the intent of the code, and the code tests the intent of the tests. This works because it is a human that makes both entries! The human must state the intent twice, but in two complementary forms. This vastly reduces many kinds of errors; as well as providing significant insight into improved design.

Using a test generator breaks this concept because the generator writes the test using the production code as input. The generated test is not a human restatement, it is an automatic translation. The human states intent only once, and therefore does not gain insights from restatement, nor does the generated test check that the intent of the code was achieved. It is true that the human must verify the observations, but compared to TDD that is a far more passive action, providing far less insight into defects, design and intent.

I conclude from this that automated test generation is neither equivalent to TDD, nor is it a way to make TDD more efficient. What you gain by trying to generate the 33% test code, you lose in defect elimination, restatement of intent, and design insight. You also sacrifice depth of test coverage, because of the periphery problem.

This does not mean that test generators aren’t useful. As I said earlier, I think they can help to partially characterize a large base of legacy code. But these tools are not TDD tools. The tests they generate are not equivalent to tests written using TDD. And many of the benefits of TDD are not achieved through test generation.

"Generated Tests and TDD" by grad school personal statement

Fri, 23 Dec 2011 09:37:11 -0600

I am absolutely amazed at how terrific the stuff is on this site. I have saved this webpage and I truly intend on visiting the site in the upcoming days. Keep up the excellent work!

"Generated Tests and TDD" by christian louboutin

Thu, 03 Nov 2011 09:11:08 -0500

The professional design make you foot more comfortable. Even more tantalizing,this pattern make your legs look as long as you can,it will make you looked more attractive.Moveover,it has reasonable price.If you are a popular woman,do not miss it.

Technical details of Christian Louboutin Velours Scrunch Suede Boots Coffee:

Color: Coffee
Material: Suede
4(100mm) heel
Signature red sole x

Fashion, delicate, luxurious Christian louboutins shoes on sale, one of its series is Christian Louboutin Tall Boots, is urbanism collocation. This Christian louboutins shoes design makes people new and refreshing. Red soles shoes is personality, your charm will be wonderful performance.

"Generated Tests and TDD" by Ashley Bowling

Sun, 16 Oct 2011 01:46:56 -0500

A fool and his money are soon parted A friend in need is a friend indeed A golden key can open any door

"Generated Tests and TDD" by african Mango dr oz

Tue, 27 Sep 2011 12:02:02 -0500

Cloudsourcing combines on-demand business process outsourcing (BPO) with crowdsourcing technologies to enable companies to purchase quality BPO services on-demand through a pay-per-use model.

"Generated Tests and TDD" by Diablo3

Wed, 14 Sep 2011 14:34:03 -0500

hmm ,i’m not sure if this is what i’m looking for but anyway this is interresting and could be useful some day,thanks for taking time to write such cool stuff

"Generated Tests and TDD" by beats by dr dre headphones

Wed, 08 Jun 2011 21:38:32 -0500

Beats by dr dre studio with look after talk in white. extra attributes on Monster Beats By Dr. Dre Pro Headphones Black a specific tri-fold design and design and carrying circumstance which make for compact and uncomplicated safe-keeping when not in use. Beats by dr dre solo .

"Generated Tests and TDD" by Jewellery

Sat, 04 Jun 2011 05:33:07 -0500

i am happy for your responses

"Generated Tests and TDD" by Discont Louis Vuitton Scarves

Tue, 17 May 2011 21:18:29 -0500

Thank you for sharing information :-

"Generated Tests and TDD" by f350 leveling kit

Thu, 12 May 2011 23:31:20 -0500

Thank you for sharing information :-)

"Generated Tests and TDD" by leveling kit f250

Thu, 12 May 2011 23:30:58 -0500

Cool website very informative :-)

"Generated Tests and TDD" by leveling kit ford

Thu, 12 May 2011 23:30:40 -0500

Thanks you, its very formative :-)

"Generated Tests and TDD" by ford leveling kit

Thu, 12 May 2011 23:30:24 -0500

I always read your blogs and I like it. thank you :-)

"Generated Tests and TDD" by okey oyunu oyna

Wed, 27 Apr 2011 12:10:39 -0500

thank you very much

internette görüntülü olarak okey oyunu oyna, gerçek kisilerle tanis, turnuva heyecanini yasa.

"Generated Tests and TDD" by dory

Sun, 03 Apr 2011 18:45:36 -0500

Useful information will I follow your posts. Social Network

"Generated Tests and TDD" by SEO Firm India

Fri, 18 Mar 2011 13:12:50 -0500

Excellently written article, if only all blogger offered the same content as you, the internet would be a much better place. Please keep it up!

"Generated Tests and TDD" by Sunglass

Wed, 09 Mar 2011 05:19:26 -0600

Buy $10 Replica Designer Sunglasses with 3-day FREE SHIPPING

"Generated Tests and TDD" by cable ties

Thu, 03 Mar 2011 05:30:14 -0600

great effort exerted.

"Generated Tests and TDD" by Tenant Screening

Thu, 17 Feb 2011 15:25:02 -0600

Another important use of TDD is to ensure that you let user stories drive the requirements. In this case, you’d write a user story (e.g. using FIT or the RSpec Story Runner) first, then write a view spec, then a controller spec, then finally a model spec if required.

"Generated Tests and TDD" by Criminal Records

Fri, 11 Feb 2011 12:31:04 -0600

Some of the tools will even write the observations as JUnit tests, so that you can run them as a standard test suite.

"Generated Tests and TDD" by Low Carb Meals

Sun, 30 Jan 2011 23:43:26 -0600

It is proven time and time again that information’s worth is not the main factor which impacts article

promotion results.

"Generated Tests and TDD" by Criminal Check

Thu, 27 Jan 2011 17:01:51 -0600

The tests they generate are not equivalent to tests written using TDD.

"Generated Tests and TDD" by axial fan

Thu, 27 Jan 2011 01:03:09 -0600

This article is very usefull for me! I can see that you are putting a lots of efforts into your blog. I will keep watching in your blog, thanks.

"Generated Tests and TDD" by iPad PDF Transfer for Mac

Mon, 24 Jan 2011 02:30:45 -0600

I really like this essay. Thank you for writing it so seriously. I want to recommend it for my friends strongly. iPad PDF Transfer for Mac can help you transfer ebooks in PDF format from ipad to mac/iTunes.

"Generated Tests and TDD" by iPad to mac transfer

Mon, 24 Jan 2011 02:27:10 -0600

Thanks for shareing! I agree with you. The artical improve me so much! I will come here frequently. iPad to Mac Transfer lets you transfer music, movie, photo, ePub, PDF, Audiobook, Podcast and TV Show from iPad to Mac or iPad to iTunes.

"Generated Tests and TDD" by http://www.blacktowhiteiphone4.com

Thu, 23 Dec 2010 01:48:51 -0600

Enjoy the christmas time and enjoy the hottest white iphone 4. All you need to do is the white iphone 4 conversion kit home.