Object Mentor Blog: Generated Tests and TDD

Generated Tests and TDD

Uncle Bob — Thu, 10 Jan 2008 13:59:30 +0000

TDD has become quite popular, and many companies are attempting to adopt it. However, some folks worry that it takes a long time to write all those unit tests and are looking to test-generation tools as a way to decrease that burden.

The burden is not insignificant. FitNesse, an application created using TDD, is comprised of 45,000 lines of Java code, 15,000 of which are unit tests. Simple math suggests that TDD increases the coding burden by a full third!

Of course this is a naive analysis. The benefits of using TDD are significant, and far outweigh the burden of writing the extra code. But that 33% still feels “extra” and tempts people to find ways to shrink it without losing any of the benefits.

Test Generators.

Some folks have put their hope in tools that automatically generate tests by inspecting code. These tools are very clever. They generate random calls to methods and remember the results. They can automatically build mocks and stubs to break the dependencies between modules. They use remarkably clever algorithms to choose their random test data. They even provide ways for programmers to write plugins that adjust those algorithms to be a better fit for their applications.

The end result of running such a tool is a set of observations. The tool observes how the instance variables of a class change when calls are made to its methods with certain arguments. It notes the return values, changes to instance variables, and outgoing calls to stubs and mocks. And it presents these observations to the user.

The user must look through these observations and determine which are correct, which are irrelevant, and which are bugs. Once the bugs are fixed, these observations can be checked over and over again by re-running the tests. This is very similar to the record-playback model used by GUI testers. Once you have registered all the correct observations, you can play the tests back and make sure those observations are still being observed.

Some of the tools will even write the observations as JUnit tests, so that you can run them as a standard test suite. Just like TDD, right? Well, not so fast…

Make no mistake, tools like this can be very useful. If you have a wad of untested legacy code, then generating a suite of JUnit tests that verifies some portion of the behavior of that code can be a great boon!

The Periphery Problem

On the other hand, no matter how clever the test generator is, the tests it generates will always be more naive than the tests that a human can write. As a simple example of this, I have tried to generate tests for the bowling game program using two of the better known test generation tools. The interface to the Bowling Game looks like this:

  public class BowlingGame {
    public void roll(int pins) {...}
    public int score() {...}
  }

The idea is that you call roll each time the balls gets rolled, and you call score at the end of the game to get the score for that game.

The test generators could not randomly generate valid games. It’s not hard to see why. A valid game is a sequence of between 12 and 21 rolls, all of which must be integers between 0 and 10. What’s more, within a given frame, the sum of rolls cannot exceed 10. These constraints are just too tight for a random generator to achieve within the current age of the universe.

I could have written a plugin that guided the generator to create valid games; but such an algorithm would embody much of the logic of the BowlingGame itself, so it’s not clear that the economics are advantageous.

To generalize this, the test generators have trouble getting inside algorithms that have any kind of protocol, calling sequence, or state semantics. They can generate tests around the periphery of the classes; but can’t get into the guts without help.

TDD?

The real question is whether or not such generated tests help you with Test Driven Development. TDD is the act of using tests as a way to drive the development of the system. You write unit test code first, and then you write the application code that makes that code pass. Clearly generating tests from existing code violates that simple rule. So in some philosophical sense, using test generators is not TDD. But who cares so long as the tests get written, right? Well, hang on…

One of the reasons that TDD works so well is that it is similar to the accounting practice of dual entry bookkeeping. Accountants make every entry twice; once on the credit side, and once on the debit side. These two entries follow separate mathematical pathways. In the end a magical subtraction yields a zero if all the entries were made correctly.

In TDD, programmers state their intent twice; once in the test code, and again in the production code. These two statements of intent verify each other. The tests, test the intent of the code, and the code tests the intent of the tests. This works because it is a human that makes both entries! The human must state the intent twice, but in two complementary forms. This vastly reduces many kinds of errors; as well as providing significant insight into improved design.

Using a test generator breaks this concept because the generator writes the test using the production code as input. The generated test is not a human restatement, it is an automatic translation. The human states intent only once, and therefore does not gain insights from restatement, nor does the generated test check that the intent of the code was achieved. It is true that the human must verify the observations, but compared to TDD that is a far more passive action, providing far less insight into defects, design and intent.

I conclude from this that automated test generation is neither equivalent to TDD, nor is it a way to make TDD more efficient. What you gain by trying to generate the 33% test code, you lose in defect elimination, restatement of intent, and design insight. You also sacrifice depth of test coverage, because of the periphery problem.

This does not mean that test generators aren’t useful. As I said earlier, I think they can help to partially characterize a large base of legacy code. But these tools are not TDD tools. The tests they generate are not equivalent to tests written using TDD. And many of the benefits of TDD are not achieved through test generation.

"Generated Tests and TDD" by Curtis Cooley

Fri, 27 Jun 2008 15:49:53 +0000

The double entry accounting analogy is pure brilliance!

"Generated Tests and TDD" by subway coupons

Mon, 05 May 2008 09:02:11 +0000

i also interested too

"Generated Tests and TDD" by 88250

Tue, 18 Mar 2008 22:34:12 +0000

en, I am a Chinese newbie in TDD. I wanna translate this entry into chinese, may i?

"Generated Tests and TDD" by mmorpg

Wed, 05 Mar 2008 16:30:45 +0000

funny how something with 33% more code can be that much more efficient…some of these algorithms people are developing have me in severe coder envy.

"Generated Tests and TDD" by Yazid

Fri, 22 Feb 2008 05:49:26 +0000

Hello,

I love TDD (or Test first), recently Microsoft research team created something called a tool called PEX.

This a description of Pex

Pex (Program EXploration) is an intelligent assistant to the programmer. By automatically generating unit tests, it helps to find bugs early. In addition, it suggests to the programmer how to fix the bugs and here is a link

http://research.microsoft.com/Pex/

Can this tool be useful or does it defeat the goal of TDD?

Thx Dr Y Arezki

"Generated Tests and TDD" by Yet another Bob

Fri, 15 Feb 2008 03:55:01 +0000

Bob,

What about code generation? Some people suggest that for certain solutions where metaprogramming is being applied you should generate tests and the corresponding code you want to test – but then you are not really expressing the intent twice (weeellll, the generating code consists of two parts, with one part generating the tests and another one generating the code).

Of course you can TDD the generation code and let it generate simple examples that you test automatically as well.

What’s your take on this?

"Generated Tests and TDD" by unclebob

Tue, 05 Feb 2008 15:59:17 +0000

For example the problem of the number of pins is easily solved by creating an integer value type that only has the range of 0-10. So when it comes time to generate a random value it can only generate valid numbers.

The problem is that a valid game is not just a sequence of rolls between 0 and 10. Within any frame (usually two rolls) the sum of the rolls cannot exceed 10; but if the first roll is 10, then that rule doesn’t count.

Trying to capture all the constraints for valid rolls is tantamount to writing the scoring algorithm.

"Generated Tests and TDD" by Ross MacGregor

Wed, 30 Jan 2008 01:22:36 +0000

Bob, I’m not sure the Periphery Problem as you’ve stated it is really a good argument.

“The test generators could not randomly generate valid games. Itâ€™s not hard to see why. A valid game is a sequence of between 12 and 21 rolls, all of which must be integers between 0 and 10. Whatâ€™s more, within a given frame, the sum of rolls cannot exceed 10. These constraints are just too tight for a random generator to achieve within the current age of the universe.”

Here you assume these constraints cannot be captured by the system, but why couldn’t they? Perhaps we need languages with more powerful constraint systems like Eiffel that has championed Design by Contract programming.

For example the problem of the number of pins is easily solved by creating an integer value type that only has the range of 0-10. So when it comes time to generate a random value it can only generate valid numbers.

DBC like TDD is a methodology for designing software that promises to increase the quality of the design. Perhaps with enough practice with DBC programming one would be able to capture most of these elusive constraints you speak of. You want to capture these constraints programmatically so that you can verify that your software is not operating in an invalid state.

If DBC programming languages were more common perhaps these tools may actually work well and be able to auto generate most of the unit tests.

"Generated Tests and TDD" by choy

Sun, 13 Jan 2008 16:14:35 +0000

the double-entry bookkeeping analogy is insightful. i think it’d be interesting to take this analogy too far.

"Generated Tests and TDD" by Eric Landes

Sun, 13 Jan 2008 16:09:46 +0000

Bob, a great point here. I’ve posted some more thoughts at the URL I’ve posted, that relate to TDD and using that with Visul Studio Testing (for those new to TDD).

"Generated Tests and TDD" by Amund

Sun, 13 Jan 2008 09:26:18 +0000

quote: “The burden is not insignificant. FitNesse, an application created using TDD, is comprised of 45,000 lines of Java code, 15,000 of which are unit tests. Simple math suggests that TDD increases the coding burden by a full third!”

That is not simple math, it is likely to be advanced-alternative-history-math. How do you know that it would end up with 30k lines (and still work) if it was written without TDD?

TDD also drives design and my guess is that you are likely to end with quite a different application when using other development approaches.

"Generated Tests and TDD" by DAR

Fri, 11 Jan 2008 13:35:46 +0000

Actually, the tests increased the size of the code base by 50%.

“45,000 lines of Java code, 15,000 of which are unit tests”

So that means 30K LOC without tests. 15K/30K = .5. So +15K means +50%.

I don’t have a problem with that (I’m a strong TDD advocate myself). But it doesn’t serve anybody well to have the numbers wrong.

"Generated Tests and TDD" by unclebob

Fri, 11 Jan 2008 12:54:37 +0000

The portion of TDD code may be 1/3, but the coding burden is increased by 1/2.

Damn! How do you TDD a blog?

"Generated Tests and TDD" by Steve Meuse

Fri, 11 Jan 2008 12:22:54 +0000

Arghh. The “Given”/”We Know” block lost its formatting. It should be eight lines, which Preview displays correctly:

Given:
     FitNesseWithTDD = 45,000 LOC
     TDDPortion = 15,000 LOC
We know:
     FitNesseWithoutTDD = 30,000 LOC
     RatioOfExtraTDDCode = TDD / FitNesseWithoutTDD
     = 15,000 / 30,000
     = 1/2

"Generated Tests and TDD" by Steve Meuse

Fri, 11 Jan 2008 12:13:28 +0000

Very nice article, insightful and clear. One tiny glitch: Uncle Bob wrote, “The burden is not insignificant. FitNesse, an application created using TDD, is comprised of 45,000 lines of Java code, 15,000 of which are unit tests. Simple math suggests that TDD increases the coding burden by a full third!”

The portion of TDD code may be 1/3, but the coding burden is increased by 1/2.

Given: FitNesseWithTDD = 45,000 LOC TDDPortion = 15,000 LOC We know: FitNesseWithoutTDD = 30,000 LOC RatioOfExtraTDDCode = TDD / FitNesseWithoutTDD = 15,000 / 30,000 = 1/2

Of course, this doesn’t account for the time saved finding bugs before and during the coding phase, rather than retrospectively. A hypothetical FitNesse developed with traditional testing methods could well be bigger than 30,000 LOC. Still, the delta is a bit higher than a third, just so the folks who write the checks and set the schedules know what to expect and when.

I agree with Pavel. The bookkeeping analogy is brilliant. Thanks!

"Generated Tests and TDD" by Jeff Langr

Fri, 11 Jan 2008 08:39:33 +0000

I’m wondering if FitNesse would be 75,000 total lines, no tests, were it not written test-first.

"Generated Tests and TDD" by Pavel Tcholakov

Fri, 11 Jan 2008 07:49:34 +0000

Great post, and the double entry bookkeeping analogy is excellent!

"Generated Tests and TDD" by Johan Samyn

Thu, 10 Jan 2008 15:15:10 +0000

A comment from a somewhat unusual angle :
Two things I like most about this post : the comparison with the accounting practice (great for advocacy), and the referring to the human factor. The second the most. Indeed, we humans are the most important factor, and will stay so for quite some time I believe. This helps me understand why you run a successful company : you seem to value people, the single most important asset there is. And that’s a great thing. You consider humans as an important factor in the process of writing software. Not just the languages, tools, methodologies and so on. But those people using all that. They are the binding glue, the commanding factor in the process. The importance of us, as valuable human beings, can’t be stressed enough. That’s why other/new tools and so can’t beat the fact that you can get more out of a team by educating them, because that is helping to make those good people (the best factor in the game) even better. Which is not always understood.

"Generated Tests and TDD" by swombat

Thu, 10 Jan 2008 14:28:46 +0000

Very true. As a dedicated RSpec BDDer, it is a bit shocking to hear of people trying to take the shortcut of generating their tests. If I was in a smart-arse mood, I might comment that they should take an even quicker shortcut and just generate the test reports directly! Why bother running the tests at all? They’ve already cut out 90% or so of the benefits of testing, why not go all the way? :-)

One element which is missing from your article is the use of TDD as a design process. This is especially the case in BDD, but as BDD is supposed to simply be “TDD done right”, with a better adapted vocabulary, what’s true of BDD tends to hold for TDD as well. When you write tests first, it makes you think about the design of the item you’re writing in a way that’s immensely helpful.

Another important use of TDD is to ensure that you let user stories drive the requirements. In this case, you’d write a user story (e.g. using FIT or the RSpec Story Runner) first, then write a view spec, then a controller spec, then finally a model spec if required. Thus, every line of new code that you write is driven by a clear user benefit, and you waste no time implementing features that You Ain’t Gonna Need (It). In my experience, this goes a long way towards reducing cruft and keeping your codebase tight and focused on user benefits.

Daniel