Object Mentor Blog: TDD Derangement Syndrome

TDD Derangement Syndrome

Uncle Bob — Wed, 07 Oct 2009 08:32:00 -0500

My recent blog about TDD, Design Patterns, Concurrency, and Sudoku seemed to draw the ire of a few vocal TDD detractors. Some of these people were rude, insulting, derisive, dismissive, and immature. Well, Halloween is not too far away.

In spite of their self-righteous snickering they did ask a few reasonable questions. To be fair I thought it would be appropriate for me to answer them.

Is there any research on TDD?

It turns out that there is a fair bit.

One simple google search led me to this blog by Phil Haack in which he reviewed a TDD research paper. Quoting from the paper:

We found that test-first students on average wrote more tests and, in turn, students who wrote more tests tended to be more productive. We also observed that the minimum quality increased linearly with the number of programmer tests, independent of the development strategy employed.

The same google search led me to this blog by Matt Hawley, in which he reviewed several other research papers. Part of his summary:

* 87.5% of developers reported better requirements understanding. * 95.8% of developers reported reduced debugging efforts. * 78% of developers reported TDD improved overall productivity. * 50% of developers found that it decreased overall development time. * 92% of developers felt that TDD yielded high-quality code. * 79% of developers believed TDD promoted simpler design.

Actually, I recognize some of Matt’s results as coming from a rather famous 2003 study (also in the list of google results) by Laurie Wiliams and Boby George. This study describes a controlled experiment that they conducted in three different companies. Though Matt’s summary above is based (in part) on that study, there is more to say.

In the George-William study teams that practiced TDD took 16% longer to claim that they were done than the teams that did not practice TDD. Apparently tests are more accurate than claims since the non-TDD teams failed to pass one third of the researcher’s hidden acceptance tests, whereas the TDD teams passed about 6 out of 7. To paraphrase Kent Beck: “If it doesn’t have to work, I can get it done a lot faster!”

Another point of interest in this study is that the TDD teams produced a suite of automated tests with very high test coverage (close to 100% in most cases) whereas most of the non-TDD teams did not produce such a suite; even though they had been instructed to.

Jim Shore wrote a review of yet another research summary which I found in the same google search. This one combines 7 different studies (including George-Williams). Here the results range from dramatically improved quality and productivity to no observed effect.

Finally, there is this 2008 case Study of TDD at IBM and Microsoft which shows that TDDers enjoy a defect density reduction ranging from 30% to 90% (as measured by defect tracking tools) and a productivity cost of between 15% and 35% (the subjective opinion of the managers). I refer you back to Kent Beck’s comment above.

I’m sure there is more research out there. After all this was just one google search. I think it’s odd that the TDD detractors didn’t find anything when they did their google searches.

Oh yeah, and then there was that whole issue of IEEE Software that was dedicated to papers and research on TDD.

What projects have been written with TDD, hmmm?

Quite a few, actually. The following is a list of projects that have an automated suite of unit tests with very high coverage. Those that I know for a fact use TDD, I have noted as such. The others, I can only surmise. If you know of any others, please post a comment here.

JUnit. This one is kind of obvious. JUnit was written by Kent Beck and Erich Gamma using TDD throughout. If you measure software success by sheer distribution, this particular program is wildly successful.

Fit. Written by Ward Cunningham. The progenitor of most current acceptance testing frameworks.

FitNesse. This testing framework has tens of thousands of users. It is 70,000 lines of java code, with 90%+ code coverage. TDD throughout. Very small bug-list. Again, if you measure by distribution, another raving success.

Cucumber,
Rspec. These two are Testing frameworks in Ruby. Of course you’d expect a testing framework to be written with TDD, wouldn’t you? I know these were. TDD throughout.

Limelight. A gui framework in JRUby. TDD throughout.
jfreechart.
Spring
JRuby
Smallsql
Ant
MarsProject
Log4J
Jmock

Are there others? I’m sure there are. This was just a quick web search. Again, if you know of more, please add a comment.

"TDD Derangement Syndrome" by Mania

Sun, 07 Mar 2010 06:09:18 -0600

The problem is, not everybody considering this/

"TDD Derangement Syndrome" by tieTYT

Thu, 15 Oct 2009 17:59:17 -0500

Uncle Bob: I wish you’d respond to the comment @triple-dot made (or at least the article he linked to). I think it was the most interesting comment in this whole thread.

"TDD Derangement Syndrome" by NotherBob

Mon, 12 Oct 2009 15:04:09 -0500

Don’t know what/when Kent said it, but this was the punch line of a story that went around before Kent was born. Here’s the version I remember:

It seems a team of developers had implemented a complex set of business rules but their code was full of bugs and kept crashing. So another developer reimplemented the rules as a table-driven process. When he presented it to the team, the leader said, “But ours is implemented in straight-line code. Your table-driven code will be a lot slower.” The developer replied, “But mine works.” They went back and forth like this for a few rounds, until the developer finally said, “Hey, if it doesn’t have to work, I can make mine as fast as you want.”

You’ve repurposed the joke to apply to development time. Still works.

"TDD Derangement Syndrome" by Joseph Beckenbach

Mon, 12 Oct 2009 08:19:19 -0500

Another app suite using TDD (as part of 18 months of small-team dials-to-11 Extreme Programming joy-ride): Eidogen’s TIP. (bioinformatics)

As of my last involvement, April 2005: 250K lines of Java, all TDD by 350K lines of unit test code. 100% coverage by team fiat, only two defects (both cosmetic) shipped to customers, with some millions in revenues earned. (The 18 months before XP and TDD had far less code and few tests of any sort, with nothing released and no revenue.)

In early 2005, we solved the SARS virus, ready to find drug candidates in silico, six months before the crystallographers. TDD made that capability happen.

"TDD Derangement Syndrome" by Stephane

Sun, 11 Oct 2009 17:10:09 -0500

Moq, one of the major Mocking framework in .NET has been written with TDD the whole way according to his author in this article. But I never checked the code myself :)

http://www.clariusconsulting.net/blogs/kzu/archive/2009/09/29/AreyousmartenoughtodowithoutTDD.aspx

"TDD Derangement Syndrome" by Philip Schwarz

Thu, 08 Oct 2009 19:21:20 -0500

@Steve Seymour

You said:

Oh my God, Uncle Bob. Are you still trying to convince the world that TDD is a wonderful technique that all disciplined, professional software engineers should follow? And are there still legions of people arguing against this?

These battles last decades! Less than a month ago, in this interview, Kent Beck, the father of TDD, said:

every idea that I have had that has taken off has taken 20 years, at least, so, TDD at this point is close to that old, and I would say that it is pretty much on track with the other things that I have done ...

and TDD seems to be like the second thing [after continuous integration] that is going to have some impact

"TDD Derangement Syndrome" by Chuck van der Linden

Thu, 08 Oct 2009 13:43:11 -0500

>>I can’t write a TDD suite before coding when I’m not entirely sure what I’ll be coding.

If you’re not entirely sure what you’ll be coding, you don’t understand the problem you’re trying to solve, or how what you are about to create is supposed to behave.

I would argue that in such a situation is is even MORE important to write the tests first as it forces you to gain a better understanding of what you are trying to create. Of how what you are about to build is supposed to behave, and how you know it’s working correctly or not. How you know you are done.

That’s a major part of TDD, it’s not just the writing of the tests, it’s that in order to write the tests you have to get yourself to a point that you a good enough understanding of what you’re supposed to be building that you CAN write the tests. It makes you better define the problem space.

If you run off and start coding before you know what you’re doing, how do you know you’ve written the right code? how do you avoid ending up writing a bunch of stuff that doesn’t matter? sure you could get lucky and fall into none of those traps, but having the test there first makes that a lot less likely.

Another value of writing the test first btw is you can watch it fail. and knowing the test fails without the code working has value all it’s own and is difficult to do after the fact. That in itself tends to create better quality unit tests that are less likely to fail to detect problems if they occur.

(although I’ve heard of some cool tools that make changes to the code and then report if tests don’t fail as a result, which no matter if you are doing tdd or not, is a great concept for making sure you have good tests)

"TDD Derangement Syndrome" by Rick T

Thu, 08 Oct 2009 08:49:11 -0500

TDD is much more obvious, and easy to accomplish with clarity and focus, for those projects that are simply a rewriting of existing code in a different language. The tests are self-evident, as is the destination.

That is the problem with TDD. While it is an obvious win for trivial tasks, for less-trivial, less-obvious tasks the merits are much less obvious. I can’t write a TDD suite before coding when I’m not entirely sure what I’ll be coding.

I guess my point is that if you’re working at an outsourcing sweatshop and the client says “take A and B and sum them to C”, the advantages and route to TDD is self-evident. If that isn’t your genre, it’s much more fuzzy. So when someone gives some claim that TDD is what “professional” developers do, they’re really betraying the simplicity of their task.

"TDD Derangement Syndrome" by Steve Seymour

Thu, 08 Oct 2009 08:06:20 -0500

Oh my God, Uncle Bob. Are you still trying to convince the world that TDD is a wonderful technique that all disciplined, professional software engineers should follow? And are there still legions of people arguing against this? And does everyone belonging to these legions not have the dedication, perseverance, and professionalism to actually commit themselves to TDD for an acceptable period of time before judging it honestly? Would these people rather play with TDD for a few hours or maybe, just maybe, a few days, and then throw it away (like I did during my first exposure to it, though I did it with the idea of returning to it someday, thankfully which I did) and form an inexperienced opinion of it?

I know, I know, you’re in a position of leadership and you are trying desperately to right the ship of the sad software world, a world severely lacking professionalism. But, I can’t help but wonder how much time, energy, and emotion you sink into this debate that is long over for anyone that has sincerely practiced TDD, and what other things you could be doing with your passion. Why not just let those that argue against it go and reserve your time (and mine, for responding to this post) for better things?

And yup, I’m a TDD snob: Anyone not regularly practicing TDD is not a professional developer, and anyone practicing it is. Those that practice it aren’t necessarily great developers, but they are at least professional in their approach. And I must say that some who don’t practice TDD are great developers, but clearly they could be better if they TDDed.

It really is that simple. Anyone that has seriously undertaken the TDD craft knows it, and anyone else doesn’t have the right to voice an opinion.

Please, Uncle Bob, go back to writing down thoughts concerning something else. You are a wonderful writer with often profound thoughts that I very much enjoy reading. Let the lazies go back to their work. Some of them are good developers and most are not, and there is enough written on this topic (by you alone!) to initiate at least a portion of them to try TDD and to realize its vast benefits.

And don’t give too much credit to Joel in order to soften your chastising of him: his original post wasn’t that good.

"TDD Derangement Syndrome" by Esko Luontola

Thu, 08 Oct 2009 03:55:58 -0500

Miško Hevery posted recently his measurements about how much of his time is taken by writing tests. The magic number is about 10%. A big reason is that the test code is always very simple compared to the production code – no conditionals, no loops. Just a couple of method calls and assertions linearly. He also goes to outline the benefits, which make the extra 10% well worth it.

http://googletesting.blogspot.com/2009/10/cost-of-testing.html

"TDD Derangement Syndrome" by Wedge

Thu, 08 Oct 2009 03:46:34 -0500

TDD proponents often come off as arrogant, dogmatic cultists, which I think explains much of the strength of the backlash against TDD.

TDD likely has very real benefits, but even so it is no silver bullet, nor is it a panacea for every software development malady. Pragmatists recognize that different tools and different methodologies are needed in different situations, and over-engineering can be just as dangerous as under-engineering (a carbon-fiber toilet is just as dumb as a construction crane made of toothpicks and glue).

More so, when experienced developers who have long histories of putting out successful, profitable, useful software hear the message that it’s not possible to be a serious software engineer without practicing TDD or that every single software project on Earth should be using TDD they may get a little defensive. Or perhaps more than a little. Indeed, they may mentally put the TDD proponents into the bozo bin.

Again, TDD likely has very real benefits, but those benefits are incremental, and they are not necessarily realized on every software project (few methodologies are). Once TDDers begin to see the world as something other than a universe of nails, then perhaps TDD detractors will have less to complain about, and pragmatists may start taking TDD more seriously.

"TDD Derangement Syndrome" by Jens

Thu, 08 Oct 2009 01:05:24 -0500

Though being quite positive towards TDD I have severe doubts in the results of many of the experiments.

In Karlsruhe, Germany, there was a quasi-experiment showing that TDD research results from experiments with students are not really generalisable to a population of professional developers. http://www.ipd.uka.de/Tichy/uploads/publikationen/136/MuellerHoefer2007.pdf

"TDD Derangement Syndrome" by James

Wed, 07 Oct 2009 23:00:22 -0500

> I’m going to have to test the code I write anyway, and I’m motivated to avoid manual work so I might as well automate it. Besides, I like writing code (writing tests is writing code).

Doesn’t really say much about test first or last. Writting code for the sake of writting code is wasting the client’s time if it can be avoided. This include writting duplicate test cases that test the same functionality in different places.

> I’ll know when I’m done. Then I’ll have a chance to go back and use hindsight to simplify my code.

If you have to go back and simplify your coee, then you’re not done. You might have a functionally correct implementation but a poor one that might not meet the system qualities (non-functional requirements.) You know when you can visibly see your code working through your portotypes. Testing first is not a neccesary indicator.

> I’m addicted to the thrill of passing tests (green bar!)

Even false positives? If you have a psychological need to see green bars then TDD might help you.

> I’d rather write code than documentation or javadocs

This sounds like hacking to me. It is easier to look through diagrams than to look through 100 test cases to immediately know what your code is supposed to do and the relationships between your objects. Furthermore, have you used a library that is poorly documented? Imagine another team is using your component. It’s unprofessional unless you work in a small close-nit team

> I’d rather work on new features, not being stuck in a debugger fixing things I broke because I didn’t have tests.

Didn’t really have anything to do with TDD

> My tests are canned thoughts that I can bottle up and then open up all at once without using brain cycles for thinking through the contexts and complexities I have already thought about. While my tests run, I can think about the next steps or go get a coffee and feel like I’m still being productive.

Sounds to me like you have problem articulating ideas and thoughts without touching the keyboard. In addition, you’re using tests as a procrastination to doing the actual thinking.

> TDD helps me eliminate waste by catching defects before I have even written them.

This doesn’t make sense. Code testing code. If you have bad tests then even with good code, it would still fail sending you on a wild goose chase. This happens to me before. Why do trust your test code more than you trust your production code?

> Better design decisions: TDD helps me keep code clean, simple and loosely coupled.

No it doesn’t. Experience and knowledge in design will help you keep your code clean and loosely couple. If you lack these skills you’re just gonna stuck with bad tests and bad code.

> Lately I have been back-filling missing tests on a legacy code base and I am feeling the pain of writing tests for code that was not written with TDD, and is very hard to test. Because of this, writing the tests takes a lot longer. It would have been much easier to write the tests if they were test-driven in the first place (we have experienced this because there are parts of the system in which we reimplemented with code that was written using TDD)

It’s hard to test because it was designed badly, not because it wasn’t done using TDD. I’ve seen teams who did TDD and ends up writing tests for their tests.

"TDD Derangement Syndrome" by Tim Andersen

Wed, 07 Oct 2009 22:05:50 -0500

I like TDD for the following reasons:

I’m going to have to test the code I write anyway, and I’m motivated to avoid manual work so I might as well automate it. Besides, I like writing code (writing tests is writing code).
I’ll know when I’m done. Then I’ll have a chance to go back and use hindsight to simplify my code.
I’m addicted to the thrill of passing tests (green bar!)
I’d rather write code than documentation or javadocs
I’d rather work on new features, not being stuck in a debugger fixing things I broke because I didn’t have tests.
My tests are canned thoughts that I can bottle up and then open up all at once without using brain cycles for thinking through the contexts and complexities I have already thought about. While my tests run, I can think about the next steps or go get a coffee and feel like I’m still being productive.
TDD helps me eliminate waste by catching defects before I have even written them.
Better design decisions: TDD helps me keep code clean, simple and loosely coupled. Lately I have been back-filling missing tests on a legacy code base and I am feeling the pain of writing tests for code that was not written with TDD, and is very hard to test. Because of this, writing the tests takes a lot longer. It would have been much easier to write the tests if they were test-driven in the first place (we have experienced this because there are parts of the system in which we reimplemented with code that was written using TDD)

"TDD Derangement Syndrome" by James

Wed, 07 Oct 2009 19:56:21 -0500

@Philip Schwarz

The article you linked to makes no sense. It work on a flaw logic of: C works for A, B resembles A therefore C should work for B as well. C in this case is the logic for Queuing Theory, A is the Queuing Theory and B is TDD. The author basically saying that since TDD looks similar to the process described by Queuing Theory, therefore the logic of Queuing Theory must validate TDD.

The important parts of the statement missed by author is highlighted by me.

“Given a process B, which follows a process A, sometimes in performing B we need to perform some of A again. We can remove the need to rework by taking some portion of process B and performing it before process A1.”

Here’s my observation:

1. Test is code. Since code is testing code it doesn’t matter what occurs before or after. 2. writing only the code we need should be a given. The only time when you’re not doing that is when you’re hacking and have no idea what you are doing. 3. Test first only works if you can assert against a known solution. Working out the 1000th largest prime does not have a known solution at the start. 4. Short cycles is not exclusive to TDD. 5. Imagine a list of requirements [1,2,3,4, ... n]. TDD start with the first requirement and works it way through. There is a high chance of reaching a requirement j (1 < j < n) that would obsolete the requirements before it. TDD would produce wasteful code if the programmer didn’t look at the big picture.

"TDD Derangement Syndrome" by Philip Schwarz

Wed, 07 Oct 2009 19:32:42 -0500

@James

You said: I think you missed this bit:“When I started Max I didn’t have any automated tests for the first month. I did all of my testing manually. After I got the first few subscribers I went back and wrote tests for the existing functionality.”

I did not: Junit and Junit Max are two different products. Re-read Beck’s post to see how the different natures of the two products justify using TDD for one, but not for the other.

"TDD Derangement Syndrome" by Philip Schwarz

Wed, 07 Oct 2009 19:21:51 -0500

@Geir Hedemark

You said: I can just hear the cynics already: “But what real-world, not-a-library-projects have used TDD”?

Is the following, from Misconceptions with Test Driven Development, part of the problem?:

So if TDD can be used on a large scale and to help drive architecture there must be examples. The leading members of the community (JB Rainsberger, Nat Pryce and Lasse Koskela) all have clients with products built and architected using TDD. Unfortunately these are all closed source or clients that don’t want to be talked about publicly.

"TDD Derangement Syndrome" by Philip Schwarz

Wed, 07 Oct 2009 19:13:34 -0500

@asdf You said: TDD is just another religion for hacks to sell books and conferences on

From JB Rainsberger, one of the leading members of the TDD community:

It surprises me, from time to time, how much I still need to justify test-driven development to prospects and would-be course attendees. Many feel that TDD has crossed the chasm, while others still see TDD as a cultish practice worth marginalizing. I take some blame for those who find TDD cultish, because until now I haven’t had a strong, sensible, theoretical basis to justify TDD as an idea. I could do no better than “it works for me” or “my friends like it”. That has changed…

If you buy in to ideas from Theory of Constraints or Lean Manufacturing, then I think I now have a stronger argument to justify the core programming practices in extreme programming in particular and agile software development in general. I don’t even need all of the Theory of Constraints but rather a simple appeal to fundamental concepts in Queuing Theory.

Read the full post, which includes the following take-home point:

Where test-first programming helps most teams most of the time reduce their mistake count to near zero, test-driven development helps them reduce their design inventory—mostly code that gets in our way because it doesn’t actively help us deliver a feature—to near zero. This further increases productivity and improves their ability to deliver by helping them waste less time agonizing over design problems they find costly to fix.

"TDD Derangement Syndrome" by Grok2

Wed, 07 Oct 2009 18:57:33 -0500

I would like real-world examples of projects done by large teams that used TDD. It’s simply point-of-view when a single person or a small team (3-4 cohesive members) say they used TDD for a product or web-site, but it really proves the point if there is a real-world example of a large team using TDD successfully.

"TDD Derangement Syndrome" by noone@none.org

Wed, 07 Oct 2009 18:24:59 -0500

“It seems that Uncle Bob is censoring messages that mention the critique of his first link study by Jacob Proffitt.”

I doubt it. As of now it is very visible in a message by triple-dot above. I think he is completely deluded in his fixation on TDD and denigration of programmers who don’t use it, but give the man his due. He doesn’t censor critical comments (afaik) and has the guts to face his detractors.

"TDD Derangement Syndrome" by James

Wed, 07 Oct 2009 18:15:38 -0500

It seems that Uncle Bob is censoring messages that mention the critique of his first link study by Jacob Proffitt.

If you want to read it, use google and search for “TDD Proven Effective! Or is it?”

"TDD Derangement Syndrome" by James

Wed, 07 Oct 2009 18:13:20 -0500

TDD Proven Effective! Or is it?

http://www.theruntime.com/blogs/jacob/archive/2008/01/22/tdd-proven-effective-or-is-it.aspx

"TDD Derangement Syndrome" by James

Wed, 07 Oct 2009 18:11:45 -0500

@Philip Schwarz

I think you missed this bit.

“When I started Max I didn’t have any automated tests for the first month. I did all of my testing manually. After I got the first few subscribers I went back and wrote tests for the existing functionality.”

"TDD Derangement Syndrome" by Philip Schwarz

Wed, 07 Oct 2009 18:00:18 -0500

@James

You said: How do you manage to write Junit using TDD?

My guess is that you use some form of boostrapping.

"TDD Derangement Syndrome" by Philip Schwarz

Wed, 07 Oct 2009 17:57:34 -0500

@James

You said: How do you manage to write Junit using TDD?...I know that this isn’t wikipedia, but can you provide any source to ANYTHING that you claimed to be done using TDD?

In this blog entry, Kent Beck, one of the authors of JUnit, said:

Working on JUnit, the whole bag of XP practices makes sense. We always test-drive development. We refactor whenever we can, sometimes trying 3-4 approaches before hitting one we are willing to live with.