Bugs Kill Productivity and Schedule Accuracy

Dean Wampler — Sun, 25 Nov 2007 11:29:07 +0000

The title may seem obvious. Here is some hard data.

SD Times recently reported on a survey done by Forrester Research on the pernicious effects of bugs on development productivity. The developers and managers surveyed said they take an average of six days to resolve issues and they spend an average 3 out of every 10 hours working on various stages of troubleshooting, including initial investigation, documenting their findings, and resolving the problems.

The survey puts some data behind an intuitive sense (of gloom…) that many teams have about the problem.

For me, this is further evidence of why the practices of Extreme Programming (XP), especially Test-Driven Development (TDD), are so important. To a high degree, TDD ensures that the software is really working and that regressions are caught and fixed quickly. TDD also provides an unambiguous definition of “done”; do the automated tests pass? The feedback on which stories are really done tells the team its true velocity and hence how many story points will be finished by a given target date. Also, you spend far less unplanned and unpredictable time debugging problems.

Hence, productivity and schedule accuracy tend to be much better for XP teams. Over time, as the software grows in size and complexity, the automated test suite will keep delivering “dividends”, by continuing to catch regressions and by empowering the team to do the aggressive refactorings that are necessary to keep the software evolving to meet new requirements.

The SD Times article goes on to say that

... two-thirds of the responding managers indicated that a solution that reduced the time spent on resolving application problems would be of interest if it created significant efficiencies and improved quality.

The article concludes with a quote that automated testing is one solution. I agree. Just make sure it is the XP kind of automated testing.

Why you have time for TDD (but may not know it yet...)

Dean Wampler — Sun, 30 Sep 2007 20:01:02 +0000

Note: Updated 9/30/2007 to improve the graphs and to clarify the content.

A common objection to TDD is this; “We don’t have time to write so many tests. We don’t even have enough time to write features!”

Here’s why people who say this probably already have enough time in the (real) schedule, they just don’t know it yet.

Let’s start with an idealized Scrum-style “burn-down chart” for a fictional project run in a “traditional” way (even though traditional projects don’t use burn-down charts…).

We have time increasing on the x axis and the number of “features” remaining to implement on the y axis (it could also be hours or “story points” remaining). During a project, a nice feature of burn-down charts is that you can extend the line to see where it intersects the x axis, which is a rough indicator of when you’ll actually finish.

The optimistic planners for our fictional project plan to give the software to QA near the end of the project. They expect QA to find nothing serious, so the release will occur soon thereafter on date T0.

Of course, it never works out that way:

The red line is the actual effort for our fictional project. It’s quite natural for the planned list of features to change as the team reacts to market changes, etc.. This is why the line goes up sometimes (in “good” projects, too!). Since this is a “traditional” project, I’m assuming that there are no automated tests that actually prove that a given feature is really done. We’re effectively running “open loop”, without the feedback of tests.

Inevitably, the project goes over budget and th planned QA drop comes late. Then things get ugly. Without our automated unit tests, there are lots of little bugs in the code. Without our automated integration tests, there are problems when the subsystems are run together. Without our acceptance tests, the implemented features don’t quite match the actual requirements for them.

Hence, a chaotic, end-of-project “birthing” period ensues, where QA reports a list of big and small problems, followed by a frantic effort (usually involving weekends…) by the developers to address the problems, followed by another QA drop, followed by…, and so forth.

Finally, out of exhaustion and because everyone else is angry at the painful schedule slip, the team declares “victory” and ships it, at time T1.

We’ve all lived through projects like this one.

Now, if you remember your calculus classes (sorry to bring up painful memories), you will recall that the area under the curve is the total quantity of whatever the curve represents. So, the actual total feature work required for our project corresponds to the area under the red line, while the planned work corresponds to the area under the black line. So, we really did have more time than we originally thought.

Now consider a Test-Driven Development (TDD) project [1]:

Here, the blue line is similar to the red line, at least early in the project. Now we have frequent “milestones” where we verify the state of the project with the three kinds of automated tests mentioned above. Each milestone is the end of an iteration (usually 1-4 weeks apart). Not shown are the 5-minute TDD cycles and the feedback from the continuous integration process that does our builds and runs all our tests after every block of commits to version control (many times a day).

The graph suggests that the total amount of effort will be higher than the expected effort without tests, which may be true [2]. However, because of the constant feedback during the whole life of the project, we really know where we actually are at any time. By measuring our progress in this way, we will know early whether or not we can meet the target date with the planned feature set. With early warnings, we can adjust accordingly, either dropping features or moving the target date, with relatively little pain. Whereas, without this feedback, we really don’t know what’s done until something, e.g., the QA process, gives us that feedback. Hence, at time T0, just before the big QA drop, the traditional project has little certainty about what features are really completed.

So, we’ll experience less of the traditional end-of-project chaos, because there will be fewer surprises. Without the feedback from automated tests, QA is find lots of problems, causing the chaotic and painful end-of-project experience. Finding and trying to fix major problems late in the game can even kill a project.

So, TDD converts that unknown schedule time at the end into known time early in the project. You really do have time for automated tests and your tests will make your projects more predictable and less painful at the end.

Note: I appreciate the early comments and questions that helped me clarify this post.

[1] As one commenter remarked, this post doesn’t actually make the case for TDD itself vs. alternative “test-heavy” strategies, but I think it’s pretty clear that TDD is the best of the known test-heavy strategies, as argued elsewhere.

[2] There is some evidence that TDD and pair programming lead to smaller applications, because they help avoid unnecessary features. Also, they provide constant feedback to the team, including the stake holders, on what the feature set should really be and which features are most important to complete first.

Object Mentor Blog: Tag schedules

Bugs Kill Productivity and Schedule Accuracy

Why you have time for TDD (but may not know it yet...)