Object Mentor Blog: To Test The Untestable Bug.

To Test The Untestable Bug.

Uncle Bob — Thu, 13 Aug 2009 13:56:12 -0500

I found a bug that couldn’t be tested yesterday.

There is nothing quite so satisfying as isolating a bug, writing a test that fails because of that bug, and then making the test pass by fixing the bug. The fact that the test is passing is proof that the bug is fixed. The existence of that test in the test suite ensures that the bug will never come back again.

For ten years I have worked this way. For ten years I have taught, and coached, and otherwise cajoled software developers to work this way too. Sometimes those developers would ask me about bugs that couldn’t be tested; and I would reject the whole concept. “If you can write a line of code, you can test that line of code.” I would reply.

But yesterday I found a bug that I couldn’t write a test for.

Actually, that’s not quite fair. I was able to write a test for the bug. The problem was that the test passed, and I couldn’t make it fail.

But let me set the stage first…

My son Justin is apprenticing with me this summer. He’s been helping me write new features for FitNesse. We have had many pleasant weeks working together.

About a week ago I asked him to rework an area of the FitNesse code that includes SuiteSetUp and SuiteTearDown pages into test suites. These pages are executed once per suite run. They are similar to @BeforeClass and @AfterClass functions in JUnit.

The old mechanism started with the suite page and found the nearest inherited SuiteSetUp and SuiteTearDown for that page. Then it would include the SuiteSetUp, all the tests, and finally the SuiteTearDown. The new mechanism needed to be smarter. It would look at every test page and determine the most appropriate SuiteSetUp and SuiteTearDown for that page. Then it would group the test pages by common SuiteSetUp and SuiteTearDown pairs and include them in the suite.

OK, I know this is a bit detailed, but I’m almost done. The point is that the old mechanism created a list of tests that looked like this:

Utttt...ttttD

Whereas the new mechanism created a list of tests that looked like this:

UtttDUttDUtttttDUtD...

Where U and D are SuiteSetUp and SuiteTearDown pages, and the t’s are test pages.

Justin got all his unit tests passing, cleaned up the code, and then showed it to me. We spent a little more time cleaning it up, until we were both satisfied with it.

Then Justin said to me: “We should write an acceptance test for this.”

We write a lot of acceptance tests for FitNesse; and, of course, we use FitNesse to write those tests. There are over 200 FitNesse tests for FitNesse. But until Justin said it, it didn’t occur to me that this feature should have an acceptance test. But after some thought, it was clear that it should.

So Justin and I set about to write the acceptance test for this feature. It was actually pretty straight forward. We simply used a shadow instance of FitNesse to create a test suite with several test pages and appropriate SuiteSetUp and SuiteTearDown pages. Then we executed the suite in the shadow instance, and dissected the HTML to ensure that the setups and teardowns were included in the right order. Easy.

We used a Slim OrderedQuery table to ensure that the list of tests, setups, and teardowns was in the right order. One of the Ordered Query tables looked like this:

!|ordered query:pages run in suite|SuiteChildOne  |
|page name                                        |
|SuiteChildOne.SuiteSetUp                         |
|SuiteChildOne.TestOneOne                         |
|SuiteChildOne.TestOneTwo                         |
|SuiteChildOne.SuiteTearDown                      |                                     
|SuiteChildOne.SuiteSetUp                         |
|SuiteChildOne.TestOneThree                       |
|SuiteChildOne.SuiteTearDown                      |

The TestOneOne and TestOneTwo pages were Fit tests. The TestOneThree page was a slim test. These kinds of tests execute in different JVMs, therefore they needed to be run separately, with their own execution of SuiteSetUp and SuiteTearDown.

Oddly, this table failed, and the failure was inexplicable. It kept complaining that the rows were out of order, when they clearly were correct.

After a little inspection and fiddling, Justin said: “I think the Ordered Query table doesn’t like duplicates.”

Having written the Ordered Query table about 6 months ago, I reviewed the code in my mind and realized he probably had a point. The Ordered Query table was derived from Query table. Query tables check that all specified rows exist in a query, but don’t care about order. Ordered Query tables ensure that order is correct too. Having based the Ordered Query algorithm on the Query algorithm, I realized that there might indeed be a duplication issue.

So, we modified the test to eliminate the need for duplicates; and I wrote a lighthouse task that described the bug.

Yesterday I went to lighthouse and saw that bug there. I decided to fix it. So I wrote an acceptance tests expecially for it. It looked like this:

!|Ordered Query:duplicate rows|A|
|x                              |
|SuiteChildOne.SuiteSetUp       |
|SuiteChildOne.TestOneOne       |
|SuiteChildOne.TestOneTwo       |
|SuiteChildOne.SuiteTearDown    |
|SuiteChildOne.SuiteSetUp       |
|SuiteChildOne.TestOneThree     |
|SuiteChildOne.SuiteTearDown    |

Note the similarity?

Oddly, the test passed. I scratched my head about this, and tried several other different combinations. They all worked just fine. It was as though the bug had disappeared.

I began to doubt that the bug had ever existed. I started rationalizing that perhaps Justin and I had been so focussed on ordering the suites that we didn’t really diagnose the problem properly, and that we had made some silly cockpit error and mistaken it for a bug in OrderedQuery.

So I went to lighthouse and invalidated the bug. Fortunately for me, I left the new acceptance test in place!...

I completed a few more lighthouse tasks, did a git commit, and prepared to push the changes to github. It is our rule that you never push to github without running an ant release. This is the same script that our Hudson server runs whenever it detects a change in github. So, in theory, the hudson build should never fail because nobody pushes to github without running the build.

The build takes about three minutes to run, and it executes every unit and acceptance test in the system. Indeed, this same script is the sole criterion we use for deciding whether FitNesse is ready for release. If the build succeeds, we ship!

Anyway, the ant release failed; and it was the new acceptance test that was the cause!

The new test history feature of fitnesse is really useful. I was able to look at the test run that failed and see exactly what the problem was. And the problem was precisely the same problem that Justin and I saw a week ago. It looked like Ordered Query really did have a problem with duplicates. Sometimes.

So I quickly re-validated the lighthouse task and set about to diagnose the problem with a bit more care. I started up FitNesse again and ran the failing test. It passed!

Oh fratz! How was I going to debug this thing if I couldn’t get it to reliably fail? I tried several different configurations of rows and queries, but they all passed just fine. What’s more, a subsequent run of ant release passed too! I only saw it fail twice, and each time a simple recompile made the problem go away.

So I fell back on a very old skill that I haven’t had to use in over ten years. I started to desk check the code! OK, it’s a bit of an exaggeration to say that I haven’t desk checked in over ten years. Actually I desk check all the time. The point is I always have failing tests to support me. This time was different.

Actually the problem (well, I think it was the problem) was not too hard to find. Deep in the algorithm that matched query rows with table rows was a HashSet of the row numbers that had not yet been matched – creatively called unmatchedRows. Every time I matched a row, I removed the number of that row from this set.

This algorithm was written for Query tables and was inherited into Ordered Query. However, in Ordered Query I copied that set into a List for one reason or another.

All this works fine until there is a duplicate. As you might expect, in the Ordered Query the order in which the duplicates are processed is important. Unfortunately, the order of the rows in the list that was copied from the set is indeterminate.

Now, maybe you missed the importance of that last paragraph. It described the bug. So let me re-emphasize. In the case of duplicate rows I was depending on the order of elements in a list; but I built the list from a HashSet. HashSets don’t order their elements. So the order of the list was indeterminate.

The fix was simple. I changed the HashSet into an ArrayList. That fixed it. I think…

The problem is that I had no way to reliably create the symptom. I could not write a test that failed because I was using a HashSet instead of an ArrayList. I could not mock out the HashSet because the bug was in the use of the HashSet. I could not run the application 1000 times to statistically see the fault, because the behavior did not change from run to run. The only thing that seemed able to sometimes expose the fault was a recompile! And I wasn’t about to put my system into a recompile-til-fail loop.

Why a recompile? I suppose it has something to do with the values of internal addresses and their effect upon the hash values within the HashSet.

So, here, finally, after 10 years of being a die-hard TDDer, I have encountered a bug that I could not TDD my way out of. To solve it, I had to guess. To solve it, I had to go open-loop. To solve it, I had to assume that I had found the cause, without being sure. As a result, I am not certain that I have actually solved the problem. It may be that a day or a year from now that acceptance test will fail again; and I’ll be back scratching my head and debugging.

(sigh).

"To Test The Untestable Bug." by Polo Ralph Lauren Pas Cher

Thu, 12 Jan 2012 20:01:37 -0600

Thanks for this beautiful website. I have enjoyed reading through a few of the articles.

"To Test The Untestable Bug." by cheap gucci

Sat, 26 Nov 2011 03:45:08 -0600

, often large drops of perspiration extract jordan release dates 2011 . Arms, neck pain is jordan 6 often air jordan 5 so tired already, but jordan 3 air jordan 2011 a fe…

"To Test The Untestable Bug." by christian louboutin

Wed, 09 Nov 2011 09:01:29 -0600

The professional design make you foot more comfortable. Even more tantalizing,this pattern make your legs look as long as you can,it will make you looked more attractive.Moveover,it has reasonable price.If you are a popular woman,do not miss it.

Technical details of Christian Louboutin Velours Scrunch Suede Boots Coffee:

Color: Coffee
Material: Suede
4(100mm) heel
Signature red sole x

Fashion, delicate, luxurious Christian louboutins shoes on sale, one of its series is Christian Louboutin Tall Boots, is urbanism collocation. This Christian louboutins shoes design makes people new and refreshing. Red soles shoes is personality, your charm will be wonderful performance.

"To Test The Untestable Bug." by Dr.Dre Casque

Tue, 08 Nov 2011 19:36:35 -0600

They actual beats through Beats de Dr.Dre Solo HD Graphiteprovide a classy and cozy pattern also as extraordinarily crispy sound reactionbeats by dre Solo Noir. beats by dre Solo pourpretop of the range related to audio is balanced, in conjunction with snug mids and conjointly walloping striped bassbeats by dre Studio Pourpre. Beats By Dre Studio PinkProvided completely are a nice travel case also as a brand new music-telephone-compatible cableBeats By Dre Studio Michael Jackson. the actual Beast Conquer headphones couldn’t be taken devoid of battery powerBeats By Dre Studio Yellow. Beats By Dre Studio All RedPolished African yank style and elegance is quite smudge-prone,beats by dre Pro Blanco and a few songs sound unpleasant. Right dr dre headphones rattles whereas you go. beats by dre Studio MoradosYou’re initial and trendy Huge Surpasses by means of Medical skilled. Dre earphones turn out stable sound, helpful add-ons, and conjointly a look that is not at all imitator.beats by dre LeBron James pro mate relating to fashion-frontward folks that have make the most acquire in order that you’ll spare,beats by dre Studio Blanco they’re fantastic alternative.

"To Test The Untestable Bug." by Tips For Bowling

Tue, 18 Oct 2011 12:25:31 -0500

No, there’s not much competition between puppeteers in general because everybody’s working their own style.

"To Test The Untestable Bug." by shanewatson404@gmail.com

Sat, 01 Oct 2011 01:01:52 -0500

Wow, this is really superb info is visible in this blog Director of Operations Job Description that to amazing Accounting Job Description technology and using the wonderful messages for this website. I was searching the great info and providing the nice info in this blog and read Graphic Designer Job Description this info you Legal Assistant Job Description can never forget this info

"To Test The Untestable Bug." by best sleep aid

Fri, 30 Sep 2011 13:30:16 -0500

When I at first left a comment I clicked the “Notify me when new comments are added” checkbox and now each time a comment is added I get three notification emails with the same comment. Is there any way you can take away people from that service? Thanks a lot!

"To Test The Untestable Bug." by DR OZ african Mango

Tue, 27 Sep 2011 11:45:47 -0500

Corporations are challenging existing business models as they seek ways to speed innovation, focus on their core competencies, and scale to capitalize on opportunities and outpace competitors.

"To Test The Untestable Bug." by Bowtrol

Mon, 08 Aug 2011 15:49:44 -0500

it needs a bokmark so i can come back to it later ,nice stuff

"To Test The Untestable Bug." by adres zoeken op naam

Tue, 02 Aug 2011 18:06:17 -0500

Thanks for this one!

"To Test The Untestable Bug." by Christian louboutin shoes on sale

Fri, 10 Jun 2011 21:23:21 -0500

What a wonderful article! And are doing well on the whole, but their work also has shortcomings.

"To Test The Untestable Bug." by beats by dr dre headphones

Wed, 08 Jun 2011 21:03:06 -0500

I attempted these Beats by dr dre studio out in several genres thinking about which i listen to an eclectic mix Beats by dr dre solo. A all natural suit With Solos, you really feel the music, not the Monster Beats By Dr. Dre Pro Headphones Black.

"To Test The Untestable Bug." by okey oyunu oyna

Thu, 28 Apr 2011 15:31:33 -0500

informative article. Thanks

internette görüntülü olarak okey oyunu oyna, gerçek kisilerle tanis, turnuva heyecanini yasa.

"To Test The Untestable Bug." by dswehfhh

Wed, 09 Mar 2011 13:45:59 -0600

We are the professional shirts manufacturer, shirts supplier, shirts factory, custom shirts.

"To Test The Untestable Bug." by mobile screening equipment

Mon, 28 Feb 2011 23:37:09 -0600

KEESTRACK?manufacturing expert of mobile crusher and mobile screening equipment. Company engaged in the research & development, manufacture, sales of track screening machine,mobile screening,cone crusher,jaw crusher,impact crusher,mobile screening equipment,mobile crushing equipment, till now we are proud to say we are at front of the industry.

"To Test The Untestable Bug." by massager

Mon, 28 Feb 2011 23:35:20 -0600

Your site is amazing.I am very impressed to see this,i want to come back for visiting your site.Keep doing Good as well as you can.

"To Test The Untestable Bug." by mechanical Seal

Mon, 28 Feb 2011 23:30:27 -0600

Thanks for writing this blog post, it was informative, enjoyable, and most importantly – a good length!

"To Test The Untestable Bug." by Silicone Molding

Wed, 29 Dec 2010 21:14:56 -0600

Mold making is the core business of Intertech (Taiwan). With world level technology, Intertech enjoys a very good reputation for making Injection Mold and Plastic Moldsfor their worldwide customers.

"To Test The Untestable Bug." by ???? ??????

Wed, 15 Dec 2010 08:08:15 -0600

Appreciate your unusual and at the same time creative way of writing, this means much!

"To Test The Untestable Bug." by Pandora

Thu, 02 Dec 2010 01:45:46 -0600

with their own execution of SuiteSetUp and SuiteTearDown.

"To Test The Untestable Bug." by moncler

Sat, 27 Nov 2010 01:18:10 -0600

Very few has the knack of writing such beautiful post like you do.. !!moncler mens I have been following this blog and i feel different energy while reading your post. Keep it dear buddy.. !!moncler women

"To Test The Untestable Bug." by Dog bark collar

Fri, 26 Nov 2010 09:17:59 -0600

The HashSet implementation can change in the future, or some one else can change the HashSet implementation and the test start to fail random again.

"To Test The Untestable Bug." by Designer Bags

Sat, 13 Nov 2010 23:41:16 -0600

Awesome topic.Thanks for sharing!

"To Test The Untestable Bug." by supplynflshop

Tue, 31 Aug 2010 22:14:52 -0500

Originally, the word emblem meant “inlaid ornamental work,” that is Wholesale NFL Jerseys, a

symbol of something else – in this case, that of wealth, because of the time and skill required to do it . Badge probably became

synonymous with emblem in the 15th century Cheap NFL Jerseys. Decal, an abbreviated version of

“decalcomania,”a French word that came into use in the early 20th century referred to the English practice of “transfer printing”

invented in the 18th century. This process transferred the ink from a design or drawing to the glass or porcelain when it was fired in the

kiln NFL Jerseys.

"To Test The Untestable Bug." by FLV to ipad converter

Fri, 16 Apr 2010 03:57:18 -0500

you can have a try