Object Mentor Blog: Specs vs. Tests

Specs vs. Tests

Uncle Bob — Thu, 01 Feb 2007 09:32:07 -0600

There’s something to this BDD kool-aid that people have been drinking lately…

As part of the Rails project I’ve been working on for the last few weeks, I’ve been using RSpec. RSpec is a unit testing tool similar in spirit to JUnit or Test/Unit. However RSpec uses an alternative syntax that reads more like a specification than like a test. Let me show you what I mean.

In Java, using JUnit, we might write the following unit test:

public class BowlingGameTest extends TestCase {
  private Game g;

  protected void setUp() throws Exception {
    g = new Game();
  }

  private void rollMany(int n, int pins) {
    for (int i=0; i< n; i++) {
      g.roll(pins);
    }
  }

  public void testGutterGame() throws Exception {
    rollMany(20, 0);
    assertEquals(0, g.score());
    assertTrue(g.isComplete());
  }

  public void testAllOnes() throws Exception {
    rollMany(20,1);
    assertEquals(20, g.score());
    assertTrue(g.isComplete());
  }
}

This is pretty typical for a Java unit test. The setup function builds the Game object, and then the various test functions make sure that it works in each different scenario. In Ruby however, this might be expressed using RSpec as:

require 'rubygems'
require_gem "rspec" 
require 'game'

context "When a gutter game is rolled" do
  setup do
    @g = Game.new
    20.times {@g.roll 0}
  end

  specify "score should be zero" do
    @g.score.should == 0
  end

  specify "game should be complete" do
    @g.complete?.should_be true
  end
end

context "When all ones are rolled" do
  setup do
    @g = Game.new
    20.times{@g.roll 1}
  end

  specify "score should be 20" do
    @g.score.should == 20
  end

  specify "game should be complete" do
    @g.complete?.should_be true
  end
end

At first blush the difference seems small. Indeed, the RSpec code might seem too verbose and fine-grained. At least that was my first impression when I first saw RSpec. However, having used it now for several months I have a different reaction.

First, let’s looks a the semantic differences. In JUnit you have TestCase derivatives, and test functions. Each TestCase derivative has a setUp and tearDown function, and a suite of test functions. In RSpec you have what appears to be an extra layer. You have the test script, which is composed of context blocks. The contexts have setup, teardown, and specify blocks.

At first you might think that the RSpec context block coresponds to the Java TestCase derivative since they are semantically equivalent. However Java throws something of a curve at us by only allowing one public class per file. So from an organizational point of view there is a stronger equivalence between the TestCase derivative and the whole RSpec test script.

This might seem petty. After all, I can write Java code that is semantically equivalent to the RSpec code simply by creating two TestCase derivatives in two different files. But separating those two test cases into two different files makes a big difference to me. It breaks apart things that otherwise want to stay together.

Now it’s true that I could keep the TestCase derivatives in the same file by making them package scope, and manually put them into a public TestSuite class. But who wants to do that? After all, my IDE is nice enough to find and execute all the public TestCase derivatives, which completely eliminates the need for me to build suites—at least at first.

Note: The JDave tool provides BDD syntax for Java.

Again, this might seem petty; and if that were the only benefit to the RSpec syntax I would agree. But it’s not the only benefit.

Strange though it may seem, the next benefit is the strings that describe the context and specify blocks. At first I thought these strings were just noise, like the strings in the JUnit assert functions. I seldom, if ever, use the JUnit assert strings, so why would I use the context and specify strings? But over the last few weeks I have come to find that, unlike the JUnit assert strings, the RSpec strings put a subtle force on me to create better test designs.

Stable State: An Emergent Rule.

When a spec fails, the message that gets printed is the concatenation of the context string and the specify string. For example: 'When a gutter game is rolled game should be complete' FAILED. If you word the context and specify strings properly, these error message make nice sentences. Since, in TDD, we almost always start out with our tests failing, I see these error message a lot. So there is a pressure on me to word them well.

But by wording them well, I am constrained to obey a rule that JUnit never put pressure on me to obey. Indeed, I didn’t know it was a rule until I started using RSpec. I call this rule Stable State, it is:

Tests don’t change the state.

In other words, the functions that make assertions about the state of the system, do not also change the state of the system. The state of the system is set up once in the setUp function, and then only interrogated by the test functions.

If you look carefully at the specification of the Bowling Game you will see that the state of the Game is changed only by the setup block within the context blocks. The specify blocks simply interrogate and verify state. This is in stark contrast to the JUnit tests in which the test methods both change and verify the state of the Game.

If you don’t follow this rule it is hard to get the strings on the context and specify blocks to create error messages that read well. On the other hand, if you make sure that the specify blocks don’t change the state, then you can find simple sentences that describe each context and specify block. And so the subtle pressure of the strings has a significant impact on the structure of the tests.

I can’t claim to have discovered the pressure of these strings. Indeed, Dan North’s original article on the topic is captivating. However, I felt the pressure and came to the same conclusion he did, well before I read his article; simply by using a tool inspired by his work.

The benefit of Stable State is that for each set of assertions there is one, and only one place where the state of the system is changed. Moreover the three level structure provides natural places for groups of state, states, and asserts.

The demise of the One Assert rule.

There have been other rules like this before. One that circulated a few years back was:

One assert per test.

I never bought into this rule, and I still don’t. It seems arbitrary and inefficient. Why should I put each assert statement into it’s own test method when I can just as well put the assert statement into a single test method.

In other words, why prefer this:

  public void testGutterGameScoreIsZero() throws Exception {
    rollMany(20, 0);
    assertEquals(0, g.score());
  }

  public void testGutterGameIsComplete() throws Exception {
    rollMany(20, 0);
    assertTrue(g.isComplete());
  }

over this:

  public void testGutterGame() throws Exception {
    rollMany(20, 0);
    assertEquals(0, g.score());
    assertTrue(g.isComplete());
  }

I think the authors of the One Assert rule were trying to achieve the benefits of Stable State, but missed the mark. It’s as though they could smell the rule out there, but couldn’t quite pinpoint it.

The State Machine metaphor

When you follow the Stable State rule your specifications (tests) become a description of a Finite State Machine. Each context block describes how to drive the SUT to a given state, and then the specify blocks describe the attributes of that state.

Dan North calls this the Given-When-Then metaphor. Consider the following triplet:

Given a Bowling Game: When 20 gutter balls are rolled, Then the score should be zero and the game should be complete.

This triplet corresponds nicely to a row in a state transition table. Consider, for example, the subway turnstile state machine:

Current State	Event	New State
Locked	coin	Unlocked
Unlocked	pass	Locked
Locked	pass	Alarm
Unlocked	coin	Unlocked

We can read this as follows:

GIVEN we are in the Locked state, WHEN we get a coin event, THEN we should be in the Unlocked state. — GIVEN we are in the Unlocked state, WHEN we get a pass event, THEN we should be in the Locked state. — etc.

Describing a system as a finite state machine has certain benefits.

We can enumerate the states and the events, and then make sure that every combination of state and event is handled properly.
We can formalize the behavior of the system into a well known tabular format that can be read and interpreted by machines.
- I am, of course, thinking about FitNesse
There are well known mechanisms for implementing finite state machines.

The point is that organizing the system description in terms of a finite state machine can have a profound impact on the system design and implementation.

The Butterfly Effect.

I find it remarkable that two dumb annoying little strings put a subtle pressure on me to adjust the style of my tests. That change in style eventually caused me to see the design and implementation of the system I was writing in a very new and interesting light.

"Specs vs. Tests" by Learn Spanish Language FREE

Mon, 30 Jan 2012 03:05:14 -0600

thanks for ur sharing, I like your blog, content is very rich. http://www.youtube.com/watch?v=AYNtk_LMrho

"Specs vs. Tests" by Harris

Wed, 28 Dec 2011 22:44:23 -0600

I really like the Stable State idea. a thought for the day

"Specs vs. Tests" by Wound Bandages

Fri, 16 Dec 2011 04:46:15 -0600

At first I would congratulate you on writing such a brilliant piece of write-up. You have got some exceptional writing skills that have made your site worth reading.

"Specs vs. Tests" by LiMelia

Sun, 27 Nov 2011 04:34:49 -0600

thanks for ur sharing, I like your blog, content is very rich, allow me to leave a message well, wish you are lucky!!!!! http://www.junyuetrade.com/

"Specs vs. Tests" by lewis

Thu, 10 Nov 2011 19:53:19 -0600

Nice pearl at http://www.cnwpearl.com http://www.cnwpearl.com/freshwater-pearl-necklace/c1/index.html http://www.cnwpearl.com/freshwater-pearl-bracelets/c7/index.html

"Specs vs. Tests" by renlewei

Tue, 20 Sep 2011 04:37:41 -0500

Gucci Top Handles Gucci Shoulder Bags Gucci Clutches http://www.saleguccinewbags.com/gucci-boston-bags-c-58.html">Gucci Boston Bags Gucci Messenger Bags authentic discount gucci bags

"Specs vs. Tests" by canada goose coat

Thu, 15 Sep 2011 21:26:46 -0500

Canada Goose Outlet is Marmot 8000M Parka. The Marmot 8000M Parka is really a waterproof, breathable jacket with 800 fill canada goose jacket feathers. It truly is design and light colored shell is produced for trendy, but uncomplicated, protection from cold temperatures. Reinforced shoulders, elbows and adjustable waist and hem make the Marmot a perfect alternate for skiing and other outdoor sports that want fairly a bit of arm motion. The 8000M Parka weighs three lbs., comes in bonfire and black colours and might be stuffed and stored like a sleeping bag to your convenience.This is one of well-know and prime down jacket brands.Hope our friends like its!Like canada goose womens and Canada Goose Expedition Parka.There are wholesale canada goose.

"Specs vs. Tests" by Diablo3

Wed, 14 Sep 2011 12:19:36 -0500

Blog posts about wedding and bridal are always rare to find , at least with great quality,you qualify for a great blog post writer title,kep the great job happening

"Specs vs. Tests" by Crescent Processing Company

Fri, 12 Aug 2011 09:18:35 -0500

Crescent Processing Company You deserve the best and I know this will just add to your very proud accomplishments in your already beautiful and deserving blessed life. I wish you all the best and again. Thanks a lot.. Crescent Processing Company

"Specs vs. Tests" by kohls printable coupons

Sat, 11 Jun 2011 19:37:26 -0500

Thanks for the guide, it’s very helpful.

"Specs vs. Tests" by funny pictures

Tue, 03 May 2011 03:41:31 -0500

Yes these are very important indeed

"Specs vs. Tests" by okey oyunu oyna

Fri, 22 Apr 2011 13:32:20 -0500

thanks for this post :)

Dünyan?n en büyük online okey oyunu bu sitede sizleri bekliyor. Gerçek ki?ilerle sohbet ederek Okey Oyunu Oyna ve internette online oyun oynaman?n zevkini ç?kar

"Specs vs. Tests" by Nike Sneakers Outlet

Wed, 13 Apr 2011 21:38:52 -0500

Thank you very good and a healthy writing. I’ll definitely keep track of posts and the occasional visit. Looking forward to reading your next publish.Nike Sneakers Outlet

"Specs vs. Tests" by True Religion Jeans Outlet Online

Tue, 12 Apr 2011 22:48:16 -0500

it is a useful and wonderful website.thanks for your information.

"Specs vs. Tests" by Hotel Bucuresti

Fri, 01 Apr 2011 09:34:41 -0500

Normally, the tests must reveal the same values as the specs said. However, the reality seems to be very difficult to understand because sometimes the differences are very big.

"Specs vs. Tests" by SEO Firm India

Fri, 18 Mar 2011 12:31:48 -0500

enjoyed reading it. I need to read more on this topic…I admiring time and effort you put in your blog, because it is obviously one great place where I can find lot of useful info..

"Specs vs. Tests" by cable ties

Thu, 03 Mar 2011 05:20:49 -0600

Great discussion in here.

"Specs vs. Tests" by cable ties

Thu, 03 Mar 2011 05:20:33 -0600

Great information in here~

"Specs vs. Tests" by Tenant Screening

Fri, 18 Feb 2011 10:56:58 -0600

Test-driven development is the most profound and auspicious thing to happen to the software industry since I’ve been a programmer. I think it’s even more important than OO.

"Specs vs. Tests" by Criminal Records

Thu, 17 Feb 2011 17:34:16 -0600

It may very well be that the “tests” for a particular class, then, get spread out across many fixtures, instead of feeling like you have to place all tests for a given class in the same TestCase derivative. The fixture names corresponding to system states, not classes. At least that’s how I’ve read it.

"Specs vs. Tests" by Criminal Check

Thu, 27 Jan 2011 16:59:07 -0600

In other words, the functions that make assertions about the state of the system, do not also change the state of the system.

"Specs vs. Tests" by Criminal Check

Fri, 21 Jan 2011 10:56:13 -0600

So from an organizational point of view there is a stronger equivalence between the TestCase derivative and the whole RSpec test script.

"Specs vs. Tests" by hermes replica watches

Tue, 04 Jan 2011 02:45:41 -0600

no longer visits the fake hermes bracelet the communal spaces.”I’m sure there are some replica hermes some nice people here, but they have hermes neck scarf have 13 or

"Specs vs. Tests" by puma

Mon, 20 Dec 2010 01:36:52 -0600

If you mean to find great shoes for your children puma speed trainers also provides a mixture of finicky and affordable shoes for them. There are a lot of choices, it is up ring call,Ugg Boots, after by people that indigence an incredible quantity of column. This will make the customers happier. If you are often tangled in Singapore womens puma future cat shoes sale at Sainte Marie that could enhance operational efficiency, range visibility and turnaround time,” said Desmond Chan, managing boss, South Asia, Menlo Worldwide Logistics. “Our multi-client facility in Boon Lay Way provides puma trainers with different flag. puma uk’s youngest targets are toddlers. The puma for sale shoes are incredibly affordable, yet they still hold the grace. Wearing comfortable shoes will help children exploit better.

"Specs vs. Tests" by iPad video converter for Mac

Fri, 03 Dec 2010 06:40:10 -0600

When I come to here, I think I am in the right place. the web gives me a lot of infomation, it is very informative. I think lots of people can learn much here. I will come to here again. Thanks.