Object Mentor Blog: Mocking Mocking and Testing Outcomes.

Mocking Mocking and Testing Outcomes.

Uncle Bob — Sat, 23 Jan 2010 11:32:00 -0600

The number of mocking frameworks has proliferated in recent years. This pleases me because it is a symptom that testing in general, and TDD in particular, have become prevalent enough to support a rich panoply of third-party products.

On the other hand, all frameworks carry a disease with them that I call The Mount Everest Syndrome: “I use it because it’s there.” The more mocking frameworks that appear, the more I see them enthusiastically used. Yet the prolific use of mocking frameworks is a rather serious design smell…

Lately I have seen several books and articles that present TDD through the lens of a mocking framework. If you were a newbie to TDD, these writings might give you the idea that TDD was defined by the use of mocking tools, rather than by the disciplines of TDD.

So when should use use a mocking framework? The answer is the same for any other framework. You use a framework only when that framework will give you a significant advantage.

Why so austere? Why shouldn’t you use frameworks “just because they are there”? Because frameworks always come with a cost. They must be learned by the author, and by all the readers. They become part of the configuration and have to be maintained. They must be tracked from version to version. But perhaps the most significant reason is that once you have a hammer, everything starts to look like a nail. The framework will put you into a constraining mindset that prevents you from seeing other, better solutions.

Consider, for example, this lovely bit of code that I’ve been reviewing recently. It uses the Moq framework to initialize a test double:


var vehicleMock = Mocks.Create<IClientVehicle>()
 .WithPersistentKey()
 .WithLogicalKey().WithLogicalName()
 .WithRentalSessionManager(rsm =>
    {
      var rs = Mocks.Create<IRentalSession>();
      rsm.Setup(o => o.GetCurrentSession()).Returns(rs.Object);
      rsm.Setup(o =>
       o.GetLogicalKeyOfSessionMember(It.IsAny<string>(),
        It.IsAny<int>())).Returns("Rental");
    })
 .AddVehicleMember<IRoadFactory>()
 .AddVehicleMember<IRoadItemFactory>(rf => rf.Setup(t => 
    t.CreateItems(It.IsAny<IRoad>())).Returns(pac))
 .AddVehicleMember<ILegacyCorporateRental>()
 .AddVehicleMember<IRentalStation>(
    m => m.Setup(k => k.Facility.FacilityID).Returns(0))
 .AddVehicleMember<IRoadManager>(m=>
    m.Setup(k=>k.GetRoundedBalanceDue(25,It.IsAny<IRoad>())).Returns(25));

Some of you might think I’m setting up a straw-man. I’m not. I realize that bad code can be written in any language or framework, and that you can’t blame the language or framework for bad code.

The point I am making is that code like this was the way that all unit tests in this application were written. The team was new to TDD, and they got hold of a tool, and perhaps read a book or article, and decided that TDD was done by using a mocking tool. This team is not the first team I’ve seen who have fallen into this trap. In fact, I think that the TDD industry as a whole has fallen into this trap to one degree or another.

Now don’t get me wrong. I like mocking tools. I use them in Ruby, Java, and .Net. I think they provide a convenient way to make test-doubles in situations where more direct means are difficult.

For example, I recently wrote the following unit test in FitNesse using the Mockito framework.


  @Before
  public void setUp() {
    manager = mock(GSSManager.class);
    properties = new Properties();
  }

  @Test
  public void credentialsShouldBeNullIfNoServiceName() throws Exception {
    NegotiateAuthenticator authenticator = 
      new NegotiateAuthenticator(manager, properties);
    assertNull(authenticator.getServerCredentials());
    verify(manager, never()).createName(
      anyString(), (Oid) anyObject(), (Oid) anyObject());
  }

The first line in the setUp function is lovely. It’s kind of hard to get prettier than that. Anybody reading it understands that manager will be a mock of the GSSManager class.

It’s not too hard to understand the test itself. Apparently we are happy to have the manager be a dummy object with the constraint that createName is never called by NegotiateAuthenticator. The anyString() and anyObject() calls are pretty self explanatory.

On the other hand, I wish I could have said this:

assertTrue(manager.createNameWasNotCalled());

That statement does not require my poor readers to understand anything about Mockito. Of course it does require me to hand-roll a manager mock. Would that be hard? Let’s try.

First I need to create a dummy.


  private class MockGSSManager extends GSSManager {
    public Oid[] getMechs() {
      return new Oid[0];
    }

    public Oid[] getNamesForMech(Oid oid) throws GSSException {
      return new Oid[0];
    }

    public Oid[] getMechsForName(Oid oid) {
      return new Oid[0];
    }

    public GSSName createName(String s, Oid oid) throws GSSException {
      return null;
    }

    public GSSName createName(byte[] bytes, Oid oid) throws GSSException {
      return null;
    }

    public GSSName createName(String s, Oid oid, Oid oid1) throws GSSException {
      return null;
    }

    public GSSName createName(byte[] bytes, Oid oid, Oid oid1) throws GSSException {
      return null;
    }

    public GSSCredential createCredential(int i) throws GSSException {
      return null;
    }

    public GSSCredential createCredential(GSSName gssName, int i, Oid oid, int i1) throws GSSException {
      return null;
    }

    public GSSCredential createCredential(GSSName gssName, int i, Oid[] oids, int i1) throws GSSException {
      return null;
    }

    public GSSContext createContext(GSSName gssName, Oid oid, GSSCredential gssCredential, int i) throws GSSException {
      return null;
    }

    public GSSContext createContext(GSSCredential gssCredential) throws GSSException {
      return null;
    }

    public GSSContext createContext(byte[] bytes) throws GSSException {
      return null;
    }

    public void addProviderAtFront(Provider provider, Oid oid) throws GSSException {
    }

    public void addProviderAtEnd(Provider provider, Oid oid) throws GSSException {
    }
  }

“Oh, ick!” you say. Yes, I agree it’s a lot of code. On the other hand, it took me just a single keystroke on my IDE to generate all those dummy methods. (In IntelliJ it was simply command-I to implement all unimplemented methods.) So it wasn’t particularly hard. And, of course, I can put this code somewhere where nobody had to look at it unless they want to. It has the advantage that anybody who knows Java can understand it, and can look right at the methods to see what they are returning. No “special” knowledge of the mocking framework is necessary.

Next, let’s’ make a test double that does precisely what this test needs.


  private class GSSManagerSpy extends MockGSSManager {
    public boolean createNameWasCalled;

    public GSSName createName(String s, Oid oid) throws GSSException {
      createNameWasCalled = true;
      return null;
    }
  }

Well, that just wasn’t that hard. It’s really easy to understand too. Now, let’s rewrite the test.


  @Test
  public void credentialsShouldBeNullIfNoServiceNameWithHandRolledMocks() throws Exception {
    NegotiateAuthenticator authenticator = new NegotiateAuthenticator(managerSpy, properties);
    assertNull(authenticator.getServerCredentials());
    assertFalse(managerSpy.createNameWasCalled);
  }

Well, that test is just a load easier to read than verify(manager, never()).createName(anyString(), (Oid) anyObject(), (Oid) anyObject());.

“But Uncle Bob!” I hear you say. “That scenario is too simple. What if there were lots of dependencies and things…” I’m glad you asked that question, because the very next test is just such a situation.


  @Test
  public void credentialsShouldBeNonNullIfServiceNamePresent() throws Exception {
    properties.setProperty("NegotiateAuthenticator.serviceName", "service");
    properties.setProperty("NegotiateAuthenticator.serviceNameType", "1.1");
    properties.setProperty("NegotiateAuthenticator.mechanism", "1.2");
    GSSName gssName = mock(GSSName.class);
    GSSCredential gssCredential = mock(GSSCredential.class);
    when(manager.createName(anyString(), (Oid) anyObject(), (Oid) anyObject())).thenReturn(gssName);
    when(manager.createCredential((GSSName) anyObject(), anyInt(), (Oid) anyObject(), anyInt())).thenReturn(gssCredential);
    NegotiateAuthenticator authenticator = new NegotiateAuthenticator(manager, properties);
    Oid serviceNameType = authenticator.getServiceNameType();
    Oid mechanism = authenticator.getMechanism();
    verify(manager).createName("service", serviceNameType, mechanism);
    assertEquals("1.1", serviceNameType.toString());
    assertEquals("1.2", mechanism.toString());
    verify(manager).createCredential(gssName, GSSCredential.INDEFINITE_LIFETIME, mechanism, GSSCredential.ACCEPT_ONLY);
    assertEquals(gssCredential, authenticator.getServerCredentials());
  }

Now I’ve got three test doubles that interact with each other; and I am verifying that the code under test is manipulating them all correctly. I could create hand-rolled test doubles for this; but the wiring between them would be scattered in the various test-double derivatives. I’d also have to write a significant number of accessors to get the values of the arguments to createName and createCredential. In short, the hand-rolled test-double code would be harder to understand than the Mockito code. The Mockito code puts the whole story in one simple test method rather than scattering it hither and yon in a plethora of little derivatives.

What’s more, since it’s clear that I should use a mocking framework for this test, I think I should be consistent and use if for all the tests in this file. So the hand-rolled MockGSSManager and ManagerSpy are history.

“But Uncle Bob, aren’t we always going to have dependencies like that? So aren’t we always going to have to use a mocking framework?”

That, my dear reader, is the real point of this blog. The answer to that salient questions is a profound: “No!“

Why did I have to use Mockito for these tests? Because the number of objects in play was large. The module under test (NegotiateAuthenticator) used GSSName, GSSCredential, and GSSManager. In other words the coupling between the module under test and the test itself was high. (I see lightbulbs above some of your heads.) That’s right, boys and girls, we don’t want coupling to be high!

It is the high coupling between modules and tests that creates the need for a mocking framework. This high coupling is also the cause of the dreaded “Fragile Test” problem. How many tests break when you change a module? If the number is high, then the coupling between your modules and tests in high. Therefore, I conclude that those systems that make prolific use of mocking frameworks are likely to suffer from fragile tests.

Of the 277 unit test files in FitNesse, only 11 use Mockito. The reason for small number is two-fold. First, we test outcomes more often than we test mechanisms. That means we test how a small group of classes behaves, rather than testing the dance of method calls between those classes. The second reason is that our test doubles have no middle class. They are either very simple stubs and spies or they are moderately complex fakes.

Testing outcomes is a traditional decoupling technique. The test doesn’t care how the end result is calculated, so long as the end result is correct. There may be a dance of several method calls between a few different objects; but the test is oblivious since it only checks the answer. Therefore the tests are not strongly coupled to the solution and are not fragile.

Keeping middle-class test doubles (i.e. Mocks) to a minimum is another way of decoupling. Mocks, by their very nature, are coupled to mechanisms instead of outcomes. Mocks, or the setup code that builds them, have deep knowledge of the inner workings of several different classes. That knowledge is the very definition of high-coupling.

What is a “moderately complex fake” and why does it help to reduce coupling? One example within FitNesse is MockSocket. (The name of this class is historical. Nowadays it should be called FakeSocket.) This class derives from Socket and implements all its methods either to remember what was sent to the socket, or to allow a user to read some canned data. This is a “fake” because it simulates the behavior of a socket. It is not a mock because it has no coupling to any mechanisms. You don’t ask it whether it succeeded or failed, you ask it to send or recieve a string. This allows our unit tests to test outcomes rather than mechanisms.

The moral of this story is that the point at which you start to really need a mocking framework is the very point at which the coupling between your tests and code is getting too high. There are times when you can’t avoid this coupling, and those are the times when mocking frameworks really pay off. However, you should strive to keep the coupling between your code and tests low enough that you don’t need to use the mocking framework very often.

You do this by testing outcomes instead of mechanisms.

"Mocking Mocking and Testing Outcomes." by Android News

Sat, 21 Jan 2012 01:47:46 -0600

Thank you.

I recentlly had an eye opener at work, where I was shown how our extensive use of Rhino Mock’s garbled the readability of our testing.

After rewriting the test without Rhino mock’s it became apparent to me, how our learned behaviour to use mocks as much as we did affected the quality of our testing code.

I’m now looking at mocking suspiciously.

"Mocking Mocking and Testing Outcomes." by free cell phone spy

Wed, 18 Jan 2012 02:25:21 -0600

This is a very informative article.I was looking for these things and here I found it. I am doing a project and this information is very useful me. I always try to find information and hopefully I found more about what I am locking for..!

"Mocking Mocking and Testing Outcomes." by iPhone contacts backup

Mon, 09 Jan 2012 00:14:35 -0600

How many people understand this? it is a good choice to make a copy of it and save it on computer. We can retrieve them when necessary.

"Mocking Mocking and Testing Outcomes." by nikeheelsdunk

Mon, 26 Dec 2011 02:48:43 -0600

A first aid kit is a must for small and accidental injuries. Of course it’s unavoidable to scrape your arm on rough bark or falling off the tree stand, at least there’s an available remedy in that box. Always make sure to dispose of the Nike heels for women material properly.

Other equipment such as rifles or bows must be kept unloaded. Most States will commission a guide for hunters to carry the ammunition and probably help the hunter carry some of the necessary equipment. It’s not like having a Jordan Heels For Women caddy carrying the bag, but he is there to ensure the safety of the hunter as well as the forest.

Never drink anything that may compromise or deteriorate your physical or mental faculties.

"Mocking Mocking and Testing Outcomes." by nikeheelsdunk

Mon, 26 Dec 2011 02:48:14 -0600

"Mocking Mocking and Testing Outcomes." by hAllohi

Mon, 19 Dec 2011 17:17:45 -0600

The best social network…..

"Mocking Mocking and Testing Outcomes." by tv show

Sat, 26 Nov 2011 09:48:17 -0600

very interesting article. thanks alot for sharing and keep up the good work!

best r, Danielle

"Mocking Mocking and Testing Outcomes." by Haseeb

Wed, 09 Nov 2011 12:42:21 -0600

object mentor is very nice and there are number of frameworks so it is very good for me and for my friends.

aircraft for sale in Ireland

"Mocking Mocking and Testing Outcomes." by Jools

Tue, 08 Nov 2011 10:47:53 -0600

:) I understand that Moq is supposed to be simple to use – but after 2 hours I am dropping this thing – something goes wrong all the time :/

anyway, tx for sharing Jools

"Mocking Mocking and Testing Outcomes." by Wiphone

Sun, 06 Nov 2011 05:03:45 -0600

The point I am making is that code like this was the way that all unit tests in this application were written. The team was new to TDD, and they got hold of a tool, and perhaps read a book or article, and decided that TDD was done by using a mocking tool. This team is not the first team I’ve seen who have fallen into this trap. In fact, I think that the TDD industry as a whole has fallen into this trap to one degree or another.

"Mocking Mocking and Testing Outcomes." by Free Download Website

Sun, 06 Nov 2011 04:53:09 -0600

Such a wonderful post. Thanks for sharing this blog post.

Status ngau hung of Tran Dang Khoa | Free Download | Free Download Website | Unlock dien thoai nhat | dien thoai nhat

"Mocking Mocking and Testing Outcomes." by Download Android Apps

Wed, 02 Nov 2011 20:47:54 -0500

Interesting article and one which should be more widely known about in my view. Your level of detail is good and the clarity of writing is excellent. I have bookmarked it for you so that others will be able to see what you have to say

"Mocking Mocking and Testing Outcomes." by Andy Yousuf

Mon, 31 Oct 2011 09:30:42 -0500

Learning a framework to the core is definitely a hard task! loves writing articles

"Mocking Mocking and Testing Outcomes." by xsw123

Sun, 30 Oct 2011 04:31:09 -0500

??? ????1856??????????? ????????1835??????????? 1924??” ??? ????”??????????????????????????? ??????????? ??? ? ( Burberry) ??? ??????1856????????? ??? ???????????? ??? ??????????????????? ???????????? ? ??????????? ??? ???.
?? ?? ??????????????? ?? ?????????? ?? ?1968??????? ?? ???????????? ????????? ?? ???????? ?? ??????? ?? ??????? ?? ???????? ?????? ?????5 ?????????TOMS ??????????? ????????????????????????? ????????????Supra ????????? ?????????????? ??????

"Mocking Mocking and Testing Outcomes." by wallyhandbags

Thu, 27 Oct 2011 13:30:27 -0500

Fashion handbags to http://www.wallyhandbags.com ,free shipping any order ,ten of the speecial offer everyday!

"Mocking Mocking and Testing Outcomes." by [url=http://www.belstaff-outlet.de/]belstaff[/url]

Wed, 26 Oct 2011 22:14:30 -0500

If you are a woman is different, Belstaff jacket, your best choice,stylish and comfortable.

"Mocking Mocking and Testing Outcomes." by [url=http://www.belstaff-outlet.de/]belstaff[/url]

Wed, 26 Oct 2011 22:12:55 -0500

If you are a woman is different, Belstaff jacket, your best choice,stylish and comfortable.

"Mocking Mocking and Testing Outcomes." by Harry

Fri, 21 Oct 2011 08:55:37 -0500

Great post Uncle Bob! I’d estimate about 99.9% of it went straight over my head, but I’m sure it was useful to someone!

"Mocking Mocking and Testing Outcomes." by nice cheap

Thu, 20 Oct 2011 06:41:26 -0500

Thanks for the code written by Moderator write. Useful to me.

"Mocking Mocking and Testing Outcomes." by Edo

Tue, 18 Oct 2011 22:27:51 -0500

I agree with most of the things that Uncle Bob says, but creating a test double manually for the sake of readability is imho overkill. It may also confuse the reader, who will have to look at the test double class to see that createNameWasNotCalled() is not a member of the GSSManager class.

"Mocking Mocking and Testing Outcomes." by euro air

Mon, 17 Oct 2011 10:07:21 -0500

unique content is always best

"Mocking Mocking and Testing Outcomes." by Buy lasix 40mg

Mon, 17 Oct 2011 06:50:30 -0500

I hope most people would agree that record/playback has not shown a lot of promise for long term automation strategies. On the other hand writing good maintainable loosely coupled GUI automation code works great

"Mocking Mocking and Testing Outcomes." by Air Socks

Fri, 14 Oct 2011 06:12:29 -0500

what is a mocking framework? i’m guessing it is different to a mocking bird, is it for the bird to stand on?

"Mocking Mocking and Testing Outcomes." by dsi ????

Wed, 12 Oct 2011 02:55:15 -0500

Every time I am finding this blog.Great post. Please write more and more about this. I had never see a blog better than this one.Thank you for your post. ???????????????????

"Mocking Mocking and Testing Outcomes." by iraq29

Tue, 11 Oct 2011 06:43:56 -0500

Parkour and Freerunning are different but not entirely. Parkour was disciplined anterior to Freerunning by David Belle. It consists of hurdles and bounds. The big philosophical system