Object Mentor Blog: Category Clean Code

20% more bugs? Or 20% less features?

Uncle Bob — Tue, 06 Apr 2010 22:03:23 -0500

People often make the argument that time to market is more important that quality. I’m not sure just what they mean by that. Do they mean that it’s ok if 20% of the features don’t work so long as they deliver quickly? If so, that’s just stupid. Why not develop 20% fewer features, and develop them well. It seems to me that choosing which 20% you are not going to develop and then choosing to develop the other 80% to a high standard of quality is a better management decision than telling the developers to work sloppily.

Software on the Cheap

Uncle Bob — Mon, 01 Feb 2010 16:17:00 -0600

When it comes to software, you get what you pay for.

Have you ever stopped to wonder how much a line of code costs? It ought to be easy to figure out.

In the last 14 months, I have written about 20KSLOC in FitNesse. Of course that was part time. My real job is running Object Mentor, consulting, teaching, mentoring, writing, and a whole load of other things. Programming takes up perhaps 15% of my time.

On the other hand most programmers have lots of other things to do. They go to meetings, and then they go to more meetings. When they are done with those meetings, they go to meetings. And then there are the meetings to go to. Oh yeah, and then there’s all the fiddling around with time accounting tools, and horrific source code control systems that perform like a salamander crawling through frozen mud.

So, maybe 15% isn’t such a bad ratio.

The loaded rate (Salary plus everything else) for a typical programmer is on the order of $200K. (I know that sounds like a lot, but you can look it up.) So $200K / (20KSLOC / 14mo * 12mo) = $11.66/SLOC.

Let’s look at one of those lines: StringBuffer nameBuffer = new StringBuffer(); Does that look like $11.66 to you? Would you pay that much for it? Well, don’t answer yet, because for each StringBuffer line you buy, you get import java.lang.StringBuffer; absolutely free!

Some factories pay their employees a “piece rate”. Would you accept $11.66 per line of code instead of a salary? Of course it couldn’t just be any old line of code. It’d have to be tested!

Hey, I bet all programmers would do TDD if we paid them a piece rate!

Down to business.

The point of that silly analysis was to demonstrate that software is expensive. Even the dumbest little app will likely require more than 1,000 lines of code; and that means it could cost $12K to write!

Imagine that you aren’t a programmer, but you have a clever idea for a new website that’ll make you a zillion dollars. You’ve storyboarded it all out. You’ve worked out all the details. Now all you need is some high-school kid to zip out the code for you. Right? Hell, you could pay him minimum wage! The little twerp would be happy to get it!

That tragic comedy is altogether too common. Too many people have borrowed money against their father’s retirement account to fund a terrible implementation of a good idea. Appalled at how much the reputable firms charge per hour ($100 or more) they go looking for a cheap solution.

“After all, this software is simple.” Or so the reasoning goes. “It’s not like we’re trying to send a rocket to the moon or anything. And, besides, those expensive guys were just out to cheat us. Software just isn’t that hard to write.” Uh huh.

So the poor schmuck finds some freshman in college, or maybe a bored housewife who read a book on HTML last year and created a cute website to show off her kittens. Have these programmers heard about TDD? Have they heard about Design Patterns? Principles? How about source code control?

Clearly they haven’t. They’re going to sling a bunch of horrific code together, without any tests, versioning, or control. The project will start well, with exciting initial results. But then it will slowly grind to a halt, while the cash continues out the door unabated.

In the end the website isn’t going to get built (and the poor schmuck’s father won’t be retiring as soon as he thought). It will be a disaster that will either be terminated, or will require double or triple the investment to get right.

The Bottom Line.

The bottom line is that, when it comes to software, you get what you pay for. If you want good software done well, then you are going to pay for it, and it will probably cost you $12/line or more. And, believe me, that’s the cheapest way to get your software done.

If you go out hunting for the cheap solution, then you’re going to end up paying more, and losing time. Software is one of those things that costs a fortune to write well, and double that to write poorly. If you go for cheap, you’re going to pay double; and maybe even triple.

Mocking Mocking and Testing Outcomes.

Uncle Bob — Sat, 23 Jan 2010 11:32:00 -0600

The number of mocking frameworks has proliferated in recent years. This pleases me because it is a symptom that testing in general, and TDD in particular, have become prevalent enough to support a rich panoply of third-party products.

On the other hand, all frameworks carry a disease with them that I call The Mount Everest Syndrome: “I use it because it’s there.” The more mocking frameworks that appear, the more I see them enthusiastically used. Yet the prolific use of mocking frameworks is a rather serious design smell…

Lately I have seen several books and articles that present TDD through the lens of a mocking framework. If you were a newbie to TDD, these writings might give you the idea that TDD was defined by the use of mocking tools, rather than by the disciplines of TDD.

So when should use use a mocking framework? The answer is the same for any other framework. You use a framework only when that framework will give you a significant advantage.

Why so austere? Why shouldn’t you use frameworks “just because they are there”? Because frameworks always come with a cost. They must be learned by the author, and by all the readers. They become part of the configuration and have to be maintained. They must be tracked from version to version. But perhaps the most significant reason is that once you have a hammer, everything starts to look like a nail. The framework will put you into a constraining mindset that prevents you from seeing other, better solutions.

Consider, for example, this lovely bit of code that I’ve been reviewing recently. It uses the Moq framework to initialize a test double:


var vehicleMock = Mocks.Create<IClientVehicle>()
 .WithPersistentKey()
 .WithLogicalKey().WithLogicalName()
 .WithRentalSessionManager(rsm =>
    {
      var rs = Mocks.Create<IRentalSession>();
      rsm.Setup(o => o.GetCurrentSession()).Returns(rs.Object);
      rsm.Setup(o =>
       o.GetLogicalKeyOfSessionMember(It.IsAny<string>(),
        It.IsAny<int>())).Returns("Rental");
    })
 .AddVehicleMember<IRoadFactory>()
 .AddVehicleMember<IRoadItemFactory>(rf => rf.Setup(t => 
    t.CreateItems(It.IsAny<IRoad>())).Returns(pac))
 .AddVehicleMember<ILegacyCorporateRental>()
 .AddVehicleMember<IRentalStation>(
    m => m.Setup(k => k.Facility.FacilityID).Returns(0))
 .AddVehicleMember<IRoadManager>(m=>
    m.Setup(k=>k.GetRoundedBalanceDue(25,It.IsAny<IRoad>())).Returns(25));

Some of you might think I’m setting up a straw-man. I’m not. I realize that bad code can be written in any language or framework, and that you can’t blame the language or framework for bad code.

The point I am making is that code like this was the way that all unit tests in this application were written. The team was new to TDD, and they got hold of a tool, and perhaps read a book or article, and decided that TDD was done by using a mocking tool. This team is not the first team I’ve seen who have fallen into this trap. In fact, I think that the TDD industry as a whole has fallen into this trap to one degree or another.

Now don’t get me wrong. I like mocking tools. I use them in Ruby, Java, and .Net. I think they provide a convenient way to make test-doubles in situations where more direct means are difficult.

For example, I recently wrote the following unit test in FitNesse using the Mockito framework.


  @Before
  public void setUp() {
    manager = mock(GSSManager.class);
    properties = new Properties();
  }

  @Test
  public void credentialsShouldBeNullIfNoServiceName() throws Exception {
    NegotiateAuthenticator authenticator = 
      new NegotiateAuthenticator(manager, properties);
    assertNull(authenticator.getServerCredentials());
    verify(manager, never()).createName(
      anyString(), (Oid) anyObject(), (Oid) anyObject());
  }

The first line in the setUp function is lovely. It’s kind of hard to get prettier than that. Anybody reading it understands that manager will be a mock of the GSSManager class.

It’s not too hard to understand the test itself. Apparently we are happy to have the manager be a dummy object with the constraint that createName is never called by NegotiateAuthenticator. The anyString() and anyObject() calls are pretty self explanatory.

On the other hand, I wish I could have said this:

assertTrue(manager.createNameWasNotCalled());

That statement does not require my poor readers to understand anything about Mockito. Of course it does require me to hand-roll a manager mock. Would that be hard? Let’s try.

First I need to create a dummy.


  private class MockGSSManager extends GSSManager {
    public Oid[] getMechs() {
      return new Oid[0];
    }

    public Oid[] getNamesForMech(Oid oid) throws GSSException {
      return new Oid[0];
    }

    public Oid[] getMechsForName(Oid oid) {
      return new Oid[0];
    }

    public GSSName createName(String s, Oid oid) throws GSSException {
      return null;
    }

    public GSSName createName(byte[] bytes, Oid oid) throws GSSException {
      return null;
    }

    public GSSName createName(String s, Oid oid, Oid oid1) throws GSSException {
      return null;
    }

    public GSSName createName(byte[] bytes, Oid oid, Oid oid1) throws GSSException {
      return null;
    }

    public GSSCredential createCredential(int i) throws GSSException {
      return null;
    }

    public GSSCredential createCredential(GSSName gssName, int i, Oid oid, int i1) throws GSSException {
      return null;
    }

    public GSSCredential createCredential(GSSName gssName, int i, Oid[] oids, int i1) throws GSSException {
      return null;
    }

    public GSSContext createContext(GSSName gssName, Oid oid, GSSCredential gssCredential, int i) throws GSSException {
      return null;
    }

    public GSSContext createContext(GSSCredential gssCredential) throws GSSException {
      return null;
    }

    public GSSContext createContext(byte[] bytes) throws GSSException {
      return null;
    }

    public void addProviderAtFront(Provider provider, Oid oid) throws GSSException {
    }

    public void addProviderAtEnd(Provider provider, Oid oid) throws GSSException {
    }
  }

“Oh, ick!” you say. Yes, I agree it’s a lot of code. On the other hand, it took me just a single keystroke on my IDE to generate all those dummy methods. (In IntelliJ it was simply command-I to implement all unimplemented methods.) So it wasn’t particularly hard. And, of course, I can put this code somewhere where nobody had to look at it unless they want to. It has the advantage that anybody who knows Java can understand it, and can look right at the methods to see what they are returning. No “special” knowledge of the mocking framework is necessary.

Next, let’s’ make a test double that does precisely what this test needs.


  private class GSSManagerSpy extends MockGSSManager {
    public boolean createNameWasCalled;

    public GSSName createName(String s, Oid oid) throws GSSException {
      createNameWasCalled = true;
      return null;
    }
  }

Well, that just wasn’t that hard. It’s really easy to understand too. Now, let’s rewrite the test.


  @Test
  public void credentialsShouldBeNullIfNoServiceNameWithHandRolledMocks() throws Exception {
    NegotiateAuthenticator authenticator = new NegotiateAuthenticator(managerSpy, properties);
    assertNull(authenticator.getServerCredentials());
    assertFalse(managerSpy.createNameWasCalled);
  }

Well, that test is just a load easier to read than verify(manager, never()).createName(anyString(), (Oid) anyObject(), (Oid) anyObject());.

“But Uncle Bob!” I hear you say. “That scenario is too simple. What if there were lots of dependencies and things…” I’m glad you asked that question, because the very next test is just such a situation.


  @Test
  public void credentialsShouldBeNonNullIfServiceNamePresent() throws Exception {
    properties.setProperty("NegotiateAuthenticator.serviceName", "service");
    properties.setProperty("NegotiateAuthenticator.serviceNameType", "1.1");
    properties.setProperty("NegotiateAuthenticator.mechanism", "1.2");
    GSSName gssName = mock(GSSName.class);
    GSSCredential gssCredential = mock(GSSCredential.class);
    when(manager.createName(anyString(), (Oid) anyObject(), (Oid) anyObject())).thenReturn(gssName);
    when(manager.createCredential((GSSName) anyObject(), anyInt(), (Oid) anyObject(), anyInt())).thenReturn(gssCredential);
    NegotiateAuthenticator authenticator = new NegotiateAuthenticator(manager, properties);
    Oid serviceNameType = authenticator.getServiceNameType();
    Oid mechanism = authenticator.getMechanism();
    verify(manager).createName("service", serviceNameType, mechanism);
    assertEquals("1.1", serviceNameType.toString());
    assertEquals("1.2", mechanism.toString());
    verify(manager).createCredential(gssName, GSSCredential.INDEFINITE_LIFETIME, mechanism, GSSCredential.ACCEPT_ONLY);
    assertEquals(gssCredential, authenticator.getServerCredentials());
  }

Now I’ve got three test doubles that interact with each other; and I am verifying that the code under test is manipulating them all correctly. I could create hand-rolled test doubles for this; but the wiring between them would be scattered in the various test-double derivatives. I’d also have to write a significant number of accessors to get the values of the arguments to createName and createCredential. In short, the hand-rolled test-double code would be harder to understand than the Mockito code. The Mockito code puts the whole story in one simple test method rather than scattering it hither and yon in a plethora of little derivatives.

What’s more, since it’s clear that I should use a mocking framework for this test, I think I should be consistent and use if for all the tests in this file. So the hand-rolled MockGSSManager and ManagerSpy are history.

“But Uncle Bob, aren’t we always going to have dependencies like that? So aren’t we always going to have to use a mocking framework?”

That, my dear reader, is the real point of this blog. The answer to that salient questions is a profound: “No!“

Why did I have to use Mockito for these tests? Because the number of objects in play was large. The module under test (NegotiateAuthenticator) used GSSName, GSSCredential, and GSSManager. In other words the coupling between the module under test and the test itself was high. (I see lightbulbs above some of your heads.) That’s right, boys and girls, we don’t want coupling to be high!

It is the high coupling between modules and tests that creates the need for a mocking framework. This high coupling is also the cause of the dreaded “Fragile Test” problem. How many tests break when you change a module? If the number is high, then the coupling between your modules and tests in high. Therefore, I conclude that those systems that make prolific use of mocking frameworks are likely to suffer from fragile tests.

Of the 277 unit test files in FitNesse, only 11 use Mockito. The reason for small number is two-fold. First, we test outcomes more often than we test mechanisms. That means we test how a small group of classes behaves, rather than testing the dance of method calls between those classes. The second reason is that our test doubles have no middle class. They are either very simple stubs and spies or they are moderately complex fakes.

Testing outcomes is a traditional decoupling technique. The test doesn’t care how the end result is calculated, so long as the end result is correct. There may be a dance of several method calls between a few different objects; but the test is oblivious since it only checks the answer. Therefore the tests are not strongly coupled to the solution and are not fragile.

Keeping middle-class test doubles (i.e. Mocks) to a minimum is another way of decoupling. Mocks, by their very nature, are coupled to mechanisms instead of outcomes. Mocks, or the setup code that builds them, have deep knowledge of the inner workings of several different classes. That knowledge is the very definition of high-coupling.

What is a “moderately complex fake” and why does it help to reduce coupling? One example within FitNesse is MockSocket. (The name of this class is historical. Nowadays it should be called FakeSocket.) This class derives from Socket and implements all its methods either to remember what was sent to the socket, or to allow a user to read some canned data. This is a “fake” because it simulates the behavior of a socket. It is not a mock because it has no coupling to any mechanisms. You don’t ask it whether it succeeded or failed, you ask it to send or recieve a string. This allows our unit tests to test outcomes rather than mechanisms.

The moral of this story is that the point at which you start to really need a mocking framework is the very point at which the coupling between your tests and code is getting too high. There are times when you can’t avoid this coupling, and those are the times when mocking frameworks really pay off. However, you should strive to keep the coupling between your code and tests low enough that you don’t need to use the mocking framework very often.

You do this by testing outcomes instead of mechanisms.

Dependency Injection Inversion

Uncle Bob — Sun, 17 Jan 2010 12:42:00 -0600

Dependency Injection is all the rage. There are several frameworks that will help you inject dependencies into your system. Some use XML (God help us) to specify those dependencies. Others use simple statements in code. In either case, the goal of these frameworks is to help you create instances without having to resort to new or Factories.

I think these frameworks are great tools. But I also think you should carefully restrict how and where you use them.

Consider, for example, this simple example using Google’s Guice framework.


 public class BillingApplication {
   public static void main(String[] args) {
    Injector injector = Guice.createInjector(new BillingModule());
    BillingService billingService = injector.getInstance(BillingService.class);
    billingService.processCharge(2034, "Bob");
  }
 }

My goal is to create an instance of BillingService. To do this, I first get an Injector from Guice. Then I use the injector to get an instance of my BillingService class. What’s so great about this? Well, take a look at the constructor of the BillingService class.


class BillingService {
  private CreditCardProcessor processor;
  private TransactionLog transactionLog;

  @Inject
  BillingService(CreditCardProcessor processor, TransactionLog transactionLog) {
    this.processor = processor;
    this.transactionLog = transactionLog;
  }

  public void processCharge(int amount, String id) {
    boolean approval = processor.approve(amount, id);
    transactionLog.log(
      String.format("Transaction by %s for %d %s",
      id, amount, approvalCode(approval)));
  }

  private String approvalCode(boolean approval) {
    return approval?"approved":"denied";
  }
}

Oh ho! The BillingService constructor requires two arguments! A CreditCardProcessor and a TransactionLog. How was the main program able to create an instance of BillingService without those two arguments? That’s the magic of Guice (and of all Dependency Injection frameworks). Guice knows that the BillingService needs those two arguments, and it knows how to create them. Did you see that funky @Inject attribute above the constructor? That’s how it got connected into Guice.

And here’s the magic module that tells Guice how to create the arguments for the BillingService


public class BillingModule extends AbstractModule {
  protected void configure() {
    bind(TransactionLog.class).to(DatabaseTransactionLog.class);
    bind(CreditCardProcessor.class).to(MyCreditCardProcessor.class);
  }
}

Clever these Google-folk! The two bind functions tell Guice that whenever we need an instance of a TransactionLog it should use an instance of DatabaseTransactionLog. Whenever it needs a CreditCardProcessor it should use an instance of MyCreditCardProcessor.

Isn’t that cool! Now you don’t have to build factories. You don’t have to use new. You just tell Guice how to map interfaces to implementations, and which constructors to inject those implementations in to, and then call Injector.getInstance(SomeClass.class); and voila! You have your instance automatically constructed for you. Cool.

Well, yes it’s cool. On the other hand, consider this code:


public class BillingApplicationNoGuice {
  public static void main(String[] args) {
    CreditCardProcessor cp = new MyCreditCardProcessor();
    TransactionLog tl = new DatabaseTransactionLog();
    BillingService bs = new BillingService(cp, tl);
    bs.processCharge(9000, "Bob");
  }
}

Why is this worse? It seems to me it’s better.

But Uncle Bob, you’ve violated DIP by creating concrete instances!

True, but you have to mention concrete instances somewhere. main seems like a perfectly good place for that. Indeed, it seems better than hiding the concrete references in BillingModule.

I don’t want a bunch of secret modules with bind calls scattered all around my code. I don’t want to have to hunt for the particular bind call for the Zapple interface when I’m looking at some module. I want to know where all the instances are created.

But Uncle Bob, You’d know where they are because this is a Guice application.

I don’t want to write a Guice application. Guice is a framework, and I don’t want framework code smeared all through my application. I want to keep frameworks nicely decoupled and at arms-length from the main body of my code. I don’t want to have @Inject attributes everywhere and bind calls hidden under rocks.

But Uncle Bob, What if I want to get an instance of BillingService from deep in the bowels of my application? With Guice I can just say injector.getInstance(BillingService.class);.

True, but I don’t want to have createInstance calls scattered all through my code. I don’t want Guice to be poured all over my app. I want my app to be clean, not soaked in Guice.

But Uncle Bob, That means I have to use new or factories, or pass globals around.

You think the injector is not a global? You think BillingService.class is not a global? There will always be globals to deal with. You can’t write systems without them. You just need to manage them nicely.

And, no, I don’t have to use new everywhere, and I don’t need factories. I can do something as simple as:


public class BillingApplicationNoGuice {
  public static void main(String[] args) {
    CreditCardProcessor cp = new MyCreditCardProcessor();
    TransactionLog tl = new DatabaseTransactionLog();
    BillingService.instance = new BillingService(cp, tl);

    // Deep in the bowels of my system.
    BillingService.instance.processCharge(9000, "Bob");
  }
}

But Uncle Bob, what if you want to create many instances of BillingService rather than just that one singleton?

Then I’d use a factory, like so:


public class BillingApplication {
   public static void main(String[] args) {
    Injector injector = Guice.createInjector(new BillingModule());
    BillingService.factory = new BillingServiceFactory(injector);

    // Deep in the bowels of my code.
    BillingService billingService = BillingService.factory.make();
    billingService.processCharge(2034, "Bob");
  }
}

But Uncle Bob, I thought the whole idea was to avoid factories!

Hardly. After all, Guice is just a big factory. But you didn’t let me finish. Did you notice that I passed the Guice injector into the factory? Here’s the factory implementation.


public class BillingServiceFactory extends AbstractModule {
  private Injector injector;

  public BillingServiceFactory(Injector injector) {
    this.injector = injector;
  }

  protected void configure() {
    bind(TransactionLog.class).to(DatabaseTransactionLog.class);
    bind(CreditCardProcessor.class).to(MyCreditCardProcessor.class);
  }

  public BillingService make() {
    return injector.getInstance(BillingService.class);
  }
}

I like this because now all the Guice is in one well understood place. I don’t have Guice all over my application. Rather, I’ve got factories that contain the Guice. Guicey factories that keep the Guice from being smeared all through my application.

What’s more, if I wanted to replace Guice with some other DI framework, I know exactly what classes would need to change, and how to change them. So I’ve kept Guice uncoupled from my application.

Indeed, using this form allows me to defer using Guice until I think it’s necessary. I can just build the factories the good old GOF way until the need to externalize dependencies emerges.

But Uncle Bob, don’t you think Dependency Injection is a good thing?

Of course I do. Dependency Injection is just a special case of Dependency Inversion. I think Dependency Inversion is so important that I want to invert the dependencies on Guice! I don’t want lots of concrete Guice dependencies scattered through my code.

BTW, did you notice that I was using Dependency Injection even when I wasn’t using Guice at all? This is nice and simple manual dependency injection. Here’s that code again in case you don’t want to look back:



public class BillingApplicationNoGuice {
  public static void main(String[] args) {
    CreditCardProcessor cp = new MyCreditCardProcessor();
    TransactionLog tl = new DatabaseTransactionLog();
    BillingService bs = new BillingService(cp, tl);
    bs.processCharge(9000, "Bob");
  }
}

Dependency Injection doesn’t require a framework; it just requires that you invert your dependencies and then construct and pass your arguments to deeper layers. Consider, for example, that the following test works just fine in all the cases above. It does not rely on Guice, it only relies on the fact that dependencies were inverted and can be injected into BillingService



public class BillingServiceTest {
  private LogSpy log;

  @Before
  public void setup() {
    log = new LogSpy();
  }

  @Test
  public void approval() throws Exception {
    BillingService bs = new BillingService(new Approver(), log);
    bs.processCharge(9000, "Bob");
    assertEquals("Transaction by Bob for 9000 approved", log.getLogged());
  }

  @Test
  public void denial() throws Exception {
    BillingService bs = new BillingService(new Denier(), log);
    bs.processCharge(9000, "Bob");
    assertEquals("Transaction by Bob for 9000 denied", log.getLogged());    
  }
}

class Approver implements CreditCardProcessor {
  public boolean approve(int amount, String id) {
    return true;
  }
}

class Denier implements CreditCardProcessor {
  public boolean approve(int amount, String id) {
    return false;
  }
}

class LogSpy implements TransactionLog {
  private String logged;

  public void log(String s) {
    logged = s;
  }

  public String getLogged() {
    return logged;
  }
}

Also notice that I rolled my own Test Doubles (we used to call them mocks, but we’re not allowed to anymore.) It would have been tragic to use a mocking framework for such a simple set of tests.

Most of the time the best kind of Dependency Injection to use, is the manual kind. Externalized dependency injection of the kind that Guice provides is appropriate for those classes that you know will be extension points for your system.

But for classes that aren’t obvious extension points, you will simply know the concrete type you need, and can create it at a relatively high level and inject it down as an interface to the lower levels. If, one day, you find that you need to externalize that dependency, it’ll be easy because you’ve already inverted and injected it.

Archeological Dig

Uncle Bob — Wed, 11 Nov 2009 10:39:00 -0600

I was going through some old files today, and I stumbled upon some acetate slides from 1995. They were entitled: “Managing OO Projects”. Wow! What a difference fifteen years makes! (Or does it?) ...

In 1995-99 I was frequently asked to speak to managers about what a transition to OO (usually from C to C++) would do for (or to) them. I would spend a half day to a day going over the issues, costs, and benefits.

One part of that talk (usually about 90 min) was a discussion about software process. It was the process part of the talk that those acetate slides that I found described.

1995 was during the ascendency of Waterfall. Waterfall thinking was king. RUP had not yet been conceived as an acronym. And though Booch was beating the drum for incrementalism, most people (even many within Rational) were thinking in terms of six to eighteen month waterfalls.

So, here are the slides that I uncovered deep within an old filing cabinet. I scanned them in. They were produced on a Macintosh using the old “More” program. (Where is that program now? It was so good.)

Go ahead and read them now. Then come back here and continue…

What struck me about those slides was the consistency of the message with today. It was all about iterative development. Small iterations (though I never deigned to define the length in the slides, I frequently told people 2 weeks), measured results, etc. etc. Any Agile evangelist could use those slides today. He or she would have to dance quickly around a few statements, but overall the message has not shifted very much.

What’s even more interesting is the coupling between the process, and OO. The slides talk a lot about dependency management and dependency structure. There are hints of the SOLID principles contained in those slides. (Indeed several of the principles had already been identified by that time.) This coupling between process and software structure was a harbinger of the current craftsmanship/clean-code movement.

Of course the one glaring omission from these slides is TDD. That makes me think that TDD was the true catalyst of change, and the bridge that conveyed our industry from then to now.

Anyway, I guess the more things change, the more they stay the same.

Comments please!

Excuse me sir, What Planet is this?

Uncle Bob — Thu, 05 Nov 2009 10:35:00 -0600

Update 12 hours later.

I’m not very proud of this blog (or as one commenter correctly called it “blart”). It is derisive, sneering, and petulant. It is unprofessional. I guess I was having a bad morning. I slipped. I didn’t check with my green band.

So I apologize to the audience at large, and to Cashto. You should expect better from me.

I thought about pulling the blog down; but I think I’ll leave it up here as an example of how not to write a blog.

Some folks on twitter have been asking me to respond to this blough (don’t bother to read it right now, I’ll give you the capsule summary below. Read it later if you must). It’s a typical screed complete with all the usual complaints, pejoratives, and illogic. Generally I don’t respond to blarts like this because I don’t imagine that any readers take them very seriously. But it appears that this blelch has made the twitter rounds and that I’m going to have to say something.

Here are the writer’s main points:

He likens unit tests to training wheels and says you can’t use them to win the Tour de France.
- I think winning the Tour de France has much more to do with self-discipline than he imagines it does. I mean it’s not really just as simple as: “Get on a bike and ride like hell!”

He says testing is popular amongst college students
- I’d like to see his data!

He goes on to say: (cue John Cleese): “(ahem) unit tests lose their effectiveness around level four of the Dreyfus model of skill acquisition”.
- (blank stunned stare). Is this a joke? All false erudition aside, WTF is he talking about?

He says that unit tests don’t give us confidence in refactoring because they over-specify behavior and are too fine-grained.
- He apparently prefers hyphenations to green bars.

He says they mostly follow the “happy path” and therefore don’t find bugs.
- Maybe when he writes them! This is a big clue that the author can’t spell TDD.

He complains about Jester
- without getting the joke!

He says unit tests “encourage some pretty questionable practices.” He flings a few design principles around and says that unit testing doesn’t help with them.
- as the author, editor, and/or advocate of many of those principles; I have a slightly different view.

He says that “many are starting to discover that functional programming teaches far better design principles than unit testing ever will”
- Oh no! Not the old “My language teaches design.” claim. We’ve heard it all before. They said it about C++, Java, COM (?!), etc… The lesson of the ‘90s? Languages don’t teach design. You can make a mess in any language.

He says: “tests can have negative ROI. Not only do they cost a lot to write, they’re fragile, they’re always broken, they get in the way of your refactoring, they’re always having you chase down bogus failures, and the only way to get anything done is to ignore them.”
- In one of my favorite episodes of Dr. Who, Tom Baker exits the Tardis, walks up to someone on the street and says: “Excuse me sir, what planet is this?”

He says: “What I’m saying is that it’s okay if you don’t write unit tests for everything. You probably have already suspected this for a long time, but now you know. I don’t want you to feel guilty about it any more.”
- Translation: “I don’t want to feel guilty about it anymore so I’m going to try to convince you…” I sincerely doubt this author has your best interests at heart.

He says: “Debugging is easy, at least in comparison to writing all those tedious tests.”
- Refer back to the Dr. Who quote.

To quote Barack Obama: “Enough!”

Has this guy ever done TDD? I rather doubt it. Or if he did, he was so inept at it that his tests were “fragile”, “always broken”, and “in the way of refactoring”. I think he should give it another try and this time spend a bit more time on test design.

Perhaps he’s one of those guys who thought that unit tests were best written after the code. Certainly his list of complains makes a lot of sense in that light. Hint: If you want to fail at unit testing, write them last.

The bottom line is that the guy probably had a bad experience writing unit tests. He’s tired of writing them and wants to write fewer of them. He’d rather debug. He thinks he can refactor without tests (which is definitively false). He thinks he can go faster by writing fewer tests. Fine, that’s his choice. And he’s found a rationalization to support his desires. Great.

I predict that his results will not compare well with those who adopt the discipline of TDD. I predict that after a few years he’ll either change his mind, or go into management.

Oh, and to the author: Gesundheit!

Manual Mocking: Resisting the Invasion of Dots and Parentheses

Uncle Bob — Wed, 28 Oct 2009 18:12:16 -0500

The twittersphere has been all abuzz today because of something I tweeted early this morning (follow @unclebobmartin). In my tweet I said that I hand-roll most of my own mock objects in Java, rather than using a mocking framework like mockito.

The replies were numerous and vociferous. Dave Astels poignantly stated that hand-rolling mocks is so 2001!

So why do I roll my own mocks?

Consider the following two tests:

public class SelectorTest {
  private List<Object> list;

  @Before
  public void setup() {
    list = new ArrayList<Object>();
    list.add(new Object());
  }

  @Test
  public void falseMatcherShouldSelectNoElements_mockist() {
    Matcher<Object> falseMatcher = mock(Matcher.class);
    Selector<Object> selector = new Selector<Object>(falseMatcher);
    when(falseMatcher.match(anyObject())).thenReturn(false);
    List<Object> selection = selector.select(list);
    assertThat(selection.size(), equalTo(0));
  }

  @Test
  public void falseMatcherShouldSelectNoElements_classic() {
    Matcher<Object> falseMatcher = new FalseMatcher();
    Selector<Object> selector = new Selector<Object>(falseMatcher);
    List<Object> selection = selector.select(list);
    assertThat(selection.size(), equalTo(0));}

  private static class FalseMatcher implements Matcher<Object> {
    public boolean match(Object element) {
      return false;
    }
  }
}

The first test shows the really cool power of mockito (which is my current favorite in the menagerie of java mocking frameworks). Just in case you can’t parse the syntax, let me describe it for you:

falseMatcher is assigned the return value of the “mock” function. This is a very cool function that takes the argument class and builds a new stubbed object that derives from it. In mockito, the argument can be a class or an interface. Cool!

Now don’t get all panicy about the strange parenthetic syntax of the ‘when’ statement. The ‘when’ statement simply tells the mock what to do when a method is called on it. In this case it instructs the falseMatcher to return false when the ‘match’ function is called with any object at all.

The second test needs no explanation.

...

And that’s kind of the point. Why would I include a bizzare, dot-ridden, parentheses-laden syntax into my tests, when I can just as easily hand-roll the stub in pure and simple java? How hard was it to hand-roll that stub? Frankly, it took a lot less time and effort to hand-roll it than it took to write the (when(myobj.mymethod(anyx())).)()).))); statement.

OK, I’m poking a little fun here. But it’s true. My IDE (InteliJ) generated the stub for me. I simply started with:

Matcher<Object> falseMatcher = new Matcher<Object>() {};

InteliJ complained that some methods weren’t implemented and offered to implement them for me. I told it to go ahead. It wrote the ‘match’ method exactly as you see it. Then I chose “Convert Anonymous to Inner…” from the refactoring menu and named the new class FalseMatcher. Voila! No muss, no fuss, no parenthetic maze of dots.

Now look, I’m not saying you shouldn’t use mockito, or any of these other mocking tools. I use them myself when I must. Here, for example, is a test I wrote in FitNesse. I was forced to use a mocking framework because I did not have the source code of the classes I was mocking.

  @Before
  public void setUp() {
    manager = mock(GSSManager.class);
    properties = new Properties();
  }

  @Test
  public void credentialsShouldBeNonNullIfServiceNamePresent() throws Exception {
    properties.setProperty("NegotiateAuthenticator.serviceName", "service");
    properties.setProperty("NegotiateAuthenticator.serviceNameType", "1.1");
    properties.setProperty("NegotiateAuthenticator.mechanism", "1.2");
    GSSName gssName = mock(GSSName.class);
    GSSCredential gssCredential = mock(GSSCredential.class);
    when(manager.createName(anyString(), (Oid) anyObject(), (Oid) anyObject())).thenReturn(gssName);
    when(manager.createCredential((GSSName) anyObject(), anyInt(), (Oid) anyObject(), anyInt())).thenReturn(gssCredential);
    NegotiateAuthenticator authenticator = new NegotiateAuthenticator(manager, properties);
    Oid serviceNameType = authenticator.getServiceNameType();
    Oid mechanism = authenticator.getMechanism();
    verify(manager).createName("service", serviceNameType, mechanism);
    assertEquals("1.1", serviceNameType.toString());
    assertEquals("1.2", mechanism.toString());
    verify(manager).createCredential(gssName, GSSCredential.INDEFINITE_LIFETIME, mechanism, GSSCredential.ACCEPT_ONLY);
    assertEquals(gssCredential, authenticator.getServerCredentials());
  }

If I’d had the source code of the GSS classes, I could have created some very simple stubs and spies that would have allowed me to make these tests a lot cleaner than they currently appear. Indeed, I might have been able to test the true behavior of the classes rather than simply testing that I was calling them appropriately…

Mockism

That last bit is pretty important. Some time ago Martin Fowler wrote a blog about the Mockist and Classical style of TDD. In short, Mockists don’t test the behavior of the system so much as they test that their classes “dance” well with other classes. That is, they mock/stub out all the other classes that the class under test uses, and then make sure that all the right functions are called in all the right orders with all the right arguments. etc. There is value to doing this in many cases. However you can get pretty badly carried away with the approach.

The classical approach is to test for desired behavior, and trust that if the test passes, then the class being tested must be dancing well with its partners.

Personally, I don’t belong to either camp. I sometimes test the choreography, and I sometimes test the behavior. I test the choreography when I am trying to isolate one part of the system from another. I test for the behavior when such isolation is not important to me.

The point of all this is that I have observed that a heavy dependence on mocking frameworks tends to tempt you towards testing the dance when you should be testing behavior. Tools can drive the way we think. So remember, you dominate the tool; don’t let the tool dominate you!

But aren’t hand-rolled mocks fragile?

Yes, they can be. If you are mocking a class or interface that it very volatile (i.e. you are adding new methods, or modifying method signatures a lot) then you’ll have to go back and maintain all your hand-rolled mocks every time you make such a change. On the other hand, if you use a mocking framework, the framework will take care of that for you unless one of the methods you are specifically testing is modified.

But here’s the thing. Interfaces should not usually be volatile. They should not continue to grow and grow, and the methods should not change much. OK, I realize that’s wishful thinking. But, yes, I wish for the kind of a design in which interfaces are the least volatile source files that you have. That’s kind of the point of interfaces after all… You create interfaces so that you can separate volatile implementations from non-volatile clients. (Or at least that’s one reason.)

So if you are tempted to use a mocking framework because you don’t want to maintain your volatile interfaces, perhaps you should be asking yourself the more pertinent question about why your interfaces are so volatile.

Still, if you’ve got volatile interfaces, and there’s just no way around it, then a mocking framework may be the right choice for you.

So here’s the bottom line.

It’s easy to roll your own stubs and mocks. Your IDE will help you and they’ll be easier and more natural to read than the dots and parentheses that the mocking frameworks impose upon you.

Mocking frameworks drive you towards testing choreography rather than behavior. This can be useful, but it’s not always appropriate. And besides, even when you are testing choreography, the hand-rolled stubs and mocks are probably easier to write and read.

There are special cases where mocking tools are invaluable, specifically when you have to test choreography with objects that you have no source for or when your design has left you with a plethora of volatile interfaces.

Am I telling you to avoid using mocking frameworks? No, not at all. I’m just telling you that you should drive tools, tools should not drive you.

If you have a situation where a mocking tool is the right choice, by all means use it. But don’t use it because you think it’s “agile”, or because you think it’s “right” or because you somehow think you are supposed to. And remember, hand-rolling often results in simpler tests without the litter of dots and parentheses!

We must ship now and deal with consequences

Uncle Bob — Thu, 15 Oct 2009 06:17:00 -0500

Martin Fowler has written a good blog about technical debt. He suggests that there are two axes of debt: deliberate and prudent. This creates four quadrants: deliberate-prudent, deliberate-imprudent, inadvertent-prudent, and inadvertent-imprudent. I agree with just about everything in his blog except for one particular caption…

Inadvertent-Imprudent Debt.

There is more of this debt than any other kind. It is all too common that software developers create a mess and don’t know they are doing it. They have not developed a nose that identifies code smells. They don’t know design principles, or design patterns. They think that the reek of rotten code is normal, and don’t even identify it as smelling bad. They think that their slow pace through the thick morass of tangled code is the norm, and have no idea they could move faster. These people destroy projects and bring whole companies to their knees. Their name is Doom.

Deliberate-Imprudent Debt.

There is a meme in our industry (call it the DI meme) that tells young software developers that rushing to the finish line at all costs is the right thing to do. This is far worse than the ignorance of the first group because these folks willfully create debt without counting the cost. Worse, this meme is contagious. People who are infected with it tend to infect others, causing an epidemic of deliberately imprudent debtors (sound familiar?) The end result, as we are now all know, is economic catastrophe, inflation (of estimates) and crushing interest (maintenance) payments. They have become death, the destroyer of worlds.

Inadvertent-Prudent Debt.

This is something of an oxymoron. Ironically, it is also the best of all possible states. The fact is that no matter how careful we are, there is always a better solution that we will stumble upon later. How many times have you finished a system only to realize that if you wrote it again, you’d do it very differently, and much better?

The result is that we are always creating a debt, because our hindsight will always show us a better option after it is too late. So even the best outcome still leaves us owing. (Mother Earth will eventually collect that debt!)

Deliberate-Prudent Debt.

This is the quadrant that I have the biggest problem with. And it is this quadrant in which Martin uses the caption I don’t like. The Caption is: “We must ship now and deal with consequences.”

Does this happen? Yes. Should it happen? Rarely, yes. But it damned well better not happen very often, and it damned well better not happen out of some misplaced urge to get done without counting the cost.

The problem I have with this quadrant (DP) is that people who are really in quadrant DI think they are in DP, and use words such as those that appear in the caption as an excuse to rack up a huge imprudent debt.

The real issue is the definition of the word: Imprudent.

So let me ask you a question. How prudent is debt? There is a very simple formula for determining whether debt is prudent or imprudent. You can use this formula in real life, in business, and in programming. The formula is: Does the debt increase your net worth, and can you make the payments?

People often focus on the first criterion, without properly considering the second. Buying a house is almost certain to increase your net worth despite the debt (though lately…). On the other hand, if you cannot make the payments, you won’t keep that house for long. The reason for our current economic woes has a lot to do with people trying to increase their net worth despite the fact that they couldn’t afford the payments. (indeed, they were encouraged by a meme very similar to the DI meme!)

Bad code is always imprudent.

Writing bad code never increases your net worth; and the interest rate is really high. People who write bad code are like those twenty-somethings who max out all their credit cards. Every transaction decreases net worth, and has horrendous consequences for cash flow. In the end, the vast bulk of your effort goes to paying the interest (the inevitable slow down of the team as they push the messes around). Paying down the principle becomes infeasible. (Just the way credit card companies like it.)

Some Suboptimal Design Decision are Prudent Debt.

But most are not. Every once in awhile there is a suboptimal design decision that will increase the net worth of the project by getting that project into customer’s hand’s early.

This is not the same as delivering software that is under-featured. It is often prudent to increase the net worth of a project by giving customers early access to a system without a full and rich feature set. This is not debt. This is more like a savings account that earns interest.

Indeed, this is one reason that most technical debt is imprudent. If you are truly concerned about getting to market early, it is almost always better to do it with fewer features, than with suboptimal design. Missing features are a promise that can be kept. Paying back suboptimal designs creates interest payments that often submerge any attempts at payback and can slow the team to the breaking point.

But there are some cases where a sub-optimal design can increase your net worth by allowing you to deliver early. However, the interest rate needs to be very low, and the principle payments need to be affordable, and big enough to pay back the debt in short order.

What does a low interest rate mean? It means that the sub-optimal design does not infiltrate every part of your system. It means that you can put the sub-optimal design off in a corner where it doesn’t impact your daily development life.

For example, I recently implemented a feature in FitNesse using HTML Frames. This is sub-optimal. On the other hand, the feature is constrained to one small part of the system, and it simply doesn’t impact any other part of the system. It does not impede my progress. There is no mess for me to move around. The interest rate is almost zero! (nice deal if you can get it!)

Implementing that feature with ajax is a much larger project. I would have had to invest a great deal of time and effort, and would have had to restructure massive amounts of the internal code. So the choice was a good one.

Better yet, the customer experience has pretty much been a big yawn. I thought people would really like the feature and would drive me to expand upon it. Instead, the customer base has virtually ignored it.

So my solution will be to pay back this debt by eliminating the feature. It was a cheap experiment, that resulted in my not having to spend a lot of time and effort on a new architecture! Net worth indeed!

But it might have gone the other way. My customers may have said: “Wow, Great! We want more!” At that point it would have been terrible to expand on the HTML Frames! That decision would have been in the DI quadrant. Deliberate imprudence! Rather, my strategy would have been to replace the suboptimal Frames design of the feature with an isolated ajax implementation, and then to gradually migrate the ajax solution throughout the project. That would have been annoying, but loan payments always are.

Summary

So, don’t let the caption in the DP quadrant be an excuse. Don’t fall for the DI meme that says “We just gotta bite the bullet”. Tread very carefully when you enter the DP quadrant. Look around at all your options, because it’s easy to think you are in the DP quadrant when you are really in the DI quadrant.

Remember: Murphy shall send you strong delusion, that you should believe you are in DP; so that you will be damned in DI.

TDD Derangement Syndrome

Uncle Bob — Wed, 07 Oct 2009 08:32:00 -0500

My recent blog about TDD, Design Patterns, Concurrency, and Sudoku seemed to draw the ire of a few vocal TDD detractors. Some of these people were rude, insulting, derisive, dismissive, and immature. Well, Halloween is not too far away.

In spite of their self-righteous snickering they did ask a few reasonable questions. To be fair I thought it would be appropriate for me to answer them.

Is there any research on TDD?

It turns out that there is a fair bit.

One simple google search led me to this blog by Phil Haack in which he reviewed a TDD research paper. Quoting from the paper:

We found that test-first students on average wrote more tests and, in turn, students who wrote more tests tended to be more productive. We also observed that the minimum quality increased linearly with the number of programmer tests, independent of the development strategy employed.

The same google search led me to this blog by Matt Hawley, in which he reviewed several other research papers. Part of his summary:

* 87.5% of developers reported better requirements understanding. * 95.8% of developers reported reduced debugging efforts. * 78% of developers reported TDD improved overall productivity. * 50% of developers found that it decreased overall development time. * 92% of developers felt that TDD yielded high-quality code. * 79% of developers believed TDD promoted simpler design.

Actually, I recognize some of Matt’s results as coming from a rather famous 2003 study (also in the list of google results) by Laurie Wiliams and Boby George. This study describes a controlled experiment that they conducted in three different companies. Though Matt’s summary above is based (in part) on that study, there is more to say.

In the George-William study teams that practiced TDD took 16% longer to claim that they were done than the teams that did not practice TDD. Apparently tests are more accurate than claims since the non-TDD teams failed to pass one third of the researcher’s hidden acceptance tests, whereas the TDD teams passed about 6 out of 7. To paraphrase Kent Beck: “If it doesn’t have to work, I can get it done a lot faster!”

Another point of interest in this study is that the TDD teams produced a suite of automated tests with very high test coverage (close to 100% in most cases) whereas most of the non-TDD teams did not produce such a suite; even though they had been instructed to.

Jim Shore wrote a review of yet another research summary which I found in the same google search. This one combines 7 different studies (including George-Williams). Here the results range from dramatically improved quality and productivity to no observed effect.

Finally, there is this 2008 case Study of TDD at IBM and Microsoft which shows that TDDers enjoy a defect density reduction ranging from 30% to 90% (as measured by defect tracking tools) and a productivity cost of between 15% and 35% (the subjective opinion of the managers). I refer you back to Kent Beck’s comment above.

I’m sure there is more research out there. After all this was just one google search. I think it’s odd that the TDD detractors didn’t find anything when they did their google searches.

Oh yeah, and then there was that whole issue of IEEE Software that was dedicated to papers and research on TDD.

What projects have been written with TDD, hmmm?

Quite a few, actually. The following is a list of projects that have an automated suite of unit tests with very high coverage. Those that I know for a fact use TDD, I have noted as such. The others, I can only surmise. If you know of any others, please post a comment here.

JUnit. This one is kind of obvious. JUnit was written by Kent Beck and Erich Gamma using TDD throughout. If you measure software success by sheer distribution, this particular program is wildly successful.

Fit. Written by Ward Cunningham. The progenitor of most current acceptance testing frameworks.

FitNesse. This testing framework has tens of thousands of users. It is 70,000 lines of java code, with 90%+ code coverage. TDD throughout. Very small bug-list. Again, if you measure by distribution, another raving success.

Cucumber,
Rspec. These two are Testing frameworks in Ruby. Of course you’d expect a testing framework to be written with TDD, wouldn’t you? I know these were. TDD throughout.

Limelight. A gui framework in JRUby. TDD throughout.
jfreechart.
Spring
JRuby
Smallsql
Ant
MarsProject
Log4J
Jmock

Are there others? I’m sure there are. This was just a quick web search. Again, if you know of more, please add a comment.

Echoes from the Stone Age

Uncle Bob — Tue, 06 Oct 2009 11:07:29 -0500

The echoes from Joel Spolsky’s Duct Tape blog continue to bounce off the blogosphere and twitterverse. Tim Bray and Peter Seibel have both written responses to Joel, me, and each other.

Here are some stray thoughts…

TDD

Anyone who continues to think that TDD slows you down is living in the stone age. Sorry, that’s just the truth. TDD does not slow you down, it speeds you up.

Look, TDD is not my religion, it is one of my disciplines. It’s like dual entry bookkeeping for accountants, or sterile procedure for surgeons. Professionals adopt such disciplines because they understand the theory behind them, and have directly experienced the benefits of using them.

I have experienced the tremendous benefit that TDD has had in my work, and I have observed it in others. I have seen and experienced the way that TDD helps programmers conceive their designs. I have seen and experienced the way it documents their decisions. I have seen and experienced the decouplings imposed by the tests, and I have seen and experienced the fearlessness with which TDDers can change and clean their code.

To be fair, I don’t think TDD is always appropriate. There are situations when I break the discipline and write code before tests. I’ll write about these situations in another blog. However, these situations are few and far between. In general, for me and many others, TDD is a way to go fast, well, and sure.

The upshot of all this is simple. TDD is a professional discipline. TDD works. TDD makes you faster. TDD is not going away. And anyone who has not really tried it, and yet claims that it would slow them down, is simply being willfully ignorant. I don’t care if your name is Don Knuth, Jamie Zawinski, Peter Seibel, or Peter Pan. Give it a real try, and then you have the right to comment.

Let me put this another way. And now I’m talking directly to those who make the claim that TDD would slow them down. Are you really such a good programmer that you don’t need to thoroughly check your work? Can you conceive of a better way to check your work than to express your intent in terms of an executable test? And can you think of a better way to ensure that you can write that test other than to write it first?

If you can, then I want to hear all about it. but I don’t want to hear that you write a few unit tests after the fact. I don’t want to hear that you manually check your code. I don’t want to hear that you do design and therefore don’t need to write tests. Those are all stone-age concepts. I know. I’ve been there.

So there. <grin>

The Design Pattern Religion

Tim Bray said:

My experience suggests that there are few surer ways to doom a big software project than via the Design Patterns religion.

He’s right of course. The Design Patterns religion is a foul bird that ravages teams and cuts down young projects in their prime. But let’s be clear about what that religion is. The Design Patterns religion is the ardent belief that the use of design patterns is good.

Here’s a clue. Design Patterns aren’t good. They also aren’t bad. They just are. Given a particular software design situation, there may be a pattern that fits and is beneficial. There may also be patterns that would be detrimental. It’s quite possible that none of the currently documented patterns are appropriate and that you should close the book and just solve the problem.

Here’s another clue. You don’t use patterns. You don’t apply patterns. Patterns just are. If a particular pattern is appropriate to solve a given problem, then it will be obvious. Indeed it is often so obvious that you don’t realize that the pattern is in place until you are done. You look back at your code and realize: “Oh, that’s a Decorator!”.

So am I saying that Design Patterns are useless?

NO! I want you to read the patterns books. I want you to know those patterns inside and out. If I point at you and say “Visitor” I want you at the board drawing all the different variants of the pattern without hesitation. I want you to get all the names and roles right. I want you to know patterns.

But I don’t want you to use patterns. I don’t want you to believe in patterns. I don’t want you to make patterns into a religion. Rather I want you to be able to recognize them when they appear, and to regularize them in your code so that others can recognize them too.

Design Patterns have a huge benefit. They have names. If you are reading code, and you see the word “Composite”, and if the author took care to regularize the code to the accepted names and roles of the “Composite” pattern, then you will know what that part of the code is doing instantly. And that is powerful!

Minimizing Concurrency.

In my first Duct Tape blog I made the statement:

I found myself annoyed at Joel’s notion that most programmers aren’t smart enough to use templates, design patterns, multi-threading, COM, etc. I don’t think that’s the case. I think that any programmer that’s not smart enough to use tools like that is probably not smart enough to be a programmer period.

Tim responds with:

...multi-threading is part of the problem, not part of the solution; that essentially no application programmer understands threads well enough to avoid deadlocks and races and horrible non-repeatable bugs. And that COM was one of the most colossal piles of crap my profession ever foisted on itself.

Is concurrency really part of the problem? Yes! Concurrency is a really big part of the problem. Indeed, the first rule of concurrency is: DON’T. The second rule is: REALLY, DON’T.

The problem is that some times you have no choice. And in those situations, where you absolutely must use concurrency, you should know it inside and out!

I completely and utterly reject the notion that ignorance is the best defense. I reject that lack of skill can ever be an advantage. So I want you to know concurrency. I want to shout “Dining Philosophers” and have you run to the board without hesitation and show me all the different solutions. If I holler “Deadlock”, I want you to quickly identify the causes and solutions.

Here’s a clue. If you want to avoid using something, know that something cold.

Sudoku

At the end of his blog, Peter jumps on the pile of bodies already crushing Ron Jeffries regarding the Sudoku problem from July of 2006.

I find the pile-up disturbing. Ron had the courage to fail in public. Indeed he announced up front that he might “crash and burn”. And yet he got lambasted for it by people who hid behind someone else’s work. The responses to Ron’s tutorial blogs were completely unfair because the authors of those blogs had everything worked out for them by Dr. Peter Norvig before they published their screeds. They were comparing apples to oranges because their responses were about the solution whereas Ron’s blogs were about the process.

Which one of us has not gone down a rat-hole when hunting for a solution to a complex problem? Let that person write the first blog. Everyone else ought to be a bit more humble.

Do the people on the pile think that Ron is unable to solve the Sudoku problem? (Some have said as much.) Then they don’t know Ron very well. Ron could code them all under the table with one hand tied behind his back.

Personal issues aside, I find the discussion fascinating in it’s own right. Ron had attempted to solve the Sudoku problem by gaining insight into that problem through the process of coding intermediate solutions. This is a common enough TDD approach. Indeed, the Bowling Game and the Prime Factors Kata are both examples where this approach can work reasonably well.

This approach follows the advice of no less than Grady Booch who (quoting Heinlein) said: “when faced with a problem you do not understand, do any part of it you do understand, then look at it again.“

Ron was attempting to use TDD to probe into the problem to see if he could gain any insight. This technique often bears fruit. Sometimes it does not.

Here is a classic example. Imagine you were going to write a sort algorithm test first:

Test 1: Sort an empty array. Solution: Return the input array.

Test 2: Sort an array with one element. Solution: Return the input array.

Test 3: Sort an array with two elements. Solution: Compare the two elements and swap if out of order. Return the result.

Test 4: Sort an array with three elements. Solution: Compare the first two and swap if out of order. Compare the second two and swap if out of order. Compare the first two again and swap if out of order. Return the result.

Test 5: Sort an array with four elements. Solution: Put the compare and swap operations into a nested loop. Return the result.

The end result is a bubble sort. The algorithm virtually self assembles. If you had never heard of a bubble sort before, this simple set of tests would have driven you to implement it naturally.

Problems like Bowling, Prime Factors, and Bubble Sort hold out the interesting promise that TDD may be a way to derive algorithmms from first principles!

On the other hand, what set of tests would drive you to implement a QuickSort? There are none that I know of. QuickSort and Sudoku may require a serious amount of introspection and concentrated thought before the solution is apparent. They may belong to a class of algorithms that do not self-assemble like Bowling, Prime Factors, and Bubble Sort.

This blog by Kurt Christensen provides all the links to the various Sudoku articles, and sums it up this way.

TDD may not be the best tool for inventing new algorithms, it may very well be the best tool for applying those algorithms to the problem at hand.

Actually I think TDD is a good way to find out if an algorithm will self-assemble or not. It usually doesn’t take a lot of time to figure out which it’s going to be.

A Mess is not a Technical Debt.

Uncle Bob — Tue, 22 Sep 2009 08:15:19 -0500

The term Technical Debt was created by Ward Cunningham to describe the engineering trade-off’s that software developers and business stakeholders must often make in order to meet schedules and customer expectations. In short, you may need to use suboptimal designs in the short term, because the schedule does not allow longer term designs to be used. As a simple example, your initial website design may need to be frames based because you don’t have time to build an Ajax framework.

Clearly this causes a debt. If the customer is looking for a web 2.0 system, then frames just aren’t going to cut it for long. So time is going to have to be carved out of a future schedule to refit the system with an Ajax solution.

In short, the business has decided that it can afford to delay release 2 in order to accelerate release 1. Is this wise?

Businesses make this kind of trade-off all the time; and there’s nothing inherently unwise about it. If the early release of 1.0 drives the business that pays for the development of 2.0 then the business has won. So this kind of reasoned technical debt may indeed be appropriate.

Unfortunately there is another situation that is sometimes called “technical debt” but that is neither reasoned nor wise. A mess.

Technical debt may be necessary, but it had also better be clean! If you are going to implement a frames solution instead of an AJAX solution, then make sure that the workmanship of the frames solution is top-notch. Make sure the design is well balanced, and the code is clean. If you make a mess while implementing that frames solution, you’ll never be able to replace it with an AJAX framework. The mess will impede your progress forever.

A mess is not a technical debt. A mess is just a mess. Technical debt decisions are made based on real project constraints. They are risky, but they can be beneficial. The decision to make a mess is never rational, is always based on laziness and unprofessionalism, and has no chance of paying of in the future. A mess is always a loss.

When you buy a house and take on a big mortgage debt, you tighten up all your spending and accounting. You clean up your books and your budgets. You behave with increased discipline. The same is true of technical debt. The more technical debt you take on, the tighter your disciplines need to be. You should do more testing, and more pairing and more refactoring. Technical debt is not a license to make a mess. Technical debt creates the need for even greater cleanliness.

When you decide to take on a technical debt, you had better make sure that your code stays squeaky clean. Keeping the system clean is the only way you will pay down that debt.

One Thing: Extract till you Drop.

Uncle Bob — Fri, 11 Sep 2009 13:33:00 -0500

For years authors and consultants (like me) have been telling us that functions should do one thing. They should do it well. They should do it only.

The question is: What the hell does “one thing” mean?

After all, one man’s “one thing” might be someone else’s “two things”.

Consider this class:

  class SymbolReplacer {
    protected String stringToReplace;
    protected List<String> alreadyReplaced = new ArrayList<String>();

    SymbolReplacer(String s) {
      this.stringToReplace = s;
    }

    String replace() {
      Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");
      Matcher symbolMatcher = symbolPattern.matcher(stringToReplace);
      while (symbolMatcher.find()) {
        String symbolName = symbolMatcher.group(1);
        if (getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName)) {
          alreadyReplaced.add(symbolName);
          stringToReplace = stringToReplace.replace("$" + symbolName, translate(symbolName));
        }
      }
      return stringToReplace;
    }

    protected String translate(String symbolName) {
      return getSymbol(symbolName);
    }
  }

It’s not too hard to understand. The replace() function searches through a string looking for $NAME and replaces each instance with the appropriate translation of NAME. It also makes sure that it doesn’t replace a name more than once. Simple.

Of course the words “It also…” pretty much proves that this function does more than one thing. So we can probably split the function up into two functions as follows:

    String replace() {
      Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");
      Matcher symbolMatcher = symbolPattern.matcher(stringToReplace);
      while (symbolMatcher.find()) {
        String symbolName = symbolMatcher.group(1);
        replaceAllInstances(symbolName);
      }
      return stringToReplace;
    }

    private void replaceAllInstances(String symbolName) {
      if (getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName)) {
        alreadyReplaced.add(symbolName);
        stringToReplace = stringToReplace.replace("$" + symbolName, translate(symbolName));
      }
    }

OK, so now the replace() function simply finds all the symbols that need replacing, and the replaceAllInstances() function replaces them if they haven’t already been replaced. So do these function do one thing each?

Well, the replace() compiles the pattern and build the Matcher() Maybe those actions should be moved into the constructor?

  class SymbolReplacer {
    protected String stringToReplace;
    protected List<String> alreadyReplaced = new ArrayList<String>();
    private Matcher symbolMatcher;
    private final Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");

    SymbolReplacer(String s) {
      this.stringToReplace = s;
      symbolMatcher = symbolPattern.matcher(s);
    }

    String replace() {
      while (symbolMatcher.find()) {
        String symbolName = symbolMatcher.group(1);
        replaceAllInstances(symbolName);
      }
      return stringToReplace;
    }

    private void replaceAllInstances(String symbolName) {
      if (getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName)) {
        alreadyReplaced.add(symbolName);
        stringToReplace = stringToReplace.replace("$" + symbolName, translate(symbolName));
      }
    }

    protected String translate(String symbolName) {
      return getSymbol(symbolName);
    }
  }

OK, so now certainly the replace() function is doing one thing? Ah, but I see at least two. It loops, extracts the symbolName and then does the replace. OK, so how about this?

    String replace() {
      for (String symbolName = nextSymbol(); symbolName != null; symbolName = nextSymbol())
        replaceAllInstances(symbolName);

      return stringToReplace;
    }

    private String nextSymbol() {
      return symbolMatcher.find() ? symbolMatcher.group(1) : null;
    }

I had to restructure things a little bit. The loop is a bit ugly. I wish I could have said for (String symbolName : symbolMatcher) but I guess Matchers don’t work that way.

I kind of like the nextSymbol() function. It gets the Matcher nicely out of the way.

So now the replace() and nextSymbol() functions are certainly doing one thing. Aren’t they?

Well, I suppose I could separate the loop from the return in replace().

    String replace() {
      replaceAllSymbols();
      return stringToReplace;
    }

    private void replaceAllSymbols() {
      for (String symbolName = nextSymbol(); symbolName != null; symbolName = nextSymbol())
        replaceAllInstances(symbolName);
    }

I don’t see how I could make these functions smaller. They must be doing one thing. There’s no way to extract any other functions from them!

Uh… Wait. Is that the definition of one thing? Is a function doing one thing if, and only if, you simply cannot extract any other functions from it? What else could “one thing” mean? After all, If I can extract one function out of another, the original function must have been doing more than one thing.

So does that mean that for all these years the authors and consultants (like me) have been telling us to extract until you can’t extract anymore?

Let’s try that with the rest of this class and see what it looks like…

  class SymbolReplacer {
    protected String stringToReplace;
    protected List<String> alreadyReplaced = new ArrayList<String>();
    private Matcher symbolMatcher;
    private final Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");

    SymbolReplacer(String s) {
      this.stringToReplace = s;
      symbolMatcher = symbolPattern.matcher(s);
    }

    String replace() {
      replaceAllSymbols();
      return stringToReplace;
    }

    private void replaceAllSymbols() {
      for (String symbolName = nextSymbol(); symbolName != null; symbolName = nextSymbol())
        replaceAllInstances(symbolName);
    }

    private String nextSymbol() {
      return symbolMatcher.find() ? symbolMatcher.group(1) : null;
    }

    private void replaceAllInstances(String symbolName) {
      if (shouldReplaceSymbol(symbolName))
        replaceSymbol(symbolName);
    }

    private boolean shouldReplaceSymbol(String symbolName) {
      return getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName);
    }

    private void replaceSymbol(String symbolName) {
      alreadyReplaced.add(symbolName);
      stringToReplace = stringToReplace.replace(
        symbolExpression(symbolName), 
        translate(symbolName));
    }

    private String symbolExpression(String symbolName) {
      return "$" + symbolName;
    }

    protected String translate(String symbolName) {
      return getSymbol(symbolName);
    }
  }

Well, I think it’s pretty clear that each of these functions is doing one thing. I’m not sure how I’d extract anything further from any of them.

Perhaps you think this is taking things too far. I used to think so too. But after programming for over 40+ years, I’m beginning to come to the conclusion that this level of extraction is not taking things too far at all. In fact, to me, it looks just about right.

So, my advice: Extract till you just can’t extract any more. Extract till you drop.

After all, with modern tools it takes very little time. It makes each function almost trivial. The code reads very nicely. It forces you to put little snippets of code into nicely named functions. And, well gosh, extracting till you drop is kind of fun!

As the tests get more specific, the code gets more generic.

Uncle Bob — Thu, 06 Aug 2009 14:23:00 -0500

I tweeted this not too long ago. The basic idea is that as you add tests, the tests get more and more specific. This makes sense since tests are, after all, specifications. The more specifications you have, the more specific the whole body of specifications becomes.

As a general rule, good design dictates that the more specific your requirements become, the more general your code needs to be. This is saying roughly the same thing as Greenspun’s Tenth Rule of Programming: “Any sufficiently complicated [...] program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp.” Or rather, as more and more constraints pile upon a program, the designers look for ways to push those constraints out of the program itself and into the data.

In return for my tweet people asked for examples.

One of the better examples (though perhaps a bit trivial) is the Prime Factors Kata. This lovely little experiment grows and grows as you add test cases, and then suddenly collapses into an elegant three line algorithm.

The tests continue to become ever more specific. The production code starts out just as specific as the tests. But with the second or third test the programmer must make a decision. He can write the production code to mirror the tests (i.e. writing it as an if/else statement that detects which test is running and supplying the expected answer) or he can come up with some kind of more general algorithm that satisfies the tests without looking anything like them.

The algorithm grows and warps and twists; and then, just when it looks like it’s destined to become a wretched mess; it simply evaporates into a lovely little three line nested loop.

We see the principle at work in other ways as well. Often the programmers have a whole list of tests that they know must pass. As they write them one by one, they write the production code that satisfies them. Then, as in the Bowling Game Kata the tests start to pass unexpectedly. You were done with the code, and you weren’t aware of it. You continue writing tests, expecting one to fail, but they all pass. The test code grows, but the production code remains the same.

Sometimes this happens in a less surprising way. Sometimes you know that you have implemented an algorithm that will pass all remaining tests, but you write those tests anyway because they are part of the specification

The point is that test code and production code do not grow at the same rate. Indeed, as the application increases in complexity, the test code grows at a rate that is faster than the production code. Sometimes the production code actually shrinks as the test code grows because the programmers moved a load of functionality out of the code and into the data.

Consider FitNesse. A year ago there were 45,000 lines of code, of which 15,000 were tests, so 33% of the total were tests.

Now Fitnesse is 58,000 lines of code of which 26,000 are tests. We added 13,000 lines of code overall, but 8,000 (61%), are tests! The tests have grown to over 44% of the total.

Uncle Bob, JSPS: Learning Clojure

Uncle Bob — Sun, 19 Jul 2009 21:10:00 -0500

(JSPS: Just Some Poor Schmuck)

I’m trying to learn Clojure, and I’m finding the effort challenging. Perhaps you can help me.

Programming in Clojure (A derivative of Lisp) requires a certain “inside-out” thinking. I’m finding the combination of the shift in thought process and TDD quite difficult to resolve. So I thought I’d ask for your help.

Below is the Bowling Game written in Clojure complete with unit tests. I’m not at all pleased with the result. The tests, especially, look bizzare and backwards to me.

Please post your own solutions to this so that we can all compare and contrast them.

Tests

(ns bowling-game)
(use 'clojure.contrib.test-is)
(use 'bowling-game)

(defn roll-list [game list]
  (if (empty? list)
    game
    (roll-list (roll game (first list)) (rest list))
    ))

(defn roll-many [game n pins]
  (loop [i n g game]
    (if (zero? i)
      g
      (recur (dec i) (roll g pins)))))

(defn gutter-game [game] (roll-many game 20 0))

(deftest can-create-game
  (is (not (nil? (new-game)))))

(deftest gutter-game-should-score-0
  (is (= 0 (score (gutter-game (new-game))))))

(deftest all-ones-should-score-20
  (is (= 20 (score (roll-many (new-game) 20 1)))))

(deftest one-spare
  (is (= 16 (score
    (roll-many
      (roll-list (new-game) [5 5 3])
      17 0)))))

(deftest one_strike
  (is (= 24 (score
    (roll-many
      (roll-list (new-game) [10 3 4])
      16 0)))))

(deftest perfect-game
  (is (= 300 (score (roll-many (new-game) 12 10)))))

(run-tests 'bowling-game)

Code

(ns bowling-game)
(defn new-game [] [])

(defn next-two-balls [rolls]
  (+ (rolls 0) (rolls 1)))

(defn score-strike [rolls]
  [1 (+ 10 (+ (rolls 1) (rolls 2)))])

(defn score-spare [rolls]
  [2 (+ 10 (rolls 2))])

(defn score-no-mark [rolls]
  [2 (next-two-balls rolls)])

(defn score-next-frame [rolls]
  (if (= 10 (first rolls))
    (score-strike rolls)
    (if (= 10 (next-two-balls rolls))
      (score-spare rolls)
      (score-no-mark rolls))))

(defn score [game]
  (loop [frame 1 rolls game score 0]
    (if (> frame 10)
      score
      (let [frame-score (score-next-frame rolls)]
        (recur (inc frame) (subvec rolls (frame-score 0)) (+ score (frame-score 1)))))))

(defn roll [game pins]
  (conj game pins))

The Rush

Uncle Bob — Fri, 26 Jun 2009 08:40:13 -0500

There’s nothing like the feeling of achievement when you get a complex software system working. It’s the feeling of the hunter making a hard fought kill. By the sheer power of your intellect you have imposed your will upon inanimate nature, and forced it to do your bidding. You feel the rush of power, the satisfaction of victory. You are the master!

That’s the good news.

The bad news is that you’ve made a mess of things along the way.

This is inevitable. It takes a great deal of focus and endurance to get a system working just right. While you are consumed by that focus, you have little room for niceties like small functions, nice names, decoupling, and cleanliness.

My apprentice and I just finished achieving a major milestone in FitNesse. It used to be that each new release of FitNesse was contained in a zip file that had all the files and directories laid out just perfectly. This is great for a brand new installation, but doesn’t work so well when you are upgrading. What we managed to get working last night was the ability for FitNesse to self-install from it’s jar file. Now, whether you are installing for the first time, or just upgrading, FitNesse will install itself into your environment appropriately.

If you’ve ever written a self-install like this, you know there are lots of niggling little details to attend to. While the problem is conceptually simple, the realization is a maze of complexity.

Last night, with dinner waiting on the table and getting cold, we got the last little problem licked, and saw the whole self-installation work as we wanted. We congratulated each other with a high-five, and then went upstairs to eat a well-deserved meal. Later that evening I went down to the office and checked in the code.

This morning I woke up, and while in the shower, I realized that we had a lot of work left to do. We need to go back over all that code and clean it up. I’m sure we left a swath of detritus while on the hunt.

The rush of victory should never be confused with the cold and crystalline concept of done. Once you get your systems to work, you still have to go back and clean up the wreckage left behind by the victorious battle. You are not done until the victorious code has been cleaned, polished, and oiled.

Clean Code and Battle Scarred Architecture

Uncle Bob — Wed, 20 May 2009 17:35:00 -0500

If you go here you’ll see that I struggle to keep the CRAP out of FitNesse. Despite the whimsical name of this metric (which I take a perverse delight in), I find it to be remarkably useful.

CRAP is a metric that is applied to each function in the system. The formula for CRAP is: CRAP = comp(f)^2. X (1 – cov(f)/100)^3. + comp(f)

Where comp(f) = the cyclomatic complexity of the function f. and cov(f) = the unit test coverage of function f.

So a function’s CRAP will be small iff the cyclomatic complexity is low and the test coverage is high. CRAP will be huge if cyclomatic complexity is high, and there is no coverage.

What does this have to do with architecture? Read on…

I work very hard to keep the ratio of crappy methods near .1%. Of the 5643 methods in FitNesse, only 6 are crappy, and five of those I have no control over.

If you study the graph you can see how quickly I react to even the slightest uptick in crap. I don’t tolerate it because it means that either I’ve got a horrifically complex method that needs to be refactored, or (and this is far more likely) I’ve got a method that isn’t sufficiently tested.

Why am I so fastidious about this? Why am I so concerned about keeping the crap out of FitNesse? The reason is pretty simple. It’s the least I can do.

If you look inside of FitNesse, you’ll find that there are lots of structures and decisions that don’t seem to make a lot of sense at first reading. There are complexities and abstractions that will leave you shaking your head.

For example. We generate all our HTML in code. Yes, you read that correctly. We write Java code that constructs HTML. And yes, that means we are slinging angle brackets around.

To be fair, we’ve managed to move most of the angle bracket slingers into a single module that hides the HTML construction behind an abstraction barrier. This helps a lot, but cripe who would sling angle brackets when template system are so prevalent? I hope nobody. But FitNesse was not conceived at a time when template systems made sense (at least to us).

Fear not, I am working through the Fitnesse code replacing the HTML generation with Velocity templates. It’ll take some time, but I’ll get it done. The point is, that just like every other software system you’ve seen, FitNesse is a collection of historical compromises. The architecture shows the scars of many decisions that have since had to be reversed or deeply modified.

What does this have to do with CRAP? Simply this. The battle scarred architecture is something that will never really go away. I can stop the bleeding, and disinfect the wounds, but there will always be evidence of the battle.

That scarring makes the system hard to understand. It complicates the job of adding features and fixing bugs. It decreases the effectiveness of the developers who work on FitNesse. And though I work hard to massage the scars and bandage the wounds, the war goes on.

But I can keep the CRAP out! I can keep the code so clean and simple at the micro level, that the poor folks who try to make sense out of the macro scale aren’t impeded by huge deeply nested functions that aren’t tested!

Think of it this way. CRAP is disease at the cellular level. CRAP is a rot so pervasive that it can infest every nook and cranny of the system. My system may have scars, but it’s not diseased! In fact, despite the evidence of battles long past, the FitNesse code is very healthy.

And I aim to keep it that way.

Crap Code Inevitable? Rumblings from ACCU.

Uncle Bob — Thu, 23 Apr 2009 04:56:22 -0500

I gave the opening Keynote at ACCU 2009 on Wednesday. It was entitled: The Birth of Craftsmanship. Nicolai Josuttis finshed the day with the closing keynote: Welcome Crappy Code – The Death of Code Quality. It was like a blow to the gut.

In my keynote I attempted to show the historical trajectory that has led to the emergence of the software craftsmanship movement. My argument was that since the business practices of SCRUM have been widely adopted, and since teams who follow those practices but do not follow the technical practices of XP experience a relentless decrease in velocity, and since that decrease in velocity is exposed by the transparency of scrum, then if follows that the eventual adoption of those technical XP practices is virtually assured. My conclusion was that Craftsmanship was the “next big thing” (tm) that would capture the attention of our industry for the next few years, driven by the business need to increase velocity. (See Martin Fowler’s blog on Flaccid Scrum) In short, we are on a trajectory towards a higher degree of professionalism and craftsmanship.

Nicolai’s thesis was the exact opposite of mine. His argument was that we are all ruled by marketing and that businesses will do whatever it takes to cut costs and increase revenue, and therefore businesses will drive software quality inexorably downward. He stipulated that this will necessarily create a crisis as the defect rates and deadline slips increased, but that all attempts to improve quality would be short lived and followed by a larger drive to decrease quality even further.

Josuttis’ talk was an hour of highly depressing rhetoric couched in articulate delivery and brilliant humor. One of the more memorable moments came when he playacted how a manger would respond to a developer’s plea to let them write clean code like Uncle Bob says. The manager replies: “I don’t care what Uncle Bob says, and if you don’t like it you can leave and take Uncle Bob with you.”

One of the funnier moments came when Josuttis came up with his rules for crap code, one of which was “Praise Copy and Paste”. Here he showed the evolution of a module from the viewpoint of clean code, and then from the viewpoint of copy-paste. His conclusion, delivered with a lovely irony, was the the copy-paste solution was more maintainable because it was clear which code belonged to which version.

It was at this point that I thought that this whole talk was a ribald joke, an elaborate spoof. I predicted that he was about to turn the tables on everyone and ringingly endorse the craftsmanship movement.

Alas, it was not so. In the end he said that he was serious about his claims, and that he was convinced that crap code would dominate our future. And then he gave his closing plea which went like this:

We finally accepted that requirements change, and so we invented Agile.

We must finally accept that code will be crap and so we must ???

He left the question marks on the screen and closed the talk.

This was like a blow to the gut. The mood of the conference changed, at least for me, from a high of enthralled geekery, to depths of hoplessness and feelings of futile striving against the inevitable. Our cause was lost. Defeat was imminent. There was no hope.

Bulls Bollocks!

To his credit, there are a few things that Josuttis got right. There is a lot of crap code out there. And there is a growing cohort of crappy coders writing that crap code.

But the solution to that is not to give up and become one of them. The solution to that is to design our systems so that they don’t require an army of maintainers slinging code. Instead we need to design our systems such that the vast majority of changes can be implemented in DSLs that are tuned to business needs, and do not require “programmers” to maintain.

The thing that Josuttis got completely wrong, in my mildly arrogant opinion, is the notion that low quality code is cheaper than high quality code. Low quality code is not cheaper; it is vastly more expensive, even in the short term. Bad code slows everyone down from the minute that it is written. It creates a continuous and copious drag on further progress. It requires armies of coders to overcome that drag; and those armies must grow exponentially to maintain constant velocity against that drag.

This strikes at the very heart of Josuttis’ argument. His claim that crappy code is inevitable is based on the notion that crappy code is cheaper than clean code, and that therefore businesses will demand the crap every time. But it has generally not been business that has demanded crappy code. Rather it has been developers who mistakenly thought that the business’ need for speed meant that they had to produce crappy code. Once we, as professional developers, realize that the only way to go fast is to create clean and well designed code, then we will see the business’ need for speed as a demand for high quality code.

My vision of the future is quite different from Josuttis’. I see software developers working together to create a discipline of craftsmanship, professionalism, and quality similar to the way that doctors, lawyers, architects, and many other professionals and artisans have done. I see a future where team velocities increase while development costs decrease because of the steadily increasing skill of the teams. I see a future where large software systems are engineered by relatively small teams of craftsmen, and are configured and customized by business people using DSLs tuned to their needs.

I see a future of Clean Code, Craftsmanship, Professionalism, and an overriding imperative for Code Quality.

Is the Supremacy of Object-Oriented Programming Over?

Dean Wampler — Mon, 20 Apr 2009 21:45:00 -0500

I never expected to see this. When I started my career, Object-Oriented Programming (OOP) was going mainstream. For many problems, it was and still is a natural way to modularize an application. It grew to (mostly) rule the world. Now it seems that the supremacy of objects may be coming to an end, of sorts.

I say this because of recent trends in our industry and my hands-on experience with many enterprise and Internet applications, mostly at client sites. You might be thinking that I’m referring to the mainstream breakout of Functional Programming (FP), which is happening right now. The killer app for FP is concurrency. We’ve all heard that more and more applications must be concurrent these days (which doesn’t necessarily mean multithreaded). When we remove side effects from functions and disallow mutable variables, our concurrency issues largely go away. The success of the Actor model of concurrency, as used to great effect in Erlang, is one example of a functional-style approach. The rise of map-reduce computations is another example of a functional technique going mainstream. A related phenomenon is the emergence of key-value store databases, like BigTable and CouchDB, is a reaction to the overhead of SQL databases, when the performance cost of the Relational Model isn’t justified. These databases are typically managed with functional techniques, like map-reduce.

But actually, I’m thinking of something else. Hybrid languages like Scala, F#, and OCaml have demonstrated that OOP and FP can complement each other. In a given context, they let you use the idioms that make the most sense for your particular needs. For example, immutable “objects” and functional-style pattern matching is a killer combination.

What’s really got me thinking that objects are losing their supremacy is a very mundane problem. It’s a problem that isn’t new, but like concurrency, it just seems to grow worse and worse.

The problem is that there is never a stable, clear object model in applications any more. What constitutes a BankAccount or Customer or whatever is fluid. It changes with each iteration. It’s different from one subsystem to another even within the same iteration! I see a lot of misfit object models that try to be all things to all people, so they are bloated and the teams that own them can’t be agile. The other extreme is “balkanization”, where each subsystem has its own model. We tend to think the latter case is bad. However, is lean and mean, but non-standard, worse than bloated, yet standardized?

The fact is, for a lot of these applications, it’s just data. The ceremony of object wrappers doesn’t carry its weight. Just put the data in a hash map (or a list if you don’t need the bits “labeled”) and then process the collection with your iterate, map, and reduce functions. This may sound heretical, but how much Java code could you delete today if you replaced it with a stored procedure?

These alternatives won’t work for all situations, of course. Sometimes polymorphism carries its weight. Unfortunately, it’s too tempting to use objects as if more is always better, like cow bell.

So what would replace objects for supremacy? Well, my point is really that there is no one true way. We’ve led ourselves down the wrong path. Or, to be more precise, we followed a single, very good path, but we didn’t know when to take a different path.

Increasingly, the best, most nimble designs I see use objects with a light touch; shallow hierarchies, small objects that try to obey the Single Responsibility Principle, composition rather than inheritance, etc. Coupled with a liberal use of functional idioms (like iterate, map, and reduce), these designs strike the right balance between the protection of data hiding vs. openness for easy processing. By the way, you can build these designs in almost any of our popular languages. Some languages make this easier than others, of course.

Despite the hype, I think Domain-Specific Languages (DSLs) are also very important and worth mentioning in this context. (Language-Oriented Programming – LOP – generalizes these ideas). It’s true that people drink the DSL Kool-Aid and create a mess. However, when used appropriately, DSLs reduce a program to its essential complexity, while hiding and modularizing the accidental complexity of the implementation. When it becomes easy to write a user story in code, we won’t obsess as much over the details of a BankAccount as they change from one story to another. We will embrace more flexible data persistence models, too.

Back to OOP and FP, I see the potential for their combination to lead to a rebirth of the old vision of software components, but that’s a topic for another blog post.

Let's Hear it for the Zealots!

Uncle Bob — Tue, 10 Mar 2009 16:58:00 -0500

In his March 1st, 2009 column in SDTimes, Andrew Binstock takes the “Zealots of Agile” to task for claiming that Agile is the one true way. He made the counter-argument that non-agile projects have succeeded, and that agile projects have failed. He implied that the attitudes of the agile zealots are blind to these facts.

What a load of dingoes kidneys!

There is a difference between a zealot, and a religious fanatic. A religious fanatic cannot envision themselves to be wrong. We in the agile community may indeed be zealots, but we know we can be wrong. We know that projects succeeded before agile, and we know that agile projects have failed. We know that there are other schools of thought that are just as valid as ours. Indeed, most of us expect (even hope!) that agile will be superseded by something better one day.

Some of you may remember my keynote address at the 2002 XPUniverse in Chicago. My son, Micah, and I got up in front of the group dressed in Jiu Jistsu garb, and commenced to do a martial arts demonstration.

After the demonstration I made the point that a student of Karate could have given an equally spellbinding, though quite different, show. A student of Tae Kwon Do would amaze us with still other wonderful moves. The morale, of course, was that there are many ways to get the job done. There are many equally valid schools of thought.

However, the knowledge that there are other valid schools of thought does not dilute the zeal a student has for the school of thought he has chosen to study. Do you think that a Jiu Jistsu master isn’t a zealot? You’d better believe he is! To be great, one must have zeal. To rail against zeal is to rail against the desire for greatness. It is zeal that makes you go the extra distance to become great. The Jiu Jitsu master may respect the Karate master, but his goal is to become great at Jiu Jitsu.

So I think it is healthy that agile proponents are vociferous advocates of their school of thought. They are like Karate students who are excited about their new skills, and want to teach those skills to others. If their zeal to excel can infect others with similar zeal, so much the better. We need people in this industry who want to excel.

In his column Andrew derided agile zealots for claiming that Agile was the one true path. I don’t know of any prominent agile advocate who has ever made that claim. Indeed, when Kent Beck and I sat down in 1999 to plan out the XP Immersion training, we were very concerned that we might turn the XP movement into a religion. We did not want to create the impression that XPers were the chosen people and that wisdom would die with us. So we were, in fact, very careful to avoid any hint of the “one true path” argument.

As evidence of this, consider the Snowbird meeting in 2001. That meeting was inclusive, not exclusive. We invited people from all over the spectrum to join us there. DSDM, FDD, Scrum, Crystal, MDA, & RUP were all represented; and the agile manifesto (Yes, Andrew, Manifesto![1]) was the result. That manifesto was a simple statement of values, not a claim to deific knowledge.

Am I an Agile Zealot? Damned right I am! Do I think Agile is the best way to get software done? Damned right I do! I wouldn’t be doing Agile if I didn’t think it was best. Do I think Agile is better than someone else’s school of thought? Let me put it this way: I’m quite happy to go head to head against someone who follows another school of thought. I believe I’d code him right into the ground! Isn’t that what any good student of his craft would think? But I also recognize that one day I’ll lose that competition, and that will be a great day for me because it will give me the opportunity to learn something new and better.

Andrew also mentioned that Kent Beck and I are so obsessed with our “evangelical fervor” that we: “endlessly test and endlessly refine and refactor code to the point that it becomes an all-consuming diversion that does not advance the project”. WTF? I’ve been making monthly releases of FitNesse, and Kent has been cranking out releases of JunitMax; neither of us has been so busy gilding our respective lilies that we are failing to move our respective projects forward. Look around Andrew, we’re shipping code over here brother.

As part of his rant against our “evangelical fervor” Andrew asked this rather snide question: “I broke up that code into a separate class and added seven methods for what benefit, again?” This would appear to be a dig against refactoring – or perhaps excess refactoring. Still the question has a simple answer. We would only break up a module into a class with seven methods if that module was big, ugly, and complex, and by breaking it up we made it simple and easy to understand and modify. It seems to me that leaving the module in a messy state is irresponsible and unprofessional.

Andrew complains that there is very little Agile experience with projects over six million lines of code. I’m not sure where he got that number from, perhaps he’s been watching Steve Austin re-runs. Actually, the majority of my 2008 consulting business has been with very large companies with products well over six million lines of code. Agile-in-the-Large is the name of the game in those companies!

Finally, Andrew says Michael Feathers is a “moderate” Agile proponent. This is the same Michael Feathers who wrote a book defining Legacy code as code without tests – hardly a moderate point of view. Michael is a master, not a moderate.

I think Andrew has fallen into a common trap. He sees the zeal of the agile proponents and mistakes it for religious fanaticism. Instead of respecting that zeal as the outward manifestation of the inward desire to excel, he rails against it as being myopic and exclusive and then views all the statements of the agile proponents through that clouded viewport. Once you have decided that someone is a religious fanatic, it is difficult to accept anything they say.

I think standing against religious fanaticism in the software community is a good thing. I also think that those of us who are zealous must constantly watch that we are not also becoming blind to our own weaknesses and exclusive in our outlook. But I could wish that those who fear religious fanaticism as much as I do would stop confusing it with professional zeal.

So here’s the bottom line. People who excel are, by definition, zealots. People who aren’t zealous, do not excel. So there’s nothing wrong with being a zealot; indeed, zeal is a very positive emotion. The trick to being a zealot is to be mindful that the day will eventually come when some other zealot from a different school of thought will code you into the ground. When that happens, you should thank him for showing you a better way.

¹ I kicked off the Snowbird meeting by challenging the group to write a “manifesto”. I chose the word “manifesto” because some years earlier I had read The Object Oriented Database System Manifesto and I thought it was a clever use of the word.

Quality: It's alive! It's ALIVE!

Uncle Bob — Sat, 07 Mar 2009 15:25:00 -0600

Recently James Bach wrote a compelling post entitled Quality is Dead. As much as I’d like to agree, something interesting has just happened that tempts me to believe in a rebirth.

Just when James finally declares the death of quality, along comes the Manifesto for Software Craftsmanship. This simple document that builds upon the four values declared in the Agile Manifesto went live late on Friday evening. Now, at 3PM on Saturday over 600 people have signed it!

What does this mean? I think it means that many of us have seen what James was talking about, and are tired of it. We don’t want our users posting blogs – as James did – crying that they HATE us. We want our users to be astonished by the elegance and simplicity of our creations. We want other software developers to look under the hood and be amazed and impressed at the clarity and simplicity they see.

We want to be proud of our work!

We want our quality back!

And so a few weeks ago some software developers gathered at the offices of 8th Light in Libertyville, Illinois, and began the process of creating this manifesto. There followed a very active and passionate discussion in the Software Craftsmanship email group. The end result was this manifesto.

I encourage all of you to read and sign the manifesto, join the email group, and help bring quality back to life in the software profession.

Fudge anyone?

Uncle Bob — Wed, 31 Dec 2008 16:01:00 -0600

Back in September, when I was just staring the Slim project, I made a crucial architectural decision. I made it dead wrong. And then life turned to fudge…

The issue was simple. Slim tables are, well, tables just like Fit tables are. How should I parse these tables? Fit parses HTML. FitNesse parses wiki text into HTML and delivers it to Fit. Where should Slim fit in all of this?

Keep in mind that Fit must parse HTML since it lives on the far side of the FitNesse/SUT boundary. Fit doesn’t have access to wiki text. Slim tables, on the other hand, live on the near side of the FitNesse/SUT boundary, and so have full access to wiki text, and the built in parsers that parse that text.

So it seemed to me that I had two options.

Follow the example of Fit and parse wiki text into HTML, and then have the Slim Tables parse the HTML in order to process the tables.
Take advantage of the built in wiki text parser inside of FitNesse and bypass HTML altogether.

I chose the latter of the two because the Parsing system of FitNesse is trivial to use. You just hand it a string of wiki text, and it hands you a nice little parse tree of wiki widgets. All I had to do was walk that parse three and process my tables. Voila!

This worked great! In a matter of hours I was making significant progress on processing Slim decision tables. Instead of worrying about parsing HTML and building my own parse tree, I could focus on the problems of translating tables into Slim directives and then using the return values from Slim to colorize the table.

Generating html was no problem since that’s what FitNesse does anyway. All I had to do was modify the elements of the parse tree and then simply tell the tree to convert itself to HTML. What a dream.

Or so it seemed. Although things started well, progress started to slow before the week was out. The problem was that the FitNesse parser is tuned to the esoteric needs of FitNesse. The parser makes choices that are perfectly fine if your goal is to generate HTML and pass it to Fit, but that aren’t quite so nice when you’re goal is to use the parse tree to process Slim tables. As a simple example, consider the problem of literals.

In FitNesse, any camel case phrase fits the pattern of a wiki word and will be turned into a wiki link. Sometimes, though, you want to use a camel case phrase and you don’t want it converted to a link. In that case you surround the phrase with literal marks as follows: !- FitNesse-!. Anything between literal marks is simply ignored by the parser and passed through to the end.

Indeed, things inside of literals are not even escaped for html! If you put hi into a wiki page, it will escape the text you’ll see hi on the screen instead of a bold “hi”. On the other hand, if you put !- hi-! on a page, then the HTML is left unescaped and a boldfaced “hi” will appear on the screen.

I’m telling you all of this because the devil is in the details—so bear with me a bit longer.

You know how the C and C++ languages have a preprocessor? This preprocessor handles all the #include and #define statements and then hands the resulting text off to the true compiler. Well, the wiki parser works the same way! Literals and !define variables are processed first, by a different pass of the parser. Then all the rest of the wiki widgets are parsed by the true parser. The reason that we had to do this is even more devilishly detailed; so I’ll spare you. Suffice it to say that the reasons we need that preprocessor are similar to the reasons that C and C++ need it.

What does the preprocessor do with a literal? It converts it into a special widget. That widget looks like this: !lit?23? What does that mean? It means replace me with the contents of literal #23. You see, when FitNesse sees !- FitNesse-! it replaces it with !lit?nn? and squirrels FitNesse away in literal slot nn. During the second pass, that !lit?nn? is replaced with the contents of literal slot nn. Simple, right?

OK, now back to SLIM table processing. There I was, in Norway, teaching a class during the day and coding Slim at night, and everything was going just great. And then, during one of my tests, I happened to put a literal into one of the test tables. This is perfectly normal, I didn’t want some camel case phrase turned into a link. But this perfectly normal gesture made a unit test fail for a very strange reason. I got this wonderful message from junit: expected MyPage but was !lit?3?

I knew exactly what this meant. It meant that the value MyPage had been squirreled away by the first pass of the parser. I also knew that I had utterly no reasonable way of getting at it. So I did the only thing any sane programmer would do. I wrote my own preprocessor and used it instead of the standard one. This was “safe” since in the end I simply reconvert all the colorized tables back into wiki text and let the normal parser render it into HTML.

It was a bit of work, but I got it done at one am on a cold Norwegian night. Tests passing, everything great!

Ah, but no. By writing my own preprocessor, I broke the !define variable processing – subtly. And when I found and fixed that I had re-broken literal processing – subtly.

If you were following my tweets at the time you saw me twitter ever more frantically about literals. It was the proverbial ball of Mercury. Every time I put my thumb on it¹ it would squirt away and take some new form.

I fell into a trap. I call it the fudge trap. It goes like this:

forever do {“I can make this work! Just one more little fudge right here!”}

I was in this trap for two months! I was making progress, and getting lots of new features to work, but I was also running into strange little quirks and unexpected bizarre behaviors caused by the fudging I was doing. So I’d fudge a little more and then keep working. But each little fudge added to the next until, eventually, I had a really nasty house of cards (or rather: pile of fudge) ready to topple every time I touched anything else. I started to fear my own code². It was time to stop!

I knew what I had to do. I had to go back to my original architectural decision and make it the other way. There was no escape from this. The FitNesse parser was too coupled to the wiki-ness, and there was no sane way to repurpose it for test table processing.

I dreaded this. It was such a big change. I had built so much code in my house of fudge. All of it would have to be changed or deleted. And, worse, I needed to write an HTML parser.

I was lamenting to Micah about this one day in late November, and he said: “Dad, there are HTML parsers out there you know.”.

Uh…

So I went to google and typed Html Parser. Duh. There they were. Lots and lots of shiny HTML parsers free for the using.

I picked one and started to fiddle with it. It was easy to use.

Now comes the good part. I had not been a complete dolt. Even when I was using the FitNesse parse tree, I ate my own dogfood and wrapped it in an abstract interface. No part of the Slim Table processing code actually knew that it was dealing with a FitNesse parse tree. It simply used the abstract interface to get its work done.

That meant that pulling out the wiki parser and putting in the HTML parser was a matter of re-implementing the abstract interface with the output of the new parser (which happened to be another parse tree!). This took me about a day.

There came a magic moment when I had both the wiki text version of the parser, and the HTML version of the parser working. I could switch back and forth between the two by changing one word in one module. When I got all my tests passing with both parsers, I knew I was done. And then the fun really began!

I deleted ever stitch of that wiki parse tree pile of fudge. I tore it loose and rent it asunder. It was gone, never to darken my door with it’s fudge again.

It took me a day. A day. And the result is 400 fewer lines of code, and a set of Slim tables that actually work the way they are supposed to.

Moral #1: “Fudge tastes good while you are eating it, but it makes you fat, slow, and dumb.”

Moral #2: “Eat the damned dog food. It’ll save your posterior from your own maladroit decisions.

¹ I do not recommend that you actually put your thumb on any Mercury. Never mind that I used to play with liquid Mercury as a child, sloshing it around from hand to hand and endlessly stirring it with my finger. Wiser heads than I have determined that merely being in the same room with liquid Mercury can cause severe brain damage, genetic corruption, and birth defects in your children, grandchildren, and pets.

² Fearing your own code is an indicator that you are headed for ruin. This fear is followed by self-loathing, project-loathing, career-loathing, divorce, infanticide, and finally chicken farming.

How to Guarantee That Your Software Will Suck

Uncle Bob — Mon, 08 Dec 2008 19:35:17 -0600

This blog is a quick comment about Justin Etheredge’s blog by the same name.

I thought the blog was good. Really. No, I did. It’s a pretty good blog. Honestly.

My problem with is is that it points the finger outwards. It’s as though software developers have no responsibility. The blog seems to suggest that projects fail because managers do dumb-ass things like not buying dual monitors, setting deadlines, and requiring documentation.

Reading a blog like Justin’s may make you feel like high-five-ing and doing a little touch-down jig. OK, fine. And, after all, there’s some truth to what Justin has to say. But there’s another side of the coin too. A pretty big side.

Your software will suck if you write it badly. Yes, you should have good tools. Yes, you should work under realistic schedules. Yes, you should have time for social interaction. But these aren’t the things that make software suck. YOU, make your software suck.

Can you write good software with just one monitor? Of course you can. It might not be ideal, but what is?

Can you write good software if the deadlines are unreasonable? Of course you can! The definition of an unreasonable deadline is a deadline you won’t make, so you might as well make the code as good as it can be in the time you’ve got. If we’ve learned anything in the last 50 years it’s that rushing won’t get you to the deadline faster.

Can you write good software if you also have to write documentation? Can you write good software if your machine isn’t top-of-the-line? Can you write good software while standing on your head under water? (er, well, I’ll give you that might be tough, but for all the others:) Of course you can!

Don’t get me wrong. I think short-shrifting on tools, monitors, and schedules is stupid. I think Justin’s points are all valid. But the burden doesn’t fall solely upon management. We also have to do our jobs well.

Marick's Law

Uncle Bob — Sat, 29 Nov 2008 14:12:00 -0600

A month ago I was deep in the throes of shipping the current release of FitNesse. I just wanted to get it done. I was close to delivery when I spotted a subtle flaw. To fix this flaw I decided to insert identical if statements into each of 9 implementations of an abstract function.

My green wrist band was glowing a nasty shade of puke. I knew I was duplicating code. I knew that I should use the Template Method pattern. But that just seemed too hard. I was convinced that it would be faster to spew the duplicated code out into the derivatives, get the release done, and then clean it up later.

So this morning I was doing something else, and I spotted this duplicated code. I sighed, as I looked down at my green wrist band, and thought to myself that I’d better eat my own dog food and clean this mess up before it gets any worse. I was dreading it.

I made sure that every occurrence of the statement was identical. Then I went to the base class with the intention of refactoring a Template Method. When, what to my wondering eyes should appear, but a Template Method that was already there.

I sheepishly copied and pasted the if statement from one of the derivatives into the Template Method, and then deleted the other eight instances.

I ran the tests.

They all passed.

Damn.

My green wrist band is shouting: “I TOLD YOU SO!”

For my penance I did 20 recitations of Marick’s law. “When it comes to code it never pays to rush.”

The Truth about BDD

Uncle Bob — Thu, 27 Nov 2008 16:25:38 -0600

I really like the concept of BDD (Behavior Driven Development). I think Dan North is brilliant, and had done us all a great service by presenting the concept.

OK, you can “feel” the “but” coming, can’t you?

It’s not so much a “but” as an “aha!”. (The punch line is at the end of this article, so don’t give up in the middle.)

BDD is a variation on TDD. Whereas in TDD we drive the development of a module by “first” stating the requirements as unit tests, in BDD we drive that development by first stating the requirements as, well, requirements. The form of those requirements is fairly rigid, allowing them to be interpreted by a tool that can execute them in a manner that is similar to unit tests.

For example,

  GIVEN an employee named Bob making $12 per hour.
  WHEN Bob works 40 hours in one week;
  THEN Bob will be paid $480 on Friday evening.

The Given/When/Then convention is central to the notion of BDD. It connects the human concept of cause and effect, to the software concept of input/process/output. With enough formality, a tool can be (and has been) written that interprets the intent of the requirement and then drives the system under test (SUT) to ensure that the requirement works as stated.

The argued benefit is that the language you use affects the way you think (See this. and so if you use a language closer to the way humans think about problems, you’ll get better thought processes and therefore better results.

To say this differently, the Given/When/Then convention stimulates better thought processes than the AssertEquals(expected, actual); convention.

But enough of the overview. This isn’t what I wanted to talk about. What struck me the other day was this…

The Given/When/Then syntax of BDD seemed eerily familiar when I first heard about it several years ago. It’s been tickling at the back of my brain since then. Something about that triplet was trying to resonate with something else in my brain.

Then yesterday I realized that Given/When/Then is very similar to If/And/Then; a convention that I have used for the last 20+ years to read state transition tables.

Consider my old standard state transition table: The Subway Turnstile:

Current State	Event	New State	Action
LOCKED	COIN	UNLOCKED	Unlock
LOCKED	PASS	LOCKED	Alarm
UNLOCKED	COIN	UNLOCKED	Thankyou
UNLOCKED	PASS	LOCKED	Lock

I read this as a set of If/And/Then sentences as follows:

If we are in the LOCKED state, and we get a COIN event, then we go to the UNLOCKED state, and we invoke the Unlock action.
If we are in the LOCKED state, and we get a PASS event, then we stay in the UNLOCKED state, and we invoke the Alarm action.
etc.

This strange similarity caused me to realize that GIVEN/WHEN/THEN is simply a state transition, and that BDD is really just a way to describe a finite state machine. Clearly “GIVEN” is the current state of the system to be explored. “WHEN” describes an event or stimulus to that system. “THEN” describes the resulting state of the system. GIVEN/WHEN/THEN is nothing more than a description of a state transition, and the sum of all the GIVEN/WHEN/THEN statement is nothing more than a Finite State Machine.

Perhaps if I rephrase this you might see why I think this is a bit more than a ho-hum.

Some of the brightest minds in our industry, people like Dan North, Dave Astels, David Chelimsky, Aslak Hellesoy, and a host of others, have been pursuing the notion that if we use a better language to describe automated requirements, we will improve the way we think about those requirements, and therefore write better requirements. The better language that they have chosen and used for the last several years uses the Given/When/Then form which, as we have seen, is a description of a finite state machine. And so it all comes back to Turing. It all comes back to marks on a tape. We’ve decided that the best way to describe the requirements of a system is to describe it as a turing machine.

OK, perhaps I overdid the irony there. Clearly we don’t need to resort to marks on a tape. But still there is a grand irony in all this. The massive churning of neurons struggling with this problem over years and decades have reconnected the circle to the thoughts of that brave pioneer from the 1940s.

But enough of irony. Is this useful? I think it may be. You see, one of the great benefits of describing a problem as a Finite State Machine (FSM) is that you can complete the logic of the problem. That is, if you can enumerate the states and the events, then you know that the number of paths through the system is no larger than S * E. Or, rather, there are no more than S*E transitions from one state to another. More importantly, enumerating them is simply a matter of creating a transition for every combination of state and event.

One of the more persistent problems in BDD (and TDD for that matter) is knowing when you are done. That is, how do you know that you have written enough scenarios (tests). Perhaps there is some condition that you have forgotten to explore, some pathway through the system that you have not described.

This problem is precisely the kind of problem that FSMs are very good at resolving. If you can enumerate the states, and events, then you know the number of paths though the system. So if Given/When/Then statements are truly nothing more than state transitios, all we need to do is enumerate the number of GIVENs and the number of WHENs. The number of scenarios will simply be the product of the two.

To my knowledge, (which is clearly inadequate) this is not something we’ve ever tried before at the level of a business requirements document. But even if we have, the BDD mindset may make it easier to apply. Indeed, if we can formally enumerate all the Givens and Whens, then a tool could determine whether our requirements document has executed every path, and could find those paths that we had missed.

So, in conclusion, TDD has led us on an interesting path. TDD was adopted as a way to help us phrase low level requirements and drive the development of software based on those requirements. BDD, a variation of TDD, was created to help us think better about higher level requirements, and drive the development of systems using a language better than unit tests. But BDD is really a variation of Finite State Machine specifications, and FSMs can be shown, mathematically, to be complete. Therefore, we may have a way to conclusively demonstrate that our requirements are complete and consistent. (Apologies to Godel).

In the end, the BDDers may have been right that language improves the way we think about things. Certainly in my silly case, it was the language of BDD that resonated with the language of FSM.

Dirty Rotten ScrumDrels

Uncle Bob — Sun, 16 Nov 2008 06:50:14 -0600

So we’ve finally found the answer. We know who’s to blame. It’s SCRUM! SCRUM is the reason that the agile movement is failing. SCRUM is the reason that agile teams are making a mess. SCRUM is the root cause behind all the problems and disasters. SCRUM is the cause of the “Decline and Fall of Agile.”

Yeah, and I’ve got a bridge to sell you.

Scrum is not the problem. Scrum never was the problem. Scrum never will be the problem. The problem, dear craftsmen, is our own laziness.

It makes no sense to blame Scrum for the fact that we don’t write tests and don’t keep our code clean. We can’t blame scrum for technical debt. Technical debt was around long before there was scrum, and it’ll be around long after. No it’s not scrum that’s to blame. The culprits remain the same as they ever were: us.

Of course it’s true that a two day certification course is neither necessary nor sufficient to create a good software leader. It’s also true that the certificate you get for attending a CSM course is good for little more than showing that you paid to attend a two day CSM course. It’s also true that scrum leaves a lot to be desired when it comes to engineering practices. But it is neither the purpose of scrum nor of CSMs to make engineers out of us, or to instill the disciplines of craftsmanship within us. That’s our job!

Some have implied that if all those scrum teams had adopted XP instead of scrum, they wouldn’t be having all the technical debt they are seeing. BALDERDASH!

Let me be more precise. ASININE, INANE, ABSURDITY. BALONEY. DINGOES KIDNEYS.

Let me tell you all, here, now, and forevermore, it is quite possible to do XP badly. It’s easy to build technical debt while going through the motions of TDD. It’s a no brainer to create a wasteland of code with your pair partner. And, believe me, you can such a Simple Design that You Aint Gonna Maintain It. And I’m not speaking metaphorically.

Do you want to know the real secret behind writing good software? Do you want to know the process that will keep your code clean? Do you want the magic bullet, the secret sauce, the once and for all one and only truth?

OK, here it is. Are you ready? The secret is…

The secret is…

Do a good job.

Oh, yeah, and stop blaming everything (and everybody) else for your own laziness.

Upcoming Speaking Engagements

Dean Wampler — Mon, 03 Nov 2008 18:13:00 -0600

I’m speaking this Friday at RubyConf on Better Ruby Through Functional Programming. I’ll introduce long-overlooked ideas from FP, why they are important for Ruby programmers, and how to use them in Ruby.

In two weeks, I’m speaking on Wednesday, 11/19 at QCon San Francisco on Radical Simplification Through Polyglot and Poly-paradigm Programming. The idea of this talk is that combining the right languages and modularity paradigms (i.e., objects, functions, aspects) can simplify your code base and reduce the amount of code you have to write and manage, providing numerous benefits.

Back in Chicago, I’m speaking at the Polyglot Programmer’s meeting on The Seductions of Scala, 11/13. It’s an intro to the Scala language, which could become the language of choice for the JVM. I’m repeating this talk at the Chicago Java User’s Group on 12/16. I’m co-writing a book on Scala with Alex Payne. O’Reilly will be the publisher.

Incidently, Bob Martin is also speaking in Chicago on 11/13 at the APLN Chicago meeting on Software Professionalism.

A Scala-style "with" Construct for Ruby

Dean Wampler — Mon, 29 Sep 2008 22:41:00 -0500

Scala has a “mixin” construct called traits, which are roughly analogous to Ruby modules. They allow you to create reusable, modular bits of state and behavior and use them to compose classes and other traits or modules.

The syntax for using Scala traits is quite elegant. It’s straightforward to implement the same syntax in Ruby and doing so has a few useful advantages.

For example, here is a Scala example that uses a trait to trace calls to a Worker.work method.

    
// run with "scala example.scala" 

class Worker {
    def work() = "work" 
}

trait WorkerTracer extends Worker {
    override def work() = "Before, " + super.work() + ", After" 
}

val worker = new Worker with WorkerTracer

println(worker.work())        // => Before, work, After

Note that WorkerTracer extends Worker so it can override the work method. Since Scala is statically typed, you can’t just define an override method and call super unless the compiler knows there really is a “super” method!

Here’s a Ruby equivalent.

    
# run with "ruby example.rb" 

module WorkerTracer
    def work; "Before, #{super}, After"; end
end

class Worker 
    def work; "work"; end
end

class TracedWorker < Worker 
  include WorkerTracer
end

worker = TracedWorker.new

puts worker.work          # => Before, work, After

Note that we have to create a subclass, which isn’t required for the Scala case (but can be done when desired).

If you know that you will always want to trace calls to work in the Ruby case, you might be tempted to dispense with the subclass and just add include WorkerTracer in Worker. Unfortunately, this won’t work. Due to the way that Ruby resolves methods, the version of work in the module will not be found before the version defined in Worker itself. Hence the subclass seems to be the only option.

However, we can work around this using metaprogramming. We can use WorkerTracer#append_features(...). What goes in the argument list? If we pass Worker, then all instances of Worker will be effected, but actually we’ll still have the problem with the method resolution rules.

If we just want to affect one object and work around the method resolution roles, then we need to pass the singleton class (or eigenclass or metaclass ...) for the object, which you can get with the following expression.

    
metaclass = class << worker; self; end

So, to encapsulate all this and to get back to the original goal of implementing with-style semantics, here is an implementation that adds a with method to Object, wrapped in an rspec example.

    
# run with "spec ruby_with_spec.rb" 

require 'rubygems'
require 'spec'

# Warning, monkeypatching Object, especially with a name
# that might be commonly used is fraught with peril!!

class Object
  def with *modules
    metaclass = class << self; self; end
    modules.flatten.each do |m|
      m.send :append_features, metaclass
    end
    self
  end
end

module WorkerTracer
    def work; "Before, #{super}, After"; end
end

module WorkerTracer1
    def work; "Before1, #{super}, After1"; end
end

class Worker 
    def work; "work"; end
end

describe "Object#with" do
  it "should make no changes to an object if no modules are specified" do
    worker = Worker.new.with
    worker.work.should == "work" 
  end

  it "should override any methods with a module's methods of the same name" do
    worker = Worker.new.with WorkerTracer
    worker.work.should == "Before, work, After" 
  end

  it "should stack overrides for multiple modules" do
    worker = Worker.new.with(WorkerTracer).with(WorkerTracer1)
    worker.work.should == "Before1, Before, work, After, After1" 
  end

  it "should stack overrides for a list of modules" do
    worker = Worker.new.with WorkerTracer, WorkerTracer1
    worker.work.should == "Before1, Before, work, After, After1" 
  end

  it "should stack overrides for an array of modules" do
    worker = Worker.new.with [WorkerTracer, WorkerTracer1]
    worker.work.should == "Before1, Before, work, After, After1" 
  end
end

You should carefully consider the warning about monkeypatching Object! Also, note that Module.append_features is actually private, so I had to use m.send :append_features, ... instead.

The syntax is reasonably intuitive and it eliminates the need for an explicit subclass. You can pass a single module, or a list or array of them. Because with returns the object, you can also chain with calls.

A final note; many developers steer clear of metaprogramming and reflection features in their languages, out of fear. While prudence is definitely wise, the power of these tools can dramatically accelerate your productivity. Metaprogramming is just programming. Every developer should master it.

Traits vs. Aspects in Scala

Dean Wampler — Sat, 27 Sep 2008 22:33:00 -0500

Scala traits provide a mixin composition mechanism that has been missing in Java. Roughly speaking, you can think of traits as analogous to Java interfaces, but with implementations.

Aspects, e.g., those written in AspectJ, are another mechanism for mixin composition in Java. How do aspects and traits compare?

Let’s look at an example trait first, then re-implement the same behavior using an AspectJ aspect, and finally compare the two approaches.

Observing with Traits

In a previous post on Scala, I gave an example of the Observer Pattern implemented using a trait. Chris Shorrock and James Iry provided improved versions in the comments. I’ll use James’ example here.

To keep things as simple as possible, let’s observe a simple Counter, which increments an internal count variable by the number input to an add method.

    
package example

class Counter {
    var count = 0
    def add(i: Int) = count += i
}

The count field is actually public, but I will only write to it through add.

Here is James’ Subject trait that implements the Observer Pattern.

    
package example

trait Subject {
  type Observer = { def receiveUpdate(subject:Any) }

  private var observers = List[Observer]()
  def addObserver(observer:Observer) = observers ::= observer
  def notifyObservers = observers foreach (_.receiveUpdate(this))
}

Effectively, this says that we can use any object as an Observer as long as it matches the structural type { def receiveUpdate(subject:Any) }. Think of structural types as anonymous interfaces. Here, a valid observer is one that has a receiveUpdate method taking an argument of Any type.

The rest of the trait manages a list of observers and defines a notifyObservers method. The expression observers ::= observer uses the List :: (“cons”) operator to prepend an item to the list. (Note, I am using the default immutable List, so a new copy is created everytime.)

The notifyObservers method iterates through the observers, calling receiveUpdate on each one. The _ that gets replaced with each observer during the iteration.

Finally, here is a specs file that exercises the code.

    
package example

import org.specs._

object CounterObserverSpec extends Specification {
    "A Counter Observer" should {
        "observe counter increments" in {
            class CounterObserver {
                var updates = 0
                def receiveUpdate(subject:Any) = updates += 1
            }
            class WatchedCounter extends Counter with Subject {
                override def add(i: Int) = { 
                    super.add(i)
                    notifyObservers
                }
            }
            var watchedCounter = new WatchedCounter
            var counterObserver = new CounterObserver
            watchedCounter.addObserver(counterObserver)
            for (i <- 1 to 3) watchedCounter.add(i)
            counterObserver.updates must_== 3
            watchedCounter.count must_== 6
    }
  }
}

The specs library is a BDD tool inspired by rspec in Rubyland.

I won’t discuss it all the specs-specific details here, but hopefully you’ll get the general idea of what it’s doing.

Inside the "observe counter increments" in {...}, I start by declaring two classes, CounterObserver and WatchedCounter. CounterObserver satisfies our required structural type, i.e., it provides a receiveUpdate method.

WatchedCounter subclasses Counter and mixes in the Subject trait. It overrides the add method, where it calls Counter’s add first, then notifies the observers. No parentheses are used in the invocation of notifyObservers because the method was not defined to take any!

Next, I create an instance of each class, add the observer to the WatchedCounter, and make 3 calls to watchedCounter.add.

Finally, I use the “actual must_== expected” idiom to test the results. The observer should have seen 3 updates, while the counter should have a total of 6.

The following simple bash shell script will build and run the code.

    
SCALA_HOME=...
SCALA_SPECS_HOME=...
CP=$SCALA_HOME/lib/scala-library.jar:$SCALA_SPECS_HOME/specs-1.3.1.jar:bin
rm -rf bin
mkdir -p bin
scalac -d bin -cp $CP src/example/*.scala
scala -cp $CP example/CounterObserverSpec

Note that I put all the sources in a src/example directory. Also, I’m using v1.3.1 of specs, as well as v2.7.1 of Scala. You should get the following output.

    
Specification "CounterObserverSpec" 
  A Counter Observer should
  + observe counter increments

Total for specification "CounterObserverSpec":
Finished in 0 second, 60 ms
1 example, 2 assertions, 0 failure, 0 error

Observing with Aspects

Because Scala compiles to Java byte code, I can use AspectJ to advice Scala code! For this to work, you have to be aware of how Scala represents its concepts in byte code. For example, object declarations, e.g., object Foo {...} become static final classes. Also, method names like + become $plus in byte code.

However, most Scala type, method, and variable names can be used as is in AspectJ. This is true for my example.

Here is an aspect that observes calls to Counter.add.

    
package example

public aspect CounterObserver {
    after(Object counter, int value): 
        call(void *.add(int)) && target(counter) && args(value) {

        RecordedObservations.record("adding "+value);
    }
}

You can read this aspect as follows, after calling Counter.add (and keeping track of the Counter object that was called, and the value passed to the method), call the static method record on the RecordedObservations.

I’m using a separate Scala object RecordedObservations

    
package example

object RecordedObservations {
    private var messages = List[String]()
    def record(message: String):Unit = messages ::= message
    def count() = messages.length
    def reset():Unit = messages = Nil
}

Recall that this is effectively a static final Java class. I need this separate object, rather than keeping information in the aspect itself, because of the simple-minded way I’m building the code. ;) However, it’s generally a good idea with aspects to delegate most of the work to Java or Scala code anyway.

Now, the “spec” file is:

    
package example

import org.specs._

object CounterObserverSpec extends Specification {
    "A Counter Observer" should {
        "observe counter increments" in {
            RecordedObservations.reset()
            var counter = new Counter
            for (i <- 1 to 3) counter.add(i)
            RecordedObservations.count() must_== 3
            counter.count must_== 6
    }
  }
}

This time, I don’t need two more classes for the adding a mixin trait or defining an observer. Also, I call RecordedObservations.count to ensure it was called 3 times.

The build script is also slightly different to add the AspectJ compilation.

    
SCALA_HOME=...
SCALA_SPECS_HOME=...
ASPECTJ_HOME=...
CP=$SCALA_HOME/lib/scala-library.jar:$SCALA_SPECS_HOME/specs-1.3.1.jar:$ASPECTJ_HOME/lib/aspectjrt.jar:bin
rm -rf bin app.jar
mkdir -p bin
scalac -d bin -cp $CP src/example/*.scala 
ajc -1.5 -outjar app.jar -cp $CP -inpath bin src/example/CounterObserver.aj
aj -cp $ASPECTJ_HOME/lib/aspectjweaver.jar:app.jar:$CP example.CounterObserverSpec

The ajc command not only compiles the aspect, but it “weaves” into the compiled Scala classes in the bin directory. Actually, it only affects the Counter class. Then it writes all the woven and unmodified class files to app.jar, which is used to execute the test. Note that for production use, you might prefer load-time weaving.

The output is the same as before (except for the milliseconds), so I won’t show it here.

Comparing Traits with Aspects

So far, both approaches are equally viable. The traits approach obviously doesn’t require a separate language and corresponding tool set.

However, traits have one important limitation with respect to aspects. Aspects let you define pointcuts that are queries over all possible points where new behavior or modifications might be desired. These points are called join points in aspect terminology. The aspect I showed above has a simple pointcut that selects one join point, calls to the Counter.add method.

However, what if I wanted to observe all state changes in all classes in a package? Defining traits for each case would be tedious and error prone, since it would be easy to overlook some cases. With an aspect framework like AspectJ, I can implement observation at all the points I care about in a modular way.

Aspect frameworks support this by providing wildcard mechanisms. I won’t go into the details here, but the * in the previous aspect is an example, matching any type. Also, one of the most powerful techniques for writing robust aspects is to use pointcuts that reference only annotations, a form of abstraction. As a final example, if I add an annotation Adder to Counter.add,

    
package example

class Counter {
    var count = 0
    @Adder def add(i: Int) = count += i
}

Then I can rewrite the aspect as follows.

    
package example

public aspect CounterObserver {
    after(Object counter, int value): 
        call(@Adder void *.*(int)) && target(counter) && args(value) {

        RecordedObservations.record("adding "+value);
    }
}

Now, there are no type and method names in the pointcut. Any instance method on any visible type that takes one int (or Scala Int) argument and is annotated with Adder will get matched.

Note: Scala requires that you create any custom annotations as normal Java annotations. Also, if you intend to use them with Aspects, use runtime retention policy, which will be necessary if you use load-time weaving.

Conclusion

If you need to mix in behavior in a specific, relatively-localized set of classes, Scala traits are probably all you need and you don’t need another language. If you need more “pervasive” modifications (e.g., tracing, policy enforcement, security), consider using aspects.

Acknowledgements

Thanks to Ramnivas Laddad, whose forthcoming 2^nd Edition of AspectJ in Action got me thinking about this topic.

Type Scum

Michael Feathers — Mon, 08 Sep 2008 17:43:00 -0500

Getting existing code under test is hard work, but it is fruitful. Yes, you get code that is easier to change, but more importantly, you get knowledge – you learn things about programming which make you better at avoiding common traps. Sadly, many of these traps aren’t well recognized yet.

The trap that I am going to write about today is one that I call type scum. It’s most prevalent in C and C++ but it can attack in any of the traditional statically typed languages.

Type Scum is the cruft in a code base which makes it impossible to compile a single file without an entire sub-stratum of defined types. I’m not talking about the primary, or even the secondary abstractions in your system, but rather the 200 or so basic types and structs that your abstractions depend upon.

Again, the problem is worst in C++ and C. At some point, every C or C++ developer feels the urge to isolate him or herself from the basic types of the language. The unsigned int type becomes uint32 and unsigned char * becomes uchar16_ptr. And, if that was all, it would be okay. But no, people define data transfer objects which aggregate these type pseudonyms together into large muddles. No file can compile without bringing in a world of types which cushion the code from dangerous things like the platform and testing harnesses.

No wait, testing harnesses are good. What can we do?

The unfortunate thing is that it is very hard to pull type scum out of a system once it’s been infected, but we can learn how to avoid it or at least manage it better:

If you must provide a sub-stratum of basic types in your system, do it in one place. There should be a single library (and associated headers) that you include whenever you need it. This library (and headers) should contain nothing else.
If you must create DTO (Data Transfer Object) types, minimize them. A good general purpose structure can carry a wide variety of different types of data and simplify testing.
Push the DTOs to the edges. There are some systems where you really do care whether an internal computation happens in unsigned long int or unsigned long long int but they are rare. Basic data types and tolerances matter when two systems need to agree upon them, and that happens at component boundaries. In many systems, the internal code can use platform types directly.

There you go. Type scum bad.

I’m sure that some people reading this will say “Hey, isn’t this the exact opposite of the advice that people give for a system with the primitive obsession code smell?” The answer is “yes.” But, to me, primitive obsession is a different problem. It’s something which is the result of a lack of real behavioral abstractions in a system, not the lack of larger data holders.

Different problem.

Type scum bad.

The Open-Closed Principle for Languages with Open Classes

Dean Wampler — Thu, 04 Sep 2008 21:42:00 -0500

We’ve been having a discussion inside Object Mentor World Design Headquarters about the meaning of the OCP for dynamic languages, like Ruby, with open classes.

For example, in Ruby it’s normal to define a class or module, e.g.,

    
# foo.rb
class Foo
    def method1 *args
        ...
    end
end

and later re-open the class and add (or redefine) methods,

    
# foo2.rb
class Foo
    def method2 *args
        ...
    end
end

Users of Foo see all the methods, as if Foo had one definition.

    
foo = Foo.new
foo.method1 :arg1, :arg2
foo.method2 :arg1, :arg2

Do open classes violate the Open-Closed Principle? Bertrand Meyer articulated OCP. Here is his definition¹.

Software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification.

He elaborated on it here.

... This is the open-closed principle, which in my opinion is one of the central innovations of object technology: the ability to use a software component as it is, while retaining the possibility of adding to it later through inheritance. Unlike the records or structures of other approaches, a class of object technology is both closed and open: closed because we can start using it for other components (its clients); open because we can at any time add new properties without invalidating its existing clients.

Tell Less, Say More: The Power of Implicitness

So, if one client require’s only foo.rb and only uses method1, that client doesn’t care what foo2.rb does. However, if the client also require’s foo2.rb, perhaps indirectly through another require, problems will ensue unless the client is unaffected by what foo2.rb does. This looks a lot like the way “good” inheritance should behave.

So, the answer is no, we aren’t violating OCP, as long as we extend a re-opened class following the same rules we would use when inheriting from it.

If we use inheritance instead:

    
# foo.rb
class Foo
    def method1 *args
        ...
    end
end
...
class DerivedFoo < Foo
    def method2 *args
        ...
    end
end
...
foo = SubFoo.new    # Instantiate different class...
foo.method1 :arg1, :arg2
foo.method2 :arg1, :arg2

One notable difference is that we have to instantiate a different class. This is an important difference. While you can often just use inheritance, and maybe you should prefer it, inheritance only works if you have full control over what types get instantiated and it’s easy to change which types you use. Of course, inheritance is also the best approach when you need all behavioral variants simulateneously, i.e., each variant in one or more objects.

Sometimes you want to affect the behavior of all instances transparently, without changing the types that are instantiated. A slightly better example, logging method calls, illustrates the point. Here we use the “famous” alias_method in Ruby.

    
# foo.rb
class Foo
    def method1 *args
        ...
    end
end
# logging_foo.rb
class Foo
    alias_method :old_method1, :method1
    def method1 *args
        p "Inside method1(#{args.inspect})" 
        old_method1 *args
    end
end
...
foo = Foo.new
foo.method1 :arg1, :arg2

Foo.method1 behaves like a subclass override, with extended behavior that still obeys the Liskov-Substitution Principle (LSP).

So, I think the OCP can be reworded slightly.

Software entities (classes, modules, functions, etc.) should be open for extension, but closed for source modification.

We should not re-open the original source, but adding functionality through a separate source file is okay.

Actually, I prefer a slightly different wording.

Software entities (classes, modules, functions, etc.) should be open for extension, but closed for source and contract modification.

The extra and contract is redundant with LSP. I don’t think this kind of redundancy is necessarily bad. ;) The contract is the set of behavioral expectations between the “entity” and its client(s). Just as it is bad to break the contract with inheritance, it is also bad to break it through open classes.

OCP and LSP together are our most important design principles for effective organization of similar vs. variant behaviors. Inheritance is one way we do this. Open classes provide another way. Aspects provide a third way and are subject to the same design issues.

¹ Meyer, Bertrand (1988). Object-Oriented Software Construction. Prentice Hall. ISBN 0136290493.

There will be code

Uncle Bob — Thu, 28 Aug 2008 21:02:45 -0500

During the last three decades, several things about software development have changed, and several other things have not. The things that have changed are startling. The things that have not are even more startling.

What has changed? Three decades have seen a 1000 fold increase in speed, another 1000 fold increase in memory. Yet another 1000 fold decrease in size (by volume), and yet another 1000 fold decrease in power consumption. Adding up all those zeros implies that the resources we have to play with have increased by twelve orders of magnitude. Even if I have over estimated by five orders of magnitude the remaining seven are still and astounding increase.

I remember building RSX-11M on a PDP-11/60 from source. It took several hours. Nowadays I can build huge java applications in a matter of seconds. I remember that compiling small C programs required dozens of minutes. Now much larger programs compile in a eyeblink. I remember painstakingly editing assembly language on punch cards. Now I use refactorings in huge java programs without thinking about it. I remember when 10,000 lines of code was five boxes of cards that weighed 50 pounds. Now, such a program is considered trivial.

Nowadays we have tools! We have editors that compile our code while we type, and complete our thoughts for us. We have analyzers that will find code duplication in huge systems, and identify flaws and weaknesses. We have code coverage tools that will tell us each line of code that our unit tests fail to execute. We have refactoring browsers that allow us to manipulate our code with unprecedented power and convenience.

But in the face of all this massive change, this rampant growth, this almost unlimited wealth of resources, there is something that hasn’t changed much at all. Code.

Fortran, Algol, and Lisp are over fifty years old. These language are the clear progenitors of the static and dynamic languages we use today. The roots of C++, Java, and C# clearly lie in Algol and Fortan. The connection between Lisp, and Ruby, Python, and smalltalk may be less obvious, but only slightly so. Today’s modern language may be rich with features and power, but they are not 12 orders of magnitude better than their ancestors. Indeed, it’s hard to say that they are even ONE order of magnitude better.

When it comes down to it. We still write programs made out of calculations, ‘if’ statements, and ‘for’ loops. We still assign values into variables and pass arguments into functions. Programmers from 30 years ago might be surprised that we use lower case letters in our programs, but little else would startle them about the code we write.

We are like carpenters who started out using hammers and saws, and have progressed to using air-hammers and power saws. These power tools help a lot; but in the end we are still cutting wood and nailing it together. And we probably will be for the next 30 years.

Looking back we see that what we do hasn’t changed all that much. What has changed are the tools and resources we can apply to the task. Looking forward I anticipate that the current trend will continue. The tools will get better, but the code will still be code. We may see some “minor” improvements in languages and frameworks, but we will still be slinging code.

Some folks have put a great deal of hope in technologies like MDA. I don’t. The reason is that I don’t see MDA as anything more than a different kind of computer language. To be effective it will still need ‘if’ and ‘for’ statements of some kind. And ‘programmers’ will still need to write programs in that language, because details will still need to be managed. There is no language that can eliminate the programming step, because the programming step is the translation from requirements to systems irrespective of language. MDA does not change this.

Some folks have speculated that we’ll have “intelligent agents” based on some kind of AI technology, and that these agents will be able to write portions of our programs for us. The problem with this is that we already have intelligent agents that write programs for us. They are called programmers. It’s difficult to imagine a program that is able to communicate to a customer and write a program better than a human programmer.

So, for the foreseeable future I think software will remain the art of crafting code to meet the requirements of our customers.

There is something else that needs to change, however. And I believe it is changing. Our professionalism.

In some ways our “profession” has paralleled that of medicine. 300 years ago there were a few thinkers, and far too many practitioners. There were no standards, no common rituals or behaviors, no common disciplines. If you got sick you might go to a barber, or a healer. Perhaps he’d let out some blood, or ask you to wear some garlic. Over time, however, the few thinkers gained knowledge, discipline, and skill. They adopted standards and rituals. They set up a system for policing and maintaining those standards, and for training and accepting new members.

THIS is the change that I hope the next thirty years holds for software development.

Baubles in Orbit

Uncle Bob — Tue, 19 Aug 2008 01:29:32 -0500

I have put together a nice little demonstration of the Bauble concept. You may recall that I first wrote about it here. Baubles are a simple component scheme for Ruby, good for when you want a component, but don’t need something as heavy as a gem.

orbit.zip contains all the files for this demonstration. I suggest you download and unpack it.

First you need to install the Bauble gem. Don’t worry, it won’t hurt anything. Just say gem install orbit/bauble/Bauble-0.1.gem (You’ll probably have to do it with sudo.) That should install Bauble. From now on you only need to say require 'bauble' in your ruby scripts that make use of it.

Now you should be able to run the orbital simulator. Just type:


cd orbit/MultipleBodyOrbit/lib
jruby multiple_body_orbit.rb

A swing window should pop up and you should be able to watch an orbital simulation. Every run shows a different random scenario, so you can kill a lot of time by watching worlds in collision.

The thing to note, if you are a ruby programmer, is the use of the term Bauble::use(-some_directory-). If you look in the multiple_body_orbit.rb file you’ll see I use two Baubles, the Physics bauble does the raw calculation for all the gravity, forces, collisions, etc. The cellular_automaton bauble provides a very simple Swing framework for drawing dots on a screen. (Yes, this is jruby).

If you look in either of the two Baubles, you’ll see that the require statements within them do not know (or care) about the directory they live in. There is none of that horrible __FILE__ nonsense that pollutes so many ruby scripts. This is because the Bauble::use function puts the directory path in the LOAD_PATH so that subsquent require statements can simply eliminate the directory spec.

Take a look at the Bauble source code. It’s no great shakes.

Also take a look at the two baubles. They show a pretty nice way to decouple business rules from gui. You might recognize the MVP pattern. The multiple_body_orbit.rb file contains the presenter. Clearly the Physics module is the model. And the cellular_automaton module is the view. (There is no controller, because there is no input.)

Quintessence: The fifth element for the Agile Manifesto

Uncle Bob — Thu, 14 Aug 2008 19:01:00 -0500

“Quintessence” was the name of the keynote I gave at the banquet of Agile 2008. Though the talk covered a lot of ground, the upshot was that we need an addition to the agile manifesto…

As you know the Agile Manifesto is composed of four balanced value statements. Here they are:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

In my talk I proposed the following addition

Craftsmanship over Crap.

From this you can probably tell that my talk was primarily about behaving professionally, and writing clean code. This should not be a big surprise since I just finished writing a book entitled Clean Code .

The problem with my proposal is that it is not a balanced value statement. In the other four statements we value the second item. We just value the first item more. But in my proposed addition, we simply don’t value crap at all.

So I hereby change my original proposal, which was made for dramatic effect, to:

Craftsmanship over Execution

Most software development teams execute, but they don’t take care. We value execution, but we value craftsmanship more.

Comments? Anyone have a better word?

(BTW, I used a wonderful web tool named Visual Thesaurus to find the word “execute”. Check it out!)

The Seductions of Scala, Part II - Functional Programming

Dean Wampler — Tue, 05 Aug 2008 20:32:00 -0500

A Functional Programming Language for the JVM

In my last blog post, I discussed Scala’s support for OOP and general improvements compared to Java. In this post, which I’m posting from Agile 2008, I discuss Scala’s support for functional programming (FP) and why it should be of interest to OO developers.

A Brief Overview of Functional Programming

You might ask, don’t most programming languages have functions? FP uses the term in the mathematical sense of the word. I hate to bring up bad memories, but you might recall from your school days that when you solved a function like

    
y = sin(x)

for y, given a value of x, you could input the same value of x an arbitrary number of times and you would get the same value of y. This means that sin(x) has no side effects. In other words, unlike our imperative OO or procedural code, no global or object state gets changed. All the work that a mathematical function does has to be returned in the result.

Similarly, the idea of a variable is a little different than what we’re used to in imperative code. While the value of y will vary with the value of x, once you have fixed x, you have also fixed y. The implication for FP is that “variables” are immutable; once assigned, they cannot be changed. I’ll call such immutable variables value objects.

Now, it would actually be hard for a “pure” FP language to have no side effects, ever. I/O would be rather difficult, for example, since the state of the input or output stream changes with each operation. So, in practice, all “pure” FP languages provide some mechanisms for breaking the rules in a controlled way.

Functions are first-class objects in FP. You can create named or anonymous functions (e.g., closures or blocks), assign them to variables, pass them as arguments to other functions, etc. Java doesn’t support this. You have to create objects that wrap the methods you want to invoke.

Functional programs tend to be much more declarative in nature than imperative programs. This is perhaps more obvious in pure FP languages, like Erlang and Haskell, than it is in Scala.

For example, the definition of Fibonacci numbers is the following.

    
F(n) = F(n-1) + F(n-2) where F(1)=1 and F(2)=1

An here is a complete implementation in Haskell.

    
module Main where 
-- Function f returns the n'th Fibonacci number. 
-- It uses binary recursion. 
f n | n <= 2 = 1 
    | n >  2 = f (n-1) + f (n-2)

Without understanding the intricacies of Haskell syntax, you can see that the code closely matches the “specification” above it. The f n | ... syntax defines the function f taking an argument n and the two cases of n values are shown on separate lines, where one case is for n <= 2 and the other case if for n > 2.

The code uses the recursive relationship between different values of the function and the special-case values when n = 1 and n = 2. The Haskell runtime does the rest of the work.

It’s interesting that most domain-specific languages are also declarative in nature. Think of how JMock, EasyMock or Rails’ ActiveRecord code look. The code is more succinct and it lets the “system” do most of the heavy lifting.

Functional Programming’s Benefits for You

Value Objects and Side-Effect Free Functions

It’s the immutable variables and side-effect free functions that help solve the multicore problem. Synchronized access to shared state is not required if there is no state to manage. This makes robust concurrent programs far easier to write.

I’ll discuss concurrency in Scala in my third post. For now, let’s discuss other ways that FP in Scala helps to improve code, concurrent or not.

Value objects are beneficial because you can pass one around without worrying that someone will change it in a way that breaks other users of the object. Value objects aren’t unique to FP, of course. They have been promoted in Domain Driven Design (DDD), for example.

Similarly, side-effect free functions are safer to use. There is less risk that a caller will change some state inappropriately. The caller doesn’t have to worry as much about calling a function. There are fewer surprises and everything of “consequence” that the function does is returned to the caller. It’s easier to keep to the Single Responsibility Principle when writing side-effect free functions.

Of course, you can write side-effect free methods and immutable variables in Java code, but it’s mostly a matter of discipline; the language doesn’t give you any enforcement mechanisms.

Scala gives you a helpful enforcement mechanism; the ability to declare variables as val’s (i.e., “values”) vs. var’s (i.e., “variables”, um… back to the imperative programming sense of the word…). In fact, val is the default, where neither is required by the language. Also, the Scala library contains both immutable and mutable collections and it “encourages” you to use the immutable collections.

However, because Scala combines both OOP and FP, it doesn’t force FP purity. The upside is that you get to use the approach that best fits the problem you’re trying to solve. It’s interesting that some of the Scala library classes expose FP-style interfaces, immutability and side-effect free functions, while using more traditional imperative code to implement them!

Closures and First-Class Functions

True to its functional side, Scala gives you true closures and first-class functions. If you’re a Groovy or Ruby programmer, you’re used to the following kind of code.

    
class ExpensiveResource {
    def open(worker: () => Unit) = {
        try {
            println("Doing expensive initialization")
            worker()
        } finally {
            close()
        }
    }
    def close() = {
        println("Doing expensive cleanup")
    }
}
// Example use:
try {
    (new ExpensiveResource()) open { () =>        // 1
        println("Using Resource")                 // 2
        throw new Exception("Thrown exception")   // 3
    }                                             // 4
} catch {
    case ex: Throwable => println("Exception caught: "+ex)
}

Running this code will yield:

    
Doing expensive initialization
Using Resource
Doing expensive cleanup
Exception caught: java.lang.Exception: Thrown exception

The ExpensiveResource.open method invokes the user-specified worker function. The syntax worker: () => Unit defines the worker parameter as a function that takes no arguments and returns nothing (recall that Unit is the equivalent of void).

ExpensiveResource.open handles the details of initializing the resource, invoking the worker, and doing the necessary cleanup.

The example marked with the comment // 1 creates a new ExpensiveResource, then calls open, passing it an anonymous function, called a function literal in Scala terminology. The function literal is of the form (arg_list_) => function body or () => println(...) ..., in our case.

A special syntax trick is used on this line; if a method takes one argument, you can change expressions of the form object.method(arg) to object method {arg}. This syntax is supported to allow user-defined methods to read like control structures (think for statements – see the next section). If you’re familiar with Ruby, the four commented lines read a lot like Ruby syntax for passing blocks to methods.

Idioms like this are very important. A library writer can encapsulate all complex, error-prone logic and allow the user to specify only the unique work required in a given situation. For example, How many times have you written code that opened an I/O stream or a database connection, used it, then cleaned up. How many times did you get the idiom wrong, especially the proper cleanup when an exception is thrown? First-class functions allow writers of I/O, database and other resource libraries to do the correct implementation once, eliminating user error and duplication. Here’s a rhetorical question I always ask myself:

How can I make it impossible for the user of this API to fail?

Iterations

Iteration through collections, Lists in particular, is even more common in FP than in imperative languages. Hence, iteration is highly evolved. Consider this example:

    
object RequireWordsStartingWithPrefix {
    def main(args: Array[String]) = {
        val prefix = args(0)
        for {
            i <- 1 to (args.length - 1)   // no semicolon
            if args(i).startsWith(prefix)
        } println("args("+i+"): "+args(i))
    }
}

Compiling this code with scalac and then running it on the command line with the command

    
scala RequireWordsStartingWithPrefix xx xy1 xx1 yy1 xx2 xy2

produces the result

    
args(2): xx1
args(5): xx2

The for loop assigns a loop variable i with each argument, but only if the if statement is true. Instead of curly braces, the for loop argument list could also be parenthesized, but then each line as shown would have to be separated by a semi-colon, like we’re used to seeing with Java for loops.

We can have an arbitrary number of assignments and conditionals. In fact, it’s quite common to filter lists:

    
object RequireWordsStartingWithPrefix2 {
    def main(args: Array[String]) = {
        val prefix = args(0)
        args.slice(1, args.length)
            .filter((arg: String) => arg.startsWith(prefix))
            .foreach((arg: String) => println("arg: "+arg))
    }
}

This version yields the same result. In this case, the args array is sliced (loping off the search prefix), the resulting array is filtered using a function literal and the filtered array is iterated over to print out the matching arguments, again using a function literal. This version of the algorithm should look familiar to Ruby programmers.

Rolling Your Own Function Objects

Scala still has to support the constraints of the JVM. As a comment to the first blog post said, the Scala compiler wraps closures and “bare” functions in Function objects. You can also make other objects behave like functions. If your object implements the apply method, that method will be invoked if you put parentheses with an matching argument list on the object, as in the following example.

    
class HelloFunction {
    def apply() = "hello" 
    def apply(name: String) = "hello "+name
}
val hello = new HelloFunction
println(hello())        // => "hello" 
println(hello("Dean"))  // => "hello Dean"

Option, None, Some…

Null pointer exceptions suck. You can still get them in Scala code, because Scala runs on the JVM and interoperates with Java libraries, but Scala offers a better way.

Typically, a reference might be null when there is nothing appropriate to assign to it. Following the conventions in some FP languages, Scala has an Option type with two subtypes, Some, which wraps a value, and None, which is used instead of null. The following example, which also demonstrates Scala’s Map support, shows these types in action.

    
val hotLangs = Map(
    "Scala" -> "Rocks", 
    "Haskell" -> "Ethereal", 
    "Java" -> null)
println(hotLangs.get("Scala"))          // => Some(Rocks)
println(hotLangs.get("Java"))           // => Some(null)
println(hotLangs.get("C++"))            // => None

Note that Map stores values in Options objects, as shown by the println statements.

By the way, those -> aren’t special operators; they’re methods. Like ::, valid method names aren’t limited to alphanumerics, _, and $.

Pattern Matching

The last FP feature I’ll discuss in this post is pattern matching, which is exploited more fully in FP languages than in imperative languages.

Using our previous definition of hotLangs, here’s how you might use matching.

    
def show(key: String) = {
    val value: Option[String] = hotLangs.get(key)
    value match {
        case Some(x) => x
        case None => "No hotness found" 
    }
}
println(show("Scala"))  // => "Rocks" 
println(show("Java"))   // => "null" 
println(show("C++"))    // => "No hotness found"

The first case statement, case Some(x) => x, says “if the value I’m matching against is a Some that could be constructed with the Some[+String](x: A) constructor, then return the x, the thing the Some contains.” Okay, there’s a lot going on here, so more background information is in order.

In Scala, like Ruby and other languages, the last value computed in a function is returned by it. Also, almost everything returns a value, including match statements, so when the Some(x) => x case is chosen, x is returned by the match and hence by the function.

Some is a generic class and the show function returns a String, so the match is to Some[+String]. The + in the +String expression is analogous to Java’s extends, i.e., <? extends String>. Capiche?

Idioms like case Some(x) => x are called extractors in Scala and are used a lot in Scala, as well as in FP, in general. Here’s another example using Lists and our friend ::, the “cons” operator.

    
def countScalas(list: List[String]): Int = {
    list match {
        case "Scala" :: tail => countScalas(tail) + 1
        case _ :: tail       => countScalas(tail)
        case Nil             => 0
    }
}
val langs = List("Scala", "Java", "C++", "Scala", "Python", "Ruby")
val count = countScalas(langs)
println(count)    // => 2

We’re counting the number of occurrences of “Scala” in a list of strings, using matching and recursion and no explicit iteration. An expression of the form head :: tail applied to a list returns the first element set as the head variable and the rest of the list set as the tail variable. In our case, the first case statement looks for the particular case where the head equals Scala. The second case matches all lists, except for the empty list (Nil). Since matches are eager, the first case will always pick out the List("Scala", ...) case first. Note that in the second case, we don’t actually care about the value, so we use the placeholder _. Both the first and second case’s call countScalas recursively.

Pattern matching like this is powerful, yet succinct and elegant. We’ll see more examples of matching in the next blog post on concurrency using message passing.

Recap of Scala’s Functional Programming

I’ve just touched the tip of the iceberg concerning functional programming (and I hope I got all the details right!). Hopefully, you can begin to see why we’ve overlooked FP for too long!

In my last post, I’ll wrap up with a look at Scala’s approach to concurrency, the Actor model of message passing.

Bauble, Bauble...

Uncle Bob — Sun, 20 Jul 2008 15:42:29 -0500

In Ruby, I hate require statements that look like this:


require File.dirname(__FILE__)+"myComponent/component.rb"

So I decided to do something about it.

This all started when my Son, Micah, told me about his Limelight project. Limelight is a jruby/swing GUI framework. If you want to build a fancy GUI in Ruby, consider this tool.

I have neither the time nor inclination to write a framework like this; but my curiosity was piqued. So in order to see what it was like to do Swing in JRuby I spent a few hours cobbling together an implementation of Langton’s Ant. This turned out to be quite simple.

The result, however, was a mess. There was swing code mixed up with “ant” code, in the classic GUI/Business-rule goulash that we “clean-coders” hate so much. Despite the fact that this was throw-away code, I could not leave it in that state – the moral outrage was just too great. So I spent some more time separating the program into two modules.

The first module knew all about Langton’s ant, but nothing about Swing. The second module was a tiny framework for implementing cellular automata in Swing. (Here are all the files).

I was quite happy with the separation, but did not like the horrible require statements that I had to use. The cellular_automaton component had two classes, in two separate files. In order to get the require right, I had to either use absolute directory paths, or the horrible File.dirname(__FILE__)... structure.

What I wanted was for cellular_automaton to behave like a gem. But I didn’t want to make it into a gem. Gem’s are kind of “heavy” for a dumb little thing like “cellular_automaton”.

So I created a module named “Bauble” which gave me some gem-like behaviors. Here it is:


module Bauble
  def self.use(bauble)
    bauble_name = File.basename(bauble)
    ensure_in_path "#{bauble}/lib" 
    require bauble_name
  end

  def self.ensure_in_path(path)
    $LOAD_PATH << path unless $LOAD_PATH.include? path
  end
end

This is no great shakes, but it solved my problem. Now, in my langton’s ant program all I need to do is this:


require 'bauble'
Bauble.use('../cellular_automaton')

All the ugly requires are gone.

I’m thinking about turning Bauble into a rubyforge project, and making a publicly available gem out of it in order to give folks a standard way to avoid those horrible __FILE__ requires. I think there are several other utilities that could be placed in Bauble such as require_relative etc.

Anyway, what do you think?

Now 'and' for something completely different.

Uncle Bob — Thu, 26 Jun 2008 13:16:00 -0500

My son Justin is working as a Ruby apprentice for my son Micah at 8th light. We were sitting at the kitchen table, and he showed me a function he was writing. In the midst of the function I saw this:

handle_batch(item) and display_batch(item) while items_remaining?

I looked hard at this and then I said: “Justin, I don’t think you understand what and does.

He said: “I think I do.” and he pointed me to a website which showed 21 Ruby tricks “you should be using in your own code.”

Trick #9 was the use of the and keyword to couple statements together to make “one liners”. It was billed as a trick that [cough] “more confident” Ruby programmers use.

So I got the pickaxe book out and looked up the and keyword. I showed Justin where it said that the second clause won’t be executed if the first clause is false (or nil!) So if handle_batch ever returned nil, then display_batch would never be called.

Warning bells were going off in my head. This was clearly an unintended use of the and keyword that could have rather nasty repercussions. I thought it was somewhat irresponsible for a website that boasted expertise in programming to tell people they should be using a dangerous stunt like this.

However, handle_batch did not return nil, so the and worked well enough in this case; and we had more to do. So I made my point to Justin and then we kept working, leaving the and in place.

An hour later we were making changes to a function. Suddenly a whole bunch of our tests broke. (You know what I’m going to say, don’t you?) The failure mode didn’t make any sense. Suddenly a whole bunch of processing simply wasn’t getting done.

It was Justin who said: “Wow, I’ll bet it’s that and. And it was. The change we had made had indirectly caused handle_batch to return a nil.

OK, this was a fun little story. You might think the moral is “don’t use and for one-liners” or “don’t trust websites that claim expertise”. Yes, those would be good conclusions to draw. But I’m concerned about something else.

The “and trick” is clever. In programming, clever != smart. (OK, sorry, that was clever…) Using a keyword like and in a manner for which it was not intended, and in a way that is not guaranteed to work in all cases, is risky. It bothers me.

By the same token it bothers me that so many Ruby programmers use the ||= trick for lazy initialization. I know it works. I know it’s a standard idiom. I’m not trying to stop people from doing it. Hell, I use it too because it’s become a standard idiom. But it bothers me nonetheless because it entered our vocabulary of idioms because it was clever trick; and clever little tricks have a way of turning into nasty little surprises.

Ruby is a fun and powerful language. But that doesn’t mean we should go out of our way to be clever. We shouldn’t be eager to adopt quirky little idioms and erudite little stunts just because they are cute, or neat, or nifty. Code is hard enough to understand without having to think sideways through the next novel application of the ?: operator.

Professionals write clear and clean code. They use their language well. They use their language efficiently. But they don’t aspire to be master tricksters. Rather, they prove their professionalism by writing code that needs no explanation.

So... You want your code to be maintainable.

Uncle Bob — Tue, 24 Jun 2008 23:07:46 -0500

We know that maintenance is 90% of the software lifecycle, and 90% of the cost. We know that our systems need to be flexible, reusable, and maintainable. Indeed, that’s why we spend so much of our time trying to get the design and architecture just right. Because we all know that good design and architecture is the key to flexibility, reusability, and maintainability…right?

Of course. Good design and architecture is what makes software easy to change. Good design and architecture separates the things that change for one reason from the things that change for another reason (The Single Responsibility Principle). Good design allows us to add new features without changing a lot of old code (Open Closed Principle). Good design makes sure that high level policy does not depend on low level detail (Dependency Inversion Principle), etc. etc.

So how do we get good design? Well, that’s tricky. Oh it’s not too tricky to get a good design in place at first. The tricky part is to keep the design good. That’s the problem, you see. It’s not that the design starts out so bad (although sometimes…) rather it is that the design degrades over time as the system changes.

Systems change. Often they change in ways that thwart the original intent of the design. Unfortunately, changing the design to align to these changes is hard. So we wind up hacking the new features into the system and thwarting the design. And that’s how even the best designed systems rot.

So how do we keep the design from rotting? How do we make sure we can migrate the design as the system changes? Simple. Tests.

When you have a suite of tests that covers >90% of the code in the system, you are not afraid to make changes. Every time you make a little change you run those tests, and you know that you have not broken anything. This gives you the confidence to make the next change, and the next, and the next. It gives you the confidence to change the design!

Nothing makes a system more flexible than a suite of tests. Nothing. Good architecture and design are important; but the affect of a robust suite of tests is an order of magnitude greater. It’s so much greater because those tests enable you to improve the design.

This can’t be overstated. If you want your systems to be flexible, write tests. If you want your systems to be reusable, write tests. If you want your systems to be maintainable, write tests.

And write your tests using the Three Laws of TDD.

What We Can Learn from the Ojibwe Language

Dean Wampler — Sat, 03 May 2008 09:48:57 -0500

Ojibwe (sometimes spelled Ojibwa; the last syllable is pronounced “way”) is one of the few Native American languages that isn’t immediately threatened with extinction. It is spoken by about 10,000 people around the Great Lakes region. Brothers David and Anton Treuer are helping to keep it alive, as they discussed in a recent Fresh Air interview.

Ojibwe is a language that is optimized for an aboriginal people whose lives and livelihoods depend on an intimate awareness of their environment, especially the weather and water conditions. They have many nouns and verbs for fine gradations of rain, snow, ice conditions, the way water waves look and sound, etc. You would want this clarity of detail if you ventured out on a lake every day to fish for dinner.

In the past, speaking languages like Ojibwe was actively suppressed by the government, in an attempt to assimilate Native Americans. Today, the threat of extinction is more from the sheer ubiquity of English. I think there is another force at play, too. People living a modern, so-called “developed” lifestyle just don’t need to be so aware of their environment anymore. In fact, most of us are pretty “tone deaf” to the nuances of weather and water, which is sad in a way. We just don’t perceive the need for the richness of an Ojibwe to communicate what’s important to us, like sports trivia and fashion tips.

So, what does Ojibwe have to do with programming languages? Our language choices inform the way we frame problem solving and design. I was reminded of this recently while reading Ted Neward’s series of articles on Scala. Scala is a JVM language that provides first-class support for functional programming and object-oriented design refinements like traits, which provide mixin behavior.

While you can write Java-like code in Scala, Neward demonstrates how exploiting Scala features can result in very different code for many problems. The Scala examples are simpler, but sometimes that simplicity only becomes apparent after you grasp the underlying design principle in use, like closures or functional idioms.

One of the best pieces of advice in the Pragmatic Programmer is to learn a new language every year. You should pick a language that is very different from what you know already, not one that is fundamentally similar. Even if you won’t use that language in your professional work, understanding its principles, patterns, and idioms will inform your work in whatever languages you actually use.

For example, there is a lot of fretting these days about concurrent programming, given the rise of multi-core CPUs and multiprocessor computers. We know how to write concurrent programs in our most popular imperative languages, like Java and C++, but that knowledge is somewhat specialized and not widely known in the community. This is the main reason that functional programming is suddenly interesting again. It is inherently easier to write concurrent applications using side-effect-free code. I expect that we will fail to meet the concurrency challenge if we rely exclusively on the mechanisms in our imperative languages.

So, you could adopt a functional language for all or part of your concurrent application. Or, if you can’t use Scala (or Haskell or Erlang or …) you could at least apply functional idioms, like side-effect-free functions, avoidance of mutable objects, etc. in your current imperative language. However, even that won’t be an option unless you understand those principles in the first place.

Learning a new language is more than learning a new vocabulary. It’s even more than learning new design techniques. It’s also learning to see common things from a fresh perspective, with greater clarity.

Clean Code. Whew!

Uncle Bob — Tue, 08 Apr 2008 00:10:16 -0500

I’ve been working on this book for several years now. After a flurry of effort (you might have noticed I’ve been quiet lately) I’m very pleased to say that I’m done with the writing and am preparing the manuscript for production. See The Prentice Hall Listing

Table of Contents

Clean Code    1
    There Will Be Code    1
    Bad Code    2
    The Total Cost of Owning a Mess.    3
    Schools of Thought.    11
    We are Authors.    12
    The Boy Scout Rule    13
    Prequel and Principles    14
    Conclusion    14
    Bibliography    15
Meaningful Names by Tim Ottinger    17
    Introduction    17
    Use Intention-revealing Names    17
    Avoid Disinformation    19
    Make Meaningful Distinctions    20
    Use Pronounceable Names    21
    Use Searchable Names    22
    Avoid Encodings    23
    Avoid Mental Mapping    24
    Class Names    25
    Method Names    25
    Don't Be Cute    25
    Pick One Word Per Concept    26
    Don't Pun    26
    Use Solution Domain Names    27
    Use Problem Domain Names    27
    Add Meaningful Context    27
    Don't add Gratuitous Context    29
    Final Words ...    30
Functions    31
    Small!    34
    Do one thing.    35
    One level of abstraction per function.    36
    Switch Statements.    37
    Use descriptive names.    39
    Function Arguments.    39
    Have no side-effects.    43
    Command Query Separation    44
    Prefer exceptions to returning error codes.    45
    Don't Repeat Yourself.    47
    Structured Programming    48
    How do you write functions like this?    48
    Conclusion    49
    SetupTeardownIncluder    49
    Bibliography    52
Comments    53
    Comments do not make up for bad code.    55
    Explain yourself in code.    55
    Good Comments    55
    Bad Comments    59
    Example    71
    Bibliography    74
Formatting    75
    The Purpose of Formatting    76
    Vertical Formatting    76
    Horizontal Formatting    84
    Team Rules    89
    Uncle Bob's Formatting Rules.    90
Objects and Data Structures    93
    Data Abstraction    93
    Data/Object anti-symmetry.    95
    The Law of Demeter    97
    Data Transfer Objects    99
    Conclusion    101
    Bibliography    101
Error Handling by Michael Feathers    103
    Use Exceptions Rather than Return Codes    103
    Write Your Try-Catch-Finally Statement First    105
    Use Unchecked Exceptions    106
    Provide Context with Exceptions    107
    Define Exception Classes In Terms of a Caller's Needs.    107
    Define the Normal Flow    109
    Don't Return Null    110
    Don't Pass Null    111
    Conclusion    112
    Bibliography    112
Boundaries by James Grenning    113
    Bibliography    119
Unit Tests    121
    The Three Laws of TDD    122
    Keeping Tests Clean    123
    Clean Tests    124
    One Assert per Test    129
    F.I.R.S.T.    132
    Conclusion    132
    Bibliography    133
Classes    135
    Class Organization    135
    Classes should be Small!    136
    Organizing for Change    146
    Bibliography    150
Systems    By Dean Wampler 151
    How would you build a city?    151
    Separate constructing a system from using it    152
    Scaling Up    155
    Java Proxies    158
    Pure Java AOP Frameworks    160
    AspectJ Aspects    163
    Test-drive the system architecture    164
    Optimize decision making    165
    Use standards wisely, when they add demonstrable value    165
    Systems need Domain-Specific Languages    166
    Conclusion    166
    Bibliography    167
Emergence By Jeff Langr    169
    Getting Clean via Emergent Design    169
    Simple Design Rule 1: Runs all the tests    170
    Simple Design Rules 2-4: Refactoring    170
    No Duplication    170
    Expressive    173
    Minimal Classes and Methods    174
    Conclusion    174
    Bibliography    174
Concurrency    by Brett Schuchert 175
    Why Concurrency?    176
    Challenges    177
    Concurrency Defense Principles    178
    Know Your Library    180
    Know Your Execution Models    181
    Beware Dependencies between Syncrhonized Methods    182
    Keep Synchronized Sections Small    183
    Writing Correct Shut-Down Code is Hard    183
    Testing Threaded Code    184
    Conclusion    188
    Bibliography    189
Successive Refinement    191
    Args Implementation    192
    Args: the rough draft.    198
    String Arguments    212
     Conclusion    246
JUnit Internals    249
    Conclusion    262
Refactoring SerialDate    263
    Conclusion    280
    Bibliography    281
Smells and Heuristics    283
    Comments    283
    Environment    284
    Functions    285
    General    285
    Java    304
    Names    306
    Tests    310
    Conclusion    311
    Bibliography    312
Concurrency II    by Brett Schuchert 313
    Client/Server Example    313
    Possible Paths of Execution    317
    Knowing Your Library    322
    Dependencies between methods can break concurrent code    325
    Increasing Throughput    329
    Deadlock    331
    Testing Multi-Threaded Code    335
    Tool Support for Testing Thread-Based Code    337
    Conclusion    338
    Tutorial: Full Code Examples    339
org.jfree.date.SerialDate    345
Cross References of Heuristics    406

Clean Code. Whew!

Uncle Bob — Tue, 08 Apr 2008 00:10:16 -0500

Table of Contents

Clean Code    1
    There Will Be Code    1
    Bad Code    2
    The Total Cost of Owning a Mess.    3
    Schools of Thought.    11
    We are Authors.    12
    The Boy Scout Rule    13
    Prequel and Principles    14
    Conclusion    14
    Bibliography    15
Meaningful Names by Tim Ottinger    17
    Introduction    17
    Use Intention-revealing Names    17
    Avoid Disinformation    19
    Make Meaningful Distinctions    20
    Use Pronounceable Names    21
    Use Searchable Names    22
    Avoid Encodings    23
    Avoid Mental Mapping    24
    Class Names    25
    Method Names    25
    Don't Be Cute    25
    Pick One Word Per Concept    26
    Don't Pun    26
    Use Solution Domain Names    27
    Use Problem Domain Names    27
    Add Meaningful Context    27
    Don't add Gratuitous Context    29
    Final Words ...    30
Functions    31
    Small!    34
    Do one thing.    35
    One level of abstraction per function.    36
    Switch Statements.    37
    Use descriptive names.    39
    Function Arguments.    39
    Have no side-effects.    43
    Command Query Separation    44
    Prefer exceptions to returning error codes.    45
    Don't Repeat Yourself.    47
    Structured Programming    48
    How do you write functions like this?    48
    Conclusion    49
    SetupTeardownIncluder    49
    Bibliography    52
Comments    53
    Comments do not make up for bad code.    55
    Explain yourself in code.    55
    Good Comments    55
    Bad Comments    59
    Example    71
    Bibliography    74
Formatting    75
    The Purpose of Formatting    76
    Vertical Formatting    76
    Horizontal Formatting    84
    Team Rules    89
    Uncle Bob's Formatting Rules.    90
Objects and Data Structures    93
    Data Abstraction    93
    Data/Object anti-symmetry.    95
    The Law of Demeter    97
    Data Transfer Objects    99
    Conclusion    101
    Bibliography    101
Error Handling by Michael Feathers    103
    Use Exceptions Rather than Return Codes    103
    Write Your Try-Catch-Finally Statement First    105
    Use Unchecked Exceptions    106
    Provide Context with Exceptions    107
    Define Exception Classes In Terms of a Caller's Needs.    107
    Define the Normal Flow    109
    Don't Return Null    110
    Don't Pass Null    111
    Conclusion    112
    Bibliography    112
Boundaries by James Grenning    113
    Bibliography    119
Unit Tests    121
    The Three Laws of TDD    122
    Keeping Tests Clean    123
    Clean Tests    124
    One Assert per Test    129
    F.I.R.S.T.    132
    Conclusion    132
    Bibliography    133
Classes    135
    Class Organization    135
    Classes should be Small!    136
    Organizing for Change    146
    Bibliography    150
Systems    By Dean Wampler 151
    How would you build a city?    151
    Separate constructing a system from using it    152
    Scaling Up    155
    Java Proxies    158
    Pure Java AOP Frameworks    160
    AspectJ Aspects    163
    Test-drive the system architecture    164
    Optimize decision making    165
    Use standards wisely, when they add demonstrable value    165
    Systems need Domain-Specific Languages    166
    Conclusion    166
    Bibliography    167
Emergence By Jeff Langr    169
    Getting Clean via Emergent Design    169
    Simple Design Rule 1: Runs all the tests    170
    Simple Design Rules 2-4: Refactoring    170
    No Duplication    170
    Expressive    173
    Minimal Classes and Methods    174
    Conclusion    174
    Bibliography    174
Concurrency    by Brett Schuchert 175
    Why Concurrency?    176
    Challenges    177
    Concurrency Defense Principles    178
    Know Your Library    180
    Know Your Execution Models    181
    Beware Dependencies between Syncrhonized Methods    182
    Keep Synchronized Sections Small    183
    Writing Correct Shut-Down Code is Hard    183
    Testing Threaded Code    184
    Conclusion    188
    Bibliography    189
Successive Refinement    191
    Args Implementation    192
    Args: the rough draft.    198
    String Arguments    212
     Conclusion    246
JUnit Internals    249
    Conclusion    262
Refactoring SerialDate    263
    Conclusion    280
    Bibliography    281
Smells and Heuristics    283
    Comments    283
    Environment    284
    Functions    285
    General    285
    Java    304
    Names    306
    Tests    310
    Conclusion    311
    Bibliography    312
Concurrency II    by Brett Schuchert 313
    Client/Server Example    313
    Possible Paths of Execution    317
    Knowing Your Library    322
    Dependencies between methods can break concurrent code    325
    Increasing Throughput    329
    Deadlock    331
    Testing Multi-Threaded Code    335
    Tool Support for Testing Thread-Based Code    337
    Conclusion    338
    Tutorial: Full Code Examples    339
org.jfree.date.SerialDate    345
Cross References of Heuristics    406

Clean Code. Whew!

Uncle Bob — Tue, 08 Apr 2008 00:10:16 -0500

Table of Contents

Clean Code    1
    There Will Be Code    1
    Bad Code    2
    The Total Cost of Owning a Mess.    3
    Schools of Thought.    11
    We are Authors.    12
    The Boy Scout Rule    13
    Prequel and Principles    14
    Conclusion    14
    Bibliography    15
Meaningful Names by Tim Ottinger    17
    Introduction    17
    Use Intention-revealing Names    17
    Avoid Disinformation    19
    Make Meaningful Distinctions    20
    Use Pronounceable Names    21
    Use Searchable Names    22
    Avoid Encodings    23
    Avoid Mental Mapping    24
    Class Names    25
    Method Names    25
    Don't Be Cute    25
    Pick One Word Per Concept    26
    Don't Pun    26
    Use Solution Domain Names    27
    Use Problem Domain Names    27
    Add Meaningful Context    27
    Don't add Gratuitous Context    29
    Final Words ...    30
Functions    31
    Small!    34
    Do one thing.    35
    One level of abstraction per function.    36
    Switch Statements.    37
    Use descriptive names.    39
    Function Arguments.    39
    Have no side-effects.    43
    Command Query Separation    44
    Prefer exceptions to returning error codes.    45
    Don't Repeat Yourself.    47
    Structured Programming    48
    How do you write functions like this?    48
    Conclusion    49
    SetupTeardownIncluder    49
    Bibliography    52
Comments    53
    Comments do not make up for bad code.    55
    Explain yourself in code.    55
    Good Comments    55
    Bad Comments    59
    Example    71
    Bibliography    74
Formatting    75
    The Purpose of Formatting    76
    Vertical Formatting    76
    Horizontal Formatting    84
    Team Rules    89
    Uncle Bob's Formatting Rules.    90
Objects and Data Structures    93
    Data Abstraction    93
    Data/Object anti-symmetry.    95
    The Law of Demeter    97
    Data Transfer Objects    99
    Conclusion    101
    Bibliography    101
Error Handling by Michael Feathers    103
    Use Exceptions Rather than Return Codes    103
    Write Your Try-Catch-Finally Statement First    105
    Use Unchecked Exceptions    106
    Provide Context with Exceptions    107
    Define Exception Classes In Terms of a Caller's Needs.    107
    Define the Normal Flow    109
    Don't Return Null    110
    Don't Pass Null    111
    Conclusion    112
    Bibliography    112
Boundaries by James Grenning    113
    Bibliography    119
Unit Tests    121
    The Three Laws of TDD    122
    Keeping Tests Clean    123
    Clean Tests    124
    One Assert per Test    129
    F.I.R.S.T.    132
    Conclusion    132
    Bibliography    133
Classes    135
    Class Organization    135
    Classes should be Small!    136
    Organizing for Change    146
    Bibliography    150
Systems    By Dean Wampler 151
    How would you build a city?    151
    Separate constructing a system from using it    152
    Scaling Up    155
    Java Proxies    158
    Pure Java AOP Frameworks    160
    AspectJ Aspects    163
    Test-drive the system architecture    164
    Optimize decision making    165
    Use standards wisely, when they add demonstrable value    165
    Systems need Domain-Specific Languages    166
    Conclusion    166
    Bibliography    167
Emergence By Jeff Langr    169
    Getting Clean via Emergent Design    169
    Simple Design Rule 1: Runs all the tests    170
    Simple Design Rules 2-4: Refactoring    170
    No Duplication    170
    Expressive    173
    Minimal Classes and Methods    174
    Conclusion    174
    Bibliography    174
Concurrency    by Brett Schuchert 175
    Why Concurrency?    176
    Challenges    177
    Concurrency Defense Principles    178
    Know Your Library    180
    Know Your Execution Models    181
    Beware Dependencies between Syncrhonized Methods    182
    Keep Synchronized Sections Small    183
    Writing Correct Shut-Down Code is Hard    183
    Testing Threaded Code    184
    Conclusion    188
    Bibliography    189
Successive Refinement    191
    Args Implementation    192
    Args: the rough draft.    198
    String Arguments    212
     Conclusion    246
JUnit Internals    249
    Conclusion    262
Refactoring SerialDate    263
    Conclusion    280
    Bibliography    281
Smells and Heuristics    283
    Comments    283
    Environment    284
    Functions    285
    General    285
    Java    304
    Names    306
    Tests    310
    Conclusion    311
    Bibliography    312
Concurrency II    by Brett Schuchert 313
    Client/Server Example    313
    Possible Paths of Execution    317
    Knowing Your Library    322
    Dependencies between methods can break concurrent code    325
    Increasing Throughput    329
    Deadlock    331
    Testing Multi-Threaded Code    335
    Tool Support for Testing Thread-Based Code    337
    Conclusion    338
    Tutorial: Full Code Examples    339
org.jfree.date.SerialDate    345
Cross References of Heuristics    406

Clean Code. Whew!

Uncle Bob — Tue, 08 Apr 2008 00:10:16 -0500

Table of Contents

Clean Code    1
    There Will Be Code    1
    Bad Code    2
    The Total Cost of Owning a Mess.    3
    Schools of Thought.    11
    We are Authors.    12
    The Boy Scout Rule    13
    Prequel and Principles    14
    Conclusion    14
    Bibliography    15
Meaningful Names by Tim Ottinger    17
    Introduction    17
    Use Intention-revealing Names    17
    Avoid Disinformation    19
    Make Meaningful Distinctions    20
    Use Pronounceable Names    21
    Use Searchable Names    22
    Avoid Encodings    23
    Avoid Mental Mapping    24
    Class Names    25
    Method Names    25
    Don't Be Cute    25
    Pick One Word Per Concept    26
    Don't Pun    26
    Use Solution Domain Names    27
    Use Problem Domain Names    27
    Add Meaningful Context    27
    Don't add Gratuitous Context    29
    Final Words ...    30
Functions    31
    Small!    34
    Do one thing.    35
    One level of abstraction per function.    36
    Switch Statements.    37
    Use descriptive names.    39
    Function Arguments.    39
    Have no side-effects.    43
    Command Query Separation    44
    Prefer exceptions to returning error codes.    45
    Don't Repeat Yourself.    47
    Structured Programming    48
    How do you write functions like this?    48
    Conclusion    49
    SetupTeardownIncluder    49
    Bibliography    52
Comments    53
    Comments do not make up for bad code.    55
    Explain yourself in code.    55
    Good Comments    55
    Bad Comments    59
    Example    71
    Bibliography    74
Formatting    75
    The Purpose of Formatting    76
    Vertical Formatting    76
    Horizontal Formatting    84
    Team Rules    89
    Uncle Bob's Formatting Rules.    90
Objects and Data Structures    93
    Data Abstraction    93
    Data/Object anti-symmetry.    95
    The Law of Demeter    97
    Data Transfer Objects    99
    Conclusion    101
    Bibliography    101
Error Handling by Michael Feathers    103
    Use Exceptions Rather than Return Codes    103
    Write Your Try-Catch-Finally Statement First    105
    Use Unchecked Exceptions    106
    Provide Context with Exceptions    107
    Define Exception Classes In Terms of a Caller's Needs.    107
    Define the Normal Flow    109
    Don't Return Null    110
    Don't Pass Null    111
    Conclusion    112
    Bibliography    112
Boundaries by James Grenning    113
    Bibliography    119
Unit Tests    121
    The Three Laws of TDD    122
    Keeping Tests Clean    123
    Clean Tests    124
    One Assert per Test    129
    F.I.R.S.T.    132
    Conclusion    132
    Bibliography    133
Classes    135
    Class Organization    135
    Classes should be Small!    136
    Organizing for Change    146
    Bibliography    150
Systems    By Dean Wampler 151
    How would you build a city?    151
    Separate constructing a system from using it    152
    Scaling Up    155
    Java Proxies    158
    Pure Java AOP Frameworks    160
    AspectJ Aspects    163
    Test-drive the system architecture    164
    Optimize decision making    165
    Use standards wisely, when they add demonstrable value    165
    Systems need Domain-Specific Languages    166
    Conclusion    166
    Bibliography    167
Emergence By Jeff Langr    169
    Getting Clean via Emergent Design    169
    Simple Design Rule 1: Runs all the tests    170
    Simple Design Rules 2-4: Refactoring    170
    No Duplication    170
    Expressive    173
    Minimal Classes and Methods    174
    Conclusion    174
    Bibliography    174
Concurrency    by Brett Schuchert 175
    Why Concurrency?    176
    Challenges    177
    Concurrency Defense Principles    178
    Know Your Library    180
    Know Your Execution Models    181
    Beware Dependencies between Syncrhonized Methods    182
    Keep Synchronized Sections Small    183
    Writing Correct Shut-Down Code is Hard    183
    Testing Threaded Code    184
    Conclusion    188
    Bibliography    189
Successive Refinement    191
    Args Implementation    192
    Args: the rough draft.    198
    String Arguments    212
     Conclusion    246
JUnit Internals    249
    Conclusion    262
Refactoring SerialDate    263
    Conclusion    280
    Bibliography    281
Smells and Heuristics    283
    Comments    283
    Environment    284
    Functions    285
    General    285
    Java    304
    Names    306
    Tests    310
    Conclusion    311
    Bibliography    312
Concurrency II    by Brett Schuchert 313
    Client/Server Example    313
    Possible Paths of Execution    317
    Knowing Your Library    322
    Dependencies between methods can break concurrent code    325
    Increasing Throughput    329
    Deadlock    331
    Testing Multi-Threaded Code    335
    Tool Support for Testing Thread-Based Code    337
    Conclusion    338
    Tutorial: Full Code Examples    339
org.jfree.date.SerialDate    345
Cross References of Heuristics    406

Business software is Messy and Ugly.

Uncle Bob — Thu, 13 Dec 2007 09:41:00 -0600

I was at a client recently. They are a successful startup who have gone through a huge growth spurt. Their software grew rapidly, through a significant hack-and-slash program. Now they have a mess, and it is slowing them way down. Defects are high. Unintended consequences of change are high. Productivity is low.

I spent two days advising them how to adopt TDD and Clean Code techniques to improve their code-base and their situation. We discussed strategies for gradual clean up, and the notion that big refactoring projects and big redesign projects have a high risk of failure. We talked about ways to clean things up over time, while incrementally insinuating tests into the existing code base.

During the sessions they told me of a software manager who is famed for having said:

“There’s a clean way to do this, and a quick-and-dirty way to do this. I want you to do it the quick-and-dirty way.”

The attitude engendered by this statement has spread throughout the company and has become a significant part of their culture. If hack-and-slash is what management wants, then that’s what they get! I spent a long time with these folks countering that attitude and trying to engender an attitude of craftsmanship and professionalism.

The developers responded to my message with enthusiasm. They want to do a good job (of course!) They just didn’t know they were authorized to do good work. They thought they had to make messes. But I told them that the only way to get things done quickly, and keep getting things done quickly, is to create the cleanest code they can, to work as well as possible, and keep the quality very high. I told them that quick-and-dirty is an oxymoron. Dirty always means slow.

On the last day of my visit the infamous manager (now the CTO) stopped into our conference room. We talked over the issues. He was constantly trying to find a quick way out. He was manipulative and cajoling. “What if we did this?” or “What if we did that?” He’d set up straw man after straw man, trying to convince his folks that there was a time and place for good code, but this was not it.

I wanted to hit him.

Then he made the dumbest, most profoundly irresponsible statement I’ve (all too often) heard come out of a CTOs mouth. He said:

“Business software is messy and ugly.”

No, it’s not! The rules can be complicated, arbitrary, and ad-hoc; but the code does not need to be messy and ugly. Indeed, the more arbitrary, complex, and ad-hoc the business rules are, the cleaner the code needs to be. You cannot manage the mess of the rules if they are contained by another mess! The only way to get a handle on the messy rules is to express them in the cleanest and clearest code you can.

In the end, he backed down. At least while I was there. But I have no doubt he’ll continue his manipulations. I hope the developers have the will to resist.

One of the developers asked the question point blank: “What do you do when your managers tell you to make a mess?” I responded: “You don’t take it. Behave like a doctor who’s hospital administrator has just told him that hand-washing is too expensive, and he should stop doing it.”

Active Record vs Objects

Uncle Bob — Fri, 02 Nov 2007 11:29:31 -0500

Active Record is a well known data persistence pattern. It has been adopted by Rails, Hibernate, and many other ORM tools. It has proven it’s usefulness over and over again. And yet I have a philosophical problem with it.

The Active Record pattern is a way to map database rows to objects. For example, let’s say we have an Employee object with name and address fields:

public class Employee extends ActiveRecord {
  private String name;
  private String address;
  ...
}

We should be able to fetch a given employee from the database by using a call like:

Employee bob = Employee.findByName("Bob Martin");

We should also be able to modify that employee and save it as follows:

bob.setName("Robert C. Martin");
bob.save();

In short, every column of the Employee table becomes a field of the Employee class. There are static methods (or some magical reflection) on the ActiveRecord class that allow you to find instances. There are also methods that provide CRUD functions.

Even shorter: There is a 1:1 correspondence between tables and classes, columns and fields. (Or very nearly so).

It is this 1:1 correspondence that bothers me. Indeed, it bothers me about all ORM tools. Why? Because this mapping presumes that tables and objects are isomorphic.

The Difference between Objects and Data Structures

From the beginning of OO we learned that the data in an object should be hidden, and the public interface should be methods. In other words: objects export behavior, not data. An object has hidden data and exposed behavior.

Data structures, on the other hand, have exposed data, and no behavior. In languages like C++ and C# the struct keyword is used to describe a data structure with public fields. If there are any methods, they are typically navigational. They don’t contain business rules.

Thus, data structures and objects are diametrically opposed. They are virtual opposites. One exposes behavior and hides data, the other exposes data and has no behavior. But that’s not the only thing that is opposite about them.

Algorithms that deal with objects have the luxury of not needing to know the kind of object they are dealing with. The old example: shape.draw(); makes the point. The caller has no idea what kind of shape is being drawn. Indeed, if I add new types of shapes, the algorithms that call draw() are not aware of the change, and do not need to be rebuilt, retested, or redeployed. In short, algorithms that employ objects are immune to the addition of new types.

By the same token, if I add new methods to the shape class, then all derivatives of shape must be modified. So objects are not immune to the addition of new functions.

Now consider an algorithm that uses a data structure.


switch(s.type) {
  case SQUARE: Shape.drawSquare((Square)s); break;
  case CIRCLE: Shape.drawCircle((Circle)s); break;
}

We usually sneer at code like this because it is not OO. But that disparagement might be a bit over-confident. Consider what happens if we add a new set of functions, such as Shape.eraseXXX(). None of the existing code is effected. Indeed, it does not need to be recompiled, retested, or redeployed. Algorithms that use data structures are immune to the addition of new functions.

By the same token if I add a new type of shape, I must find every algorithm and add the new shape to the corresponding switch statement. So algorithms that employ data structures are not immune to the addition of new types.

Again, note the almost diametrical opposition. Objects and Data structures convey nearly opposite immunities and vulnerabilities.

Good designers uses this opposition to construct systems that are appropriately immune to the various forces that impinge upon them. Those portions of the system that are likely to be subject to new types, should be oriented around objects. On the other hand, any part of the system that is likely to need new functions ought to be oriented around data structures. Indeed, much of good design is about how to mix and match the different vulnerabilities and immunities of the different styles.

Active Record Confusion

The problem I have with Active Record is that it creates confusion about these two very different styles of programming. A database table is a data structure. It has exposed data and no behavior. But an Active Record appears to be an object. It has “hidden” data, and exposed behavior. I put the word “hidden” in quotes because the data is, in fact, not hidden. Almost all ActiveRecord derivatives export the database columns through accessors and mutators. Indeed, the Active Record is meant to be used like a data structure.

On the other hand, many people put business rule methods in their Active Record classes; which makes them appear to be objects. This leads to a dilemma. On which side of the line does the Active Record really fall? Is it an object? Or is it a data structure?

This dilemma is the basis for the oft-cited impedance mismatch between relational databases and object oriented languages. Tables are data structures, not classes. Objects are encapsulated behavior, not database rows.

At this point you might be saying: “So what Uncle Bob? Active Record works great. So what’s the problem if I mix data structures and objects?” Good question.

Missed Opportunity

The problem is that Active Records are data structures. Putting business rule methods in them doesn’t turn them into true objects. In the end, the algorithms that employ Active Records are vulnerable to changes in schema, and changes in type. They are not immune to changes in type, the way algorithms that use objects are.

You can prove this to yourself by realizing how difficult it is to implement an polymorphic hierarchy in a relational database. It’s not impossible of course, but every trick for doing it is a hack. The end result is that few database schemae, and therefore few uses of Active Record, employ the kind of polymorphism that conveys the immunity of changes to type.

So applications built around ActiveRecord are applications built around data structures. And applications that are built around data structures are procedural—they are not object oriented. The opportunity we miss when we structure our applications around Active Record is the opportunity to use object oriented design.

No, I haven’t gone off the deep end.

I am not recommending against the use of Active Record. As I said in the first part of this blog I think the pattern is very useful. What I am advocating is a separation between the application and Active Record.

Active Record belongs in the layer that separates the database from the application. It makes a very convenient halfway-house between the hard data structures of database tables, and the behavior exposing objects in the application.

Applications should be designed and structured around objects, not data structures. Those objects should expose business behaviors, and hide any vestige of the database. The fact that we have Employee tables in the database, does not mean that we must have Employee classes in the application proper. We may have Active Records that hold Employee rows in the database interface layer, but by the time that information gets to the application, it may be in very different kinds of objects.

Conclusion

So, in the end, I am not against the use of Active Record. I just don’t want Active Record to be the organizing principle of the application. It makes a fine transport mechanism between the database and the application; but I don’t want the application knowing about Active Records. I want the application oriented around objects that expose behavior and hide data. I generally want the application immune to type changes; and I want to structure the application so that new features can be added by adding new types. (See: The Open Closed Principle)

Architecture is a Second Order Effect

Uncle Bob — Sat, 20 Oct 2007 01:58:29 -0500

We often argue that in order to achieve flexibility, maintainability, and reusability, it is important to have a good software architecture. This is certainly true. Without a well ordered architecture, software systems become masses of tangled modules and code. However, the effect of architecture on the ‘ilities is secondary to the effect that a good, fast, suite of tests has.

Why don’t we clean our code? When we see an ugly mass of code that we know is going to cause of problems, our first reaction is “This needs to be cleaned up.” Our second reaction is: “If I touch this code I’ll be spending the next two weeks trying to get it to work again.” We don’t clean code because we are afraid we’ll break it.

In this way bad code hooks itself into our systems and refuses to go away. Nothing stops clean code from going bad, but once it’s bad, we seldom have the time, energy, or nerve to clean it. In that sense, bad code is permanent.

What if you had a button? If you push this button a little light instantly turns red or green. Green means your system works. Red means it’s broken. If you had a button like this, you could make small changes to the system, and prove that they didn’t break anything. If you saw a batch of tangled messy code, you could start to clean it. You’d simply make a tiny improvement and then push the button. If the light was green you’d make the next tiny change, and the next, and the next.

I have a button like that! It’s called a test suite. I can run it any time I like, and within seconds it tells me, with very high certainty, that my system works.

Of course, to be effective, that test suite has to cover all the code in the system. And, indeed, that’s what I have. My code coverage tools tell me that 89% of the 45,000 lines in my system are covered by my test suite. (89% is pretty good number given that my coverage tool counts un-executable lines like interfaces.)

Can I be absolutely sure that the green light means that the system works? Certainly not. But given the coverage of my test suite, I am reasonably sure that the changes I make are not introducing any bugs. And that reasonable surety is enough for me to be fearless about making changes.

This fearlessness is something that needs to be experienced to understand. I feel no reluctance at all about cleaning up the code in my system. I frequently take whole classes and restructure them. I change the names of classes, variables, and functions on a whim. I extract super-classes and derivatives any time I like.

In short, the test suite makes it easy to make changes to my code. It makes my code flexible and easy to maintain.

So does my architecture of course. The design and structure of my system is very easy to deal with, and allows me to make changes without undue impact. The reason my architecture is so friendly, is that I’ve been fearless about changing it. And that’s because I have a test suite that runs in seconds, and that I trust!

So the clean architecture of my system is a result of on-going efforts supported by the test suite. I can keep the architecture clean and relevant because I have tests. I can improve the architecture when I see a better approach because I have tests. It is the tests that enable architectural improvement.

Yes, architecture enables flexibility, maintainability, and reusability; but test suites enable architecture. Architecture is a second order effect.