Object Mentor Blog: Category Uncle Bob's Blatherings

Software Calculus - The Missing Abstraction.

Uncle Bob — Mon, 05 Jul 2010 20:26:15 -0500

The problem of infinity plagued mathematicians for millennia. Consider Xeno’s paradox; the one with Achilles and the tortoise. While it was intuitively clear that Achilles would pass the Tortoise quickly, the algebra and logic of the day seemed to suggest that the Tortoise would win every race given a head start. Every time Achilles got to where the tortoise was, the tortoise would have moved on. The ancients had no way to see that an infinite sum could be finite.

Then came Leibnitz and everything changed. Suddenly infinity was tractable. Suddenly you could sum infinite series and write the equations that showed Achilles passing the tortoise. Suddenly a whole range of calculations that had either been impossible or intractable became trivial.

Calculus was a watershed invention for mathematics. It opened up vistas that we have yet to fully plumb. It made possible things like Newtonian mechanics, Maxwell’s equations, special and general relativity and quantum mechanics. It supports the entire framework of our modern world. We need a watershed like that for software.

If you listen to my keynote: Twenty-Five Zeros you’ll hear me go on and on about how even though software has changed a lot in form over the last fifty years, it has changed little in substance. Software is still the organization of sequence, selection, and iteration.

For fifty years we have been inventing new languages, notations, and formulations to manage Sequence, Selection, and Iteration (SSI). Structured Programming is simply a way to organize SSI. Objects are another way to organize SSI. Functional is still another. Indeed, almost all of our software technologies are just different ways of organizing Sequence, Selection, and Iteration.

This is similar to Algebra in the days before calculus. We knew how to solve linear and polynomial equations. We knew how to complete squares and find roots. But in the end it was all just different ways to organize adding. That may sound simplistic, but it’s not. Subtracting is just adding in reverse. Multiplying is just adding repeatedly. Division is just multiplication in reverse. In short, Algebra is an organizational strategy for adding.

Algebra went through many different languages and notations too, just like software has. Think about Roman and Greek numerals. Think how long it took to invent the concept of zero, or the positional exponential notation we use today.

And then one day Newton saw an apple fall, and he changed the way we thought about mathematics. Suddenly it wasn’t about adding anymore. Suddenly it was about infinities and differentials. Mathematical reasoning was raised to a new order of abstraction.

Where is that apple for software (pun intended). Where is the Newton or Leibnitz that will transform everything about the way we think about software. Where is that long-sought new level of abstraction?

For awhile we thought it would be MDA. Bzzzzt, wrong. We thought it would be logic programming like prolog¹. Bzzzt. We thought it would be database scripts and 4GLs. Bzzzt. None of those did it. None of those can do it. They are still just various ways of organizing sequence, selection, and iteration.

Some people have set their sights on quantum computing. While I’ll grant you that computations with bits that can be both states simultaneously is interesting, in the end I think this is just another hardware trick to increase throughput as opposed to a whole new way to think about software.

This software transformation, whatever it is, is coming. It must come; because we simply cannot keep piling complexity upon complexity. We need some new organizing principle that revamps the very foundations of the way we think about software and opens up vistas that will take us into the 22nd century.

¹ Prolog comes closest to being something more than a simple reorganization of sequence, selection, and iteration. At first look logic programming seems very different. In the end, however, an algorithm expressed in prolog can be translated into any of the other languages, demonstrating the eventual equivalence.

TDD in Clojure

Uncle Bob — Thu, 03 Jun 2010 12:33:00 -0500

OO is a tell-don’t-ask paradigm. Yes, I know people don’t always use it that way, but one of Kay’s original concepts was that objects were like cells in a living creature. The cells in a living creature do not ask any questions. They simply tell each other what to do. Neurons are tellers, not askers. Hormones are tellers not askers. In biological systems, (and in Kay’s original concept for OO) communication was half-duplex.

Clojure is a functional language. Functional languages are ask-dont-tell. Indeed, the whole notion of “tell” is to change the state of the system. In a functional program there is no state to change. So “telling” makes little sense.

When we use TDD to develop a tell-don’t-ask system, we start at the high level and write tests using mocks to make sure we are issuing the correct “tells”. We proceed from the top of the system to the bottom of the system. The last tests we write are for the utilities at the very bottom.

In an ask-don’t-tell system, data starts at the bottom and flows upwards. The operation of each function depends on the data fed to it by the lower level functions. There is no mocking framework. So we write tests that start at the bottom, and we work our way up the the top.

Therein lies the rub.

In a tell-don’t-ask system, the tells at the high level are relatively complex. They branch out into lower subsystems getting simpler, but more numerous as they descend. Testing these tells using mocks is not particularly difficult because we don’t need to depend on the lower level functions being there. The mocks make them irrelevant.

In an ask-don’t-tell system the asks at the low level are simple, but as the data moves upwards it gets grouped and composed into lists, maps, sets, and other complex data structures. At the top the data is in it’s most complex form. Writing tests against that complex data is difficult at best. And there is currently no way to mock out the lower levels¹ so all tests written at the high level depend on all the functions below.

The perception of writing tests from the bottom to the top can be horrific at first. Consider, for example, the Orbit program I just wrote. This program simulates N-body gravitation. Imagine that I am writing tests at the top level. I have three bodies at position Pa, Pb, and Pc. They have masses Ma, Mb, and Mc. They have velocity vectors of Va, Vb, Vc. The test I want to write needs to make sure that new positions Pa’, Pb’, Pc’, and new Velocity vectors Va’, Vb’, and Vc’ are computed correctly. How do I do that?

Should I write a test that looks like this?

test-update {
  Pa = (1,1)
  Ma = 2
  Va = (0,0)

  Pb = (1,2)
  Mb = 3
  Vb = (0,0)

  Pc = (4,5)
  Mc = 4
  Vc = (0,0)

  update-all

  Pa should == (1.096, 4.128)
  Va should == (0.096, 3.128)

  Pb should == (1.1571348402636772, 0.1571348402636774)
  Vb should == (0.15713484026367727, -1.8428651597363226)

  Pc should == (3.834148869802242, 4.818148869802242)
  Vc should == (-0.16585113019775796, -0.18185113019775795)
}

A test like this is awful. It’s loaded with magic numbers, and secret information. It tells me nothing about how the update-all function is working. It only tells me that it generated certain numbers. Are those numbers correct? How would I know?

But wait! I’m working in a functional language. That means that every function I call with certain inputs will always return the same value; no matter how many times I call it. Functions don’t change state! And that means that I can write my tests quite differently.

How does update-all work? Simple, given a list of objects it performs the following operations (written statefully):

  update-all(objects) {
    for each object in objects {
      accumulate-forces(object, objects)
    }
    for each object in objects {
      accelerate(object)
      reposition(object)
    }
  }

This is written in stateful form to make is easier for our non-functional friends to follow. First we accumulate the force of gravity between all the objects. This amounts to evaluating Newton’s F=Gm1m1/r^2 formula for each pair of objects, and adding up the force vectors.

Then, for each object we accelerate that object by applying the force vector to it’s mass, and adding the resultant delta-v vector to it’s velocity vector.

Then, for each object we reposition that object by applying the velocity vector to it’s current position.

Here’s the clojure code for update-all

(defn update-all [os]
  (reposition-all (accelerate-all (calculate-forces-on-all os))))

In this code you can clearly see the bottom-to-top flow of the application. First we calculate forces, then we accelerate, and finally we reposition.

Now, what do these -all functions look like? Here they are:

(defn calculate-forces-on-all [os]
  (map #(accumulate-forces % os) os))

(defn accelerate-all [os]
  (map accelerate os))

(defn reposition-all [os]
  (map reposition os))

If you don’t read clojure, don’t worry. the map function simply creates a new list from an old list by applying a function to each element of the old list. So in the case of reposition-all it simply calls reposition on the list of objects (os) producing a new list of objects that have been repositioned.

From this we can determine that the function of update-all is to call the three functions (accumulate-forces, accelerate, and reposition) on each element of the input list, producing a new list.

Notice how similar that is to a statement we might make about a high level method in an OO program. (It’s got to call these three functions on each element of the list). In an OO language we would mock out the three functions and just make sure they’d been called for each element. The calculations would be bypassed as irrelevant.

Oddly, we can make the same statement in clojure. Here’s the test for update-all

(testing "update-all" 
  (let [
    o1 (make-object ...)
    o2 (make-object ...)
    o3 (make-object ...)
    os [o1 o2 o3]
    us (update-all os)
    ]
    (is (= (nth us 0) (reposition (accelerate (accumulate-forces os o1)
    (is (= (nth us 1) (reposition (accelerate (accumulate-forces os o2)
    (is (= (nth us 2) (reposition (accelerate (accumulate-forces os o3)
    )
  )

If you don’t read clojure don’t worry. All this is saying is that we test the update-all function by calling the appropriate functions for each input object, and then see if the elements in the output list match them.

In an OO program we’d find this dangerous because of side-effects. We couldn’t be sure that the functions could safely be called without changing the state of some object in the system. But in a functional language it doesn’t matter how many times you call a function. So long as you pass in the same data, you will get the same result.

So this test simply checks that the appropriate three functions are getting called on each element of the list. This is exactly the same thing an OO programmer would do with a mock object!

Is TDD necessary in Clojure?

If you follow the code in the Orbit example, you’ll note that I wrote tests for all the computations, but did not write tests for the Swing-Gui. This is typical of the way that I work. I try to test all business rules, but I “fiddle” with the GUI until I like it.

If you look carefully you’ll find that amidst the GUI functions there are some “presentation” functions that could have been tested, but that I neglected to write with TDD[2]. These functions were the worst to get working. I continuously encountered NPEs and Illegal Cast exceptions while trying to get them to work.

My conclusion is that Clojure without TDD is just as much a nightmare as Java or Ruby without TDD.

Summary

In OO we tend to TDD our way from the top to the bottom by using Mocks. In Clojure we tend to TDD our way from the bottom to the top. In either case we can compose our tests in terms of the functions they should call on the lower level objects. In the case of OO we use mocks to tell us if the functions have been called properly. This protects us from side-effects and allows us to decouple our tests from the whole system. In clojure we can rely on the fact that the language is functional, and that no matter how many times you call a function it will return the same value.

¹ Brian Marick is working on something that looks a lot like a mocking framework for clojure. If his ideas pan out, we may be able to TDD from the top to the bottom in Clojure.

² This is an unconscious game we all play with ourselves. When we have a segment of code that we consider to be immune to TDD (like GUI) then we unconsciously move lots of otherwise testable code into that segment. Yes, I heard my green band complain every time I did it; but I ignored it because I was in the GUI. Whoops.

Orbit in Clojure

Uncle Bob — Wed, 02 Jun 2010 17:08:56 -0500

I spent the last two days (in between the usual BS) writing a simple orbital simulator in Clojure using Java interop with Swing. This was a very pleasant experience, and I like the way the code turned out – even the swing code!

You can see the source code here

Those of you who are experienced with Clojure, I’d like your opinion on my use of namespaces and modules and other issues of style.

Those of you who are not experienced with Clojure, should start. You might want to use this application as a tutorial.

And just have fun watching the simulation of the coalescence of an accretion disk around a newly formed star.

Clojure Prime Factors

Uncle Bob — Sat, 15 May 2010 19:09:59 -0500

Can anyone create a simpler version of prime factors in Clojure?

(ns primeFactors)

(defn of
  ([n]
    (of [] n 2))
  ([factors n candidate]
    (cond
      (= n 1) factors
      (= 0 (rem n candidate)) (recur (conj factors candidate) (quot n candidate) candidate)
      (> candidate (Math/sqrt n)) (conj factors n)
      :else (recur factors n (inc candidate))
      )
    )
  )

Sufficient Design means Damned Good Design.

Uncle Bob — Wed, 28 Apr 2010 18:02:10 -0500

@JoshuaKerievsky wrote a blog entitled “Sufficient Design”.

Josh makes this point:

‘Yet some programmers argue that the software design quality of every last piece of code ought to be as high as possible. “Produce clean code or you are not a software craftsman!”’

He goes on to say:

“Yet ultimately the craftsmanship advice fails to consider simple economics: If you take the time to craft code of little or moderate value to users, you’re wasting time and money.”

Now this sounds like heresy, and I can imagine that software craftsmanship supporters (like me) are ready to storm the halls of Industrial Logic and string the blighter up by his toenails!

But hold on there craftsmen, don’t get the pitchforks out yet. Look at the scenario that Josh describes. It’s quite revealing.

Josh’s example of “not being a craftsman” is his niggling worry over a function that returns a string but in one derivative case ought to return a void.

Horrors! Horrors! He left a bit of evolved cruft in the design! Revoke his craftsman license and sick the dogs on him!

Ah, but wait. He also says that he spent a half-day trying to refactor this into a better shape but eventually had to give up.

The fool! The nincompoop! The anti-craftsman! A pox on him and all his ilk!

OK, enough hyperbole. It seems to me that Josh was behaving exactly as a craftsman should. He was worrying about exactly the kinds of things a craftsman ought to worry about. He was making the kinds of pragmatic decisions a craftsman ought to make. He was not leaving a huge mess in the system and rushing on to the next feature to implement. Instead he was taking care of his code. The fact that he put so much energy, time, and thought into such a small issue as an inconsistent return type speaks volumes for Josh’s integrity as a software craftsman.

So, as far as Josh Kerievsky is concerned “Sufficient Design” is “Pretty Damned Good Design”.

Look, all our software will have little tiny warts. There’s no such thing as a perfect system. Craftsmen like Josh work hard to keep those warts as small as possible, and to sequester them into parts of the system where they can do the least harm.

So the only aspect of Josh’s post that I disagree with is his contention that the “craftsman” message is one of unrelenting perfection. Craftsmen are first and foremost pragmatists. They seek very high quality code; but they are not so consumed with perfection that they make foolish economic tradeoffs.

Certification: Don't Waste Your Time!

Uncle Bob — Tue, 27 Apr 2010 15:11:00 -0500

As I have said before, there’s nothing particularly wrong with the current mania for certification. If you want to be certified at the cost of a 2-day course, by all means get certified. If you want to certify people for attending your 2-day course, by all means hold the course and hand out the certificates. It’s all good. Make money! Be fruitful and multiply!

But be careful not to waste your time.

How could certification be a waste of time? That depends on your motive.

If you are getting certified in order to impress someone, like a hiring manager, or a recruiter, or your peers, you are wasting your time. Nobody worth impressing is going to find this certification impressive. Indeed, the people you really want to impress are likely to find it a bit mundane.
If you are getting certified in order to get hired, you are wasting your time. Nobody is going to hire you simply because of that “C”, and nobody worth working for is going to require that “C”.
If you have decided to hire only certified people, you are wasting your time. The population of certified people is not richer in talent, skill, or knowledge. Indeed, it may be poorer. Remember, those who have talent don’t need certification as much as those who don’t.

What part of certification is not a waste of time?

The primary benefit you are getting is the instruction; but be careful: There are lots of pretty mediocre instructors out there. Some of the instructors teach pretty good courses, but others are just hoping that all you care about is the certification.
So do a little research and find the best instructors. You may, in fact, find that some of the best instructors and courses do not offer certification. That shouldn’t stop you from considering them.

But wait, aren’t the instructors certified as trainers?

Sure. They paid the money and took the course to become a certified trainer.
But that doesn’t necessarily mean that they:
- are a good instructor.
- know what they are teaching.
- have done what they are teaching.
- are qualified to teach you.

OK, but isn’t there some benefit to the certification itself?

Sure. A nice piece of paper.

The Tricky Bit

Uncle Bob — Fri, 23 Apr 2010 04:28:35 -0500

I once heard a story about the early days of the Concorde. A british MP was on board for a demonstration flight. As the jet went supersonic he disappointedly commented to one of the designers that it didn’t feel any different at all. The designer beamed and said: “Yes, that was the tricky bit.”

I wonder if the MP would have been happier if the plane had begun to vibrate and shake and make terrifying noises.

While at #openvolcano10, Gojko Adzic and Steve Freeman told the story of a company in which one team, among many, had gone agile. Over time that team managed to get into a nice rhythm, passing it’s continuous builds, adding lots of business value, working normal hours, and keeping their customers happy. Meanwhile other teams at this company were facing delays, defects, frustrated customers, and lots of overtime.

The managers at this company looked at the agile team, working like a well-oiled machine, and looked at all the other teams toiling in vain at the bilge pump, and came to the conclusion that the project that the agile team was working on was too simple to keep them busy. That the other teams’ projects were far more difficult. It didn’t even occur to them that the agile team had figured out the tricky bit.

The R.E.A.L.I.T.Y. principles of Agile Software Certification.

Uncle Bob — Wed, 21 Apr 2010 05:29:00 -0500

As you are probably aware the Scrum Alliance is planning to offer a Certified Scrum Developer program. You can read about it here.

Interestingly enough Microsoft, in collaboration with Ken Schwaber, is offering a Professional Scrum Developer program. You can read about that here (look carefully at the url).

The scrum alliance program seems to be about agile skills. The Microsoft program seems to be about Visual Studio. So all is right with the world.

Is there anything wrong with this? Is this somehow immoral or bad? Not at all. This is just people doing what people do: Selling services and making money. I’m all for it. I expect the courses to be of a quality that you’d expect from professionals; and I’m sure that students will learn useful information. Indeed, from time to time, even I have offered CSM courses taught by people I trust and respect.

There are, however, some claims that need to be dealt with. The most important is that these programs shorten the recruiting process.

I’ve been thinking about offering a Certified Recruiter for Agile Professionals course. This fifty-two week course will teach, and reteach, a number of principles of Agile Software Certification. I call this repertoire of principles: R.E.A.L.I.T.Y.

The first principle is the Redaction of Certification Principle (RCP). The principle states:

Certifications generally certify nothing whatever about experience, knowledge, or skill. Generally they certify that the certificate holder has attended (or at least paid to attend) a course. Perhaps they took (and maybe even passed) an exam based on that course.

Based on this principle, any recruiter taking my Certified Recruiter for Agile Professionals course will not be allowed to complete the course or receive their certification until they have taken the following oath.

As a Cerftified Recruiter for Agile Professionals I solemnly swear before Knuth and all here present:

That before I read a resume, I will find every use of the word “certified” on that document and redact it with whiteout.

That if a candidate uses the word “certified” in an interview, I will ask the candidate to repeat him- or herself without using that word.

That I will pay no attention whatever to any implications of that word, nor will that word in any way influence my opinions regarding that candidate.

This oath is an exemplar of the Certification Nullification Principle (CNP), which is another of the R.E.A.L.I.T.Y. principles of common sense and moderately competent thinking. The principle states:

The word “certified”, when used in the context of Agile Software Development, is a filler word that has no bearing on anything salient or interesting about individuals, skills, knowledge, or anything else. It is a marketing word similar to “new”, “improved”, “natural” and “organic”. It can safely be removed from all documents and discussions without changing their meaning.

This principle leads to the well known Certification Uncertainty Principle (CUP), yet another of the R.E.A.L.I.T.Y. principles. This principle states:

Any acronym that includes the letter ‘C’ standing for the word “Certification” can be safely changed into a similar acronym that eliminates the letter ‘C’ and puts a ’?’ at the end. This transformation of the acronym vastly improves the meaning of the orginal.

That last sentence probably requires some justification. After all, extraordinary claims require extraordinary substantiation. So lets try a few experiments:

Statement                    Ac'nm   CUP  Resulting Statement 
---------------------------- -----   ---- ---------------------
Certified Scrum Master        CSM    SM?  Scrum Master?
Certified Scrum Developer     CSD    SD?  Scrum Developer?
Certified Scrum Trainer       CST    ST?  Scrum Trainer?
Certified Scrum Professional  CSP    SP?  Scrum Professional?
Certified Scrum Product Owner CSPO   SPO? Scrum Product Owner?
Certified Scrum Coach         CSC    SC?  Scrum Coach?

As we can see, the meanings of the statements are indeed clarified. In my course the Certified Recruiter for Agile Professionals is taught that the question mark is the most significant aspect of each of the resultant statements.

In recent weeks we have uncovered a potentially profound new principle which we may find necessary to include in the R.E.A.L.I.T.Y. repertoire. This is the so-called Scrum Exclusion Principle (SEP) which tentatively states:

Wherever the word SCRUM appears in any statement, or is represented within any acronym, it can be safely excluded without changing any meaning.

The following table show just how profound the effects of this principle are:

Statement                    Ac'nm   CUP  SEP   Resulting Statement 
---------------------------- -----   ---- ----  ---------------------
Certified Scrum Master        CSM    SM?  M?    Master?
Certified Scrum Developer     CSD    SD?  D?    Developer?
Certified Scrum Trainer       CST    ST?  T?    Trainer?
Certified Scrum Professional  CSP    SP?  P?    Professional?
Certified Scrum Product Owner CSPO   SPO? PO?   Product Owner?
Certified Scrum Coach         CSC    SC?  C?    Coach?

We are not yet willing to say that these transformation are a reliable feature of nature. We can only say that, at this point, there are very troubling implications that support it.

Our final advice to Certified Recruiters for Agile Professionals is based on one final principle: The Smoke and Mirrors principle, (SAM) which states:

Well… You know what it states.

The Mayans are Coming!

Uncle Bob — Sat, 17 Apr 2010 11:22:59 -0500

This one is just a quickie; too long for a tweet.

I gave a lightning keynote (actually a key-rant) today at #accu2010. It was about a bet between John Lakos and Kevlin Henney. Kevlin “won” the bet last night by showing that he could get a C++ program to behave in a way that John said couldn’t be done. My rant was about how both of them were wrong (not technically, but morally) and so they both owed me the sum of the bet. (I’m always happy to insert myself into any available revenue stream.)

The problem they had coded, and for which I presented a far superior solution in Clojure, was that old work-horse of prob/stat: “What’s the probability that out of N people in a room two will have the same birthday?”

Now, just to show you how strange James Bach is (and why his strangeness is in some bizarre way actually valuable); once my talk was done, James got up and said “What if some of the people in the room are ancient Mayans who use a completely different kind of calendar making it impossible to compare their birthdays?”

I’ve long said that testers were deviant souls with twisted minds that delight in reducing helpless software systems into piles of smoking slag. I believe James may be the exemplar.

Sapient Testing: The "Professionalism" meme.

Uncle Bob — Thu, 15 Apr 2010 06:18:00 -0500

James Bach gave a stirring keynote today at ACCU 2010. He described a vision of testing that our industry sorely needs. Towit: Testing requires sapience.

Testing, according to Bach, is not about assuring conformance to requirements; rather it is about understanding the requirements. Even that’s not quite right. It is not sufficient to simply understand and verify the requirements. A good tester uses the behavior of the system and the descriptions in the requirements, (and face-to-face interaction with the authors of both) to understand the motivation behind the system. Ultimately it is the tester’s job to divine the system that the customer imagined; and then to illuminate those parts of the system that are not consistent with that imagination.

It seems to me that James is attempting to define “professionalism” as it applies to testing. A professional tester does not blindly follow a test plan. A professional tester does not simply write test plans that reflect the stated requirements. Rather a professional tester takes responsibility for interpreting the requirements with intelligence. He tests, not only the system, but also (and more importantly) the assumptions of the programmers, and specifiers.

I like this view. I like it a lot. I like the fact that testers are seeking professionalism in the same way that developer are. I like the fact that testing is becoming a craft, and that people like James are passionate about that craft. There may yet be hope for our industry!

There has been a long standing frission between James’ view of testing and the Agile emphasis on TDD and automated tests. Agilists have been very focussed on creating suites of automated tests, and exposing the insanity (and inanity) of huge manual testing suites. This focus can be (and has been) misinterpreted as an anti-tester bias.

It seems to me that professional testers are completely compatible with agile development. No, that’s wrong. I think professional testers are utterly essential to agile development. I don’t want testers who rotely execute brain-dead manual test plans. I want testers using their brains! I want testers to be partners in the effort to create world-class, high-quality software. As a professional developer I want – I need – professional testers helping me find my blind spots, illuminating the naivete of my assumptions, and partnering with me to satisfy the needs of our customers.

20% more bugs? Or 20% less features?

Uncle Bob — Tue, 06 Apr 2010 22:03:23 -0500

People often make the argument that time to market is more important that quality. I’m not sure just what they mean by that. Do they mean that it’s ok if 20% of the features don’t work so long as they deliver quickly? If so, that’s just stupid. Why not develop 20% fewer features, and develop them well. It seems to me that choosing which 20% you are not going to develop and then choosing to develop the other 80% to a high standard of quality is a better management decision than telling the developers to work sloppily.

Developer Certification WTF?

Uncle Bob — Sun, 07 Mar 2010 12:16:43 -0600

Watch the following video. It will convince you that we have to do something about the horrible state of software engineering.

Bad Code from unclebob on Vimeo.

How could any team of disciplined professionals have produced a wretched mess like that? Clearly they were ignorant of good practices. Clearly they were inexperienced novices. Clearly their priorities and values were all wrong. If only they had been taught good coding practices, and good development skills. If only we could have gotten to them before they made such a horrible mess.

What we need to do is create a certification program that provides developers with the knowledge and skills that they need. This program will involve a course that teaches good development practice, and a certification that they are now knowledgeable developers. They can use that certification to prove to their employers and their fellow professionals that they are worthy of being considered true and clean software developers.

That’s what we need. Right?....

Eagle’s Entrails! Deer Droppings! and Elephant Phlatulence!

I’m sorry, am I being vague? Is my opinion not clear enough? All right then, allow me to elaborate.

What Problem does Certification solve?

Certainly there are certification organizations who have the problem of dwindling revenues. A new developer certification program would likely solve their problem.

And just as certainly there are tin-men who need hearts, lions who need courage, scarecrows who need brains, and developers who need self-esteem. A new developer certification program might just help those lackluster developers.

But who else is served by a developer certification program? Employers?

The Trials of Hercules.

Consider Doctors and Lawyers. When a medical student earns an M.D., or a law student passes the Bar, that’s an achievement. The certification, in those cases is deeply meaningful because the certification is not easily won. To get it you have to spend many years, and many tens of thousands of dollars. What’s more, you have to acquire in-depth knowledge and at least a modicum of provable skill. You have to do something significant.

If we had a developer certification like that, then employers might find it useful. But I know of no developer certification program (except for one that I’ll mention later) that offers proof that the developer has accomplished anything of significance. Most certification programs prove little more than that the “developer” paid to attend a 3-5 day course. As an employer, I’m not particularly interested in the ability of a developer to pay to attend a 3-5 day course. As an employer I want to know what the developer can do.

Michaelangelo’s Apprentice.

If you were hiring a guitarist for your band, you might be very impressed with someone who toured with Clapton. You don’t get to tour with Clapton unless you are talented. But more than just talented, you have to show up every day, you have to work well with others, you have to be willing to get the job done under pressure. Clapton doesn’t work with slouches!

A signed letter of recommendation from Clapton is a certification of significance! Not just anyone can get one of those. That letter will impress potential employers forever after.

As an employer, I want to hire people who have succeeded at working with others in the past. The more I trust those others, the better! No developer certification program that I know of (except for one that I’ll mention later) provides that kind of credible reference.

Over-promise and Under-deliver.

The certification programs that I know of (except for one that I’ll mention later) make implicit promises that they can’t hope to begin to deliver. The implied promise is that the certificant has been shown to have some significant skill, ability, or knowledge. The reality is that you have to look at the word “significant” through the wrong end of some very powerful binoculars before it matches the truth. Indeed, there are certification programs that show absolutely nothing about the certificant except that they bought a seat in a class.

Notice that I said that the promises were implicit. That’s the whole key to making lightweight certifications profitable. The goal is to imply that the benefits of certification are so huge that people will pay to get the certification. Often that implication can be as simple and subtle as the choice of words used to name the certification.

Consider, for example, the title of “Certified Development Chief” imprinted on the certificate of someone who just took a “Development Chief” course. Who wouldn’t want to be a Certified Development Chief? And if you aren’t a Certified Development Chief, then what kind of Development Chief are you?

A course, a card, and a Bridge to sell.

What significant accomplishment can a developer make by paying to attend (or even actually attending) an 3-5 day course? How credible a recommendation about any of the 20-odd students could an instructor of such a course make? I can answer both questions in two words, with apologies to SNL. “Jack” and “Cheese”. The accomplishment is miniscule, and the recommendation is meaningless.

So as an employer of developers I think you can take your course and card and show it to some other sucker. It don’t impress me much. Go away and come back later when you’ve accomplished the trials of Hercules and apprenticed under Michaelangelo. And make sure you bring letters of reference. NEXT!

The one I said I’d mention later.

I’ve got an idea for a certification program that just might work. Now, bear with me because this is a little complicated. It’s an idea so revolutionary, so different, so incredibly new that I’m getting giddy just thinking about it.

What if we asked young graduates to actually get jobs as “interns” for awhile. What if we gave out responsibility to them in small incremental measures. What if they gradually learned more and more things while working for us. What if we guided them to ever greater accomplishments. What if we slowly gave them trials of Hercules to follows, and had them work with Michaelangelos.

Of course some of our people might leave and go to greener pastures. We’ll need to replace them. Also, our company might succeed and grow, so we’ll need to hire more good developers. So what if we…

OK, now hold on to your hat because this is where it gets really tricky. You might want to get up and walk around a bit, get a diet coke, eat a twinkie, or something to get your brain working.

Ready? OK, here we go…

So what if we… interviewed ... candidate developers? What if we asked them what accomplishments they’d achieved in previous employment. What if we asked them who they used to work with. And… ok, this is even wilder ... what if we called those references and checked them out?

Are you still with me?

What if we….

Naaaaahhhhh. Dumb idea.

Honestly, I think I’ve wasted your time. It’s a stupid idea. Never mind. We should all just go get certified instead.

Software on the Cheap

Uncle Bob — Mon, 01 Feb 2010 16:17:00 -0600

When it comes to software, you get what you pay for.

Have you ever stopped to wonder how much a line of code costs? It ought to be easy to figure out.

In the last 14 months, I have written about 20KSLOC in FitNesse. Of course that was part time. My real job is running Object Mentor, consulting, teaching, mentoring, writing, and a whole load of other things. Programming takes up perhaps 15% of my time.

On the other hand most programmers have lots of other things to do. They go to meetings, and then they go to more meetings. When they are done with those meetings, they go to meetings. And then there are the meetings to go to. Oh yeah, and then there’s all the fiddling around with time accounting tools, and horrific source code control systems that perform like a salamander crawling through frozen mud.

So, maybe 15% isn’t such a bad ratio.

The loaded rate (Salary plus everything else) for a typical programmer is on the order of $200K. (I know that sounds like a lot, but you can look it up.) So $200K / (20KSLOC / 14mo * 12mo) = $11.66/SLOC.

Let’s look at one of those lines: StringBuffer nameBuffer = new StringBuffer(); Does that look like $11.66 to you? Would you pay that much for it? Well, don’t answer yet, because for each StringBuffer line you buy, you get import java.lang.StringBuffer; absolutely free!

Some factories pay their employees a “piece rate”. Would you accept $11.66 per line of code instead of a salary? Of course it couldn’t just be any old line of code. It’d have to be tested!

Hey, I bet all programmers would do TDD if we paid them a piece rate!

Down to business.

The point of that silly analysis was to demonstrate that software is expensive. Even the dumbest little app will likely require more than 1,000 lines of code; and that means it could cost $12K to write!

Imagine that you aren’t a programmer, but you have a clever idea for a new website that’ll make you a zillion dollars. You’ve storyboarded it all out. You’ve worked out all the details. Now all you need is some high-school kid to zip out the code for you. Right? Hell, you could pay him minimum wage! The little twerp would be happy to get it!

That tragic comedy is altogether too common. Too many people have borrowed money against their father’s retirement account to fund a terrible implementation of a good idea. Appalled at how much the reputable firms charge per hour ($100 or more) they go looking for a cheap solution.

“After all, this software is simple.” Or so the reasoning goes. “It’s not like we’re trying to send a rocket to the moon or anything. And, besides, those expensive guys were just out to cheat us. Software just isn’t that hard to write.” Uh huh.

So the poor schmuck finds some freshman in college, or maybe a bored housewife who read a book on HTML last year and created a cute website to show off her kittens. Have these programmers heard about TDD? Have they heard about Design Patterns? Principles? How about source code control?

Clearly they haven’t. They’re going to sling a bunch of horrific code together, without any tests, versioning, or control. The project will start well, with exciting initial results. But then it will slowly grind to a halt, while the cash continues out the door unabated.

In the end the website isn’t going to get built (and the poor schmuck’s father won’t be retiring as soon as he thought). It will be a disaster that will either be terminated, or will require double or triple the investment to get right.

The Bottom Line.

The bottom line is that, when it comes to software, you get what you pay for. If you want good software done well, then you are going to pay for it, and it will probably cost you $12/line or more. And, believe me, that’s the cheapest way to get your software done.

If you go out hunting for the cheap solution, then you’re going to end up paying more, and losing time. Software is one of those things that costs a fortune to write well, and double that to write poorly. If you go for cheap, you’re going to pay double; and maybe even triple.

Mocking Mocking and Testing Outcomes.

Uncle Bob — Sat, 23 Jan 2010 11:32:00 -0600

The number of mocking frameworks has proliferated in recent years. This pleases me because it is a symptom that testing in general, and TDD in particular, have become prevalent enough to support a rich panoply of third-party products.

On the other hand, all frameworks carry a disease with them that I call The Mount Everest Syndrome: “I use it because it’s there.” The more mocking frameworks that appear, the more I see them enthusiastically used. Yet the prolific use of mocking frameworks is a rather serious design smell…

Lately I have seen several books and articles that present TDD through the lens of a mocking framework. If you were a newbie to TDD, these writings might give you the idea that TDD was defined by the use of mocking tools, rather than by the disciplines of TDD.

So when should use use a mocking framework? The answer is the same for any other framework. You use a framework only when that framework will give you a significant advantage.

Why so austere? Why shouldn’t you use frameworks “just because they are there”? Because frameworks always come with a cost. They must be learned by the author, and by all the readers. They become part of the configuration and have to be maintained. They must be tracked from version to version. But perhaps the most significant reason is that once you have a hammer, everything starts to look like a nail. The framework will put you into a constraining mindset that prevents you from seeing other, better solutions.

Consider, for example, this lovely bit of code that I’ve been reviewing recently. It uses the Moq framework to initialize a test double:


var vehicleMock = Mocks.Create<IClientVehicle>()
 .WithPersistentKey()
 .WithLogicalKey().WithLogicalName()
 .WithRentalSessionManager(rsm =>
    {
      var rs = Mocks.Create<IRentalSession>();
      rsm.Setup(o => o.GetCurrentSession()).Returns(rs.Object);
      rsm.Setup(o =>
       o.GetLogicalKeyOfSessionMember(It.IsAny<string>(),
        It.IsAny<int>())).Returns("Rental");
    })
 .AddVehicleMember<IRoadFactory>()
 .AddVehicleMember<IRoadItemFactory>(rf => rf.Setup(t => 
    t.CreateItems(It.IsAny<IRoad>())).Returns(pac))
 .AddVehicleMember<ILegacyCorporateRental>()
 .AddVehicleMember<IRentalStation>(
    m => m.Setup(k => k.Facility.FacilityID).Returns(0))
 .AddVehicleMember<IRoadManager>(m=>
    m.Setup(k=>k.GetRoundedBalanceDue(25,It.IsAny<IRoad>())).Returns(25));

Some of you might think I’m setting up a straw-man. I’m not. I realize that bad code can be written in any language or framework, and that you can’t blame the language or framework for bad code.

The point I am making is that code like this was the way that all unit tests in this application were written. The team was new to TDD, and they got hold of a tool, and perhaps read a book or article, and decided that TDD was done by using a mocking tool. This team is not the first team I’ve seen who have fallen into this trap. In fact, I think that the TDD industry as a whole has fallen into this trap to one degree or another.

Now don’t get me wrong. I like mocking tools. I use them in Ruby, Java, and .Net. I think they provide a convenient way to make test-doubles in situations where more direct means are difficult.

For example, I recently wrote the following unit test in FitNesse using the Mockito framework.


  @Before
  public void setUp() {
    manager = mock(GSSManager.class);
    properties = new Properties();
  }

  @Test
  public void credentialsShouldBeNullIfNoServiceName() throws Exception {
    NegotiateAuthenticator authenticator = 
      new NegotiateAuthenticator(manager, properties);
    assertNull(authenticator.getServerCredentials());
    verify(manager, never()).createName(
      anyString(), (Oid) anyObject(), (Oid) anyObject());
  }

The first line in the setUp function is lovely. It’s kind of hard to get prettier than that. Anybody reading it understands that manager will be a mock of the GSSManager class.

It’s not too hard to understand the test itself. Apparently we are happy to have the manager be a dummy object with the constraint that createName is never called by NegotiateAuthenticator. The anyString() and anyObject() calls are pretty self explanatory.

On the other hand, I wish I could have said this:

assertTrue(manager.createNameWasNotCalled());

That statement does not require my poor readers to understand anything about Mockito. Of course it does require me to hand-roll a manager mock. Would that be hard? Let’s try.

First I need to create a dummy.


  private class MockGSSManager extends GSSManager {
    public Oid[] getMechs() {
      return new Oid[0];
    }

    public Oid[] getNamesForMech(Oid oid) throws GSSException {
      return new Oid[0];
    }

    public Oid[] getMechsForName(Oid oid) {
      return new Oid[0];
    }

    public GSSName createName(String s, Oid oid) throws GSSException {
      return null;
    }

    public GSSName createName(byte[] bytes, Oid oid) throws GSSException {
      return null;
    }

    public GSSName createName(String s, Oid oid, Oid oid1) throws GSSException {
      return null;
    }

    public GSSName createName(byte[] bytes, Oid oid, Oid oid1) throws GSSException {
      return null;
    }

    public GSSCredential createCredential(int i) throws GSSException {
      return null;
    }

    public GSSCredential createCredential(GSSName gssName, int i, Oid oid, int i1) throws GSSException {
      return null;
    }

    public GSSCredential createCredential(GSSName gssName, int i, Oid[] oids, int i1) throws GSSException {
      return null;
    }

    public GSSContext createContext(GSSName gssName, Oid oid, GSSCredential gssCredential, int i) throws GSSException {
      return null;
    }

    public GSSContext createContext(GSSCredential gssCredential) throws GSSException {
      return null;
    }

    public GSSContext createContext(byte[] bytes) throws GSSException {
      return null;
    }

    public void addProviderAtFront(Provider provider, Oid oid) throws GSSException {
    }

    public void addProviderAtEnd(Provider provider, Oid oid) throws GSSException {
    }
  }

“Oh, ick!” you say. Yes, I agree it’s a lot of code. On the other hand, it took me just a single keystroke on my IDE to generate all those dummy methods. (In IntelliJ it was simply command-I to implement all unimplemented methods.) So it wasn’t particularly hard. And, of course, I can put this code somewhere where nobody had to look at it unless they want to. It has the advantage that anybody who knows Java can understand it, and can look right at the methods to see what they are returning. No “special” knowledge of the mocking framework is necessary.

Next, let’s’ make a test double that does precisely what this test needs.


  private class GSSManagerSpy extends MockGSSManager {
    public boolean createNameWasCalled;

    public GSSName createName(String s, Oid oid) throws GSSException {
      createNameWasCalled = true;
      return null;
    }
  }

Well, that just wasn’t that hard. It’s really easy to understand too. Now, let’s rewrite the test.


  @Test
  public void credentialsShouldBeNullIfNoServiceNameWithHandRolledMocks() throws Exception {
    NegotiateAuthenticator authenticator = new NegotiateAuthenticator(managerSpy, properties);
    assertNull(authenticator.getServerCredentials());
    assertFalse(managerSpy.createNameWasCalled);
  }

Well, that test is just a load easier to read than verify(manager, never()).createName(anyString(), (Oid) anyObject(), (Oid) anyObject());.

“But Uncle Bob!” I hear you say. “That scenario is too simple. What if there were lots of dependencies and things…” I’m glad you asked that question, because the very next test is just such a situation.


  @Test
  public void credentialsShouldBeNonNullIfServiceNamePresent() throws Exception {
    properties.setProperty("NegotiateAuthenticator.serviceName", "service");
    properties.setProperty("NegotiateAuthenticator.serviceNameType", "1.1");
    properties.setProperty("NegotiateAuthenticator.mechanism", "1.2");
    GSSName gssName = mock(GSSName.class);
    GSSCredential gssCredential = mock(GSSCredential.class);
    when(manager.createName(anyString(), (Oid) anyObject(), (Oid) anyObject())).thenReturn(gssName);
    when(manager.createCredential((GSSName) anyObject(), anyInt(), (Oid) anyObject(), anyInt())).thenReturn(gssCredential);
    NegotiateAuthenticator authenticator = new NegotiateAuthenticator(manager, properties);
    Oid serviceNameType = authenticator.getServiceNameType();
    Oid mechanism = authenticator.getMechanism();
    verify(manager).createName("service", serviceNameType, mechanism);
    assertEquals("1.1", serviceNameType.toString());
    assertEquals("1.2", mechanism.toString());
    verify(manager).createCredential(gssName, GSSCredential.INDEFINITE_LIFETIME, mechanism, GSSCredential.ACCEPT_ONLY);
    assertEquals(gssCredential, authenticator.getServerCredentials());
  }

Now I’ve got three test doubles that interact with each other; and I am verifying that the code under test is manipulating them all correctly. I could create hand-rolled test doubles for this; but the wiring between them would be scattered in the various test-double derivatives. I’d also have to write a significant number of accessors to get the values of the arguments to createName and createCredential. In short, the hand-rolled test-double code would be harder to understand than the Mockito code. The Mockito code puts the whole story in one simple test method rather than scattering it hither and yon in a plethora of little derivatives.

What’s more, since it’s clear that I should use a mocking framework for this test, I think I should be consistent and use if for all the tests in this file. So the hand-rolled MockGSSManager and ManagerSpy are history.

“But Uncle Bob, aren’t we always going to have dependencies like that? So aren’t we always going to have to use a mocking framework?”

That, my dear reader, is the real point of this blog. The answer to that salient questions is a profound: “No!“

Why did I have to use Mockito for these tests? Because the number of objects in play was large. The module under test (NegotiateAuthenticator) used GSSName, GSSCredential, and GSSManager. In other words the coupling between the module under test and the test itself was high. (I see lightbulbs above some of your heads.) That’s right, boys and girls, we don’t want coupling to be high!

It is the high coupling between modules and tests that creates the need for a mocking framework. This high coupling is also the cause of the dreaded “Fragile Test” problem. How many tests break when you change a module? If the number is high, then the coupling between your modules and tests in high. Therefore, I conclude that those systems that make prolific use of mocking frameworks are likely to suffer from fragile tests.

Of the 277 unit test files in FitNesse, only 11 use Mockito. The reason for small number is two-fold. First, we test outcomes more often than we test mechanisms. That means we test how a small group of classes behaves, rather than testing the dance of method calls between those classes. The second reason is that our test doubles have no middle class. They are either very simple stubs and spies or they are moderately complex fakes.

Testing outcomes is a traditional decoupling technique. The test doesn’t care how the end result is calculated, so long as the end result is correct. There may be a dance of several method calls between a few different objects; but the test is oblivious since it only checks the answer. Therefore the tests are not strongly coupled to the solution and are not fragile.

Keeping middle-class test doubles (i.e. Mocks) to a minimum is another way of decoupling. Mocks, by their very nature, are coupled to mechanisms instead of outcomes. Mocks, or the setup code that builds them, have deep knowledge of the inner workings of several different classes. That knowledge is the very definition of high-coupling.

What is a “moderately complex fake” and why does it help to reduce coupling? One example within FitNesse is MockSocket. (The name of this class is historical. Nowadays it should be called FakeSocket.) This class derives from Socket and implements all its methods either to remember what was sent to the socket, or to allow a user to read some canned data. This is a “fake” because it simulates the behavior of a socket. It is not a mock because it has no coupling to any mechanisms. You don’t ask it whether it succeeded or failed, you ask it to send or recieve a string. This allows our unit tests to test outcomes rather than mechanisms.

The moral of this story is that the point at which you start to really need a mocking framework is the very point at which the coupling between your tests and code is getting too high. There are times when you can’t avoid this coupling, and those are the times when mocking frameworks really pay off. However, you should strive to keep the coupling between your code and tests low enough that you don’t need to use the mocking framework very often.

You do this by testing outcomes instead of mechanisms.

Dependency Injection Inversion

Uncle Bob — Sun, 17 Jan 2010 12:42:00 -0600

Dependency Injection is all the rage. There are several frameworks that will help you inject dependencies into your system. Some use XML (God help us) to specify those dependencies. Others use simple statements in code. In either case, the goal of these frameworks is to help you create instances without having to resort to new or Factories.

I think these frameworks are great tools. But I also think you should carefully restrict how and where you use them.

Consider, for example, this simple example using Google’s Guice framework.


 public class BillingApplication {
   public static void main(String[] args) {
    Injector injector = Guice.createInjector(new BillingModule());
    BillingService billingService = injector.getInstance(BillingService.class);
    billingService.processCharge(2034, "Bob");
  }
 }

My goal is to create an instance of BillingService. To do this, I first get an Injector from Guice. Then I use the injector to get an instance of my BillingService class. What’s so great about this? Well, take a look at the constructor of the BillingService class.


class BillingService {
  private CreditCardProcessor processor;
  private TransactionLog transactionLog;

  @Inject
  BillingService(CreditCardProcessor processor, TransactionLog transactionLog) {
    this.processor = processor;
    this.transactionLog = transactionLog;
  }

  public void processCharge(int amount, String id) {
    boolean approval = processor.approve(amount, id);
    transactionLog.log(
      String.format("Transaction by %s for %d %s",
      id, amount, approvalCode(approval)));
  }

  private String approvalCode(boolean approval) {
    return approval?"approved":"denied";
  }
}

Oh ho! The BillingService constructor requires two arguments! A CreditCardProcessor and a TransactionLog. How was the main program able to create an instance of BillingService without those two arguments? That’s the magic of Guice (and of all Dependency Injection frameworks). Guice knows that the BillingService needs those two arguments, and it knows how to create them. Did you see that funky @Inject attribute above the constructor? That’s how it got connected into Guice.

And here’s the magic module that tells Guice how to create the arguments for the BillingService


public class BillingModule extends AbstractModule {
  protected void configure() {
    bind(TransactionLog.class).to(DatabaseTransactionLog.class);
    bind(CreditCardProcessor.class).to(MyCreditCardProcessor.class);
  }
}

Clever these Google-folk! The two bind functions tell Guice that whenever we need an instance of a TransactionLog it should use an instance of DatabaseTransactionLog. Whenever it needs a CreditCardProcessor it should use an instance of MyCreditCardProcessor.

Isn’t that cool! Now you don’t have to build factories. You don’t have to use new. You just tell Guice how to map interfaces to implementations, and which constructors to inject those implementations in to, and then call Injector.getInstance(SomeClass.class); and voila! You have your instance automatically constructed for you. Cool.

Well, yes it’s cool. On the other hand, consider this code:


public class BillingApplicationNoGuice {
  public static void main(String[] args) {
    CreditCardProcessor cp = new MyCreditCardProcessor();
    TransactionLog tl = new DatabaseTransactionLog();
    BillingService bs = new BillingService(cp, tl);
    bs.processCharge(9000, "Bob");
  }
}

Why is this worse? It seems to me it’s better.

But Uncle Bob, you’ve violated DIP by creating concrete instances!

True, but you have to mention concrete instances somewhere. main seems like a perfectly good place for that. Indeed, it seems better than hiding the concrete references in BillingModule.

I don’t want a bunch of secret modules with bind calls scattered all around my code. I don’t want to have to hunt for the particular bind call for the Zapple interface when I’m looking at some module. I want to know where all the instances are created.

But Uncle Bob, You’d know where they are because this is a Guice application.

I don’t want to write a Guice application. Guice is a framework, and I don’t want framework code smeared all through my application. I want to keep frameworks nicely decoupled and at arms-length from the main body of my code. I don’t want to have @Inject attributes everywhere and bind calls hidden under rocks.

But Uncle Bob, What if I want to get an instance of BillingService from deep in the bowels of my application? With Guice I can just say injector.getInstance(BillingService.class);.

True, but I don’t want to have createInstance calls scattered all through my code. I don’t want Guice to be poured all over my app. I want my app to be clean, not soaked in Guice.

But Uncle Bob, That means I have to use new or factories, or pass globals around.

You think the injector is not a global? You think BillingService.class is not a global? There will always be globals to deal with. You can’t write systems without them. You just need to manage them nicely.

And, no, I don’t have to use new everywhere, and I don’t need factories. I can do something as simple as:


public class BillingApplicationNoGuice {
  public static void main(String[] args) {
    CreditCardProcessor cp = new MyCreditCardProcessor();
    TransactionLog tl = new DatabaseTransactionLog();
    BillingService.instance = new BillingService(cp, tl);

    // Deep in the bowels of my system.
    BillingService.instance.processCharge(9000, "Bob");
  }
}

But Uncle Bob, what if you want to create many instances of BillingService rather than just that one singleton?

Then I’d use a factory, like so:


public class BillingApplication {
   public static void main(String[] args) {
    Injector injector = Guice.createInjector(new BillingModule());
    BillingService.factory = new BillingServiceFactory(injector);

    // Deep in the bowels of my code.
    BillingService billingService = BillingService.factory.make();
    billingService.processCharge(2034, "Bob");
  }
}

But Uncle Bob, I thought the whole idea was to avoid factories!

Hardly. After all, Guice is just a big factory. But you didn’t let me finish. Did you notice that I passed the Guice injector into the factory? Here’s the factory implementation.


public class BillingServiceFactory extends AbstractModule {
  private Injector injector;

  public BillingServiceFactory(Injector injector) {
    this.injector = injector;
  }

  protected void configure() {
    bind(TransactionLog.class).to(DatabaseTransactionLog.class);
    bind(CreditCardProcessor.class).to(MyCreditCardProcessor.class);
  }

  public BillingService make() {
    return injector.getInstance(BillingService.class);
  }
}

I like this because now all the Guice is in one well understood place. I don’t have Guice all over my application. Rather, I’ve got factories that contain the Guice. Guicey factories that keep the Guice from being smeared all through my application.

What’s more, if I wanted to replace Guice with some other DI framework, I know exactly what classes would need to change, and how to change them. So I’ve kept Guice uncoupled from my application.

Indeed, using this form allows me to defer using Guice until I think it’s necessary. I can just build the factories the good old GOF way until the need to externalize dependencies emerges.

But Uncle Bob, don’t you think Dependency Injection is a good thing?

Of course I do. Dependency Injection is just a special case of Dependency Inversion. I think Dependency Inversion is so important that I want to invert the dependencies on Guice! I don’t want lots of concrete Guice dependencies scattered through my code.

BTW, did you notice that I was using Dependency Injection even when I wasn’t using Guice at all? This is nice and simple manual dependency injection. Here’s that code again in case you don’t want to look back:



public class BillingApplicationNoGuice {
  public static void main(String[] args) {
    CreditCardProcessor cp = new MyCreditCardProcessor();
    TransactionLog tl = new DatabaseTransactionLog();
    BillingService bs = new BillingService(cp, tl);
    bs.processCharge(9000, "Bob");
  }
}

Dependency Injection doesn’t require a framework; it just requires that you invert your dependencies and then construct and pass your arguments to deeper layers. Consider, for example, that the following test works just fine in all the cases above. It does not rely on Guice, it only relies on the fact that dependencies were inverted and can be injected into BillingService



public class BillingServiceTest {
  private LogSpy log;

  @Before
  public void setup() {
    log = new LogSpy();
  }

  @Test
  public void approval() throws Exception {
    BillingService bs = new BillingService(new Approver(), log);
    bs.processCharge(9000, "Bob");
    assertEquals("Transaction by Bob for 9000 approved", log.getLogged());
  }

  @Test
  public void denial() throws Exception {
    BillingService bs = new BillingService(new Denier(), log);
    bs.processCharge(9000, "Bob");
    assertEquals("Transaction by Bob for 9000 denied", log.getLogged());    
  }
}

class Approver implements CreditCardProcessor {
  public boolean approve(int amount, String id) {
    return true;
  }
}

class Denier implements CreditCardProcessor {
  public boolean approve(int amount, String id) {
    return false;
  }
}

class LogSpy implements TransactionLog {
  private String logged;

  public void log(String s) {
    logged = s;
  }

  public String getLogged() {
    return logged;
  }
}

Also notice that I rolled my own Test Doubles (we used to call them mocks, but we’re not allowed to anymore.) It would have been tragic to use a mocking framework for such a simple set of tests.

Most of the time the best kind of Dependency Injection to use, is the manual kind. Externalized dependency injection of the kind that Guice provides is appropriate for those classes that you know will be extension points for your system.

But for classes that aren’t obvious extension points, you will simply know the concrete type you need, and can create it at a relatively high level and inject it down as an interface to the lower levels. If, one day, you find that you need to externalize that dependency, it’ll be easy because you’ve already inverted and injected it.

Saying "NO".

Uncle Bob — Fri, 04 Dec 2009 08:35:55 -0600

I saw this cartoon in a tweet today. It’s the story of how a boss corrupts the work of a professional. It’s a funny cartoon, and a sad story that happens all too often in our profession. But who, exactly, was at fault?...

The difference between a laborer and a professional is that a laborer takes orders from his boss, and a professional provides input to his superiors. Laborers are hired to take direction. Professionals are hired to ensure that the direction chosen makes sense.

Imagine this conversation between a patient and a doctor:

Patient: “My arm hurts.” Doctor: “What would you like me to do about it?” Patient: “Make my arm stop hurting.” Doctor: “Do you want me to cut it off?, I can do that.” Patient: “No, I just want it to stop hurting.” Doctor: “I could cut all the nerves to your arm. That’ll stop it.” Patient: “Isn’t there something less drastic you could do?” Doctor: “Ooops, sorry, time for my break.”

Clearly we don’t expect doctors to behave this way. Even though the patient is the boss, the patient expects the doctor to have the answers and help set the direction.

Here’s another version of the conversation:

Patient: “I want you to cut my arm off.” Doctor: “What’s wrong with your arm?” Patient: “It hurts. I’m tired of it. Just cut it off.” Doctor: “Let me see your arm. Hmmm. Looks like you’ve got a sprain or perhaps a hairline fracture. We should take some X-Rays.” Patient: “No, just cut it off.” Doctor: “Sir, I do not cut off healthy arms.” Patient: “But I’m paying you. You have to do what I say!” Doctor: “No, sir, I don’t. Cutting off your arm would violate my oath.”

Which of these two doctors would you rather be? Now project these two doctors into your own profession, and which would you rather be?

Programmers are professionals. They know more about designing and implementing software systems than their bosses do. Indeed, they are hired for this knowledge and expertise. And they have a solemn duty to prevent their managers from doing things that would be harmful.

All this boils down to one simple thing. Professionals are willing to say “No”. When their managers come to them with direction that makes no sense, a professional programmer will refuse the direction.

Is this risky? Sure. But part of being a professional is the willingness to stand on principle. There are lines that a professional will not cross.

Of course saying “No.” is only one side of the coin. Professionals are also expected to explain their positions, and come up with viable alternatives. Professionals negotiate with their superiors until both parties are satisfied with the chosen direction.

The poor web-designer schmuck in that cartoon was not behaving as a professional. He was behaving as a laborer. The fiasco at the end was his fault. He should have said “No.” and started a negotiation with his customer instead of just doing everything the customer said.

The cartoonist painted the web-designer as a wise but impotent victim, and the boss as the overbearing dufus. The reality is that the web-designer took the role of the victim voluntarily and shirked his responsibility to refuse direction that he considered harmful.

If you are a professional, you never allow yourself to be put in the role of the victim.

What's all this nonsense about Katas?

Uncle Bob — Sat, 21 Nov 2009 04:32:00 -0600

There has been an increase in interest in Software Katas of late. What is all this buzz about, and why might it be important?

Several years ago (Pragmatic) Dave Thomas started a special blog about coding katas. The idea was simple: Professionals practice.

This thought might not have occurred to you before; but it’s self-evidently true. Professional musicians practice. Professional dancers practice. Doctors practice. Soldiers practice. Indeed, any kind of professional craftsman or artisan must practice their adopted trade in order to execute it well when it counts.

Dave’s point was that professional programmers need to practice like any other professional. He set forth a number of simple exercises and problems that programmers could solve during their practice time. He also suggested that practice time needs to be scheduled and protected as part of the normal routine of a software developer.

At XP2005 in Shefield, UK, I attended Laurent Bossavit’s and Emmanuel Gaillot’s Coding Dojo session. I blogged about it at the time. I left that session with a different view of what a Kata might be. In Dave Thomas’ view a kata is an exercise to solve. Each time you solve it you try a different approach. Dave’s notion is that a kata is a way to explore the solution space of a problem. But in Martial arts a kata is something different; it is a set of motions that one memorizes in minute detail. A martial arts kata is a precisely executed dance. I left Sheffield with the idea that a coding kata could be more like a martial arts kata.

About a year ago, Corey Haines (our wandering minstrel of code) stopped by and told me about some mini-conferences that had adopted the notion of coding kata. What he described sounded more like a group dance than a bunch of people independently solving a given problem. This got me thinking. So at the next Software Craftsmanship meeting at 8th Light (Which, not coincidentally, takes it’s name from a martial arts philosophy) I conducted a simple kata session in dojo style.

I, acting as the Sen Sei, had everyone file in to the room as they would to a dojo. I had them put their green bands on their wrists (mine was black!). I bade them to sit, and then gestured for silence. For the next 15 minutes I led them through the Prime Factors Kata in Java without uttering a word (well, almost). I would write a simple test case, then gesture for them to do the same. Then I would make the test pass, and again gesture for them to repeat my moves.

Then, this Fall, I attended Programming with the Stars at Agile 2009. As I watched the contestants perform their programming sessions, I realized that they were not simply demonstrating programming prowess; indeed they were competing in how well they performed their session. That’s when it struck me. This was a performance art!

It seems absurd to think of programming as a performance art. And yet watching those people up on stage working through routines that they had clearly practiced over and over again, was delightfully entertaining to me, and to the crowd. There were cheers and applause when a performing pair did something especially well. There was laughter when a pair executed a clever joke in code, or in behavior. For programmers, watching other programmers perform well practiced routines was fun!

Now I had seen martial arts katas executed as performance art during competitions. Indeed, it is a beautiful thing to watch a skilled and well practiced martial artist peform a kata. And that started me thinking. Why would a martial artist practice these forms so intensely that they could be performed as an art?

The goal of martial arts is not to perform on stage. The goal of martial arts is self defense! And yet, when a martial artist is at the peak of his skill, his practice takes on the quality of a performance art. When you watch a skilled martial artist perform, you know you don’t want to fight him. The performance is a demonstration, and a realization, of mastery.

And yet the performance is not the goal. No martial artist practices his art so that they can perform on stage. A martial artist practices to achieve personal perfection in the art of self defense. The fact that the practice can be performed is a (pleasant) side effect.

Could it be that these coding kata that we’ve been dabbling with for the last few years could be performed? Is there a benefit to practicing them so much that performance is an option? Is there really any reason for a programmer to create and then memorize a routine down to the level of individual keystrokes? Is there profit to be gained from practicing that routine so thoroughly that you can do it at high speed with virtually no errors? This August I decided to find out.

I started with the Prime Factors Kata in Java, and set it to music inspired by this video of dancing kites. The piece is about 4 minutes long, and I figured I’d need about twice that, so I found two different version of the music and played them back to back. But working in Java is ponderous, and I could not complete the kata before the two pieces had finished. So I switched to Ruby. Using Ruby and Rspec I was able to finish the Kata with plenty of time to spare.

Thereupon began an evolution. Since I had time, I added more features to the kata. As I improved in my practice I found I had even more time, so I added even more features. I refined and polished. I refactored my refactorings. I trimmed keystrokes, and learned many new keyboard shortcuts in the process. I completely abandoned the mouse; and then later reacquired it for flourishes and highlights (but never for actual coding).

Week after week, I set aside 30 minutes or so every day to practice, and practice, and practice—just the way I used to practice my Jiu Jitsu Katas. I would practice on airplanes while flying to clients. I would practice in bed with my laptop in my lap. I would practice late at night in hotel rooms after a long day consulting for clients. I would practice early in the morning before shower and breakfast. And the practice started to pay off.

Ten weeks later I finally decided I was ready to create a screencast. I reduced my screen resolution to 1024X768. I configured SnapZ ProX to record the whole screen at 80% size. And I began to record.

Performing a coding kata in time to music is a very difficult thing to do. Maintaining the timeline is critical. There are very few errors that you can effectively recover from. If you miss a stroke, the whole sequence melts down. And melt down it did. Over and over and over again. I spent weeks trying to record a reasonably good session. I must have done hundreds of takes. It was very frustrating.

But day by day I got better and better at it. Finally, after weeks of trying, I got what I consider to be a recording that, though not perfect, is good enough to present. You can watch that recording here.

Has this paid off in the regular programming I do? I think it has. My mouse usage is much less than it used to be. I know a much larger number of keyboard shortcuts than I used to, and I can use them almost instinctively. My typing and accuracy have improved quite a bit, and I now type more punctuation and number keys without looking. So, yes, I think the effort has paid off; though I’m not at all sure the payoff compensates for the effort.

But something else compensates for the effort. Making this recording was fun—it was a lot of fun. And that’s probably the real answer to the question posed in the title of this blog.

Archeological Dig

Uncle Bob — Wed, 11 Nov 2009 10:39:00 -0600

I was going through some old files today, and I stumbled upon some acetate slides from 1995. They were entitled: “Managing OO Projects”. Wow! What a difference fifteen years makes! (Or does it?) ...

In 1995-99 I was frequently asked to speak to managers about what a transition to OO (usually from C to C++) would do for (or to) them. I would spend a half day to a day going over the issues, costs, and benefits.

One part of that talk (usually about 90 min) was a discussion about software process. It was the process part of the talk that those acetate slides that I found described.

1995 was during the ascendency of Waterfall. Waterfall thinking was king. RUP had not yet been conceived as an acronym. And though Booch was beating the drum for incrementalism, most people (even many within Rational) were thinking in terms of six to eighteen month waterfalls.

So, here are the slides that I uncovered deep within an old filing cabinet. I scanned them in. They were produced on a Macintosh using the old “More” program. (Where is that program now? It was so good.)

Go ahead and read them now. Then come back here and continue…

What struck me about those slides was the consistency of the message with today. It was all about iterative development. Small iterations (though I never deigned to define the length in the slides, I frequently told people 2 weeks), measured results, etc. etc. Any Agile evangelist could use those slides today. He or she would have to dance quickly around a few statements, but overall the message has not shifted very much.

What’s even more interesting is the coupling between the process, and OO. The slides talk a lot about dependency management and dependency structure. There are hints of the SOLID principles contained in those slides. (Indeed several of the principles had already been identified by that time.) This coupling between process and software structure was a harbinger of the current craftsmanship/clean-code movement.

Of course the one glaring omission from these slides is TDD. That makes me think that TDD was the true catalyst of change, and the bridge that conveyed our industry from then to now.

Anyway, I guess the more things change, the more they stay the same.

Comments please!

Excuse me sir, What Planet is this?

Uncle Bob — Thu, 05 Nov 2009 10:35:00 -0600

Update 12 hours later.

I’m not very proud of this blog (or as one commenter correctly called it “blart”). It is derisive, sneering, and petulant. It is unprofessional. I guess I was having a bad morning. I slipped. I didn’t check with my green band.

So I apologize to the audience at large, and to Cashto. You should expect better from me.

I thought about pulling the blog down; but I think I’ll leave it up here as an example of how not to write a blog.

Some folks on twitter have been asking me to respond to this blough (don’t bother to read it right now, I’ll give you the capsule summary below. Read it later if you must). It’s a typical screed complete with all the usual complaints, pejoratives, and illogic. Generally I don’t respond to blarts like this because I don’t imagine that any readers take them very seriously. But it appears that this blelch has made the twitter rounds and that I’m going to have to say something.

Here are the writer’s main points:

He likens unit tests to training wheels and says you can’t use them to win the Tour de France.
- I think winning the Tour de France has much more to do with self-discipline than he imagines it does. I mean it’s not really just as simple as: “Get on a bike and ride like hell!”

He says testing is popular amongst college students
- I’d like to see his data!

He goes on to say: (cue John Cleese): “(ahem) unit tests lose their effectiveness around level four of the Dreyfus model of skill acquisition”.
- (blank stunned stare). Is this a joke? All false erudition aside, WTF is he talking about?

He says that unit tests don’t give us confidence in refactoring because they over-specify behavior and are too fine-grained.
- He apparently prefers hyphenations to green bars.

He says they mostly follow the “happy path” and therefore don’t find bugs.
- Maybe when he writes them! This is a big clue that the author can’t spell TDD.

He complains about Jester
- without getting the joke!

He says unit tests “encourage some pretty questionable practices.” He flings a few design principles around and says that unit testing doesn’t help with them.
- as the author, editor, and/or advocate of many of those principles; I have a slightly different view.

He says that “many are starting to discover that functional programming teaches far better design principles than unit testing ever will”
- Oh no! Not the old “My language teaches design.” claim. We’ve heard it all before. They said it about C++, Java, COM (?!), etc… The lesson of the ‘90s? Languages don’t teach design. You can make a mess in any language.

He says: “tests can have negative ROI. Not only do they cost a lot to write, they’re fragile, they’re always broken, they get in the way of your refactoring, they’re always having you chase down bogus failures, and the only way to get anything done is to ignore them.”
- In one of my favorite episodes of Dr. Who, Tom Baker exits the Tardis, walks up to someone on the street and says: “Excuse me sir, what planet is this?”

He says: “What I’m saying is that it’s okay if you don’t write unit tests for everything. You probably have already suspected this for a long time, but now you know. I don’t want you to feel guilty about it any more.”
- Translation: “I don’t want to feel guilty about it anymore so I’m going to try to convince you…” I sincerely doubt this author has your best interests at heart.

He says: “Debugging is easy, at least in comparison to writing all those tedious tests.”
- Refer back to the Dr. Who quote.

To quote Barack Obama: “Enough!”

Has this guy ever done TDD? I rather doubt it. Or if he did, he was so inept at it that his tests were “fragile”, “always broken”, and “in the way of refactoring”. I think he should give it another try and this time spend a bit more time on test design.

Perhaps he’s one of those guys who thought that unit tests were best written after the code. Certainly his list of complains makes a lot of sense in that light. Hint: If you want to fail at unit testing, write them last.

The bottom line is that the guy probably had a bad experience writing unit tests. He’s tired of writing them and wants to write fewer of them. He’d rather debug. He thinks he can refactor without tests (which is definitively false). He thinks he can go faster by writing fewer tests. Fine, that’s his choice. And he’s found a rationalization to support his desires. Great.

I predict that his results will not compare well with those who adopt the discipline of TDD. I predict that after a few years he’ll either change his mind, or go into management.

Oh, and to the author: Gesundheit!

Manual Mocking: Resisting the Invasion of Dots and Parentheses

Uncle Bob — Wed, 28 Oct 2009 18:12:16 -0500

The twittersphere has been all abuzz today because of something I tweeted early this morning (follow @unclebobmartin). In my tweet I said that I hand-roll most of my own mock objects in Java, rather than using a mocking framework like mockito.

The replies were numerous and vociferous. Dave Astels poignantly stated that hand-rolling mocks is so 2001!

So why do I roll my own mocks?

Consider the following two tests:

public class SelectorTest {
  private List<Object> list;

  @Before
  public void setup() {
    list = new ArrayList<Object>();
    list.add(new Object());
  }

  @Test
  public void falseMatcherShouldSelectNoElements_mockist() {
    Matcher<Object> falseMatcher = mock(Matcher.class);
    Selector<Object> selector = new Selector<Object>(falseMatcher);
    when(falseMatcher.match(anyObject())).thenReturn(false);
    List<Object> selection = selector.select(list);
    assertThat(selection.size(), equalTo(0));
  }

  @Test
  public void falseMatcherShouldSelectNoElements_classic() {
    Matcher<Object> falseMatcher = new FalseMatcher();
    Selector<Object> selector = new Selector<Object>(falseMatcher);
    List<Object> selection = selector.select(list);
    assertThat(selection.size(), equalTo(0));}

  private static class FalseMatcher implements Matcher<Object> {
    public boolean match(Object element) {
      return false;
    }
  }
}

The first test shows the really cool power of mockito (which is my current favorite in the menagerie of java mocking frameworks). Just in case you can’t parse the syntax, let me describe it for you:

falseMatcher is assigned the return value of the “mock” function. This is a very cool function that takes the argument class and builds a new stubbed object that derives from it. In mockito, the argument can be a class or an interface. Cool!

Now don’t get all panicy about the strange parenthetic syntax of the ‘when’ statement. The ‘when’ statement simply tells the mock what to do when a method is called on it. In this case it instructs the falseMatcher to return false when the ‘match’ function is called with any object at all.

The second test needs no explanation.

...

And that’s kind of the point. Why would I include a bizzare, dot-ridden, parentheses-laden syntax into my tests, when I can just as easily hand-roll the stub in pure and simple java? How hard was it to hand-roll that stub? Frankly, it took a lot less time and effort to hand-roll it than it took to write the (when(myobj.mymethod(anyx())).)()).))); statement.

OK, I’m poking a little fun here. But it’s true. My IDE (InteliJ) generated the stub for me. I simply started with:

Matcher<Object> falseMatcher = new Matcher<Object>() {};

InteliJ complained that some methods weren’t implemented and offered to implement them for me. I told it to go ahead. It wrote the ‘match’ method exactly as you see it. Then I chose “Convert Anonymous to Inner…” from the refactoring menu and named the new class FalseMatcher. Voila! No muss, no fuss, no parenthetic maze of dots.

Now look, I’m not saying you shouldn’t use mockito, or any of these other mocking tools. I use them myself when I must. Here, for example, is a test I wrote in FitNesse. I was forced to use a mocking framework because I did not have the source code of the classes I was mocking.

  @Before
  public void setUp() {
    manager = mock(GSSManager.class);
    properties = new Properties();
  }

  @Test
  public void credentialsShouldBeNonNullIfServiceNamePresent() throws Exception {
    properties.setProperty("NegotiateAuthenticator.serviceName", "service");
    properties.setProperty("NegotiateAuthenticator.serviceNameType", "1.1");
    properties.setProperty("NegotiateAuthenticator.mechanism", "1.2");
    GSSName gssName = mock(GSSName.class);
    GSSCredential gssCredential = mock(GSSCredential.class);
    when(manager.createName(anyString(), (Oid) anyObject(), (Oid) anyObject())).thenReturn(gssName);
    when(manager.createCredential((GSSName) anyObject(), anyInt(), (Oid) anyObject(), anyInt())).thenReturn(gssCredential);
    NegotiateAuthenticator authenticator = new NegotiateAuthenticator(manager, properties);
    Oid serviceNameType = authenticator.getServiceNameType();
    Oid mechanism = authenticator.getMechanism();
    verify(manager).createName("service", serviceNameType, mechanism);
    assertEquals("1.1", serviceNameType.toString());
    assertEquals("1.2", mechanism.toString());
    verify(manager).createCredential(gssName, GSSCredential.INDEFINITE_LIFETIME, mechanism, GSSCredential.ACCEPT_ONLY);
    assertEquals(gssCredential, authenticator.getServerCredentials());
  }

If I’d had the source code of the GSS classes, I could have created some very simple stubs and spies that would have allowed me to make these tests a lot cleaner than they currently appear. Indeed, I might have been able to test the true behavior of the classes rather than simply testing that I was calling them appropriately…

Mockism

That last bit is pretty important. Some time ago Martin Fowler wrote a blog about the Mockist and Classical style of TDD. In short, Mockists don’t test the behavior of the system so much as they test that their classes “dance” well with other classes. That is, they mock/stub out all the other classes that the class under test uses, and then make sure that all the right functions are called in all the right orders with all the right arguments. etc. There is value to doing this in many cases. However you can get pretty badly carried away with the approach.

The classical approach is to test for desired behavior, and trust that if the test passes, then the class being tested must be dancing well with its partners.

Personally, I don’t belong to either camp. I sometimes test the choreography, and I sometimes test the behavior. I test the choreography when I am trying to isolate one part of the system from another. I test for the behavior when such isolation is not important to me.

The point of all this is that I have observed that a heavy dependence on mocking frameworks tends to tempt you towards testing the dance when you should be testing behavior. Tools can drive the way we think. So remember, you dominate the tool; don’t let the tool dominate you!

But aren’t hand-rolled mocks fragile?

Yes, they can be. If you are mocking a class or interface that it very volatile (i.e. you are adding new methods, or modifying method signatures a lot) then you’ll have to go back and maintain all your hand-rolled mocks every time you make such a change. On the other hand, if you use a mocking framework, the framework will take care of that for you unless one of the methods you are specifically testing is modified.

But here’s the thing. Interfaces should not usually be volatile. They should not continue to grow and grow, and the methods should not change much. OK, I realize that’s wishful thinking. But, yes, I wish for the kind of a design in which interfaces are the least volatile source files that you have. That’s kind of the point of interfaces after all… You create interfaces so that you can separate volatile implementations from non-volatile clients. (Or at least that’s one reason.)

So if you are tempted to use a mocking framework because you don’t want to maintain your volatile interfaces, perhaps you should be asking yourself the more pertinent question about why your interfaces are so volatile.

Still, if you’ve got volatile interfaces, and there’s just no way around it, then a mocking framework may be the right choice for you.

So here’s the bottom line.

It’s easy to roll your own stubs and mocks. Your IDE will help you and they’ll be easier and more natural to read than the dots and parentheses that the mocking frameworks impose upon you.

Mocking frameworks drive you towards testing choreography rather than behavior. This can be useful, but it’s not always appropriate. And besides, even when you are testing choreography, the hand-rolled stubs and mocks are probably easier to write and read.

There are special cases where mocking tools are invaluable, specifically when you have to test choreography with objects that you have no source for or when your design has left you with a plethora of volatile interfaces.

Am I telling you to avoid using mocking frameworks? No, not at all. I’m just telling you that you should drive tools, tools should not drive you.

If you have a situation where a mocking tool is the right choice, by all means use it. But don’t use it because you think it’s “agile”, or because you think it’s “right” or because you somehow think you are supposed to. And remember, hand-rolling often results in simpler tests without the litter of dots and parentheses!

We must ship now and deal with consequences

Uncle Bob — Thu, 15 Oct 2009 06:17:00 -0500

Martin Fowler has written a good blog about technical debt. He suggests that there are two axes of debt: deliberate and prudent. This creates four quadrants: deliberate-prudent, deliberate-imprudent, inadvertent-prudent, and inadvertent-imprudent. I agree with just about everything in his blog except for one particular caption…

Inadvertent-Imprudent Debt.

There is more of this debt than any other kind. It is all too common that software developers create a mess and don’t know they are doing it. They have not developed a nose that identifies code smells. They don’t know design principles, or design patterns. They think that the reek of rotten code is normal, and don’t even identify it as smelling bad. They think that their slow pace through the thick morass of tangled code is the norm, and have no idea they could move faster. These people destroy projects and bring whole companies to their knees. Their name is Doom.

Deliberate-Imprudent Debt.

There is a meme in our industry (call it the DI meme) that tells young software developers that rushing to the finish line at all costs is the right thing to do. This is far worse than the ignorance of the first group because these folks willfully create debt without counting the cost. Worse, this meme is contagious. People who are infected with it tend to infect others, causing an epidemic of deliberately imprudent debtors (sound familiar?) The end result, as we are now all know, is economic catastrophe, inflation (of estimates) and crushing interest (maintenance) payments. They have become death, the destroyer of worlds.

Inadvertent-Prudent Debt.

This is something of an oxymoron. Ironically, it is also the best of all possible states. The fact is that no matter how careful we are, there is always a better solution that we will stumble upon later. How many times have you finished a system only to realize that if you wrote it again, you’d do it very differently, and much better?

The result is that we are always creating a debt, because our hindsight will always show us a better option after it is too late. So even the best outcome still leaves us owing. (Mother Earth will eventually collect that debt!)

Deliberate-Prudent Debt.

This is the quadrant that I have the biggest problem with. And it is this quadrant in which Martin uses the caption I don’t like. The Caption is: “We must ship now and deal with consequences.”

Does this happen? Yes. Should it happen? Rarely, yes. But it damned well better not happen very often, and it damned well better not happen out of some misplaced urge to get done without counting the cost.

The problem I have with this quadrant (DP) is that people who are really in quadrant DI think they are in DP, and use words such as those that appear in the caption as an excuse to rack up a huge imprudent debt.

The real issue is the definition of the word: Imprudent.

So let me ask you a question. How prudent is debt? There is a very simple formula for determining whether debt is prudent or imprudent. You can use this formula in real life, in business, and in programming. The formula is: Does the debt increase your net worth, and can you make the payments?

People often focus on the first criterion, without properly considering the second. Buying a house is almost certain to increase your net worth despite the debt (though lately…). On the other hand, if you cannot make the payments, you won’t keep that house for long. The reason for our current economic woes has a lot to do with people trying to increase their net worth despite the fact that they couldn’t afford the payments. (indeed, they were encouraged by a meme very similar to the DI meme!)

Bad code is always imprudent.

Writing bad code never increases your net worth; and the interest rate is really high. People who write bad code are like those twenty-somethings who max out all their credit cards. Every transaction decreases net worth, and has horrendous consequences for cash flow. In the end, the vast bulk of your effort goes to paying the interest (the inevitable slow down of the team as they push the messes around). Paying down the principle becomes infeasible. (Just the way credit card companies like it.)

Some Suboptimal Design Decision are Prudent Debt.

But most are not. Every once in awhile there is a suboptimal design decision that will increase the net worth of the project by getting that project into customer’s hand’s early.

This is not the same as delivering software that is under-featured. It is often prudent to increase the net worth of a project by giving customers early access to a system without a full and rich feature set. This is not debt. This is more like a savings account that earns interest.

Indeed, this is one reason that most technical debt is imprudent. If you are truly concerned about getting to market early, it is almost always better to do it with fewer features, than with suboptimal design. Missing features are a promise that can be kept. Paying back suboptimal designs creates interest payments that often submerge any attempts at payback and can slow the team to the breaking point.

But there are some cases where a sub-optimal design can increase your net worth by allowing you to deliver early. However, the interest rate needs to be very low, and the principle payments need to be affordable, and big enough to pay back the debt in short order.

What does a low interest rate mean? It means that the sub-optimal design does not infiltrate every part of your system. It means that you can put the sub-optimal design off in a corner where it doesn’t impact your daily development life.

For example, I recently implemented a feature in FitNesse using HTML Frames. This is sub-optimal. On the other hand, the feature is constrained to one small part of the system, and it simply doesn’t impact any other part of the system. It does not impede my progress. There is no mess for me to move around. The interest rate is almost zero! (nice deal if you can get it!)

Implementing that feature with ajax is a much larger project. I would have had to invest a great deal of time and effort, and would have had to restructure massive amounts of the internal code. So the choice was a good one.

Better yet, the customer experience has pretty much been a big yawn. I thought people would really like the feature and would drive me to expand upon it. Instead, the customer base has virtually ignored it.

So my solution will be to pay back this debt by eliminating the feature. It was a cheap experiment, that resulted in my not having to spend a lot of time and effort on a new architecture! Net worth indeed!

But it might have gone the other way. My customers may have said: “Wow, Great! We want more!” At that point it would have been terrible to expand on the HTML Frames! That decision would have been in the DI quadrant. Deliberate imprudence! Rather, my strategy would have been to replace the suboptimal Frames design of the feature with an isolated ajax implementation, and then to gradually migrate the ajax solution throughout the project. That would have been annoying, but loan payments always are.

Summary

So, don’t let the caption in the DP quadrant be an excuse. Don’t fall for the DI meme that says “We just gotta bite the bullet”. Tread very carefully when you enter the DP quadrant. Look around at all your options, because it’s easy to think you are in the DP quadrant when you are really in the DI quadrant.

Remember: Murphy shall send you strong delusion, that you should believe you are in DP; so that you will be damned in DI.

TDD Derangement Syndrome

Uncle Bob — Wed, 07 Oct 2009 08:32:00 -0500

My recent blog about TDD, Design Patterns, Concurrency, and Sudoku seemed to draw the ire of a few vocal TDD detractors. Some of these people were rude, insulting, derisive, dismissive, and immature. Well, Halloween is not too far away.

In spite of their self-righteous snickering they did ask a few reasonable questions. To be fair I thought it would be appropriate for me to answer them.

Is there any research on TDD?

It turns out that there is a fair bit.

One simple google search led me to this blog by Phil Haack in which he reviewed a TDD research paper. Quoting from the paper:

We found that test-first students on average wrote more tests and, in turn, students who wrote more tests tended to be more productive. We also observed that the minimum quality increased linearly with the number of programmer tests, independent of the development strategy employed.

The same google search led me to this blog by Matt Hawley, in which he reviewed several other research papers. Part of his summary:

* 87.5% of developers reported better requirements understanding. * 95.8% of developers reported reduced debugging efforts. * 78% of developers reported TDD improved overall productivity. * 50% of developers found that it decreased overall development time. * 92% of developers felt that TDD yielded high-quality code. * 79% of developers believed TDD promoted simpler design.

Actually, I recognize some of Matt’s results as coming from a rather famous 2003 study (also in the list of google results) by Laurie Wiliams and Boby George. This study describes a controlled experiment that they conducted in three different companies. Though Matt’s summary above is based (in part) on that study, there is more to say.

In the George-William study teams that practiced TDD took 16% longer to claim that they were done than the teams that did not practice TDD. Apparently tests are more accurate than claims since the non-TDD teams failed to pass one third of the researcher’s hidden acceptance tests, whereas the TDD teams passed about 6 out of 7. To paraphrase Kent Beck: “If it doesn’t have to work, I can get it done a lot faster!”

Another point of interest in this study is that the TDD teams produced a suite of automated tests with very high test coverage (close to 100% in most cases) whereas most of the non-TDD teams did not produce such a suite; even though they had been instructed to.

Jim Shore wrote a review of yet another research summary which I found in the same google search. This one combines 7 different studies (including George-Williams). Here the results range from dramatically improved quality and productivity to no observed effect.

Finally, there is this 2008 case Study of TDD at IBM and Microsoft which shows that TDDers enjoy a defect density reduction ranging from 30% to 90% (as measured by defect tracking tools) and a productivity cost of between 15% and 35% (the subjective opinion of the managers). I refer you back to Kent Beck’s comment above.

I’m sure there is more research out there. After all this was just one google search. I think it’s odd that the TDD detractors didn’t find anything when they did their google searches.

Oh yeah, and then there was that whole issue of IEEE Software that was dedicated to papers and research on TDD.

What projects have been written with TDD, hmmm?

Quite a few, actually. The following is a list of projects that have an automated suite of unit tests with very high coverage. Those that I know for a fact use TDD, I have noted as such. The others, I can only surmise. If you know of any others, please post a comment here.

JUnit. This one is kind of obvious. JUnit was written by Kent Beck and Erich Gamma using TDD throughout. If you measure software success by sheer distribution, this particular program is wildly successful.

Fit. Written by Ward Cunningham. The progenitor of most current acceptance testing frameworks.

FitNesse. This testing framework has tens of thousands of users. It is 70,000 lines of java code, with 90%+ code coverage. TDD throughout. Very small bug-list. Again, if you measure by distribution, another raving success.

Cucumber,
Rspec. These two are Testing frameworks in Ruby. Of course you’d expect a testing framework to be written with TDD, wouldn’t you? I know these were. TDD throughout.

Limelight. A gui framework in JRUby. TDD throughout.
jfreechart.
Spring
JRuby
Smallsql
Ant
MarsProject
Log4J
Jmock

Are there others? I’m sure there are. This was just a quick web search. Again, if you know of more, please add a comment.

Echoes from the Stone Age

Uncle Bob — Tue, 06 Oct 2009 11:07:29 -0500

The echoes from Joel Spolsky’s Duct Tape blog continue to bounce off the blogosphere and twitterverse. Tim Bray and Peter Seibel have both written responses to Joel, me, and each other.

Here are some stray thoughts…

TDD

Anyone who continues to think that TDD slows you down is living in the stone age. Sorry, that’s just the truth. TDD does not slow you down, it speeds you up.

Look, TDD is not my religion, it is one of my disciplines. It’s like dual entry bookkeeping for accountants, or sterile procedure for surgeons. Professionals adopt such disciplines because they understand the theory behind them, and have directly experienced the benefits of using them.

I have experienced the tremendous benefit that TDD has had in my work, and I have observed it in others. I have seen and experienced the way that TDD helps programmers conceive their designs. I have seen and experienced the way it documents their decisions. I have seen and experienced the decouplings imposed by the tests, and I have seen and experienced the fearlessness with which TDDers can change and clean their code.

To be fair, I don’t think TDD is always appropriate. There are situations when I break the discipline and write code before tests. I’ll write about these situations in another blog. However, these situations are few and far between. In general, for me and many others, TDD is a way to go fast, well, and sure.

The upshot of all this is simple. TDD is a professional discipline. TDD works. TDD makes you faster. TDD is not going away. And anyone who has not really tried it, and yet claims that it would slow them down, is simply being willfully ignorant. I don’t care if your name is Don Knuth, Jamie Zawinski, Peter Seibel, or Peter Pan. Give it a real try, and then you have the right to comment.

Let me put this another way. And now I’m talking directly to those who make the claim that TDD would slow them down. Are you really such a good programmer that you don’t need to thoroughly check your work? Can you conceive of a better way to check your work than to express your intent in terms of an executable test? And can you think of a better way to ensure that you can write that test other than to write it first?

If you can, then I want to hear all about it. but I don’t want to hear that you write a few unit tests after the fact. I don’t want to hear that you manually check your code. I don’t want to hear that you do design and therefore don’t need to write tests. Those are all stone-age concepts. I know. I’ve been there.

So there. <grin>

The Design Pattern Religion

Tim Bray said:

My experience suggests that there are few surer ways to doom a big software project than via the Design Patterns religion.

He’s right of course. The Design Patterns religion is a foul bird that ravages teams and cuts down young projects in their prime. But let’s be clear about what that religion is. The Design Patterns religion is the ardent belief that the use of design patterns is good.

Here’s a clue. Design Patterns aren’t good. They also aren’t bad. They just are. Given a particular software design situation, there may be a pattern that fits and is beneficial. There may also be patterns that would be detrimental. It’s quite possible that none of the currently documented patterns are appropriate and that you should close the book and just solve the problem.

Here’s another clue. You don’t use patterns. You don’t apply patterns. Patterns just are. If a particular pattern is appropriate to solve a given problem, then it will be obvious. Indeed it is often so obvious that you don’t realize that the pattern is in place until you are done. You look back at your code and realize: “Oh, that’s a Decorator!”.

So am I saying that Design Patterns are useless?

NO! I want you to read the patterns books. I want you to know those patterns inside and out. If I point at you and say “Visitor” I want you at the board drawing all the different variants of the pattern without hesitation. I want you to get all the names and roles right. I want you to know patterns.

But I don’t want you to use patterns. I don’t want you to believe in patterns. I don’t want you to make patterns into a religion. Rather I want you to be able to recognize them when they appear, and to regularize them in your code so that others can recognize them too.

Design Patterns have a huge benefit. They have names. If you are reading code, and you see the word “Composite”, and if the author took care to regularize the code to the accepted names and roles of the “Composite” pattern, then you will know what that part of the code is doing instantly. And that is powerful!

Minimizing Concurrency.

In my first Duct Tape blog I made the statement:

I found myself annoyed at Joel’s notion that most programmers aren’t smart enough to use templates, design patterns, multi-threading, COM, etc. I don’t think that’s the case. I think that any programmer that’s not smart enough to use tools like that is probably not smart enough to be a programmer period.

Tim responds with:

...multi-threading is part of the problem, not part of the solution; that essentially no application programmer understands threads well enough to avoid deadlocks and races and horrible non-repeatable bugs. And that COM was one of the most colossal piles of crap my profession ever foisted on itself.

Is concurrency really part of the problem? Yes! Concurrency is a really big part of the problem. Indeed, the first rule of concurrency is: DON’T. The second rule is: REALLY, DON’T.

The problem is that some times you have no choice. And in those situations, where you absolutely must use concurrency, you should know it inside and out!

I completely and utterly reject the notion that ignorance is the best defense. I reject that lack of skill can ever be an advantage. So I want you to know concurrency. I want to shout “Dining Philosophers” and have you run to the board without hesitation and show me all the different solutions. If I holler “Deadlock”, I want you to quickly identify the causes and solutions.

Here’s a clue. If you want to avoid using something, know that something cold.

Sudoku

At the end of his blog, Peter jumps on the pile of bodies already crushing Ron Jeffries regarding the Sudoku problem from July of 2006.

I find the pile-up disturbing. Ron had the courage to fail in public. Indeed he announced up front that he might “crash and burn”. And yet he got lambasted for it by people who hid behind someone else’s work. The responses to Ron’s tutorial blogs were completely unfair because the authors of those blogs had everything worked out for them by Dr. Peter Norvig before they published their screeds. They were comparing apples to oranges because their responses were about the solution whereas Ron’s blogs were about the process.

Which one of us has not gone down a rat-hole when hunting for a solution to a complex problem? Let that person write the first blog. Everyone else ought to be a bit more humble.

Do the people on the pile think that Ron is unable to solve the Sudoku problem? (Some have said as much.) Then they don’t know Ron very well. Ron could code them all under the table with one hand tied behind his back.

Personal issues aside, I find the discussion fascinating in it’s own right. Ron had attempted to solve the Sudoku problem by gaining insight into that problem through the process of coding intermediate solutions. This is a common enough TDD approach. Indeed, the Bowling Game and the Prime Factors Kata are both examples where this approach can work reasonably well.

This approach follows the advice of no less than Grady Booch who (quoting Heinlein) said: “when faced with a problem you do not understand, do any part of it you do understand, then look at it again.“

Ron was attempting to use TDD to probe into the problem to see if he could gain any insight. This technique often bears fruit. Sometimes it does not.

Here is a classic example. Imagine you were going to write a sort algorithm test first:

Test 1: Sort an empty array. Solution: Return the input array.

Test 2: Sort an array with one element. Solution: Return the input array.

Test 3: Sort an array with two elements. Solution: Compare the two elements and swap if out of order. Return the result.

Test 4: Sort an array with three elements. Solution: Compare the first two and swap if out of order. Compare the second two and swap if out of order. Compare the first two again and swap if out of order. Return the result.

Test 5: Sort an array with four elements. Solution: Put the compare and swap operations into a nested loop. Return the result.

The end result is a bubble sort. The algorithm virtually self assembles. If you had never heard of a bubble sort before, this simple set of tests would have driven you to implement it naturally.

Problems like Bowling, Prime Factors, and Bubble Sort hold out the interesting promise that TDD may be a way to derive algorithmms from first principles!

On the other hand, what set of tests would drive you to implement a QuickSort? There are none that I know of. QuickSort and Sudoku may require a serious amount of introspection and concentrated thought before the solution is apparent. They may belong to a class of algorithms that do not self-assemble like Bowling, Prime Factors, and Bubble Sort.

This blog by Kurt Christensen provides all the links to the various Sudoku articles, and sums it up this way.

TDD may not be the best tool for inventing new algorithms, it may very well be the best tool for applying those algorithms to the problem at hand.

Actually I think TDD is a good way to find out if an algorithm will self-assemble or not. It usually doesn’t take a lot of time to figure out which it’s going to be.

A Mess is not a Technical Debt.

Uncle Bob — Tue, 22 Sep 2009 08:15:19 -0500

The term Technical Debt was created by Ward Cunningham to describe the engineering trade-off’s that software developers and business stakeholders must often make in order to meet schedules and customer expectations. In short, you may need to use suboptimal designs in the short term, because the schedule does not allow longer term designs to be used. As a simple example, your initial website design may need to be frames based because you don’t have time to build an Ajax framework.

Clearly this causes a debt. If the customer is looking for a web 2.0 system, then frames just aren’t going to cut it for long. So time is going to have to be carved out of a future schedule to refit the system with an Ajax solution.

In short, the business has decided that it can afford to delay release 2 in order to accelerate release 1. Is this wise?

Businesses make this kind of trade-off all the time; and there’s nothing inherently unwise about it. If the early release of 1.0 drives the business that pays for the development of 2.0 then the business has won. So this kind of reasoned technical debt may indeed be appropriate.

Unfortunately there is another situation that is sometimes called “technical debt” but that is neither reasoned nor wise. A mess.

Technical debt may be necessary, but it had also better be clean! If you are going to implement a frames solution instead of an AJAX solution, then make sure that the workmanship of the frames solution is top-notch. Make sure the design is well balanced, and the code is clean. If you make a mess while implementing that frames solution, you’ll never be able to replace it with an AJAX framework. The mess will impede your progress forever.

A mess is not a technical debt. A mess is just a mess. Technical debt decisions are made based on real project constraints. They are risky, but they can be beneficial. The decision to make a mess is never rational, is always based on laziness and unprofessionalism, and has no chance of paying of in the future. A mess is always a loss.

When you buy a house and take on a big mortgage debt, you tighten up all your spending and accounting. You clean up your books and your budgets. You behave with increased discipline. The same is true of technical debt. The more technical debt you take on, the tighter your disciplines need to be. You should do more testing, and more pairing and more refactoring. Technical debt is not a license to make a mess. Technical debt creates the need for even greater cleanliness.

When you decide to take on a technical debt, you had better make sure that your code stays squeaky clean. Keeping the system clean is the only way you will pay down that debt.

The CSM Integrity Deficit

Uncle Bob — Fri, 18 Sep 2009 08:44:43 -0500

Scott Ambler wrote a blog, and an editorial about the dirty dealings and desperate deception of the Scrum Alliance and their slimy certification scam. He rightly points out that the certification means little more than the applicant’s check didn’t bounce.

He goes on to imply that the entire agile community is guilty of keeping silent while this huge chicanery was foisted upon an innocent industry. He calls this conspiratorial silence: “integrity debt”.

Oh bollux! What an incredible load of Dingoes Kidneys!

Look. I’m not a big fan of CSM. I think it’s a gimmick. I am not a CSM myself, and have no intention of joining their ranks. When I meet someone who proclaims themselves to be a CSM, I’m not particularly impressed. I know what that certification means, and I take it with a grain of salt. To me, the title of CSM is worth little more than a shrug.

We at Object Mentor do a lot of training in things like Test Driven Development, Agile Methods. Object Oriented Principles, Java, C#, etc. etc. At the end of every course we often sign and pass out certificates to the students. Those certificates proclaim that the student attended the course. I see no difference between that certificate (which is a certification after all) and the CSM certificate. I suppose the students who take our TDD course could claim to be Object Mentor Certified TDDers; and they’d be right.

Have we created an “Integrity Debt” by handing out those certificates? Of course not. Everybody knows exactly what they mean. Nobody misrepresents their intent. They are an honest statement of fact. And the same is true of the CSM certificate.

Is it troubling that some HR people are starting to put CSM requirements on Job postings? Not at all! It is perfectly within the rights of any company to decide that they want to hire people who have been appropriately trained. Are there some HR people who overestimate the value of CSM? Probably, but that’s their own fault.

In my humble opinion there is no significant integrity issue here. Oh it wouldn’t surprise me to learn that there might have been some back-door deals in the early days of CSM. Perhaps some people were given CST status, or CSM status without careful controls. If that happened, I chalk it up to birthing pains which the Scrum Alliance is striving to correct. I don’t think anybody was out to scam anybody else. I don’t think CSM is a far flung conspiracy to ruin the software industry, and I don’t think the US government flew those jets into the twin towers.

Bottom line. There is no “Integrity Debt” here. What there is is a group of honest and caring folks who are trying to figure out the best ways to get Agile concepts adopted in an industry that sorely needs them.

In that regard I think that the agile movement has enjoyed a significant boost because of the interest generated by the CSM program. There are more companies doing Agile today because of CSM. So if anybody owes a debt here, it may be the Agile community owing a debt to the CSM program.

Maybe, instead of accusing and castigating and pointing the finger of judgement and doom we ought give a salute to Ken Schwaber, and say: “Thanks Ken.”

One Thing: Extract till you Drop.

Uncle Bob — Fri, 11 Sep 2009 13:33:00 -0500

For years authors and consultants (like me) have been telling us that functions should do one thing. They should do it well. They should do it only.

The question is: What the hell does “one thing” mean?

After all, one man’s “one thing” might be someone else’s “two things”.

Consider this class:

  class SymbolReplacer {
    protected String stringToReplace;
    protected List<String> alreadyReplaced = new ArrayList<String>();

    SymbolReplacer(String s) {
      this.stringToReplace = s;
    }

    String replace() {
      Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");
      Matcher symbolMatcher = symbolPattern.matcher(stringToReplace);
      while (symbolMatcher.find()) {
        String symbolName = symbolMatcher.group(1);
        if (getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName)) {
          alreadyReplaced.add(symbolName);
          stringToReplace = stringToReplace.replace("$" + symbolName, translate(symbolName));
        }
      }
      return stringToReplace;
    }

    protected String translate(String symbolName) {
      return getSymbol(symbolName);
    }
  }

It’s not too hard to understand. The replace() function searches through a string looking for $NAME and replaces each instance with the appropriate translation of NAME. It also makes sure that it doesn’t replace a name more than once. Simple.

Of course the words “It also…” pretty much proves that this function does more than one thing. So we can probably split the function up into two functions as follows:

    String replace() {
      Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");
      Matcher symbolMatcher = symbolPattern.matcher(stringToReplace);
      while (symbolMatcher.find()) {
        String symbolName = symbolMatcher.group(1);
        replaceAllInstances(symbolName);
      }
      return stringToReplace;
    }

    private void replaceAllInstances(String symbolName) {
      if (getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName)) {
        alreadyReplaced.add(symbolName);
        stringToReplace = stringToReplace.replace("$" + symbolName, translate(symbolName));
      }
    }

OK, so now the replace() function simply finds all the symbols that need replacing, and the replaceAllInstances() function replaces them if they haven’t already been replaced. So do these function do one thing each?

Well, the replace() compiles the pattern and build the Matcher() Maybe those actions should be moved into the constructor?

  class SymbolReplacer {
    protected String stringToReplace;
    protected List<String> alreadyReplaced = new ArrayList<String>();
    private Matcher symbolMatcher;
    private final Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");

    SymbolReplacer(String s) {
      this.stringToReplace = s;
      symbolMatcher = symbolPattern.matcher(s);
    }

    String replace() {
      while (symbolMatcher.find()) {
        String symbolName = symbolMatcher.group(1);
        replaceAllInstances(symbolName);
      }
      return stringToReplace;
    }

    private void replaceAllInstances(String symbolName) {
      if (getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName)) {
        alreadyReplaced.add(symbolName);
        stringToReplace = stringToReplace.replace("$" + symbolName, translate(symbolName));
      }
    }

    protected String translate(String symbolName) {
      return getSymbol(symbolName);
    }
  }

OK, so now certainly the replace() function is doing one thing? Ah, but I see at least two. It loops, extracts the symbolName and then does the replace. OK, so how about this?

    String replace() {
      for (String symbolName = nextSymbol(); symbolName != null; symbolName = nextSymbol())
        replaceAllInstances(symbolName);

      return stringToReplace;
    }

    private String nextSymbol() {
      return symbolMatcher.find() ? symbolMatcher.group(1) : null;
    }

I had to restructure things a little bit. The loop is a bit ugly. I wish I could have said for (String symbolName : symbolMatcher) but I guess Matchers don’t work that way.

I kind of like the nextSymbol() function. It gets the Matcher nicely out of the way.

So now the replace() and nextSymbol() functions are certainly doing one thing. Aren’t they?

Well, I suppose I could separate the loop from the return in replace().

    String replace() {
      replaceAllSymbols();
      return stringToReplace;
    }

    private void replaceAllSymbols() {
      for (String symbolName = nextSymbol(); symbolName != null; symbolName = nextSymbol())
        replaceAllInstances(symbolName);
    }

I don’t see how I could make these functions smaller. They must be doing one thing. There’s no way to extract any other functions from them!

Uh… Wait. Is that the definition of one thing? Is a function doing one thing if, and only if, you simply cannot extract any other functions from it? What else could “one thing” mean? After all, If I can extract one function out of another, the original function must have been doing more than one thing.

So does that mean that for all these years the authors and consultants (like me) have been telling us to extract until you can’t extract anymore?

Let’s try that with the rest of this class and see what it looks like…

  class SymbolReplacer {
    protected String stringToReplace;
    protected List<String> alreadyReplaced = new ArrayList<String>();
    private Matcher symbolMatcher;
    private final Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");

    SymbolReplacer(String s) {
      this.stringToReplace = s;
      symbolMatcher = symbolPattern.matcher(s);
    }

    String replace() {
      replaceAllSymbols();
      return stringToReplace;
    }

    private void replaceAllSymbols() {
      for (String symbolName = nextSymbol(); symbolName != null; symbolName = nextSymbol())
        replaceAllInstances(symbolName);
    }

    private String nextSymbol() {
      return symbolMatcher.find() ? symbolMatcher.group(1) : null;
    }

    private void replaceAllInstances(String symbolName) {
      if (shouldReplaceSymbol(symbolName))
        replaceSymbol(symbolName);
    }

    private boolean shouldReplaceSymbol(String symbolName) {
      return getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName);
    }

    private void replaceSymbol(String symbolName) {
      alreadyReplaced.add(symbolName);
      stringToReplace = stringToReplace.replace(
        symbolExpression(symbolName), 
        translate(symbolName));
    }

    private String symbolExpression(String symbolName) {
      return "$" + symbolName;
    }

    protected String translate(String symbolName) {
      return getSymbol(symbolName);
    }
  }

Well, I think it’s pretty clear that each of these functions is doing one thing. I’m not sure how I’d extract anything further from any of them.

Perhaps you think this is taking things too far. I used to think so too. But after programming for over 40+ years, I’m beginning to come to the conclusion that this level of extraction is not taking things too far at all. In fact, to me, it looks just about right.

So, my advice: Extract till you just can’t extract any more. Extract till you drop.

After all, with modern tools it takes very little time. It makes each function almost trivial. The code reads very nicely. It forces you to put little snippets of code into nicely named functions. And, well gosh, extracting till you drop is kind of fun!

Jarvis March in Clojure

Uncle Bob — Tue, 11 Aug 2009 21:10:21 -0500

OK, all you Clojure gurus, I need your help. I need to speed this algorithm up.

Just for fun I thought I’d write the Jarvis March algorithm in Clojure. This algorithm is for finding the convex hull of a collection of points. The basic idea is really simple. Imagine that all the points in the collection are represented by nails in a board. Find the left-most nail and tie a string to it. Then wind the string around the nails. The string will only touch nails that are part of the convex hull.

The details are not really that much more difficult. You start at the left-most point and calculate the angle from vertical required to touch every other point. The point with the minimum angle is the next point. You keep going around looking finding points with the minimum angle that is greater than the previous angle. When you get back to the starting point you are done.

Calculating angles can be time consuming, so I use a psuedo-angle algorithm. It doesn’t calculate the actual angle, rather it is a function that increases with the true angle, and goes from [0, 4).

The code is pretty simple.

(ns convexHull.convex-hull
  (:use clojure.contrib.math))

(defn quadrant-one-pseudo-angle [dx dy]
  (/ dx (+ dy dx)))

(defn pseudo-angle [[dx dy]]
  (cond
    (and (= dx 0) (= dy 0))
    0

    (and (>= dx 0) (> dy 0))
    (quadrant-one-pseudo-angle dx dy)

    (and (> dx 0) (<= dy 0))
    (+ 1 (quadrant-one-pseudo-angle (abs dy) dx))

    (and (<= dx 0) (< dy 0))
    (+ 2 (quadrant-one-pseudo-angle (abs dx) (abs dy)))

    (and (< dx 0) (>= dy 0))
    (+ 3 (quadrant-one-pseudo-angle dy (abs dx)))

    :else nil))

(defn point-min [[x1 y1 :as p1] [x2 y2 :as p2]]
  (cond
    (< x1 x2)
    p1

    (= x1 x2)
    (if (< y1 y2) p1 p2)

    :else
    p2))

(defn find-min-point [points]
  (reduce point-min points))

(defn delta-point [[x1 y1] [x2 y2]]
  [(- x1 x2) (- y1 y2)])

(defn angle-and-point [point base]
  [(pseudo-angle (delta-point point base)) point])

(defn min-angle-and-point [ap1 ap2]
  (if (< (first ap1) (first ap2)) ap1 ap2))

(defn find-point-with-least-angle-from [base angle points]
  (reduce min-angle-and-point
    (remove
      #(< (first %) angle)
      (map #(angle-and-point % base)
        (remove
          (fn [p] (= base p))
          points)))))

(defn hull [points]
  (println "Start")
  (let [starting-point (find-min-point points)]
    (println starting-point)
    (loop [hull-list [starting-point] angle 0 last-point starting-point]
      (let [[angle next-point] (find-point-with-least-angle-from last-point angle points)]
        (if (= next-point (first hull-list))
          hull-list
          (recur (conj hull-list next-point) angle next-point))))))

I execute it with this:

(ns convexHull.time-hull
  (:use convexHull.convex-hull))

(def r (java.util.Random.))
(defn rands [] (repeatedly #(.nextGaussian r)))
(defn points [] (take 400000 (partition 2 (rands))))
(let [hull-points (time (hull (points)))]
  (printf "Points: %d\n" (count hull-points))
  (doseq [x hull-points] (println x)))

This takes way too long to run. The equivalent java program will do a million points in half a second. This one is taking 25 seconds to do a half-million points. It won’t even do a million points. It slows way way down and then runs out of memory. (There must be some kind of disk caching going on or something.)

Anyway, I’d be interested in seeing how a real Clojure programmer would speed this program up.

Using Clojure for Scripting

Uncle Bob — Sat, 08 Aug 2009 14:14:07 -0500

Clojure makes a nice language for writing scripts that explore things on the filesystem. For example, here is a cute little program that explores the FitNesse source tree counting the lines of production java code and comparing it to the lines of unit test code.


(use 'clojure.contrib.duck-streams)

(def file-counts
  (map
    #(list (.toString %) (count (read-lines %)))
    (remove #(.isDirectory %)
      (file-seq (file-str "~/projects/FitNesseGit/src/fitnesse")))))

(defn count-lines-that-end-with [file-counts suffix]
  (reduce +
    (map second
      (filter #(.endsWith (first %) suffix) file-counts))))

(def java-count (count-lines-that-end-with file-counts ".java"))
(def test-count (count-lines-that-end-with file-counts "Test.java"))

(printf "Java lines: %d\n" java-count)
(printf "Test lines: %d\n" test-count)
(printf "Test pct: %.1f\n" (double (* 100 (/ test-count java-count))))

Java calling Clojure.

Uncle Bob — Fri, 07 Aug 2009 17:00:43 -0500

While I think Clojure is an interesting language, in order for it to be of real practical use, I must be able to use it in conjunction with other systems I am working on. Specifically, I’d like to write some FitNesse tools in Clojure; but for that to work, I’ll need to call into my Clojure code from Java.

Today, my son Justin and I managed to do just that by following the instruction in Stuart Halloway’s book, the Clojure api website, and Mark Volkmann’s very useful site.

Be advised, that it takes a bit of fiddling to get this to work. You will have to jockey around with your classpaths and output directories. But it’s not actually that hard to do, and the results can be very rewarding.

We implemented the Bowling game (again), but this time we wrote the unit tests in Java, and had them call into Clojure. From the point of view of the Java tests, it looked just like we were calling into java code. The tests had no idea that this was Clojure.

Here are the tests:

package bowling;

import static org.junit.Assert.assertEquals;
import org.junit.Before;
import org.junit.Test;

public class BowlingTest {
  private Game g;

  @Before
  public void setup() {
    g = new Game();
  }

  @Test
  public void roll() throws Exception {
    g.roll(0);
  }

  @Test
  public void gutterGame() throws Exception {
    rollMany(20, 0);
    assertEquals(0, g.score());
  }

  private void rollMany(int n, int pins) {
    for (int i=0; i< n; i++) {
      g.roll(pins);
    }
  }

  @Test
  public void allOnes() throws Exception {
    rollMany(20, 1);
    assertEquals(20, g.score());    
  }

  @Test
  public void oneSpare() throws Exception {
    g.roll(5);
    g.roll(5);
    g.roll(3);
    rollMany(17, 0);
    assertEquals(16, g.score());
  }

  @Test
  public void oneStrike() throws Exception {
    g.roll(10);
    g.roll(5);
    rollMany(17,0);
    assertEquals(20, g.score());
  }

  @Test
  public void perfectGame() throws Exception {
    rollMany(12, 10);
    assertEquals(300, g.score());
  }

  @Test
  public void allSpares() throws Exception {
    rollMany(21, 5);
    assertEquals(150, g.score());
  }
}

The clojure code to make them pass looks like this:

(ns bowling.Game
  (:gen-class
    :state state
    :init init
    :methods [
    [roll [int] void]
    [score [] int]
    ]))

(defn -init [] [[] (atom [])])

(defn -roll [this pins]
  (reset! (.state this) (conj @(.state this) pins)))

(declare score-frames)

(defn -score [this]
  (reduce + (take 10 (score-frames [] (conj @(.state this) 0)))))

(defn score-frames [scores [first second third :as rolls]]
  (cond
    (or (empty? rolls) (nil? second) (nil? third)) scores
    (= 10 first) (recur (conj scores (+ 10 second third)) (next rolls))
    (= 10 (+ first second)) (recur (conj scores (+ 10 third)) (nnext rolls))
    :else
    (recur (conj scores (+ first second)) (nnext rolls))))

The magic is all up in that (ns bowling.game… block.

The name of the java class is Game in the bowling package, or just bowling.Game.
Clojure will create a special member function named state that you can use to squirrel away the state variables of the class.
A function named -init will be called when your class is constructed. You must write this function to return a vector containing 1) a vector of all the arguments to pass to the base class constructor (in my case none), and the initial state of the newly created instance, (in my case an empty vector stuffed into an atom)
The class will have two new methods, roll, which takes an int and returns void, and score which takes nothing and returns an int.

When a method is called on the instance, Clojure automatically invokes a function of the same name, but prefixed by a -, so -roll and -score are the implementations of the roll and score methods. Note that each of these functions take a this argument. You can get the state out of this by saying (.state this).

From a source code point of view, that’s about all there is to it. But how do you compile this into a java .class file?

Here’s what we did. We created a new file named compile.clj that looks like this:

(set! *compile-path* "../../classes")
(compile 'bowling.Game)

We can run this file from our IDE (IntelliJ using the LAClojure plugin) and it will compile quite nicely. But there are a few things you have to make sure you do.

Find out where your IDE puts the .class files, and set the compile-path variable to that directory.
Also make sure that directory is in your classpath.
Make sure that directory exists!
Also make sure that your source file directory (the directory that contains the packages) is in your classpath. (I know… but that’s apparently what it takes.)

You will get errors. And the errors aren’t particularly informative. The key to understanding them is to look at the backtrace of the exceptions they throw. Notice, for example, if the function “WriteClassFile” (or something like that) is buried in the backtrace. If so, you are probably having trouble writing your class file.

In the end, we wrote up an ant script that compiled our clojure and our java together. (You have to compile them in the right order! If java calls Clojure, those .class files have to exist before the java will compile!)

The ant script we used was created for us by IntelliJ, and then we modified it to get it to work. I include it here with the caveat that we made it work, but we didn’t make it “right”. You can ignore most of the stuff at the beginning. The interesting part is the Clojure target, and the clean target.

<?xml version="1.0" encoding="UTF-8"?>
<project name="bowlingforjunit" default="all">

  <property file="bowlingforjunit.properties"/>
  <!-- Uncomment the following property if no tests compilation is needed -->
  <!-- 
  <property name="skip.tests" value="true"/>
   -->

  <!-- Compiler options -->

  <property name="compiler.debug" value="on"/>
  <property name="compiler.generate.no.warnings" value="off"/>
  <property name="compiler.args" value=""/>
  <property name="compiler.max.memory" value="128m"/>
  <patternset id="ignored.files">
    <exclude name="**/CVS/**"/>
    <exclude name="**/SCCS/**"/>
    <exclude name="**/RCS/**"/>
    <exclude name="**/rcs/**"/>
    <exclude name="**/.DS_Store/**"/>
    <exclude name="**/.svn/**"/>
    <exclude name="**/.pyc/**"/>
    <exclude name="**/.pyo/**"/>
    <exclude name="**/*.pyc/**"/>
    <exclude name="**/*.pyo/**"/>
    <exclude name="**/.git/**"/>
    <exclude name="**/.sbas/**"/>
    <exclude name="**/.IJI.*/**"/>
    <exclude name="**/vssver.scc/**"/>
    <exclude name="**/vssver2.scc/**"/>
  </patternset>
  <patternset id="library.patterns">
    <include name="*.zip"/>
    <include name="*.war"/>
    <include name="*.egg"/>
    <include name="*.ear"/>
    <include name="*.swc"/>
    <include name="*.jar"/>
  </patternset>
  <patternset id="compiler.resources">
    <include name="**/?*.properties"/>
    <include name="**/?*.xml"/>
    <include name="**/?*.gif"/>
    <include name="**/?*.png"/>
    <include name="**/?*.jpeg"/>
    <include name="**/?*.jpg"/>
    <include name="**/?*.html"/>
    <include name="**/?*.dtd"/>
    <include name="**/?*.tld"/>
    <include name="**/?*.ftl"/>
  </patternset>

  <!-- JDK definitions -->

  <property name="jdk.bin.1.6" value="${jdk.home.1.6}/bin"/>
  <path id="jdk.classpath.1.6">
    <fileset dir="${jdk.home.1.6}">
      <include name="lib/deploy.jar"/>
      <include name="lib/dt.jar"/>
      <include name="lib/javaws.jar"/>
      <include name="lib/jce.jar"/>
      <include name="lib/management-agent.jar"/>
      <include name="lib/plugin.jar"/>
      <include name="lib/sa-jdi.jar"/>
      <include name="../Classes/charsets.jar"/>
      <include name="../Classes/classes.jar"/>
      <include name="../Classes/dt.jar"/>
      <include name="../Classes/jce.jar"/>
      <include name="../Classes/jconsole.jar"/>
      <include name="../Classes/jsse.jar"/>
      <include name="../Classes/laf.jar"/>
      <include name="../Classes/management-agent.jar"/>
      <include name="../Classes/ui.jar"/>
      <include name="lib/ext/apple_provider.jar"/>
      <include name="lib/ext/dnsns.jar"/>
      <include name="lib/ext/localedata.jar"/>
      <include name="lib/ext/sunjce_provider.jar"/>
      <include name="lib/ext/sunpkcs11.jar"/>
    </fileset>
  </path>

  <property name="project.jdk.home" value="${jdk.home.1.6}"/>
  <property name="project.jdk.bin" value="${jdk.bin.1.6}"/>
  <property name="project.jdk.classpath" value="jdk.classpath.1.6"/>

  <!-- Project Libraries -->

  <!-- Global Libraries -->

  <path id="library.clojure.classpath">
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/ant-launcher.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/ant.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/clojure-contrib.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/clojure.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/commons-codec-1.3.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/commons-fileupload-1.2.1.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/commons-io-1.4.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/compojure.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/hsqldb.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/jetty-6.1.14.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/jetty-util-6.1.14.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/jline-0.9.94.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/servlet-api-2.5-6.1.14.jar"/>
  </path>

  <!-- Modules -->

  <!-- Module BowlingForJunit -->

  <dirname property="module.bowlingforjunit.basedir" file="${ant.file}"/>

  <property name="module.jdk.home.bowlingforjunit" value="${project.jdk.home}"/>
  <property name="module.jdk.bin.bowlingforjunit" value="${project.jdk.bin}"/>
  <property name="module.jdk.classpath.bowlingforjunit" value="${project.jdk.classpath}"/>

  <property name="compiler.args.bowlingforjunit" value="${compiler.args}"/>

  <property name="bowlingforjunit.output.dir" value="${module.bowlingforjunit.basedir}/classes"/>
  <property name="bowlingforjunit.testoutput.dir" value="${module.bowlingforjunit.basedir}/classes"/>

  <path id="bowlingforjunit.module.bootclasspath">
    <!-- Paths to be included in compilation bootclasspath -->
  </path>

  <path id="bowlingforjunit.module.classpath">
    <path refid="${module.jdk.classpath.bowlingforjunit}"/>
    <pathelement location="/Library/junit4.6/junit-4.6.jar"/>
    <path refid="library.clojure.classpath"/>
    <pathelement location="${basedir}/classes"/>
  </path>

  <path id="bowlingforjunit.runtime.module.classpath">
    <pathelement location="${bowlingforjunit.output.dir}"/>
    <pathelement location="/Library/junit4.6/junit-4.6.jar"/>
    <path refid="library.clojure.classpath"/>
    <pathelement location="${basedir}/classes"/>
  </path>

  <patternset id="excluded.from.module.bowlingforjunit">
    <patternset refid="ignored.files"/>
  </patternset>

  <patternset id="excluded.from.compilation.bowlingforjunit">
    <patternset refid="excluded.from.module.bowlingforjunit"/>
  </patternset>

  <path id="bowlingforjunit.module.sourcepath">
    <dirset dir="${module.bowlingforjunit.basedir}">
      <include name="src"/>
    </dirset>
  </path>

  <target name="compile" depends="clojure, compile.java" description="Compile module BowlingForJunit"/>

  <target name="compile.java" description="Compile module BowlingForJunit; production classes">
    <mkdir dir="${bowlingforjunit.output.dir}"/>
    <javac destdir="${bowlingforjunit.output.dir}" debug="${compiler.debug}" nowarn="${compiler.generate.no.warnings}" 
           memorymaximumsize="${compiler.max.memory}" fork="true" executable="${module.jdk.bin.bowlingforjunit}/javac">
      <compilerarg line="${compiler.args.bowlingforjunit}"/>
      <bootclasspath refid="bowlingforjunit.module.bootclasspath"/>
      <classpath refid="bowlingforjunit.module.classpath"/>
      <src refid="bowlingforjunit.module.sourcepath"/>
      <patternset refid="excluded.from.compilation.bowlingforjunit"/>
    </javac>

    <copy todir="${bowlingforjunit.output.dir}">
      <fileset dir="${module.bowlingforjunit.basedir}/src">
        <patternset refid="compiler.resources"/>
        <type type="file"/>
      </fileset>
    </copy>
  </target>

  <target name="clojure">
    <java classname="clojure.lang.Compile">
      <classpath>
        <path location="/Users/unclebob/projects/clojure/BowlingForJunit/classes"/>
        <path location="/Users/unclebob/projects/clojure/BowlingForJunit/src"/>        
        <path location="/Users/unclebob/projects/clojure/BowlingForJunit/src/bowling"/>
        <path location="/Users/unclebob/projects/clojure-build/lib/clojure.jar"/>
        <path location="/Users/unclebob/projects/clojure-build/lib/clojure-contrib.jar"/>  
      </classpath>
      <sysproperty key="clojure.compile.path" value="/Users/unclebob/projects/clojure/BowlingForJunit/classes"/>
      <sysproperty key="java.awt.headless" value="true"/>
      <arg value="bowling.Game"/>
    </java>
  </target>

  <target name="clean" description="cleanup module">
    <delete dir="${bowlingforjunit.output.dir}"/>
    <delete dir="${bowlingforjunit.testoutput.dir}"/>
    <mkdir  dir="${bowlingforjunit.output.dir}"/>
  </target>

  <target name="all" depends="clean, compile" description="build all"/>
</project>

As the tests get more specific, the code gets more generic.

Uncle Bob — Thu, 06 Aug 2009 14:23:00 -0500

I tweeted this not too long ago. The basic idea is that as you add tests, the tests get more and more specific. This makes sense since tests are, after all, specifications. The more specifications you have, the more specific the whole body of specifications becomes.

As a general rule, good design dictates that the more specific your requirements become, the more general your code needs to be. This is saying roughly the same thing as Greenspun’s Tenth Rule of Programming: “Any sufficiently complicated [...] program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp.” Or rather, as more and more constraints pile upon a program, the designers look for ways to push those constraints out of the program itself and into the data.

In return for my tweet people asked for examples.

One of the better examples (though perhaps a bit trivial) is the Prime Factors Kata. This lovely little experiment grows and grows as you add test cases, and then suddenly collapses into an elegant three line algorithm.

The tests continue to become ever more specific. The production code starts out just as specific as the tests. But with the second or third test the programmer must make a decision. He can write the production code to mirror the tests (i.e. writing it as an if/else statement that detects which test is running and supplying the expected answer) or he can come up with some kind of more general algorithm that satisfies the tests without looking anything like them.

The algorithm grows and warps and twists; and then, just when it looks like it’s destined to become a wretched mess; it simply evaporates into a lovely little three line nested loop.

We see the principle at work in other ways as well. Often the programmers have a whole list of tests that they know must pass. As they write them one by one, they write the production code that satisfies them. Then, as in the Bowling Game Kata the tests start to pass unexpectedly. You were done with the code, and you weren’t aware of it. You continue writing tests, expecting one to fail, but they all pass. The test code grows, but the production code remains the same.

Sometimes this happens in a less surprising way. Sometimes you know that you have implemented an algorithm that will pass all remaining tests, but you write those tests anyway because they are part of the specification

The point is that test code and production code do not grow at the same rate. Indeed, as the application increases in complexity, the test code grows at a rate that is faster than the production code. Sometimes the production code actually shrinks as the test code grows because the programmers moved a load of functionality out of the code and into the data.

Consider FitNesse. A year ago there were 45,000 lines of code, of which 15,000 were tests, so 33% of the total were tests.

Now Fitnesse is 58,000 lines of code of which 26,000 are tests. We added 13,000 lines of code overall, but 8,000 (61%), are tests! The tests have grown to over 44% of the total.

Don't be lazy about lazy sequences.

Uncle Bob — Thu, 06 Aug 2009 11:54:41 -0500

Coding with infinite sequences is very powerful, but also very bizarre. If you aren’t careful you can get algorithms that work fine, but are O(N^{2) instead of O(N).}

Consider the (partial-sums) function that I used in my previous blog on generating pi.

(defn partial-sums [s]
  (lazy-seq
    (cons (first s) (add-streams (next s) (partial-sums s)))))

This function takes in a list of (a b c…) and returns a list of partial sums (a a+b a+b+c…). Trying to figure out how this function works is a bit mind-bending; at least for me; but here’s how I visualize it.

It adds (next s) to (partial-sums s).

                   (next s)
                      |
                      V
              s: (a   b    c     d ...)
(partial-sum s):     (a   a+b  a+b+c ...)

Notice how the vertical columns are the sums that we want. Notice also that each result is the sum of the previous result and the next element of s. So verbally and visually this all makes sense. But algorithmically it can still be challenging to understand. I mean, how do those values get calculated really?

Oh… And I should note… The function above doesn’t work this way at all. It’s actually horrifically inefficient. But I’ll get to that, and to the solution. First things first.

What allows this function to run is lazy-evaluation. All that really means is that an element of a list does not really exist until you ask for it. At that point the program calculates what the element of the list should be and then returns it to you as if it was there all along.

In Clojure, when you build a sequence inside the (lazy-seq) form, it simply defers the execution of the procedures that generate the values of the sequence until those values are requested. Look again at the function above. When you call it, it simply returns immediately without doing anything. The (lazy-seq) form simply short-circuits the execution of the function, deferring it for later.

So when (partial-sums) returns it gives you a sequence, but there’s nothing in it! However, as soon as you ask for the first element, the body of the (partial-sums) function will execute just enough to give you the value of the first element. No other elements are calculated. When you ask for the next value, (partial-sums) will execute again, just enough to give you that value. And so on.

Knowing that, let’s trace the execution.

We call (partial-sums s) which returns a list (let’s call it ps)
To get the first element of ps, (partial-sums) is lazily-invoked and simply returns (first s)
For the second element it returns the first element of the sequence produced by (add-streams (next s) (partial-sums s))
which will be: (+ (first (next s)) (first s))
For the third element it returns the second element of the added streams, but the element is composed of yet another stream addition invoked by the recursive (partial-sums) So it looks like this:
(+ (first (next (next s)) (+ (first (next s)) (first s))

“Uh… Wait.” I hear you say. “You mean that in order to calculate the third element, you have to add three elements? Didn’t we say that we could add the next element of s to the previous element of the partial sums? Why do we have to do three additions? Why does it seem to be doing this:”

(a b c)
  (a b c)
    (a b c)

Notice that the vertical columns are the sums we are looking for; but calculating them requires a lot more work than just adding the next value to the previous sum!.

“But wait,” you say again. “I know something about Clojure. Clojure memoizes the list values. Once you ask for a list element the first time, it never has to recalculate it again! So when we get the third element of the result it shouldn’t have to recalculate the second.”

Partly right. Once you lazily evaluate a list element, that element never has to be evaluated again. It’s in the list, and will simply be returned from the list. But let me ask you this: In the function above, where is the list that being asked again? Look closely. There is no partial-sums result that is being asked for a value more than once. Indeed, every time we need a value we create a new (partial-sums) list. So we aren’t taking advantage of the previous values because we’re always creating a new list.

If you’d like to prove this to yourself, you can run this fun little program.

(defn p [a b] (println a '+ b) (+ a b))

(defn add_streams [a b] (map p a b))

(defn partial-sums [s]
    (lazy-seq
      (cons (first s)
        (add_streams (next s) (partial-sums s)))))

(def w (iterate inc 1))
(println (take 5 (partial-sums w)))

Notice how I print every addition in the (p) function. If you run this you’ll get the following result:

The jumble of numbers is evidence that the lazy evaluation is working. As the println marches through the 5 partial sums, it causes each to be evaluated, causing each sum to be printed. Notice how many sums are taken! Ten additions for 5 partial sums. It’ll be 15 for 6, and 21 for 7. Indeed, the number of additions required to calculate ps[n] is equal to ps[n-1]. A little thought will convince you that this is O(n2).

To reduce this back to O(n), we need to stop creating a new list of partial-sums at each recursion, and use a single copy of the partial-sums result. We can do this by saving the partial sums result in a var, and then using that in the add-streams call.

(defn partial-sums [s]
    (def result (lazy-seq
      (cons (first s)
        (add_streams (next s) result)))) result)

This may bend your brain just a little bit more. We are using the result before we are done calculating it! But that’s ok, since we never use more than has already been calculated…

Running the program with this change yields the following output.

That’s the O(n) behavior we wanted! Each sum is calculated by adding the next element to the previous sum! Hooray!

There’s a lesson here. You don’t get the benefit of lazy sequences unless you use the lazy sequences. Creating a brand new one each time may generate the right answer, but it’s lazy, and can burn a lot of execution cycles.

One problem I have with this solution is the creation of the global var ‘result’. I’m sure there’s a way to create a local result; but I haven’t found it yet. (I tried using various let forms, but I haven’t stumbled upon one that works yet.)

Generating PI in Clojure

Uncle Bob — Wed, 05 Aug 2009 08:07:00 -0500

The SICP book shows a cute algorithm for computing PI using infinite sequences (lazy evaluation). I have adapted it for Clojure and modified it to use Java’s BigDecimal class, so that I can generate arbitrary precision.

First we define a function that generates an “infinite” list of terms of the form 1/1 + 1/3 – 1/5 + 1/7 – ... This is a well known series that very slowly converges on Pi. Notice that I’m using the ‘M’ suffix to denote BigDecimal, and I’m restricting the precision to 300 places. Notice also that I’m using lazy-seq. This function delays the execution of the ‘cons’ term until that part of the list is actually used. This allows the list to appear to be infinite.

(defn pi-summands [n]
  (lazy-seq
    (cons (with-precision 300 (/ 1M n))
      (map - (pi-summands (+ n 2))))))

Next we have two utility functions. The first multiplies the terms of a list by a scale factor. The other adds two lists together producing a list of sums. If the input lists are ‘infinite’, the output lists will also be ‘infinite’.

(defn scale-stream [stream factor]
  (map (fn [x] (* x factor)) stream))

(defn add-streams [p q]
  (map +  p q))

partial-sums takes a list of terms produces a list of the sums of those terms. Given [a b c] it produces [a a+b a+b+c]. And, again, the lists can be infinite.

(defn partial-sums [s]
  (lazy-seq
    (cons (first s) (add-streams (next s) (partial-sums s)))))

pi-stream mutiplies the partial-sums of the pi-summands by 4. Turning the summing sequence into 4/1 + 4/3 – 4/5…

(def pi-stream
  (scale-stream (partial-sums (pi-summands 1)) 4))

A simple squaring function.

(defn square [s] (* s s))

Euler’s transform is a fancy way to average three terms of a +/- sequence. It can dramatically increase the precision of an otherwise slowly converging sequence. The euler-transform function produces a sequence of results from the pi-stream. Again, note the constraint I place on BigDecimal precision.

(defn euler-transform [s]
  (let [s0 (nth s 0)
        s1 (nth s 1)
        s2 (nth s 2)]
    (lazy-seq
      (cons (with-precision 300 (- s2 (/ (square (- s2 s1))
        (+ s0 (* -2 s1) s2))))
        (euler-transform (next s))))))

It turns out that you can use Euler’s transform on it’s own results to further increase precision. The make-tableau function arranges the reapplication of a transform so that it can be repeatedly applied to itself.

(defn make-tableau [transform s]
  (lazy-seq
    (cons s (make-tableau transform (transform s)))))

And now we grab the first element of each repetition.

(defn accelerated-sequence [transform s]
  (map first (make-tableau transform s)))

A little utility to print the elements of a list.

(defn print-list [s] (doseq [n s] (println n)))

And now finally we print the first 150 iterations of the accelerated euler-transformation of the pi sequence.

(print-list (take 150
  (accelerated-sequence euler-transform pi-stream)))

The 150th result (the first two lines of which are accurate) 3.14159265358979323846264338327950288419716939 93751058209749445923078164062862089986280348253 25256352154500957038098311477352066337027814620 46050836612955894790973622135628850706785335611 34661971615744147647325427374216175843967838084 09640502874355811103562027043480098593797655792 36344321165429010895M

Scenario Tables for FitNesse

Uncle Bob — Tue, 04 Aug 2009 15:00:00 -0500

Scenario Tables are a very powerful new feature of FitNesse, allowing a whole new level of abstraction in the specification of tests. Scenario tables can be used on any page that uses the Slim test system.

One of the tests among the many acceptance tests for FitNesse itself ensures that certain operations require authentication. For example, if you declare a page to be read-locked, then you should not be able to read that page unless you have logged in.

Here is an example of an old-style FitNesse test that ensures that you can not save a page that is write-locked unless you log in.

Notice the detailed, step by step, programmerish nature of the test. 1. It creates a user with a password. 2. It creates a page that is write-locked. 3. It requests the page without any authentication and expects a 401. 4. It requests the page with bad authentication and expects a 401. 5. It requests the page with good authentication and expects a 303.

This test is not too hard to understand, and any good QA/BA ought to be able to grasp what it is doing. But it is certainly circuitous and indirect. Worse, this algorithm is repeated on a new page for each of fifteen other secure operations, creating a suite of very similar pages. In other words, lots of duplication.

In sweeping through the acceptance tests of FitNesse, I decided to use scenarios to simplify them. My first attempt at this was to adopt a Given/When/Then (BDD) approach. The same kind of approach that you might use with Cucumber of JBehave.

This test is much shorter, easier to read, and gets right to the point. It’s also much less programmerish since it doesn’t mention 401s or 303s. I was able to accomplish this by creating a set of scenarios that look something like the ones below. (They aren’t the exact ones, but you get the idea.)

All the programmerish stuff is hidden in the scenarios! That’s nice. If you look closely at the scenarios and substitute them into the test, you can see that the whole test simply turns into something very much like the original test.

Though this is better, we still have a lot of similar pages. The duplication has not been removed.

So then I added a third level of indirection. This one solves all the problems.

It’s hard to imagine a simpler test than this. Instead of a suite of sixteen different tests, I have one single test. The scenario is simply executed once of reach row of the “Operation is authenticated” table.

Here are the scenarios that allowed me to do this.

Once again, if you simply replace the scenarios in each row of the table, you’ll wind up with sixteen tests that look a lot like the original test. But now they are very direct, absurdly simple to read, and are virtually free of programmerish gobblety-gook.

Critics may rightly point out that you can get the same results by simply writing a ColumnFixture that interprets the final table. True, but then you put the burden of test-abstraction on the programmers. What is interesting about the scenario approach is that I was able to reduce the programmerish and highly duplicated tests into a single simple test table without writing any new fixtures!

In other words, programmers can supply a set of APIs, in the form of fixtures, and then competent test engineers can shape the tests appropriately with scenario tables that eventually call the low level fixture APIs while keeping the tests readable and free of duplication.

More Clojure from Uncle Bob JSPS

Uncle Bob — Fri, 31 Jul 2009 08:12:44 -0500

Thanks to all who posted their solutions to the Bowling Game. Just to be contrary, I decided to implement yet another kind of solution. This one clearly shows my OO roots.

h2. Here are the tests.

(ns bowling-game)
(use 'clojure.contrib.test-is)
(use 'bowling-game)

(defn roll-list [game list]
  (vec (concat game list)))

(defn roll-many [game n pins]
  (roll-list game (repeat n pins)))

(deftest can-create-game
  (is (not (nil? (new-game)))))

(deftest gutter-game-should-score-0
  (is (= 0 (score (-> (new-game) (roll-many 20 0))))))

(deftest all-ones-should-score-20
  (is (= 20 (score (-> (new-game) (roll-many 20 1))))))

(deftest one-spare
  (is (= 16 (score (-> (new-game) (roll-list [5 5 3]) (roll-many 17 0))))))

(deftest one_strike
  (is (= 24 (score (-> (new-game) (roll-list [10 3 4]) (roll-many 16 0))))))

(deftest perfect-game
  (is (= 300 (score (-> (new-game) (roll-many 12 10))))))

(run-tests 'bowling-game)

And here is the code.

(ns bowling-game)
(defn new-game [] [])

(defn frame-type [[first second]]
  (cond
    (= 10 first) :strike
    (= 10 (+ first second)) :spare
    :else :no-mark))

(defmulti frame-size frame-type)
(defmethod frame-size :strike [_] 1)
(defmethod frame-size :spare [_] 2)
(defmethod frame-size :no-mark [_] 2)

(defmulti frame-score frame-type)
(defmethod frame-score :strike [[_ first second]] (+ 10 (+ first second)))
(defmethod frame-score :spare [[_ _ next]] (+ 10 next))
(defmethod frame-score :no-mark [[first second]] (+ first second))

(defn score [game]
  (loop [frame 1 rolls game score 0]
    (if (> frame 10)
      score
      (recur (inc frame) (subvec rolls (frame-size rolls)) (+ score (frame-score rolls))))))

Uncle Bob, JSPS: Learning Clojure

Uncle Bob — Sun, 19 Jul 2009 21:10:00 -0500

(JSPS: Just Some Poor Schmuck)

I’m trying to learn Clojure, and I’m finding the effort challenging. Perhaps you can help me.

Programming in Clojure (A derivative of Lisp) requires a certain “inside-out” thinking. I’m finding the combination of the shift in thought process and TDD quite difficult to resolve. So I thought I’d ask for your help.

Below is the Bowling Game written in Clojure complete with unit tests. I’m not at all pleased with the result. The tests, especially, look bizzare and backwards to me.

Please post your own solutions to this so that we can all compare and contrast them.

Tests

(ns bowling-game)
(use 'clojure.contrib.test-is)
(use 'bowling-game)

(defn roll-list [game list]
  (if (empty? list)
    game
    (roll-list (roll game (first list)) (rest list))
    ))

(defn roll-many [game n pins]
  (loop [i n g game]
    (if (zero? i)
      g
      (recur (dec i) (roll g pins)))))

(defn gutter-game [game] (roll-many game 20 0))

(deftest can-create-game
  (is (not (nil? (new-game)))))

(deftest gutter-game-should-score-0
  (is (= 0 (score (gutter-game (new-game))))))

(deftest all-ones-should-score-20
  (is (= 20 (score (roll-many (new-game) 20 1)))))

(deftest one-spare
  (is (= 16 (score
    (roll-many
      (roll-list (new-game) [5 5 3])
      17 0)))))

(deftest one_strike
  (is (= 24 (score
    (roll-many
      (roll-list (new-game) [10 3 4])
      16 0)))))

(deftest perfect-game
  (is (= 300 (score (roll-many (new-game) 12 10)))))

(run-tests 'bowling-game)

Code

(ns bowling-game)
(defn new-game [] [])

(defn next-two-balls [rolls]
  (+ (rolls 0) (rolls 1)))

(defn score-strike [rolls]
  [1 (+ 10 (+ (rolls 1) (rolls 2)))])

(defn score-spare [rolls]
  [2 (+ 10 (rolls 2))])

(defn score-no-mark [rolls]
  [2 (next-two-balls rolls)])

(defn score-next-frame [rolls]
  (if (= 10 (first rolls))
    (score-strike rolls)
    (if (= 10 (next-two-balls rolls))
      (score-spare rolls)
      (score-no-mark rolls))))

(defn score [game]
  (loop [frame 1 rolls game score 0]
    (if (> frame 10)
      score
      (let [frame-score (score-next-frame rolls)]
        (recur (inc frame) (subvec rolls (frame-score 0)) (+ score (frame-score 1)))))))

(defn roll [game pins]
  (conj game pins))

The Rush

Uncle Bob — Fri, 26 Jun 2009 08:40:13 -0500

There’s nothing like the feeling of achievement when you get a complex software system working. It’s the feeling of the hunter making a hard fought kill. By the sheer power of your intellect you have imposed your will upon inanimate nature, and forced it to do your bidding. You feel the rush of power, the satisfaction of victory. You are the master!

That’s the good news.

The bad news is that you’ve made a mess of things along the way.

This is inevitable. It takes a great deal of focus and endurance to get a system working just right. While you are consumed by that focus, you have little room for niceties like small functions, nice names, decoupling, and cleanliness.

My apprentice and I just finished achieving a major milestone in FitNesse. It used to be that each new release of FitNesse was contained in a zip file that had all the files and directories laid out just perfectly. This is great for a brand new installation, but doesn’t work so well when you are upgrading. What we managed to get working last night was the ability for FitNesse to self-install from it’s jar file. Now, whether you are installing for the first time, or just upgrading, FitNesse will install itself into your environment appropriately.

If you’ve ever written a self-install like this, you know there are lots of niggling little details to attend to. While the problem is conceptually simple, the realization is a maze of complexity.

Last night, with dinner waiting on the table and getting cold, we got the last little problem licked, and saw the whole self-installation work as we wanted. We congratulated each other with a high-five, and then went upstairs to eat a well-deserved meal. Later that evening I went down to the office and checked in the code.

This morning I woke up, and while in the shower, I realized that we had a lot of work left to do. We need to go back over all that code and clean it up. I’m sure we left a swath of detritus while on the hunt.

The rush of victory should never be confused with the cold and crystalline concept of done. Once you get your systems to work, you still have to go back and clean up the wreckage left behind by the victorious battle. You are not done until the victorious code has been cleaned, polished, and oiled.

Clean Code and Battle Scarred Architecture

Uncle Bob — Wed, 20 May 2009 17:35:00 -0500

If you go here you’ll see that I struggle to keep the CRAP out of FitNesse. Despite the whimsical name of this metric (which I take a perverse delight in), I find it to be remarkably useful.

CRAP is a metric that is applied to each function in the system. The formula for CRAP is: CRAP = comp(f)^2. X (1 – cov(f)/100)^3. + comp(f)

Where comp(f) = the cyclomatic complexity of the function f. and cov(f) = the unit test coverage of function f.

So a function’s CRAP will be small iff the cyclomatic complexity is low and the test coverage is high. CRAP will be huge if cyclomatic complexity is high, and there is no coverage.

What does this have to do with architecture? Read on…

I work very hard to keep the ratio of crappy methods near .1%. Of the 5643 methods in FitNesse, only 6 are crappy, and five of those I have no control over.

If you study the graph you can see how quickly I react to even the slightest uptick in crap. I don’t tolerate it because it means that either I’ve got a horrifically complex method that needs to be refactored, or (and this is far more likely) I’ve got a method that isn’t sufficiently tested.

Why am I so fastidious about this? Why am I so concerned about keeping the crap out of FitNesse? The reason is pretty simple. It’s the least I can do.

If you look inside of FitNesse, you’ll find that there are lots of structures and decisions that don’t seem to make a lot of sense at first reading. There are complexities and abstractions that will leave you shaking your head.

For example. We generate all our HTML in code. Yes, you read that correctly. We write Java code that constructs HTML. And yes, that means we are slinging angle brackets around.

To be fair, we’ve managed to move most of the angle bracket slingers into a single module that hides the HTML construction behind an abstraction barrier. This helps a lot, but cripe who would sling angle brackets when template system are so prevalent? I hope nobody. But FitNesse was not conceived at a time when template systems made sense (at least to us).

Fear not, I am working through the Fitnesse code replacing the HTML generation with Velocity templates. It’ll take some time, but I’ll get it done. The point is, that just like every other software system you’ve seen, FitNesse is a collection of historical compromises. The architecture shows the scars of many decisions that have since had to be reversed or deeply modified.

What does this have to do with CRAP? Simply this. The battle scarred architecture is something that will never really go away. I can stop the bleeding, and disinfect the wounds, but there will always be evidence of the battle.

That scarring makes the system hard to understand. It complicates the job of adding features and fixing bugs. It decreases the effectiveness of the developers who work on FitNesse. And though I work hard to massage the scars and bandage the wounds, the war goes on.

But I can keep the CRAP out! I can keep the code so clean and simple at the micro level, that the poor folks who try to make sense out of the macro scale aren’t impeded by huge deeply nested functions that aren’t tested!

Think of it this way. CRAP is disease at the cellular level. CRAP is a rot so pervasive that it can infest every nook and cranny of the system. My system may have scars, but it’s not diseased! In fact, despite the evidence of battles long past, the FitNesse code is very healthy.

And I aim to keep it that way.

Why the sea is boiling hot

Uncle Bob — Mon, 11 May 2009 11:09:57 -0500

In my keynote at the Rails Conference last Wednesday, I spoke about What Killed Smalltalk and could it Kill Ruby Too?. In this keynote, as is my wont, I spoke about the need for our industry to define itself as a profession, and for us to define ourselves as professionals. I closed the keynote with this statement: “Professionalism does not mean rigid formalism. Professionalism does not mean adhering to beaurocrasy. Professionalism is honor. Professionalism is being honest with yourself and disciplined in the way you work. Professionalism is not letting fear take over.”

David Heinemeier Hansson, inventor of Rails, and affectionately referred to as @dhh has posted a blog in response in which he asserts that we need artists as well as professionals, and draws a dichotomy between the two. The dichotomy is a false one…

We are a community of artisans. We make things with our hands! We all strive, like @dhh, to make things of great beauty and utility. In no way, and by no means, do I wish to assail that artistry. Indeed, my hope is to free it. I want to convince all programmers that the desire for, and the pursuit of beauty is the way you satisfy your customers. The only way to go fast, is to go well.

I want developers to take pride in their work. I also want them to take pride in the way that they work. I want them to be able to look back on the last few days, weeks, and months, and be able to say to themselves, “I made something beautiful, and I made it well.”

We are a quirky lot. Some of us wear faded jeans with yesterday’s spaghetti on them. Others wear T-shirts that have PI wrapped around them. There are beards, tatoos, tongue-studs and hair in all shapes and colors. There are hawaiian shirts and sandals. There are jackets and ties and sometimes even suits. Some of us speak carefully. Some of us drop F-bombs at a whim. Some of us are liberal, and some are conservative. Some of us relish in being seen, and some of us are glad to be overlooked. In short, we are a group of diverse people who are drawn together by our common passion for code.

There is nothing wrong with this diversity. Indeed it’s healthy. The fact that we all think differently about styles, language, appearance, and values means that there are a zillion different ways that we can learn from each other. And in that learning grows the seed of our profession.

So @dhh is right, at least about the diversity. We should all relish the opportunity to share ideas with people who think differently than we do. But @dhh is wrong when he draws the dichotomy between artists and engineers. Every engineer is an artist, and every artist is an engineer. Every engineer strives for elegance and beauty. Every artist has the need to make their art actually work. The two are inextricably tied. You cannot be one without also being the other.

Now, certainly there are environments where the engineering side of things is emphasized over the artistry side. In extreme cases the artistry is repressed into near non-existence. Such places are soul-searing hell-holes that every programmer should strive to escape (or for the brave: change from within!) Indeed, @dhh implies that he worked in such places and found he was “faking [his] way along in this world.” I completely understand that.

But then @dhh makes his biggest mistake. He equates the professionalism I was talking about in my keynote with those repressive environments. He seems to think that professionalism and artistry are mutually exclusive. That wearing a green band means you give up on beauty. That discipline somehow saps programmer happiness.

But remember what I said when I closed my keynote: “Professionalism does not mean rigid formalism. Professionalism does not mean adhering to beaurocrasy. Professionalism is honor. Professionalism is being honest with yourself and disciplined in the way you work. Professionalism is not letting fear take over.”

This is not a complete definition; but it will serve for my current purposes. Because, you see, I made a big mistake during that keynote. And it is in how we deal with our own errors that the claim of professionalism is most frequently, and most thoroughly tested.

In my keynote I used a metaphor to link hormones and languages. I said that C++ was a testosterone language, but Java was an estrogen language. And then I used the word “insipid” to describe Java.

Now clearly C++ and testosterone have very little in common. My use of this metaphor was an oratory device – a joke. As far as the operation of that device is concerned, it was a success. The vast majority of the audience laughed, demonstrating to me that they were a) listening and b) understanding and c) open.

There is a kind of artistry in making oratory devices like this, and I take a certain amount pride in it. Such devices need to be timed appropriately, delivered skillfully, and used to gauge the audience. They can help to turn virtually any dry topic into a compelling speech.

On the other hand, the construction of this device had a significant flaw. I had equated women with weakness. This was not intentional. I do not think of women as weak. But there it was: Estrogen === Insipid. If you were a woman in that audience, how could you come to any conclusion other than “Uncle Bob thinks women are weak.”

How did I make this error? Lack of discipline. I did not test this keynote adequately. I should at least have run it past my wife! I mis-engineered my art! (Or perhaps my engineering was artless <grin>).

Within minutes of concluding my talk, the complaints appeared on twitter. Women who had been offended and hurt by the remark were tweeting their dissatisfaction. Some men were joining them.

There were two ways I could have responded to this. I could have asserted that these people were just being too sensitive; that they should have realized that this was just an oratory device and that I didn’t mean any harm; that they should just recognize me for who I am and not get so hung up in their own fears and values.

But I think that reaction would have been unprofessional. Why? Because it would have been dishonest. The truth was that this was just a stupid mistake. I built the device badly. I executed the device without testing it properly. I screwed up; and I needed to own up that. So I immediately tweeted apologies to those concerned and ate an appropriate amount of humble pie.

The reason I told you this story (at the risk of sounding somewhat self-aggrandizing) is so that I could use it to help define professionalism. The construction of my oratory device was unprofessional. I should have tested it better. I should have realized the danger of using gender comparisons and taken greater care with their use. I could have done better.

We professionals aren’t prefect. We all make mistakes. We recover from those mistakes by owning up to them as mistakes. We do not cover those mistakes by claiming that everyone else is wrong.

Confronting your mistakes and taking appropriate action is a discipline. It is a discipline of honor and self-honesty. It is a demonstration that we do not let the fear of our own errors take us over.

The Scatology of Agile Architecture

Uncle Bob — Sat, 25 Apr 2009 15:23:09 -0500

One of the more insidious and persistent myths of agile development is that up-front architecture and design are bad; that you should never spend time up front making architectural decisions. That instead you should evolve your architecture and design from nothing, one test-case at a time.

Pardon me, but that’s Horse Shit.

This myth is not part of agile at all. Rather it is an hyper-zealous response to the real Agile proscription of Big Design Up Front (BDUF). There should be no doubt that BDUF is harmful. It makes no sense at all for designers and architects to spend month after month spinning system designs based on a daisy-chain of untested hypotheses. To paraphrase John Gall: Complex systems designed from scratch never work.

However, there are architectural issues that need to be resolved up front. There are design decisions that must be made early. It is possible to code yourself into a very nasty cul-de-sac that you might avoid with a little forethought.

Notice the emphasis on size here. Size matters! ‘B’ is bad, but ‘L’ is good. Indeed, LDUF is absolutely essential.

How big are these B’s and L’s? It depends on the size of the project of course. For most projects of moderate size I think a few days ought to be sufficient to think through the most important architectural issues and start testing them with iterations. On the other hand, for very large projects, I seen nothing wrong with spending anywhere from a week to even a month, thinking through architectural issues.

In some circles this early spate of architectural thought is called Iteration 0. The goal is to make sure you’ve got your ducks in a row before you go off half-cocked and code yourself into a nightmare.

When I work on FitNesse, I spend a lot of time thinking about how I should implement a new feature. For most features I spend an hour or two considering alternative implementations. For larger features I’ve spent one or two days batting notions back and forth. There have been times when I’ve even drawn UML diagrams.

On the other hand, I don’t allow those early design plans to dominate once I start TDDing. Often enough the TDD process leads me in a direction different from those plans. That’s OK, I’m glad I made those earlier plans. Even if I don’t follow them they helped me to understand and constrain the problem. They gave me the context to evaluate the new solution that TDD helped me to discover. To paraphrase Eisenhower: Individual plans may not turn out to be helpful, but the act of planning is always indispensable.

So here’s the bottom line. If you are working in an Agile team, don’t feel guilty about taking a day or two to think some issues through. Indeed, feel guilty if you don’t take a little time to think things through. Don’t feel that TDD is the only way to design. On the other hand, don’t let yourself get too vested in your designs. Allow TDD to change your plans if it leads you in a different direction.

Crap Code Inevitable? Rumblings from ACCU.

Uncle Bob — Thu, 23 Apr 2009 04:56:22 -0500

I gave the opening Keynote at ACCU 2009 on Wednesday. It was entitled: The Birth of Craftsmanship. Nicolai Josuttis finshed the day with the closing keynote: Welcome Crappy Code – The Death of Code Quality. It was like a blow to the gut.

In my keynote I attempted to show the historical trajectory that has led to the emergence of the software craftsmanship movement. My argument was that since the business practices of SCRUM have been widely adopted, and since teams who follow those practices but do not follow the technical practices of XP experience a relentless decrease in velocity, and since that decrease in velocity is exposed by the transparency of scrum, then if follows that the eventual adoption of those technical XP practices is virtually assured. My conclusion was that Craftsmanship was the “next big thing” (tm) that would capture the attention of our industry for the next few years, driven by the business need to increase velocity. (See Martin Fowler’s blog on Flaccid Scrum) In short, we are on a trajectory towards a higher degree of professionalism and craftsmanship.

Nicolai’s thesis was the exact opposite of mine. His argument was that we are all ruled by marketing and that businesses will do whatever it takes to cut costs and increase revenue, and therefore businesses will drive software quality inexorably downward. He stipulated that this will necessarily create a crisis as the defect rates and deadline slips increased, but that all attempts to improve quality would be short lived and followed by a larger drive to decrease quality even further.

Josuttis’ talk was an hour of highly depressing rhetoric couched in articulate delivery and brilliant humor. One of the more memorable moments came when he playacted how a manger would respond to a developer’s plea to let them write clean code like Uncle Bob says. The manager replies: “I don’t care what Uncle Bob says, and if you don’t like it you can leave and take Uncle Bob with you.”

One of the funnier moments came when Josuttis came up with his rules for crap code, one of which was “Praise Copy and Paste”. Here he showed the evolution of a module from the viewpoint of clean code, and then from the viewpoint of copy-paste. His conclusion, delivered with a lovely irony, was the the copy-paste solution was more maintainable because it was clear which code belonged to which version.

It was at this point that I thought that this whole talk was a ribald joke, an elaborate spoof. I predicted that he was about to turn the tables on everyone and ringingly endorse the craftsmanship movement.

Alas, it was not so. In the end he said that he was serious about his claims, and that he was convinced that crap code would dominate our future. And then he gave his closing plea which went like this:

We finally accepted that requirements change, and so we invented Agile.

We must finally accept that code will be crap and so we must ???

He left the question marks on the screen and closed the talk.

This was like a blow to the gut. The mood of the conference changed, at least for me, from a high of enthralled geekery, to depths of hoplessness and feelings of futile striving against the inevitable. Our cause was lost. Defeat was imminent. There was no hope.

Bulls Bollocks!

To his credit, there are a few things that Josuttis got right. There is a lot of crap code out there. And there is a growing cohort of crappy coders writing that crap code.

But the solution to that is not to give up and become one of them. The solution to that is to design our systems so that they don’t require an army of maintainers slinging code. Instead we need to design our systems such that the vast majority of changes can be implemented in DSLs that are tuned to business needs, and do not require “programmers” to maintain.

The thing that Josuttis got completely wrong, in my mildly arrogant opinion, is the notion that low quality code is cheaper than high quality code. Low quality code is not cheaper; it is vastly more expensive, even in the short term. Bad code slows everyone down from the minute that it is written. It creates a continuous and copious drag on further progress. It requires armies of coders to overcome that drag; and those armies must grow exponentially to maintain constant velocity against that drag.

This strikes at the very heart of Josuttis’ argument. His claim that crappy code is inevitable is based on the notion that crappy code is cheaper than clean code, and that therefore businesses will demand the crap every time. But it has generally not been business that has demanded crappy code. Rather it has been developers who mistakenly thought that the business’ need for speed meant that they had to produce crappy code. Once we, as professional developers, realize that the only way to go fast is to create clean and well designed code, then we will see the business’ need for speed as a demand for high quality code.

My vision of the future is quite different from Josuttis’. I see software developers working together to create a discipline of craftsmanship, professionalism, and quality similar to the way that doctors, lawyers, architects, and many other professionals and artisans have done. I see a future where team velocities increase while development costs decrease because of the steadily increasing skill of the teams. I see a future where large software systems are engineered by relatively small teams of craftsmen, and are configured and customized by business people using DSLs tuned to their needs.

I see a future of Clean Code, Craftsmanship, Professionalism, and an overriding imperative for Code Quality.

Master Craftsman Teams.

Uncle Bob — Wed, 01 Apr 2009 16:51:00 -0500

In the early ‘70s F. T. Baker, and Fred Brooks wrote about the notion of Chief Programmer teams. The idea was that a highly skilled chief programer would be assisted by a team of helpers. The chief would write most of the code, delegating simpler tasks to the helpers.

Though the idea was tried at IBM and shown to work well, the idea has never really caught on. One reason may be that the economics don’t appear to be all that good. A team of four juniors making $50K supporting a senior making $100K costs $300K. That just seems like a lot of money for one well supported chief programmer writing most of the code.

But in light of the movement towards software craftsmanship, perhaps it’s time to dust off this old idea and look at it from a different perspective. Consider the following thought experiment…

We are in the throes of a recession that is challenging us to think differently about things. At the same time our universities are raising tuitions and increasing faculty salaries, while requiring ever less from both the faculty and the students. Today’s professors spend about 9 hours per week in the classroom. And the quality of the CS majors graduating from these institutions is, to put it bluntly, wretched.

Why should a young aspiring software professional spend four years and $200K+ to attend an institution that will teach them less about their chosen profession than 3 months of working on a real project with talented mentors? Indeed, why should employers pay $50K for undertrained programmers who are sure to make horrific messes for the next three years of their career?

Consider instead a team of craftspeople. At the center of this team is a master programmer. This is someone who has been programming for two decades or more. This person understand systems at a gut level, and can quickly make technical judgements without agonizing over them. Such a person can direct a team with the kind of calm confidence that only comes with years of experience and seasoning.

Our master has three lieutenant journeymen who themselves each have three apprentices. Our journeymen have at least five years of experience, and are capable both of taking and giving direction. They can make short term tactical decisions, and direct their apprentices accordingly. At the same time they take technical and architectural direction from the master.

The journeymen write a lot of code – almost as much as the master. They also throw away two thirds of the code written by the apprentices, and drive them to redo it better – always better. They teach the apprentices how to refactor, and often pair with them and sit with them while reviewing their code.

The master probably writes more code than any of the team members. This code is often architectural in nature, and helps to establish the seed about which the system will crystalize. The master also sets technical direction and overall architecture, and makes all critical technical decisions. However, the chief job of the master is to relentlessly drive the journeymen towards higher quality and productivity by establishing and enforcing the appropriate disciplines and practices. From the master’s point of view, it’s “my way or the highway.”

Yes, I know this is very anti-scrum. But this is a case where I think scrum in particular, and agile in general just got it wrong. No battle was won by gaining the consensus of the soldiers. And if you don’t think developing a system is a battle against time, resources, and attitudes, then you’ve never built a system. No great team has ever succeeded without a strong coach who imposed discipline, set the vision, and had the final authority over who played and who sat on the bench. Consensus and team rule may be useful tools some of the time, but only in the context of effective leadership.

The apprentices write a significant amount of code—most of which is discarded by the journeymen. The journeymen act as committers. When an apprentice delivers code that the journeyman finds acceptable, the journeyman will commit that code to their main line. Otherwise he sends it back to the apprentice to do over. By the same token the master accepts subsystems from the journeymen and, if acceptable, commits it to the master main line.

The master sees every line of code in the system. The Journeymen see every line of code written by their apprentices, and they write a significant amount themselves. Pairing is prevalent though not universal. Anyone on the team can pair with anyone else. It should not be uncommon for the master to pair with the apprentices from time to time.

Apprentices begin at a young age—perhaps 18 or 19. After all, the idea is that they aren’t going to a university. Rather, the brightest would start as apprentices right out of high school, while others might opt for two years of community college first.

Apprenticeship would last 2-4 years, depending on how quickly the apprentice learns. Apprentices earn no more than minimum wage. After all, they are young, barely productive, and are deriving great benefit from working with experienced journeyman under the guidance of a master.

Attaining journeyman status is something for the master to decide, with advice from the journeymen. The decision is bittersweet because it is a decision to eject the apprentice from the team so that he or she can start the journey. Usually the new journeyman would join the team of a different master.

Some apprentices simply won’t attain journeyman status. In the end, if they fail to make progress, they must be removed from the team. No one should stay an apprentice year after year. It is the responsibility of the journeymen and master to decide if an apprentice should be removed from the team.

The skills of new journeymen will gradually increase. At first they should have only one apprentice. Over the months and years their increasing leadership skills will allow them to take two or three at a time. The most experienced Journeymen might play the role of master in smaller projects.

Attaining master status is no small feat. It requires the consensus of other masters, and must be demonstrated through several successful accomplishments.

I expect a journeyman would make $50-100K depending on their experience. I expect a master would make $150-$250K or more.

Now imagine a team of 13. One master, three journeymen, and nine apprentices. This is a powerful unit. Its size is right at the sweet spot for hyper productive teams. It has the right balance of experience. And… it’s pretty cheap. This team costs ~$600K per year, or about $50K per person. Or (consulting companies take notice) about $23/person/hour.

It seems to me that such a team could do much more, much better, than a team of 12 graduates lead by a tech-lead with five years experience. The productivity and efficiency (and efficiencies of scale!) of this approach should be obvious. Moreover, I think that being an apprentice on a team like this would provide a much better, much faster, and much cheaper education than a university degree. Finally, I think that using teams of this kind could be a remarkably cost-effective way to get very high quality software written very quickly.

Let's Hear it for the Zealots!

Uncle Bob — Tue, 10 Mar 2009 16:58:00 -0500

In his March 1st, 2009 column in SDTimes, Andrew Binstock takes the “Zealots of Agile” to task for claiming that Agile is the one true way. He made the counter-argument that non-agile projects have succeeded, and that agile projects have failed. He implied that the attitudes of the agile zealots are blind to these facts.

What a load of dingoes kidneys!

There is a difference between a zealot, and a religious fanatic. A religious fanatic cannot envision themselves to be wrong. We in the agile community may indeed be zealots, but we know we can be wrong. We know that projects succeeded before agile, and we know that agile projects have failed. We know that there are other schools of thought that are just as valid as ours. Indeed, most of us expect (even hope!) that agile will be superseded by something better one day.

Some of you may remember my keynote address at the 2002 XPUniverse in Chicago. My son, Micah, and I got up in front of the group dressed in Jiu Jistsu garb, and commenced to do a martial arts demonstration.

After the demonstration I made the point that a student of Karate could have given an equally spellbinding, though quite different, show. A student of Tae Kwon Do would amaze us with still other wonderful moves. The morale, of course, was that there are many ways to get the job done. There are many equally valid schools of thought.

However, the knowledge that there are other valid schools of thought does not dilute the zeal a student has for the school of thought he has chosen to study. Do you think that a Jiu Jistsu master isn’t a zealot? You’d better believe he is! To be great, one must have zeal. To rail against zeal is to rail against the desire for greatness. It is zeal that makes you go the extra distance to become great. The Jiu Jitsu master may respect the Karate master, but his goal is to become great at Jiu Jitsu.

So I think it is healthy that agile proponents are vociferous advocates of their school of thought. They are like Karate students who are excited about their new skills, and want to teach those skills to others. If their zeal to excel can infect others with similar zeal, so much the better. We need people in this industry who want to excel.

In his column Andrew derided agile zealots for claiming that Agile was the one true path. I don’t know of any prominent agile advocate who has ever made that claim. Indeed, when Kent Beck and I sat down in 1999 to plan out the XP Immersion training, we were very concerned that we might turn the XP movement into a religion. We did not want to create the impression that XPers were the chosen people and that wisdom would die with us. So we were, in fact, very careful to avoid any hint of the “one true path” argument.

As evidence of this, consider the Snowbird meeting in 2001. That meeting was inclusive, not exclusive. We invited people from all over the spectrum to join us there. DSDM, FDD, Scrum, Crystal, MDA, & RUP were all represented; and the agile manifesto (Yes, Andrew, Manifesto![1]) was the result. That manifesto was a simple statement of values, not a claim to deific knowledge.

Am I an Agile Zealot? Damned right I am! Do I think Agile is the best way to get software done? Damned right I do! I wouldn’t be doing Agile if I didn’t think it was best. Do I think Agile is better than someone else’s school of thought? Let me put it this way: I’m quite happy to go head to head against someone who follows another school of thought. I believe I’d code him right into the ground! Isn’t that what any good student of his craft would think? But I also recognize that one day I’ll lose that competition, and that will be a great day for me because it will give me the opportunity to learn something new and better.

Andrew also mentioned that Kent Beck and I are so obsessed with our “evangelical fervor” that we: “endlessly test and endlessly refine and refactor code to the point that it becomes an all-consuming diversion that does not advance the project”. WTF? I’ve been making monthly releases of FitNesse, and Kent has been cranking out releases of JunitMax; neither of us has been so busy gilding our respective lilies that we are failing to move our respective projects forward. Look around Andrew, we’re shipping code over here brother.

As part of his rant against our “evangelical fervor” Andrew asked this rather snide question: “I broke up that code into a separate class and added seven methods for what benefit, again?” This would appear to be a dig against refactoring – or perhaps excess refactoring. Still the question has a simple answer. We would only break up a module into a class with seven methods if that module was big, ugly, and complex, and by breaking it up we made it simple and easy to understand and modify. It seems to me that leaving the module in a messy state is irresponsible and unprofessional.

Andrew complains that there is very little Agile experience with projects over six million lines of code. I’m not sure where he got that number from, perhaps he’s been watching Steve Austin re-runs. Actually, the majority of my 2008 consulting business has been with very large companies with products well over six million lines of code. Agile-in-the-Large is the name of the game in those companies!

Finally, Andrew says Michael Feathers is a “moderate” Agile proponent. This is the same Michael Feathers who wrote a book defining Legacy code as code without tests – hardly a moderate point of view. Michael is a master, not a moderate.

I think Andrew has fallen into a common trap. He sees the zeal of the agile proponents and mistakes it for religious fanaticism. Instead of respecting that zeal as the outward manifestation of the inward desire to excel, he rails against it as being myopic and exclusive and then views all the statements of the agile proponents through that clouded viewport. Once you have decided that someone is a religious fanatic, it is difficult to accept anything they say.

I think standing against religious fanaticism in the software community is a good thing. I also think that those of us who are zealous must constantly watch that we are not also becoming blind to our own weaknesses and exclusive in our outlook. But I could wish that those who fear religious fanaticism as much as I do would stop confusing it with professional zeal.

So here’s the bottom line. People who excel are, by definition, zealots. People who aren’t zealous, do not excel. So there’s nothing wrong with being a zealot; indeed, zeal is a very positive emotion. The trick to being a zealot is to be mindful that the day will eventually come when some other zealot from a different school of thought will code you into the ground. When that happens, you should thank him for showing you a better way.

¹ I kicked off the Snowbird meeting by challenging the group to write a “manifesto”. I chose the word “manifesto” because some years earlier I had read The Object Oriented Database System Manifesto and I thought it was a clever use of the word.

Quality: It's alive! It's ALIVE!

Uncle Bob — Sat, 07 Mar 2009 15:25:00 -0600

Recently James Bach wrote a compelling post entitled Quality is Dead. As much as I’d like to agree, something interesting has just happened that tempts me to believe in a rebirth.

Just when James finally declares the death of quality, along comes the Manifesto for Software Craftsmanship. This simple document that builds upon the four values declared in the Agile Manifesto went live late on Friday evening. Now, at 3PM on Saturday over 600 people have signed it!

What does this mean? I think it means that many of us have seen what James was talking about, and are tired of it. We don’t want our users posting blogs – as James did – crying that they HATE us. We want our users to be astonished by the elegance and simplicity of our creations. We want other software developers to look under the hood and be amazed and impressed at the clarity and simplicity they see.

We want to be proud of our work!

We want our quality back!

And so a few weeks ago some software developers gathered at the offices of 8th Light in Libertyville, Illinois, and began the process of creating this manifesto. There followed a very active and passionate discussion in the Software Craftsmanship email group. The end result was this manifesto.

I encourage all of you to read and sign the manifesto, join the email group, and help bring quality back to life in the software profession.

Whiners that Fail

Uncle Bob — Fri, 27 Feb 2009 18:55:40 -0600

The acronym is not accidental.

Recently, Michael Feathers posted a hugely valuable blog entitled: 10 Papers every Programmer should read at least twice. Do you have any idea how valuable this is? Michael has read hundreds, if not thousands, of papers, and he is freely offering his opinion about which ten are the best. A wise man would pay thousands of dollars for this information. But some people prefer to whine.

Get a load of this gem posted by someone named David:

... you should have actual pdfs of these papers, not links that require you to pay. If these papers were truly important, then you would be able to read them without paying … your tone is arrogant and annoying. Get over yourself.

Clearly this is a troll. Normally I do not feed the trolls. But this one pushed my buttons (I know, I know, that’s the intent of a troll—well it worked.) But it wasn’t only this response. Here’s another one by someone named Mik:

Thanks for the list, but please don’t link to ACM portal, it is a pay site. Always link to a free version if it is available. Yes, these papers are worth spending a minute or two searching for a free version, and yes they are probably worth spending real money to read as well, but if a free version exists, save us all the time and money and link to it.
One small moment for an author, or one giant waste of time for mankind.

What a bunch of whiners!

Here is this golden nugget laid in front of them, and they complain that it’s too heavy to pick up. Fools! Losers!

I’m sorry for the anger, but the attitude makes me crazy. Are we professionals who stand on our own and take responsibility for our careers? Or are we children who expect our parents to wipe our bums?

I know I’m preaching to the choir. The people who take the time to read this blog don’t generally need to be told this. But on the off-chance that this might actually reach someone and change their attitude…here goes.

YOU, and NO ONE ELSE, is responsible for your career. Your employer is not responsible for it. You should not depend on your employer to advance your career. You should not depend on your employer to buy you books, it’s great if they do, but it’s not really their responsibility. If they won’t buy them, YOU buy them! It’s not your employers responsibility to teach you a new language. It’s great if they send you to a training course, but if they don’t YOU teach the language to your self!

I fear greatly that our culture of entitlement has created a bunch of whining sissy programmers who think it’s unfair that they have to pay for a copyrighted article. (Pay? Who me? That’s my employer’s job! That’s my teacher’s job! That’s Michael Feathers’ job! I mean if they want me to be a good programmer then they’d better not expect ME to pay for these articles! They’d better not expect ME to do a google search for an article! They’d better come right over here to my cubicle between 9am and 10am and read the article to me while stroking my hair!)

The world does not owe you a living. Your employer does not owe you a career. And Michael Feathers’ does not owe you access to free articles.

Getting a SOLID start.

Uncle Bob — Thu, 12 Feb 2009 20:43:35 -0600

I am often asked: “How should I get started with SOLID principles?” Given the recent interest and controversy about the issue, it’s probably time I gave a written answer.

First things first.

You can read about the Solid principles here. This is a paper I wrote nearly a decade ago. It uses a somewhat dated form of UML, and examples are in C++. Also the concepts are severely abbreviated. This is the Cliff Notes. Still, it should give you an initial notion of the names, definitions, and concepts.

There are quite a few papers here that explain the principles in much more detail. Just click on the “Design Principles” topic to see them all. Though I think you’ll find the other topics pretty interesting too.

Finally, the principles are definitively described in two books: Agile Software Development: Principles, Patterns, and Practices, and Agile Principles Patterns, and Practices in C#. You can see descriptions of these books here

What do I mean by “Principle”

The SOLID principles are not rules. They are not laws. They are not perfect truths. The are statements on the order of “An apple a day keeps the doctor away.” This is a good principle, it is good advice, but it’s not a pure truth, nor is it a rule.

The principles are mental cubby-holes. They give a name to a concept so that you can talk and reason about that concept. They provide a place to hang the feelings we have about good and bad code. They attempt to categorize those feelings into concrete advice. In that sense, the principles are a kind of anodyne. Given some code or design that you feel bad about, you may be able to find a principle that explains that bad feeling and advises you about how to feel better.

These principles are heuristics. They are common-sense solutions to common problems. They are common-sense disciplines that can help you stay out of trouble. But like any heuristic, they are empirical in nature. They have been observed to work in many cases; but there is no proof that they always work, nor any proof that they should always be followed.

Following the rules on the paint can won’t teach you how to paint.

This is an important point. Principles will not turn a bad programmer into a good programmer. Principles have to be applied with judgement. If they are applied by rote it is just as bad as if they are not applied at all.

Having said that, if you want to paint well, I suggest you learn the rules on the paint can. You may not agree with them all. You may not always apply the ones you do agree with. But you’d better know them. Knowledge of the principles and patterns gives you the justification to decide when and where to apply them. If you don’t know them, your decisions are much more arbitrary.

So how do I get started?

“There is no royal road to Geometry” Euclid once said to a King who wanted the short version. Don’t expect to skim through the papers, or thumb through the books, and come out with any real knowledge. If you want to learn these principles well enough to be able to apply them, then you have to study them. The books are full of coded examples of principles done right and wrong. Work through those examples, and follow the reasoning carefully. This is not easy, but it is rewarding.

Search your own code, and the code of others, for applications and violations of the principles. Determine whether those applications and violations were justified. Improve the designs by applying one or more principles, and see if the result is actually better. If you’d like to study a code base that was written by people who’ve been immersed in these principles for years, then download the source for FitNesse from fitnesse.org.

Conduct or join a discussion group at work. Do brown-bags, lunch-n-learns, reading-groups, etc. People who learn in groups learn much more, and much more quickly, than people who study alone. Never underestimate the power of the other guy’s viewpoint.

By the same token, join a user group with people from different companies. Attend monthly meetings and listen to the speakers, or participate in the discussions. Again, you’ll learn an awful lot that way.

Practice, practice, practice, practice. Be prepared to make lots of mistakes.

And, of course, apply what you’ve learned on the job, and ask your peers to review your work. Pair program with them if at all possible. “As iron sharpens iron, so one man sharpens another.” (Proverbs 27:17)

An Open Letter to Joel Spolsky and Jeff Atwood

Uncle Bob — Fri, 06 Feb 2009 20:53:00 -0600

Joel, Jeff,

Here is an open letter to the two of you that I hope we can use in our upcoming StackOverflow #40 podcast. I won’t post it on my blog until we’ve been able to iterate it a bit. I’ve spent several hours on this trying to balance the tone. It used to be a lot longer <grin>.

Dear Joel and Jeff

In the Stack Overflow Podcast #38 you said: “Last week I was listening to a podcast on Hanselminutes, with Robert Martin talking about the SOLID principles … they all sounded to me like extremely bureaucratic programming that came from the mind of somebody that has not written a lot of code, frankly.”

And again later: ”...it seems to me like a lot of the Object Oriented Design principles you’re hearing lately from people like Robert Martin and Kent Beck and so forth have gone off the deep end into architecture for architecture’s sake.”

And yet again: “People that say things like this have just never written a heck of a lot of code.”

And still again: “One of the SOLID principles, (a butchering of the Single Responsibility Principle) and it’s just… idiotic! You can’t build software that way!”

I hope you’ve had time to perform a little due diligence since then, and that you now realize that your statements were unfair, harmful, and were pretty much crap.

I understand that you were trying to make a valid point and that the words just got away from you a bit. I quite agree that a dogmatic or religious application of the SOLID principles is neither realistic nor beneficial. If you had read through any of the articles and/or books I’ve written on these principles over the last 15 years, you’d have found that I don’t recommend the religious or dogmatic approach you blamed me for. In short, you jumped to a erroneous conclusion about me, and about the principles, because you weren’t familiar with the material.

I understand that this was a podcast, and that you were both just jawing. However, your podcasts are a product that you ship to everyone in the world. The content of that product can do great good; but it can also do great and permanent harm. One would think that you’d want to be careful with such a product. Yet in the intensity of the moment you got a bit careless and spewed some crap instead. That’s fine, everybody makes mistakes. But then you shipped it! You shipped a product that had a huge bug in it. You should have had tests!

Could it be that when you said ”...quality just doesn’t matter that much…” you were being serious. Clearly the quality of podcast #38 didn’t matter much to you, since you shipped it without verifying that your information was accurate. As a result, you did unjust harm to me. Were I a more litigious person, we might be having this discussion in court.

Now I’m quite certain that you don’t want to ship bad product. I just think that you’ve been careless with your production process. You haven’t put in place a mechanism that will stop you from shipping crap. So I suggest that you practice Test Driven Development with your podcast. Write down the acceptance criteria beforehand (I suggest lawsuit avoidance would be a high priority), and then review the product against those criteria afterwards. Yes, this will take some time, and there will be a cost. However, it will pay back handsomely because your product will be far better, and you will avoid embarrassments like this one (or worse). And, after all, products deserve and require this kind of care. So do your listeners. This is just simple professionalism. Don’t ship shit. —-——-

Some of the other points we could talk about in the podcast:

TDD for real this time. You guys can just throw your complaints at me and I’ll address them. * For example, I’d like to explain how one would test Joel’s JPeg compression feature.
ISP and SRP, the two principles that Joel butchered.
Mike Connie’s observation (in #39) about interfaces.
How, when, and why to apply Principles and Patterns.
From Stack Overflow: * Large Switch statements: Bad OOP? * JSON is used only for JavaScript? (Rant on XML) * ASP.Net – How to effectively use design patterns without over-engineering!

Speed Kills

Uncle Bob — Tue, 03 Feb 2009 17:03:33 -0600

Ron Jeffries wrote a wonderful blog about how we kid ourselves that there is a tradeoff between quality and speed. Isn’t there?

That depends on what you mean by “speed”. If you mean rushingtogetawholebunchofshitcodedandsortofrunning, then quality is not for you. On the other hand, if by “speed” you mean delivering working software quickly and repeatably release after release after release; then maintaining high quality is your only option.

The first course is that of the adrenline junkie. The hacker stereotype of the twinkie eating cola guzzling nerd who cackles like the Wicked Witch of the West while tunnel visioning into his screen and keyboard. The second course is that of the professional programmer, the craftsman who with calm demeanor and deliberate confidence servers his clients, and his career.

We all feel the pressure to rush. It’s like a drug, really. It feels good to through off the disciplines and justcodeupawholebunchofshit. We get the adrenline high, and we feel as though we are skating on the edge between success and failure, sailing so close to the wind that we’re on the verge of tipping, getting the rush of speed. When we get our code working we feel victorious!.

What we should be feeling is stupid. Because we actually were skating on the edge between success and failure. We should not feel victorious, we should feel shame.

We justify this addiction to speed by telling ourselves that our bosses want us to go fast. We tell ourselves that time-to-market requires that we sail close to the wind. And then we put the needle in our arm and let fly another dose of speed—oblivious to the consequences.

I teach a lot of programming classes in which I give the students programs to write. I tell them that the goal is not to finish the programs, but to learn the skills of programming. I tell them that the code they are writing will all be deleted after they leave. I tell them to focus on the techniques and disciplines they are learning, not on finishing the programs. And yet on Friday afternoon there are always a group of folks who are flailing away at their keyboards trying desperately to finish something that they know will be deleted as soon as they leave.

These people have surrendered to the drug of speed. They need to conquer. They manufactured their own deadline pressure and behaved as if their boss was breathing down their necks.

How do you really get things done quickly? You listen to your grandparents. Remember what they told you? “Slow and steady wins the race.” and “Anything worth doing is worth doing well.” How does a professional craftsman get things done quickly? He/she adopts an attitude of calm, focuses on the problem to be solved, and then step by step solves that problem without rushing, without yielding to the need for speed, without surrendering to the desire for a quick conquest.

When you feel the temptation to rush, resist it. Leave the keyboard and walk around. Distract yourslef with something else. Do not give in to the call of your addiction.

If you want to be a craftsman. If you want to be someone who builds software quickly, accurately, and repeatably. If you want to be someone that your employer respects and values. If you want to respect yourself. Then remember this simple fact. Speed Kills.

Multi-dimensional Seniority

Uncle Bob — Tue, 20 Jan 2009 23:10:38 -0600

I was at a meeting of the Chicago Software Craftsmanship group last night. Dave Hoover and Paul Pagel were talking about the different apprenticeship models at Obtiva and 8thLight.

The 8thLight model uses a 90 day probation period during which an “apprentice” has an appointed mentor who guides the apprentice to learn enough to become a “craftsman”.

The Obtiva model is less structured, but there’s no probationary period. Once an apprentice had delivered something meaningful for a client, and had been recognized by the others, he/she would become a “consultant”, and eventually a “senior consultant”

Both speakers used the term “journeyman” but neither defined it. Both speakers agreed that 90 days, or a year, or even two years was insufficient for an apprentice to become a true journeyman.

After the talk I started to ponder this question of seniority. What makes a software developer senior? Certainly the amount of time spent in a career is a big factor. Someone with 15 years of experience is certainly senior.

However, there’s that old joke about the guy who had 15 years of experience, the same year fifteen times in a row. That’s a real concern. It is perfectly possible to work for 15 years and dig yourself into a rut you can’t get out of. Think of all those Cobol programmers who never learned anything else.

Both Paul and Dave talked about how they planned for their apprentices to grow with the company. They both lamented apprentices that had left the company to do other things.

This made me think about the apprentices we’ve had at Object Mentor. (Paul was one of them. Dave was almost another.) Over the years we’ve had seven or so apprentices. Today, we have none. Not a single apprentice has stayed at Object Mentor. Is this bad?

It hasn’t been bad for the apprentices. Two went on to form their own company (8thLight). Others have made names for themselves in the community. More than one has written a book. Nearly all are passionate craftsmen who care about their profession, and who are making an impact.

It wasn’t bad for Object Mentor either. The apprentices moved on when it was a good time for them to move on, and we have kept good relationships with them. More than one customer has come our way because of referrals from past apprentices.

So at the end of the meeting, when everyone had milled away from the central area, I went to the whiteboard and scratched out a simple diagram. It was an graph with Seniority on the horizontal axis, and Journey on the vertical axis.

Then I drew a nearly horizontal line starting from the origin and hugging the Seniority axis. This is the career path of the guy who has the same year of experience 15 times in a row. Let’s call him Sam. Then I drew another line which was a stair-step that went towards the upper right at something less then a 45 degree angle. This is the career path of a developer who changes jobs a few times during his/her career. Let’s call her Jasmine.

I don’t think there is a vertical path. The only way to go on a journey is to spend some time at it, and that drives you horizontally.

OK, now maybe I’ve just spent a dozen paragraphs explaining the obvious. To-wit: A journeyman is a journeyman because he/she’s gone on a journey. Does that journey have to involve changing employers? I think it probably does. You see, there’s a problem.

The student/teacher relationship is persistent. At least in my experience it is. The only way for a teacher and student to ever consider themselves to be peers is for the relationship to break and then reform later in a new context.

Sam, has remained an apprentice, even though he’s got lots of experience. Why? Because he still works with his original mentors who consider him to be their student. Jasmine, on the other hand, has seen the world, experienced many different situations in many different places. She could become a master.

OK, ok. I’m belaboring the point. So let me conclude. If you are considering an apprenticeship program it seems to me that the most important aspect of that program is that the relationship is temporary, and results in a journey. In other words, don’t expect to hold on to your apprentices for very long.

Fudge anyone?

Uncle Bob — Wed, 31 Dec 2008 16:01:00 -0600

Back in September, when I was just staring the Slim project, I made a crucial architectural decision. I made it dead wrong. And then life turned to fudge…

The issue was simple. Slim tables are, well, tables just like Fit tables are. How should I parse these tables? Fit parses HTML. FitNesse parses wiki text into HTML and delivers it to Fit. Where should Slim fit in all of this?

Keep in mind that Fit must parse HTML since it lives on the far side of the FitNesse/SUT boundary. Fit doesn’t have access to wiki text. Slim tables, on the other hand, live on the near side of the FitNesse/SUT boundary, and so have full access to wiki text, and the built in parsers that parse that text.

So it seemed to me that I had two options.

Follow the example of Fit and parse wiki text into HTML, and then have the Slim Tables parse the HTML in order to process the tables.
Take advantage of the built in wiki text parser inside of FitNesse and bypass HTML altogether.

I chose the latter of the two because the Parsing system of FitNesse is trivial to use. You just hand it a string of wiki text, and it hands you a nice little parse tree of wiki widgets. All I had to do was walk that parse three and process my tables. Voila!

This worked great! In a matter of hours I was making significant progress on processing Slim decision tables. Instead of worrying about parsing HTML and building my own parse tree, I could focus on the problems of translating tables into Slim directives and then using the return values from Slim to colorize the table.

Generating html was no problem since that’s what FitNesse does anyway. All I had to do was modify the elements of the parse tree and then simply tell the tree to convert itself to HTML. What a dream.

Or so it seemed. Although things started well, progress started to slow before the week was out. The problem was that the FitNesse parser is tuned to the esoteric needs of FitNesse. The parser makes choices that are perfectly fine if your goal is to generate HTML and pass it to Fit, but that aren’t quite so nice when you’re goal is to use the parse tree to process Slim tables. As a simple example, consider the problem of literals.

In FitNesse, any camel case phrase fits the pattern of a wiki word and will be turned into a wiki link. Sometimes, though, you want to use a camel case phrase and you don’t want it converted to a link. In that case you surround the phrase with literal marks as follows: !- FitNesse-!. Anything between literal marks is simply ignored by the parser and passed through to the end.

Indeed, things inside of literals are not even escaped for html! If you put hi into a wiki page, it will escape the text you’ll see hi on the screen instead of a bold “hi”. On the other hand, if you put !- hi-! on a page, then the HTML is left unescaped and a boldfaced “hi” will appear on the screen.

I’m telling you all of this because the devil is in the details—so bear with me a bit longer.

You know how the C and C++ languages have a preprocessor? This preprocessor handles all the #include and #define statements and then hands the resulting text off to the true compiler. Well, the wiki parser works the same way! Literals and !define variables are processed first, by a different pass of the parser. Then all the rest of the wiki widgets are parsed by the true parser. The reason that we had to do this is even more devilishly detailed; so I’ll spare you. Suffice it to say that the reasons we need that preprocessor are similar to the reasons that C and C++ need it.

What does the preprocessor do with a literal? It converts it into a special widget. That widget looks like this: !lit?23? What does that mean? It means replace me with the contents of literal #23. You see, when FitNesse sees !- FitNesse-! it replaces it with !lit?nn? and squirrels FitNesse away in literal slot nn. During the second pass, that !lit?nn? is replaced with the contents of literal slot nn. Simple, right?

OK, now back to SLIM table processing. There I was, in Norway, teaching a class during the day and coding Slim at night, and everything was going just great. And then, during one of my tests, I happened to put a literal into one of the test tables. This is perfectly normal, I didn’t want some camel case phrase turned into a link. But this perfectly normal gesture made a unit test fail for a very strange reason. I got this wonderful message from junit: expected MyPage but was !lit?3?

I knew exactly what this meant. It meant that the value MyPage had been squirreled away by the first pass of the parser. I also knew that I had utterly no reasonable way of getting at it. So I did the only thing any sane programmer would do. I wrote my own preprocessor and used it instead of the standard one. This was “safe” since in the end I simply reconvert all the colorized tables back into wiki text and let the normal parser render it into HTML.

It was a bit of work, but I got it done at one am on a cold Norwegian night. Tests passing, everything great!

Ah, but no. By writing my own preprocessor, I broke the !define variable processing – subtly. And when I found and fixed that I had re-broken literal processing – subtly.

If you were following my tweets at the time you saw me twitter ever more frantically about literals. It was the proverbial ball of Mercury. Every time I put my thumb on it¹ it would squirt away and take some new form.

I fell into a trap. I call it the fudge trap. It goes like this:

forever do {“I can make this work! Just one more little fudge right here!”}

I was in this trap for two months! I was making progress, and getting lots of new features to work, but I was also running into strange little quirks and unexpected bizarre behaviors caused by the fudging I was doing. So I’d fudge a little more and then keep working. But each little fudge added to the next until, eventually, I had a really nasty house of cards (or rather: pile of fudge) ready to topple every time I touched anything else. I started to fear my own code². It was time to stop!

I knew what I had to do. I had to go back to my original architectural decision and make it the other way. There was no escape from this. The FitNesse parser was too coupled to the wiki-ness, and there was no sane way to repurpose it for test table processing.

I dreaded this. It was such a big change. I had built so much code in my house of fudge. All of it would have to be changed or deleted. And, worse, I needed to write an HTML parser.

I was lamenting to Micah about this one day in late November, and he said: “Dad, there are HTML parsers out there you know.”.

Uh…

So I went to google and typed Html Parser. Duh. There they were. Lots and lots of shiny HTML parsers free for the using.

I picked one and started to fiddle with it. It was easy to use.

Now comes the good part. I had not been a complete dolt. Even when I was using the FitNesse parse tree, I ate my own dogfood and wrapped it in an abstract interface. No part of the Slim Table processing code actually knew that it was dealing with a FitNesse parse tree. It simply used the abstract interface to get its work done.

That meant that pulling out the wiki parser and putting in the HTML parser was a matter of re-implementing the abstract interface with the output of the new parser (which happened to be another parse tree!). This took me about a day.

There came a magic moment when I had both the wiki text version of the parser, and the HTML version of the parser working. I could switch back and forth between the two by changing one word in one module. When I got all my tests passing with both parsers, I knew I was done. And then the fun really began!

I deleted ever stitch of that wiki parse tree pile of fudge. I tore it loose and rent it asunder. It was gone, never to darken my door with it’s fudge again.

It took me a day. A day. And the result is 400 fewer lines of code, and a set of Slim tables that actually work the way they are supposed to.

Moral #1: “Fudge tastes good while you are eating it, but it makes you fat, slow, and dumb.”

Moral #2: “Eat the damned dog food. It’ll save your posterior from your own maladroit decisions.

¹ I do not recommend that you actually put your thumb on any Mercury. Never mind that I used to play with liquid Mercury as a child, sloshing it around from hand to hand and endlessly stirring it with my finger. Wiser heads than I have determined that merely being in the same room with liquid Mercury can cause severe brain damage, genetic corruption, and birth defects in your children, grandchildren, and pets.

² Fearing your own code is an indicator that you are headed for ruin. This fear is followed by self-loathing, project-loathing, career-loathing, divorce, infanticide, and finally chicken farming.

Hudson -- A Very Quick Demo

Uncle Bob — Thu, 11 Dec 2008 20:04:00 -0600

I recently set up Hudson as the continuous integration server for FitNesse. I was impressed at how very very simple this was to do. The Hudson folks have adopted the same philosophy that we adopted for FitNesse: download-and-go.

So I recorded a quick (<5min) video showing you how I set this up. In this video you'll see me start Hudson from scratch, create the job to build FitNesse, and then run the FitNesse build.

Hudson—A Very Quick Demo from unclebob on Vimeo.