Object Mentor Blog: Category Software Craftsmanship

Sufficient Design means Damned Good Design.

Uncle Bob — Wed, 28 Apr 2010 18:02:10 -0500

@JoshuaKerievsky wrote a blog entitled “Sufficient Design”.

Josh makes this point:

‘Yet some programmers argue that the software design quality of every last piece of code ought to be as high as possible. “Produce clean code or you are not a software craftsman!”’

He goes on to say:

“Yet ultimately the craftsmanship advice fails to consider simple economics: If you take the time to craft code of little or moderate value to users, you’re wasting time and money.”

Now this sounds like heresy, and I can imagine that software craftsmanship supporters (like me) are ready to storm the halls of Industrial Logic and string the blighter up by his toenails!

But hold on there craftsmen, don’t get the pitchforks out yet. Look at the scenario that Josh describes. It’s quite revealing.

Josh’s example of “not being a craftsman” is his niggling worry over a function that returns a string but in one derivative case ought to return a void.

Horrors! Horrors! He left a bit of evolved cruft in the design! Revoke his craftsman license and sick the dogs on him!

Ah, but wait. He also says that he spent a half-day trying to refactor this into a better shape but eventually had to give up.

The fool! The nincompoop! The anti-craftsman! A pox on him and all his ilk!

OK, enough hyperbole. It seems to me that Josh was behaving exactly as a craftsman should. He was worrying about exactly the kinds of things a craftsman ought to worry about. He was making the kinds of pragmatic decisions a craftsman ought to make. He was not leaving a huge mess in the system and rushing on to the next feature to implement. Instead he was taking care of his code. The fact that he put so much energy, time, and thought into such a small issue as an inconsistent return type speaks volumes for Josh’s integrity as a software craftsman.

So, as far as Josh Kerievsky is concerned “Sufficient Design” is “Pretty Damned Good Design”.

Look, all our software will have little tiny warts. There’s no such thing as a perfect system. Craftsmen like Josh work hard to keep those warts as small as possible, and to sequester them into parts of the system where they can do the least harm.

So the only aspect of Josh’s post that I disagree with is his contention that the “craftsman” message is one of unrelenting perfection. Craftsmen are first and foremost pragmatists. They seek very high quality code; but they are not so consumed with perfection that they make foolish economic tradeoffs.

Sapient Testing: The "Professionalism" meme.

Uncle Bob — Thu, 15 Apr 2010 06:18:00 -0500

James Bach gave a stirring keynote today at ACCU 2010. He described a vision of testing that our industry sorely needs. Towit: Testing requires sapience.

Testing, according to Bach, is not about assuring conformance to requirements; rather it is about understanding the requirements. Even that’s not quite right. It is not sufficient to simply understand and verify the requirements. A good tester uses the behavior of the system and the descriptions in the requirements, (and face-to-face interaction with the authors of both) to understand the motivation behind the system. Ultimately it is the tester’s job to divine the system that the customer imagined; and then to illuminate those parts of the system that are not consistent with that imagination.

It seems to me that James is attempting to define “professionalism” as it applies to testing. A professional tester does not blindly follow a test plan. A professional tester does not simply write test plans that reflect the stated requirements. Rather a professional tester takes responsibility for interpreting the requirements with intelligence. He tests, not only the system, but also (and more importantly) the assumptions of the programmers, and specifiers.

I like this view. I like it a lot. I like the fact that testers are seeking professionalism in the same way that developer are. I like the fact that testing is becoming a craft, and that people like James are passionate about that craft. There may yet be hope for our industry!

There has been a long standing frission between James’ view of testing and the Agile emphasis on TDD and automated tests. Agilists have been very focussed on creating suites of automated tests, and exposing the insanity (and inanity) of huge manual testing suites. This focus can be (and has been) misinterpreted as an anti-tester bias.

It seems to me that professional testers are completely compatible with agile development. No, that’s wrong. I think professional testers are utterly essential to agile development. I don’t want testers who rotely execute brain-dead manual test plans. I want testers using their brains! I want testers to be partners in the effort to create world-class, high-quality software. As a professional developer I want – I need – professional testers helping me find my blind spots, illuminating the naivete of my assumptions, and partnering with me to satisfy the needs of our customers.

Saying "NO".

Uncle Bob — Fri, 04 Dec 2009 08:35:55 -0600

I saw this cartoon in a tweet today. It’s the story of how a boss corrupts the work of a professional. It’s a funny cartoon, and a sad story that happens all too often in our profession. But who, exactly, was at fault?...

The difference between a laborer and a professional is that a laborer takes orders from his boss, and a professional provides input to his superiors. Laborers are hired to take direction. Professionals are hired to ensure that the direction chosen makes sense.

Imagine this conversation between a patient and a doctor:

Patient: “My arm hurts.” Doctor: “What would you like me to do about it?” Patient: “Make my arm stop hurting.” Doctor: “Do you want me to cut it off?, I can do that.” Patient: “No, I just want it to stop hurting.” Doctor: “I could cut all the nerves to your arm. That’ll stop it.” Patient: “Isn’t there something less drastic you could do?” Doctor: “Ooops, sorry, time for my break.”

Clearly we don’t expect doctors to behave this way. Even though the patient is the boss, the patient expects the doctor to have the answers and help set the direction.

Here’s another version of the conversation:

Patient: “I want you to cut my arm off.” Doctor: “What’s wrong with your arm?” Patient: “It hurts. I’m tired of it. Just cut it off.” Doctor: “Let me see your arm. Hmmm. Looks like you’ve got a sprain or perhaps a hairline fracture. We should take some X-Rays.” Patient: “No, just cut it off.” Doctor: “Sir, I do not cut off healthy arms.” Patient: “But I’m paying you. You have to do what I say!” Doctor: “No, sir, I don’t. Cutting off your arm would violate my oath.”

Which of these two doctors would you rather be? Now project these two doctors into your own profession, and which would you rather be?

Programmers are professionals. They know more about designing and implementing software systems than their bosses do. Indeed, they are hired for this knowledge and expertise. And they have a solemn duty to prevent their managers from doing things that would be harmful.

All this boils down to one simple thing. Professionals are willing to say “No”. When their managers come to them with direction that makes no sense, a professional programmer will refuse the direction.

Is this risky? Sure. But part of being a professional is the willingness to stand on principle. There are lines that a professional will not cross.

Of course saying “No.” is only one side of the coin. Professionals are also expected to explain their positions, and come up with viable alternatives. Professionals negotiate with their superiors until both parties are satisfied with the chosen direction.

The poor web-designer schmuck in that cartoon was not behaving as a professional. He was behaving as a laborer. The fiasco at the end was his fault. He should have said “No.” and started a negotiation with his customer instead of just doing everything the customer said.

The cartoonist painted the web-designer as a wise but impotent victim, and the boss as the overbearing dufus. The reality is that the web-designer took the role of the victim voluntarily and shirked his responsibility to refuse direction that he considered harmful.

If you are a professional, you never allow yourself to be put in the role of the victim.

What's all this nonsense about Katas?

Uncle Bob — Sat, 21 Nov 2009 04:32:00 -0600

There has been an increase in interest in Software Katas of late. What is all this buzz about, and why might it be important?

Several years ago (Pragmatic) Dave Thomas started a special blog about coding katas. The idea was simple: Professionals practice.

This thought might not have occurred to you before; but it’s self-evidently true. Professional musicians practice. Professional dancers practice. Doctors practice. Soldiers practice. Indeed, any kind of professional craftsman or artisan must practice their adopted trade in order to execute it well when it counts.

Dave’s point was that professional programmers need to practice like any other professional. He set forth a number of simple exercises and problems that programmers could solve during their practice time. He also suggested that practice time needs to be scheduled and protected as part of the normal routine of a software developer.

At XP2005 in Shefield, UK, I attended Laurent Bossavit’s and Emmanuel Gaillot’s Coding Dojo session. I blogged about it at the time. I left that session with a different view of what a Kata might be. In Dave Thomas’ view a kata is an exercise to solve. Each time you solve it you try a different approach. Dave’s notion is that a kata is a way to explore the solution space of a problem. But in Martial arts a kata is something different; it is a set of motions that one memorizes in minute detail. A martial arts kata is a precisely executed dance. I left Sheffield with the idea that a coding kata could be more like a martial arts kata.

About a year ago, Corey Haines (our wandering minstrel of code) stopped by and told me about some mini-conferences that had adopted the notion of coding kata. What he described sounded more like a group dance than a bunch of people independently solving a given problem. This got me thinking. So at the next Software Craftsmanship meeting at 8th Light (Which, not coincidentally, takes it’s name from a martial arts philosophy) I conducted a simple kata session in dojo style.

I, acting as the Sen Sei, had everyone file in to the room as they would to a dojo. I had them put their green bands on their wrists (mine was black!). I bade them to sit, and then gestured for silence. For the next 15 minutes I led them through the Prime Factors Kata in Java without uttering a word (well, almost). I would write a simple test case, then gesture for them to do the same. Then I would make the test pass, and again gesture for them to repeat my moves.

Then, this Fall, I attended Programming with the Stars at Agile 2009. As I watched the contestants perform their programming sessions, I realized that they were not simply demonstrating programming prowess; indeed they were competing in how well they performed their session. That’s when it struck me. This was a performance art!

It seems absurd to think of programming as a performance art. And yet watching those people up on stage working through routines that they had clearly practiced over and over again, was delightfully entertaining to me, and to the crowd. There were cheers and applause when a performing pair did something especially well. There was laughter when a pair executed a clever joke in code, or in behavior. For programmers, watching other programmers perform well practiced routines was fun!

Now I had seen martial arts katas executed as performance art during competitions. Indeed, it is a beautiful thing to watch a skilled and well practiced martial artist peform a kata. And that started me thinking. Why would a martial artist practice these forms so intensely that they could be performed as an art?

The goal of martial arts is not to perform on stage. The goal of martial arts is self defense! And yet, when a martial artist is at the peak of his skill, his practice takes on the quality of a performance art. When you watch a skilled martial artist perform, you know you don’t want to fight him. The performance is a demonstration, and a realization, of mastery.

And yet the performance is not the goal. No martial artist practices his art so that they can perform on stage. A martial artist practices to achieve personal perfection in the art of self defense. The fact that the practice can be performed is a (pleasant) side effect.

Could it be that these coding kata that we’ve been dabbling with for the last few years could be performed? Is there a benefit to practicing them so much that performance is an option? Is there really any reason for a programmer to create and then memorize a routine down to the level of individual keystrokes? Is there profit to be gained from practicing that routine so thoroughly that you can do it at high speed with virtually no errors? This August I decided to find out.

I started with the Prime Factors Kata in Java, and set it to music inspired by this video of dancing kites. The piece is about 4 minutes long, and I figured I’d need about twice that, so I found two different version of the music and played them back to back. But working in Java is ponderous, and I could not complete the kata before the two pieces had finished. So I switched to Ruby. Using Ruby and Rspec I was able to finish the Kata with plenty of time to spare.

Thereupon began an evolution. Since I had time, I added more features to the kata. As I improved in my practice I found I had even more time, so I added even more features. I refined and polished. I refactored my refactorings. I trimmed keystrokes, and learned many new keyboard shortcuts in the process. I completely abandoned the mouse; and then later reacquired it for flourishes and highlights (but never for actual coding).

Week after week, I set aside 30 minutes or so every day to practice, and practice, and practice—just the way I used to practice my Jiu Jitsu Katas. I would practice on airplanes while flying to clients. I would practice in bed with my laptop in my lap. I would practice late at night in hotel rooms after a long day consulting for clients. I would practice early in the morning before shower and breakfast. And the practice started to pay off.

Ten weeks later I finally decided I was ready to create a screencast. I reduced my screen resolution to 1024X768. I configured SnapZ ProX to record the whole screen at 80% size. And I began to record.

Performing a coding kata in time to music is a very difficult thing to do. Maintaining the timeline is critical. There are very few errors that you can effectively recover from. If you miss a stroke, the whole sequence melts down. And melt down it did. Over and over and over again. I spent weeks trying to record a reasonably good session. I must have done hundreds of takes. It was very frustrating.

But day by day I got better and better at it. Finally, after weeks of trying, I got what I consider to be a recording that, though not perfect, is good enough to present. You can watch that recording here.

Has this paid off in the regular programming I do? I think it has. My mouse usage is much less than it used to be. I know a much larger number of keyboard shortcuts than I used to, and I can use them almost instinctively. My typing and accuracy have improved quite a bit, and I now type more punctuation and number keys without looking. So, yes, I think the effort has paid off; though I’m not at all sure the payoff compensates for the effort.

But something else compensates for the effort. Making this recording was fun—it was a lot of fun. And that’s probably the real answer to the question posed in the title of this blog.

Archeological Dig

Uncle Bob — Wed, 11 Nov 2009 10:39:00 -0600

I was going through some old files today, and I stumbled upon some acetate slides from 1995. They were entitled: “Managing OO Projects”. Wow! What a difference fifteen years makes! (Or does it?) ...

In 1995-99 I was frequently asked to speak to managers about what a transition to OO (usually from C to C++) would do for (or to) them. I would spend a half day to a day going over the issues, costs, and benefits.

One part of that talk (usually about 90 min) was a discussion about software process. It was the process part of the talk that those acetate slides that I found described.

1995 was during the ascendency of Waterfall. Waterfall thinking was king. RUP had not yet been conceived as an acronym. And though Booch was beating the drum for incrementalism, most people (even many within Rational) were thinking in terms of six to eighteen month waterfalls.

So, here are the slides that I uncovered deep within an old filing cabinet. I scanned them in. They were produced on a Macintosh using the old “More” program. (Where is that program now? It was so good.)

Go ahead and read them now. Then come back here and continue…

What struck me about those slides was the consistency of the message with today. It was all about iterative development. Small iterations (though I never deigned to define the length in the slides, I frequently told people 2 weeks), measured results, etc. etc. Any Agile evangelist could use those slides today. He or she would have to dance quickly around a few statements, but overall the message has not shifted very much.

What’s even more interesting is the coupling between the process, and OO. The slides talk a lot about dependency management and dependency structure. There are hints of the SOLID principles contained in those slides. (Indeed several of the principles had already been identified by that time.) This coupling between process and software structure was a harbinger of the current craftsmanship/clean-code movement.

Of course the one glaring omission from these slides is TDD. That makes me think that TDD was the true catalyst of change, and the bridge that conveyed our industry from then to now.

Anyway, I guess the more things change, the more they stay the same.

Comments please!

We must ship now and deal with consequences

Uncle Bob — Thu, 15 Oct 2009 06:17:00 -0500

Martin Fowler has written a good blog about technical debt. He suggests that there are two axes of debt: deliberate and prudent. This creates four quadrants: deliberate-prudent, deliberate-imprudent, inadvertent-prudent, and inadvertent-imprudent. I agree with just about everything in his blog except for one particular caption…

Inadvertent-Imprudent Debt.

There is more of this debt than any other kind. It is all too common that software developers create a mess and don’t know they are doing it. They have not developed a nose that identifies code smells. They don’t know design principles, or design patterns. They think that the reek of rotten code is normal, and don’t even identify it as smelling bad. They think that their slow pace through the thick morass of tangled code is the norm, and have no idea they could move faster. These people destroy projects and bring whole companies to their knees. Their name is Doom.

Deliberate-Imprudent Debt.

There is a meme in our industry (call it the DI meme) that tells young software developers that rushing to the finish line at all costs is the right thing to do. This is far worse than the ignorance of the first group because these folks willfully create debt without counting the cost. Worse, this meme is contagious. People who are infected with it tend to infect others, causing an epidemic of deliberately imprudent debtors (sound familiar?) The end result, as we are now all know, is economic catastrophe, inflation (of estimates) and crushing interest (maintenance) payments. They have become death, the destroyer of worlds.

Inadvertent-Prudent Debt.

This is something of an oxymoron. Ironically, it is also the best of all possible states. The fact is that no matter how careful we are, there is always a better solution that we will stumble upon later. How many times have you finished a system only to realize that if you wrote it again, you’d do it very differently, and much better?

The result is that we are always creating a debt, because our hindsight will always show us a better option after it is too late. So even the best outcome still leaves us owing. (Mother Earth will eventually collect that debt!)

Deliberate-Prudent Debt.

This is the quadrant that I have the biggest problem with. And it is this quadrant in which Martin uses the caption I don’t like. The Caption is: “We must ship now and deal with consequences.”

Does this happen? Yes. Should it happen? Rarely, yes. But it damned well better not happen very often, and it damned well better not happen out of some misplaced urge to get done without counting the cost.

The problem I have with this quadrant (DP) is that people who are really in quadrant DI think they are in DP, and use words such as those that appear in the caption as an excuse to rack up a huge imprudent debt.

The real issue is the definition of the word: Imprudent.

So let me ask you a question. How prudent is debt? There is a very simple formula for determining whether debt is prudent or imprudent. You can use this formula in real life, in business, and in programming. The formula is: Does the debt increase your net worth, and can you make the payments?

People often focus on the first criterion, without properly considering the second. Buying a house is almost certain to increase your net worth despite the debt (though lately…). On the other hand, if you cannot make the payments, you won’t keep that house for long. The reason for our current economic woes has a lot to do with people trying to increase their net worth despite the fact that they couldn’t afford the payments. (indeed, they were encouraged by a meme very similar to the DI meme!)

Bad code is always imprudent.

Writing bad code never increases your net worth; and the interest rate is really high. People who write bad code are like those twenty-somethings who max out all their credit cards. Every transaction decreases net worth, and has horrendous consequences for cash flow. In the end, the vast bulk of your effort goes to paying the interest (the inevitable slow down of the team as they push the messes around). Paying down the principle becomes infeasible. (Just the way credit card companies like it.)

Some Suboptimal Design Decision are Prudent Debt.

But most are not. Every once in awhile there is a suboptimal design decision that will increase the net worth of the project by getting that project into customer’s hand’s early.

This is not the same as delivering software that is under-featured. It is often prudent to increase the net worth of a project by giving customers early access to a system without a full and rich feature set. This is not debt. This is more like a savings account that earns interest.

Indeed, this is one reason that most technical debt is imprudent. If you are truly concerned about getting to market early, it is almost always better to do it with fewer features, than with suboptimal design. Missing features are a promise that can be kept. Paying back suboptimal designs creates interest payments that often submerge any attempts at payback and can slow the team to the breaking point.

But there are some cases where a sub-optimal design can increase your net worth by allowing you to deliver early. However, the interest rate needs to be very low, and the principle payments need to be affordable, and big enough to pay back the debt in short order.

What does a low interest rate mean? It means that the sub-optimal design does not infiltrate every part of your system. It means that you can put the sub-optimal design off in a corner where it doesn’t impact your daily development life.

For example, I recently implemented a feature in FitNesse using HTML Frames. This is sub-optimal. On the other hand, the feature is constrained to one small part of the system, and it simply doesn’t impact any other part of the system. It does not impede my progress. There is no mess for me to move around. The interest rate is almost zero! (nice deal if you can get it!)

Implementing that feature with ajax is a much larger project. I would have had to invest a great deal of time and effort, and would have had to restructure massive amounts of the internal code. So the choice was a good one.

Better yet, the customer experience has pretty much been a big yawn. I thought people would really like the feature and would drive me to expand upon it. Instead, the customer base has virtually ignored it.

So my solution will be to pay back this debt by eliminating the feature. It was a cheap experiment, that resulted in my not having to spend a lot of time and effort on a new architecture! Net worth indeed!

But it might have gone the other way. My customers may have said: “Wow, Great! We want more!” At that point it would have been terrible to expand on the HTML Frames! That decision would have been in the DI quadrant. Deliberate imprudence! Rather, my strategy would have been to replace the suboptimal Frames design of the feature with an isolated ajax implementation, and then to gradually migrate the ajax solution throughout the project. That would have been annoying, but loan payments always are.

Summary

So, don’t let the caption in the DP quadrant be an excuse. Don’t fall for the DI meme that says “We just gotta bite the bullet”. Tread very carefully when you enter the DP quadrant. Look around at all your options, because it’s easy to think you are in the DP quadrant when you are really in the DI quadrant.

Remember: Murphy shall send you strong delusion, that you should believe you are in DP; so that you will be damned in DI.

TDD Derangement Syndrome

Uncle Bob — Wed, 07 Oct 2009 08:32:00 -0500

My recent blog about TDD, Design Patterns, Concurrency, and Sudoku seemed to draw the ire of a few vocal TDD detractors. Some of these people were rude, insulting, derisive, dismissive, and immature. Well, Halloween is not too far away.

In spite of their self-righteous snickering they did ask a few reasonable questions. To be fair I thought it would be appropriate for me to answer them.

Is there any research on TDD?

It turns out that there is a fair bit.

One simple google search led me to this blog by Phil Haack in which he reviewed a TDD research paper. Quoting from the paper:

We found that test-first students on average wrote more tests and, in turn, students who wrote more tests tended to be more productive. We also observed that the minimum quality increased linearly with the number of programmer tests, independent of the development strategy employed.

The same google search led me to this blog by Matt Hawley, in which he reviewed several other research papers. Part of his summary:

* 87.5% of developers reported better requirements understanding. * 95.8% of developers reported reduced debugging efforts. * 78% of developers reported TDD improved overall productivity. * 50% of developers found that it decreased overall development time. * 92% of developers felt that TDD yielded high-quality code. * 79% of developers believed TDD promoted simpler design.

Actually, I recognize some of Matt’s results as coming from a rather famous 2003 study (also in the list of google results) by Laurie Wiliams and Boby George. This study describes a controlled experiment that they conducted in three different companies. Though Matt’s summary above is based (in part) on that study, there is more to say.

In the George-William study teams that practiced TDD took 16% longer to claim that they were done than the teams that did not practice TDD. Apparently tests are more accurate than claims since the non-TDD teams failed to pass one third of the researcher’s hidden acceptance tests, whereas the TDD teams passed about 6 out of 7. To paraphrase Kent Beck: “If it doesn’t have to work, I can get it done a lot faster!”

Another point of interest in this study is that the TDD teams produced a suite of automated tests with very high test coverage (close to 100% in most cases) whereas most of the non-TDD teams did not produce such a suite; even though they had been instructed to.

Jim Shore wrote a review of yet another research summary which I found in the same google search. This one combines 7 different studies (including George-Williams). Here the results range from dramatically improved quality and productivity to no observed effect.

Finally, there is this 2008 case Study of TDD at IBM and Microsoft which shows that TDDers enjoy a defect density reduction ranging from 30% to 90% (as measured by defect tracking tools) and a productivity cost of between 15% and 35% (the subjective opinion of the managers). I refer you back to Kent Beck’s comment above.

I’m sure there is more research out there. After all this was just one google search. I think it’s odd that the TDD detractors didn’t find anything when they did their google searches.

Oh yeah, and then there was that whole issue of IEEE Software that was dedicated to papers and research on TDD.

What projects have been written with TDD, hmmm?

Quite a few, actually. The following is a list of projects that have an automated suite of unit tests with very high coverage. Those that I know for a fact use TDD, I have noted as such. The others, I can only surmise. If you know of any others, please post a comment here.

JUnit. This one is kind of obvious. JUnit was written by Kent Beck and Erich Gamma using TDD throughout. If you measure software success by sheer distribution, this particular program is wildly successful.

Fit. Written by Ward Cunningham. The progenitor of most current acceptance testing frameworks.

FitNesse. This testing framework has tens of thousands of users. It is 70,000 lines of java code, with 90%+ code coverage. TDD throughout. Very small bug-list. Again, if you measure by distribution, another raving success.

Cucumber,
Rspec. These two are Testing frameworks in Ruby. Of course you’d expect a testing framework to be written with TDD, wouldn’t you? I know these were. TDD throughout.

Limelight. A gui framework in JRUby. TDD throughout.
jfreechart.
Spring
JRuby
Smallsql
Ant
MarsProject
Log4J
Jmock

Are there others? I’m sure there are. This was just a quick web search. Again, if you know of more, please add a comment.

Echoes from the Stone Age

Uncle Bob — Tue, 06 Oct 2009 11:07:29 -0500

The echoes from Joel Spolsky’s Duct Tape blog continue to bounce off the blogosphere and twitterverse. Tim Bray and Peter Seibel have both written responses to Joel, me, and each other.

Here are some stray thoughts…

TDD

Anyone who continues to think that TDD slows you down is living in the stone age. Sorry, that’s just the truth. TDD does not slow you down, it speeds you up.

Look, TDD is not my religion, it is one of my disciplines. It’s like dual entry bookkeeping for accountants, or sterile procedure for surgeons. Professionals adopt such disciplines because they understand the theory behind them, and have directly experienced the benefits of using them.

I have experienced the tremendous benefit that TDD has had in my work, and I have observed it in others. I have seen and experienced the way that TDD helps programmers conceive their designs. I have seen and experienced the way it documents their decisions. I have seen and experienced the decouplings imposed by the tests, and I have seen and experienced the fearlessness with which TDDers can change and clean their code.

To be fair, I don’t think TDD is always appropriate. There are situations when I break the discipline and write code before tests. I’ll write about these situations in another blog. However, these situations are few and far between. In general, for me and many others, TDD is a way to go fast, well, and sure.

The upshot of all this is simple. TDD is a professional discipline. TDD works. TDD makes you faster. TDD is not going away. And anyone who has not really tried it, and yet claims that it would slow them down, is simply being willfully ignorant. I don’t care if your name is Don Knuth, Jamie Zawinski, Peter Seibel, or Peter Pan. Give it a real try, and then you have the right to comment.

Let me put this another way. And now I’m talking directly to those who make the claim that TDD would slow them down. Are you really such a good programmer that you don’t need to thoroughly check your work? Can you conceive of a better way to check your work than to express your intent in terms of an executable test? And can you think of a better way to ensure that you can write that test other than to write it first?

If you can, then I want to hear all about it. but I don’t want to hear that you write a few unit tests after the fact. I don’t want to hear that you manually check your code. I don’t want to hear that you do design and therefore don’t need to write tests. Those are all stone-age concepts. I know. I’ve been there.

So there. <grin>

The Design Pattern Religion

Tim Bray said:

My experience suggests that there are few surer ways to doom a big software project than via the Design Patterns religion.

He’s right of course. The Design Patterns religion is a foul bird that ravages teams and cuts down young projects in their prime. But let’s be clear about what that religion is. The Design Patterns religion is the ardent belief that the use of design patterns is good.

Here’s a clue. Design Patterns aren’t good. They also aren’t bad. They just are. Given a particular software design situation, there may be a pattern that fits and is beneficial. There may also be patterns that would be detrimental. It’s quite possible that none of the currently documented patterns are appropriate and that you should close the book and just solve the problem.

Here’s another clue. You don’t use patterns. You don’t apply patterns. Patterns just are. If a particular pattern is appropriate to solve a given problem, then it will be obvious. Indeed it is often so obvious that you don’t realize that the pattern is in place until you are done. You look back at your code and realize: “Oh, that’s a Decorator!”.

So am I saying that Design Patterns are useless?

NO! I want you to read the patterns books. I want you to know those patterns inside and out. If I point at you and say “Visitor” I want you at the board drawing all the different variants of the pattern without hesitation. I want you to get all the names and roles right. I want you to know patterns.

But I don’t want you to use patterns. I don’t want you to believe in patterns. I don’t want you to make patterns into a religion. Rather I want you to be able to recognize them when they appear, and to regularize them in your code so that others can recognize them too.

Design Patterns have a huge benefit. They have names. If you are reading code, and you see the word “Composite”, and if the author took care to regularize the code to the accepted names and roles of the “Composite” pattern, then you will know what that part of the code is doing instantly. And that is powerful!

Minimizing Concurrency.

In my first Duct Tape blog I made the statement:

I found myself annoyed at Joel’s notion that most programmers aren’t smart enough to use templates, design patterns, multi-threading, COM, etc. I don’t think that’s the case. I think that any programmer that’s not smart enough to use tools like that is probably not smart enough to be a programmer period.

Tim responds with:

...multi-threading is part of the problem, not part of the solution; that essentially no application programmer understands threads well enough to avoid deadlocks and races and horrible non-repeatable bugs. And that COM was one of the most colossal piles of crap my profession ever foisted on itself.

Is concurrency really part of the problem? Yes! Concurrency is a really big part of the problem. Indeed, the first rule of concurrency is: DON’T. The second rule is: REALLY, DON’T.

The problem is that some times you have no choice. And in those situations, where you absolutely must use concurrency, you should know it inside and out!

I completely and utterly reject the notion that ignorance is the best defense. I reject that lack of skill can ever be an advantage. So I want you to know concurrency. I want to shout “Dining Philosophers” and have you run to the board without hesitation and show me all the different solutions. If I holler “Deadlock”, I want you to quickly identify the causes and solutions.

Here’s a clue. If you want to avoid using something, know that something cold.

Sudoku

At the end of his blog, Peter jumps on the pile of bodies already crushing Ron Jeffries regarding the Sudoku problem from July of 2006.

I find the pile-up disturbing. Ron had the courage to fail in public. Indeed he announced up front that he might “crash and burn”. And yet he got lambasted for it by people who hid behind someone else’s work. The responses to Ron’s tutorial blogs were completely unfair because the authors of those blogs had everything worked out for them by Dr. Peter Norvig before they published their screeds. They were comparing apples to oranges because their responses were about the solution whereas Ron’s blogs were about the process.

Which one of us has not gone down a rat-hole when hunting for a solution to a complex problem? Let that person write the first blog. Everyone else ought to be a bit more humble.

Do the people on the pile think that Ron is unable to solve the Sudoku problem? (Some have said as much.) Then they don’t know Ron very well. Ron could code them all under the table with one hand tied behind his back.

Personal issues aside, I find the discussion fascinating in it’s own right. Ron had attempted to solve the Sudoku problem by gaining insight into that problem through the process of coding intermediate solutions. This is a common enough TDD approach. Indeed, the Bowling Game and the Prime Factors Kata are both examples where this approach can work reasonably well.

This approach follows the advice of no less than Grady Booch who (quoting Heinlein) said: “when faced with a problem you do not understand, do any part of it you do understand, then look at it again.“

Ron was attempting to use TDD to probe into the problem to see if he could gain any insight. This technique often bears fruit. Sometimes it does not.

Here is a classic example. Imagine you were going to write a sort algorithm test first:

Test 1: Sort an empty array. Solution: Return the input array.

Test 2: Sort an array with one element. Solution: Return the input array.

Test 3: Sort an array with two elements. Solution: Compare the two elements and swap if out of order. Return the result.

Test 4: Sort an array with three elements. Solution: Compare the first two and swap if out of order. Compare the second two and swap if out of order. Compare the first two again and swap if out of order. Return the result.

Test 5: Sort an array with four elements. Solution: Put the compare and swap operations into a nested loop. Return the result.

The end result is a bubble sort. The algorithm virtually self assembles. If you had never heard of a bubble sort before, this simple set of tests would have driven you to implement it naturally.

Problems like Bowling, Prime Factors, and Bubble Sort hold out the interesting promise that TDD may be a way to derive algorithmms from first principles!

On the other hand, what set of tests would drive you to implement a QuickSort? There are none that I know of. QuickSort and Sudoku may require a serious amount of introspection and concentrated thought before the solution is apparent. They may belong to a class of algorithms that do not self-assemble like Bowling, Prime Factors, and Bubble Sort.

This blog by Kurt Christensen provides all the links to the various Sudoku articles, and sums it up this way.

TDD may not be the best tool for inventing new algorithms, it may very well be the best tool for applying those algorithms to the problem at hand.

Actually I think TDD is a good way to find out if an algorithm will self-assemble or not. It usually doesn’t take a lot of time to figure out which it’s going to be.

One Thing: Extract till you Drop.

Uncle Bob — Fri, 11 Sep 2009 13:33:00 -0500

For years authors and consultants (like me) have been telling us that functions should do one thing. They should do it well. They should do it only.

The question is: What the hell does “one thing” mean?

After all, one man’s “one thing” might be someone else’s “two things”.

Consider this class:

  class SymbolReplacer {
    protected String stringToReplace;
    protected List<String> alreadyReplaced = new ArrayList<String>();

    SymbolReplacer(String s) {
      this.stringToReplace = s;
    }

    String replace() {
      Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");
      Matcher symbolMatcher = symbolPattern.matcher(stringToReplace);
      while (symbolMatcher.find()) {
        String symbolName = symbolMatcher.group(1);
        if (getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName)) {
          alreadyReplaced.add(symbolName);
          stringToReplace = stringToReplace.replace("$" + symbolName, translate(symbolName));
        }
      }
      return stringToReplace;
    }

    protected String translate(String symbolName) {
      return getSymbol(symbolName);
    }
  }

It’s not too hard to understand. The replace() function searches through a string looking for $NAME and replaces each instance with the appropriate translation of NAME. It also makes sure that it doesn’t replace a name more than once. Simple.

Of course the words “It also…” pretty much proves that this function does more than one thing. So we can probably split the function up into two functions as follows:

    String replace() {
      Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");
      Matcher symbolMatcher = symbolPattern.matcher(stringToReplace);
      while (symbolMatcher.find()) {
        String symbolName = symbolMatcher.group(1);
        replaceAllInstances(symbolName);
      }
      return stringToReplace;
    }

    private void replaceAllInstances(String symbolName) {
      if (getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName)) {
        alreadyReplaced.add(symbolName);
        stringToReplace = stringToReplace.replace("$" + symbolName, translate(symbolName));
      }
    }

OK, so now the replace() function simply finds all the symbols that need replacing, and the replaceAllInstances() function replaces them if they haven’t already been replaced. So do these function do one thing each?

Well, the replace() compiles the pattern and build the Matcher() Maybe those actions should be moved into the constructor?

  class SymbolReplacer {
    protected String stringToReplace;
    protected List<String> alreadyReplaced = new ArrayList<String>();
    private Matcher symbolMatcher;
    private final Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");

    SymbolReplacer(String s) {
      this.stringToReplace = s;
      symbolMatcher = symbolPattern.matcher(s);
    }

    String replace() {
      while (symbolMatcher.find()) {
        String symbolName = symbolMatcher.group(1);
        replaceAllInstances(symbolName);
      }
      return stringToReplace;
    }

    private void replaceAllInstances(String symbolName) {
      if (getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName)) {
        alreadyReplaced.add(symbolName);
        stringToReplace = stringToReplace.replace("$" + symbolName, translate(symbolName));
      }
    }

    protected String translate(String symbolName) {
      return getSymbol(symbolName);
    }
  }

OK, so now certainly the replace() function is doing one thing? Ah, but I see at least two. It loops, extracts the symbolName and then does the replace. OK, so how about this?

    String replace() {
      for (String symbolName = nextSymbol(); symbolName != null; symbolName = nextSymbol())
        replaceAllInstances(symbolName);

      return stringToReplace;
    }

    private String nextSymbol() {
      return symbolMatcher.find() ? symbolMatcher.group(1) : null;
    }

I had to restructure things a little bit. The loop is a bit ugly. I wish I could have said for (String symbolName : symbolMatcher) but I guess Matchers don’t work that way.

I kind of like the nextSymbol() function. It gets the Matcher nicely out of the way.

So now the replace() and nextSymbol() functions are certainly doing one thing. Aren’t they?

Well, I suppose I could separate the loop from the return in replace().

    String replace() {
      replaceAllSymbols();
      return stringToReplace;
    }

    private void replaceAllSymbols() {
      for (String symbolName = nextSymbol(); symbolName != null; symbolName = nextSymbol())
        replaceAllInstances(symbolName);
    }

I don’t see how I could make these functions smaller. They must be doing one thing. There’s no way to extract any other functions from them!

Uh… Wait. Is that the definition of one thing? Is a function doing one thing if, and only if, you simply cannot extract any other functions from it? What else could “one thing” mean? After all, If I can extract one function out of another, the original function must have been doing more than one thing.

So does that mean that for all these years the authors and consultants (like me) have been telling us to extract until you can’t extract anymore?

Let’s try that with the rest of this class and see what it looks like…

  class SymbolReplacer {
    protected String stringToReplace;
    protected List<String> alreadyReplaced = new ArrayList<String>();
    private Matcher symbolMatcher;
    private final Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");

    SymbolReplacer(String s) {
      this.stringToReplace = s;
      symbolMatcher = symbolPattern.matcher(s);
    }

    String replace() {
      replaceAllSymbols();
      return stringToReplace;
    }

    private void replaceAllSymbols() {
      for (String symbolName = nextSymbol(); symbolName != null; symbolName = nextSymbol())
        replaceAllInstances(symbolName);
    }

    private String nextSymbol() {
      return symbolMatcher.find() ? symbolMatcher.group(1) : null;
    }

    private void replaceAllInstances(String symbolName) {
      if (shouldReplaceSymbol(symbolName))
        replaceSymbol(symbolName);
    }

    private boolean shouldReplaceSymbol(String symbolName) {
      return getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName);
    }

    private void replaceSymbol(String symbolName) {
      alreadyReplaced.add(symbolName);
      stringToReplace = stringToReplace.replace(
        symbolExpression(symbolName), 
        translate(symbolName));
    }

    private String symbolExpression(String symbolName) {
      return "$" + symbolName;
    }

    protected String translate(String symbolName) {
      return getSymbol(symbolName);
    }
  }

Well, I think it’s pretty clear that each of these functions is doing one thing. I’m not sure how I’d extract anything further from any of them.

Perhaps you think this is taking things too far. I used to think so too. But after programming for over 40+ years, I’m beginning to come to the conclusion that this level of extraction is not taking things too far at all. In fact, to me, it looks just about right.

So, my advice: Extract till you just can’t extract any more. Extract till you drop.

After all, with modern tools it takes very little time. It makes each function almost trivial. The code reads very nicely. It forces you to put little snippets of code into nicely named functions. And, well gosh, extracting till you drop is kind of fun!

The Rush

Uncle Bob — Fri, 26 Jun 2009 08:40:13 -0500

There’s nothing like the feeling of achievement when you get a complex software system working. It’s the feeling of the hunter making a hard fought kill. By the sheer power of your intellect you have imposed your will upon inanimate nature, and forced it to do your bidding. You feel the rush of power, the satisfaction of victory. You are the master!

That’s the good news.

The bad news is that you’ve made a mess of things along the way.

This is inevitable. It takes a great deal of focus and endurance to get a system working just right. While you are consumed by that focus, you have little room for niceties like small functions, nice names, decoupling, and cleanliness.

My apprentice and I just finished achieving a major milestone in FitNesse. It used to be that each new release of FitNesse was contained in a zip file that had all the files and directories laid out just perfectly. This is great for a brand new installation, but doesn’t work so well when you are upgrading. What we managed to get working last night was the ability for FitNesse to self-install from it’s jar file. Now, whether you are installing for the first time, or just upgrading, FitNesse will install itself into your environment appropriately.

If you’ve ever written a self-install like this, you know there are lots of niggling little details to attend to. While the problem is conceptually simple, the realization is a maze of complexity.

Last night, with dinner waiting on the table and getting cold, we got the last little problem licked, and saw the whole self-installation work as we wanted. We congratulated each other with a high-five, and then went upstairs to eat a well-deserved meal. Later that evening I went down to the office and checked in the code.

This morning I woke up, and while in the shower, I realized that we had a lot of work left to do. We need to go back over all that code and clean it up. I’m sure we left a swath of detritus while on the hunt.

The rush of victory should never be confused with the cold and crystalline concept of done. Once you get your systems to work, you still have to go back and clean up the wreckage left behind by the victorious battle. You are not done until the victorious code has been cleaned, polished, and oiled.

Rich Hickey on Testing

Dean Wampler — Fri, 05 Jun 2009 12:40:00 -0500

It was an interesting week at JavaOne, with lots of talks and hallway discussions about new languages on the JVM. One of those languages is Clojure.

Rich Hickey, the creator of Clojure, gave a talk at the Bay Area Clojure User Group Wednesday evening. During the Q&A part, he said that he’s not big on writing tests, although he always runs the tests that other people have written before he commits changes.

Of course, there are many people, including us Object Mentors, who consider TDD to be an essential part of professional software development. Obviously not everyone agrees. James Coplien has been another critic of this view.

We should never accept any dogma. Why is TDD considered important? What does it purport to do? TDD provides two important benefits.

Driving the design.
Building a suite of automated regression tests.

So, if you can satisfy both requirements without TDD, then technically you don’t need it. In Rich’s case, he said he spends a lot of time thinking about what he’s going to do before he does it. In this way, he satisfies the first requirement, driving the design. I had a spirited discussion with some Thoughtworkers afterwards and Ola Bini said what a lot of us think, “I do that thinking by writing tests.” I’ll freely admit that when I am really experimenting with ideas, I might just write code, but once I know how to proceed, I return to the beginning and test drive the “production” code.

Rich also made an off-hand comment that if he screws something up, he’s got thousands of users who will let him know! That ex post facto testing, along with the Rich’s own devotion to doing high-quality work, does a good job of handling regressions.

But Rich mentioned something else that is also very important. In a functional language, where values are immutability and mutable state is handled in specific, principled ways, regressions don’t happen nearly as often. Clojure has one of the most deeply thought out approaches for handling state, which is the genius of Clojure.

I asked Rich how long he worked on Clojure before releasing it to the world. He spent about 2 1/2 years, much of that time working exclusively on Clojure (and eating through his savings). When he finally “announced” it, his “marketing” consisted of one email to some friends in the Common Lisp community. The rest was viral, a testament to the justified excitement Clojure has generated.

For me, I’ll probably always do my design thinking through tests, especially when I’m writing code in imperative languages, like Java and Ruby. I’ll continue to encourage my clients to use TDD, because I find that TDD is the most productive way to achieve high quality. I want the safety net of a good test suite. I’m also writing more and more of my code in a functional style, with minimal side effects and mutable data. You should, too.

Why the sea is boiling hot

Uncle Bob — Mon, 11 May 2009 11:09:57 -0500

In my keynote at the Rails Conference last Wednesday, I spoke about What Killed Smalltalk and could it Kill Ruby Too?. In this keynote, as is my wont, I spoke about the need for our industry to define itself as a profession, and for us to define ourselves as professionals. I closed the keynote with this statement: “Professionalism does not mean rigid formalism. Professionalism does not mean adhering to beaurocrasy. Professionalism is honor. Professionalism is being honest with yourself and disciplined in the way you work. Professionalism is not letting fear take over.”

David Heinemeier Hansson, inventor of Rails, and affectionately referred to as @dhh has posted a blog in response in which he asserts that we need artists as well as professionals, and draws a dichotomy between the two. The dichotomy is a false one…

We are a community of artisans. We make things with our hands! We all strive, like @dhh, to make things of great beauty and utility. In no way, and by no means, do I wish to assail that artistry. Indeed, my hope is to free it. I want to convince all programmers that the desire for, and the pursuit of beauty is the way you satisfy your customers. The only way to go fast, is to go well.

I want developers to take pride in their work. I also want them to take pride in the way that they work. I want them to be able to look back on the last few days, weeks, and months, and be able to say to themselves, “I made something beautiful, and I made it well.”

We are a quirky lot. Some of us wear faded jeans with yesterday’s spaghetti on them. Others wear T-shirts that have PI wrapped around them. There are beards, tatoos, tongue-studs and hair in all shapes and colors. There are hawaiian shirts and sandals. There are jackets and ties and sometimes even suits. Some of us speak carefully. Some of us drop F-bombs at a whim. Some of us are liberal, and some are conservative. Some of us relish in being seen, and some of us are glad to be overlooked. In short, we are a group of diverse people who are drawn together by our common passion for code.

There is nothing wrong with this diversity. Indeed it’s healthy. The fact that we all think differently about styles, language, appearance, and values means that there are a zillion different ways that we can learn from each other. And in that learning grows the seed of our profession.

So @dhh is right, at least about the diversity. We should all relish the opportunity to share ideas with people who think differently than we do. But @dhh is wrong when he draws the dichotomy between artists and engineers. Every engineer is an artist, and every artist is an engineer. Every engineer strives for elegance and beauty. Every artist has the need to make their art actually work. The two are inextricably tied. You cannot be one without also being the other.

Now, certainly there are environments where the engineering side of things is emphasized over the artistry side. In extreme cases the artistry is repressed into near non-existence. Such places are soul-searing hell-holes that every programmer should strive to escape (or for the brave: change from within!) Indeed, @dhh implies that he worked in such places and found he was “faking [his] way along in this world.” I completely understand that.

But then @dhh makes his biggest mistake. He equates the professionalism I was talking about in my keynote with those repressive environments. He seems to think that professionalism and artistry are mutually exclusive. That wearing a green band means you give up on beauty. That discipline somehow saps programmer happiness.

But remember what I said when I closed my keynote: “Professionalism does not mean rigid formalism. Professionalism does not mean adhering to beaurocrasy. Professionalism is honor. Professionalism is being honest with yourself and disciplined in the way you work. Professionalism is not letting fear take over.”

This is not a complete definition; but it will serve for my current purposes. Because, you see, I made a big mistake during that keynote. And it is in how we deal with our own errors that the claim of professionalism is most frequently, and most thoroughly tested.

In my keynote I used a metaphor to link hormones and languages. I said that C++ was a testosterone language, but Java was an estrogen language. And then I used the word “insipid” to describe Java.

Now clearly C++ and testosterone have very little in common. My use of this metaphor was an oratory device – a joke. As far as the operation of that device is concerned, it was a success. The vast majority of the audience laughed, demonstrating to me that they were a) listening and b) understanding and c) open.

There is a kind of artistry in making oratory devices like this, and I take a certain amount pride in it. Such devices need to be timed appropriately, delivered skillfully, and used to gauge the audience. They can help to turn virtually any dry topic into a compelling speech.

On the other hand, the construction of this device had a significant flaw. I had equated women with weakness. This was not intentional. I do not think of women as weak. But there it was: Estrogen === Insipid. If you were a woman in that audience, how could you come to any conclusion other than “Uncle Bob thinks women are weak.”

How did I make this error? Lack of discipline. I did not test this keynote adequately. I should at least have run it past my wife! I mis-engineered my art! (Or perhaps my engineering was artless <grin>).

Within minutes of concluding my talk, the complaints appeared on twitter. Women who had been offended and hurt by the remark were tweeting their dissatisfaction. Some men were joining them.

There were two ways I could have responded to this. I could have asserted that these people were just being too sensitive; that they should have realized that this was just an oratory device and that I didn’t mean any harm; that they should just recognize me for who I am and not get so hung up in their own fears and values.

But I think that reaction would have been unprofessional. Why? Because it would have been dishonest. The truth was that this was just a stupid mistake. I built the device badly. I executed the device without testing it properly. I screwed up; and I needed to own up that. So I immediately tweeted apologies to those concerned and ate an appropriate amount of humble pie.

The reason I told you this story (at the risk of sounding somewhat self-aggrandizing) is so that I could use it to help define professionalism. The construction of my oratory device was unprofessional. I should have tested it better. I should have realized the danger of using gender comparisons and taken greater care with their use. I could have done better.

We professionals aren’t prefect. We all make mistakes. We recover from those mistakes by owning up to them as mistakes. We do not cover those mistakes by claiming that everyone else is wrong.

Confronting your mistakes and taking appropriate action is a discipline. It is a discipline of honor and self-honesty. It is a demonstration that we do not let the fear of our own errors take us over.

Crap Code Inevitable? Rumblings from ACCU.

Uncle Bob — Thu, 23 Apr 2009 04:56:22 -0500

I gave the opening Keynote at ACCU 2009 on Wednesday. It was entitled: The Birth of Craftsmanship. Nicolai Josuttis finshed the day with the closing keynote: Welcome Crappy Code – The Death of Code Quality. It was like a blow to the gut.

In my keynote I attempted to show the historical trajectory that has led to the emergence of the software craftsmanship movement. My argument was that since the business practices of SCRUM have been widely adopted, and since teams who follow those practices but do not follow the technical practices of XP experience a relentless decrease in velocity, and since that decrease in velocity is exposed by the transparency of scrum, then if follows that the eventual adoption of those technical XP practices is virtually assured. My conclusion was that Craftsmanship was the “next big thing” (tm) that would capture the attention of our industry for the next few years, driven by the business need to increase velocity. (See Martin Fowler’s blog on Flaccid Scrum) In short, we are on a trajectory towards a higher degree of professionalism and craftsmanship.

Nicolai’s thesis was the exact opposite of mine. His argument was that we are all ruled by marketing and that businesses will do whatever it takes to cut costs and increase revenue, and therefore businesses will drive software quality inexorably downward. He stipulated that this will necessarily create a crisis as the defect rates and deadline slips increased, but that all attempts to improve quality would be short lived and followed by a larger drive to decrease quality even further.

Josuttis’ talk was an hour of highly depressing rhetoric couched in articulate delivery and brilliant humor. One of the more memorable moments came when he playacted how a manger would respond to a developer’s plea to let them write clean code like Uncle Bob says. The manager replies: “I don’t care what Uncle Bob says, and if you don’t like it you can leave and take Uncle Bob with you.”

One of the funnier moments came when Josuttis came up with his rules for crap code, one of which was “Praise Copy and Paste”. Here he showed the evolution of a module from the viewpoint of clean code, and then from the viewpoint of copy-paste. His conclusion, delivered with a lovely irony, was the the copy-paste solution was more maintainable because it was clear which code belonged to which version.

It was at this point that I thought that this whole talk was a ribald joke, an elaborate spoof. I predicted that he was about to turn the tables on everyone and ringingly endorse the craftsmanship movement.

Alas, it was not so. In the end he said that he was serious about his claims, and that he was convinced that crap code would dominate our future. And then he gave his closing plea which went like this:

We finally accepted that requirements change, and so we invented Agile.

We must finally accept that code will be crap and so we must ???

He left the question marks on the screen and closed the talk.

This was like a blow to the gut. The mood of the conference changed, at least for me, from a high of enthralled geekery, to depths of hoplessness and feelings of futile striving against the inevitable. Our cause was lost. Defeat was imminent. There was no hope.

Bulls Bollocks!

To his credit, there are a few things that Josuttis got right. There is a lot of crap code out there. And there is a growing cohort of crappy coders writing that crap code.

But the solution to that is not to give up and become one of them. The solution to that is to design our systems so that they don’t require an army of maintainers slinging code. Instead we need to design our systems such that the vast majority of changes can be implemented in DSLs that are tuned to business needs, and do not require “programmers” to maintain.

The thing that Josuttis got completely wrong, in my mildly arrogant opinion, is the notion that low quality code is cheaper than high quality code. Low quality code is not cheaper; it is vastly more expensive, even in the short term. Bad code slows everyone down from the minute that it is written. It creates a continuous and copious drag on further progress. It requires armies of coders to overcome that drag; and those armies must grow exponentially to maintain constant velocity against that drag.

This strikes at the very heart of Josuttis’ argument. His claim that crappy code is inevitable is based on the notion that crappy code is cheaper than clean code, and that therefore businesses will demand the crap every time. But it has generally not been business that has demanded crappy code. Rather it has been developers who mistakenly thought that the business’ need for speed meant that they had to produce crappy code. Once we, as professional developers, realize that the only way to go fast is to create clean and well designed code, then we will see the business’ need for speed as a demand for high quality code.

My vision of the future is quite different from Josuttis’. I see software developers working together to create a discipline of craftsmanship, professionalism, and quality similar to the way that doctors, lawyers, architects, and many other professionals and artisans have done. I see a future where team velocities increase while development costs decrease because of the steadily increasing skill of the teams. I see a future where large software systems are engineered by relatively small teams of craftsmen, and are configured and customized by business people using DSLs tuned to their needs.

I see a future of Clean Code, Craftsmanship, Professionalism, and an overriding imperative for Code Quality.

Master Craftsman Teams.

Uncle Bob — Wed, 01 Apr 2009 16:51:00 -0500

In the early ‘70s F. T. Baker, and Fred Brooks wrote about the notion of Chief Programmer teams. The idea was that a highly skilled chief programer would be assisted by a team of helpers. The chief would write most of the code, delegating simpler tasks to the helpers.

Though the idea was tried at IBM and shown to work well, the idea has never really caught on. One reason may be that the economics don’t appear to be all that good. A team of four juniors making $50K supporting a senior making $100K costs $300K. That just seems like a lot of money for one well supported chief programmer writing most of the code.

But in light of the movement towards software craftsmanship, perhaps it’s time to dust off this old idea and look at it from a different perspective. Consider the following thought experiment…

We are in the throes of a recession that is challenging us to think differently about things. At the same time our universities are raising tuitions and increasing faculty salaries, while requiring ever less from both the faculty and the students. Today’s professors spend about 9 hours per week in the classroom. And the quality of the CS majors graduating from these institutions is, to put it bluntly, wretched.

Why should a young aspiring software professional spend four years and $200K+ to attend an institution that will teach them less about their chosen profession than 3 months of working on a real project with talented mentors? Indeed, why should employers pay $50K for undertrained programmers who are sure to make horrific messes for the next three years of their career?

Consider instead a team of craftspeople. At the center of this team is a master programmer. This is someone who has been programming for two decades or more. This person understand systems at a gut level, and can quickly make technical judgements without agonizing over them. Such a person can direct a team with the kind of calm confidence that only comes with years of experience and seasoning.

Our master has three lieutenant journeymen who themselves each have three apprentices. Our journeymen have at least five years of experience, and are capable both of taking and giving direction. They can make short term tactical decisions, and direct their apprentices accordingly. At the same time they take technical and architectural direction from the master.

The journeymen write a lot of code – almost as much as the master. They also throw away two thirds of the code written by the apprentices, and drive them to redo it better – always better. They teach the apprentices how to refactor, and often pair with them and sit with them while reviewing their code.

The master probably writes more code than any of the team members. This code is often architectural in nature, and helps to establish the seed about which the system will crystalize. The master also sets technical direction and overall architecture, and makes all critical technical decisions. However, the chief job of the master is to relentlessly drive the journeymen towards higher quality and productivity by establishing and enforcing the appropriate disciplines and practices. From the master’s point of view, it’s “my way or the highway.”

Yes, I know this is very anti-scrum. But this is a case where I think scrum in particular, and agile in general just got it wrong. No battle was won by gaining the consensus of the soldiers. And if you don’t think developing a system is a battle against time, resources, and attitudes, then you’ve never built a system. No great team has ever succeeded without a strong coach who imposed discipline, set the vision, and had the final authority over who played and who sat on the bench. Consensus and team rule may be useful tools some of the time, but only in the context of effective leadership.

The apprentices write a significant amount of code—most of which is discarded by the journeymen. The journeymen act as committers. When an apprentice delivers code that the journeyman finds acceptable, the journeyman will commit that code to their main line. Otherwise he sends it back to the apprentice to do over. By the same token the master accepts subsystems from the journeymen and, if acceptable, commits it to the master main line.

The master sees every line of code in the system. The Journeymen see every line of code written by their apprentices, and they write a significant amount themselves. Pairing is prevalent though not universal. Anyone on the team can pair with anyone else. It should not be uncommon for the master to pair with the apprentices from time to time.

Apprentices begin at a young age—perhaps 18 or 19. After all, the idea is that they aren’t going to a university. Rather, the brightest would start as apprentices right out of high school, while others might opt for two years of community college first.

Apprenticeship would last 2-4 years, depending on how quickly the apprentice learns. Apprentices earn no more than minimum wage. After all, they are young, barely productive, and are deriving great benefit from working with experienced journeyman under the guidance of a master.

Attaining journeyman status is something for the master to decide, with advice from the journeymen. The decision is bittersweet because it is a decision to eject the apprentice from the team so that he or she can start the journey. Usually the new journeyman would join the team of a different master.

Some apprentices simply won’t attain journeyman status. In the end, if they fail to make progress, they must be removed from the team. No one should stay an apprentice year after year. It is the responsibility of the journeymen and master to decide if an apprentice should be removed from the team.

The skills of new journeymen will gradually increase. At first they should have only one apprentice. Over the months and years their increasing leadership skills will allow them to take two or three at a time. The most experienced Journeymen might play the role of master in smaller projects.

Attaining master status is no small feat. It requires the consensus of other masters, and must be demonstrated through several successful accomplishments.

I expect a journeyman would make $50-100K depending on their experience. I expect a master would make $150-$250K or more.

Now imagine a team of 13. One master, three journeymen, and nine apprentices. This is a powerful unit. Its size is right at the sweet spot for hyper productive teams. It has the right balance of experience. And… it’s pretty cheap. This team costs ~$600K per year, or about $50K per person. Or (consulting companies take notice) about $23/person/hour.

It seems to me that such a team could do much more, much better, than a team of 12 graduates lead by a tech-lead with five years experience. The productivity and efficiency (and efficiencies of scale!) of this approach should be obvious. Moreover, I think that being an apprentice on a team like this would provide a much better, much faster, and much cheaper education than a university degree. Finally, I think that using teams of this kind could be a remarkably cost-effective way to get very high quality software written very quickly.

Let's Hear it for the Zealots!

Uncle Bob — Tue, 10 Mar 2009 16:58:00 -0500

In his March 1st, 2009 column in SDTimes, Andrew Binstock takes the “Zealots of Agile” to task for claiming that Agile is the one true way. He made the counter-argument that non-agile projects have succeeded, and that agile projects have failed. He implied that the attitudes of the agile zealots are blind to these facts.

What a load of dingoes kidneys!

There is a difference between a zealot, and a religious fanatic. A religious fanatic cannot envision themselves to be wrong. We in the agile community may indeed be zealots, but we know we can be wrong. We know that projects succeeded before agile, and we know that agile projects have failed. We know that there are other schools of thought that are just as valid as ours. Indeed, most of us expect (even hope!) that agile will be superseded by something better one day.

Some of you may remember my keynote address at the 2002 XPUniverse in Chicago. My son, Micah, and I got up in front of the group dressed in Jiu Jistsu garb, and commenced to do a martial arts demonstration.

After the demonstration I made the point that a student of Karate could have given an equally spellbinding, though quite different, show. A student of Tae Kwon Do would amaze us with still other wonderful moves. The morale, of course, was that there are many ways to get the job done. There are many equally valid schools of thought.

However, the knowledge that there are other valid schools of thought does not dilute the zeal a student has for the school of thought he has chosen to study. Do you think that a Jiu Jistsu master isn’t a zealot? You’d better believe he is! To be great, one must have zeal. To rail against zeal is to rail against the desire for greatness. It is zeal that makes you go the extra distance to become great. The Jiu Jitsu master may respect the Karate master, but his goal is to become great at Jiu Jitsu.

So I think it is healthy that agile proponents are vociferous advocates of their school of thought. They are like Karate students who are excited about their new skills, and want to teach those skills to others. If their zeal to excel can infect others with similar zeal, so much the better. We need people in this industry who want to excel.

In his column Andrew derided agile zealots for claiming that Agile was the one true path. I don’t know of any prominent agile advocate who has ever made that claim. Indeed, when Kent Beck and I sat down in 1999 to plan out the XP Immersion training, we were very concerned that we might turn the XP movement into a religion. We did not want to create the impression that XPers were the chosen people and that wisdom would die with us. So we were, in fact, very careful to avoid any hint of the “one true path” argument.

As evidence of this, consider the Snowbird meeting in 2001. That meeting was inclusive, not exclusive. We invited people from all over the spectrum to join us there. DSDM, FDD, Scrum, Crystal, MDA, & RUP were all represented; and the agile manifesto (Yes, Andrew, Manifesto![1]) was the result. That manifesto was a simple statement of values, not a claim to deific knowledge.

Am I an Agile Zealot? Damned right I am! Do I think Agile is the best way to get software done? Damned right I do! I wouldn’t be doing Agile if I didn’t think it was best. Do I think Agile is better than someone else’s school of thought? Let me put it this way: I’m quite happy to go head to head against someone who follows another school of thought. I believe I’d code him right into the ground! Isn’t that what any good student of his craft would think? But I also recognize that one day I’ll lose that competition, and that will be a great day for me because it will give me the opportunity to learn something new and better.

Andrew also mentioned that Kent Beck and I are so obsessed with our “evangelical fervor” that we: “endlessly test and endlessly refine and refactor code to the point that it becomes an all-consuming diversion that does not advance the project”. WTF? I’ve been making monthly releases of FitNesse, and Kent has been cranking out releases of JunitMax; neither of us has been so busy gilding our respective lilies that we are failing to move our respective projects forward. Look around Andrew, we’re shipping code over here brother.

As part of his rant against our “evangelical fervor” Andrew asked this rather snide question: “I broke up that code into a separate class and added seven methods for what benefit, again?” This would appear to be a dig against refactoring – or perhaps excess refactoring. Still the question has a simple answer. We would only break up a module into a class with seven methods if that module was big, ugly, and complex, and by breaking it up we made it simple and easy to understand and modify. It seems to me that leaving the module in a messy state is irresponsible and unprofessional.

Andrew complains that there is very little Agile experience with projects over six million lines of code. I’m not sure where he got that number from, perhaps he’s been watching Steve Austin re-runs. Actually, the majority of my 2008 consulting business has been with very large companies with products well over six million lines of code. Agile-in-the-Large is the name of the game in those companies!

Finally, Andrew says Michael Feathers is a “moderate” Agile proponent. This is the same Michael Feathers who wrote a book defining Legacy code as code without tests – hardly a moderate point of view. Michael is a master, not a moderate.

I think Andrew has fallen into a common trap. He sees the zeal of the agile proponents and mistakes it for religious fanaticism. Instead of respecting that zeal as the outward manifestation of the inward desire to excel, he rails against it as being myopic and exclusive and then views all the statements of the agile proponents through that clouded viewport. Once you have decided that someone is a religious fanatic, it is difficult to accept anything they say.

I think standing against religious fanaticism in the software community is a good thing. I also think that those of us who are zealous must constantly watch that we are not also becoming blind to our own weaknesses and exclusive in our outlook. But I could wish that those who fear religious fanaticism as much as I do would stop confusing it with professional zeal.

So here’s the bottom line. People who excel are, by definition, zealots. People who aren’t zealous, do not excel. So there’s nothing wrong with being a zealot; indeed, zeal is a very positive emotion. The trick to being a zealot is to be mindful that the day will eventually come when some other zealot from a different school of thought will code you into the ground. When that happens, you should thank him for showing you a better way.

¹ I kicked off the Snowbird meeting by challenging the group to write a “manifesto”. I chose the word “manifesto” because some years earlier I had read The Object Oriented Database System Manifesto and I thought it was a clever use of the word.

Quality: It's alive! It's ALIVE!

Uncle Bob — Sat, 07 Mar 2009 15:25:00 -0600

Recently James Bach wrote a compelling post entitled Quality is Dead. As much as I’d like to agree, something interesting has just happened that tempts me to believe in a rebirth.

Just when James finally declares the death of quality, along comes the Manifesto for Software Craftsmanship. This simple document that builds upon the four values declared in the Agile Manifesto went live late on Friday evening. Now, at 3PM on Saturday over 600 people have signed it!

What does this mean? I think it means that many of us have seen what James was talking about, and are tired of it. We don’t want our users posting blogs – as James did – crying that they HATE us. We want our users to be astonished by the elegance and simplicity of our creations. We want other software developers to look under the hood and be amazed and impressed at the clarity and simplicity they see.

We want to be proud of our work!

We want our quality back!

And so a few weeks ago some software developers gathered at the offices of 8th Light in Libertyville, Illinois, and began the process of creating this manifesto. There followed a very active and passionate discussion in the Software Craftsmanship email group. The end result was this manifesto.

I encourage all of you to read and sign the manifesto, join the email group, and help bring quality back to life in the software profession.