Inadvertent-Imprudent Debt.

There is more of this debt than any other kind. It is all too common that software developers create a mess and don’t know they are doing it. They have not developed a nose that identifies code smells. They don’t know design principles, or design patterns. They think that the reek of rotten code is normal, and don’t even identify it as smelling bad. They think that their slow pace through the thick morass of tangled code is the norm, and have no idea they could move faster. These people destroy projects and bring whole companies to their knees. Their name is Doom.

Deliberate-Imprudent Debt.

There is a meme in our industry (call it the DI meme) that tells young software developers that rushing to the finish line at all costs is the right thing to do. This is far worse than the ignorance of the first group because these folks willfully create debt without counting the cost. Worse, this meme is contagious. People who are infected with it tend to infect others, causing an epidemic of deliberately imprudent debtors (sound familiar?) The end result, as we are now all know, is economic catastrophe, inflation (of estimates) and crushing interest (maintenance) payments. They have become death, the destroyer of worlds.

Inadvertent-Prudent Debt.

This is something of an oxymoron. Ironically, it is also the best of all possible states. The fact is that no matter how careful we are, there is always a better solution that we will stumble upon later. How many times have you finished a system only to realize that if you wrote it again, you’d do it very differently, and much better?

The result is that we are always creating a debt, because our hindsight will always show us a better option after it is too late. So even the best outcome still leaves us owing. (Mother Earth will eventually collect that debt!)

Deliberate-Prudent Debt.

This is the quadrant that I have the biggest problem with. And it is this quadrant in which Martin uses the caption I don’t like. The Caption is: “We must ship now and deal with consequences.”

Does this happen? Yes. Should it happen? Rarely, yes. But it damned well better not happen very often, and it damned well better not happen out of some misplaced urge to get done without counting the cost.

The problem I have with this quadrant (DP) is that people who are really in quadrant DI think they are in DP, and use words such as those that appear in the caption as an excuse to rack up a huge imprudent debt.

The real issue is the definition of the word: Imprudent.

So let me ask you a question. How prudent is debt? There is a very simple formula for determining whether debt is prudent or imprudent. You can use this formula in real life, in business, and in programming. The formula is: Does the debt increase your net worth, and can you make the payments?

People often focus on the first criterion, without properly considering the second. Buying a house is almost certain to increase your net worth despite the debt (though lately…). On the other hand, if you cannot make the payments, you won’t keep that house for long. The reason for our current economic woes has a lot to do with people trying to increase their net worth despite the fact that they couldn’t afford the payments. (indeed, they were encouraged by a meme very similar to the DI meme!)

Bad code is always imprudent.

Writing bad code never increases your net worth; and the interest rate is really high. People who write bad code are like those twenty-somethings who max out all their credit cards. Every transaction decreases net worth, and has horrendous consequences for cash flow. In the end, the vast bulk of your effort goes to paying the interest (the inevitable slow down of the team as they push the messes around). Paying down the principle becomes infeasible. (Just the way credit card companies like it.)

Some Suboptimal Design Decision are Prudent Debt.

But most are not. Every once in awhile there is a suboptimal design decision that will increase the net worth of the project by getting that project into customer’s hand’s early.

This is not the same as delivering software that is under-featured. It is often prudent to increase the net worth of a project by giving customers early access to a system without a full and rich feature set. This is not debt. This is more like a savings account that earns interest.

Indeed, this is one reason that most technical debt is imprudent. If you are truly concerned about getting to market early, it is almost always better to do it with fewer features, than with suboptimal design. Missing features are a promise that can be kept. Paying back suboptimal designs creates interest payments that often submerge any attempts at payback and can slow the team to the breaking point.

But there are some cases where a sub-optimal design can increase your net worth by allowing you to deliver early. However, the interest rate needs to be very low, and the principle payments need to be affordable, and big enough to pay back the debt in short order.

What does a low interest rate mean? It means that the sub-optimal design does not infiltrate every part of your system. It means that you can put the sub-optimal design off in a corner where it doesn’t impact your daily development life.

For example, I recently implemented a feature in FitNesse using HTML Frames. This is sub-optimal. On the other hand, the feature is constrained to one small part of the system, and it simply doesn’t impact any other part of the system. It does not impede my progress. There is no mess for me to move around. The interest rate is almost zero! (nice deal if you can get it!)

Implementing that feature with ajax is a much larger project. I would have had to invest a great deal of time and effort, and would have had to restructure massive amounts of the internal code. So the choice was a good one.

Better yet, the customer experience has pretty much been a big yawn. I thought people would really like the feature and would drive me to expand upon it. Instead, the customer base has virtually ignored it.

So my solution will be to pay back this debt by eliminating the feature. It was a cheap experiment, that resulted in my not having to spend a lot of time and effort on a new architecture! Net worth indeed!

But it might have gone the other way. My customers may have said: “Wow, Great! We want more!” At that point it would have been terrible to expand on the HTML Frames! That decision would have been in the DI quadrant. Deliberate imprudence! Rather, my strategy would have been to replace the suboptimal Frames design of the feature with an isolated ajax implementation, and then to gradually migrate the ajax solution throughout the project. That would have been annoying, but loan payments always are.

Summary

So, don’t let the caption in the DP quadrant be an excuse. Don’t fall for the DI meme that says “We just gotta bite the bullet”. Tread very carefully when you enter the DP quadrant. Look around at all your options, because it’s easy to think you are in the DP quadrant when you are really in the DI quadrant.

Remember: Murphy shall send you strong delusion, that you should believe you are in DP; so that you will be damned in DI.

Is there any research on TDD?

It turns out that there is a fair bit.

• One simple google search led me to this blog by Phil Haack in which he reviewed a TDD research paper. Quoting from the paper:

We found that test-first students on average wrote more tests and, in turn, students who wrote more tests tended to be more productive. We also observed that the minimum quality increased linearly with the number of programmer tests, independent of the development strategy employed.

• The same google search led me to this blog by Matt Hawley, in which he reviewed several other research papers. Part of his summary:

* 87.5% of developers reported better requirements understanding. * 95.8% of developers reported reduced debugging efforts. * 78% of developers reported TDD improved overall productivity. * 50% of developers found that it decreased overall development time. * 92% of developers felt that TDD yielded high-quality code. * 79% of developers believed TDD promoted simpler design.

Actually, I recognize some of Matt’s results as coming from a rather famous 2003 study (also in the list of google results) by Laurie Wiliams and Boby George. This study describes a controlled experiment that they conducted in three different companies. Though Matt’s summary above is based (in part) on that study, there is more to say.

In the George-William study teams that practiced TDD took 16% longer to claim that they were done than the teams that did not practice TDD. Apparently tests are more accurate than claims since the non-TDD teams failed to pass one third of the researcher’s hidden acceptance tests, whereas the TDD teams passed about 6 out of 7. To paraphrase Kent Beck: “If it doesn’t have to work, I can get it done a lot faster!”

Another point of interest in this study is that the TDD teams produced a suite of automated tests with very high test coverage (close to 100% in most cases) whereas most of the non-TDD teams did not produce such a suite; even though they had been instructed to.

• Jim Shore wrote a review of yet another research summary which I found in the same google search. This one combines 7 different studies (including George-Williams). Here the results range from dramatically improved quality and productivity to no observed effect.

• Finally, there is this 2008 case Study of TDD at IBM and Microsoft which shows that TDDers enjoy a defect density reduction ranging from 30% to 90% (as measured by defect tracking tools) and a productivity cost of between 15% and 35% (the subjective opinion of the managers). I refer you back to Kent Beck’s comment above.

I’m sure there is more research out there. After all this was just one google search. I think it’s odd that the TDD detractors didn’t find anything when they did their google searches.

• Oh yeah, and then there was that whole issue of IEEE Software that was dedicated to papers and research on TDD.

What projects have been written with TDD, hmmm?

Quite a few, actually. The following is a list of projects that have an automated suite of unit tests with very high coverage. Those that I know for a fact use TDD, I have noted as such. The others, I can only surmise. If you know of any others, please post a comment here.

• JUnit. This one is kind of obvious. JUnit was written by Kent Beck and Erich Gamma using TDD throughout. If you measure software success by sheer distribution, this particular program is wildly successful.

• Fit. Written by Ward Cunningham. The progenitor of most current acceptance testing frameworks.

• FitNesse. This testing framework has tens of thousands of users. It is 70,000 lines of java code, with 90%+ code coverage. TDD throughout. Very small bug-list. Again, if you measure by distribution, another raving success.

• Cucumber,
• Rspec. These two are Testing frameworks in Ruby. Of course you’d expect a testing framework to be written with TDD, wouldn’t you? I know these were. TDD throughout.

• Limelight. A gui framework in JRUby. TDD throughout.
• jfreechart.
• Spring
• JRuby
• Smallsql
• Ant
• MarsProject
• Log4J
• Jmock

Are there others? I’m sure there are. This was just a quick web search. Again, if you know of more, please add a comment.

TDD

Anyone who continues to think that TDD slows you down is living in the stone age. Sorry, that’s just the truth. TDD does not slow you down, it speeds you up.

Look, TDD is not my religion, it is one of my disciplines. It’s like dual entry bookkeeping for accountants, or sterile procedure for surgeons. Professionals adopt such disciplines because they understand the theory behind them, and have directly experienced the benefits of using them.

I have experienced the tremendous benefit that TDD has had in my work, and I have observed it in others. I have seen and experienced the way that TDD helps programmers conceive their designs. I have seen and experienced the way it documents their decisions. I have seen and experienced the decouplings imposed by the tests, and I have seen and experienced the fearlessness with which TDDers can change and clean their code.

To be fair, I don’t think TDD is always appropriate. There are situations when I break the discipline and write code before tests. I’ll write about these situations in another blog. However, these situations are few and far between. In general, for me and many others, TDD is a way to go fast, well, and sure.

The upshot of all this is simple. TDD is a professional discipline. TDD works. TDD makes you faster. TDD is not going away. And anyone who has not really tried it, and yet claims that it would slow them down, is simply being willfully ignorant. I don’t care if your name is Don Knuth, Jamie Zawinski, Peter Seibel, or Peter Pan. Give it a real try, and then you have the right to comment.

Let me put this another way. And now I’m talking directly to those who make the claim that TDD would slow them down. Are you really such a good programmer that you don’t need to thoroughly check your work? Can you conceive of a better way to check your work than to express your intent in terms of an executable test? And can you think of a better way to ensure that you can write that test other than to write it first?

If you can, then I want to hear all about it. but I don’t want to hear that you write a few unit tests after the fact. I don’t want to hear that you manually check your code. I don’t want to hear that you do design and therefore don’t need to write tests. Those are all stone-age concepts. I know. I’ve been there.

So there. <grin>

The Design Pattern Religion

Tim Bray said:

My experience suggests that there are few surer ways to doom a big software project than via the Design Patterns religion.

He’s right of course. The Design Patterns religion is a foul bird that ravages teams and cuts down young projects in their prime. But let’s be clear about what that religion is. The Design Patterns religion is the ardent belief that the use of design patterns is good.

Here’s a clue. Design Patterns aren’t good. They also aren’t bad. They just are. Given a particular software design situation, there may be a pattern that fits and is beneficial. There may also be patterns that would be detrimental. It’s quite possible that none of the currently documented patterns are appropriate and that you should close the book and just solve the problem.

Here’s another clue. You don’t use patterns. You don’t apply patterns. Patterns just are. If a particular pattern is appropriate to solve a given problem, then it will be obvious. Indeed it is often so obvious that you don’t realize that the pattern is in place until you are done. You look back at your code and realize: “Oh, that’s a Decorator!”.

So am I saying that Design Patterns are useless?

NO! I want you to read the patterns books. I want you to know those patterns inside and out. If I point at you and say “Visitor” I want you at the board drawing all the different variants of the pattern without hesitation. I want you to get all the names and roles right. I want you to know patterns.

But I don’t want you to use patterns. I don’t want you to believe in patterns. I don’t want you to make patterns into a religion. Rather I want you to be able to recognize them when they appear, and to regularize them in your code so that others can recognize them too.

Design Patterns have a huge benefit. They have names. If you are reading code, and you see the word “Composite”, and if the author took care to regularize the code to the accepted names and roles of the “Composite” pattern, then you will know what that part of the code is doing instantly. And that is powerful!

Minimizing Concurrency.

In my first Duct Tape blog I made the statement:

I found myself annoyed at Joel’s notion that most programmers aren’t smart enough to use templates, design patterns, multi-threading, COM, etc. I don’t think that’s the case. I think that any programmer that’s not smart enough to use tools like that is probably not smart enough to be a programmer period.

Tim responds with:

...multi-threading is part of the problem, not part of the solution; that essentially no application programmer understands threads well enough to avoid deadlocks and races and horrible non-repeatable bugs. And that COM was one of the most colossal piles of crap my profession ever foisted on itself.

Is concurrency really part of the problem? Yes! Concurrency is a really big part of the problem. Indeed, the first rule of concurrency is: DON’T. The second rule is: REALLY, DON’T.

The problem is that some times you have no choice. And in those situations, where you absolutely must use concurrency, you should know it inside and out!

I completely and utterly reject the notion that ignorance is the best defense. I reject that lack of skill can ever be an advantage. So I want you to know concurrency. I want to shout “Dining Philosophers” and have you run to the board without hesitation and show me all the different solutions. If I holler “Deadlock”, I want you to quickly identify the causes and solutions.

Here’s a clue. If you want to avoid using something, know that something cold.

Sudoku

At the end of his blog, Peter jumps on the pile of bodies already crushing Ron Jeffries regarding the Sudoku problem from July of 2006.

I find the pile-up disturbing. Ron had the courage to fail in public. Indeed he announced up front that he might “crash and burn”. And yet he got lambasted for it by people who hid behind someone else’s work. The responses to Ron’s tutorial blogs were completely unfair because the authors of those blogs had everything worked out for them by Dr. Peter Norvig before they published their screeds. They were comparing apples to oranges because their responses were about the solution whereas Ron’s blogs were about the process.

Which one of us has not gone down a rat-hole when hunting for a solution to a complex problem? Let that person write the first blog. Everyone else ought to be a bit more humble.

Do the people on the pile think that Ron is unable to solve the Sudoku problem? (Some have said as much.) Then they don’t know Ron very well. Ron could code them all under the table with one hand tied behind his back.

Personal issues aside, I find the discussion fascinating in it’s own right. Ron had attempted to solve the Sudoku problem by gaining insight into that problem through the process of coding intermediate solutions. This is a common enough TDD approach. Indeed, the Bowling Game and the Prime Factors Kata are both examples where this approach can work reasonably well.

This approach follows the advice of no less than Grady Booch who (quoting Heinlein) said: “when faced with a problem you do not understand, do any part of it you do understand, then look at it again.“

Ron was attempting to use TDD to probe into the problem to see if he could gain any insight. This technique often bears fruit. Sometimes it does not.

Here is a classic example. Imagine you were going to write a sort algorithm test first:

• Test 1: Sort an empty array. Solution: Return the input array.

• Test 2: Sort an array with one element. Solution: Return the input array.

• Test 3: Sort an array with two elements. Solution: Compare the two elements and swap if out of order. Return the result.

• Test 4: Sort an array with three elements. Solution: Compare the first two and swap if out of order. Compare the second two and swap if out of order. Compare the first two again and swap if out of order. Return the result.

• Test 5: Sort an array with four elements. Solution: Put the compare and swap operations into a nested loop. Return the result.

The end result is a bubble sort. The algorithm virtually self assembles. If you had never heard of a bubble sort before, this simple set of tests would have driven you to implement it naturally.

Problems like Bowling, Prime Factors, and Bubble Sort hold out the interesting promise that TDD may be a way to derive algorithmms from first principles!

On the other hand, what set of tests would drive you to implement a QuickSort? There are none that I know of. QuickSort and Sudoku may require a serious amount of introspection and concentrated thought before the solution is apparent. They may belong to a class of algorithms that do not self-assemble like Bowling, Prime Factors, and Bubble Sort.

This blog by Kurt Christensen provides all the links to the various Sudoku articles, and sums it up this way.

TDD may not be the best tool for inventing new algorithms, it may very well be the best tool for applying those algorithms to the problem at hand.

Actually I think TDD is a good way to find out if an algorithm will self-assemble or not. It usually doesn’t take a lot of time to figure out which it’s going to be.

Businesses make this kind of trade-off all the time; and there’s nothing inherently unwise about it. If the early release of 1.0 drives the business that pays for the development of 2.0 then the business has won. So this kind of reasoned technical debt may indeed be appropriate.

Unfortunately there is another situation that is sometimes called “technical debt” but that is neither reasoned nor wise. A mess.

Technical debt may be necessary, but it had also better be clean! If you are going to implement a frames solution instead of an AJAX solution, then make sure that the workmanship of the frames solution is top-notch. Make sure the design is well balanced, and the code is clean. If you make a mess while implementing that frames solution, you’ll never be able to replace it with an AJAX framework. The mess will impede your progress forever.

A mess is not a technical debt. A mess is just a mess. Technical debt decisions are made based on real project constraints. They are risky, but they can be beneficial. The decision to make a mess is never rational, is always based on laziness and unprofessionalism, and has no chance of paying of in the future. A mess is always a loss.

When you buy a house and take on a big mortgage debt, you tighten up all your spending and accounting. You clean up your books and your budgets. You behave with increased discipline. The same is true of technical debt. The more technical debt you take on, the tighter your disciplines need to be. You should do more testing, and more pairing and more refactoring. Technical debt is not a license to make a mess. Technical debt creates the need for even greater cleanliness.

When you decide to take on a technical debt, you had better make sure that your code stays squeaky clean. Keeping the system clean is the only way you will pay down that debt.

Look. I’m not a big fan of CSM. I think it’s a gimmick. I am not a CSM myself, and have no intention of joining their ranks. When I meet someone who proclaims themselves to be a CSM, I’m not particularly impressed. I know what that certification means, and I take it with a grain of salt. To me, the title of CSM is worth little more than a shrug.

We at Object Mentor do a lot of training in things like Test Driven Development, Agile Methods. Object Oriented Principles, Java, C#, etc. etc. At the end of every course we often sign and pass out certificates to the students. Those certificates proclaim that the student attended the course. I see no difference between that certificate (which is a certification after all) and the CSM certificate. I suppose the students who take our TDD course could claim to be Object Mentor Certified TDDers; and they’d be right.

Have we created an “Integrity Debt” by handing out those certificates? Of course not. Everybody knows exactly what they mean. Nobody misrepresents their intent. They are an honest statement of fact. And the same is true of the CSM certificate.

Is it troubling that some HR people are starting to put CSM requirements on Job postings? Not at all! It is perfectly within the rights of any company to decide that they want to hire people who have been appropriately trained. Are there some HR people who overestimate the value of CSM? Probably, but that’s their own fault.

In my humble opinion there is no significant integrity issue here. Oh it wouldn’t surprise me to learn that there might have been some back-door deals in the early days of CSM. Perhaps some people were given CST status, or CSM status without careful controls. If that happened, I chalk it up to birthing pains which the Scrum Alliance is striving to correct. I don’t think anybody was out to scam anybody else. I don’t think CSM is a far flung conspiracy to ruin the software industry, and I don’t think the US government flew those jets into the twin towers.

Bottom line. There is no “Integrity Debt” here. What there is is a group of honest and caring folks who are trying to figure out the best ways to get Agile concepts adopted in an industry that sorely needs them.

In that regard I think that the agile movement has enjoyed a significant boost because of the interest generated by the CSM program. There are more companies doing Agile today because of CSM. So if anybody owes a debt here, it may be the Agile community owing a debt to the CSM program.

Maybe, instead of accusing and castigating and pointing the finger of judgement and doom we ought give a salute to Ken Schwaber, and say: “Thanks Ken.”

After all, one man’s “one thing” might be someone else’s “two things”.

Consider this class:

  class SymbolReplacer {
    protected String stringToReplace;
    protected List<String> alreadyReplaced = new ArrayList<String>();

    SymbolReplacer(String s) {
      this.stringToReplace = s;
    }

    String replace() {
      Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");
      Matcher symbolMatcher = symbolPattern.matcher(stringToReplace);
      while (symbolMatcher.find()) {
        String symbolName = symbolMatcher.group(1);
        if (getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName)) {
          alreadyReplaced.add(symbolName);
          stringToReplace = stringToReplace.replace("$" + symbolName, translate(symbolName));
        }
      }
      return stringToReplace;
    }

    protected String translate(String symbolName) {
      return getSymbol(symbolName);
    }
  }

It’s not too hard to understand. The replace() function searches through a string looking for $NAME and replaces each instance with the appropriate translation of NAME. It also makes sure that it doesn’t replace a name more than once. Simple.

Of course the words “It also…” pretty much proves that this function does more than one thing. So we can probably split the function up into two functions as follows:

    String replace() {
      Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");
      Matcher symbolMatcher = symbolPattern.matcher(stringToReplace);
      while (symbolMatcher.find()) {
        String symbolName = symbolMatcher.group(1);
        replaceAllInstances(symbolName);
      }
      return stringToReplace;
    }

    private void replaceAllInstances(String symbolName) {
      if (getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName)) {
        alreadyReplaced.add(symbolName);
        stringToReplace = stringToReplace.replace("$" + symbolName, translate(symbolName));
      }
    }

OK, so now the replace() function simply finds all the symbols that need replacing, and the replaceAllInstances() function replaces them if they haven’t already been replaced. So do these function do one thing each?

Well, the replace() compiles the pattern and build the Matcher() Maybe those actions should be moved into the constructor?

  class SymbolReplacer {
    protected String stringToReplace;
    protected List<String> alreadyReplaced = new ArrayList<String>();
    private Matcher symbolMatcher;
    private final Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");

    SymbolReplacer(String s) {
      this.stringToReplace = s;
      symbolMatcher = symbolPattern.matcher(s);
    }

    String replace() {
      while (symbolMatcher.find()) {
        String symbolName = symbolMatcher.group(1);
        replaceAllInstances(symbolName);
      }
      return stringToReplace;
    }

    private void replaceAllInstances(String symbolName) {
      if (getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName)) {
        alreadyReplaced.add(symbolName);
        stringToReplace = stringToReplace.replace("$" + symbolName, translate(symbolName));
      }
    }

    protected String translate(String symbolName) {
      return getSymbol(symbolName);
    }
  }

OK, so now certainly the replace() function is doing one thing? Ah, but I see at least two. It loops, extracts the symbolName and then does the replace. OK, so how about this?

    String replace() {
      for (String symbolName = nextSymbol(); symbolName != null; symbolName = nextSymbol())
        replaceAllInstances(symbolName);

      return stringToReplace;
    }

    private String nextSymbol() {
      return symbolMatcher.find() ? symbolMatcher.group(1) : null;
    }

I had to restructure things a little bit. The loop is a bit ugly. I wish I could have said for (String symbolName : symbolMatcher) but I guess Matchers don’t work that way.

I kind of like the nextSymbol() function. It gets the Matcher nicely out of the way.

So now the replace() and nextSymbol() functions are certainly doing one thing. Aren’t they?

Well, I suppose I could separate the loop from the return in replace().

    String replace() {
      replaceAllSymbols();
      return stringToReplace;
    }

    private void replaceAllSymbols() {
      for (String symbolName = nextSymbol(); symbolName != null; symbolName = nextSymbol())
        replaceAllInstances(symbolName);
    }

I don’t see how I could make these functions smaller. They must be doing one thing. There’s no way to extract any other functions from them!

Uh… Wait. Is that the definition of one thing? Is a function doing one thing if, and only if, you simply cannot extract any other functions from it? What else could “one thing” mean? After all, If I can extract one function out of another, the original function must have been doing more than one thing.

So does that mean that for all these years the authors and consultants (like me) have been telling us to extract until you can’t extract anymore?

Let’s try that with the rest of this class and see what it looks like…

  class SymbolReplacer {
    protected String stringToReplace;
    protected List<String> alreadyReplaced = new ArrayList<String>();
    private Matcher symbolMatcher;
    private final Pattern symbolPattern = Pattern.compile("\\$([a-zA-Z]\\w*)");

    SymbolReplacer(String s) {
      this.stringToReplace = s;
      symbolMatcher = symbolPattern.matcher(s);
    }

    String replace() {
      replaceAllSymbols();
      return stringToReplace;
    }

    private void replaceAllSymbols() {
      for (String symbolName = nextSymbol(); symbolName != null; symbolName = nextSymbol())
        replaceAllInstances(symbolName);
    }

    private String nextSymbol() {
      return symbolMatcher.find() ? symbolMatcher.group(1) : null;
    }

    private void replaceAllInstances(String symbolName) {
      if (shouldReplaceSymbol(symbolName))
        replaceSymbol(symbolName);
    }

    private boolean shouldReplaceSymbol(String symbolName) {
      return getSymbol(symbolName) != null && !alreadyReplaced.contains(symbolName);
    }

    private void replaceSymbol(String symbolName) {
      alreadyReplaced.add(symbolName);
      stringToReplace = stringToReplace.replace(
        symbolExpression(symbolName), 
        translate(symbolName));
    }

    private String symbolExpression(String symbolName) {
      return "$" + symbolName;
    }

    protected String translate(String symbolName) {
      return getSymbol(symbolName);
    }
  }

Well, I think it’s pretty clear that each of these functions is doing one thing. I’m not sure how I’d extract anything further from any of them.

Perhaps you think this is taking things too far. I used to think so too. But after programming for over 40+ years, I’m beginning to come to the conclusion that this level of extraction is not taking things too far at all. In fact, to me, it looks just about right.

So, my advice: Extract till you just can’t extract any more. Extract till you drop.

After all, with modern tools it takes very little time. It makes each function almost trivial. The code reads very nicely. It forces you to put little snippets of code into nicely named functions. And, well gosh, extracting till you drop is kind of fun!

The details are not really that much more difficult. You start at the left-most point and calculate the angle from vertical required to touch every other point. The point with the minimum angle is the next point. You keep going around looking finding points with the minimum angle that is greater than the previous angle. When you get back to the starting point you are done.

Calculating angles can be time consuming, so I use a psuedo-angle algorithm. It doesn’t calculate the actual angle, rather it is a function that increases with the true angle, and goes from [0, 4).

The code is pretty simple.

(ns convexHull.convex-hull
  (:use clojure.contrib.math))

(defn quadrant-one-pseudo-angle [dx dy]
  (/ dx (+ dy dx)))

(defn pseudo-angle [[dx dy]]
  (cond
    (and (= dx 0) (= dy 0))
    0

    (and (>= dx 0) (> dy 0))
    (quadrant-one-pseudo-angle dx dy)

    (and (> dx 0) (<= dy 0))
    (+ 1 (quadrant-one-pseudo-angle (abs dy) dx))

    (and (<= dx 0) (< dy 0))
    (+ 2 (quadrant-one-pseudo-angle (abs dx) (abs dy)))

    (and (< dx 0) (>= dy 0))
    (+ 3 (quadrant-one-pseudo-angle dy (abs dx)))

    :else nil))

(defn point-min [[x1 y1 :as p1] [x2 y2 :as p2]]
  (cond
    (< x1 x2)
    p1

    (= x1 x2)
    (if (< y1 y2) p1 p2)

    :else
    p2))

(defn find-min-point [points]
  (reduce point-min points))

(defn delta-point [[x1 y1] [x2 y2]]
  [(- x1 x2) (- y1 y2)])

(defn angle-and-point [point base]
  [(pseudo-angle (delta-point point base)) point])

(defn min-angle-and-point [ap1 ap2]
  (if (< (first ap1) (first ap2)) ap1 ap2))

(defn find-point-with-least-angle-from [base angle points]
  (reduce min-angle-and-point
    (remove
      #(< (first %) angle)
      (map #(angle-and-point % base)
        (remove
          (fn [p] (= base p))
          points)))))

(defn hull [points]
  (println "Start")
  (let [starting-point (find-min-point points)]
    (println starting-point)
    (loop [hull-list [starting-point] angle 0 last-point starting-point]
      (let [[angle next-point] (find-point-with-least-angle-from last-point angle points)]
        (if (= next-point (first hull-list))
          hull-list
          (recur (conj hull-list next-point) angle next-point))))))

I execute it with this:

(ns convexHull.time-hull
  (:use convexHull.convex-hull))

(def r (java.util.Random.))
(defn rands [] (repeatedly #(.nextGaussian r)))
(defn points [] (take 400000 (partition 2 (rands))))
(let [hull-points (time (hull (points)))]
  (printf "Points: %d\n" (count hull-points))
  (doseq [x hull-points] (println x)))

This takes way too long to run. The equivalent java program will do a million points in half a second. This one is taking 25 seconds to do a half-million points. It won’t even do a million points. It slows way way down and then runs out of memory. (There must be some kind of disk caching going on or something.)

Anyway, I’d be interested in seeing how a real Clojure programmer would speed this program up.

(use 'clojure.contrib.duck-streams)

(def file-counts
  (map
    #(list (.toString %) (count (read-lines %)))
    (remove #(.isDirectory %)
      (file-seq (file-str "~/projects/FitNesseGit/src/fitnesse")))))

(defn count-lines-that-end-with [file-counts suffix]
  (reduce +
    (map second
      (filter #(.endsWith (first %) suffix) file-counts))))

(def java-count (count-lines-that-end-with file-counts ".java"))
(def test-count (count-lines-that-end-with file-counts "Test.java"))

(printf "Java lines: %d\n" java-count)
(printf "Test lines: %d\n" test-count)
(printf "Test pct: %.1f\n" (double (* 100 (/ test-count java-count))))

Today, my son Justin and I managed to do just that by following the instruction in Stuart Halloway’s book, the Clojure api website, and Mark Volkmann’s very useful site.

Be advised, that it takes a bit of fiddling to get this to work. You will have to jockey around with your classpaths and output directories. But it’s not actually that hard to do, and the results can be very rewarding.

We implemented the Bowling game (again), but this time we wrote the unit tests in Java, and had them call into Clojure. From the point of view of the Java tests, it looked just like we were calling into java code. The tests had no idea that this was Clojure.

Here are the tests:

package bowling;

import static org.junit.Assert.assertEquals;
import org.junit.Before;
import org.junit.Test;

public class BowlingTest {
  private Game g;

  @Before
  public void setup() {
    g = new Game();
  }

  @Test
  public void roll() throws Exception {
    g.roll(0);
  }

  @Test
  public void gutterGame() throws Exception {
    rollMany(20, 0);
    assertEquals(0, g.score());
  }

  private void rollMany(int n, int pins) {
    for (int i=0; i< n; i++) {
      g.roll(pins);
    }
  }

  @Test
  public void allOnes() throws Exception {
    rollMany(20, 1);
    assertEquals(20, g.score());    
  }

  @Test
  public void oneSpare() throws Exception {
    g.roll(5);
    g.roll(5);
    g.roll(3);
    rollMany(17, 0);
    assertEquals(16, g.score());
  }

  @Test
  public void oneStrike() throws Exception {
    g.roll(10);
    g.roll(5);
    rollMany(17,0);
    assertEquals(20, g.score());
  }

  @Test
  public void perfectGame() throws Exception {
    rollMany(12, 10);
    assertEquals(300, g.score());
  }

  @Test
  public void allSpares() throws Exception {
    rollMany(21, 5);
    assertEquals(150, g.score());
  }
}

The clojure code to make them pass looks like this:

(ns bowling.Game
  (:gen-class
    :state state
    :init init
    :methods [
    [roll [int] void]
    [score [] int]
    ]))

(defn -init [] [[] (atom [])])

(defn -roll [this pins]
  (reset! (.state this) (conj @(.state this) pins)))

(declare score-frames)

(defn -score [this]
  (reduce + (take 10 (score-frames [] (conj @(.state this) 0)))))

(defn score-frames [scores [first second third :as rolls]]
  (cond
    (or (empty? rolls) (nil? second) (nil? third)) scores
    (= 10 first) (recur (conj scores (+ 10 second third)) (next rolls))
    (= 10 (+ first second)) (recur (conj scores (+ 10 third)) (nnext rolls))
    :else
    (recur (conj scores (+ first second)) (nnext rolls))))

The magic is all up in that (ns bowling.game… block.

• The name of the java class is Game in the bowling package, or just bowling.Game.
• Clojure will create a special member function named state that you can use to squirrel away the state variables of the class.
• A function named -init will be called when your class is constructed. You must write this function to return a vector containing 1) a vector of all the arguments to pass to the base class constructor (in my case none), and the initial state of the newly created instance, (in my case an empty vector stuffed into an atom)
• The class will have two new methods, roll, which takes an int and returns void, and score which takes nothing and returns an int.

When a method is called on the instance, Clojure automatically invokes a function of the same name, but prefixed by a -, so -roll and -score are the implementations of the roll and score methods. Note that each of these functions take a this argument. You can get the state out of this by saying (.state this).

From a source code point of view, that’s about all there is to it. But how do you compile this into a java .class file?

Here’s what we did. We created a new file named compile.clj that looks like this:

(set! *compile-path* "../../classes")
(compile 'bowling.Game)

We can run this file from our IDE (IntelliJ using the LAClojure plugin) and it will compile quite nicely. But there are a few things you have to make sure you do.

• Find out where your IDE puts the .class files, and set the compile-path variable to that directory.
• Also make sure that directory is in your classpath.
• Make sure that directory exists!
• Also make sure that your source file directory (the directory that contains the packages) is in your classpath. (I know… but that’s apparently what it takes.)

You will get errors. And the errors aren’t particularly informative. The key to understanding them is to look at the backtrace of the exceptions they throw. Notice, for example, if the function “WriteClassFile” (or something like that) is buried in the backtrace. If so, you are probably having trouble writing your class file.

In the end, we wrote up an ant script that compiled our clojure and our java together. (You have to compile them in the right order! If java calls Clojure, those .class files have to exist before the java will compile!)

The ant script we used was created for us by IntelliJ, and then we modified it to get it to work. I include it here with the caveat that we made it work, but we didn’t make it “right”. You can ignore most of the stuff at the beginning. The interesting part is the Clojure target, and the clean target.

<?xml version="1.0" encoding="UTF-8"?>
<project name="bowlingforjunit" default="all">

  <property file="bowlingforjunit.properties"/>
  <!-- Uncomment the following property if no tests compilation is needed -->
  <!-- 
  <property name="skip.tests" value="true"/>
   -->

  <!-- Compiler options -->

  <property name="compiler.debug" value="on"/>
  <property name="compiler.generate.no.warnings" value="off"/>
  <property name="compiler.args" value=""/>
  <property name="compiler.max.memory" value="128m"/>
  <patternset id="ignored.files">
    <exclude name="**/CVS/**"/>
    <exclude name="**/SCCS/**"/>
    <exclude name="**/RCS/**"/>
    <exclude name="**/rcs/**"/>
    <exclude name="**/.DS_Store/**"/>
    <exclude name="**/.svn/**"/>
    <exclude name="**/.pyc/**"/>
    <exclude name="**/.pyo/**"/>
    <exclude name="**/*.pyc/**"/>
    <exclude name="**/*.pyo/**"/>
    <exclude name="**/.git/**"/>
    <exclude name="**/.sbas/**"/>
    <exclude name="**/.IJI.*/**"/>
    <exclude name="**/vssver.scc/**"/>
    <exclude name="**/vssver2.scc/**"/>
  </patternset>
  <patternset id="library.patterns">
    <include name="*.zip"/>
    <include name="*.war"/>
    <include name="*.egg"/>
    <include name="*.ear"/>
    <include name="*.swc"/>
    <include name="*.jar"/>
  </patternset>
  <patternset id="compiler.resources">
    <include name="**/?*.properties"/>
    <include name="**/?*.xml"/>
    <include name="**/?*.gif"/>
    <include name="**/?*.png"/>
    <include name="**/?*.jpeg"/>
    <include name="**/?*.jpg"/>
    <include name="**/?*.html"/>
    <include name="**/?*.dtd"/>
    <include name="**/?*.tld"/>
    <include name="**/?*.ftl"/>
  </patternset>

  <!-- JDK definitions -->

  <property name="jdk.bin.1.6" value="${jdk.home.1.6}/bin"/>
  <path id="jdk.classpath.1.6">
    <fileset dir="${jdk.home.1.6}">
      <include name="lib/deploy.jar"/>
      <include name="lib/dt.jar"/>
      <include name="lib/javaws.jar"/>
      <include name="lib/jce.jar"/>
      <include name="lib/management-agent.jar"/>
      <include name="lib/plugin.jar"/>
      <include name="lib/sa-jdi.jar"/>
      <include name="../Classes/charsets.jar"/>
      <include name="../Classes/classes.jar"/>
      <include name="../Classes/dt.jar"/>
      <include name="../Classes/jce.jar"/>
      <include name="../Classes/jconsole.jar"/>
      <include name="../Classes/jsse.jar"/>
      <include name="../Classes/laf.jar"/>
      <include name="../Classes/management-agent.jar"/>
      <include name="../Classes/ui.jar"/>
      <include name="lib/ext/apple_provider.jar"/>
      <include name="lib/ext/dnsns.jar"/>
      <include name="lib/ext/localedata.jar"/>
      <include name="lib/ext/sunjce_provider.jar"/>
      <include name="lib/ext/sunpkcs11.jar"/>
    </fileset>
  </path>

  <property name="project.jdk.home" value="${jdk.home.1.6}"/>
  <property name="project.jdk.bin" value="${jdk.bin.1.6}"/>
  <property name="project.jdk.classpath" value="jdk.classpath.1.6"/>

  <!-- Project Libraries -->

  <!-- Global Libraries -->

  <path id="library.clojure.classpath">
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/ant-launcher.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/ant.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/clojure-contrib.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/clojure.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/commons-codec-1.3.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/commons-fileupload-1.2.1.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/commons-io-1.4.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/compojure.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/hsqldb.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/jetty-6.1.14.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/jetty-util-6.1.14.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/jline-0.9.94.jar"/>
    <pathelement location="/Users/unclebob/projects/clojure-build/lib/servlet-api-2.5-6.1.14.jar"/>
  </path>

  <!-- Modules -->

  <!-- Module BowlingForJunit -->

  <dirname property="module.bowlingforjunit.basedir" file="${ant.file}"/>

  <property name="module.jdk.home.bowlingforjunit" value="${project.jdk.home}"/>
  <property name="module.jdk.bin.bowlingforjunit" value="${project.jdk.bin}"/>
  <property name="module.jdk.classpath.bowlingforjunit" value="${project.jdk.classpath}"/>

  <property name="compiler.args.bowlingforjunit" value="${compiler.args}"/>

  <property name="bowlingforjunit.output.dir" value="${module.bowlingforjunit.basedir}/classes"/>
  <property name="bowlingforjunit.testoutput.dir" value="${module.bowlingforjunit.basedir}/classes"/>

  <path id="bowlingforjunit.module.bootclasspath">
    <!-- Paths to be included in compilation bootclasspath -->
  </path>

  <path id="bowlingforjunit.module.classpath">
    <path refid="${module.jdk.classpath.bowlingforjunit}"/>
    <pathelement location="/Library/junit4.6/junit-4.6.jar"/>
    <path refid="library.clojure.classpath"/>
    <pathelement location="${basedir}/classes"/>
  </path>

  <path id="bowlingforjunit.runtime.module.classpath">
    <pathelement location="${bowlingforjunit.output.dir}"/>
    <pathelement location="/Library/junit4.6/junit-4.6.jar"/>
    <path refid="library.clojure.classpath"/>
    <pathelement location="${basedir}/classes"/>
  </path>

  <patternset id="excluded.from.module.bowlingforjunit">
    <patternset refid="ignored.files"/>
  </patternset>

  <patternset id="excluded.from.compilation.bowlingforjunit">
    <patternset refid="excluded.from.module.bowlingforjunit"/>
  </patternset>

  <path id="bowlingforjunit.module.sourcepath">
    <dirset dir="${module.bowlingforjunit.basedir}">
      <include name="src"/>
    </dirset>
  </path>

  <target name="compile" depends="clojure, compile.java" description="Compile module BowlingForJunit"/>

  <target name="compile.java" description="Compile module BowlingForJunit; production classes">
    <mkdir dir="${bowlingforjunit.output.dir}"/>
    <javac destdir="${bowlingforjunit.output.dir}" debug="${compiler.debug}" nowarn="${compiler.generate.no.warnings}" 
           memorymaximumsize="${compiler.max.memory}" fork="true" executable="${module.jdk.bin.bowlingforjunit}/javac">
      <compilerarg line="${compiler.args.bowlingforjunit}"/>
      <bootclasspath refid="bowlingforjunit.module.bootclasspath"/>
      <classpath refid="bowlingforjunit.module.classpath"/>
      <src refid="bowlingforjunit.module.sourcepath"/>
      <patternset refid="excluded.from.compilation.bowlingforjunit"/>
    </javac>

    <copy todir="${bowlingforjunit.output.dir}">
      <fileset dir="${module.bowlingforjunit.basedir}/src">
        <patternset refid="compiler.resources"/>
        <type type="file"/>
      </fileset>
    </copy>
  </target>

  <target name="clojure">
    <java classname="clojure.lang.Compile">
      <classpath>
        <path location="/Users/unclebob/projects/clojure/BowlingForJunit/classes"/>
        <path location="/Users/unclebob/projects/clojure/BowlingForJunit/src"/>        
        <path location="/Users/unclebob/projects/clojure/BowlingForJunit/src/bowling"/>
        <path location="/Users/unclebob/projects/clojure-build/lib/clojure.jar"/>
        <path location="/Users/unclebob/projects/clojure-build/lib/clojure-contrib.jar"/>  
      </classpath>
      <sysproperty key="clojure.compile.path" value="/Users/unclebob/projects/clojure/BowlingForJunit/classes"/>
      <sysproperty key="java.awt.headless" value="true"/>
      <arg value="bowling.Game"/>
    </java>
  </target>

  <target name="clean" description="cleanup module">
    <delete dir="${bowlingforjunit.output.dir}"/>
    <delete dir="${bowlingforjunit.testoutput.dir}"/>
    <mkdir  dir="${bowlingforjunit.output.dir}"/>
  </target>

  <target name="all" depends="clean, compile" description="build all"/>
</project>

One of the better examples (though perhaps a bit trivial) is the Prime Factors Kata. This lovely little experiment grows and grows as you add test cases, and then suddenly collapses into an elegant three line algorithm.

The tests continue to become ever more specific. The production code starts out just as specific as the tests. But with the second or third test the programmer must make a decision. He can write the production code to mirror the tests (i.e. writing it as an if/else statement that detects which test is running and supplying the expected answer) or he can come up with some kind of more general algorithm that satisfies the tests without looking anything like them.

The algorithm grows and warps and twists; and then, just when it looks like it’s destined to become a wretched mess; it simply evaporates into a lovely little three line nested loop.

We see the principle at work in other ways as well. Often the programmers have a whole list of tests that they know must pass. As they write them one by one, they write the production code that satisfies them. Then, as in the Bowling Game Kata the tests start to pass unexpectedly. You were done with the code, and you weren’t aware of it. You continue writing tests, expecting one to fail, but they all pass. The test code grows, but the production code remains the same.

Sometimes this happens in a less surprising way. Sometimes you know that you have implemented an algorithm that will pass all remaining tests, but you write those tests anyway because they are part of the specification

The point is that test code and production code do not grow at the same rate. Indeed, as the application increases in complexity, the test code grows at a rate that is faster than the production code. Sometimes the production code actually shrinks as the test code grows because the programmers moved a load of functionality out of the code and into the data.

Consider FitNesse. A year ago there were 45,000 lines of code, of which 15,000 were tests, so 33% of the total were tests.

Now Fitnesse is 58,000 lines of code of which 26,000 are tests. We added 13,000 lines of code overall, but 8,000 (61%), are tests! The tests have grown to over 44% of the total.