This myth is not part of agile at all. Rather it is an hyper-zealous response to the real Agile proscription of Big Design Up Front (BDUF). There should be no doubt that BDUF is harmful. It makes no sense at all for designers and architects to spend month after month spinning system designs based on a daisy-chain of untested hypotheses. To paraphrase John Gall: Complex systems designed from scratch never work.

However, there are architectural issues that need to be resolved up front. There are design decisions that must be made early. It is possible to code yourself into a very nasty cul-de-sac that you might avoid with a little forethought.

Notice the emphasis on size here. Size matters! ‘B’ is bad, but ‘L’ is good. Indeed, LDUF is absolutely essential.

How big are these B’s and L’s? It depends on the size of the project of course. For most projects of moderate size I think a few days ought to be sufficient to think through the most important architectural issues and start testing them with iterations. On the other hand, for very large projects, I seen nothing wrong with spending anywhere from a week to even a month, thinking through architectural issues.

In some circles this early spate of architectural thought is called Iteration 0. The goal is to make sure you’ve got your ducks in a row before you go off half-cocked and code yourself into a nightmare.

When I work on FitNesse, I spend a lot of time thinking about how I should implement a new feature. For most features I spend an hour or two considering alternative implementations. For larger features I’ve spent one or two days batting notions back and forth. There have been times when I’ve even drawn UML diagrams.

On the other hand, I don’t allow those early design plans to dominate once I start TDDing. Often enough the TDD process leads me in a direction different from those plans. That’s OK, I’m glad I made those earlier plans. Even if I don’t follow them they helped me to understand and constrain the problem. They gave me the context to evaluate the new solution that TDD helped me to discover. To paraphrase Eisenhower: Individual plans may not turn out to be helpful, but the act of planning is always indispensable.

So here’s the bottom line. If you are working in an Agile team, don’t feel guilty about taking a day or two to think some issues through. Indeed, feel guilty if you don’t take a little time to think things through. Don’t feel that TDD is the only way to design. On the other hand, don’t let yourself get too vested in your designs. Allow TDD to change your plans if it leads you in a different direction.

I say this because of recent trends in our industry and my hands-on experience with many enterprise and Internet applications, mostly at client sites. You might be thinking that I’m referring to the mainstream breakout of Functional Programming (FP), which is happening right now. The killer app for FP is concurrency. We’ve all heard that more and more applications must be concurrent these days (which doesn’t necessarily mean multithreaded). When we remove side effects from functions and disallow mutable variables, our concurrency issues largely go away. The success of the Actor model of concurrency, as used to great effect in Erlang, is one example of a functional-style approach. The rise of map-reduce computations is another example of a functional technique going mainstream. A related phenomenon is the emergence of key-value store databases, like BigTable and CouchDB, is a reaction to the overhead of SQL databases, when the performance cost of the Relational Model isn’t justified. These databases are typically managed with functional techniques, like map-reduce.

But actually, I’m thinking of something else. Hybrid languages like Scala, F#, and OCaml have demonstrated that OOP and FP can complement each other. In a given context, they let you use the idioms that make the most sense for your particular needs. For example, immutable “objects” and functional-style pattern matching is a killer combination.

What’s really got me thinking that objects are losing their supremacy is a very mundane problem. It’s a problem that isn’t new, but like concurrency, it just seems to grow worse and worse.

The problem is that there is never a stable, clear object model in applications any more. What constitutes a BankAccount or Customer or whatever is fluid. It changes with each iteration. It’s different from one subsystem to another even within the same iteration! I see a lot of misfit object models that try to be all things to all people, so they are bloated and the teams that own them can’t be agile. The other extreme is “balkanization”, where each subsystem has its own model. We tend to think the latter case is bad. However, is lean and mean, but non-standard, worse than bloated, yet standardized?

The fact is, for a lot of these applications, it’s just data. The ceremony of object wrappers doesn’t carry its weight. Just put the data in a hash map (or a list if you don’t need the bits “labeled”) and then process the collection with your iterate, map, and reduce functions. This may sound heretical, but how much Java code could you delete today if you replaced it with a stored procedure?

These alternatives won’t work for all situations, of course. Sometimes polymorphism carries its weight. Unfortunately, it’s too tempting to use objects as if more is always better, like cow bell.

So what would replace objects for supremacy? Well, my point is really that there is no one true way. We’ve led ourselves down the wrong path. Or, to be more precise, we followed a single, very good path, but we didn’t know when to take a different path.

Increasingly, the best, most nimble designs I see use objects with a light touch; shallow hierarchies, small objects that try to obey the Single Responsibility Principle, composition rather than inheritance, etc. Coupled with a liberal use of functional idioms (like iterate, map, and reduce), these designs strike the right balance between the protection of data hiding vs. openness for easy processing. By the way, you can build these designs in almost any of our popular languages. Some languages make this easier than others, of course.

Despite the hype, I think Domain-Specific Languages (DSLs) are also very important and worth mentioning in this context. (Language-Oriented Programming – LOP – generalizes these ideas). It’s true that people drink the DSL Kool-Aid and create a mess. However, when used appropriately, DSLs reduce a program to its essential complexity, while hiding and modularizing the accidental complexity of the implementation. When it becomes easy to write a user story in code, we won’t obsess as much over the details of a BankAccount as they change from one story to another. We will embrace more flexible data persistence models, too.

Back to OOP and FP, I see the potential for their combination to lead to a rebirth of the old vision of software components, but that’s a topic for another blog post.

The Interwebs are full of technical comparisons between Java and the different languages, e.g., why language X fixes Java’s perceived issue Y. I won’t rehash those arguments here, but I will describe some language features, as needed.

A similar “polyglot” trend is happening on the .NET platform.

The New JVM Languages

I’ll limit my discussion to these representative (and best known) alternative languages for the JVM.

1. JRuby – Ruby running on the JVM.
2. Scala – A hybrid object-oriented and functional language that runs on .NET as well as the JVM. (Disclaimer: I’m co-writing a book on Scala for O’Reilly.)
3. Clojure – A Lisp dialect.

I picked these languages because they seem to be the most likely candidates for most enterprises considering a new JVM language, although some of the languages listed below could make that claim.

There are other deserving languages besides these three, but I don’t have the time to do them justice. Hopefully, you can generalize the subsequent discussion for these other languages.

1. Groovy – A dynamically-typed language designed specifically for interoperability with Java. It will appeal to teams that want a dynamically-typed language that is closer to Java than Ruby. With Grails, you have a combination that’s comparable to Ruby on Rails.
2. Jython – The first non-Java language ported to the JVM, started by Jim Hugunin in 1997. Most of my remarks about JRuby are applicable to Jython. Django is the Python analog of Rails. If your Java shop already has a lot of Python, consider Jython.
3. Fan – A hybrid object-oriented and functional language that runs on .NET, too. It has a lot of similarities to Scala, like a scripting-language feel.
4. Ioke – (pronounced “eye-oh-key”) An innovative language developed by Ola Bini and inspired by Io and Lisp. This is the newest language discussed here. Hence, it has a small following, but a lot of potential. The Io/Lisp-flavored syntax will be more challenging to average Java developers than Scala, JRuby, Jython, Fan, and JavaScript.
5. JavaScript, e.g., Rhino – Much maligned and misunderstood (e.g., due to buggy and inconsistent browser implementations), JavaScript continues to gain converts as an alternative scripting language for Java applications. It is the default scripting language supported by the JDK 6 scripting interface.
6. Fortress – A language designed as a replacement for high-performance FORTRAN for industrial and academic “number crunching”. This one will interest scientists and engineers…

Note: Like a lot of people, I use the term scripting language to refer to languages with a lightweight syntax, usually dynamically typed. The name reflects their convenience for “scripting”, but that quality is sometimes seen as pejorative; they aren’t seen as “serious” languages. I reject this view.

To learn more about what people are doing on the JVM today (with some guest .NET presentations), a good place to start is the recent JVM Language Summit.

Criteria For Evaluating New JVM Languages

I’ll frame the discussion around a few criteria you should consider when evaluating language choices. I’ll then discuss how each of the languages address those criteria. Since we’re restricting ourselves to JVM languages, I assume that each language compiles to valid byte code, so code in the new language and code written in Java can call each other, at least at some level. The “some level” part will be one criterion. Substitute X for the language you are considering.

1. Interoperability: How easily can X code invoke Java code and vice versa? Specifically:
   1. Create objects (i.e., call new Foo(...)).
   2. Call methods on an object.
   3. Call static methods on a class.
   4. Extend a class.
   5. Implement an interface.
2. Object Model: How different is the object model of X compared to Java’s object model? (This is somewhat tied to the previous point.)
3. New “Ideas”: Does X support newer programming trends:
   1. Functional Programming.
   2. Metaprogramming.
   3. Easier approaches to writing robust concurrent applications.
   4. Easier support for processing XML, SQL queries, etc.
   5. Support internal DSL creation.
   6. Easier presentation-tier development of web and thick-client UI’s.
4. Stability: How stable is the language, in terms of:
   1. Lack of Bugs.
   2. Stability of the language’s syntax, semantics, and library API’s. (All the languages can call Java API’s.)
5. Performance: How does code written in X perform?
6. Adoption: Is X easy to learn and use?
7. Tool Support: What about editors, IDE’s, code coverage, etc.
8. Deployment: How are apps and libraries written in X deployed?
   1. Do I have to modify my existing infrastructure, management, etc.?

The Interoperability point affects ease of adoption and use with a legacy Java code base. The Object Model and Adoption points address the barrier to adoption from the learning point of view. The New “Ideas” point asks what each language brings to development that is not available in Java (or poorly supported) and is seen as valuable to the developer. Finally, Stability, Performance, and Deployment address very practical issues that a candidate production language must address.

Comparing the Languages

JRuby

JRuby is the most popular alternative JVM langauge, driven largely by interest in Ruby and Ruby on Rails.

Interoperability

Ruby’s object model is a little different than Java’s, but JRuby provides straightforward coding idioms that make it easy to call Java from JRuby. Calling JRuby from Java requires the JSR 223 scripting interface or a similar approach, unless JRuby is used to compile the Ruby code to byte code first. In that case, shortcuts are possible, which are well documented.

Object Model

Ruby’s object model is a little different than Java’s. Ruby support mixin-style modules, which behave like interfaces with implementations. So, the Ruby object model needs to be learned, but it is straightforward or the Java developer.

New Ideas

JRuby brings closures to the JVM, a much desired feature that probably won’t be added in the forthcoming Java 7. Using closures, Ruby supports a number of functional-style iterative operations, like mapping, filtering, and reducing/folding. However, Ruby does not fully support functional programming.

Ruby uses dynamic-typing instead of static-typing, which it exploits to provide extensive and powerful metaprogramming facilities.

Ruby doesn’t offer any specific enhancements over Java for safe, robust concurrent programming.

Ruby API’s make XML processing and database access relatively easy. Ruby on Rails is legendary for improving the productivity of web developers and similar benefits are available for thick-client developers using other libraries.

Ruby is also one of the best languages for defining “internal” DSL’s, which are used to great affect in Rails (e.g., ActiveRecord).

Stability

JRuby and Ruby are very stable and are widely used in production. JRuby is believed to be the best performing Ruby platform.

The Ruby syntax and API are undergoing some significant changes in the current 1.9.X release, but migration is not a major challenge.

Performance

JRuby is believed to be the best performing Ruby platform. While it is a topic of hot debate, Ruby and most dynamically-typed languages have higher runtime overhead compared to statically-typed languages. Also, the JVM has some known performance issues for dynamically-typed languages, some of which will be fixed in JDK 7.

As always, enterprises should profile code written in their languages of choice to pick the best one for each particular task.

Adoption

Ruby is very easy to learn, although effective use of advanced techniques like metaprogramming require some time to master. JRuby-specific idioms are also easy to master and are well documented.

Tool Support

Ruby is experiencing tremendous growth in tool support. IDE support still lags support for Java, but IntelliJ, NetBeans, and Eclipse are working on Ruby support. JRuby users can exploit many Java tools.

Code analysis tools and testing tools (TDD and BDD styles) are now better than Java’s.

Deployment

JRuby applications, even Ruby on Rails applications, can be deployed as jars or wars, requiring no modifications to an existing java-based infrastructure. Teams use this approach to minimize the “friction” of adopting Ruby, while also getting the performance benefits of the JVM.

Because JRuby code is byte code at runtime, it can be managed with JMX, etc.

Scala

Scala is a statically-typed language that supports an improved object model (with a full mixin mechanism called traits; similar to Ruby modules) and full support for functional programming, following a design goal of the inventor of Scala, Martin Odersky, that these two paradigms can be integrated, despite some surface incompatibilities. Odersky was involved in the design of Java generics (through earlier research languages) and he wrote the original version of the current javac. The name is a contraction of “scalable language”, but the first “a” is pronounced like “ah”, not long as in the word “hay”.

The syntax looks like a cross between Ruby (method definitions start with the def keyword) and Java (e.g., curly braces). Type inferencing and other syntactic conventions significantly reduce the “cluuter”, such as the number of explicit type declarations (“annotations”) compared to Java. Scala syntax is very succinct, sometimes even more so than Ruby! For more on Scala, see also my previous blog postings, part 1, part 2, part 3, and this related post on traits vs. aspects.

Interoperability

Scala’s has the most seamless interoperability with Java of any of the languages discussed here. This is due in part to Scala’s static typing and “closed” classes (as opposed to Ruby’s “open” classes). It is trivial to import and use Java classes, implement interfaces, etc.

Direct API calls from Java to Scala are also supported. The developer needs to know how the names of Scala methods are encoding in byte code. For example, Scala methods can have “operator” names, like ”+”. In the byte code, that name will be ”$plus”.

Object Model

Scala’s object model extends Java’s model with traits, which support flexble mixin composition. Traits behave like interfaces with implementations. The Scala object model provides other sophisticated features for building “scalable applications”.

New Ideas

Scala brings full support for functional programming to the JVM, including first-class function and closures. Other aspects of functional programming, like immutable variables and side-effect free functions, are encouraged by the language, but not mandated, as Scala is not a pure functional language. (Functional programming is very effective strategy for writing tread-safe programs, etc.) Scala’s Actor library is a port of Erlang’s Actor library, a message-based concurrency approach.

In my view, the Actor model is the best general-purpose approach to concurrency. There are times when multi-threaded code is needed for performance, but not for most concurrent applications. (Note: there are Actor libraries for Java, e.g., Kilim.)

Scala has very good support for building internal DSL’s, although it is not quite as good as Ruby’s features for this purpose. It has a combinator parser library that makes external DSL creation comparatively easy. Scala also offers some innovative API’s for XML processing and Swing development.

Stability

Scala is over 5 years old and it is very stable. The API and syntax continue to evolve, but no major, disruptive changes are expected. In fact, the structure of the language is such that almost all changes occur in libraries, not the language grammar.

There are some well-known production deployments, such as back-end services at twitter.

Performance

Scala provides comparable performance to Java, since it is very close “structurally” to Java code at the byte-code level, given the static typing and “closed” classes. Hence, Scala can exploit JVM optimizations that aren’t available to dynamically-typed languages.

However, Scala will also benefit from planned improvements to support dynamically-typed languages, such as tail-call optimizations (which Scala current does in the compiler.) Hence, Scala probably has marginally better performance than JRuby, in general. If true, Scala may be more appealing than JRuby as a general-purpose, systems language, where performance is critical.

Adoption

Scala is harder to learn and master than JRuby, because it is a more comprehensive language. It not only supports a sophisticated object model, but it also supports functional programming, type inferencing, etc. In my view, the extra effort will be rewarded with higher productivity. Also, because it is closer to Java than JRuby and Clojure, new users will be able to start using it quickly as a “better object-oriented Java”, while they continue to learn the more advanced features, like functional programming, that will accelerate their productivity over the long term.

Tool Support

Scala support in IDE’s still lags support for Java, but it is improving. IntelliJ, NetBeans, and Eclipse now support Scala with plugins. Maven and ant are widely used as the build tool for Scala applications. Several excellent TDD and BDD libraries are available.

Deployment

Scala applications are packaged and deployed just like Java applications, since Scala files are compiled to class files. A Scala runtime jar is also required.

Clojure

Of the three new JVM languages discussed here, Clojure is the least like Java, due to its Lisp syntax and innovative “programming model”. Yet it is also the most innovative and exciting new JVM language for many people. Clojure interoperates with Java code, but it emphasizes functional programming. Unlike the other languages, Clojure does not support object-oriented programming. Instead, it relies on mechanisms like multi-methods and macros to address design problems for which OOP is often used.

One exciting innovation in Clojure is support for software transactional memory, which uses a database-style transactional approach to concurrent modifications of in-memory, mutable state. STM is somewhat controversial. You can google for arguments about its practicality, etc. However, Clojure’s implementation appears to be successful.

Clojure also has other innovative ways of supporting “principled” modification of mutable data, while encouraging the use of immutable data. These features with STM are the basis of Clojure’s approach to robust concurrency.

Finally, Clojure implements several optimizations in the compiler that are important for functional programming, such as optimizing tail call recursion.

Disclaimer: I know less about Clojure than JRuby and Scala. While I have endeavored to get the facts right, there may be errors in the following analysis. Feedback is welcome.

Interoperability

Despite the Lisp syntax and functional-programming emphasis, Clojure interoperates with Java. Calling java from Clojure uses direct API calls, as for JRuby and Scala. Calling Clojure from Java is a more involved. You have to create Java proxies on the Clojure side to generate the byte code needed on the Java side. The idioms for doing this are straightforward, however.

Object Model

Clojure is not an object-oriented language. However, in order to interoperate with Java code, Clojure supports implementing interfaces and instantiating Java objects. Otherwise, Clojure offers a significant departure for develops well versed in object-oriented programming, but with little functional programming experience.

New Ideas

Clojure brings to the JVM full support for functional programming and popular Lisp concepts like macros, multi-methods, and powerful metaprogramming. It has innovative approaches to safe concurrency, including “principled” mechanisms for supporting mutable state, as discussed previously.

Clojure’s succinct syntax and built-in libraries make processing XML succinct and efficient. DSL creation is also supported using Lisp mechanisms, like macros.

Stability

Clojure is the newest of the three languages profiled here. Hence, it may be the most subject to change. However, given the nature of Lisps, it is more likely that changes will occur in libraries than the language itself. Stability in terms of bugs does not appear to be an issue.

Clojure also has the fewest known production deployments of the three languages. However, industry adoption is expected to happen rapidly.

Performance

Clojure supports type “hints” to assist in optimizing performance. The preliminary discussions I have seen suggest that Clojure offers very good performance.

Adoption

Clojure is more of a departure from Java than is Scala. It will require a motivated team that likes Lisp ;) However, such a team may learn Clojure faster than Scala, since Clojure is a simpler language, e.g., because it doesn’t have its own object model. Also, Lisps are well known for being simple languages, where the real learning comes in understanding how to use it effectively!

However, in my view, as for Scala, the extra learning effort will be rewarded with higher productivity.

Tool Support

As a new language, tool support is limited. Most Clojure developers use Emacs with its excellent Lisp support. Many Java tools can be used with Clojure.

Deployment

Clojure deployment appears to be as straightforward as for the other languages. A Clojure runtime jar is required.

Comparisons

Briefly, let’s review the points and compare the three languages.

Interoperability

All three languages make calling Java code straightforward. Scala interoperates most seamlessly. Scala code is easiest to invoke from Java code, using direct API calls, as long as you know how Scala encodes method names that have “illegal” characters (according to the JVM spec.). Calling JRuby and Clojure code from Java is more involved.

Therefore, if you expect to continue writing Java code that needs to make frequent API calls to the code in the new language, Scala will be a better choice.

Object Model

Scala is closest to Java’s object model. Ruby’s object model is superficially similar to Scala’s, but the dynamic nature of Ruby brings significant differences. Both extend Java’s object model with mixin composition through traits (Scala) or modules (Ruby), that act like interfaces with implementations.

Clojure is quite different, with an emphasis on functional programming and no direct support for object-oriented programming.

New Ideas

JRuby brings the productivity and power of a dynamically-typed language to the JVM, along with the drawbacks. It also brings some functional idioms.

Scala and Clojure bring full support for functional programming. Scala provides a complete Actor model of concurrency (as a library). Clojure brings software transactional memory and other innovations for writing robust concurrent applications. JRuby and Ruby don’t add anything specific for concurrency.

JRuby, like Ruby, is exceptionally good for writing internal DSL’s. Scala is also very good and Clojure benefits from Lisp’s support for DSL creation.

Stability

All the language implementations are of high quality. Scala is the most mature, but JRuby has the widest adoption in production.

Performance

Performance should be comparable for all, but JRuby and Clojure have to deal with some inefficiencies inherent to running dynamic languages on the JVM. Your mileage may vary, so please run realistic profiling experiments on sample implementations that are representative of your needs. Avoid “prematurely optimization” when choosing a new language. Often, team productivity and “time to market” are more important than raw performance.

Adoption

JRuby is the the easiest of the three languages to learn and adopt if you already have some Ruby or Ruby on Rails code in your environment.

Scala has the lowest barrier to adoption because it is the language that most resembles Java “philosophically” (static typing, emphasis on object-oriented programming, etc.). Adopters can start with Scala as a “better Java” and gradually learn the advanced features (mixin composition with traits and functional programming). Scala will appeal the most to teams that prefer statically-typed languages, yet want some of the benefits of dynamically-typed languages, like a succinct syntax.

However, Scala is the most complex of the three languages, while Clojure requires the biggest conceptual leap from Java.

Clojure will appeal to teams willing to explore more radical departures from what they are doing now, with potentially great payoffs!

Deployment

Deployment is easy with all three languages. Scala is most like Java, since you normally compile to class files (there is a limited interpreter mode). JRuby and Clojure code can be interpreted at runtime or compiled.

Summary and Conclusions

All three choices (or comparable substitutions from the list of other languages), will provide a Java team with a more modern language, yet fully leverage the existing investment in Java. Scala is the easiest incremental change. JRuby brings the vibrant Ruby world to the JVM. Clojure offers the most innovative departures from Java.

Hudson—A Very Quick Demo from unclebob on Vimeo.

It’s not so much a “but” as an “aha!”. (The punch line is at the end of this article, so don’t give up in the middle.)

BDD is a variation on TDD. Whereas in TDD we drive the development of a module by “first” stating the requirements as unit tests, in BDD we drive that development by first stating the requirements as, well, requirements. The form of those requirements is fairly rigid, allowing them to be interpreted by a tool that can execute them in a manner that is similar to unit tests.

For example,

  GIVEN an employee named Bob making $12 per hour.
  WHEN Bob works 40 hours in one week;
  THEN Bob will be paid $480 on Friday evening.

The Given/When/Then convention is central to the notion of BDD. It connects the human concept of cause and effect, to the software concept of input/process/output. With enough formality, a tool can be (and has been) written that interprets the intent of the requirement and then drives the system under test (SUT) to ensure that the requirement works as stated.

The argued benefit is that the language you use affects the way you think (See this. and so if you use a language closer to the way humans think about problems, you’ll get better thought processes and therefore better results.

To say this differently, the Given/When/Then convention stimulates better thought processes than the AssertEquals(expected, actual); convention.

But enough of the overview. This isn’t what I wanted to talk about. What struck me the other day was this…

The Given/When/Then syntax of BDD seemed eerily familiar when I first heard about it several years ago. It’s been tickling at the back of my brain since then. Something about that triplet was trying to resonate with something else in my brain.

Then yesterday I realized that Given/When/Then is very similar to If/And/Then; a convention that I have used for the last 20+ years to read state transition tables.

Consider my old standard state transition table: The Subway Turnstile:

Current State	Event	New State	Action
LOCKED	COIN	UNLOCKED	Unlock
LOCKED	PASS	LOCKED	Alarm
UNLOCKED	COIN	UNLOCKED	Thankyou
UNLOCKED	PASS	LOCKED	Lock

I read this as a set of If/And/Then sentences as follows:

• If we are in the LOCKED state, and we get a COIN event, then we go to the UNLOCKED state, and we invoke the Unlock action.
• If we are in the LOCKED state, and we get a PASS event, then we stay in the UNLOCKED state, and we invoke the Alarm action.
• etc.

This strange similarity caused me to realize that GIVEN/WHEN/THEN is simply a state transition, and that BDD is really just a way to describe a finite state machine. Clearly “GIVEN” is the current state of the system to be explored. “WHEN” describes an event or stimulus to that system. “THEN” describes the resulting state of the system. GIVEN/WHEN/THEN is nothing more than a description of a state transition, and the sum of all the GIVEN/WHEN/THEN statement is nothing more than a Finite State Machine.

Perhaps if I rephrase this you might see why I think this is a bit more than a ho-hum.

Some of the brightest minds in our industry, people like Dan North, Dave Astels, David Chelimsky, Aslak Hellesoy, and a host of others, have been pursuing the notion that if we use a better language to describe automated requirements, we will improve the way we think about those requirements, and therefore write better requirements. The better language that they have chosen and used for the last several years uses the Given/When/Then form which, as we have seen, is a description of a finite state machine. And so it all comes back to Turing. It all comes back to marks on a tape. We’ve decided that the best way to describe the requirements of a system is to describe it as a turing machine.

OK, perhaps I overdid the irony there. Clearly we don’t need to resort to marks on a tape. But still there is a grand irony in all this. The massive churning of neurons struggling with this problem over years and decades have reconnected the circle to the thoughts of that brave pioneer from the 1940s.

But enough of irony. Is this useful? I think it may be. You see, one of the great benefits of describing a problem as a Finite State Machine (FSM) is that you can complete the logic of the problem. That is, if you can enumerate the states and the events, then you know that the number of paths through the system is no larger than S * E. Or, rather, there are no more than S*E transitions from one state to another. More importantly, enumerating them is simply a matter of creating a transition for every combination of state and event.

One of the more persistent problems in BDD (and TDD for that matter) is knowing when you are done. That is, how do you know that you have written enough scenarios (tests). Perhaps there is some condition that you have forgotten to explore, some pathway through the system that you have not described.

This problem is precisely the kind of problem that FSMs are very good at resolving. If you can enumerate the states, and events, then you know the number of paths though the system. So if Given/When/Then statements are truly nothing more than state transitios, all we need to do is enumerate the number of GIVENs and the number of WHENs. The number of scenarios will simply be the product of the two.

To my knowledge, (which is clearly inadequate) this is not something we’ve ever tried before at the level of a business requirements document. But even if we have, the BDD mindset may make it easier to apply. Indeed, if we can formally enumerate all the Givens and Whens, then a tool could determine whether our requirements document has executed every path, and could find those paths that we had missed.

So, in conclusion, TDD has led us on an interesting path. TDD was adopted as a way to help us phrase low level requirements and drive the development of software based on those requirements. BDD, a variation of TDD, was created to help us think better about higher level requirements, and drive the development of systems using a language better than unit tests. But BDD is really a variation of Finite State Machine specifications, and FSMs can be shown, mathematically, to be complete. Therefore, we may have a way to conclusively demonstrate that our requirements are complete and consistent. (Apologies to Godel).

In the end, the BDDers may have been right that language improves the way we think about things. Certainly in my silly case, it was the language of BDD that resonated with the language of FSM.

Yeah, and I’ve got a bridge to sell you.

Scrum is not the problem. Scrum never was the problem. Scrum never will be the problem. The problem, dear craftsmen, is our own laziness.

It makes no sense to blame Scrum for the fact that we don’t write tests and don’t keep our code clean. We can’t blame scrum for technical debt. Technical debt was around long before there was scrum, and it’ll be around long after. No it’s not scrum that’s to blame. The culprits remain the same as they ever were: us.

Of course it’s true that a two day certification course is neither necessary nor sufficient to create a good software leader. It’s also true that the certificate you get for attending a CSM course is good for little more than showing that you paid to attend a two day CSM course. It’s also true that scrum leaves a lot to be desired when it comes to engineering practices. But it is neither the purpose of scrum nor of CSMs to make engineers out of us, or to instill the disciplines of craftsmanship within us. That’s our job!

Some have implied that if all those scrum teams had adopted XP instead of scrum, they wouldn’t be having all the technical debt they are seeing. BALDERDASH!

Let me be more precise. ASININE, INANE, ABSURDITY. BALONEY. DINGOES KIDNEYS.

Let me tell you all, here, now, and forevermore, it is quite possible to do XP badly. It’s easy to build technical debt while going through the motions of TDD. It’s a no brainer to create a wasteland of code with your pair partner. And, believe me, you can such a Simple Design that You Aint Gonna Maintain It. And I’m not speaking metaphorically.

Do you want to know the real secret behind writing good software? Do you want to know the process that will keep your code clean? Do you want the magic bullet, the secret sauce, the once and for all one and only truth?

OK, here it is. Are you ready? The secret is…

The secret is…

Do a good job.

Oh, yeah, and stop blaming everything (and everybody) else for your own laziness.

Now, it would be nice if the developers just pair-programmed with each other, ignoring their assigned component boundaries, but we’re not quite there yet. Also, there are other issues we are addressing, such as the granularity of feature definitions, etc., etc. Becoming truly agile will take time.

Given where we are, it’s just not feasible to estimate a single story point value for each cross-component user story, because the work for each component varies considerably. A particular story might be the equivalent of 1 point for the UI part, 8 points for the middle-tier part, 2 points for the database part, etc.

So, what we’re doing is treating the user story as an “umbrella”, with individual component stories underneath. We’re estimating and tracking the points for each component story. The total points for the user story is the sum of the component story points, plus any extra we decide is needed for the integration and final acceptance testing work.

This model allows us to track the work more closely, as long as we remember that component points mean nothing from the point of view of delivering customer value!

I prefer this approach to documenting tasks, because it keeps the focus on delivering value to the client of each story. For the component stories, the client will be another component.

Automated Acceptance Tests for Component Stories

Just as for user stories, we are defining automated acceptance tests for each component. We’re using JUnit for them, since we don’t need a customer-friendly specification format, like FitNesse or RSpec.

This is also a (sneaky…) way to get the developers from different components to pair together. Say for example that we have a component story for the midtier and the UI is the client. The UI developer and the midtier developer pair to produce the the acceptance criteria for the story.

For each component story, the pair of programmers produce the following:

1. JUnit tests that define the acceptance criteria for the component story.
2. One or more interfaces that will be used by the client of the component. They will also be implemented by concrete classes in the component.
3. A test double that passes the JUnit tests and allows the client to move forward while the component feature is being implemented.

In a sense, the “contract” of the component story is the interfaces, which specify the static structure, and the JUnit tests, which specify the dynamic behavior of the feature.

This model of pair-programming the component interface should solve the common, inefficient communication problems when component interactions need to be changed. You know the scenario; a client component developer or manager tells a server component developer or manager that a change is needed. A developer (probably a different one…) on the server component team makes up an interface, checks it into version control, and waits for feedback from the client team. Meanwhile, the server component developer starts implementing the changes.

A few days before the big drop to QA for final integration testing, the server component developer realizes that the interface is missing some essential features. At the same time, the client component developer finally gets around to using the new interface and discovers a different set of missing essential features. Hilarity ensues…

We’re just getting started with this approach, but so far it is proving to be an effective way to organize our work and to be more efficient.

This is a common symptom in teams that don’t have have good automated test coverage and don’t use continuous integration (CI). Fear is their lot in life. They’ll keep lots of little branches and only merge to integration when they’re ready for the “big bang” integration.

I spoke with a manager at the client site today who expressed frustration that no one really knows which branch they should be committing to and when they should merge to the integration branches.

In contrast, projects with good test coverage and CI do almost all their work on the integration branch. They have little fear, because any problems will get caught and fixed quickly. So, the first point of this post is:

Automated test coverage and CI drastically simplify your use of configuration management.

Here’s something else I’ve noticed, open source CM systems seem to focus on different priorities than commercial systems. Disclaimer: I know I’m generalizing a bit here.

The commercial systems tend to be really good at managing complex branching, with fancy GUI tools to help manage the big trees of branches, facilitate merges, etc. That seems to be their biggest selling feature, GUI’s to manage the complexity.

In contrast, most of the open-source CM systems have lower-tech GUI’s, if any, but the teams using them don’t seem to care that much. Usually, this is because these teams are also practicing TDD and CI, so they just don’t need the wizardry as much.

The open-source CM systems seem to be better at scalability and performance. Some are pioneering distributed CM, e.g., Git and Mercurial. Git, for example, was designed to manage a massive project called Linux. Maybe you’ve heard of it.

Distributed CM is not easy, but it’s a lot easier to do if you don’t need to worry as much about complex branch hierarchies.

Most of the commercial tools I’ve seen don’t scale well and some require way too much administration. My client is apparently the biggest user of their particular tool and the developers complain all the time about performance. This tool is not designed to scale horizontally. The only hope is use faster hardware. In this case, the vendor has focused on managing complexity. To be frank, even their GUI tool is an uninspired and slow Java fat client.

So the second point of this post is:

Avoid CM tools that encourage complexity. Pick the ones that scale.