Why don’t we clean our code? When we see an ugly mass of code that we know is going to cause of problems, our first reaction is “This needs to be cleaned up.” Our second reaction is: “If I touch this code I’ll be spending the next two weeks trying to get it to work again.” We don’t clean code because we are afraid we’ll break it.

In this way bad code hooks itself into our systems and refuses to go away. Nothing stops clean code from going bad, but once it’s bad, we seldom have the time, energy, or nerve to clean it. In that sense, bad code is permanent.

What if you had a button? If you push this button a little light instantly turns red or green. Green means your system works. Red means it’s broken. If you had a button like this, you could make small changes to the system, and prove that they didn’t break anything. If you saw a batch of tangled messy code, you could start to clean it. You’d simply make a tiny improvement and then push the button. If the light was green you’d make the next tiny change, and the next, and the next.

I have a button like that! It’s called a test suite. I can run it any time I like, and within seconds it tells me, with very high certainty, that my system works.

Of course, to be effective, that test suite has to cover all the code in the system. And, indeed, that’s what I have. My code coverage tools tell me that 89% of the 45,000 lines in my system are covered by my test suite. (89% is pretty good number given that my coverage tool counts un-executable lines like interfaces.)

Can I be absolutely sure that the green light means that the system works? Certainly not. But given the coverage of my test suite, I am reasonably sure that the changes I make are not introducing any bugs. And that reasonable surety is enough for me to be fearless about making changes.

This fearlessness is something that needs to be experienced to understand. I feel no reluctance at all about cleaning up the code in my system. I frequently take whole classes and restructure them. I change the names of classes, variables, and functions on a whim. I extract super-classes and derivatives any time I like.

In short, the test suite makes it easy to make changes to my code. It makes my code flexible and easy to maintain.

So does my architecture of course. The design and structure of my system is very easy to deal with, and allows me to make changes without undue impact. The reason my architecture is so friendly, is that I’ve been fearless about changing it. And that’s because I have a test suite that runs in seconds, and that I trust!

So the clean architecture of my system is a result of on-going efforts supported by the test suite. I can keep the architecture clean and relevant because I have tests. I can improve the architecture when I see a better approach because I have tests. It is the tests that enable architectural improvement.

Yes, architecture enables flexibility, maintainability, and reusability; but test suites enable architecture. Architecture is a second order effect.

It’s true that the two streams of tests test the same things. Indeed, that’s the point. Unit tests are written by programers to ensure that the code does what they intend it to do. Acceptance tests are written by business people (and QA) to make sure the code does what they intend it to do. The two together make sure that the business people and programmers intend the same thing.

Of course there’s also a difference in level. Unit tests reach deep into the code and test independent units. Indeed, programmers must go to great lengths to decouple the components of the system in order to test them independently. Therefore unit tests seldom exercise large integrated chunks of the system.

Acceptance tests, on the other hand, operate on much larger integrated chunks of the system. They typically drive the system from it’s inputs (or a point very close to it’s inputs) and verify operation from it’s outputs (or again, very close to it’s outputs). So, though the acceptance tests may be testing the same things as the unit tests, the execution pathways are very different.

Process

Acceptance tests should be written at the start of each iteration. QA and Business analysts should take the stories chosen during the planning meeting, and turn them into automated acceptance tests written in FitNesse, or Selenium or some other appropriate automation tool.

The first few acceptance tests should arrive within a day of the planning meeting. More should arrive each day thereafter. They should all be complete by the midpoint of the iteration. If they aren’t, then some developers should change hats and help the business people finish writing the acceptance tests.

Using developers in this way is an automatic safety valve. If it happens too often, then we need to add more QA or BA resources. If it never happens, we may need to add more programmers.

Programmers use the acceptance tests as requirements. They read those tests to find out what their stories are really supposed to do.

Programmers start a story by executing the acceptance tests for that story, and noting what fails. Then they write unit tests that force them to write the code that will make some small portion of the acceptance tests pass. They keep running the acceptance tests to see how much of their story is working, and they keep adding unit tests and production code until all the acceptance tests pass.

At the end of the iteration all the acceptance tests (and unit tests) are passing. There is nothing left for QA to do. There is no hand-off to QA to make sure the system does what it is supposed to. The acceptance tests already prove that the system is working.

This does not mean that QA does not put their hands on the keyboards and their eyes on the screen. They do! But they don’t follow manual test scripts! Rather, they perform exploratory testing. They get creative. They do what QA people are really good at—they find new and interesting ways to break the system. They uncover unspecified, or under-specified areas of the system.

ASIDE: Manual testing is immoral. Not only is it high stress, tedious, and error prone; it’s just wrong to turn humans into machines. If you can write a script for a test procedure, then you can write a program to execute that procedure. That program will be cheaper, faster, and more accurate than a human, and will free the human to do what humans to best: create!

So, in short, the business specifies the system with automated acceptance tests. Programmers run those tests to see what unit tests need to be written. The unit tests force them to write production code that passes both tests. In the end, all the tests pass. In the middle of the iteration, QA changes from writing automated tests, to exploratory testing.

No.

Let me restate that more clearly for those of you who may still have doubt.

No frickin way Jose!

Most software systems are a mess. SOA is not the cure for that mess. Indeed, SOA doesn’t even start to be feasible until the mess has been cleaned up.

Again, let me restate that.

SOA does not help you clean up the mess that your software is in. In order to adopt SOA you must first clean up the mess you have made. Once the mess is clean, then you can start to think about SOA.

Right now, at this very moment, some IT manager is reading about how the next release of some Enterprise Service Bus is going to solve all his woes. All he has to do is install this wonderful piece of software, tie it into the new whippy-diddle business process modeling tool. Get some clerks to write the XML files that describe their business processes, get some other clerks to write the message translators that drive that great interoperability Babel Fish in the sky. Get some business process modelers to draw pretty pictures of all the interconnections between the systems. Feed that all into the ESB, and shake vigorously with deadlines and threats. This will cause fuzzy bunnies to hop happily over the green fields of his IT systems.

Right.

The reality is that the first letter of SOA, namely “S”, stands for “Service”. Not to put to fine a point on it, SOA means an Architecture Oriented around Services. What is a service? A Service is a piece of software with the characteristics of a service? What are those characteristics? They are: Low coupling, independently deployability, stable interface, etc.

Most software professionals (if that term is not oxymoronic) will recognize this set of attributes as being identical with that of the “Components” of the 90s. Indeed, Services are the new Components. The words are synonyms.

This means that SOA is just as hard to do as COA. (Component Oriented Architecture for those of you who are acronym challenged). Indeed, SOA is COA!

Now, I know it’s been a long time. But does anybody remember what it took to do COA? No? Then let me tell you a little story to jog your memory.

In the mid ‘90s I had a client who had a 250MB executable written in C++. (Yes, I know this seems like a hello world program today, but back then 250MB was a big program). This client was vexed because every time he changed even a single line of code, he had to rebuild the entire executable, burn it onto a CD, and ship the CDs to his installed base. This was costing him lots of time and money.

What he wanted was a way to break his application into components and then ship only those components that changed. His hope was that he could ship them over the internet because they’d all be small enough to email or FTP. (This was back when people did not regularly download 4GB movies, or backup 100GB disks over the internet every night).

So he adopted COA, and the popular implementation platform of for COA, COM+ (Otherwise known as void**: Nothing to the power of nothing.) He slaved away for months breaking his application up into components that could be dynamically loaded in DLLs. And in the end, he succeeded. His application was completely—componentized.

However, there was one little snag. He forgot to decouple the components from one another. Each component depended on several others. There were cycles in the dependency graph of components. (Aside to the architecturally challenged: “That’s bad.”) So in the end, when he changed a line of code in one component, he had to rebuild and redeploy all the components. He had to burn all those components onto a CD.

In short, nothing had changed.

Long long ago, I watched a software developer write complicated macro in his highly powerful text editor. It converted all tabs to spaces. I remember watching him start it. The code he was transforming was on the screen. He ran the macro, which took several minutes to run. It displayed a series of dots running across the screen. It was very impressive, and looked like it was doing something significant. And then when it finished, the code was redisplayed on the screen. I laughed, because after all that work of translating tabs into spaces, and putting dots on the screen, the displayed code looked exactly the same as when he started. It was as if nothing had happened.

My client’s reaction to his inability to independently deploy his components had nothing to do with laughter.

The point? SOA will not cut the Gordian Knot. Indeed, it will pull that Knot even tighter. The Knot can’t be cut. There is no magic that will make the mess go away, or make the mess less of an issue. If you want the flexibility of SOA, then you must clean up the mess you have and then slowly and gradually build services from those cleaned up systems.

The real point? TANSTAASB. (There ain’t no such thing as a Silver Bullet).

A project with 10 different developers cannot withstand 10 different styles of coding, 10 different architectural visions, 10 different philosophies of OO design, or 10 different software development methods. To prevent the chaos of anarchy, one style, one vision, one philosophy and one method must be adopted.

The problem is, we know that no one style, vision, philosophy, or method is “right”. This knowledge makes us timid. We tolerate other people’s styles and attitudes rather than insisting on a single style and vision for the project. We feel that we have no right to insist on one particular way of doing things, and so we adopt a politically correct viewpoint regarding software.

Politically correct projects are muddles. They find it difficult to get traction because the team members are all pulling in different directions. This muddling continues until someone puts a stake in the ground and says “We’re doing it this way”, and makes it stick by sheer force of will. Then the team moves on and makes progress.

Was his way right? The question is irrelevant. The stake in the ground provided focus. It eliminated the chaos of too much “rightness”, of too much political correctness.

There are many different schools of martial arts. Karate, Tai Kwon Do, Jiu Jitsu, Judo, etc. Which is right? Of course the question is absurd. Right and wrong are not adjectives that apply to individual martial arts. And yet, each martial art attracts dedicated adherents. The intensity of their devotion to their art is almost religious.

Students of Karate have no rational reason to prefer it to Tai Kwon Do. Indeed, the initial decision to study one vs the other was almost certainly arbitrary. Perhaps there was a Karate studio near their home. Perhaps a friend invited them to a session. Perhaps they saw something on TV, or in the paper. In most cases the initial decision was trivial.

But that’s where the triviality ends. From then on the new student gets immersed into a set of styles, disciplines, philosophies, and methods to the exclusion of all others. There is no political correctness at this level. Teachers tolerate no dissent from their teachings. It’s the Sen Sei’s way, or the highway. Within a dojo, there is a right and a wrong.

At a larger level, everyone knows that no one martial art is clearly better than all the others. There is no “right” school of martial art. And yet, students invest huge efforts into becoming masters of one particular arbitrary discipline.

Why would a student focus on one particular school of martial art to the exclusion of all others for years of his or her life? Simple. To become a master. You can’t become a master of all martial arts at the same time. You have to separate and focus. You have to dedicate yourself to one and learn it inside and out. You have to be politically incorrect. That’s how you gain mastery.

Once a degree of mastery is attained, it becomes beneficial to study other martial arts. A karate black belt will find it beneficial to study Jiu Jitsu, or Akido. The Judo black belt will gain insights into his own art by studying Karate. So, at the level of the master, political correctness and tolerance is re-established.

There are many diverse attitudes about software. For example there are many different kinds of agile methods. Indeed, there are methods that aren’t agile at all. There’s the American vs the European style of OO. There’s functional vs OO vs procedural programming. There are those who practice hungarian notation. There are those who practice test driven development. There are those who believe in C and C++ and those who believe in Ruby. None of these attitudes and preferences are right. None of them are wrong. But without some kind of organizing principle, like a dojo, or a Sen Sei, they are chaos.

There has been a lot of talk about certification lately. Companies need to know if the people they hire are good developers or not. The problem is, there is no standard of certification. There is no right or wrong, and so there’s little to certify. We can certify that someone took a particular course, or that they have certain knowledge of C++ or Java. We can measure someone’s ability to read a book, or write a dumb example program. What we cannot do is certify whether someone is a good developer or not, because we don’t have a definition of “good”.

The different schools of martial arts have solved this problem of certification quite effectively. No school will certify that their student is a master martial artist. But they will certify that they are a master of one particular discipline. You cannot be a general black-belt, but you can be a black-belt in one particular school of Karate.

The award of rank is not based on a casual test, or the taking of a two day course. It is based on months of exposure and participation in practice sessions. It is based on the teacher’s intimate knowledge of the student’s skill. What’s more the teacher’s right to convey rank is based on his teacher’s authority. Thus, there is a chain of trust from teacher to teacher.

I don’t know if a martial-arts ranking system could ever be established in software. The problems seem significant. For example, how would teachers gain the kind of prolonged access they’d need to bestow ranks on students?

However, I do think it’s possible for experienced practitioners to put a stake in the ground and establish a school of software discipline. In effect the practitioner says: “I’ve done it my way, and it worked for me. I’ll teach you my way, perhaps it will work for you.”

Such a school would be precise and exclusive. Students would adopt a certain coding style, a certain philosophy of design, a certain architectural vision, and a certain set of methods and practices. They would adhere to those certain teachings to the exclusion of all others until they mastered them.

Thirteen years ago, while working on my first book, Jim Coplien and I were having a debate on the nature of threads and objects. He made a clarifying statement that has stuck with me since. He said: “An object is an abstraction of function. A thread is an abstraction of schedule.”

It has become the norm, in Java applications, to assume that there is a one-to-one correspondence between a thread, and a unit-of-work. This appears to make sense since every Servlet request has it’s own particular thread. Framework authors have built on this assumption by putting unit-of-work related information (e.g. session) into ThreadLocal variables.

Clearly, the more ThreadLocal variables that hold unit-of-work related information, the more that the thread and the unit-of-work are related. While very convenient, the basic assumption is dead wrong.

A unit-of-work is a task to be performed. There is no rule that says that this task must be single threaded. Nor is there any rule that says that the components of the task must run at the same priority. Indeed, it is not uncommon for a task to have a high-priority event driven part, and a low priority compute-bound part.

Consider, for example, a unit-of-work that makes a request from a third-party-service, and then processes the response with a complex number-crunching algorithm. We want the request to run at a high priority so that it is not waiting for other low priority compute-bound tasks to finish. On the other hand, we want the number-cruncher to run at a low priority so that it yields the processor as soon as possible whenever one of the high priority requests needs a few cycles.

Clearly these two tasks form a complete unit-of-work, but just as clearly they should run as two different threads—probably with a queue in between them in a standard producer-consumer model.

Where are the unit-of-work related variables? They can’t be kept in a ThreadLocal since each part of the task runs in a separate thread. They can’t be kept in static variables since there is more than one thread. The answer is that they have to be passed around between the threads as function arguments on the stack, and recorded in the data structured placed on the queue.

So, though convenient, ThreadLocal variables confuse the issue of separating function from schedule. They tempt us to couple function and schedule together. This is unfortunate since the correspondence of function and schedule is weak and accidental.

What we’d really like is to be able to create UnitOfWorkLocal variables.

In the spring of 2000 Kent Beck called a meeting at the Rogue River Lodge near his home in Medford, Oregon. He called it the Extreme Programming Leadership Conference. In attendance were: myself, Ron Jeffries, Ken Auer, Martin Fowler, and several others who had been instrumental in getting the XP movement off the ground.

In one of the sessions we discussed the creation of an organization to drive and support the adoption of XP. This was an idea that I argued in favor of. It seemed to me that the industry would benefit from a body who promoted the ideas of lightweight methods like XP. However, the idea was not met with enthusiasm amongst many of the other attendees. The session turned into a collection of painful reminiscences about a previous attempt at creating an organization around the Patterns movement. I came away convinced that a different group of people would have to drive the creation of such an organization.

Apparently I was not alone. Martin Fowler had noticed the arguments I had made in the session, and approached me during a break. We briefly discussed the idea of broadening the scope of the proposed organization to all the various “lightweight methods” such as Scrum and Crystal Light. We felt that the folks who espoused these methods would have a lot to add, and could form the core of a comprehensive movement. Martin and I determined to meet a few weeks later to work out the details.

In the meantime I contacted my friend and advisor Dave Thomas of OTI fame. (We call him “Big Dave” to differentiate him from the Pragmatic Programmer of the same name.) I mentioned the notion of a Lightweight Methods Summit. We put a tentative invitation list together. Dave suggested that we meet in February at his winter home in Anguilla.

Later that fall I met with Martin at a cafe in Chicago. We added to the invitation list and outlined the invitation email. The invitation stated the goal of creating a manifesto describing what all the various lightweight methods had in common. I sent the email that evening.

The Lightweight Process Summit invitation drew an enthusiastic response from the invitees. Alistair Cockburn was especially interested. As it happened, he was on the verge of sending out his own invitation for a similar event. His comment to me was: “Your invitation list was better than mine”. The two invitation lists were merged, and the discussion of logistics began.

Alistair became the de-facto organizer. He suggested that we change the venue from Anquilla to Snowbird in Salt Lake City. Almost all the other invitees agreed because the flights would be easier. Alistair and Jim Highsmith did the leg-work to arrange the rooms, meals, and activities. Things came together rapidly.

The meeting was very well attended by 17 out of ~20 invitees. (See www.agilemanifesto.org to see the list of attendees.) Unfortunately neither Grady Booch nor Big Dave Thomas were able to make it. However, their influence was strongly felt in the subsequent discussions.

I kicked off the meeting with a brief introduction and an appeal to the similarities between all the various lightweight processes. I reaffirmed that the goal of our two day meeting was to create a manifesto that outlined those similarities. This was to be a document, addressed to the industry at large, describing our views about the process of creating useful software.

Martin Fowler and Ward Cunningham quickly became the de-facto facilitators of the meeting. With their help we quickly worked out an agenda for the two days, and a method for making decisions.

It was actually quite a thrill to see all these people with disparate ideas working together so well. I don’t think I’ve ever been to a meeting that stayed on point and met it’s goals with such ease and with so little conflict. It was as though the pieces just somehow fell together.

One of the early discussions was about a name. No one liked the term Lightweight. Several other options were suggested, including Lean, and Adaptive. But the name Agile won the day.

The structure of the manifesto was mutually agreed; I recall that Ward played a big part framing the idea of pairs of relative value, but it may have been Martin and Pragmatic Dave Thomas who came up with the idea. Here’s PragDave’s recollection:

Actually, that was Martin and me, noodling on the whiteboard over lunch. I think we came up with the initial three. The group that then formed took it up to five, and that was then pruned down to four. In fact, Ward took the picture of everyone discussing the idea that now adorns his manifesto site.

By the end of the two days in Snowbird, the manifesto was done, the principles were outlined, and the Agile Alliance had been born. Much more would need to be done before there would be a true legal entity with a board and members, but that’s a story for another time.

For another version of this history see Jim Highsmith’s description at http://www.agilemanifesto.org/history.html