End-to-end automated integration tests are a well-established way of catching these problems early on. The team I’m mentoring has set up its own continuous-integration (CI) server and the team is getting pretty good at writing acceptance tests using FitNesse. However, these tests only cover the components provided by the team, not the true end-to-end user stories. So, they are imperfect as both acceptance tests and integration tests. Our longer-term goal is to automate true end-to-end acceptance and integration tests, across all components and services.

In this particular case, the other team is following a waterfall-style of development, with big design up front. Therefore, my team needed to give them an interface to design against, before we were ready to actually implement the service.

There are a couple of problems with this approach. First, the two teams should really “pair” to work out the interface and behavior across their components. As I said, we’re just starting to go Agile, but my goal is to have virtual feature teams, where members of the required component teams come together as needed to implement features. This would help prevent the miscommunication of one team defining an interface and sharing it with another team through documentation, etc. Getting people to communicate face-to-face and to write code together would minimize miscommunication.

Second, defining a service interface without the implementation is risky, because it’s very likely you will miss important details. The best way to work out the details of the interface is to test drive it in some way.

This suggests another technique I want to introduce to the team. When defining an interface for external consumption, don’t just deliver the “static” interface (source files, documentation, etc.), also deliver working Mock Objects that the other team can test against. You should develop these mocks as you test drive the interface, even if you aren’t yet working on the full implementation (for schedule or other reasons).

The mocks encapsulate and enforce the behavioral contract of the interface. Design by Contract is a very effective way of thinking about interface design and implementing automated enforcement of it. Test-driven development mostly serves the same practical function, but thinking in “contractual” terms brings clarity to tests that is often missing in many of the tests I see.

Many developers already use mocks for components that don’t exist yet and find that the mocks help them design the interfaces to those components, even while the mocks are being used to test clients of the components.

Of course, there is no guarantee that the mocks faithfully represent the actual behavior, but they will minimize surprises. Whether you have mocks or not, there is no substitute for running automated integration tests on real components as soon as possible.

Recently, I’ve been working to establish another magic funnel. One of the first things we did was to find the person who formerly handed out work to the developers and made him a single point of contact. In our case this is not the scrum master, but is another trusted line-manager type. He has been given a number of people to work with on the Customer side of the house and also works with technical people.

We have tried to establish story feeds from all the various stakeholders. Developers, operations people, technical writers, customers, sales people, marketing people, inside security consultants, and others have been feeding their ideas in to our point-of-contact man. This part of the funnel is working fairly well.

The next thing we have tried to do is to match the feed of work to the rate at which work can be done. This has created a fair amount of back-pressure on the stakeholders (which I believe to be healthy).

We have also worked on making the bottom of the funnel narrow, meaning that our guy doesn’t scatter new work to the team, but feeds it through the scrum master who protects the team’s productivity and keeps the work flow and tasks visible on the status board. He makes sure that the team is not expected to “absorb” changes, but that adding two points of work to the iteration results in two points of unstarted work coming off. This also creates a healthy back-pressure.

As a pun on “in-flight”, I named another area of the board “flying standby”. This is for stories that aren’t important enough to swap a story off the board (or for stories that are displaced by more important ones). If the developers finish more work than they expected, there are stories that can be picked up even though they’re not a scheduled part of the iteration. Stakeholders are told that there is no guarantee of these stories being picked up at all in this iteration, but there is some small chance that it could happen if the team discovers that it has overestimated some other stories.

The bottom of the funnel is working pretty well.

What’s missing is the “magic” bit.

The group has been working to stabilize and improve a product that was developed by a tiger team of outsider contractors and handed over to the in-house team post-facto. The developers had to overcome many obstacles to come up to speed on the code, to learn the new programming languages and tools, and to try to keep the feature set moving forward. None of them had been involved in the original design, but it was now their product, and its problems were their problems.

At the time of this meeting, I was one part of a coaching team which had introduced a great many changes. We were trying to help the organization to build a “whole team” mentality that encompassed documentation, security, systems administration, QA, customer representatives, and developers alike. We’d attacked the problem of matrix management. We reorganized the seating floorplan. I think at some point we’d been an inconvenience to just about everybody. The developers were in the middle of their second two-week iteration.

On this occasion, the team was in the midst of a recurring production difficulty, and had been gathered into the CIO’s office to work through a top-20 list of problems to solve.

“Your attitude has been affecting other departments”, he said. “And I want to thank you for it.”

I guess I get hung up on words. Essentially, “velocity” is fine. It is the rate at which we’re “burning through” our stories. We use “yesterday’s weather” to set our level, and we are always seeking any sustainable way to increase our velocity. Why would we produce only 12 points when we could produce 20? As motivated developers, we aren’t yearning to accomplish less. But somehow this doesn’t tell the right story.

When our velocity is decreased, it always seems like a tragedy. We have all felt it when our 23-point iteration is followed by a 21-point iteration. We can justify it a lot of different ways, but it always feels tragic. Worse, hitting any kind of plateau seems frustrating.

You increase velocity by pushing harder. If you push harder, and the numbers come up, then you’ve successfully increased the velocity and that seems to be intuitively all there is to it.

I’ve reflected about this for a few years before I realized that I really like the term capacity much more. It is also accurately describing the rate at which we can truly complete stories. Since we use yesterday’s weather, we can even use the term “proven capacity” to describe that we’ve demonstrated the speed. But it also says some other things.

Intuitively it seems that we know that you don’t push your people beyond their capacity to get work done. Instead, you want to increase their capacity. When we talk about longer hours, it is still obvious that working people harder diminishes their capacity. Instead, we want to be working at capacity on the one hand, and increasing our capacity on the other.

Increasing the capacity sounds like the right problem. To improve capacity, you want fresh workers working hard, pairing and communicating, building and/or using better tools (or using tools more capably), using more powerful hardware, learning, and performing the kind of housekeeping that keeps the team productive. Life balance enters into it. Sufficient team size enters into it so you’re not so small you’re short-handed, yet not so plentiful you trip over each other.

I don’t suppose that anyone is going to rename standardized eXtreme Programming terms like “velocity” just because some guy from Indiana says to, but I wonder if maybe we can’t rephrase the problem in such a way that we are intuitively driven toward more reasonable solutions. Maybe we should explain that velocity is just capacity, and then work from that point onward.

For me, this is further evidence of why the practices of Extreme Programming (XP), especially Test-Driven Development (TDD), are so important. To a high degree, TDD ensures that the software is really working and that regressions are caught and fixed quickly. TDD also provides an unambiguous definition of “done”; do the automated tests pass? The feedback on which stories are really done tells the team its true velocity and hence how many story points will be finished by a given target date. Also, you spend far less unplanned and unpredictable time debugging problems.

Hence, productivity and schedule accuracy tend to be much better for XP teams. Over time, as the software grows in size and complexity, the automated test suite will keep delivering “dividends”, by continuing to catch regressions and by empowering the team to do the aggressive refactorings that are necessary to keep the software evolving to meet new requirements.

The SD Times article goes on to say that

... two-thirds of the responding managers indicated that a solution that reduced the time spent on resolving application problems would be of interest if it created significant efficiencies and improved quality.

The article concludes with a quote that automated testing is one solution. I agree. Just make sure it is the XP kind of automated testing.

In Extreme Programming (XP), a team’s velocity is defined as the number of story points completed in an iteration. We can track the velocity for each iteration and calculate the average velocity over several iterations. If we know how many story points comprise an application, we can even project when an application can be released . Velocity is a very useful measure of a team’s capabilities. I have found it to be a much more reliable (make that infinitely more reliable) number to be used for project planning than asking developers for time estimates based on requirements.

Because velocity correlates to a team’s productivity, there is a tendency for managers to try to “goose” the velocity. The technical term for this is a stretch goal. Just like the coach of a sport’s team trying to get the team to play better, I can imagine project managers saying at a team meeting at an iteration boundary, “Come on team – you did 22 points last week, what do you say we try for 25 this week?” This same manager probably buys those motivation posters of eagles soaring and hangs them all around the team room.

The programming team takes pride in their achievements and they also like to see their velocity increasing. They will try to persuade the Customer to give them points for any bit of work they do. I can imagine a developer putting in a few minutes work to change the position of a GUI button on a screen and begging the Customer, “Can I get a point for that?”

XP teams can become obsessed with their velocity, but increasing velocity isn’t always a good thing. A team can inadvertently become so focused on getting points done that they sometimes forget the real goal – to deliver a quality product.

This became apparent to me recently when I visited a client to see how they were doing in their transition to XP. Their velocity had skyrocketed since I last saw them – and it was definitely not a good thing.

When Object Mentor does an XP transition with a client, we start with our XP Immersion course to get everybody on the same page about what our goals are. Ideally, we use two instructors, one to train the programmers in topics such as Test Driven Development and Refactoring, and the other coach teaches story writing, iteration planning, and release planning to the customer team. We then bring the two groups together for a day and have the customer team drive the programming team on a classroom exercise so everybody can experience how the process works. The instructors then stay and work with the team for a few iterations, coaching them on how to use what they learned in class on their own project.

Such was the case with the client I mentioned earlier. I was working with the programmers and the other coach had the customer team. When it came time to estimate stories, the other coach suggested using an open-ended scale. Where I like to use a scale of 1-5, this team was using a 1-infinity scale. That’s fine, it doesn’t really matter which scale you use, as long as whatever scale you choose is consistent. Stories were estimated according to their relative complexities, iterations were planned, and the programmers started cranking out code. After a few iterations the team’s velocity settled to a value of about 14 points. The team was doing fine and it was time for me to move on and let them do it alone.

When I returned to do a process checkup, their velocity had climbed to 48 points. Wow. This new process must really be resonating with the team. I purposely timed my visit to occur on an iteration boundary and we conducted a retrospection on how well the team was adhering the the XP practices. This turned out to be the bad news.

With a focus on getting more points done, it seemed that the team had abandoned many of the practices. Programmers weren’t pairing, weren’t writing unit tests, and weren’t refactoring their code. Customers were trying to stay ahead of the programmer’s ever increasing productivity abandoned writing automated acceptance tests in FitNesse and were now back to manual testing. I was heartbroken.

Beck teaches that one of the things teams have to do is to adapt the practices to the organization. Perhaps what they were doing was adapting. Adapting to the point that they weren’t even close to doing XP anymore, but adapting nonetheless. I wondered how that was working for them.

I observed the iteration planning meeting for the upcoming iteration and I noticed that all of the user stories were bug fixes from the previous iteration. No new functionality was being added to the application, the team was essentially spinning its wheels. So even though the velocity indicated a very productive team, the actual productivity was essentially zero. There must be a lesson here.

Velocity is what it is. You must take great care when encouraging a team to increase its velocity because you will always get whatever you’re trying to maximize. You have to be careful that you are maximizing the right thing. Want to maximize number of lines of code produced? Well you’re most likely going to get a lot of really crappy code. Want to maximize the number of unit tests that developers write? You’ll get a lot of them, but they’ll probably be worthless. Applying pressure to increase the velocity might make developers to subconsciously inflate their estimates or worse, abandon good development practices in order to get the numbers up.

I had this UML class diagram projected on the screen:

Employee
+ calculatePay()
+ save()
+ printReport()

I asked the class, “How many responsibilities does this class have?” Those students who had the courage to answer a question out loud (sadly, a rare subset of students) all mumbled, “Three.” I guess that makes sense, three methods means three responsibilities. My friend and colleague Dean Wampler would call this a first-order answer (he’s a physicist and that’s how he talks ;-) ). The number will increase as we dig into details. I held one finger in the air and said, “It knows the business rules for how to calculate its pay.” I put up a second finger and said, “It knows how to save its fields to some persistence store.”

“I happen to know that you folks use JDBC to talk to your Oracle database.” Another finger for, “It knows the SQL to save its data.” Another finger for, “It knows how to establish a database connection.” My entire hand is held up for, “It knows how to map its fields to SQLStatement parameters.” I start working on my other hand with, “It knows the content of the report.” Another finger for, “It knows the formatting of the report.” If this example was a real class from this company’s code base I knew I’d be taking off my shoes and socks.

Not that my answer was any better than the students’ answer, given the information at hand there can’t be a right answer because I didn’t provide any context for my question. I found our different answers interesting though. This particular company would have a design phase for a project where UML diagrams would be produced and discussed. How can any reviewer know if a class diagram is “right”?

I have this belief (hang-up?) that you don’t really know what you don’t know, until you take the next step and actually use what you currently have. Looking at UML, we can’t really say that it’s going to work or that it satisfies the Principles of Object Oriented Design, until we take the next step, i.e., write some code and see whether it works or not. The devil is in the details and those devilish dirty details just can’t be seen in a picture.

Let’s take a step back. Before there is UML specifying a design, there must have been requirements stating a problem that the design addresses. Have we captured all of the requirements? Are they complete? Are they accurate? Are they unambiguous? How do you know? I believe that you don’t know, and worse, you don’t even know what you don’t know. You don’t know until you take that next step and try to design a solution. It’s only during design that phrases like, “What’s supposed to happen here,” or, “That doesn’t seem to have been addressed in the spec,” are heard. You don’t know what you don’t know until you take that next step.

It is very easy for everybody on a project to believe that they are doing good work and that the project is going according to plan. If we don’t know what we don’t know, it’s hard to know if we’re on the right track. During requirements gathering, business analysts can crank out user stories, use cases, functional requirements, or whatever artifacts the process du jour dictates they produce. Meetings can be scheduled and documents can be approved once every last detail has been captured, discussed to death, and revised to everybody’s liking. Unfortunately, until solutions for these requirements are designed, they are but a dream. There is no way to predict how long implementation will take so project plans are really interpretations of dreams.

The same danger exists during design. Architects can be cranking out UML class diagrams, sequence diagrams, and state transition diagrams. Abstractions are captured, Design Patterns are applied, and the size of the project documentation archive grows. Good work must be happening. But are the abstractions “right”? Can the code be made to do what our diagrams require? Is the design flexible, maintainable, extensible, testable (add a few more of your favorite -able’s)? You just don’t know.

The problem with Waterfall, or at least the problem with the way most companies implement it, is that there either isn’t a feedback mechanism or the feedback loop is way too long. Projects are divided into phases and people feel that they aren’t allowed to take that crucial next step because the next step isn’t scheduled until next quarter on somebody’s PERT chart. If you don’t know what you don’t know until you use what you currently have, and the process doesn’t allow you to take the next step (more likely somebody’s interpretation of the process doesn’t let you take the next step), then we don’t have a very efficient process in place.

In order to not delude myself that I am on the right track when in reality I’m heading down a blind alley, I would like to know the error of my ways as quickly as possible. Rapid feedback is a characteristic of all of the Agile methodologies. By learning what’s missing or wrong with what I’m currently doing, I can make corrections before too much time is wasted going in the wrong direction. A short feedback loop minimizes the amount of work that I must throw away and do again.

I’m currently working with a client who wants to adopt Extreme Programming (XP) as their development methodology. What makes this difficult is that they are enhancing legacy code and the project members are geographically distributed. The legacy code aspect means that we have to figure out how we’re going to write tests and hook them into the existing code. The fact that we’re not all sitting in the same room means that we have to have more written documentation. We don’t know the nature of the test points in the system, nor do we know what to document and how much detail to provide in documentation. We can’t rely mostly on verbal communication but we don’t want to go back to writing detailed functional specs either. There are many unknowns and what makes this relevant to this discussion is, we don’t even know all of the unknowns. Rapid feedback has to be our modus operandi.

An XP project is driven by User Stories developed by the Customer Team, composed of Business Analysts and QA people. A User Story, by definition, must be sized so that the developers can complete it within an iteration. I have a sense how much work that I personally can do in a two week iteration, but I’m not the one doing the work and I don’t know how much of an obstacle the existing code is going to be. The Customer Team could go off and blindly write stories, but that would lead to a high probability that we’d have to rewrite, rework, split, and join once the developers saw the stories and gave us their estimates. To minimize the amount of rework, I suggested that the Customer Team write about twenty or so stories and then meet with the developers to go over the stories and allow them to give estimates.

My plan to get a feedback loop going on user story development worked quite well. The Customer Team actually erred on the side of stories that were too small. The developers wanted to put estimates of 0.5 on some of the stories (I have my teams estimate on a scale of 1-5. I’ll write up a blog entry on the estimation process I’ve been using.) so we combined a few of them and rewrote others to tackle a larger scope. We took the next step in our process, learned what we didn’t know, took corrective action, and moved forward.

Writing Customer Acceptance Tests didn’t go quite as smoothly, but it is yet another example of learning what didn’t work and making corrections. I advise my clients to write tests for their business logic and to specifically not write business tests through the user interface. Well guess where a lot of business logic resided – yep, in the user interface. We ran into the situation where the acceptance tests passed, but when the application was actually used through the user interface, it failed to satisfy the requirements. I’m very happy that we had a short feedback loop in place that revealed that our tests weren’t really testing anything interesting before the Customer Team had written too many FitNesse tests and the Development Team had written too many test fixtures.

Feedback is good. Rapid feedback is better. Seek it whenever you can. Learn what you don’t know, make corrections, and proceed.