How much test coverage should your code have? 80%? 90%? If you’ve been writing tests from the beginning of your project, you probably have a percentage that hovers around 90%, but what about the typical project? The project which was started years ago, and contains hundreds of thousands of lines of code? Or millions of lines of code? What can we expect from it?

One of the things that I know is that in these code bases, one could spend one’s entire working life writing tests without doing anything else. There’s simply that much untested code. It’s better to write tests for the new code that you write and write tests for existing code you have to change, at the time you have to change it. Over time, you get more coverage, but your coverage percentage isn’t a goal. The goal is to make your changes safely. In a large existing code base, you may never get more than 20% coverage over its lifetime.

Changes occur in clusters in applications. There’s some code that you will simply never change and there’s other areas of code which change quite often. The other day it occurred to me that we could use that fact to arrive at a better metric, one that is a bit less disheartening and also gives us a sense of our true progress.

The metric I’m thinking about is percentage of commits on files which are covered by tests relative to the number of commits on files without tests.

In the beginning, you can expect to have a very low percentage, but as you start to get tests in place for changes that you make, your percentage will rise rapidly. If you write tests for all of your changes, it will continue to rise. At a certain point, you may want to track only a window of commits, say, the commits which have happened only in the last year. When you do this, you can end up very close to 100%.

If you think this through, it might seem a bit dodgy on a couple of fronts. The first is that having tests for code within a file does not mean that that code is completely covered by those tests. But, I often find that the hardest part of getting started with unit testing is getting classes isolated enough from their dependencies to be testable in a harness at all. Another dodgy bit is the fact that once you get some tests in place for a file, all of the commits you’ve ever done for that file count in the percentage. Again, that’s okay for fundamentally the same reason. Once you start getting coverage, you are in a good position with that particular code.

What about the moving window? If you track this metric over, say, the last N months of commits, you’ll progressively lose information about the code that you just aren’t changing. To me, that’s fine. Coverage matters for the code that we are changing.

The metric I’m considering (maybe we can call it ‘change coverage’) gives us information about how tests are really impacting our day to day work. Moreover, it’s likely that it would be a good motivational tool, and really, that’s one of the things that a good metric should be.

Every once in a while in life, you get a lucky break. My lucky break was right as I was graduating from college. One of my favorite professors told the class that there were some people interviewing potential hires on campus and they were looking for a programmer. I remember standing dutifully outside his office, in line, waiting for my little mini-interview. When it was my turn, they didn’t see me for a little while – the door stayed closed after they let the last interviewee out. I found out later, that my professor was telling the interviewers that I was the guy that they wanted.

I came in, and showed them some code I’d written outside of schoolwork, and we talked about it. I told them about my passion for programming languages and they said that the job would be at a research department and they needed someone to design and implement a new programming language. I was over the moon. It was my dream job. And, sure enough, it was real. When I was hired, I sat down with my boss and went over ideas for syntax with him, and he pretty much left me alone for a couple of months to design the language, and implement a compiler for it. I was sitting in my first job, with some real responsibility, struggling with YACC, designing a symbol table. It was all very heady.

In retrospect, I was extremely lucky. I not only had more individual trust and responsibility than many developers ever get in their career, I also became part of an extremely good working environment. The guys who hired me, David Lopez de Quintana, and Carlos Perez, also ran the team I was part of. We were a group of about four which eventually grew to about 12. Although, we weren’t working on one big integrated project, we saw ourselves as a team and we had a great cooperative culture. We each had our area of expertise, and most of us were in our 20s, ready to stretch. We tackled all sorts of interesting problems at the intersection of software, mechanics, electronics, optics, physics, chemistry and biology.

About six months into the job, I had an odd experience. Carlos called me to his office and told me he was going to give me my review. I was a bit shocked. The way I understood things, David was actually the guy I worked for, and Carlos headed up part of the team but not my part. Carlos gave me a big smile and said “Yeah, well, you’ve been working for me for the past few months, but we didn’t tell you.”

The review went great and I happily went back to work. I always knew that, on paper we were two different teams, but Dave and Carlos treated us as one big team. We had one big meeting every week or so, and they shared responsibilities and pretty much kept us from having to care about anything outside the work we had to do.

It was a very cool experience. Sometimes, Dave would spend more time outside the team, sort of managing our interface with the rest of the organization. Part of the group wrote software to support research but we had customers outside of our group too. Dave would be off at meetings, and Carlos would handle the day-to-day stuff with our team. Sometimes they reversed the roles. Dave would handle us day to day and Carlos would manage the world outside. The thing is, no one was asking them to allocate responsibilities that way; they just found it a comfortable way of working. It was pair management. Sort of like pair-programming, but at the management level.

The funny thing is that I haven’t seen this much since. Yes, I’ve visited organizations with teams that do something close to this, but it seems like all sorts of forces are aligned against it. The thing which made it work for Dave and Carlos is that they were very good friends. They were hired at the same time in entry-level non-management positions and they rose through the ranks together. They saw each other outside work all the time and joked together at work, making it all a very rich experience. Importantly, their manager never seemed to put them in competition with each other. As I reflect on it more, I think that was the key. Managers are often in competition with each other and that can impede the sort of collaboration and culture that Dave and Carlos built up. They created a wonderful high-performing team. It was one of those once in a lifetime things, and it’s a time that I’ll always remember.

Whenever I think about team health and culture, that first team I joined is my benchmark. I don’t know where the magic came from, but it was there and I think that the culture that Dave and Carlos created with pair management was a very important part of it. In their interactions with each other, they provided a model for us, showing us how to treat each other at work.

It happens over and over again. I visit a team and I ask about their testing situation. We talk about unit tests, exploratory testing, the works. Then, I ask about automated end-to-end testing and they point at a machine in the corner. That poor machine has an installation of some highly-priced per seat testing tool (or an open source one, it doesn’t matter), and the chair in front of it is empty. We walk over, sweep the dust away from the keyboard, and load up the tool. Then, we glance through their set of test scripts and try to run them. The system falls over a couple of times and then they give me that sheepish grin and say “we tried.” I say, “don’t worry, everyone does.”

It’s a very familiar ‘rabbit hole’ in the industry. It’s sort of like the old days when you’d find a couple of classes generated by a CASE tool in every code base you visited if you looked hard enough. People started with the merry notion that they were going to round-trip code with some CASE tool and they learned (like most lucky teams do) that it just doesn’t pay for itself, it’s not worth the time or the frustration. UI Test Automation tools are in the same category. Personally, I think that in this day and age selling them is irresponsible. Developing them open-source? Well, let your conscience be your guide, but really, even though people can use them responsibly, they hardly ever do because these tools are sold with a dream, a very seductive and dangerous one.

The Dream

Janet comes into work in the morning and she sits down at her super-duper testing console. She presses a button and the testing system springs to life. The application comes up all at once across ten monitors. Cursors move, selections are made (silently) and tests run against the user interface magically, as if some eager set of ghost elves took control, mischievously burrowing through the nooks and crannies of the application, running scripts to completion, and making little notes whenever there is a failure. Janet sits back in her chair, waiting for the elves to report back to her. She stirs her coffee gently.

The Reality

Janet hasn’t gone home yet. It’s 2AM and she has to report completion of all her test cases at a meeting in the morning. She thinks she’s past the last configuration issue but she’s not sure. For the last hour, she’s been trying to make sure that a particular button is pressed at step 14 of her script, but quirky latency on the server is preventing it from happening consistently. Sadly, she has to run the script from the beginning each time. Oh, and five hours ago she discovered UI changes which invalidated 30% of the regression tests. Most of the changes were easy but she still has 12 cases to go and her 9AM meeting looms ahead of her.

This gap between the dream and the reality is not a matter of flawed execution, it’s endemic. Here’s the scoop.

UI Test Automation Tools are Brittle

You might not think this is fair but it is, really. I haven’t seen one of these tools yet which isn’t susceptible to missed events or timing madness. It just happens. The fact of the matter is, it is hard to sit on the outside of an application an instrument it. It’s a very technology sensitive problem. You need to hook into either the OS or the browser or both. Neither are ever really built from the ground up for that sort of access.

UI Based Testing Is Not the Solution That Vendors Imply It Is

This is the big issue, the one which really hurts the industry. The fact of the matter is that UI based testing should be used for UIs: that’s it. You should not be testing your full application end-to-end through a UI Testing tool. First of all, that sort of testing couples some of the most important tests in your system to one of the most volatile parts of it. It’s easy for us to see business logic as volatile, but really, the UI is the thing which twists and ripples in the winds of change. When customers want new features, often those features involve new workflows. When usability experts discover better ways of models of interaction, an agile business seizes upon them and makes the changes—if they can. You’d be surprised at the number of applications which continue to sport out of day user interfaces simply because the development organization is terrified of throwing away all of their regression tests which (by the way) go through the UI. Even if you’re not a consultant like me, visiting teams and seeing their development processes, you can see hints from the outside. Think of every website or shrink-wrap application which has “the same old workflow” and a UI that has become more cluttered over the years. Often it’s because of that lock-in.

UI Based Testing Takes More Staff and Time Than You Expect

This, really, is the most common failure case. It’s the case which explains the dust on the testing box’s keyboard. Someone, usually disconnected from the development organization, decides that “hey, we need to solve the testing problem. We have too many people doing manual testing. It’s taking forever.” So, they do their research, find a vendor with with a good licensing model and a good pitch and then they push it on the development organization. They are, of course, looking to reduce staff so when they realize that translating all of those manual tests to the tool is very labor-intensive, they are taken aback. But, of course, it is just a temporary cost, right? But, then it takes far longer than they expect. Remember Janet’s story? It’s really hard to catch up with a UI-Based testing tool. It’s hard to even stay in place with one. Typically it takes a number of people to do UI-Based automated testing for a development team in sync with an iteration and worse, they’ll always lag behind a bit because you can’t really write UI-based tests ahead of time the way you can with FIT and other beneath-the-UI testing tools. From what I’ve seen UI-based testing, done diligently, takes the effort of about one tester for every two to three developers. That’s what it seems to cost amortized across all of the maintenance of UI-induced test breakage. Oh, and by the way, if think you are going to save labor using record and playback? Nope, you aren’t. It doesn’t work.

Solutions

The fact of the matter is, you can use these tools effectively, but in a very narrow space. It’s nice to be able to test the UI—by itself. However, this sort of thing requires an architectural change.

In general, UIs are too volatile for end-to-end testing. Teams that do it well, typically develop a small task-focused scripting layer and build tests on top if it so that the actual tests don’t touch the UI directly. But, if they happen across that technique, they are lucky. Still, it isn’t an ideal solution. You really want to be below the UI working against an API which exposes the business logic. And, because of that nearly mystical synergy between testability and good design, that API layer is often useful for many things other than testing.

Conclusion/Challenge

I recognize that I’ve been rather vicious in the this blog. If you develop these tools for a living, you might not think it’s fair. But consider this. If you don’t think I’m being fair, take a look at how your tools are marketed. In particular, show me where the product literature discourages end-to-end testing through the tool. Otherwise, well, you know, you are probably developing snake-oil.

Over the past few days, I’ve been tinkering with a little project I’ve called vih – it’s a vi clone written in Haskell. Here’s the git repo.

The way I’ve started it is rather naive, and frankly I’m surprised I that I had gotten as far as I did without curses and with the simple data structure I choose. Rather than using a clever representation for a buffer, I decided to just use a String. What this means is that every time a character is inserted into a buffer, the whole thing is split and reconstituted. I’m not using Data.ByteString or Data.ByteString.Lazy either. I suppose that I’ll move toward a more sensible data structure in a while but right now typing is comfortable even with all of that work going on.

My intention in doing this was to start to get a sense of how data structures evolve in Haskell by growing something large from a small seed. The primary data structure that I’m using right now is called EditBuffer and I notice that quite a few of my functions translate an EditBuffer to another EditBuffer. This seems to work fine, however I find that I almost always have to label the buffer in the argument list and deconstruct it. I haven’t seen pervasive use of labels like that in Haskell code I’ve looked at, so I’m wondering if my mapping of functions to data structures is odd.

Right now, vih supports insert and command mode (toggled with ‘i’ and ‘ESC’), the h, j, k, l cursor movements, line deletion with ‘dd’, home with ‘gg’, end of file with ‘G’, `in line’ positioning with ‘0’ and ’$’, insert new line after with ‘o’, and delete current char with ‘x’. No file I/O yet.

Have a look, fork or contribute.

I’m off on a round of conferences, classes and talks soon, so I figured I’d list them here.

I wrote a blog the other day about functional refactoring and I had what I thought was a good example:

absPosition :: Buffer -> Int
absPosition (Buffer (x, y) contents) = x + lenPreviousLines contents
  where lenPreviousLines = 
    foldr ((+).length) 0 . take y . terminatedLines

absPosition (Buffer (x, y) contents) = x + lenPreviousLines contents
  where lenPreviousLines = 
    sum . map length . take y . terminatedLines

absPosition (Buffer (x, y) contents) = x + lenPreviousLines contents
  where lenPreviousLines = 
    foldr ((+).((+1).length)) 0 . take y . lines

terminatedLines :: String -> [String]
terminatedLines = map (++ "\n") . lines

absPosition (Buffer (x, y) contents) = 
  x + (sum . map length . take y . terminatedLines) contents

absPosition (Buffer (x, y) contents) = 
  sum . (x:) . map length . take y . terminatedLines $ contents

Here’s what wikipedia has to say about lazy evaluation:

The benefits of lazy evaluation include: performance increases due to avoiding unnecessary calculations, avoiding error conditions in the evaluation of compound expressions, the ability to construct infinite data structures, and the ability to define control structures as regular functions rather than built-in primitives.

It’s all true, but I think it misses a point. One of the nicest things about lazy evaluation is that it enables a different form of abstraction. Take a look at this code. It computes the Mandelbrot Set. There are many ways to code up this algorithm, but the code I just linked to is very typical – it’s a doubly nested loop which iterates over a specific rectangle in the complex plane. It’s hard to see anything that’s awkward about its iteration strategy in a procedural language – the algorithm does require starting and ending points in the plane.

Let’s look at a different way of approaching this. Here’s part of a Mandelbrot Set generator in Haskell:

mandelbrot :: Double -> [[Int]]
mandelbrot incrementSize =
  [[ escapeIterations $ translate offset . translate (x,y) . mandel
    | x <- increments]
    | y <- increments] 
   where increments = [0.0, incrementSize .. ]

window :: (Int, Int) -> (Int, Int) -> [[a]] -> [[a]]
window (x0, x1) (y0, y1) = range y0 y1 . map (range x0 x1)
  where range m n = take (n-m) . drop m

The interesting bit is the mandelbrot function. It contains a doubly-nested list comprehension which, essentially, mimics the behavior of a doubly-nested loop in the procedural algorithm. However, there is one interesting difference. This list comprehension actually represents the mandelbrot computation across a full quarter of the infinite plane – it goes from zero to infinity in x and y. All we have to do is pass it an increment size and it is configured to create that grid. If we want to zoom in to a particular place in the Mandelbrot Set, we can use the window function like this:

-- compute the mandelbrot from (10,10) inclusive to
-- (20,20) exclusive with an increment of 0.05 

window (10, 20) (10, 20) $ mandelbrot 0.05

The trick to this code is in window’s range computation. It takes a starting position, an ending position, and a list and it drops all of the elements of the list before the starting position, without evaluating them, and then it takes, from the front of the new list, end - start elements, giving us a sub-range from the original list. When that elements of that sub-list are evaluated, the computation for each point in the window occurs.

This code yields the computation that we want and the neat thing is, we are really only computing the piece we specified with window. We’ve essentially formed a general computation and imposed the edges later.

Looking back now at the imperative mandelbrot code, it’s easy to see that it was, in a way, mixing two concerns. The code which determined the range was mixed with the code which generated the set. Laziness allows us to separate the two concerns.

Note: There are much more direct ways of computing the Mandelbrot Set in Haskell. Here’s one which is particularly brief.

I’ve been working in Haskell recently, and I find myself doing much less refactoring. In fact, I rewrite more than I refactor.

I’m not talking about massive refactoring, but rather the sort of micro-refactoring that I do to move from one algorithm to another. If you are working in an imperative language, you can often make micro waypoints – little testable steps along the way as you change your code from one shape to another. You can add a variable, confident that behavior won’t change because you haven’t used it yet, and you can move one statement down below another, changing the order of a calculation if (again) you are confident that the overall computation is order in variant. As I TDD my code, I do those sorts of things. I often refactor by introducing a parallel computation in the same method as the old one, and if the test passes, I knock out the old way of doing things and leave the shiny new one.

In Haskell, however, I don’t do that, and it reduces the amount of time that I spend refactoring. Yes, I rename variables, add parameters, and introduce where-clauses, etc. But, refactoring in Haskell seems different, and I think I know why now. There are two reasons and they sort of intermingle.

absPosition :: Buffer -> Int
absPosition (Buffer (x, y) contents) = x + lenPreviousLines contents
  where lenPreviousLines = foldr ((+).length) 0 . take y . terminatedLines

Some sculptors prefer clay and some prefer marble. Some choose on a piece by piece basis, depending upon what the work demands. The important thing is to respect the medium and understand its affordances and limitations.

Earlier this year, I was asked to speak at a Ruby Conference. I was happy to go, but I also felt a bit out of place. I haven’t done much Ruby, but I’ve admired the language from afar. I have a number of friends who’ve left C++ and Java to jump toward Ruby and Python and for the most part, they are happy. They do great work, and they enjoy it. They are living proof that the nightmare scenarios that people imagine about dynamic languages aren’t inevitable. You can program safe, secure, high quality applications in dynamically typed languages. People do it all the time, but that’s cultural knowledge. If you are in a culture, you hear about all of the things which are normal which appear odd from outside. If you aren’t, you don’t.

This is pretty much the situation I’ve been in with Ruby, up to a point. I haven’t written a large Ruby application. I’ve tinkered around with the language and written utilities but as far as total immersion goes — no, I’ve never been totally immersed in the language but that hasn’t kept me from learning noticing interesting things at the edge. One of the striking things that I’ve noticed is that the attitude of Rubyists toward their language is a bit different. They seem to have an ethic of responsibility that I don’t see in many other language cultures.

Ethic of responsibility? What do I mean by that?

I guess I can explain it this way. In many language communities, people are very concerned with the “right way” to do things. They learn all of the warts and edges of the language and they anticipate the ways that features could be misused. Then, they starting writing advice and style guides — all the literature which tells you how to avoid problems in that language. The advice goes on and on. Much of it centers around legitimate language defects. Some languages make you work hard to use them well. Other bits of advice, though, are really extensions of culture. If a language gives you mechanisms to enforce design constraints, it doesn’t feel quite right to not use them. As an example, consider sealed and final in C# and Java. Both of those constructs do pretty much the same thing and people do go out of their way to advise people on how they should be used to protect abstractions. It’s interesting, however, that languages like Ruby, Python, and Groovy don’t have anything similar, yet people do write solid code in those languages.

Let’s leave aside, for a minute, the debate over static and dynamic typing. What I think is more important is the issue of tone. In some languages you get the sense that the language designer is telling you that some things are very dangerous, so dangerous that we should prohibit them and have tools available to prohibit misuse of those features. As a result, the entire community spends a lot of time on prescriptive advice and workarounds. And, if the language doesn’t provide all of the features needed to lock things down in the way people are accustomed to, they become irate.

I just haven’t noticed this in Ruby culture.

In Ruby, you can do nearly anything imaginable. You can change any class in the library, weave in aspect-y behavior and do absolutely insane things with meta-programming. In other words, it’s all on you. You are responsible. If something goes wrong you have only yourself to blame. You can’t blame the language designer for not adding a feature because you can do it yourself, and you can’t blame him for getting in your way because he didn’t.

So, why aren’t more people crashing and burning? I think there is a very good reason. When you can do anything, you have to become more responsible. You own your fate. And, that, I think, is a situation which promotes responsibility.

For years, in the software industry, we’ve made the assumption that some language features are just too powerful for the typical developer — too prone to misuse. C++ avoided reflection and Java/C# avoided multiple-inheritance. In each case, however, we’ve discovered that the workarounds that programmers apply when they legitimately need a missing feature are worse than what the omission was meant to solve. Blocks and closures are good immediate example. There are tens of thousands of applications in the world today which contain duplication that you can really only remove with the template method design pattern or by creating a tiny class which encapsulates the variation. If blocks or closures were available, programmers would be more likely to tackle the duplication and arrive at much less cluttered design.

Meta-programming features are yet another example. Business applications are rife with situations where you need to know the value, type, and name of a piece of data, yet we use languages in which these sorts of capabilities have to be hand-coded and over and over again. The fact that it took decades for the industry to arrive at something as useful as ActiveRecord in Rails is due primarily to the attitude that some language features are just too powerful for everyone to use.

We’ve paid a price for that attitude. Fortunately, I think we are getting past it. The newer breed of languages puts responsibility back on the developer. But, language designers do persist in this sort of reasoning — this notion that some things should not be permitted. If you want to see an example of this style of reasoning see the metaprogramming section in this blog of Bruce Eckel’s ( http://www.artima.com/weblogs/viewpost.jsp?thread=260578 ). I respect Bruce, and I realize he isn’t speaking as a language designer, but I offer that as a example of that type of reasoning — reasoning about what should be permitted in a language rather than what puts a bit more control and responsibility in the hands of programmers. Maybe decorators work in 95% of the cases where you would want to do metaprogramming in an application, but there is a price to that choice, and it isn’t just the workarounds in 5% of the cases. The additional price is a decreased sense of responsibility and ownership. I think that those human dimensions have far more impact on software than many people suspect.

The fact of the matter is this: it is possible to create a mess in every language. Language designers can’t prevent it. All they can do is determine which types of messes are possible and how hard they will be to create. But, at the moment that they make those decisions, they are far removed from the specifics of an application. Programmers aren’t. They can be responsible. Ultimately, no one else can.