Object Mentor Blog

Using Band XI's FitNesse Plugin for Eclipse

Dean Wampler — Fri, 16 May 2008 17:22:22 -0500

Our friends at Band XI have an Eclipse plugin for FitNesse. I spent a few hours trying it out for a client. Here are some notes to help you use it.

Band XI’s plugin makes it easy to run FitNesse from within Eclipse and to edit the Java code and FitNesse pages in one place. Hence, no more switching between windows while you’re making your acceptance tests pass.

The download and installation page describes how to get started. While it only lists support for Eclipse 3.1 and 3.2, it seems to work fine in Eclipse 3.3 and 3.4 M7.

After you have installed the plugin, you will have two new toolbar buttons, one to start and stop a local FitNesse server and one to start and stop a remote server.

You configure where these servers are located in the Preferences > FitNesse page. These settings apply for the whole workspace. It might be nice to be able to configure them on a project by project basis, but that’s not supported (and maybe not that important, either).

You also get a new project option in the “New” wizard. Java > FitNesse > FitNesse Example Project creates a new Java project with src, FitNesseRoot and fixtures source directories. The include all the standard example acceptance tests and code that comes with the FitNesse download.

I encountered a little oddity the first time I started the local server by clicking the toolbar button. The usual front page, with links for the examples, manual, etc. weren’t shown on the FitNesse front page.

I determined that this happens because the default FitNesseRoot in the preferences is ”.”. Instead, start the server the first time by right-clicking on the FitNesseRoot folder, then select the FitNesse menu and the Launch FitNesse runtime on this root folder option. From then on, it will launch with this folder as the default, until you select a different one.

Using your own FitNesse installation and projects

Here’s what I did to work with existing projects and use a separate FitNesse installation outside of an Eclipse project.

If you have another project with the FitNesseRoot folder, you can use it as just described (i.e., using the context menu).

Otherwise, go to the Preferences > FitNesse page and paste the full path of your alternative FitNesseRoot in the “local root directory” field. Be sure to delete the ”.” value that is present by default and also watch for spaces before and after the string. It appears that they aren’t removed, causing mysterious behavior…

To build fixtures in your Java project, add the FIT_LIB and FITNESSE_LIB “variables” (defined by the plugin) to the Java build path for your project (using the project properties dialog). The default values of these environment variables should be fine unless you are running a different version of FIT or FitNesse in your separate installation (in which case you can change the values as needed).

Finally, set your class path in your FitNesse pages to point to the correct directory for the compiled classes. For example,

!path /home/me/projects/workspace/MyFitProject/bin

That should do it!

It's All Data

Brett Schuchert — Wed, 14 May 2008 16:37:00 -0500

“So, you managed to pick stories that only involved changing data…”

“Yep.”

Great, there’s no need to test it, right?

Therefore, testing Java classes is unnecessary.

The JVM simply reads files in a particular format (the Java Class format)
The JVM processes those files
Java class files are just data to the JVM
Java source files are just data to the Java Compiler to generate data used by the JVM
We do not need to test data changes
Therefore we do not need to test Java classes.

q.e.d.

Then Norbert asked “Oh, by the way, did you version your data changes?”

</sarcasm>

Your code is not flat

Brett Schuchert — Mon, 05 May 2008 23:35:00 -0500

There are three possibilities for the universe. On the one hand it is Closed. There’s enough matter that the universe, currently expanding, will eventually stop expanding and collapse in on itself.

The universe might be open. There’s not enough matter for gravitational forces to bring the universe back in on itself.

The final option is that the universe is flat. There’s just enough mass that it will stop expanding but it will not collapse in on itself.

Assuming an infinite number of universes, the final option, the flat option, must surly exist, somewhere. Probably not here, but somewhere. So for all intents and purposes, our universe is likely open or closed but not flat.

While you were reading that, your code rotted just a little. You’ve heard of it, bit rot. The bits in a program sitting on a disk rot. It’s a well known fact. Sure the magnetic surface could give out or if it’s in memory, a stray cosmic ray could flip a bit.

But that’s not what I’m talking about. You know the experience. It was just working and now it is not working. You did not change anything so therefore it is the same as it was and so something else must have changed. This is bit rot.

Typically, bit rot is really just our expectations being crushed by reality. Never let a fact get in the way of a good theory, that’s my motto. This works especially well if you tend to be a better talker than listener (guilty).

At any rate, your code continues to rot as you continue to read this. Why? Someone using it has managed to find another thing programmers did not think about. So what was “working” before is not “working” now. Wait, was it really working? If someone was getting value out of using it then yes it was working. Is it working now? If nothing about the system has changed but it no longer serves a purpose, then it is not working.

Here’s a question. If a tree falls in the forest and nobody hears it, does it make a sound? Answer from a human perspective, no. If there’s nobody to hear it, then it didn’t make a sound. Sure it moved some air about and you could take an infinite sum of sine waves to characterize that air movement but then if you characterize it then something was there to observe it and that seems to violate the spirit of “nobody hears it.”

What about if there’s a problem in the code and nobody hits it, does your code have a defect? Using the tree as an example, then the answer is no. However, unlike the tree’s sound, which quickly dissipates (if for no other reason that then laws of thermodynamics), your code tends to stay around a bit longer so the chance for observation of a sleeping defect is higher (though to be fair, if you wait long enough your code, like the sound in the forest, will go away).

OK, but it is even worse than that. Things change. If your body is not changing then you are dead. That’s a fact. In one study by Dr. Frisen, he demonstrated the age of cells in a rib bone of a 30 year old to be just over 15 years, while the cells that line the stomach to be closer to 5 days. Your body is constantly fixing itself. Cells replace themselves. That’s why you can donate platelets every 3 days and blood every 56 days.

What about your code? Is it fixing itself? Is it repairing itself? Does it need to? If your code is not changing, then the project is dead. Sure, it takes some time for all usefulness to finally wear out of the system but at some point the system will essentially bit rot so badly that is serves no useful purpose. (While there are multiple clinical definitions of “dead” for a body, some life remains long after you are clinically dead – your fingernails will keep growing for some time after you expire.)

There are several things that cause your system to decay. Here are just a few:

What the business needs changed so until the system catches up it has less value
The act of introducing the system has changed the business so that the system needs to change
Someone misunderstood something (it’s actually no small miracle that people can communicate at all – my wife would argue she cannot get through to me)
Someone actually changed some code badly (most 1-line bug fixes introduce new defects – see Weinberg)
Someone is compelled to hack it in to meet the deadline/demo/beat the lunch crowd/...

Your code is rotting while you read this because the environment around it is changing. Add to that an observation that most changes to code introduce rather than reduce chaos and it’s no wonder your system is either closed (code collapses in on itself, weighed down by its own incidental complexity) or open (nobody can decide what to do and the project eventually sort of fizzles into nothingness).

So what are you doing to actively maintain your system’s integrity? Are you just using antibiotics (patching things quickly hoping that there’s no super bug on the horizon – there’s always a super bug on the horizon no matter how strong an alligator’s immune system might be), or are you doing what the surgeon general has been saying for years: getting enough sleep, working out regularly, eat healthily (there’s a moving target).

Your code is rotting every day in every way unless you are actively working against that tide. Talking to your users – being friendly even, writing tests (acceptance, integration, smoke, unit, load, exploratory, ... mostly automated), integrating often, refactoring, learning to see beginning of rot rather than the end of it, etc.

There are a lot of things we can do and need to do to make a system. Part of that is keeping the system alive and breathing. Unlike biological systems where nature (and natural selection) has resulted in a self-monitoring, self-healing systems, code does not really monitor itself and fix itself (well most code does not do that).

Maybe there’s been a hidden force driving the creation of these things we called code maintainers whose existence is predicated on the need to repair living code to keep it around just a bit longer.

If this is the case, then just like the waist lines of americans are increasing as our biological systems have not had enough time to adjust to the changing environment, our code bases are bloating and getting to the point where code maintainers cannot keep the systems living long enough.

It seems a shift is in order to make it possible to keep these living, but often very sick, applications alive longer. In a sense, if you are practicing keeping your code clean, you are like a doctor because you diagnosing sickness, prescribing a path to heath and, if necessary, making incisions, using slugs or just waiting for a given sickness to simply follow its natural course before going away (like the requirement that simply must be done right away, just because).

Unfortunately, if we are calling ourselves doctors, then we’re still learning the value of keeping our hands clean between procedures. Unlike doctors of the mid-19th century who thought more blood on clothes = more experience, Ignaz Semmelweis figured out that hand washing saved lives.

As a community we’re not there yet. There’s still pride is hacking something together.

Your code is getting worse, what are you going to do about it?

What We Can Learn from the Ojibwe Language

Dean Wampler — Sat, 03 May 2008 09:48:57 -0500

Ojibwe (sometimes spelled Ojibwa; the last syllable is pronounced “way”) is one of the few Native American languages that isn’t immediately threatened with extinction. It is spoken by about 10,000 people around the Great Lakes region. Brothers David and Anton Treuer are helping to keep it alive, as they discussed in a recent Fresh Air interview.

Ojibwe is a language that is optimized for an aboriginal people whose lives and livelihoods depend on an intimate awareness of their environment, especially the weather and water conditions. They have many nouns and verbs for fine gradations of rain, snow, ice conditions, the way water waves look and sound, etc. You would want this clarity of detail if you ventured out on a lake every day to fish for dinner.

In the past, speaking languages like Ojibwe was actively suppressed by the government, in an attempt to assimilate Native Americans. Today, the threat of extinction is more from the sheer ubiquity of English. I think there is another force at play, too. People living a modern, so-called “developed” lifestyle just don’t need to be so aware of their environment anymore. In fact, most of us are pretty “tone deaf” to the nuances of weather and water, which is sad in a way. We just don’t perceive the need for the richness of an Ojibwe to communicate what’s important to us, like sports trivia and fashion tips.

So, what does Ojibwe have to do with programming languages? Our language choices inform the way we frame problem solving and design. I was reminded of this recently while reading Ted Neward’s series of articles on Scala. Scala is a JVM language that provides first-class support for functional programming and object-oriented design refinements like traits, which provide mixin behavior.

While you can write Java-like code in Scala, Neward demonstrates how exploiting Scala features can result in very different code for many problems. The Scala examples are simpler, but sometimes that simplicity only becomes apparent after you grasp the underlying design principle in use, like closures or functional idioms.

One of the best pieces of advice in the Pragmatic Programmer is to learn a new language every year. You should pick a language that is very different from what you know already, not one that is fundamentally similar. Even if you won’t use that language in your professional work, understanding its principles, patterns, and idioms will inform your work in whatever languages you actually use.

For example, there is a lot of fretting these days about concurrent programming, given the rise of multi-core CPUs and multiprocessor computers. We know how to write concurrent programs in our most popular imperative languages, like Java and C++, but that knowledge is somewhat specialized and not widely known in the community. This is the main reason that functional programming is suddenly interesting again. It is inherently easier to write concurrent applications using side-effect-free code. I expect that we will fail to meet the concurrency challenge if we rely exclusively on the mechanisms in our imperative languages.

So, you could adopt a functional language for all or part of your concurrent application. Or, if you can’t use Scala (or Haskell or Erlang or …) you could at least apply functional idioms, like side-effect-free functions, avoidance of mutable objects, etc. in your current imperative language. However, even that won’t be an option unless you understand those principles in the first place.

Learning a new language is more than learning a new vocabulary. It’s even more than learning new design techniques. It’s also learning to see common things from a fresh perspective, with greater clarity.

VIM as a Diff/Merge Tool

Tim Ottinger — Wed, 30 Apr 2008 21:24:00 -0500

Do you need a really quick, clean, powerful diff/merge tool that you can use in windows, mac, and various Unixen? As a happy and pround vim user, I have been enjoying vimdiff for a number of years.

        > gvim -d file1 file2
        > vimdiff file1 file2

(G)vimdiff is wonderful. You can move changes between two versions of a file with great ease (if you know vim) and amazing speed. It can handle files of any size and does a fine job of syncing up both versions.

Vim can do just about anything. I have learned that no matter how much I use and study Vim I still am just scratching the surface of what the editor can do. I find little tricks and configuration options, new keystrokes, and fun little bits of minutae. I think it’s all new. With my memory, sometimes I just think something is new, and have to go check my past notes to see if I already knew it.

Today I (re)learned about diffopt, which makes gvimdiff more wonderful.

    :set diffopt=filler

By default, diffopt is set to “filler”. Filler doesn’t do much, it just adds vertical spaces to keep the text of the left and right pane aligned. It’s a good setting, and I don’t much like working without it. But there are other settings which are overlooked, even by experts.

    : set diffopt+=iwhite

My second favorite is ‘iwhite’, which stands for “ignore whitespace”. It makes gvimdiff ignore leading, trailing, and embedded spaces so that simple acts like retabbing or deleting vertical spaces won’t obscure actual changes. It really helps you to merge changes in situations where you’ve adopted code that was built to entirely different standards.

    :set diffopt+=icase

iCase ignores capitalization, which is not as useful as the other options, since programmers are normally interested in case, but it may be handy when someone has corrected and reformatted SQL. I don’t expect to use it much.

    :set diffopt+=horizontal

The last interesting bit is the choice between vertical splits (the normal case) and horizontal splits (for very long lines on cinema-aspect screens). I hope to never need this option again, but I have needed to deal with absurdly long lines.

If you can use VIM at all, you will find vim’s diffmode to be a very powerful and useful tool. If you can’t use vim already, then maybe vimdiff will be the reason you will learn vim.

Magic Funnel, Part 3: Covey's Miracle

Tim Ottinger — Thu, 17 Apr 2008 22:15:00 -0500

We had a team of surrogate stakeholders in the room, and all the very most important stories (as far as we could tell) on a table in front of us.

“These are the most important?” Heads nodded. “Then,” I asked, “what makes them important?” Brows furrowed, and well-considered answers poured forth.

I collected their reasons for importance. Some were about keeping promises, some were about providing increased quality (uptime, correct function), and there were other kinds of importance. We wrote them on a 3×5 card. Ultimately, we realized that the company’s reputation is their source of importance. Users needed to know that the company will do the right things, and provide software that worked and kept them working. I thought it was a good criteria.

Next I explained that urgency is different from importance. There may be a hot demand for some cosmetic changes that don’t really impact the company’s reputation one way or another. There may be some changes that in the direction we want to go. These may be hot topics, but may not be truly important.

We could see a difference between the forces of urgency and the force of importance, but there is gray area as a sufficient pent-up demand for otherwise unimportant changes could in fact effect our reputation for service. Still, we could usually see a crisp-enough boundary for any given user story.

I then (re-)introduced the team to Stephen Covey’s four-quadrant system. It has two axes. The vertical is importance (important at the top, unimportant below). The horizontal is urgency with urgent things to the left and non-urgent on the right:

Q1 Important and Urgent	Q2 Important, not Urgent
Q3 Unimportant and Urgent	Q4 Neither important nor urgent

We took our set of cards and the team determined groupings. Was this a quadrant one activity? Quadrant 3? Quadrant 2? There were some real surprises, especially when we realized our “most important” bit of work was not urgent and should wait until quadrant one activities were finished.

Covey’s recommendation is that we tackle first those items that are both important and urgent. When we knock that stack down, we move on to those that are important but not urgent. If we have time left, those non-important-but-urgent items can “fly standby” in our sprint. We reordered the priorities in accordance with Covey’s recommendation.

We had no cards that were both unimportant and non-urgent.

At this point in time, we had what looked very much like a magic funnel. Ideas came in from everywhere, and the items that moved from the planning team to the developers came out in the “magically correct” order. We agreed that the selection and the ordering were the best we could do at the time.

Our next sprint planning meeting was remarkably efficient.

Building Magic Funnels, Part 2: Pragmatic Pedantry

Tim Ottinger — Thu, 17 Apr 2008 22:00:00 -0500

The middle of the funnel we started on needed work. While it is a simple idea that the single, most important thing in the funnel is the first to emerge from the bottom, it is a multi-flavored affair to try to manage for real.

My first strategy was to get the right people in a room and have them fight it out. I think that the prioritization process should be a lot like local government (maybe a school board?) in that people should argue and complain and push and eventually settle on compromises and deals. Ultimately, I believe that people who have a strong interest in the company can make the right decisions. At least they can be right enough for the next 5 days. When you have one-week iterations, the next chance to change the agenda is never far away.

This first strategy didn’t work out, and so we went to a backup plan. A C level manager said he knew what we should do, and so we scheduled an hour or so with him, myself, our priority manager (“funnel guy”) and the senior technical developer.

Our guys used the sticky post-it 3×5 cards and papered the CIO’s windows and whiteboards. They listed the various categories of work from the various stakeholders and pasted them up in priority order.

I asked my first pedantic question: “What is the single most important thing we can work on? If you had only one story that you knew for sure would be finished this week, which would it be?” That led to a nice discussion.

When they placed the card on the table, signifying that it was definitely in the build, I asked again if there was any one card anywhere else in the room more important. I asked if that was really the single most important one.

When the answer was “yes, absolutely” I was ready for my second pedantic question: “Now that this card is off the board, what is the single most important card left for us to do, if only two stories were guaranteed to be done.” Now the pedantry was fully exposed, but the idea had carried. The team collected all the most important stories and placed them in order on the table.

Now I was ready for my next round of pedantry.

Building Magic Funnels, Part 1

Tim Ottinger — Thu, 17 Apr 2008 21:38:00 -0500

The idea of a magic funnel, as you may remember, is that there is some kind of organizational structure where many ideas and proposals and issues go into the top. Through some magic, the item that emerges from the bottom of the funnel (to go to the development team) is the single most important thing they could work on next.

This is all just backlog management and prioritization, of course. But I think it can be simpler than I’ve seen it in the past, and that real people working without magic can approximate it.

Recently, I’ve been working to establish another magic funnel. One of the first things we did was to find the person who formerly handed out work to the developers and made him a single point of contact. In our case this is not the scrum master, but is another trusted line-manager type. He has been given a number of people to work with on the Customer side of the house and also works with technical people.

We have tried to establish story feeds from all the various stakeholders. Developers, operations people, technical writers, customers, sales people, marketing people, inside security consultants, and others have been feeding their ideas in to our point-of-contact man. This part of the funnel is working fairly well.

The next thing we have tried to do is to match the feed of work to the rate at which work can be done. This has created a fair amount of back-pressure on the stakeholders (which I believe to be healthy).

We have also worked on making the bottom of the funnel narrow, meaning that our guy doesn’t scatter new work to the team, but feeds it through the scrum master who protects the team’s productivity and keeps the work flow and tasks visible on the status board. He makes sure that the team is not expected to “absorb” changes, but that adding two points of work to the iteration results in two points of unstarted work coming off. This also creates a healthy back-pressure.

As a pun on “in-flight”, I named another area of the board “flying standby”. This is for stories that aren’t important enough to swap a story off the board (or for stories that are displaced by more important ones). If the developers finish more work than they expected, there are stories that can be picked up even though they’re not a scheduled part of the iteration. Stakeholders are told that there is no guarantee of these stories being picked up at all in this iteration, but there is some small chance that it could happen if the team discovers that it has overestimated some other stories.

The bottom of the funnel is working pretty well.

What’s missing is the “magic” bit.

The Magic Funnel

Tim Ottinger — Wed, 09 Apr 2008 13:18:00 -0500

At a planning meeting, the developers on the team offer up some amount of work they are able to do. This capacity is termed velocity. The customer part of the team then selects enough user stories to consume all of the offered capacity. This is simple negotiation, but that hardly tells the story. What is happening on the customer side is a magic funnel.

All kinds of stories come pouring into the customer group. Some are from actual users in the field who want improvements to existing features. Some suggest entirely new areas of functionality into which the program could expand to good effect. Some are from the operations group, generally describing ways to make the product more reliable, configurable, or manageable. Some are from the developers who realized that some small changes in code could provide large changed in capability or usability. Some are from external regulatory bodies, some are from quality control, but for the most part they’re all good ideas and many have urgency.

When the magic funnel is working, we pour all these stories into the top, and the thing that falls out the bottom is the single most important, urgent task we could possibly do. When it comes time for a new iteration, we fill in our allotted space with whatever comes out the end of the funnel. Whatever we start doing on Monday, it is the best thing we could possibly start doing. We know that it is true because it came out of the funnel that way.

Sadly, magic funnels are hard to come by. We approximate them with human beings in multi-disciplinary small groups. If we leave the funnel purely in the hands of operations, then operational stories are the first to come out. If we leave the funnel in the hands of marketing people, we tend to do things that are more speculative even though the product users don’t really need them right now. If it’s sales, then whatever will sell more product this month will be the first thing out of the funnel. Give control to customer support and we find ourselves constantly appeasing customers and not moving forward on larger issues. If we leave it to developers, it will tend to turn out those stories that improve the experience of developing the application. Of course the problem is about conflicting perspectives (all of them right!) with competing desires being processed in fixed time.

I’m not an expert in creating magic funnels, though it would be a very interesting direction to take at this point in my agile coaching career. I believe that the answer is ultimately to be found in people working together, well-informed people arguing and debating and even trading favors.

I think that part of the answer is Covey-style prioritizing to select the urgent and important (Q1), then the important items that aren’t urgent (Q2), and holding all unimportant work until there is nothing better to do.

Importance in this case relates to how much the change will affect the future of the product and the company. This is not just a matter of closing a contract or soothing an angry customer. It may be that the future is larger and brighter than our immediate concerns. Immediacy is “urgency”, but the leverage to create a more greater future is “importance”. I had to read Covey’s First Things First several times to really understand the difference, as may the kind reader. But I did manage to absorb the fact that until you know what you want your future to be, you don’t know what is important.

But imagine how great it will be when you have established your magic funnel, and the work being assigned to each iteration really is the most important and valuable thing that you could do next. Imagine what it would be like if you really were building the future you would like to live in.

Clean Code. Whew!

Uncle Bob — Tue, 08 Apr 2008 00:10:16 -0500

I’ve been working on this book for several years now. After a flurry of effort (you might have noticed I’ve been quiet lately) I’m very pleased to say that I’m done with the writing and am preparing the manuscript for production. See The Prentice Hall Listing

Table of Contents

Clean Code    1
    There Will Be Code    1
    Bad Code    2
    The Total Cost of Owning a Mess.    3
    Schools of Thought.    11
    We are Authors.    12
    The Boy Scout Rule    13
    Prequel and Principles    14
    Conclusion    14
    Bibliography    15
Meaningful Names by Tim Ottinger    17
    Introduction    17
    Use Intention-revealing Names    17
    Avoid Disinformation    19
    Make Meaningful Distinctions    20
    Use Pronounceable Names    21
    Use Searchable Names    22
    Avoid Encodings    23
    Avoid Mental Mapping    24
    Class Names    25
    Method Names    25
    Don't Be Cute    25
    Pick One Word Per Concept    26
    Don't Pun    26
    Use Solution Domain Names    27
    Use Problem Domain Names    27
    Add Meaningful Context    27
    Don't add Gratuitous Context    29
    Final Words ...    30
Functions    31
    Small!    34
    Do one thing.    35
    One level of abstraction per function.    36
    Switch Statements.    37
    Use descriptive names.    39
    Function Arguments.    39
    Have no side-effects.    43
    Command Query Separation    44
    Prefer exceptions to returning error codes.    45
    Don't Repeat Yourself.    47
    Structured Programming    48
    How do you write functions like this?    48
    Conclusion    49
    SetupTeardownIncluder    49
    Bibliography    52
Comments    53
    Comments do not make up for bad code.    55
    Explain yourself in code.    55
    Good Comments    55
    Bad Comments    59
    Example    71
    Bibliography    74
Formatting    75
    The Purpose of Formatting    76
    Vertical Formatting    76
    Horizontal Formatting    84
    Team Rules    89
    Uncle Bob's Formatting Rules.    90
Objects and Data Structures    93
    Data Abstraction    93
    Data/Object anti-symmetry.    95
    The Law of Demeter    97
    Data Transfer Objects    99
    Conclusion    101
    Bibliography    101
Error Handling by Michael Feathers    103
    Use Exceptions Rather than Return Codes    103
    Write Your Try-Catch-Finally Statement First    105
    Use Unchecked Exceptions    106
    Provide Context with Exceptions    107
    Define Exception Classes In Terms of a Caller's Needs.    107
    Define the Normal Flow    109
    Don't Return Null    110
    Don't Pass Null    111
    Conclusion    112
    Bibliography    112
Boundaries by James Grenning    113
    Bibliography    119
Unit Tests    121
    The Three Laws of TDD    122
    Keeping Tests Clean    123
    Clean Tests    124
    One Assert per Test    129
    F.I.R.S.T.    132
    Conclusion    132
    Bibliography    133
Classes    135
    Class Organization    135
    Classes should be Small!    136
    Organizing for Change    146
    Bibliography    150
Systems    By Dean Wampler 151
    How would you build a city?    151
    Separate constructing a system from using it    152
    Scaling Up    155
    Java Proxies    158
    Pure Java AOP Frameworks    160
    AspectJ Aspects    163
    Test-drive the system architecture    164
    Optimize decision making    165
    Use standards wisely, when they add demonstrable value    165
    Systems need Domain-Specific Languages    166
    Conclusion    166
    Bibliography    167
Emergence By Jeff Langr    169
    Getting Clean via Emergent Design    169
    Simple Design Rule 1: Runs all the tests    170
    Simple Design Rules 2-4: Refactoring    170
    No Duplication    170
    Expressive    173
    Minimal Classes and Methods    174
    Conclusion    174
    Bibliography    174
Concurrency    by Brett Schuchert 175
    Why Concurrency?    176
    Challenges    177
    Concurrency Defense Principles    178
    Know Your Library    180
    Know Your Execution Models    181
    Beware Dependencies between Syncrhonized Methods    182
    Keep Synchronized Sections Small    183
    Writing Correct Shut-Down Code is Hard    183
    Testing Threaded Code    184
    Conclusion    188
    Bibliography    189
Successive Refinement    191
    Args Implementation    192
    Args: the rough draft.    198
    String Arguments    212
     Conclusion    246
JUnit Internals    249
    Conclusion    262
Refactoring SerialDate    263
    Conclusion    280
    Bibliography    281
Smells and Heuristics    283
    Comments    283
    Environment    284
    Functions    285
    General    285
    Java    304
    Names    306
    Tests    310
    Conclusion    311
    Bibliography    312
Concurrency II    by Brett Schuchert 313
    Client/Server Example    313
    Possible Paths of Execution    317
    Knowing Your Library    322
    Dependencies between methods can break concurrent code    325
    Increasing Throughput    329
    Deadlock    331
    Testing Multi-Threaded Code    335
    Tool Support for Testing Thread-Based Code    337
    Conclusion    338
    Tutorial: Full Code Examples    339
org.jfree.date.SerialDate    345
Cross References of Heuristics    406

Your Attitude is Affecting Other Departments

Tim Ottinger — Sat, 05 Apr 2008 21:17:00 -0500

The CIO looked into the eyes of his agile development staff last Friday. “Your attitude is affecting other departments” he said.

I’ve heard a lot of department-level speeches start this way, and in relatively small companies it is not unheard-of for a C-level manager to address attitudes of development teams.

The group has been working to stabilize and improve a product that was developed by a tiger team of outsider contractors and handed over to the in-house team post-facto. The developers had to overcome many obstacles to come up to speed on the code, to learn the new programming languages and tools, and to try to keep the feature set moving forward. None of them had been involved in the original design, but it was now their product, and its problems were their problems.

At the time of this meeting, I was one part of a coaching team which had introduced a great many changes. We were trying to help the organization to build a “whole team” mentality that encompassed documentation, security, systems administration, QA, customer representatives, and developers alike. We’d attacked the problem of matrix management. We reorganized the seating floorplan. I think at some point we’d been an inconvenience to just about everybody. The developers were in the middle of their second two-week iteration.

On this occasion, the team was in the midst of a recurring production difficulty, and had been gathered into the CIO’s office to work through a top-20 list of problems to solve.

“Your attitude has been affecting other departments”, he said. “And I want to thank you for it.”

Velocity is Just Capacity

Tim Ottinger — Wed, 02 Apr 2008 19:23:00 -0500

I don’t know why, but somehow velocity as a term really bothers me.

We always want higher velocity, and that’s a good thing. But with the term velocity, we think that we can create velocity with greater pressure (thrust?). Doesn’t it make sense that with a really hard push you can get greater velocity?

I guess I get hung up on words. Essentially, “velocity” is fine. It is the rate at which we’re “burning through” our stories. We use “yesterday’s weather” to set our level, and we are always seeking any sustainable way to increase our velocity. Why would we produce only 12 points when we could produce 20? As motivated developers, we aren’t yearning to accomplish less. But somehow this doesn’t tell the right story.

When our velocity is decreased, it always seems like a tragedy. We have all felt it when our 23-point iteration is followed by a 21-point iteration. We can justify it a lot of different ways, but it always feels tragic. Worse, hitting any kind of plateau seems frustrating.

You increase velocity by pushing harder. If you push harder, and the numbers come up, then you’ve successfully increased the velocity and that seems to be intuitively all there is to it.

I’ve reflected about this for a few years before I realized that I really like the term capacity much more. It is also accurately describing the rate at which we can truly complete stories. Since we use yesterday’s weather, we can even use the term “proven capacity” to describe that we’ve demonstrated the speed. But it also says some other things.

Intuitively it seems that we know that you don’t push your people beyond their capacity to get work done. Instead, you want to increase their capacity. When we talk about longer hours, it is still obvious that working people harder diminishes their capacity. Instead, we want to be working at capacity on the one hand, and increasing our capacity on the other.

Increasing the capacity sounds like the right problem. To improve capacity, you want fresh workers working hard, pairing and communicating, building and/or using better tools (or using tools more capably), using more powerful hardware, learning, and performing the kind of housekeeping that keeps the team productive. Life balance enters into it. Sufficient team size enters into it so you’re not so small you’re short-handed, yet not so plentiful you trip over each other.

I don’t suppose that anyone is going to rename standardized eXtreme Programming terms like “velocity” just because some guy from Indiana says to, but I wonder if maybe we can’t rephrase the problem in such a way that we are intuitively driven toward more reasonable solutions. Maybe we should explain that velocity is just capacity, and then work from that point onward.

Musing over Mutation

Tim Ottinger — Thu, 20 Mar 2008 22:25:00 -0500

I read a mailing list entry in which one fellow (who? I can’t remember!) asked another “Do you want to get better at what you’re doing, or find a better way to get the results you want?”

I’m a sucker for a good one-liner. That one had me thinking, and as I’ve had other conversations about innovation, I keep coming back to that line.

In our Agile practices, we work really hard for a week or two, and then hold a retrospective. The purpose of the retrospective is to find ways to work more effectively for the next two weeks. As we develop better software, we also evolve a better team. We use “tricks” such as tracking our velocity and recording blockages on our ‘waste snake’ to provide data for our decisions, and we use gut feel to evaluate those things that feel like collateral effort to us.

If the practice works, we will see incremental improvement in the team. We will develop ways of avoiding special variations, and we will learn to accept our normal variations. It will make us better at the way we do things now.

XP didn’t come from a series of incremental improvements to the waterfall technology. I wasn’t there when it happened, but it seems that they took on an change in axioms. They didn’t strengthen the contracts between groups, but pulled all the decision-makers onto the same team. They didn’t find more careful ways to examine the code they were changing, but rather decided to lean radically on volumes of tests. They didn’t build practices to improve their anticipatory design, they decided instead not to anticipate at all and simplify their design to allow future change. At the time, this was radical stuff.

I’m sure there have been many other less-successful process mutations, but there is no evolution without mutation.

The man behind the iPod, iPhone, and MacBookPro has had some less-successful product ideas, too. Some exciting high-concept products didn’t make it in the wild. But then some new ideas become category killers.

How do we learn to make the axiomatic changes that lead us to radically better ways to get what we want?

TDD on Three Index Cards

Tim Ottinger — Thu, 06 Mar 2008 19:39:00 -0600

I had the opportunity to talk to a fellow who missed part of a class on TDD. I told him that I could give him a 15-minute overview, and give him all the essentials of TDD on three index cards.

Yes, I know that volumes have been written about TDD and BDD, and that it’s a large topic with many many branches of application, but I didn’t have time for that. I had time for three index cards. I figure that an index card is a token, and it represents a conversation, and that one can always dig deeper later.

They looked more or less like this:

Card 1: Uncle Bob’s Three Laws (Object Mentor)

Write no production code except to pass a failing test.
Write only enough of a test to demonstrate a failure.
Write only enough production code to pass the test.

Card 2: FIRST Principles (Brett and Tim at Object Mentor)

Fast: Mind-numbingly fast, as in hundreds or thousands per second.
Isolated: The test isolates a fault clearly.
Repeatable: I can run it repeatedly and it will pass or fail the same way each time.
Self-verifying: The test is unambiguously pass-fail.
Timely: Produced in lockstep with tiny code changes.

Card 3: Flow (using the famous three-node circle diagram) – origin unknown.

Red: test fails
Green: test passes
Refactor: clean code and tests

Sure there is plenty more, but I didn’t know how I could provide significantly less. As is, I’m pretty happy with the exercise. Now I am wondering if I couldn’t produce most of the important information I wish to convey as a series of index cards. Would that be cool or what?

Clues For Reading New Code

Tim Ottinger — Wed, 05 Mar 2008 19:27:00 -0600

Okay, somebody just handed you a new chunk of code to work on. Your first though on opening the file is “Why, dear God, why?”. How do you get a handle on this masterpiece of clever programming? Let’s look for a few clues.

Where to start?

Documents

Why not look at the documentation? <laughs> Just kidding. You know there’s no documentation, and it’s probably not useful.

People and Interactions

You need a partner who is familiar with the code. Having a guide is better than having a map. Not only will you know where to go, but you’ll know ’’how’’ to go and where not to step. Other people are a wonderful resource. Just don’t settle for someone doing the work in front of you… the goal is to learn as much as to do.

No, really, documents!

OTOH, if you have somethng in the way of a summary or architectural overview, that might help. I’d read the abstract and look at the pictures. We’re after big-picture, so read it only, don’t trust it. Documents are seldom accurate, and seldom for long. It might be good preparation for your partner’s visit.

Use The Tests, Luke

Do you have tests? If you are in a test-driven shop, looking at unit tests is a good idea. If they’re written well, they are specifications (by example) of the system. If they’re written poorly, you know where to start your work at least. Your partner could do a great service by helping clean up the code instead of explaining it.

No tests? Uh, oh. No tests and no doc? Now you’re in trouble.

Scan the File Space

Are the classes well-named? Do you know where to begin working? If you can find code by looking at file names then you’ve got the handle you need. Your partner can help improve file naming if you find the code you’ve been handed is in ill-named files.

So you’re in a file, you have a mission. Either you have tests, or you’re ready to write some.

Sense Of Smell

What if the code is still not obvious? You have one more resource before you have to print out the listing and break out the markers to reverse-engineer your way out of code hell. You have your nose and your refactoring editor. You can use refactorings such as ‘rename’, ‘extract method’, and ‘introduce variable’ to clarify an existing method. You can spot duplication and eliminate it. Maybe you can bring it to a point of clarity, and then you will know

Deep Code Spelunking

It’s time to pour a really big cup of tea, get some food, take a couple of preemptive aspirin, and make sure your printer has paper.

Sometimes, you will have to reverse-engineer some of the code you were handed. Your only hope may be to rebuild it from the inside-out.

You might want to either copy the code to a scratch space to do this. Maybe you want to branch the project in version control. You want to be free to dig in an learn this the hard way, and make it better as you emerge from the depths.

You still can/must use refactoring tricks to capture the knowledge you gain, but you may need to take the code off line. And look for new partners.

Turn Back The Dial

Tim Ottinger — Wed, 05 Mar 2008 19:16:00 -0600

The coolest thing just happened! I broke the glass cover off of my watch. At first I thought it was awful, but then I realized that I could turn the hands.

Imagine my joy as I realized that I could make it 11:30 again, and go enjoy another lunch. Meeting at 3:30? No problem, just turn the hour hand up to 6:00 and go home! I can sleep as long as I want as long as I turn it back to 8:00 when I get to the office. All my work estimates are now “five minutes”, and I complete them every time.

My coworkers have no idea the awesome power that I’ve gained with this one happy accident. They ask “what time is it?” and I say “what time would you like it to be.”

Of course the above is a total fabrication. Pretending it’s 6:00pm when it’s 8:00am isn’t going to do anybody any good at all, and is likely to make a mess of things for me.

But people still try to mandate velocity.

Discipline often directed at the symptom, not the cause

Brett Schuchert — Sat, 01 Mar 2008 16:18:00 -0600

Have you ever heard something like?

If the developer just had a little discipline and did it the right way, we would not have this problem.

That’s often a sign that the way something is getting done is hard to do, not supported well or just plain works against against you.

Here’s an example I recently came across…

In a particular organization, part of their build system produces “project” information to allow for development on multiple platforms and with different C++ compilers. That’s excellent, it sounds like a great tool. For a developer to get started, he/she:

Checks out the source tree
Runs the script
Starts working

So far, everything is great and this part of their build system is essential to their environment – and good in general.

Here’s the next part…

To add a file to the system you have to:

Create the file
Update project information
Rerun the script to regenerate project information.

When I asked how long it takes to add a class, I was told about 5 minutes. So if I want to add a header file and source file, it takes 5 minutes. That’s a big problem. Why?

After this discussion, I heard one of the senior people lamenting that a developer had put another class in an existing file rather than creating a second file. He said something like “If the developer just had discipline, he’d to the right thing.” Those darn developers.

It takes 5 minutes to add a few files to a build. That does not include build time. That’s just the time to configure the build information.

Does 5 minutes seem like very much time?

Here are a few more thins I noticed(before I knew about the build system):

Some header files defined multiple classes
Some source files had the methods for those multiple classes
Some of the header files had names that did not match any of the classes defined in that header file

So is this a problem?

Here’s an important rule from Jerry Weinberg:

Nothing + Nothing + Nothing eventually equals something

5 minutes may not seem like a lot of time, and if it were isolated, then it’s probably not a problem. On the other hand, when you multiply it by a team and time, you end up with big problems.

Imagine, you need to use class X. Its header file is actually named Q.h and by the way, classes T U and L are defined in that file – none of which you want to know about.

So your class now has phantom dependencies on T, U and L. Also, how did you find the right header file? A quick search (time wasted). Someone changed U, so you end up having to recompile even though you don’t care about nor use (wasted time). I’m sure you can come up with a few things on your own.

So what do you do about it?

OK, first, do not throw the baby out with the bathwater. The original tool solved an important problem. But the first rule of problem solving, again from Jerry Weinberg:

Every solution introduces problems

The problems include (but are not limited to):

Time wasting adding files
It requires discipline to add new files, so it doesn’t always happen
A name is wrong, but it’s a pain to update the build configuration, so it doesn’t happen – not all the time, just every so often

Little by little, things get a bit more chaotic.

So now that we’ve observed a problem – some waste, we need to find a way to remove the need to update the build information and regenerate to even work.

I don’t know what’s going to happen with this group. They are hoping to perform some refactorings. Their system has quite a bit of coupling. One thing we can do to reduce coupling is:

Introduce interfaces
Inversion of Control
Identify seams and use some of Feather’s stuff to introduce them
Etc., the usual stuff to introduce seams and break dependencies.

But many of the refactorings they’ll want to use will involve creating new classes. Since that takes a little bit longer, it will slow everything down – or seem so daunting, it might not happen at all.

Here’s a personal example. A few years ago, I built the security system for one applications and then a suite of applications with single sign on. When I initially introduced the security system, many people wrote tests that would forget to log out, causing problems in both the real system and the simulator.

I kept grumbling. I though, “if people would just do it right, there wouldn’t be a problem.” If they just had a little discipline.

Independent of whose fault it was, it ended up being my problem – and, quite frankly, it was my fault as well. The solution was actually pretty easy:

Create an abstract test base
Change tests to use it
In the setup, the base logged in
In the teardown, the base logged out

You might think, “Duh!” and in retrospect, so did I. But that few minutes of effort:

Reduced code duplication
Increased the stability of the tests
Made it hard for people to mess up (so long as they used the correct test base – and I updated all of the tests that needed it, so going forward, people had correct examples upon which to base their work).

Ultimately, this removed a lot of my wasted time

Detecting waste is the first thing. Until you know it is a problem, you cannot do anything about it.

So, the next time you think something like:

If that person was only following “the rules”
If he/she just had a little more discipline
Stupid user, they should not have done that

Ask yourself if it’s possible that those statements are directed at the symptom, not the problem.

Writing Java Aspects ... with JRuby and Aquarium!

Dean Wampler — Mon, 25 Feb 2008 22:10:00 -0600

Aquarium V0.4.0, my AOP library for Ruby, now supports JRuby. Not only do the regular “pure Ruby” Aquarium specs run reliably under JRuby (V1.1RC2), but you can now write aspects for Java types with Aquarium!

There are some important limitations, though. Cartographers of old would mark dangerous or unknown territory on their maps with hic sunt dracones (“here be dragons”), a reference to the old practice of adorning maps with serpents around the edges.

This is true of Aqurium + Java types in JRuby, too, at least for now.

Aquarium uses Ruby’s metaprogramming API extensively and the JRuby team has done some pretty sophisticated work to integrate Java types with Ruby. Hence, it’s not too surprising there are some gotchas. Hopefully, workarounds will be possible for all of them.

The details are discussed on the JRuby page, the README on the Aquarium site, and of course the “specs” in the distribution’s jruby/spec directory. I’ll summarize them here, after discussing the pros and cons of Aquarium vs. the venerable AspectJ and showing you an example of using Aquarium for Java.

Briefly, Aquarium’s advantages over AspectJ are these:

You can add and remove advice dynamically at runtime. You can’t remove AspectJ advice.
You can advise JDK types easily with Aquarium. AspectJ won’t do this by default, but this is really more of a legacy licensing issue than a real technical limitation.
You can advise individual objects, not just types.

Aquarium’s disadvantages compared to AspectJ include:

Aquarium will be slower than using AspectJ (although this has not been studied in depth yet).
Aquarium’s pointcut language is not as full-featured as AspectJ’s.
There are the bugs and limitations I mentioned above in this initial V0.4.0 release, which I’ll elaborate shortly.

Here is an example of adding tracing calls to a method doIt in all classes that implement the Java interface com.foo.Work.


Aspect.new :before, :calls_to => [:doIt, :do_it], :in_types_and_descendents => Java::com.foo.Work do |jp, obj, *args|
  log "Entering: #{jp.target_type.name}##{jp.method_name}: object = #{object}, args = #{args.inspect}" 
end

There are two important points to notice in this example:

You can choose to refer to the method as do_it (Ruby style) or doIt, but these variants are effectively treated as separate methods; advice on one will not affect invocations of the other. So, if you want to be sure to catch all invocations, use both forms. There is a bug (18326) that happens in certain conditions if you use just the Java naming convention.
If the type is an interface, you must use :types_and_descendents (or one of the supported variants on the word types...). Since interfaces don’t have method implementations, you will match no join points unless you use the _and_descendents clause. (By default, Aquarium warns you when no join points are matched by an aspect.) However, there is a bug (18325) with this approach if Java types are subtyped in Ruby.

Limitations and Bugs

Okay, here’s the “fine print”...

In this (V0.4.0) release, there are some important limitations.

Aquarium advice on a method in a Java type will only be invoked when the method is called directly from Ruby.
To have the advice invoked when the method is called from either Java or Ruby, it is necessary to create a Ruby subclass of the Java type and override the method(s) you want to advise. These overrides can just call super. Note that it will also be necessary for instances of this Ruby type to be used throughout the application, in both the Java and Ruby code. So, you’ll have to instantiate the object in your Ruby code.

Yea, this isn’t so great, but if you’re motivated… ;)

There are also a few outstanding Aquarium bugs (which could actually be JRuby bugs or quirks of the Aquarium-JRuby “interaction”; I’m not yet sure which).

Bug #18325: If you have Ruby subclasses of Java types and you advise a Java method in the hierarchy using :types_and_descendents => MyJavaBaseClassOrInterface and you call unadvise on the aspect, the advice “infrastructure” is not correctly removed from the Ruby types. Workaround: Either don’t “unadvise” such Ruby types or only advise methods in such Ruby types where the method is explicitly overridden in the Ruby class. (The spec and the Rubyforge bug report provide examples.)
Bug #18326: Normally, you can use either Java- or Ruby-style method names (e.g., doSomething vs. do_something), for Java types. However, if you write an aspect using the Java-style for a method name and a Ruby subclass of the Java type where the method is actually defined (i.e., the Ruby class doesn’t override the method), Aquarium acts like the JoinPoint is advised, but the advice is never actually called. Workaround: Use the Ruby-style name in this scenario.

So, there is still some work to do, but it’s promising that you can use an aspect framework in one language with another. A primary goal of Aquarium is to make it easy to write simple aspects. My hope is that people who might find AspectJ daunting will still give Aquarium a try.

The Quality of TDD

Uncle Bob — Sat, 16 Feb 2008 20:23:44 -0600

Kieth Braithwaite has made an interesting observation here. The basic idea is that code that has been written with TDD has a lower Cyclomatic Complexity per function compared to code that has not been written with TDD. If this is true then it could imply lower defects because of this.

Kieth’s metric takes in the code for an entire project and boils it down to a single number. His hypothesis is that a system written with TDD will always measure above a certain threshold, indicating very low CC; whereas systems written without TDD may or may not measure above that threshold.

Kieth has built a tool that you can get here that will generate this metric for most java projects. He and others have used this tool to measure many different systems. So far the hypothesis seems to hold water.

The metric can’t tell you if TDD was used; but it might just be able to tell you that it wasn’t used.

Crash Test Dummies

Uncle Bob — Mon, 11 Feb 2008 14:25:41 -0600

Even the most dedicated TDDer would not write a unit test for “Hello, world.” Even the most fervent FitNesse wonk would not write an acceptance test for a program that prints the first 10 squares. This implies that there are programs too simple for TDD. So where do we draw the line?

Accellerating towards a brick wall.

The issue, of course, is speed. For projects that are too simple to require unit tests, or acceptance tests, you can go a lot faster by not writing them. I can write HelloWorld or SquaresOfIntegers in much less than half the time that it would take me to write unit and acceptance tests for those programs. And I know they work because I can run them and manually validate them almost instantly.

For large projects the opposite is true; unit tests and acceptance tests make us go faster. The increase in speed comes from reduced debugging time and reduced ambiguity. When programmers follow the three laws of TDD, writing tests and code in a very short cycle, they reduce their debugging time by perhaps as much as 90%. Code written this way is much easier to understand since the tests act as little code examples. It is also much easier to change since the suite of tests act as instant fault detection.

Acceptance tests act as executable requirements. When Business Analysts and QA staff write these tests at the beginning of an iteration, they are producing unambiguous requirements. Programmers cannot help but implement the features as specified, since the tests will fail otherwise. So there is much less requirements thrashing and rework.

The cutoff for unit tests is probably around 10 lines of code. That is, it is probably faster to write 10 lines of code by writing unit tests first, than to just write the 10 lines and hope for the best. As an example of this, I recently wrote a dumb stack class. I found that I had to write the simple unit tests first in order to be sure that I had properly implemented the algorithm. Writing the stack without tests just left too much room for something to go wrong.

Acceptance tests are another matter. You don’t really feel the benefit of acceptance tests for projects below a few thousand lines of code. The problem is that in such projects manual testing is cheap. The programmers and customers can just run the program for 15 minutes and prove that it works. Any team caught in a project of this size will immediately feel the drag imposed by writing automated acceptance tests. They’ll look at how much they could get done without them. They’ll envy the time lost to those tests. And they may make an obvious, but possibly catastrophic choice.

The problem is that as the project grows, the manual testing time gets longer. It gets harder and harder to remember everything that needs to be tested. So manual test plans have to be written. Following those plans becomes ever more tedious and time consuming, so customers hand this off to a QA team.

The QA team requires staffing; and as the project grows that staffing must increase. Test time also increases, from minutes to days to weeks. Eventually the cost of running manual QA tests becomes significant enough to be a large number on the project cost spreadsheet. Managers resist the addition of more QA staff and suggest working “smarter”.

With staffing throttled, QA gets ever more stressed. Development seldom delivers builds on time, yet the release date cannot move; so QA must make up the slack. They do this by choosing which tests to run and which tests to assume still work. They must quickly execute thousands of horribly tedious manual tests and interpret the results. Under pressure to deliver, the choices they make inevitably, and ironically, reduce the quality of the result.

The name of this blog is a metaphor for a team on a smallish project who decides that they can go faster if they don’t write automated acceptance tests. They’re right—for now.

Unit Testing C and C++ ... with Ruby and RSpec!

Dean Wampler — Mon, 04 Feb 2008 22:08:00 -0600

If you’re writing C/C++ code, it’s natural to write your unit tests in the same language (or use C++ for your C test code). All the well-known unit testing tools take this approach.

I think we can agree that neither language offers the best developer productivity among all the language choices out there. Most of us use either language because of perceived performance requirements, institutional and industry tradition, etc.

There’s growing interest, however, in mixing languages, tools, and paradigms to get the best tool for a particular job. <shameless-plug>I’m giving a talk March 7^th at SD West on this very topic, called Polyglot and Poly-Paradigm Programming </shameless-plug>.

So, why not use a more productive language for your C or C++ unit tests? You have more freedom in your development chores than what’s required for production. Why not use Ruby’s RSpec, a Behavior-Driven Development tool for acceptance and unit testing? Or, you could use Ruby’s version of JUnit, called Test::Unit. The hard part is integrating Ruby and C/C++. If you’ve been looking for an excuse to bring Ruby (or Tcl or Python or Java or…) into your C/C++ environment, starting with development tasks is usually the path of least resistance.

I did some experimenting over the last few days to integrate RSpec using SWIG (Simplified Wrapper and Interface Generator), a tool for bridging libraries written in C and C++ to other languages, like Ruby. The Ruby section of the SWIG manual was very helpful.

My Proof-of-Concept Code

Here is a zip file of my experiment: rspec_for_cpp.zip

This is far from a complete and working solution, but I think it shows promise. See the Current Limitations section below for details.

Unzip the file into a directory. I’ll assume you named it rspec_for_cpp. You need to have gmake, gcc, SWIG and Ruby installed, along with the RSpec “gem”. Right now, it only builds on OS X and Linux (at least the configurations on my machines running those OS’s – see the discussion below). To run the build, use the following commands:

    
        $ cd rspec_for_cpp/cpp
        $ make

You should see it finish with the lines

    
        ( cd ../spec; spec *_spec.rb )
        .........

        Finished in 0.0***** seconds

        9 examples, 0 failures

Congratulations, you’ve just tested some C and C++ code with RSpec! (Or, if you didn’t succeed, see the notes in the Makefile and the discussion below.)

The Details

I’ll briefly walk you through the files in the zip and the key steps required to make it all work.

`cexample.h`

Here is a simple C header file.

    
        /* cexample.h */
        #ifndef CEXAMPLE_H
        #define CEXAMPLE_H
        #ifdef __cplusplus
         extern "C" {
        #endif
        char* returnString(char* input);
        double returnDouble(int input);
        void  doNothing();

        #ifdef __cplusplus
         }
        #endif
        #endif

Of course, in a pure C shop, you won’t need the #ifdef __cplusplus stuff. I found this was essential in my experiment when I mixed C and C++, as you might expect.

`cpp/cexample.c`

Here is the corresponding C source file.

    
        /* cexample.h */

        char* returnString(char* input) {
            return input;
        }

        double returnDouble(int input) {
            return (double) input;
        }

        void  doNothing() {}

`cpp/CppExample.h`

Here is a C++ header file.

    
        #ifndef CPPEXAMPLE_H
        #define CPPEXAMPLE_H

        #include <string>

        class CppExample 
        {
        public:
            CppExample();
            CppExample(const CppExample& foo);
            CppExample(const char* title, int flag);
            virtual ~CppExample();

            const char* title() const;
            void        title(const char* title);
            int         flag() const;
            void        flag(int value);

            static int countOfCppExamples();
        private:
            std::string _title;
            int         _flag;
        };

        #endif

`cpp/CppExample.cpp`

Here is the corresponding C++ source file.

    
        #include "CppExample.h" 

        CppExample::CppExample() : _title("") {}
        CppExample::CppExample(const CppExample& foo): _title(foo._title) {}
        CppExample::CppExample(const char* title, int flag) : _title(title), _flag(flag) {}
        CppExample::~CppExample() {}

        const char* CppExample::title() const { return _title.c_str(); }
        void        CppExample::title(const char* name) { _title = name; }

        int  CppExample::flag() const { return _flag; }
        void CppExample::flag(int value) { _flag = value; }

        int CppExample::countOfCppExamples() { return 1; }

`cpp/example.i`

Typically in SWIG, you specify a .i file to the swig command to define the module that wraps the classes and global functions, which classes and functions to expose to the target language (usually all in our case), and other assorted customization options, which are discussed in the SWIG manual. I’ll show the swig command in a minute. For now, note that I’m going to generate an example_wrap.cpp file that will function as the bridge between the languages.

Here’s my example.i, where I named the module example.

    
        %module example
        %{
            #include "cexample.h" 
            #include "CppExample.h"    
        %}
        %include "cexample.h" 
        %include "CppExample.h"

It looks odd to have header files appear twice. The code inside the %{...%} (with a ’#’ before each include) are standard C and C++ statements, etc. that will be inserted verbatim into the generated “wrapper” file, example_wrap.cpp, so that file will compile when it references anything declared in the header files. The second case, with a ’%’ before the include statements¹, tells SWIG to make all the declarations in those header files available to the target language. (You can be more selective, if you prefer…)

Following Ruby conventions, the Ruby plugin for SWIG automatically names the module with an upper case first letter (Example), but you use require 'example' to include it, as we’ll see shortly.

Building

See the cpp/Makefile for the gory details. In a nutshell, you run the swig command like this.

    
        swig -c++ -ruby -Wall -o example_wrap.cpp example.i

Next, you create a dynamically-linked library, as appropriate for your platform, so the Ruby interpreter can load the module dynamically when required. The Makefile can do this for Linux and OS X platforms. See the Ruby section of the SWIG manual for Windows specifics.

If you test-drive your code, which tends to drive you towards minimally-coupled “modules”, then you can keep your libraries and build times small, which will make the build and test cycle very fast!

`spec/cexample_spec.rb` and `spec/cppexample_spec.rb`

Finally, here are the RSpec files that exercise the C and C++ code. (Disclaimer: these aren’t the best spec files I’ve ever written. For one thing, they don’t exercise all the CppExample methods! So sue me… :)

    
        require File.dirname(__FILE__) + '/spec_helper'
        require 'example'

        describe "Example (C functions)" do
          it "should be a constant on Module" do
            Module.constants.should include('Example')
          end
          it "should have the methods defined in the C header file" do
            Example.methods.should include('returnString')
            Example.methods.should include('returnDouble')
            Example.methods.should include('doNothing')
          end
        end

        describe Example, ".returnString" do
          it "should return the input char * string as a Ruby string unchanged" do
            Example.returnString("bar!").should == "bar!" 
          end  
        end

        describe Example, ".returnDouble" do
          it "should return the input integer as a double" do
            Example.returnDouble(10).should == 10.0
          end
        end

        describe Example, ".doNothing" do
          it "should exist, but do nothing" do
            lambda { Example.doNothing }.should_not raise_error
          end
        end

and

    
    require File.dirname(__FILE__) + '/spec_helper'
    require 'example'

    describe Example::CppExample do
      it "should be a constant on module Example" do
        Example.constants.should include('CppExample')
      end
    end

    describe Example::CppExample, ".new" do
      it "should create a new object of type CppExample" do
        example = Example::CppExample.new("example1", 1)
        example.title.should == "example1" 
        example.flag.should  == 1
      end
    end

    describe Example::CppExample, "#title(value)" do
      it "should set the title" do
        example = Example::CppExample.new("example1", 1)
        example.title("title2")
        example.title.should == "title2" 
      end
    end

    describe Example::CppExample, "#flag(value)" do
      it "should set the flag" do
        example = Example::CppExample.new("example1", 1)
        example.flag(2)
        example.flag.should == 2
      end
    end

If you love RSpec like I do, this is a very compelling thing to see!

And now for the small print:

Current Limitations

As I said, this is just an experiment at this point. Volunteers to make this battle-ready would be most welcome!

General

The Example `Makefile` File

It Must Be Hand Edited for Each New or Renamed Source File

You’ve probably already solved this problem for your own make files. Just merge in the example Makefile to pick up the SWIG- and RSpec-related targets and rules.

It Only Knows How to Build Shared Libraries for Mac OS X and Linux (and Not Very Well)

Some definitions are probably unique to my OS X and Linux machines. Windows is not supported at all. However, this is also easy rectify. Start with the notes in the Makefile itself.

The `module.i` File Must Be Hand Edited for Each File Change

Since the format is simple, a make task could fill a template file with the changed list of source files during the build.

Better Automation

It should be straightforward to provide scripts, IDE/Editor shortcuts, etc. that automate some of the tasks of adding new methods and classes to your C and C++ code when you introduce them first in your “spec” files. (The true TDD way, of course.)

Specific Issues for C Code Testing

I don’t know of any other C-specific issues, so unit testing with Ruby is most viable today for C code. Although I haven’t experimented extensively, C functions and variables are easily mapped by SWIG to a Ruby module. The Ruby section of the SWIG manual discusses this mapping in some detail.

Specific Issues for C++ Code Testing

More work will be required to make this viable. It’s important to note that SWIG cannot handle all C++ constructs (although there are workarounds for most issues, if you’re committed to this approach…). For example, namespaces, nested classes, some template and some method overloading scenarios are not supported. The SWIG manual has details.

Also, during my experiment, SWIG didn’t seem to map const std::string& objects in C++ method signatures to Ruby strings, as I would have expected (char* worked fine).

Is It a Viable Approach?

Once the General issues listed above are handled, I think this approach would work very well for C code. For C++ code, there are more issues that need to be addressed, and programmers who are committed to this strategy will need to tolerate some issues (or just use C++-language tools for some scenarios).

Conclusions: Making It Development-Team Ready

I’d like to see this approach pushed to its logical limit. I think it has the potential to really improve the productivity of C and C++ developers and the quality of their test coverage, by leveraging the productivity and power of dynamically-typed languages like Ruby. If you prefer, you could use Tcl, Python, even Java instead.

License

This code is complete open and free to use. Of course, use it at your own risk; I offer it without warranty, etc., etc. When I polish it to the point of making it an “official” project, I will probably release under the Apache license.

¹ I spent a lot of time debugging problems because I had a ’#’ where I should have had a ’%’! Caveat emptor!

The Post-it® Notes Test for UML Diagrams

Dean Wampler — Tue, 22 Jan 2008 15:58:00 -0600

A lot of teams require their developers to document their designs in UML, using Visio or another tool, before they can start coding.

Of course, this is not at all Agile. For one thing, the design is likely to change quite a bit as you learn while coding. Hardly anyone returns to the diagrams and updates them. Now they are lies, because they make claims about the designs that aren’t true.

UML still has a place in agile projects, of course. It’s a great tool for brainstorming design ideas. So, how do you decide when a diagram is worth keeping and therefore, worth maintaining? Here’s a little strategy that I recommend.

Draw the diagram during those brainstorming sessions on a white board or a poster-sized Post-it® Note. Drawing it this way means you have invested almost no additional effort, beyond the brainstorming itself, to create the diagram. Also, you won’t feel bad about lost work if you eventually throw it away.

Leave the diagram on the wall for everyone to see while they implement the design.

By the time the note is falling off the wall or the dry-erase marker is wearing off the white board, you’ll know if the ideas are still relevant or completely obsolete.

If they are obsolete, you can erase the board or toss the paper. If they are still relevant, and probably changed somewhat, you now know that the diagram is worth preserving. Go ahead and spend the time to create an updated, more permanent version in your drawing tool (but don’t spend too much time!).

Generated Tests and TDD

Uncle Bob — Thu, 10 Jan 2008 13:59:30 -0600

TDD has become quite popular, and many companies are attempting to adopt it. However, some folks worry that it takes a long time to write all those unit tests and are looking to test-generation tools as a way to decrease that burden.

The burden is not insignificant. FitNesse, an application created using TDD, is comprised of 45,000 lines of Java code, 15,000 of which are unit tests. Simple math suggests that TDD increases the coding burden by a full third!

Of course this is a naive analysis. The benefits of using TDD are significant, and far outweigh the burden of writing the extra code. But that 33% still feels “extra” and tempts people to find ways to shrink it without losing any of the benefits.

Test Generators.

Some folks have put their hope in tools that automatically generate tests by inspecting code. These tools are very clever. They generate random calls to methods and remember the results. They can automatically build mocks and stubs to break the dependencies between modules. They use remarkably clever algorithms to choose their random test data. They even provide ways for programmers to write plugins that adjust those algorithms to be a better fit for their applications.

The end result of running such a tool is a set of observations. The tool observes how the instance variables of a class change when calls are made to its methods with certain arguments. It notes the return values, changes to instance variables, and outgoing calls to stubs and mocks. And it presents these observations to the user.

The user must look through these observations and determine which are correct, which are irrelevant, and which are bugs. Once the bugs are fixed, these observations can be checked over and over again by re-running the tests. This is very similar to the record-playback model used by GUI testers. Once you have registered all the correct observations, you can play the tests back and make sure those observations are still being observed.

Some of the tools will even write the observations as JUnit tests, so that you can run them as a standard test suite. Just like TDD, right? Well, not so fast…

Make no mistake, tools like this can be very useful. If you have a wad of untested legacy code, then generating a suite of JUnit tests that verifies some portion of the behavior of that code can be a great boon!

The Periphery Problem

On the other hand, no matter how clever the test generator is, the tests it generates will always be more naive than the tests that a human can write. As a simple example of this, I have tried to generate tests for the bowling game program using two of the better known test generation tools. The interface to the Bowling Game looks like this:

  public class BowlingGame {
    public void roll(int pins) {...}
    public int score() {...}
  }

The idea is that you call roll each time the balls gets rolled, and you call score at the end of the game to get the score for that game.

The test generators could not randomly generate valid games. It’s not hard to see why. A valid game is a sequence of between 12 and 21 rolls, all of which must be integers between 0 and 10. What’s more, within a given frame, the sum of rolls cannot exceed 10. These constraints are just too tight for a random generator to achieve within the current age of the universe.

I could have written a plugin that guided the generator to create valid games; but such an algorithm would embody much of the logic of the BowlingGame itself, so it’s not clear that the economics are advantageous.

To generalize this, the test generators have trouble getting inside algorithms that have any kind of protocol, calling sequence, or state semantics. They can generate tests around the periphery of the classes; but can’t get into the guts without help.

TDD?

The real question is whether or not such generated tests help you with Test Driven Development. TDD is the act of using tests as a way to drive the development of the system. You write unit test code first, and then you write the application code that makes that code pass. Clearly generating tests from existing code violates that simple rule. So in some philosophical sense, using test generators is not TDD. But who cares so long as the tests get written, right? Well, hang on…

One of the reasons that TDD works so well is that it is similar to the accounting practice of dual entry bookkeeping. Accountants make every entry twice; once on the credit side, and once on the debit side. These two entries follow separate mathematical pathways. In the end a magical subtraction yields a zero if all the entries were made correctly.

In TDD, programmers state their intent twice; once in the test code, and again in the production code. These two statements of intent verify each other. The tests, test the intent of the code, and the code tests the intent of the tests. This works because it is a human that makes both entries! The human must state the intent twice, but in two complementary forms. This vastly reduces many kinds of errors; as well as providing significant insight into improved design.

Using a test generator breaks this concept because the generator writes the test using the production code as input. The generated test is not a human restatement, it is an automatic translation. The human states intent only once, and therefore does not gain insights from restatement, nor does the generated test check that the intent of the code was achieved. It is true that the human must verify the observations, but compared to TDD that is a far more passive action, providing far less insight into defects, design and intent.

I conclude from this that automated test generation is neither equivalent to TDD, nor is it a way to make TDD more efficient. What you gain by trying to generate the 33% test code, you lose in defect elimination, restatement of intent, and design insight. You also sacrifice depth of test coverage, because of the periphery problem.

This does not mean that test generators aren’t useful. As I said earlier, I think they can help to partially characterize a large base of legacy code. But these tools are not TDD tools. The tests they generate are not equivalent to tests written using TDD. And many of the benefits of TDD are not achieved through test generation.

So just what does synchronized do?

Brett Schuchert — Thu, 03 Jan 2008 21:27:00 -0600

Synopsis

Using synchronized turns a huge state space into a comparatively small one.

Normally, the light from a star radiates out in all directions. But what happens when a star collapses? There are several possibilities depending on the mass of the star. Our sun will turn into a red giant and then later turn into a white dwarf, giving out light from its accumulated heat for many years after living on Earth has become unbearable; mostly because of all the traffic.

If the mass of the star is around 10x our star, its destiny is just a bit different. It will begin to collapse. Along the way, it will probably have some temporary reprieves from its ultimate fate, to become a Neutron star, by consuming other heavier elements such as carbon and helium . However, the writing is on the wall. The center of star, already an extreme environment, becomes even more so; eventually, the pressure from the collapsing of material into the center of the star results in a massive explosion known as a nova or super nova – depending on the mass of the star. What is left is a neutron star. A neutron star is the heart of a pulsar, able to spin at amazing rates without pulling itself apart.

Another option is a black hole. A black hole is an often used metaphor for either the effort expended on support/fixing/updating a big ball of mud or what the development effort appears to be in most of the time. Either way, it is not considered a good thing if you’re anywhere in the vicinity.

I’m a pessimist by nature. I see a project in decline, sucking in other resources, like a black holes eating neighboring galaxies, snuffing the light out of stars, just like projects can suck the life force out of individuals.

However, energy can escape from a black hole, eventually leading to the demise of the black hole. Energy can escape along the axis of rotation in either direction. In a sense, if we see the expansion of the black hole as a bad thing, anything that diminishes a black hole is a good thing. Stephen Hawking theorized that at the event horizon of black holes, fluctuations in the space can cause the spontaneous creations of pairs of particles and their anti particles. This is another way it can appear that a black hole is losing energy since it gives off particles while the opposite particle goes into the black hole.

Now I’m off track. I wanted to talk about how synchronized give a similar effect as the massive gravity of black hole. But instead of curving space-time, they invert the possible state space of code in the presence of threads. Let’s take a look.

Here’s a simple class:

public class X {
   int value;

   public synchronized void incrementValue() {
        ++value;
    }
}

Let’s say, for the sake of argument, two threads are trying to execute the single method in this class. If the method is written like this:

   public synchronized void incrementValue() {
        ++value;
    }

Then there are only two orderings, Thread 1/Thread 2, or Thread 2/Thread 1. All of those paragraphs at the top were to make this analogy.

OK, so if we use synchronized that that’s the black hole. The number of possible paths through the code for N threads is N!. What’s the opposite? What happens if we take off the keyword and then run two threads through it? In how many ways can that single line of code get executed?

Are you ready for it?

3,432 different ways

Really? Well at least conceptually. In practice, the Just In Time compiler will probably turn the JVM instructions into native code, so the actual number of individual steps will probably be less. But there really are that many paths through the system. You might ask how?

That single line of code is represented in byte code as 7 lines. I could show you the disassembly, but really, it’s 7. You can convince yourself by having a look at the byte-code using a nice byte-code viewer plug-in for eclipse.

What happens if we change the int to a long? We still have 7 lines of byte-code, but with different instructions. We have a total of 4 long reads/writes. Each long read/write requires two individual 32-bit operations – or at lest the JVM can implement it that way. The actual number of atomic operations after the Just In Time compiler has had its way with your byte-code will probably be less, but if we stick to byte-codes and the Java memory model, the number 3,432 is now a whopping 705,432!

Let’s extend that just a bit more. What if you have several lines of code? Each line of Java results in many more lines of byte-code. What if we have have 3 lies of Java? We probably have something like 21 lines of byte-code. How many paths of execution would 21 lines of byte-code executed by two threads have? 538,257,874,440!

Where the first program ended up being a white dwarf, the version using longs was a nova. I’m not sure what to call the thee line Java method, maybe a hyper-nova?

In practice, the actual number of paths through a method will end up being smaller. And, most of the paths do not have any negative side-effects. The problem, however, is that there are a small number of hard to find paths of execution that do cause problems and finding them is like looking for a needle in a haystack.

Remember what adding the keyword synchronized did? If we use it on a three line method, it turns 538,257,874,440 into 2. It collapses the number of paths into 2 for two threads. And 2 is less than 538,257,874,440, for even vary large values of 2.

How did I derive at these magic numbers? Two ways.

I wanted to know how many possible orderings there were for 7 instructions and two threads. I knew it had to do with combinations and permutations but I just wasn’t smart enough to figure it out. Searching on Google didn’t do much for me either. So I decided to write a program to numerically derive the result. I tested my way into a program that would calculate the result.

The basic algorithm is:

Determine all possible orderings of each of the unique “nodes” in a system. For example, if I have 2 threads and 2 instructions, I can generate a natural key (ugh!) to represent each of the different combinations of threads and instructions: T1S1 T1S2 T2S1 2S2.
Produce all valid permutations of these four “nodes”.
Remove any that have invalid orderings.
Count the results.

Many of my first implementations for some of these steps were exercises in writing really inefficient code! But having those tests made it really easy to swap out better algorithms when my brain caught up with what I was doing.

What is an invalid ordering? The code generated by the single line using the pre increment operator has 7 steps. Those 7 steps are in a single sequence with no looping and no conditionals. That’s like saying A is always before B. So if we generate a list of orderings that end up including that combination, we’ve generated an ordering that cannot happen.

So you can describe your possible state space in a few steps:

Register a node for each thread, step combination. 7 threads, 4 steps = 28 nodes.
Register dependences between steps. For example, if I have seven steps and two threads executing the same sequence, I could describe this as: T1S1 -> T1S2 -> T1S3, etc. And for the second thread: T2S1 -> T2S2 -> T2>S3. Since the name is arbitrary, I just used letters, so in fact I had: a -> b -> c -> d -> e -> f -> g for one set of dependencies and h -> i -> j -> k -> l -> m -> o

In fact, here’s the test I used to describe that exact space:

    @Test
    public void twoThreadsAndSevenSteps() {
        calculator = new PathCalculator(14);
        createLine('a', 'b', 'c', 'd', 'e', 'f', 'g');
        createLine('h', 'i', 'j', 'k', 'l', 'm', 'n');
        start();
        int totalPaths = calculator.totalPaths();
        stop(2, 4, totalPaths);
    }

By the time I got to this point, I wrote some “tests” using JUnit as a harness to run my code and display timing information and a summary of the

Time:        4ms -- Threads:  3, Opcodes:  2, Total paths:      90
Time:        1ms -- Threads:  2, Opcodes:  3, Total paths:      20
Time:       54ms -- Threads:  4, Opcodes:  2, Total paths:    2520
Time:        3ms -- Threads:  2, Opcodes:  4, Total paths:      70
Time:       61ms -- Threads:  2, Opcodes:  5, Total paths:     252
Time:     1248ms -- Threads:  2, Opcodes:  6, Total paths:     924
Time:    12819ms -- Threads: 10, Opcodes:  1, Total paths: 3628800

I then used this information to fit a cure to my data and derive a formula for various combinations. For 2 threads, I came up with the following formula. If you let N be the number of steps:

Total Paths = 252 * 3.6667 ^N-5

The morning after, Uncle Bob sent an email, showing me his work on how he calculated.

If you let:

T be the number of threads
N be the number of steps

Total = (N*T)! / N! ^T

When I plugged numbers into his formula, they fit my formula. His formula generated an exact result as opposed to my estimate based on fitting a curve. Another thing his solution has going for it is the algorithm I came up with, I believe, is NP-complete. The net effect was I that while I was able to calculate result for 2 threads and 6 steps, 2 threads and 7 steps took a long time before running out of memory with a 2GB heap space. Bob’s formula gave me a so-called verifier (as did my formula – only Bob’s was better) and the algorithm seems to grow based on factorials and not polynomials.

What’s the point of all of this? The single, one-line program, is a demonstration of why going from one thread to two threads makes writing/debugging/testing programs so much harder. It also numerically demonstrates why the single responsibility principle is even more important when we start writing concurrent programs. The more you put in one place, the large the state space. Larger state spaces:

Make it harder to find that one or few number of paths that can cause your program to die, under load, in production, while you are on vacation
Make your efforts to test (which is a sampling technique anyway) for concurrency issues all the more difficult
Make it harder to find what might actually have caused a failure once you’ve notice it.

In a later post, I’ll describe how we can improve our chances of covering this huge space of potential orderings.

If you’ve made it this far, wow! Congratulations!