Several colleagues and I really groked Fusion. It started with system sequence diagrams showing interactions much like the specification I mentioned above. It also described a difference between analysis and design (and if Uncle Bob reads this, he’ll probably have some strong words about so-called object-oriented analysis, well this was 15 years ago… though I still there there is some value to be found there). Anyway, this is mostly about system sequence diagrams so I won’t say much more about that excellent process.

Each system sequence diagram represented a scenario. To represent many scenarios, Fusion offered a BNF-based syntax to express those various scenarios. (I understand that this was also because for political reasons within HP they were not allowed to include a state model, but I don’t know if that is true or not.) For several years I practiced Fusion and really I often revert back to that if I’m not trying to do anything in particular.

Spending a little time up front thinking a little about the logical interaction between the system and its various actors helps me get a big picture view of the system boundary and its general flow. I have also found it helps others as well, but your mileage may vary.

So when I viewed their protocol specification, it really brought back some good memories. And in fact, that’s how we decided to look at the problem.

(What follows is highly-idealized)

We reviewed their specification and decided we’d try to work through the initialization sequence and then work through one sequence that involved “completing a simple job.” I need to keep this high level to keep the identity of the company a secret.

There was prior work and we kept that in mind but really started from scratch. In our very first attempt, there had been some work done along the lines of using the Command pattern, so we started there. Of course, once we did our first command, we backed off and when with a more basic design that seemed to fit the complexity a bit better (starting with the command pattern at the beginning is an example of solution-problemming to use a Weinberg term – and one of the reasons I’m sometimes skeptical when people start talking in patterns).

We continued working from the request coming into the system and working its way through the system. Along the way, we wrote unit tests, driven by our end goal of trying to complete a simple job and guided by the single responsibility principle. As we thought about the system, there were several logical steps:

• Receive a message from the outside as some array of bytes
• Determine the “command” represented by the bytes
• Process the parameters within the command
• Issue a message to the simulator
• Create a logical response
• Format the logical response into the underlying protocol
• Send the response back

At the time, they were considering using JNI, so we spent just over a day validating that we could communicate bi-directionally, maintaining a single process space.

Along the way we moved from using hand-rolled test doubles to using JMock 2 to create mock objects. I mentioned this to friend of mine who lamented that there are several issues using a mock-based approach:

• It is easy to end up with a bunch of tested objects but no fully-connected system
• Sharing setup between various mocks is difficult and often not done so there’s a lot of violation of DRY
• You have to learn a new syntax

We accepted learning a new syntax because it was deemed less painful than maintaining existing hand-rolled test doubles (though there are several reasonable solution for that, ask if you want to know what it is). There is the issue of sharing setup on mocks, but we did not have enough work yet to really notice that as a problem. However, they were at least aware of that and we briefly discussed how to share common expectation-setting (it’s well supported).

Finally, there’s the issue of not having a fully connected system. We knew this was an issue so we started by writing an integration test using JUnit. We needed to design a system that:

• Could be tested up to but excluding the JNI stuff
• Could be configured to stub out JNI or use real JNI
• Was easily configurable
• Was automatically configured by C++ (since it was a C++ process that was started to get the whole system in place)

We designed that (15 minute white-board session), coded it and ended up with a few integration tests. Along the way, we built a simple factory for creating the system fully connected. That factory was used both in tests as well as by the JNI-based classes to make sure that we had a fully-connected systems when it was finally started by C++.

Near the end, we decided we wanted to demonstrate asynchronous computation, which we did using tests. I stumbled a bit but we got it done in a few hours. We demonstrated that the system receiving messages from the outside world basically queued up requests rather than making the sender wait synchronously (we demonstrated this indirectly – that might be a later blog post – let me know if you’re interested).

By the way, that was the first week. These guys were good and I had a great time.

There was still a little work to be done on the C++ side and I only had a week, so I asked them to keep me posted. The following Tuesday they had the first end-to-end interaction, system initialization.

By Wednesday (so 3 business days later), they had a complete demonstration of end-to-end interaction with a single, simple job finishing. Not long after that they demonstrated several simple jobs finishing. The next thing on their list? Completing more complex jobs, system configuration, etc.

However, it all goes back to having a well-defined protocol. After we had one system interaction described end-to-end, doing the next thing was easier:

• Select a system interaction
• List all of the steps it needs to accomplish (some requests required a response, some did not)
• Write unit tests for each “arm” of the interaction

So they had a very natural way to form the backlog:

Select a set of end-to-end interactions that add value to the user of the system

They also had an easy way to create a sprint backlog:

For each system-level interaction, enumerate all of its steps and then add implementing those steps as individual back-log items

Now some of those individual steps will end up being small (less than an hour) but some will be quite large when they start working with variable parameters and commands that need to operate at a higher priority.

But they are well on their way and I was reminded of just how much I really enjoyed using Fusion.

Does that mean all meetings need to be that way?

No

Why were we told to write agendas?

To keep focus and know we’re making progress

What purpose does a meeting serve in the delivery of value to someone who’s got $$ to spend?

That varies by the meeting

What do tests written first (unit or acceptance) accomplish?

Exress how we know somethign has worked

Can we do this with meetings? If we did, what might it look like? Would the very act of trying to express success critera for a meeting have a profound effect on how a meeting progresses? Isn’t this really just Covey’s 2nd habit (begin with the end in mind).

What if you wanted to try? How might you begin?

It’s all about he benjamins.

The first thing you need to ask is how what you’re going to do in a given meeting directly or indireclty adds value. Of coruse, to do that you need to know what is valuable.

Suppose, for example, you need to have a traditional requirements meeting of some sort (there is a traditional requirements meeting, it is usually bogged down in implementation details or it’s stuck in “how” mode versus “what” mode – the more you talk about database columns, the more you’re deep into requirements … NOT!).

OK, so take a step back. You think you need this meeting on requirements.

Why?

So we can figure out what this feature needs to do

Why?

So we know what to what to write, what schema changes we need

Why?

So we can implement it

Why?

Because customer X has a contract and we’ll be in breech of contract if we don’t write it

AH! That sounds like we’re getting close to something valuable. We have some feature that’s been promised to a current customer. OK, so before we go any further, that information needs to be part of the context for the meeting.

What’s been promised? What does the customer think has been promised?

If you cannot answer this, you are in deep trouble

OK, so you have promised some feature. Apparently that feature is ill defined (or you wouldn’t need the requirements meeting). So, what problem does that feature address?

If you know this, then you have something to grow. Maybe your first attempt at an acceptance test for your meeting is something like this:

This meeting is a success if we have described scenarios that cover what this feature will do for at the top three uses of this feature

OK, so now here’s your agend:

• This requirements meeting is to address feature X
• We need to discuss this because customer X has been promised this feature and if we do not do it by (insert a real date here), we are in breech of contract
• This meeting will be a success if we can describe scenarios covering the top three uses of this feature

That’s a start. I can think of ways to improve that. Here are a few examples:

• How do we define top 3? Is that something we know or something we need to accomplish?
• What do we mean by “describe scenarios” does it include:
  • Acceptance tests
  • One or more stories (or maybe it is just a scenario in a use case, pick your requirements coolaid)
  • Used by whom? Are there different roles? Is that important?
  • What about a timebox? (Maybe that’s implicit because it’d be an electronic request)
  • ...

Here’s the thing, do you think you’d be more inclined to think a meeting described thusly would have some value?

Even so, I’ve wondered about this quite a bit. Here are some of the reasons I’ve heard to justify this practice:

• It expresses the design intent
• It’s for performance
• The code as written cannot be safely overridden.

(Before I go any further, I am not arguing against the use of sealed or final. I will say that in my practice, it has to be justified to be used rather than the other way around.)

Back to the reasons. I generally don’t buy these reasons. Each may be perfectly valid but none of them stand on their own; each requires further discussion.

I’ve had those discussions with various people and most of the time they really don’t hold water. Here are some quip-y responses to each of them:

• It expresses design intent – you don’t know how to design
• It’s for performance – do you have any idea how a VM works?
• The code as written cannot be safely overridden – So the code is so badly written that nobody understands it?

Again, let me be clear, there are good reasons to seal things. My point is that while there are good reasons, most of the time I’ve seen the habit is a form of cargo culting; monkey see, monkey do.

This observation sat around for some time until I read some stuff in a book called CLR via C# (good reading if you want to better understand the CLR, and if you’re developing code for a virtual machine, I think it’s a good idea to understand the virtual machine at least a little).

Reading that book and my observations led to my theory:

People are sealing as much as they are because virtual table binding is done way to early.

The thing I read in the book I mentioned above relates to when virtual methods are bound. This applies to 2.0 version of the CLR, but I don’t think there’s any change for the latest version of the CLR and this seems like a pretty “hard” design decision. This book mentions that binding of virtual methods is performed at compile time.

I’m not making a distinction between compile time and link time for this discussion, but I’ll be a little more precise. The generated code stored in an assembly “knows” which virtual function it is invoking because in the generated byte code is an index into a virtual table.

How is this a problem?

I write a base class and ship it. You write a derived class (OK, maybe you use delegation, good for you, but the problem still remains). Your coupling to my class involves using a method with the virtual keyword (Java programmers, you don’t have to do this, regular methods [not static, not a constructor, ...] have the potential for dynamic dispatch unless you declare them final). At compile time, your object module (which eventually resides in an assembly that will get loaded and executed some time later) knows that it wants to call the 3rd virtual method in the v-table.

So far, so good.

Next, I add a new virtual method to my class and ship a new assembly.

Question. To use this new assembly, do you need to recompile?

Yes.

What?

Yes. What happens if my newly added method is the first virtual method? Then your compiled code that is happily living in a peaceful assembly somewhere has a stale reference. It thinks it wants to run the 3rd method in the virtual table but now it really should be using the 4th virtual method.

This is similar to a caching issue or premature optimization. The calling code has a compile-time generated index into the v-table. Why is this? Is it for performance? Not a good argument, this pre-calculation is irrelevant with modern JIT compiling techniques. I think the reason it is done this way is that C++ does things this way. C++ needs to because of its execution model (there’s not virtual machine between the code and its execution).

No big deal, you think, I’ll just recompile my code.

That will fix the problem. Or will it?

Say you’re stuck using an old version of some assembly that uses an old version of another assembly that you also use. Do you recompile both your code and that other assembly? You’ll have to but you might forget. Or you might not be able to.

.Net handles this by having an excellent dependency system to track which versions of which assemblies work with with other versions of assemblies. They need this to address problems introduced by the decision to perform such early binding.

Simply put, this is an unfortunate decision that I believe was more based on precedence or history than on sound design decisions.

This is not what happens in the Java Virtual Machine (JVM).

In the JVM, the binding is done later—much later. It is possible that my class got loaded and JIT’ed so that all of the methods that could be virtually invoked (not final) are in fact not virtually invoked. So far, no subclasses have been loaded, so there’s been no need. After my system has been running for a few weeks, a new class (a subclass) gets loaded and invalidates the JIT’d version of my class. No problem, the system will dump the old version and JIT it again, on the fly.

Even if there is a need for virtual dispatch, it might be that most of the time I use one version of method for a particularly heavily used subclass. The JVM might inline the common case and virtually dispatch the others.

Bottom line: what’s being done with JVM’s is indistinguishable from black magic for most of us.

I can use new JAR files and not have to worry about recompiling to remove stale v-table references. The method binding will just work. (This does not remove all problems with changing versions of JAR’s but it gets rid of one persnickety one.)

But it goes deeper. As I mentioned above, I’ve noticed several Microsoft shops that use a lot of sealed classes or avoid using virtual methods because “they perform poorly”.

OK, let’s talk about method invocation for just a second (or make that pico second). On my machine, I timed virtual method dispatch. I should check my numbers, but what I got was that virtual method dispatch was taking around 450 pico-seconds. So you can invoke roughly 2.2 Billion methods per second.

So:

• Virtual method dispatch is not that expensive
• The JVM can address caching issues with code analysis and inlining
• You only pay the price if your design requires it
• The people writing the JVM are better and micro optimization than certainly me, and I’d say most people

OK, but so it doesn’t help with performance. Is there a benefit to leaving things unsealed and using more virtual methods?

• Testability
• Maintainability
• Flexibility
• Extensibility

Look up the percentage of time spent “building” a system versus “maintaining” a system. The percentage has been on the rise, and it’s 80 – 95% typically. 80% of the cost of development is spent maintaining code. (I believe a lot of this percentage has to do with the definition of “done” – but that’s a whole other discussion).

You might think that locking things down make them more stable and that stability leads to maintainability.

That’s not really the case.

Things change. Requirements change.

Locking down a class to keep it from breaking is like signing off on requirements to keep them from changing. Neither thing is really a good idea.

I CAN lock down my classes in a sense. I can use tests to describe my assumptions so the class is locked down so long as we all agree that we’ll keep the tests passing. That is a fine level of granularity.

But I digress.

Back to the whole sealed thing. I was wondering why so many places that use Microsoft stuff seal their classes and do not use virtual methods (in my experience, it’s about 50%). I then read that thing about the binding of virtual methods at compile time. And then about 2 weeks later it hit me.

If you use virtual methods and you have a messed up build system, then you’ll get strange behavior. Sealing things makes those surprising behaviors go away, so seal stuff.

It makes sense. This is something that C++ was notorious about. You hoped you’d get a segmentation violation but often the wrong method could get called and the system continued to run…

Unfortunately, the “solution” – sealing – is attacking a symptom, not the actual problem. This is where following the 5-why’s of Toyota would have been a good idea.

“Seal your classes” Why?

• Because they will be more stable. More safe/it expresses my design. Why?
• Because my design says that this should not change. Why?
• Because when things change, sometimes “bad things happen.” Why?
• I don’t know…

And right there is where you know the solution is a bad idea.

In reality I suspect the habit perpetuates because once something appears to work, it is cut and pasted ad nauseam.

Should you never seal things?

Never say never.

Or better yet, never say “never, never, never.”

There are three levels of never:

• Never: Don’t do it because you may not know what you are doing. (Once you know the rules, you can do it.)
• Never, Never: You know what you’re doing most of the time, you are doing a bad thing but you really do know better.
• Never, Never, Never: DON’T DO IT!

There are few things that are Never, Never, Never. Dereferencing NULL comes to mind (but then maybe I waned a core dump if I get to this point in the code).

I’d place using sealed/final in the Never, Never camp if I’m using tests to describe the semantics of my classes. OR, if I’m writing concurrent code, using final on fields provides certain guarantees…

However, let’s say that I’ve been told not to use tests (I’ve seen it at more than one place). Then that “never, never” becomes a “sure do it to cover my ass”. Someone can’t subclass and mess things up. Someone can’t forget to compile their code when mine changes and cause their code to break…

Now is this a reason to avoid using Microsoft products? No. I really do like C# (for a statically-typed language that is). And I’m convinced that Java is a better language because of the work done by and in C#. And let’s face it, having more VM’s is a good thing. It improves competition. It raises the level at which we can expect to work.

But to quote Tim Booth/James in a rather trite way, “there’s a chain of consequence within, without.” That one thing, binding to the v-table so early, has caused a chain of events that leads to a platform that seems a little bit more sluggish to develop in to me. (I expect flames for this.)

I think this design decision is a reflection of a mind-set that permeates the development environment. For example, if I have a large solution with several projects, it sure seems that dependency checking in the build environment takes a long time. When I watch Developer Studio build things, it looks like make is running under the covers and performing a bunch of file checks to see what has changed. Where’s the incremental compilation?

I’ve seen very good work in both .Net and Java development efforts. It just seems that the frequency of unnecessary, basic kinds of environmental design debt occur more frequently in a .NET development effort than in similar Java development efforts.

Maybe it’s a sampling effect. To paraphrase Weinberg, “there’s always a sample, be aware of it, you can’t remove it.” I’m a consultant, I go places where they need consultants. I never, never go places that don’t need consultants, so my sample set is biased (I have one experience, maybe two, where I went and the people did not need consultants because they were doing fine – or maybe they didn’t need technically-oriented consultants).

So what do you think?

• Is using sealed by default a good design choice?
• Is the binding of methods early a good thing?
  • Was the decision by design or because of history?
• Has/will the CLR change?

Therefore, testing Java classes is unnecessary.

1. The JVM simply reads files in a particular format (the Java Class format)
2. The JVM processes those files
3. Java class files are just data to the JVM
4. Java source files are just data to the Java Compiler to generate data used by the JVM
5. We do not need to test data changes
6. Therefore we do not need to test Java classes.

q.e.d.

Then Norbert asked “Oh, by the way, did you version your data changes?”

</sarcasm>

But that’s not what I’m talking about. You know the experience. It was just working and now it is not working. You did not change anything so therefore it is the same as it was and so something else must have changed. This is bit rot.

Typically, bit rot is really just our expectations being crushed by reality. Never let a fact get in the way of a good theory, that’s my motto. This works especially well if you tend to be a better talker than listener (guilty).

At any rate, your code continues to rot as you continue to read this. Why? Someone using it has managed to find another thing programmers did not think about. So what was “working” before is not “working” now. Wait, was it really working? If someone was getting value out of using it then yes it was working. Is it working now? If nothing about the system has changed but it no longer serves a purpose, then it is not working.

Here’s a question. If a tree falls in the forest and nobody hears it, does it make a sound? Answer from a human perspective, no. If there’s nobody to hear it, then it didn’t make a sound. Sure it moved some air about and you could take an infinite sum of sine waves to characterize that air movement but then if you characterize it then something was there to observe it and that seems to violate the spirit of “nobody hears it.”

What about if there’s a problem in the code and nobody hits it, does your code have a defect? Using the tree as an example, then the answer is no. However, unlike the tree’s sound, which quickly dissipates (if for no other reason that then laws of thermodynamics), your code tends to stay around a bit longer so the chance for observation of a sleeping defect is higher (though to be fair, if you wait long enough your code, like the sound in the forest, will go away).

OK, but it is even worse than that. Things change. If your body is not changing then you are dead. That’s a fact. In one study by Dr. Frisen, he demonstrated the age of cells in a rib bone of a 30 year old to be just over 15 years, while the cells that line the stomach to be closer to 5 days. Your body is constantly fixing itself. Cells replace themselves. That’s why you can donate platelets every 3 days and blood every 56 days.

What about your code? Is it fixing itself? Is it repairing itself? Does it need to? If your code is not changing, then the project is dead. Sure, it takes some time for all usefulness to finally wear out of the system but at some point the system will essentially bit rot so badly that is serves no useful purpose. (While there are multiple clinical definitions of “dead” for a body, some life remains long after you are clinically dead – your fingernails will keep growing for some time after you expire.)

There are several things that cause your system to decay. Here are just a few:

• What the business needs changed so until the system catches up it has less value
• The act of introducing the system has changed the business so that the system needs to change
• Someone misunderstood something (it’s actually no small miracle that people can communicate at all – my wife would argue she cannot get through to me)
• Someone actually changed some code badly (most 1-line bug fixes introduce new defects – see Weinberg)
• Someone is compelled to hack it in to meet the deadline/demo/beat the lunch crowd/...

Your code is rotting while you read this because the environment around it is changing. Add to that an observation that most changes to code introduce rather than reduce chaos and it’s no wonder your system is either closed (code collapses in on itself, weighed down by its own incidental complexity) or open (nobody can decide what to do and the project eventually sort of fizzles into nothingness).

So what are you doing to actively maintain your system’s integrity? Are you just using antibiotics (patching things quickly hoping that there’s no super bug on the horizon – there’s always a super bug on the horizon no matter how strong an alligator’s immune system might be), or are you doing what the surgeon general has been saying for years: getting enough sleep, working out regularly, eat healthily (there’s a moving target).

Your code is rotting every day in every way unless you are actively working against that tide. Talking to your users – being friendly even, writing tests (acceptance, integration, smoke, unit, load, exploratory, ... mostly automated), integrating often, refactoring, learning to see beginning of rot rather than the end of it, etc.

There are a lot of things we can do and need to do to make a system. Part of that is keeping the system alive and breathing. Unlike biological systems where nature (and natural selection) has resulted in a self-monitoring, self-healing systems, code does not really monitor itself and fix itself (well most code does not do that).

Maybe there’s been a hidden force driving the creation of these things we called code maintainers whose existence is predicated on the need to repair living code to keep it around just a bit longer.

If this is the case, then just like the waist lines of americans are increasing as our biological systems have not had enough time to adjust to the changing environment, our code bases are bloating and getting to the point where code maintainers cannot keep the systems living long enough.

It seems a shift is in order to make it possible to keep these living, but often very sick, applications alive longer. In a sense, if you are practicing keeping your code clean, you are like a doctor because you diagnosing sickness, prescribing a path to heath and, if necessary, making incisions, using slugs or just waiting for a given sickness to simply follow its natural course before going away (like the requirement that simply must be done right away, just because).

Unfortunately, if we are calling ourselves doctors, then we’re still learning the value of keeping our hands clean between procedures. Unlike doctors of the mid-19th century who thought more blood on clothes = more experience, Ignaz Semmelweis figured out that hand washing saved lives.

As a community we’re not there yet. There’s still pride is hacking something together.

Your code is getting worse, what are you going to do about it?

The group has been working to stabilize and improve a product that was developed by a tiger team of outsider contractors and handed over to the in-house team post-facto. The developers had to overcome many obstacles to come up to speed on the code, to learn the new programming languages and tools, and to try to keep the feature set moving forward. None of them had been involved in the original design, but it was now their product, and its problems were their problems.

At the time of this meeting, I was one part of a coaching team which had introduced a great many changes. We were trying to help the organization to build a “whole team” mentality that encompassed documentation, security, systems administration, QA, customer representatives, and developers alike. We’d attacked the problem of matrix management. We reorganized the seating floorplan. I think at some point we’d been an inconvenience to just about everybody. The developers were in the middle of their second two-week iteration.

On this occasion, the team was in the midst of a recurring production difficulty, and had been gathered into the CIO’s office to work through a top-20 list of problems to solve.

“Your attitude has been affecting other departments”, he said. “And I want to thank you for it.”

In a particular organization, part of their build system produces “project” information to allow for development on multiple platforms and with different C++ compilers. That’s excellent, it sounds like a great tool. For a developer to get started, he/she:

• Checks out the source tree
• Runs the script
• Starts working

So far, everything is great and this part of their build system is essential to their environment – and good in general.

Here’s the next part…

To add a file to the system you have to:

• Create the file
• Update project information
• Rerun the script to regenerate project information.

When I asked how long it takes to add a class, I was told about 5 minutes. So if I want to add a header file and source file, it takes 5 minutes. That’s a big problem. Why?

After this discussion, I heard one of the senior people lamenting that a developer had put another class in an existing file rather than creating a second file. He said something like “If the developer just had discipline, he’d to the right thing.” Those darn developers.

It takes 5 minutes to add a few files to a build. That does not include build time. That’s just the time to configure the build information.

Does 5 minutes seem like very much time?

Here are a few more thins I noticed(before I knew about the build system):

• Some header files defined multiple classes
• Some source files had the methods for those multiple classes
• Some of the header files had names that did not match any of the classes defined in that header file

So is this a problem?

Here’s an important rule from Jerry Weinberg:

Nothing + Nothing + Nothing eventually equals something

5 minutes may not seem like a lot of time, and if it were isolated, then it’s probably not a problem. On the other hand, when you multiply it by a team and time, you end up with big problems.

Imagine, you need to use class X. Its header file is actually named Q.h and by the way, classes T U and L are defined in that file – none of which you want to know about.

So your class now has phantom dependencies on T, U and L. Also, how did you find the right header file? A quick search (time wasted). Someone changed U, so you end up having to recompile even though you don’t care about nor use (wasted time). I’m sure you can come up with a few things on your own.

So what do you do about it?

OK, first, do not throw the baby out with the bathwater. The original tool solved an important problem. But the first rule of problem solving, again from Jerry Weinberg:

Every solution introduces problems

The problems include (but are not limited to):

• Time wasting adding files
• It requires discipline to add new files, so it doesn’t always happen
• A name is wrong, but it’s a pain to update the build configuration, so it doesn’t happen – not all the time, just every so often

Little by little, things get a bit more chaotic.

So now that we’ve observed a problem – some waste, we need to find a way to remove the need to update the build information and regenerate to even work.

I don’t know what’s going to happen with this group. They are hoping to perform some refactorings. Their system has quite a bit of coupling. One thing we can do to reduce coupling is:

• Introduce interfaces
• Inversion of Control
• Identify seams and use some of Feather’s stuff to introduce them
• Etc., the usual stuff to introduce seams and break dependencies.

But many of the refactorings they’ll want to use will involve creating new classes. Since that takes a little bit longer, it will slow everything down – or seem so daunting, it might not happen at all.

Here’s a personal example. A few years ago, I built the security system for one applications and then a suite of applications with single sign on. When I initially introduced the security system, many people wrote tests that would forget to log out, causing problems in both the real system and the simulator.

I kept grumbling. I though, “if people would just do it right, there wouldn’t be a problem.” If they just had a little discipline.

Independent of whose fault it was, it ended up being my problem – and, quite frankly, it was my fault as well. The solution was actually pretty easy:

• Create an abstract test base
• Change tests to use it
• In the setup, the base logged in
• In the teardown, the base logged out

You might think, “Duh!” and in retrospect, so did I. But that few minutes of effort:

• Reduced code duplication
• Increased the stability of the tests
• Made it hard for people to mess up (so long as they used the correct test base – and I updated all of the tests that needed it, so going forward, people had correct examples upon which to base their work).

Ultimately, this removed a lot of my wasted time

Detecting waste is the first thing. Until you know it is a problem, you cannot do anything about it.

So, the next time you think something like:

• If that person was only following “the rules”
• If he/she just had a little more discipline
• Stupid user, they should not have done that

Ask yourself if it’s possible that those statements are directed at the symptom, not the problem.

I’m a pessimist by nature. I see a project in decline, sucking in other resources, like a black holes eating neighboring galaxies, snuffing the light out of stars, just like projects can suck the life force out of individuals.

However, energy can escape from a black hole, eventually leading to the demise of the black hole. Energy can escape along the axis of rotation in either direction. In a sense, if we see the expansion of the black hole as a bad thing, anything that diminishes a black hole is a good thing. Stephen Hawking theorized that at the event horizon of black holes, fluctuations in the space can cause the spontaneous creations of pairs of particles and their anti particles. This is another way it can appear that a black hole is losing energy since it gives off particles while the opposite particle goes into the black hole.

Now I’m off track. I wanted to talk about how synchronized give a similar effect as the massive gravity of black hole. But instead of curving space-time, they invert the possible state space of code in the presence of threads. Let’s take a look.

Here’s a simple class:

public class X {
   int value;

   public synchronized void incrementValue() {
        ++value;
    }
}

Let’s say, for the sake of argument, two threads are trying to execute the single method in this class. If the method is written like this:

   public synchronized void incrementValue() {
        ++value;
    }

Then there are only two orderings, Thread 1/Thread 2, or Thread 2/Thread 1. All of those paragraphs at the top were to make this analogy.

OK, so if we use synchronized that that’s the black hole. The number of possible paths through the code for N threads is N!. What’s the opposite? What happens if we take off the keyword and then run two threads through it? In how many ways can that single line of code get executed?

Are you ready for it?

3,432 different ways

Really? Well at least conceptually. In practice, the Just In Time compiler will probably turn the JVM instructions into native code, so the actual number of individual steps will probably be less. But there really are that many paths through the system. You might ask how?

That single line of code is represented in byte code as 7 lines. I could show you the disassembly, but really, it’s 7. You can convince yourself by having a look at the byte-code using a nice byte-code viewer plug-in for eclipse.

What happens if we change the int to a long? We still have 7 lines of byte-code, but with different instructions. We have a total of 4 long reads/writes. Each long read/write requires two individual 32-bit operations – or at lest the JVM can implement it that way. The actual number of atomic operations after the Just In Time compiler has had its way with your byte-code will probably be less, but if we stick to byte-codes and the Java memory model, the number 3,432 is now a whopping 705,432!

Let’s extend that just a bit more. What if you have several lines of code? Each line of Java results in many more lines of byte-code. What if we have have 3 lies of Java? We probably have something like 21 lines of byte-code. How many paths of execution would 21 lines of byte-code executed by two threads have? 538,257,874,440!

Where the first program ended up being a white dwarf, the version using longs was a nova. I’m not sure what to call the thee line Java method, maybe a hyper-nova?

In practice, the actual number of paths through a method will end up being smaller. And, most of the paths do not have any negative side-effects. The problem, however, is that there are a small number of hard to find paths of execution that do cause problems and finding them is like looking for a needle in a haystack.

Remember what adding the keyword synchronized did? If we use it on a three line method, it turns 538,257,874,440 into 2. It collapses the number of paths into 2 for two threads. And 2 is less than 538,257,874,440, for even vary large values of 2.

How did I derive at these magic numbers? Two ways.

I wanted to know how many possible orderings there were for 7 instructions and two threads. I knew it had to do with combinations and permutations but I just wasn’t smart enough to figure it out. Searching on Google didn’t do much for me either. So I decided to write a program to numerically derive the result. I tested my way into a program that would calculate the result.

The basic algorithm is:

1. Determine all possible orderings of each of the unique “nodes” in a system. For example, if I have 2 threads and 2 instructions, I can generate a natural key (ugh!) to represent each of the different combinations of threads and instructions: T1S1 T1S2 T2S1 2S2.
2. Produce all valid permutations of these four “nodes”.
3. Remove any that have invalid orderings.
4. Count the results.

Many of my first implementations for some of these steps were exercises in writing really inefficient code! But having those tests made it really easy to swap out better algorithms when my brain caught up with what I was doing.

What is an invalid ordering? The code generated by the single line using the pre increment operator has 7 steps. Those 7 steps are in a single sequence with no looping and no conditionals. That’s like saying A is always before B. So if we generate a list of orderings that end up including that combination, we’ve generated an ordering that cannot happen.

So you can describe your possible state space in a few steps:

1. Register a node for each thread, step combination. 7 threads, 4 steps = 28 nodes.
2. Register dependences between steps. For example, if I have seven steps and two threads executing the same sequence, I could describe this as: T1S1 -> T1S2 -> T1S3, etc. And for the second thread: T2S1 -> T2S2 -> T2>S3. Since the name is arbitrary, I just used letters, so in fact I had: a -> b -> c -> d -> e -> f -> g for one set of dependencies and h -> i -> j -> k -> l -> m -> o

In fact, here’s the test I used to describe that exact space:

    @Test
    public void twoThreadsAndSevenSteps() {
        calculator = new PathCalculator(14);
        createLine('a', 'b', 'c', 'd', 'e', 'f', 'g');
        createLine('h', 'i', 'j', 'k', 'l', 'm', 'n');
        start();
        int totalPaths = calculator.totalPaths();
        stop(2, 4, totalPaths);
    }

By the time I got to this point, I wrote some “tests” using JUnit as a harness to run my code and display timing information and a summary of the

Time:        4ms -- Threads:  3, Opcodes:  2, Total paths:      90
Time:        1ms -- Threads:  2, Opcodes:  3, Total paths:      20
Time:       54ms -- Threads:  4, Opcodes:  2, Total paths:    2520
Time:        3ms -- Threads:  2, Opcodes:  4, Total paths:      70
Time:       61ms -- Threads:  2, Opcodes:  5, Total paths:     252
Time:     1248ms -- Threads:  2, Opcodes:  6, Total paths:     924
Time:    12819ms -- Threads: 10, Opcodes:  1, Total paths: 3628800

I then used this information to fit a cure to my data and derive a formula for various combinations. For 2 threads, I came up with the following formula. If you let N be the number of steps:

Total Paths = 252 * 3.6667 ^N-5

The morning after, Uncle Bob sent an email, showing me his work on how he calculated.

If you let:

1. T be the number of threads
2. N be the number of steps

Total = (N*T)!   /   N! ^T

When I plugged numbers into his formula, they fit my formula. His formula generated an exact result as opposed to my estimate based on fitting a curve. Another thing his solution has going for it is the algorithm I came up with, I believe, is NP-complete. The net effect was I that while I was able to calculate result for 2 threads and 6 steps, 2 threads and 7 steps took a long time before running out of memory with a 2GB heap space. Bob’s formula gave me a so-called verifier (as did my formula – only Bob’s was better) and the algorithm seems to grow based on factorials and not polynomials.

What’s the point of all of this? The single, one-line program, is a demonstration of why going from one thread to two threads makes writing/debugging/testing programs so much harder. It also numerically demonstrates why the single responsibility principle is even more important when we start writing concurrent programs. The more you put in one place, the large the state space. Larger state spaces:

1. Make it harder to find that one or few number of paths that can cause your program to die, under load, in production, while you are on vacation
2. Make your efforts to test (which is a sampling technique anyway) for concurrency issues all the more difficult
3. Make it harder to find what might actually have caused a failure once you’ve notice it.

In a later post, I’ll describe how we can improve our chances of covering this huge space of potential orderings.

If you’ve made it this far, wow! Congratulations!

What’s the problem?

It was written in the days when we would use a pre-processing stage to instrument code and then execute tests. This is fine if I wanted to work at the command line and use ant/maven to build. However, I want to work in an IDE that makes running unit tests easy (Eclipse, though they all do it now). But if I have to remember to instrument my classes before running my tests, that leads to human error. I don’t want that, I want to just run my unit tests and have my classes dynamically instrumented. (This is not speculation, this comes from working in a group consisting of multiple teams, all of which were using some Aspects written using AspectJ and running unit tests in Eclipse – when we introduced dynamic instrumentation – pre JDK 1.5 and even working in WebSphere 5.x), it improved productivity.

What about a plug-in? Sure, there’s one for Eclipse but I’ve not been able to download it. It probably works fine – after a class is compiled, it gets instrumented – but if I’m able to write a simple Java Agent, I can create a JVM configuration with a few parameters and every time I run my tests, viola, dynamic instrumentation with only a little one-time environment configuration. Also, I don’t have to worry about waiting for a plug-in update to continue using the tool.

Will my tests run slower? Probably, but until I know how much slower, I’m willing to risk it. (I’ve used dynamic instrumentation when running over 1,000 tests on a workspace with > 1 Million lines of code, it was fine.) If it’s an issue, I can imagine using a combination of annotations and the Java agent. Something like:

@TestInstrument
public class SomeClassThatUsesThreading {}

This would allow the Java Agent to only instrument some classes, rather than all classes. This could cause problems if I forget the annotation, but it’s an option if speed is an issue.

I would not do that unless it was necessary. First, most of my tests would be testing code that is not thread-related. Those tests would not require instrumentation. The tests that require instrumentation, would be somewhere else, and I’d run them with a different frequency; I’d run them longer, with more configurations and iterations, to increase my chances of finding threading-related issues.

There’s another option. Copy what ConTest is already doing using AOP. I tried, and I cannot select the correct point-cuts using traditional point-cut syntax – try selecting synchronized blocks and then every line within the block, that’s not a typical point-cut usage scenario. I considered a combination of hand instrumentation and AOP – it’ll work but it makes the code ugly. I even considered using asm or cglib directly and at that point I knew it was more time than I wanted to spend when the developers of ConTest have years of experience instrumenting classes.

Anyway, I’m hoping the team working on this tool will publish an API soon so I can give that a try. They mentioned they would at some point.

If you want more information on writing a Java Agent (and the class that actually registeres it), have a look at Brett’s Java Agent Notes.

What?! Why should you care? Answer, you shouldn’t. Yes C++ and Java both support covariant return types, but so what? Have you ever used them? OK, I have, but then I also used and liked C++ for about 7 years, over 10 years ago. We all learn to move on.

You ever notice how something meant to help often (typically?) turns out to do exactly the opposite? Even worse, it directly supports or enables another unfortunate behavior. This is just Weinberg’s first rule of problem solving:

Every solution introduces new problems

Here’s an example I’m guilty of. Several years ago I was working on a project where we had written many unit tests. We had not followed the FIRST principles, specifically R-Reliability. Our tests were hugely affected by the environment. We had copies of real databases that would get overridden without warning. We’d have problems with MQ timeouts at certain times of the day or sometimes for days. And our tests would fail.

We wold generally have “events” that would cause tests to fail for some time. We were using continuous integration and so to “fix” the problem, I created a class we called “TimeBomb” – turns out it was the right name for the wrong reason.

You’d put a TimeBomb in a test and then comment out the test. The TimeBomb would be given a date. Until that date, the test would “pass” – or rater be ignored. At some point in the future, the TimeBomb would expire and tests would start failing again.

I was so proud of it, I even took the time to write something up about it here. I had nothing but the best intentions. I wanted an active mechanism to remind us of work that needed to be done. We had so many todo’s and warnings that anything short of an active reminder would simply be missed. I also wanted CI to “work.” But what eventually happened was that as our tests kept failing, we’d simply keep moving the TimeBomb date out.

I wrote something that enabled our project to collect more design debt. As I said, my intentions were noble. But the road to Hell is paved with good intentions. Luckily I got the name right. The thing that was really blowing up, however, was the project itself.

What has all of this got to do with “strongly typed languages”?

If you’re working with a strongly typed language, you will have discussions about covariance (well I do anyway). You’ll discuss the merits of multiple inheritance (it really is a necessary feature – let the flames rise, I’m already in Hell from my good intentions). You’ll also discuss templates/generics. The list goes on and on.

If you’re working with a dynamically typed language (the first one I used professionally was Smalltalk but I also used Self, Objective-C, which lets you choose, and several other languages whose names I do not recall).

Covariance is not even an issue. The same can be said of generics/templates and yes, even multiple inheritance is less of an issue. In a strongly-typed languages, things like Multiple Inheritance are necessary if you want your type system to be complete. (If you don’t believe me, read a copy of Object Oriented Software Construction, 2nd ed. by Bertrand Meyer – an excellent read.)

Ever created a mock object in a typed language? You either need an interface or at least a non-final class. What about Ruby or Smalltalk? Nope. Neither language cares about a class’ interface until it executes. And neither language cares how a class is able respond to a particular message, just that it does. It’s even possible in both languages to NOT have a method and still work if you mess with the meta-language protocol.

OK, but still, does this make typed languages bad?

Lets go back to that issue of enabling bad things.

Virtual Machines, like the JVM and CLR, have made amazing strides in the past few years. Reflection keeps getting faster. Just in time compilers keep getting smarter. The time required for intrinsic locks has improved and now a Java 5 compiler will support non-blocking, safe, multi-threaded updates. Modern processors support such operations using an optimistic approach. Heck, Java 6 even does some cool stuff with local object references to significantly improve Java’s memory usage. Java runs pretty fast.

Dynamic languages, generally, are not there yet. I hope they get there, but they simply are not there. But so what?! If your system runs “fast enough” then it’s fast enough. People used Smalltalk for real applications years ago – and still do to some extent. Ruby is certainly fast enough for a large class of problems.

How is that? These languages force you to write well. If you do not, then you will write code that is not “fast enough.” I’ve see very poor performing Smalltalk solutions. But it was never because of Smalltalk, it was because of poor programming practices. Are there things that won’t currently perform fast enough in dynamically typed languages? Yes. Are most applications like that? Probably not.

You can’t get away with as much in a dynamically typed language. That sounds ironic. On the one hand you have amazing power with dynamically typed languages. Of course Peter Parker learned that “With great power comes great responsibility.” This is just as true with Ruby and other dynamically typed languages.

You do not have as much to guide you in a dynamically typed language. Badly written, poorly organized code in a typed language is hard to follow but it’s possible. In a language like Ruby or Smalltalk it’s still possible but it’s a lot harder. Such poor approaches will typically fail sooner. And thats GOOD! You’ve wasted less money because you failed sooner. If you’ve got crappy design, you’re going to fail. The issue is how much time/money will you spend to get there.

Another thing that strongly typed languages offer is a false sense of security because of compiler errors. I have heard many people deride the need for unit tests because the compiler “will catch it.”

This misses a significant point that unit tests can serve as a specification of behavior rather than just a validation of as-written code.

You cannot get away with as much in a dynamically typed language. Or put another way, a dynamically typed language does not enable you to be as sloppy. It’s just a fact. In fact you can typically get away with a lot in a dynamically typed language, you just have to do it well.

Does this mean that dynamically typed languages are harder to work in? Maybe. But if you follow TDD, then the language is less important.

Do we need fast, typed languages? Clearly we do. There are applications where having such languages is necessary. Device drivers are not written in Ruby (maybe they are, I’m just trying to be more balanced).

However, how many of you were around when games were written in assembly? C was not fast enough. Then C was fast enough but C++ was still a question mark. C++ and C became mainstream but Java was just too slow. Some games are getting written in Java now. Not many, but it’s certainly possible. It is also possible to use multiple languages and put the time-critical stuff, which is generally a small part of your system, in one language and use another language for the rest of the system.

Back in the very early 90’s I worked just a little bit with Self. At that time, their Just In Time compiler would create machine code from byte code for methods that got executed. They went one step further, however. Not only did they JIT compile a method for an object, they would actually do it for combinations of receivers and type parameters.

There’s a name for this. In general it is called multi-dispatch (some languages support double-dispatch, the visitor pattern is a mechanism to turn standard polymorphism into a weak form of double dispatch, Smalltalk used a similar approach for handling numerical calculations).

Self was doing this not to support multi-dispatch but to improve performance. That means that a given method could have multiple compiled forms to handle commonly used methods on a receiver with commonly-used parameters. Yes it used more memory. But given what modern compilers do with code optimization, it just seems that this kind of technique could have huge benefits in the performance of dynamically typed languages. This is just one way a dynamic language can speed things up. There are others and they are happening NOW.

I’m hoping that in the next few years dynamic languages will get more credit for what they have to offer. I believe they are the future. I still primarily develop in Java but that’s just because I’m waiting for the dust to settle a little bit and for a clear dynamic language to start to assert itself. I like Ruby (though its support for block closures is, IMO, weak). I’m not convinced Ruby is the next big thing. I’m working with it a little bit just in case, however.

What are you currently doing that enables yourself or your co-workers to maintain the status quo?