• You could end up writing less code to accomplish the same thing
• You might write better code that is less-coupled and more maleable
• The code tends to be testable because, well, it IS tested
• The coverage of your tests will be such that making significant changes will not be too risky
• The number of defects should be quite low
• The tests serve as excellent exampls of how to use the various classes in your solution
• Less “just in case” code written, which typically doesn’t work in those cases that they targeted

Right now I do not know of a better way to accomplish all of these results more effectively than practicing TDD. Even so, this does not elevate TDD from a “how” to a “what.” TDD remains a technique to accomplish thigns I value. It is not a self-justifying practice. If someone asks me “why do we do it this way”, saying something like “we practice TDD” or “well you don’t understand TDD” is not a good answer.

• Write no production code without failing tests
• Write only enough test code so that it fails (not compiling is failing)
• Write only enough production code to get your tests to pass.

• S.O.L.I.D.
• F.I.R.S.T.
• Separation of Concerns (Square Law of Computation)
• Coupling/Cohesion
• Protected Variation
• ...

TDD is a means to an end but it is the end we care about. What is that end? Software that has few defects and is easy to change. Tests give us that. Not testing generally does not give us that. And testing in a common “QA over the wall” configuration typically does not cut it.

Since I do not know how to so easily produce those results in any other way. TDD becomes the defacto means of implementation for me. That doesn’t mean I should turn a blind eye to new ways of doing things. In lieu of any such information, however, I’ll pick TDD as a starting point. This is still a “how” and not a “what”.

• Increased confidence in the code I produce (even more than when I was simply test infected)
• Less worrying about one-off conditions and edge cases. I’ll get to them and as I think about then, they become tests
• Fun

Fun?

• I seem to produce demonstrable benefits sooner
• I actually do more analysis throughout
• I get to do more OO programming

Demonstrable Benefits Sooner

More Analysis

More OO

When I try to create a good unit test, much of what I’m doing is trying to figure out how the effect I’m shooting for can be isolated to make the test fast, independent, reliable … To do so, I make heavy use of test doubles. Sometimes I hand-roll them, sometimes I use mocking libraries. I’ve event used AOP frameworks, but not nearly as extensively.

Doing all of this allows me to use polymorphism more often. And that’s fun.

Conclusion

Those are good questions. And these are things you might want to ask yourself.

• The product owner is keeping me focused on things that add value
• Short iterations keep me reigned in
• I’m only doing as much as necessary to get the stories for an iteration implemented
• The tests I’m writing stay passing, run quickly and over time remain (become) maintainable

Even so, since TDD is a how and not a what, I still need to keep asking myself if the work I’m doing is moving us towards a working solution that will be maintainable during its lifetime.

I think it is. What about you?

I managed to get two walls done and everything moved back into place before I left for another business trip. My wife is happy because the bedroom looks better. I did no damage and made noticeable progress. I still have some Painting to do (the capital P is to indicate it will be a Pain). I eventually have to move the bed, rent scaffolding, and so on. That’s probably more in the future than I’d prefer, but I’ll do it when I know I have the time and space to do it.

Contrast this to when we bough the house back in March. I entered an empty house. I managed to get two bedrooms painted (ceilings included) and the “grand” room with 14’ vaulted ceilings. I only nearly killed myself once – don’t lean a ladder against a wall but put the legs on plastic – and it was much easier to move around. I had a clean slate.

Sometimes you’ve got to get done what you can get done to make some progress. When I was younger, my desire to finish the entire bedroom might have stopped me from making any progress. Sure, the bedroom is now half old paint and half new paint, but according to my wife it looks better – and she’s the product owner! I can probably do one more wall without having to do major lifting and when I’m finally ready to rent the scaffolding, I won’t have as much to do. I can break down the bed, rent the scaffolding and then in one day I might be able to finish the remainder of the work. (Well probably 2 days because I’ll end up wanting to apply 2 coats to the ceiling and I’ll need to wait 8 hours).

Painting is just like software development.

Several colleagues and I really groked Fusion. It started with system sequence diagrams showing interactions much like the specification I mentioned above. It also described a difference between analysis and design (and if Uncle Bob reads this, he’ll probably have some strong words about so-called object-oriented analysis, well this was 15 years ago… though I still there there is some value to be found there). Anyway, this is mostly about system sequence diagrams so I won’t say much more about that excellent process.

Each system sequence diagram represented a scenario. To represent many scenarios, Fusion offered a BNF-based syntax to express those various scenarios. (I understand that this was also because for political reasons within HP they were not allowed to include a state model, but I don’t know if that is true or not.) For several years I practiced Fusion and really I often revert back to that if I’m not trying to do anything in particular.

Spending a little time up front thinking a little about the logical interaction between the system and its various actors helps me get a big picture view of the system boundary and its general flow. I have also found it helps others as well, but your mileage may vary.

So when I viewed their protocol specification, it really brought back some good memories. And in fact, that’s how we decided to look at the problem.

(What follows is highly-idealized)

We reviewed their specification and decided we’d try to work through the initialization sequence and then work through one sequence that involved “completing a simple job.” I need to keep this high level to keep the identity of the company a secret.

There was prior work and we kept that in mind but really started from scratch. In our very first attempt, there had been some work done along the lines of using the Command pattern, so we started there. Of course, once we did our first command, we backed off and when with a more basic design that seemed to fit the complexity a bit better (starting with the command pattern at the beginning is an example of solution-problemming to use a Weinberg term – and one of the reasons I’m sometimes skeptical when people start talking in patterns).

• Receive a message from the outside as some array of bytes
• Determine the “command” represented by the bytes
• Process the parameters within the command
• Issue a message to the simulator
• Create a logical response
• Format the logical response into the underlying protocol
• Send the response back

At the time, they were considering using JNI, so we spent just over a day validating that we could communicate bi-directionally, maintaining a single process space.

• It is easy to end up with a bunch of tested objects but no fully-connected system
• Sharing setup between various mocks is difficult and often not done so there’s a lot of violation of DRY
• You have to learn a new syntax

We accepted learning a new syntax because it was deemed less painful than maintaining existing hand-rolled test doubles (though there are several reasonable solution for that, ask if you want to know what it is). There is the issue of sharing setup on mocks, but we did not have enough work yet to really notice that as a problem. However, they were at least aware of that and we briefly discussed how to share common expectation-setting (it’s well supported).

• Could be tested up to but excluding the JNI stuff
• Could be configured to stub out JNI or use real JNI
• Was easily configurable
• Was automatically configured by C++ (since it was a C++ process that was started to get the whole system in place)

We designed that (15 minute white-board session), coded it and ended up with a few integration tests. Along the way, we built a simple factory for creating the system fully connected. That factory was used both in tests as well as by the JNI-based classes to make sure that we had a fully-connected systems when it was finally started by C++.

Near the end, we decided we wanted to demonstrate asynchronous computation, which we did using tests. I stumbled a bit but we got it done in a few hours. We demonstrated that the system receiving messages from the outside world basically queued up requests rather than making the sender wait synchronously (we demonstrated this indirectly – that might be a later blog post – let me know if you’re interested).

By the way, that was the first week. These guys were good and I had a great time.

There was still a little work to be done on the C++ side and I only had a week, so I asked them to keep me posted. The following Tuesday they had the first end-to-end interaction, system initialization.

By Wednesday (so 3 business days later), they had a complete demonstration of end-to-end interaction with a single, simple job finishing. Not long after that they demonstrated several simple jobs finishing. The next thing on their list? Completing more complex jobs, system configuration, etc.

• Select a system interaction
• List all of the steps it needs to accomplish (some requests required a response, some did not)
• Write unit tests for each “arm” of the interaction

Select a set of end-to-end interactions that add value to the user of the system

For each system-level interaction, enumerate all of its steps and then add implementing those steps as individual back-log items

Now some of those individual steps will end up being small (less than an hour) but some will be quite large when they start working with variable parameters and commands that need to operate at a higher priority.

But they are well on their way and I was reminded of just how much I really enjoyed using Fusion.

No

To keep focus and know we’re making progress

That varies by the meeting

Exress how we know somethign has worked

Can we do this with meetings? If we did, what might it look like? Would the very act of trying to express success critera for a meeting have a profound effect on how a meeting progresses? Isn’t this really just Covey’s 2nd habit (begin with the end in mind).

It’s all about he benjamins.

The first thing you need to ask is how what you’re going to do in a given meeting directly or indireclty adds value. Of coruse, to do that you need to know what is valuable.

Suppose, for example, you need to have a traditional requirements meeting of some sort (there is a traditional requirements meeting, it is usually bogged down in implementation details or it’s stuck in “how” mode versus “what” mode – the more you talk about database columns, the more you’re deep into requirements … NOT!).

OK, so take a step back. You think you need this meeting on requirements.

So we can figure out what this feature needs to do

So we know what to what to write, what schema changes we need

So we can implement it

Because customer X has a contract and we’ll be in breech of contract if we don’t write it

AH! That sounds like we’re getting close to something valuable. We have some feature that’s been promised to a current customer. OK, so before we go any further, that information needs to be part of the context for the meeting.

If you cannot answer this, you are in deep trouble

OK, so you have promised some feature. Apparently that feature is ill defined (or you wouldn’t need the requirements meeting). So, what problem does that feature address?

This meeting is a success if we have described scenarios that cover what this feature will do for at the top three uses of this feature

• This requirements meeting is to address feature X
• We need to discuss this because customer X has been promised this feature and if we do not do it by (insert a real date here), we are in breech of contract
• This meeting will be a success if we can describe scenarios covering the top three uses of this feature

• How do we define top 3? Is that something we know or something we need to accomplish?
• What do we mean by “describe scenarios” does it include:
  • Acceptance tests
  • One or more stories (or maybe it is just a scenario in a use case, pick your requirements coolaid)
  • Used by whom? Are there different roles? Is that important?
  • What about a timebox? (Maybe that’s implicit because it’d be an electronic request)
  • ...

Here’s the thing, do you think you’d be more inclined to think a meeting described thusly would have some value?

• It expresses the design intent
• It’s for performance
• The code as written cannot be safely overridden.

(Before I go any further, I am not arguing against the use of sealed or final. I will say that in my practice, it has to be justified to be used rather than the other way around.)

Back to the reasons. I generally don’t buy these reasons. Each may be perfectly valid but none of them stand on their own; each requires further discussion.

• It expresses design intent – you don’t know how to design
• It’s for performance – do you have any idea how a VM works?
• The code as written cannot be safely overridden – So the code is so badly written that nobody understands it?

Again, let me be clear, there are good reasons to seal things. My point is that while there are good reasons, most of the time I’ve seen the habit is a form of cargo culting; monkey see, monkey do.

This observation sat around for some time until I read some stuff in a book called CLR via C# (good reading if you want to better understand the CLR, and if you’re developing code for a virtual machine, I think it’s a good idea to understand the virtual machine at least a little).

People are sealing as much as they are because virtual table binding is done way to early.

The thing I read in the book I mentioned above relates to when virtual methods are bound. This applies to 2.0 version of the CLR, but I don’t think there’s any change for the latest version of the CLR and this seems like a pretty “hard” design decision. This book mentions that binding of virtual methods is performed at compile time.

I’m not making a distinction between compile time and link time for this discussion, but I’ll be a little more precise. The generated code stored in an assembly “knows” which virtual function it is invoking because in the generated byte code is an index into a virtual table.

How is this a problem?

I write a base class and ship it. You write a derived class (OK, maybe you use delegation, good for you, but the problem still remains). Your coupling to my class involves using a method with the virtual keyword (Java programmers, you don’t have to do this, regular methods [not static, not a constructor, ...] have the potential for dynamic dispatch unless you declare them final). At compile time, your object module (which eventually resides in an assembly that will get loaded and executed some time later) knows that it wants to call the 3rd virtual method in the v-table.

So far, so good.

Next, I add a new virtual method to my class and ship a new assembly.

Question. To use this new assembly, do you need to recompile?

Yes.

What?

Yes. What happens if my newly added method is the first virtual method? Then your compiled code that is happily living in a peaceful assembly somewhere has a stale reference. It thinks it wants to run the 3rd method in the virtual table but now it really should be using the 4th virtual method.

This is similar to a caching issue or premature optimization. The calling code has a compile-time generated index into the v-table. Why is this? Is it for performance? Not a good argument, this pre-calculation is irrelevant with modern JIT compiling techniques. I think the reason it is done this way is that C++ does things this way. C++ needs to because of its execution model (there’s not virtual machine between the code and its execution).

No big deal, you think, I’ll just recompile my code.

That will fix the problem. Or will it?

Say you’re stuck using an old version of some assembly that uses an old version of another assembly that you also use. Do you recompile both your code and that other assembly? You’ll have to but you might forget. Or you might not be able to.

.Net handles this by having an excellent dependency system to track which versions of which assemblies work with with other versions of assemblies. They need this to address problems introduced by the decision to perform such early binding.

Simply put, this is an unfortunate decision that I believe was more based on precedence or history than on sound design decisions.

This is not what happens in the Java Virtual Machine (JVM).

In the JVM, the binding is done later—much later. It is possible that my class got loaded and JIT’ed so that all of the methods that could be virtually invoked (not final) are in fact not virtually invoked. So far, no subclasses have been loaded, so there’s been no need. After my system has been running for a few weeks, a new class (a subclass) gets loaded and invalidates the JIT’d version of my class. No problem, the system will dump the old version and JIT it again, on the fly.

Even if there is a need for virtual dispatch, it might be that most of the time I use one version of method for a particularly heavily used subclass. The JVM might inline the common case and virtually dispatch the others.

Bottom line: what’s being done with JVM’s is indistinguishable from black magic for most of us.

I can use new JAR files and not have to worry about recompiling to remove stale v-table references. The method binding will just work. (This does not remove all problems with changing versions of JAR’s but it gets rid of one persnickety one.)

But it goes deeper. As I mentioned above, I’ve noticed several Microsoft shops that use a lot of sealed classes or avoid using virtual methods because “they perform poorly”.

OK, let’s talk about method invocation for just a second (or make that pico second). On my machine, I timed virtual method dispatch. I should check my numbers, but what I got was that virtual method dispatch was taking around 450 pico-seconds. So you can invoke roughly 2.2 Billion methods per second.

• Virtual method dispatch is not that expensive
• The JVM can address caching issues with code analysis and inlining
• You only pay the price if your design requires it
• The people writing the JVM are better and micro optimization than certainly me, and I’d say most people

• Testability
• Maintainability
• Flexibility
• Extensibility

Look up the percentage of time spent “building” a system versus “maintaining” a system. The percentage has been on the rise, and it’s 80 – 95% typically. 80% of the cost of development is spent maintaining code. (I believe a lot of this percentage has to do with the definition of “done” – but that’s a whole other discussion).

You might think that locking things down make them more stable and that stability leads to maintainability.

That’s not really the case.

Things change. Requirements change.

Locking down a class to keep it from breaking is like signing off on requirements to keep them from changing. Neither thing is really a good idea.

I CAN lock down my classes in a sense. I can use tests to describe my assumptions so the class is locked down so long as we all agree that we’ll keep the tests passing. That is a fine level of granularity.

But I digress.

Back to the whole sealed thing. I was wondering why so many places that use Microsoft stuff seal their classes and do not use virtual methods (in my experience, it’s about 50%). I then read that thing about the binding of virtual methods at compile time. And then about 2 weeks later it hit me.

If you use virtual methods and you have a messed up build system, then you’ll get strange behavior. Sealing things makes those surprising behaviors go away, so seal stuff.

It makes sense. This is something that C++ was notorious about. You hoped you’d get a segmentation violation but often the wrong method could get called and the system continued to run…

Unfortunately, the “solution” – sealing – is attacking a symptom, not the actual problem. This is where following the 5-why’s of Toyota would have been a good idea.

• Because they will be more stable. More safe/it expresses my design. Why?
• Because my design says that this should not change. Why?
• Because when things change, sometimes “bad things happen.” Why?
• I don’t know…

And right there is where you know the solution is a bad idea.

In reality I suspect the habit perpetuates because once something appears to work, it is cut and pasted ad nauseam.

Should you never seal things?

Never say never.

Or better yet, never say “never, never, never.”

• Never: Don’t do it because you may not know what you are doing. (Once you know the rules, you can do it.)
• Never, Never: You know what you’re doing most of the time, you are doing a bad thing but you really do know better.
• Never, Never, Never: DON’T DO IT!

There are few things that are Never, Never, Never. Dereferencing NULL comes to mind (but then maybe I waned a core dump if I get to this point in the code).

I’d place using sealed/final in the Never, Never camp if I’m using tests to describe the semantics of my classes. OR, if I’m writing concurrent code, using final on fields provides certain guarantees…

However, let’s say that I’ve been told not to use tests (I’ve seen it at more than one place). Then that “never, never” becomes a “sure do it to cover my ass”. Someone can’t subclass and mess things up. Someone can’t forget to compile their code when mine changes and cause their code to break…

Now is this a reason to avoid using Microsoft products? No. I really do like C# (for a statically-typed language that is). And I’m convinced that Java is a better language because of the work done by and in C#. And let’s face it, having more VM’s is a good thing. It improves competition. It raises the level at which we can expect to work.

But to quote Tim Booth/James in a rather trite way, “there’s a chain of consequence within, without.” That one thing, binding to the v-table so early, has caused a chain of events that leads to a platform that seems a little bit more sluggish to develop in to me. (I expect flames for this.)

I think this design decision is a reflection of a mind-set that permeates the development environment. For example, if I have a large solution with several projects, it sure seems that dependency checking in the build environment takes a long time. When I watch Developer Studio build things, it looks like make is running under the covers and performing a bunch of file checks to see what has changed. Where’s the incremental compilation?

I’ve seen very good work in both .Net and Java development efforts. It just seems that the frequency of unnecessary, basic kinds of environmental design debt occur more frequently in a .NET development effort than in similar Java development efforts.

Maybe it’s a sampling effect. To paraphrase Weinberg, “there’s always a sample, be aware of it, you can’t remove it.” I’m a consultant, I go places where they need consultants. I never, never go places that don’t need consultants, so my sample set is biased (I have one experience, maybe two, where I went and the people did not need consultants because they were doing fine – or maybe they didn’t need technically-oriented consultants).

• Is using sealed by default a good design choice?
• Is the binding of methods early a good thing?
  • Was the decision by design or because of history?
• Has/will the CLR change?

1. The JVM simply reads files in a particular format (the Java Class format)
2. The JVM processes those files
3. Java class files are just data to the JVM
4. Java source files are just data to the Java Compiler to generate data used by the JVM
5. We do not need to test data changes
6. Therefore we do not need to test Java classes.

q.e.d.

Then Norbert asked “Oh, by the way, did you version your data changes?”

</sarcasm>

But that’s not what I’m talking about. You know the experience. It was just working and now it is not working. You did not change anything so therefore it is the same as it was and so something else must have changed. This is bit rot.

Typically, bit rot is really just our expectations being crushed by reality. Never let a fact get in the way of a good theory, that’s my motto. This works especially well if you tend to be a better talker than listener (guilty).

At any rate, your code continues to rot as you continue to read this. Why? Someone using it has managed to find another thing programmers did not think about. So what was “working” before is not “working” now. Wait, was it really working? If someone was getting value out of using it then yes it was working. Is it working now? If nothing about the system has changed but it no longer serves a purpose, then it is not working.

Here’s a question. If a tree falls in the forest and nobody hears it, does it make a sound? Answer from a human perspective, no. If there’s nobody to hear it, then it didn’t make a sound. Sure it moved some air about and you could take an infinite sum of sine waves to characterize that air movement but then if you characterize it then something was there to observe it and that seems to violate the spirit of “nobody hears it.”

What about if there’s a problem in the code and nobody hits it, does your code have a defect? Using the tree as an example, then the answer is no. However, unlike the tree’s sound, which quickly dissipates (if for no other reason that then laws of thermodynamics), your code tends to stay around a bit longer so the chance for observation of a sleeping defect is higher (though to be fair, if you wait long enough your code, like the sound in the forest, will go away).

OK, but it is even worse than that. Things change. If your body is not changing then you are dead. That’s a fact. In one study by Dr. Frisen, he demonstrated the age of cells in a rib bone of a 30 year old to be just over 15 years, while the cells that line the stomach to be closer to 5 days. Your body is constantly fixing itself. Cells replace themselves. That’s why you can donate platelets every 3 days and blood every 56 days.

What about your code? Is it fixing itself? Is it repairing itself? Does it need to? If your code is not changing, then the project is dead. Sure, it takes some time for all usefulness to finally wear out of the system but at some point the system will essentially bit rot so badly that is serves no useful purpose. (While there are multiple clinical definitions of “dead” for a body, some life remains long after you are clinically dead – your fingernails will keep growing for some time after you expire.)

• What the business needs changed so until the system catches up it has less value
• The act of introducing the system has changed the business so that the system needs to change
• Someone misunderstood something (it’s actually no small miracle that people can communicate at all – my wife would argue she cannot get through to me)
• Someone actually changed some code badly (most 1-line bug fixes introduce new defects – see Weinberg)
• Someone is compelled to hack it in to meet the deadline/demo/beat the lunch crowd/...

Your code is rotting while you read this because the environment around it is changing. Add to that an observation that most changes to code introduce rather than reduce chaos and it’s no wonder your system is either closed (code collapses in on itself, weighed down by its own incidental complexity) or open (nobody can decide what to do and the project eventually sort of fizzles into nothingness).

So what are you doing to actively maintain your system’s integrity? Are you just using antibiotics (patching things quickly hoping that there’s no super bug on the horizon – there’s always a super bug on the horizon no matter how strong an alligator’s immune system might be), or are you doing what the surgeon general has been saying for years: getting enough sleep, working out regularly, eat healthily (there’s a moving target).

Your code is rotting every day in every way unless you are actively working against that tide. Talking to your users – being friendly even, writing tests (acceptance, integration, smoke, unit, load, exploratory, ... mostly automated), integrating often, refactoring, learning to see beginning of rot rather than the end of it, etc.

There are a lot of things we can do and need to do to make a system. Part of that is keeping the system alive and breathing. Unlike biological systems where nature (and natural selection) has resulted in a self-monitoring, self-healing systems, code does not really monitor itself and fix itself (well most code does not do that).

Maybe there’s been a hidden force driving the creation of these things we called code maintainers whose existence is predicated on the need to repair living code to keep it around just a bit longer.

If this is the case, then just like the waist lines of americans are increasing as our biological systems have not had enough time to adjust to the changing environment, our code bases are bloating and getting to the point where code maintainers cannot keep the systems living long enough.

It seems a shift is in order to make it possible to keep these living, but often very sick, applications alive longer. In a sense, if you are practicing keeping your code clean, you are like a doctor because you diagnosing sickness, prescribing a path to heath and, if necessary, making incisions, using slugs or just waiting for a given sickness to simply follow its natural course before going away (like the requirement that simply must be done right away, just because).

Unfortunately, if we are calling ourselves doctors, then we’re still learning the value of keeping our hands clean between procedures. Unlike doctors of the mid-19th century who thought more blood on clothes = more experience, Ignaz Semmelweis figured out that hand washing saved lives.

As a community we’re not there yet. There’s still pride is hacking something together.

Your code is getting worse, what are you going to do about it?

The group has been working to stabilize and improve a product that was developed by a tiger team of outsider contractors and handed over to the in-house team post-facto. The developers had to overcome many obstacles to come up to speed on the code, to learn the new programming languages and tools, and to try to keep the feature set moving forward. None of them had been involved in the original design, but it was now their product, and its problems were their problems.

At the time of this meeting, I was one part of a coaching team which had introduced a great many changes. We were trying to help the organization to build a “whole team” mentality that encompassed documentation, security, systems administration, QA, customer representatives, and developers alike. We’d attacked the problem of matrix management. We reorganized the seating floorplan. I think at some point we’d been an inconvenience to just about everybody. The developers were in the middle of their second two-week iteration.

On this occasion, the team was in the midst of a recurring production difficulty, and had been gathered into the CIO’s office to work through a top-20 list of problems to solve.

“Your attitude has been affecting other departments”, he said. “And I want to thank you for it.”

• Checks out the source tree
• Runs the script
• Starts working

So far, everything is great and this part of their build system is essential to their environment – and good in general.

Here’s the next part…

• Create the file
• Update project information
• Rerun the script to regenerate project information.

When I asked how long it takes to add a class, I was told about 5 minutes. So if I want to add a header file and source file, it takes 5 minutes. That’s a big problem. Why?

After this discussion, I heard one of the senior people lamenting that a developer had put another class in an existing file rather than creating a second file. He said something like “If the developer just had discipline, he’d to the right thing.” Those darn developers.

It takes 5 minutes to add a few files to a build. That does not include build time. That’s just the time to configure the build information.

Does 5 minutes seem like very much time?

• Some header files defined multiple classes
• Some source files had the methods for those multiple classes
• Some of the header files had names that did not match any of the classes defined in that header file

So is this a problem?

Nothing + Nothing + Nothing eventually equals something

5 minutes may not seem like a lot of time, and if it were isolated, then it’s probably not a problem. On the other hand, when you multiply it by a team and time, you end up with big problems.

Imagine, you need to use class X. Its header file is actually named Q.h and by the way, classes T U and L are defined in that file – none of which you want to know about.

So your class now has phantom dependencies on T, U and L. Also, how did you find the right header file? A quick search (time wasted). Someone changed U, so you end up having to recompile even though you don’t care about nor use (wasted time). I’m sure you can come up with a few things on your own.

So what do you do about it?

Every solution introduces problems

• Time wasting adding files
• It requires discipline to add new files, so it doesn’t always happen
• A name is wrong, but it’s a pain to update the build configuration, so it doesn’t happen – not all the time, just every so often

Little by little, things get a bit more chaotic.

So now that we’ve observed a problem – some waste, we need to find a way to remove the need to update the build information and regenerate to even work.

• Introduce interfaces
• Inversion of Control
• Identify seams and use some of Feather’s stuff to introduce them
• Etc., the usual stuff to introduce seams and break dependencies.

But many of the refactorings they’ll want to use will involve creating new classes. Since that takes a little bit longer, it will slow everything down – or seem so daunting, it might not happen at all.

Here’s a personal example. A few years ago, I built the security system for one applications and then a suite of applications with single sign on. When I initially introduced the security system, many people wrote tests that would forget to log out, causing problems in both the real system and the simulator.

I kept grumbling. I though, “if people would just do it right, there wouldn’t be a problem.” If they just had a little discipline.

• Create an abstract test base
• Change tests to use it
• In the setup, the base logged in
• In the teardown, the base logged out

• Reduced code duplication
• Increased the stability of the tests
• Made it hard for people to mess up (so long as they used the correct test base – and I updated all of the tests that needed it, so going forward, people had correct examples upon which to base their work).

Ultimately, this removed a lot of my wasted time

Detecting waste is the first thing. Until you know it is a problem, you cannot do anything about it.

• If that person was only following “the rules”
• If he/she just had a little more discipline
• Stupid user, they should not have done that

Ask yourself if it’s possible that those statements are directed at the symptom, not the problem.

I’m a pessimist by nature. I see a project in decline, sucking in other resources, like a black holes eating neighboring galaxies, snuffing the light out of stars, just like projects can suck the life force out of individuals.

However, energy can escape from a black hole, eventually leading to the demise of the black hole. Energy can escape along the axis of rotation in either direction. In a sense, if we see the expansion of the black hole as a bad thing, anything that diminishes a black hole is a good thing. Stephen Hawking theorized that at the event horizon of black holes, fluctuations in the space can cause the spontaneous creations of pairs of particles and their anti particles. This is another way it can appear that a black hole is losing energy since it gives off particles while the opposite particle goes into the black hole.

Now I’m off track. I wanted to talk about how synchronized give a similar effect as the massive gravity of black hole. But instead of curving space-time, they invert the possible state space of code in the presence of threads. Let’s take a look.

public class X {
   int value;

   public synchronized void incrementValue() {
        ++value;
    }
}

   public synchronized void incrementValue() {
        ++value;
    }

Then there are only two orderings, Thread 1/Thread 2, or Thread 2/Thread 1. All of those paragraphs at the top were to make this analogy.

OK, so if we use synchronized that that’s the black hole. The number of possible paths through the code for N threads is N!. What’s the opposite? What happens if we take off the keyword and then run two threads through it? In how many ways can that single line of code get executed?

Are you ready for it?

Really? Well at least conceptually. In practice, the Just In Time compiler will probably turn the JVM instructions into native code, so the actual number of individual steps will probably be less. But there really are that many paths through the system. You might ask how?

That single line of code is represented in byte code as 7 lines. I could show you the disassembly, but really, it’s 7. You can convince yourself by having a look at the byte-code using a nice byte-code viewer plug-in for eclipse.

What happens if we change the int to a long? We still have 7 lines of byte-code, but with different instructions. We have a total of 4 long reads/writes. Each long read/write requires two individual 32-bit operations – or at lest the JVM can implement it that way. The actual number of atomic operations after the Just In Time compiler has had its way with your byte-code will probably be less, but if we stick to byte-codes and the Java memory model, the number 3,432 is now a whopping 705,432!

Let’s extend that just a bit more. What if you have several lines of code? Each line of Java results in many more lines of byte-code. What if we have have 3 lies of Java? We probably have something like 21 lines of byte-code. How many paths of execution would 21 lines of byte-code executed by two threads have? 538,257,874,440!

Where the first program ended up being a white dwarf, the version using longs was a nova. I’m not sure what to call the thee line Java method, maybe a hyper-nova?

In practice, the actual number of paths through a method will end up being smaller. And, most of the paths do not have any negative side-effects. The problem, however, is that there are a small number of hard to find paths of execution that do cause problems and finding them is like looking for a needle in a haystack.

Remember what adding the keyword synchronized did? If we use it on a three line method, it turns 538,257,874,440 into 2. It collapses the number of paths into 2 for two threads. And 2 is less than 538,257,874,440, for even vary large values of 2.

How did I derive at these magic numbers? Two ways.

I wanted to know how many possible orderings there were for 7 instructions and two threads. I knew it had to do with combinations and permutations but I just wasn’t smart enough to figure it out. Searching on Google didn’t do much for me either. So I decided to write a program to numerically derive the result. I tested my way into a program that would calculate the result.

1. Determine all possible orderings of each of the unique “nodes” in a system. For example, if I have 2 threads and 2 instructions, I can generate a natural key (ugh!) to represent each of the different combinations of threads and instructions: T1S1 T1S2 T2S1 2S2.
2. Produce all valid permutations of these four “nodes”.
3. Remove any that have invalid orderings.
4. Count the results.

Many of my first implementations for some of these steps were exercises in writing really inefficient code! But having those tests made it really easy to swap out better algorithms when my brain caught up with what I was doing.

What is an invalid ordering? The code generated by the single line using the pre increment operator has 7 steps. Those 7 steps are in a single sequence with no looping and no conditionals. That’s like saying A is always before B. So if we generate a list of orderings that end up including that combination, we’ve generated an ordering that cannot happen.

1. Register a node for each thread, step combination. 7 threads, 4 steps = 28 nodes.
2. Register dependences between steps. For example, if I have seven steps and two threads executing the same sequence, I could describe this as: T1S1 -> T1S2 -> T1S3, etc. And for the second thread: T2S1 -> T2S2 -> T2>S3. Since the name is arbitrary, I just used letters, so in fact I had: a -> b -> c -> d -> e -> f -> g for one set of dependencies and h -> i -> j -> k -> l -> m -> o

In fact, here’s the test I used to describe that exact space:

    @Test
    public void twoThreadsAndSevenSteps() {
        calculator = new PathCalculator(14);
        createLine('a', 'b', 'c', 'd', 'e', 'f', 'g');
        createLine('h', 'i', 'j', 'k', 'l', 'm', 'n');
        start();
        int totalPaths = calculator.totalPaths();
        stop(2, 4, totalPaths);
    }

By the time I got to this point, I wrote some “tests” using JUnit as a harness to run my code and display timing information and a summary of the

Time:        4ms -- Threads:  3, Opcodes:  2, Total paths:      90
Time:        1ms -- Threads:  2, Opcodes:  3, Total paths:      20
Time:       54ms -- Threads:  4, Opcodes:  2, Total paths:    2520
Time:        3ms -- Threads:  2, Opcodes:  4, Total paths:      70
Time:       61ms -- Threads:  2, Opcodes:  5, Total paths:     252
Time:     1248ms -- Threads:  2, Opcodes:  6, Total paths:     924
Time:    12819ms -- Threads: 10, Opcodes:  1, Total paths: 3628800

Total Paths = 252 * 3.6667 ^N-5

The morning after, Uncle Bob sent an email, showing me his work on how he calculated.

1. T be the number of threads
2. N be the number of steps

Total = (N*T)!   /   N! ^T

When I plugged numbers into his formula, they fit my formula. His formula generated an exact result as opposed to my estimate based on fitting a curve. Another thing his solution has going for it is the algorithm I came up with, I believe, is NP-complete. The net effect was I that while I was able to calculate result for 2 threads and 6 steps, 2 threads and 7 steps took a long time before running out of memory with a 2GB heap space. Bob’s formula gave me a so-called verifier (as did my formula – only Bob’s was better) and the algorithm seems to grow based on factorials and not polynomials.

What’s the point of all of this? The single, one-line program, is a demonstration of why going from one thread to two threads makes writing/debugging/testing programs so much harder. It also numerically demonstrates why the single responsibility principle is even more important when we start writing concurrent programs. The more you put in one place, the large the state space. Larger state spaces:

1. Make it harder to find that one or few number of paths that can cause your program to die, under load, in production, while you are on vacation
2. Make your efforts to test (which is a sampling technique anyway) for concurrency issues all the more difficult
3. Make it harder to find what might actually have caused a failure once you’ve notice it.

In a later post, I’ll describe how we can improve our chances of covering this huge space of potential orderings.

If you’ve made it this far, wow! Congratulations!