Object Mentor Blog: Category Tim's Tepid Torrent

Top Refactorings

tottinger — Thu, 26 Jun 2008 01:10:00 -0500

This has been done before by another mentor, and it was fun. I’ve been watching my use of refactoring tools, and here is my top five list of seven most-used refactorings:

Rename
Introduce Variable
Extract Method
Inline variable
Inline method
Move Method
Change method signature

You might guess that I’m doing a fair amount of refactoring on legacy code.

I was suprised how much I’m using introduce/inline variable, but quite often I’m doing that to make a block of code ready for method extraction. Having broken out small methods, I sometimes find a larger method to extract, and then I inline the use of smaller methods. It’s sometimes hard to find the right level of generality.

I’d have trouble imagining that the first three weren’t everyone’s favorite refactorings.

Quality, Timing, and a Clash of Values

tottinger — Thu, 22 May 2008 22:31:00 -0500

I came across a quote in something I was reading, and I can’t recall if it was a blog, a cartoon, a book, or what. A young woman was talking to a politician and asked if he had to make a lot of compromises. When he admitted that he had to compromise a few times, she shut him down by saying “you can only compromise your values once, and after that you don’t have any.”

I had a talk the other day with a fellow mentor, and brought up line only to have it returned to me (with top spin!) when we were talking about when would be the right time to turn on an older suite of tests in CI.

I finally had a chance to examine the decision point. Of course, it’s always the right time to turn on more tests. When do you not want to know if your code stinks? On the other hand, I knew:

these forgotten tests were not used by developers, and so are of dubious quality
the team in the middle of a big stabilization crunch
the team is in their final coding week before release.

While they need to know their code and test quality soonest (preferably before release) there was emotional foot-dragging on my part. I knew that they didn’t have the headroom to even triage the old tests to see if they should pass.

The problem was not of compromising my principles so much as choosing which of my principles would have sway. On one hand, the quality driver knows that we need more tests sooner. On the other hand, I knew how close the team was to being overwhelmed to the point of shutting down. My nurturing, coaching “spider-sense” kicked in when I wanted to turn on the tests, and my XP quality-driving “spider-sense” kicked in when I thought about not turning them on.

I decided to wait only a week, and then get the tests turned on. Was it the right decision? I can’t tell. I am truly ambivalent; both drawn and repulsed by the decision. Sometimes the good is the enemy of the best and it’s hard to see which is which.

I chose to be the kind of guy who would err on the side of compassion. It’s the more personal and superior of the two values. Having decided, I move on.

But now it’s time to get those tests running, and sooner is better.

Python's Mutable Default Problem

tottinger — Thu, 22 May 2008 22:16:00 -0500

Today I was perusing a very fine python article all professional python programmers should read. I had to grin when I saw one of my favorite python quirks/flaws.

Python only evaluates default parameters when it defines a function, leading to surprising side effects (example from saifoo):

def function(item, stuff = []):
    stuff.append(item)
    print stuff

function(1)
# prints '[1]'

function(2)
# prints '[1,2]' !!!

The grin on my face was because I was bit by this one again last Tuesday. You just don’t want to make a mutable object a default parameter. The solution I used was close to the solution from saifoo:

Saifoo.net:

def function(item, stuff=None):
   if stuff is None:
      stuff = []
   stuff.append(item)
   print stuff

This, of course, gives the correct behavior because it creates a new empty list each time it is called without a second parameter.

My solution actually combines some advice contained earlier in the article with the solution given for this problem:

Mine

def function(item, stuff=None):
   stuff = stuff or []
   stuff.append(item)
   print stuff

It’s not necessarily prettier, and I struggle with whether it is more obvious (to a Python programmer) or not. I fear it may be clever (a word I only use in the pejorative sense), but it is also clean-looking to me.

The point remains, though, that the mutable default parameter quirk is an ugly corner worth avoiding, and I suspect that any code that purposefully exploits that behavior to populate a structure that persists across calls will run a very real risk of being misunderstood.

We all know what happens when the next programmer to touch a program misunderstands it.

VIM as a Diff/Merge Tool

tottinger — Wed, 30 Apr 2008 21:24:00 -0500

Do you need a really quick, clean, powerful diff/merge tool that you can use in windows, mac, and various Unixen? As a happy and pround vim user, I have been enjoying vimdiff for a number of years.

        > gvim -d file1 file2
        > vimdiff file1 file2

(G)vimdiff is wonderful. You can move changes between two versions of a file with great ease (if you know vim) and amazing speed. It can handle files of any size and does a fine job of syncing up both versions.

Vim can do just about anything. I have learned that no matter how much I use and study Vim I still am just scratching the surface of what the editor can do. I find little tricks and configuration options, new keystrokes, and fun little bits of minutae. I think it’s all new. With my memory, sometimes I just think something is new, and have to go check my past notes to see if I already knew it.

Today I (re)learned about diffopt, which makes gvimdiff more wonderful.

    :set diffopt=filler

By default, diffopt is set to “filler”. Filler doesn’t do much, it just adds vertical spaces to keep the text of the left and right pane aligned. It’s a good setting, and I don’t much like working without it. But there are other settings which are overlooked, even by experts.

    : set diffopt+=iwhite

My second favorite is ‘iwhite’, which stands for “ignore whitespace”. It makes gvimdiff ignore leading, trailing, and embedded spaces so that simple acts like retabbing or deleting vertical spaces won’t obscure actual changes. It really helps you to merge changes in situations where you’ve adopted code that was built to entirely different standards.

    :set diffopt+=icase

iCase ignores capitalization, which is not as useful as the other options, since programmers are normally interested in case, but it may be handy when someone has corrected and reformatted SQL. I don’t expect to use it much.

    :set diffopt+=horizontal

The last interesting bit is the choice between vertical splits (the normal case) and horizontal splits (for very long lines on cinema-aspect screens). I hope to never need this option again, but I have needed to deal with absurdly long lines.

If you can use VIM at all, you will find vim’s diffmode to be a very powerful and useful tool. If you can’t use vim already, then maybe vimdiff will be the reason you will learn vim.

Magic Funnel, Part 3: Covey's Miracle

tottinger — Thu, 17 Apr 2008 22:15:00 -0500

We had a team of surrogate stakeholders in the room, and all the very most important stories (as far as we could tell) on a table in front of us.

“These are the most important?” Heads nodded. “Then,” I asked, “what makes them important?” Brows furrowed, and well-considered answers poured forth.

I collected their reasons for importance. Some were about keeping promises, some were about providing increased quality (uptime, correct function), and there were other kinds of importance. We wrote them on a 3×5 card. Ultimately, we realized that the company’s reputation is their source of importance. Users needed to know that the company will do the right things, and provide software that worked and kept them working. I thought it was a good criteria.

Next I explained that urgency is different from importance. There may be a hot demand for some cosmetic changes that don’t really impact the company’s reputation one way or another. There may be some changes that in the direction we want to go. These may be hot topics, but may not be truly important.

We could see a difference between the forces of urgency and the force of importance, but there is gray area as a sufficient pent-up demand for otherwise unimportant changes could in fact effect our reputation for service. Still, we could usually see a crisp-enough boundary for any given user story.

I then (re-)introduced the team to Stephen Covey’s four-quadrant system. It has two axes. The vertical is importance (important at the top, unimportant below). The horizontal is urgency with urgent things to the left and non-urgent on the right:

Q1 Important and Urgent	Q2 Important, not Urgent
Q3 Unimportant and Urgent	Q4 Neither important nor urgent

We took our set of cards and the team determined groupings. Was this a quadrant one activity? Quadrant 3? Quadrant 2? There were some real surprises, especially when we realized our “most important” bit of work was not urgent and should wait until quadrant one activities were finished.

Covey’s recommendation is that we tackle first those items that are both important and urgent. When we knock that stack down, we move on to those that are important but not urgent. If we have time left, those non-important-but-urgent items can “fly standby” in our sprint. We reordered the priorities in accordance with Covey’s recommendation.

We had no cards that were both unimportant and non-urgent.

At this point in time, we had what looked very much like a magic funnel. Ideas came in from everywhere, and the items that moved from the planning team to the developers came out in the “magically correct” order. We agreed that the selection and the ordering were the best we could do at the time.

Our next sprint planning meeting was remarkably efficient.

Building Magic Funnels, Part 2: Pragmatic Pedantry

tottinger — Thu, 17 Apr 2008 22:00:00 -0500

The middle of the funnel we started on needed work. While it is a simple idea that the single, most important thing in the funnel is the first to emerge from the bottom, it is a multi-flavored affair to try to manage for real.

My first strategy was to get the right people in a room and have them fight it out. I think that the prioritization process should be a lot like local government (maybe a school board?) in that people should argue and complain and push and eventually settle on compromises and deals. Ultimately, I believe that people who have a strong interest in the company can make the right decisions. At least they can be right enough for the next 5 days. When you have one-week iterations, the next chance to change the agenda is never far away.

This first strategy didn’t work out, and so we went to a backup plan. A C level manager said he knew what we should do, and so we scheduled an hour or so with him, myself, our priority manager (“funnel guy”) and the senior technical developer.

Our guys used the sticky post-it 3×5 cards and papered the CIO’s windows and whiteboards. They listed the various categories of work from the various stakeholders and pasted them up in priority order.

I asked my first pedantic question: “What is the single most important thing we can work on? If you had only one story that you knew for sure would be finished this week, which would it be?” That led to a nice discussion.

When they placed the card on the table, signifying that it was definitely in the build, I asked again if there was any one card anywhere else in the room more important. I asked if that was really the single most important one.

When the answer was “yes, absolutely” I was ready for my second pedantic question: “Now that this card is off the board, what is the single most important card left for us to do, if only two stories were guaranteed to be done.” Now the pedantry was fully exposed, but the idea had carried. The team collected all the most important stories and placed them in order on the table.

Now I was ready for my next round of pedantry.

Building Magic Funnels, Part 1

tottinger — Thu, 17 Apr 2008 21:38:00 -0500

The idea of a magic funnel, as you may remember, is that there is some kind of organizational structure where many ideas and proposals and issues go into the top. Through some magic, the item that emerges from the bottom of the funnel (to go to the development team) is the single most important thing they could work on next.

This is all just backlog management and prioritization, of course. But I think it can be simpler than I’ve seen it in the past, and that real people working without magic can approximate it.

Recently, I’ve been working to establish another magic funnel. One of the first things we did was to find the person who formerly handed out work to the developers and made him a single point of contact. In our case this is not the scrum master, but is another trusted line-manager type. He has been given a number of people to work with on the Customer side of the house and also works with technical people.

We have tried to establish story feeds from all the various stakeholders. Developers, operations people, technical writers, customers, sales people, marketing people, inside security consultants, and others have been feeding their ideas in to our point-of-contact man. This part of the funnel is working fairly well.

The next thing we have tried to do is to match the feed of work to the rate at which work can be done. This has created a fair amount of back-pressure on the stakeholders (which I believe to be healthy).

We have also worked on making the bottom of the funnel narrow, meaning that our guy doesn’t scatter new work to the team, but feeds it through the scrum master who protects the team’s productivity and keeps the work flow and tasks visible on the status board. He makes sure that the team is not expected to “absorb” changes, but that adding two points of work to the iteration results in two points of unstarted work coming off. This also creates a healthy back-pressure.

As a pun on “in-flight”, I named another area of the board “flying standby”. This is for stories that aren’t important enough to swap a story off the board (or for stories that are displaced by more important ones). If the developers finish more work than they expected, there are stories that can be picked up even though they’re not a scheduled part of the iteration. Stakeholders are told that there is no guarantee of these stories being picked up at all in this iteration, but there is some small chance that it could happen if the team discovers that it has overestimated some other stories.

The bottom of the funnel is working pretty well.

What’s missing is the “magic” bit.

The Magic Funnel

tottinger — Wed, 09 Apr 2008 13:18:00 -0500

At a planning meeting, the developers on the team offer up some amount of work they are able to do. This capacity is termed velocity. The customer part of the team then selects enough user stories to consume all of the offered capacity. This is simple negotiation, but that hardly tells the story. What is happening on the customer side is a magic funnel.

All kinds of stories come pouring into the customer group. Some are from actual users in the field who want improvements to existing features. Some suggest entirely new areas of functionality into which the program could expand to good effect. Some are from the operations group, generally describing ways to make the product more reliable, configurable, or manageable. Some are from the developers who realized that some small changes in code could provide large changed in capability or usability. Some are from external regulatory bodies, some are from quality control, but for the most part they’re all good ideas and many have urgency.

When the magic funnel is working, we pour all these stories into the top, and the thing that falls out the bottom is the single most important, urgent task we could possibly do. When it comes time for a new iteration, we fill in our allotted space with whatever comes out the end of the funnel. Whatever we start doing on Monday, it is the best thing we could possibly start doing. We know that it is true because it came out of the funnel that way.

Sadly, magic funnels are hard to come by. We approximate them with human beings in multi-disciplinary small groups. If we leave the funnel purely in the hands of operations, then operational stories are the first to come out. If we leave the funnel in the hands of marketing people, we tend to do things that are more speculative even though the product users don’t really need them right now. If it’s sales, then whatever will sell more product this month will be the first thing out of the funnel. Give control to customer support and we find ourselves constantly appeasing customers and not moving forward on larger issues. If we leave it to developers, it will tend to turn out those stories that improve the experience of developing the application. Of course the problem is about conflicting perspectives (all of them right!) with competing desires being processed in fixed time.

I’m not an expert in creating magic funnels, though it would be a very interesting direction to take at this point in my agile coaching career. I believe that the answer is ultimately to be found in people working together, well-informed people arguing and debating and even trading favors.

I think that part of the answer is Covey-style prioritizing to select the urgent and important (Q1), then the important items that aren’t urgent (Q2), and holding all unimportant work until there is nothing better to do.

Importance in this case relates to how much the change will affect the future of the product and the company. This is not just a matter of closing a contract or soothing an angry customer. It may be that the future is larger and brighter than our immediate concerns. Immediacy is â€œurgencyâ€, but the leverage to create a more greater future is â€œimportanceâ€. I had to read Covey’s First Things First several times to really understand the difference, as may the kind reader. But I did manage to absorb the fact that until you know what you want your future to be, you don’t know what is important.

But imagine how great it will be when you have established your magic funnel, and the work being assigned to each iteration really is the most important and valuable thing that you could do next. Imagine what it would be like if you really were building the future you would like to live in.

Your Attitude is Affecting Other Departments

tottinger — Sat, 05 Apr 2008 21:17:00 -0500

The CIO looked into the eyes of his agile development staff last Friday. “Your attitude is affecting other departments” he said.

I’ve heard a lot of department-level speeches start this way, and in relatively small companies it is not unheard-of for a C-level manager to address attitudes of development teams.

The group has been working to stabilize and improve a product that was developed by a tiger team of outsider contractors and handed over to the in-house team post-facto. The developers had to overcome many obstacles to come up to speed on the code, to learn the new programming languages and tools, and to try to keep the feature set moving forward. None of them had been involved in the original design, but it was now their product, and its problems were their problems.

At the time of this meeting, I was one part of a coaching team which had introduced a great many changes. We were trying to help the organization to build a “whole team” mentality that encompassed documentation, security, systems administration, QA, customer representatives, and developers alike. We’d attacked the problem of matrix management. We reorganized the seating floorplan. I think at some point we’d been an inconvenience to just about everybody. The developers were in the middle of their second two-week iteration.

On this occasion, the team was in the midst of a recurring production difficulty, and had been gathered into the CIO’s office to work through a top-20 list of problems to solve.

“Your attitude has been affecting other departments”, he said. “And I want to thank you for it.”

Velocity is Just Capacity

tottinger — Wed, 02 Apr 2008 19:23:00 -0500

I don’t know why, but somehow velocity as a term really bothers me.

We always want higher velocity, and that’s a good thing. But with the term velocity, we think that we can create velocity with greater pressure (thrust?). Doesn’t it make sense that with a really hard push you can get greater velocity?

I guess I get hung up on words. Essentially, “velocity” is fine. It is the rate at which we’re “burning through” our stories. We use “yesterday’s weather” to set our level, and we are always seeking any sustainable way to increase our velocity. Why would we produce only 12 points when we could produce 20? As motivated developers, we aren’t yearning to accomplish less. But somehow this doesn’t tell the right story.

When our velocity is decreased, it always seems like a tragedy. We have all felt it when our 23-point iteration is followed by a 21-point iteration. We can justify it a lot of different ways, but it always feels tragic. Worse, hitting any kind of plateau seems frustrating.

You increase velocity by pushing harder. If you push harder, and the numbers come up, then you’ve successfully increased the velocity and that seems to be intuitively all there is to it.

I’ve reflected about this for a few years before I realized that I really like the term capacity much more. It is also accurately describing the rate at which we can truly complete stories. Since we use yesterday’s weather, we can even use the term “proven capacity” to describe that we’ve demonstrated the speed. But it also says some other things.

Intuitively it seems that we know that you don’t push your people beyond their capacity to get work done. Instead, you want to increase their capacity. When we talk about longer hours, it is still obvious that working people harder diminishes their capacity. Instead, we want to be working at capacity on the one hand, and increasing our capacity on the other.

Increasing the capacity sounds like the right problem. To improve capacity, you want fresh workers working hard, pairing and communicating, building and/or using better tools (or using tools more capably), using more powerful hardware, learning, and performing the kind of housekeeping that keeps the team productive. Life balance enters into it. Sufficient team size enters into it so you’re not so small you’re short-handed, yet not so plentiful you trip over each other.

I don’t suppose that anyone is going to rename standardized eXtreme Programming terms like “velocity” just because some guy from Indiana says to, but I wonder if maybe we can’t rephrase the problem in such a way that we are intuitively driven toward more reasonable solutions. Maybe we should explain that velocity is just capacity, and then work from that point onward.

Musing over Mutation

tottinger — Thu, 20 Mar 2008 22:25:00 -0500

I read a mailing list entry in which one fellow (who? I can’t remember!) asked another “Do you want to get better at what you’re doing, or find a better way to get the results you want?”

I’m a sucker for a good one-liner. That one had me thinking, and as I’ve had other conversations about innovation, I keep coming back to that line.

In our Agile practices, we work really hard for a week or two, and then hold a retrospective. The purpose of the retrospective is to find ways to work more effectively for the next two weeks. As we develop better software, we also evolve a better team. We use “tricks” such as tracking our velocity and recording blockages on our ‘waste snake’ to provide data for our decisions, and we use gut feel to evaluate those things that feel like collateral effort to us.

If the practice works, we will see incremental improvement in the team. We will develop ways of avoiding special variations, and we will learn to accept our normal variations. It will make us better at the way we do things now.

XP didn’t come from a series of incremental improvements to the waterfall technology. I wasn’t there when it happened, but it seems that they took on an change in axioms. They didn’t strengthen the contracts between groups, but pulled all the decision-makers onto the same team. They didn’t find more careful ways to examine the code they were changing, but rather decided to lean radically on volumes of tests. They didn’t build practices to improve their anticipatory design, they decided instead not to anticipate at all and simplify their design to allow future change. At the time, this was radical stuff.

I’m sure there have been many other less-successful process mutations, but there is no evolution without mutation.

The man behind the iPod, iPhone, and MacBookPro has had some less-successful product ideas, too. Some exciting high-concept products didn’t make it in the wild. But then some new ideas become category killers.

How do we learn to make the axiomatic changes that lead us to radically better ways to get what we want?

TDD on Three Index Cards

tottinger — Thu, 06 Mar 2008 19:39:00 -0600

I had the opportunity to talk to a fellow who missed part of a class on TDD. I told him that I could give him a 15-minute overview, and give him all the essentials of TDD on three index cards.

Yes, I know that volumes have been written about TDD and BDD, and that it’s a large topic with many many branches of application, but I didn’t have time for that. I had time for three index cards. I figure that an index card is a token, and it represents a conversation, and that one can always dig deeper later.

They looked more or less like this:

Card 1: Uncle Bob’s Three Laws (Object Mentor)

Write no production code except to pass a failing test.
Write only enough of a test to demonstrate a failure.
Write only enough production code to pass the test.

Card 2: FIRST Principles (Brett and Tim at Object Mentor)

Fast: Mind-numbingly fast, as in hundreds or thousands per second.
Isolated: The test isolates a fault clearly.
Repeatable: I can run it repeatedly and it will pass or fail the same way each time.
Self-verifying: The test is unambiguously pass-fail.
Timely: Produced in lockstep with tiny code changes.

Card 3: Flow (using the famous three-node circle diagram) – origin unknown.

Red: test fails
Green: test passes
Refactor: clean code and tests

Sure there is plenty more, but I didn’t know how I could provide significantly less. As is, I’m pretty happy with the exercise. Now I am wondering if I couldn’t produce most of the important information I wish to convey as a series of index cards. Would that be cool or what?

Clues For Reading New Code

tottinger — Wed, 05 Mar 2008 19:27:00 -0600

Okay, somebody just handed you a new chunk of code to work on. Your first though on opening the file is “Why, dear God, why?”. How do you get a handle on this masterpiece of clever programming? Let’s look for a few clues.

Where to start?

Documents

Why not look at the documentation? <laughs> Just kidding. You know there’s no documentation, and it’s probably not useful.

People and Interactions

You need a partner who is familiar with the code. Having a guide is better than having a map. Not only will you know where to go, but you’ll know ’’how’’ to go and where not to step. Other people are a wonderful resource. Just don’t settle for someone doing the work in front of you… the goal is to learn as much as to do.

No, really, documents!

OTOH, if you have somethng in the way of a summary or architectural overview, that might help. I’d read the abstract and look at the pictures. We’re after big-picture, so read it only, don’t trust it. Documents are seldom accurate, and seldom for long. It might be good preparation for your partner’s visit.

Use The Tests, Luke

Do you have tests? If you are in a test-driven shop, looking at unit tests is a good idea. If they’re written well, they are specifications (by example) of the system. If they’re written poorly, you know where to start your work at least. Your partner could do a great service by helping clean up the code instead of explaining it.

No tests? Uh, oh. No tests and no doc? Now you’re in trouble.

Scan the File Space

Are the classes well-named? Do you know where to begin working? If you can find code by looking at file names then you’ve got the handle you need. Your partner can help improve file naming if you find the code you’ve been handed is in ill-named files.

So you’re in a file, you have a mission. Either you have tests, or you’re ready to write some.

Sense Of Smell

What if the code is still not obvious? You have one more resource before you have to print out the listing and break out the markers to reverse-engineer your way out of code hell. You have your nose and your refactoring editor. You can use refactorings such as ‘rename’, ‘extract method’, and ‘introduce variable’ to clarify an existing method. You can spot duplication and eliminate it. Maybe you can bring it to a point of clarity, and then you will know

Deep Code Spelunking

It’s time to pour a really big cup of tea, get some food, take a couple of preemptive aspirin, and make sure your printer has paper.

Sometimes, you will have to reverse-engineer some of the code you were handed. Your only hope may be to rebuild it from the inside-out.

You might want to either copy the code to a scratch space to do this. Maybe you want to branch the project in version control. You want to be free to dig in an learn this the hard way, and make it better as you emerge from the depths.

You still can/must use refactoring tricks to capture the knowledge you gain, but you may need to take the code off line. And look for new partners.

Turn Back The Dial

tottinger — Wed, 05 Mar 2008 19:16:00 -0600

The coolest thing just happened! I broke the glass cover off of my watch. At first I thought it was awful, but then I realized that I could turn the hands.

Imagine my joy as I realized that I could make it 11:30 again, and go enjoy another lunch. Meeting at 3:30? No problem, just turn the hour hand up to 6:00 and go home! I can sleep as long as I want as long as I turn it back to 8:00 when I get to the office. All my work estimates are now “five minutes”, and I complete them every time.

My coworkers have no idea the awesome power that I’ve gained with this one happy accident. They ask “what time is it?” and I say “what time would you like it to be.”

Of course the above is a total fabrication. Pretending it’s 6:00pm when it’s 8:00am isn’t going to do anybody any good at all, and is likely to make a mess of things for me.

But people still try to mandate velocity.

Shoveling Code

tottinger — Thu, 06 Dec 2007 12:56:00 -0600

I was ill when the first snow fell last weekend, and didn’t get out and scrape the drive and sidewalks. Sadly, neither did I bundle up my kids and send them out with shovels. As a result, the snow melted and refroze. When I left home on sunday morning, clearing the snow was frustrated by the presence of hard, packed ice under the snow.

Now I have to be very careful when moving in the drive or sidewalk because of the black ice in some places and thick lumpy stuff in others. I could choose to leave it that way and just walk carefully and try to avoid the area, but I really don’t want it to be like this all winter.

Instead, I bundled up and spent several painful hours trying to clear not only the six to eight inches of new snow, but the ice under it. Clearing the fresh stuff is tough because the rough and lumpy ice is under it. The ice snags the shovel and hurts my elbows and shoulders. I use the metal edge of the shovel to plane off the ice so that it’s not quite so difficult, and the work gets a little easier.

Smooth, planed ice is not a solution. It is actually going to be more slippery and dangerous when the sun hits it and it melts a little. It will then refreeze to an even smoother and slicker sheen. I can’t ignore the problem, and I can’t smooth it over and leave it at that.

I invested in some complicated salt product to lower the melting point of the ice so I can scrape it away. It wasn’t enough, but now there are some safe areas in my driveway. I continue to scrape it and hope that I can get it all squared away so that my wife and kids have safe passage, though it is tedious and unpleasant work.

Yes, it’s my fault. I should have bundled up my sick self or my children and taken care of this when the first snow fell, back when it was easy.

Then it dawned on me that my driveway is a lot like source code. If I always take care of it when it’s relatively easy and not too polluted, it will remain easier to deal with in the longer term. The more unsafe and ugly it is, the more important it is to clean it up. It won’t do to smooth it out a little or to leave it be. It needs to be safe for me and my team.

And I guess that’s my parable for today.

Naming: A Word To The Wise

tottinger — Tue, 27 Nov 2007 20:50:00 -0600

Please pronounce your variable names as words, even if they are merely abbreviations. The phonemes indicated by your spelling may correspond to an existing word with an entrenched meaning. It may be one that you do not intend to communicate.

On Being Stupid

tottinger — Mon, 10 Sep 2007 10:16:00 -0500

This was posted originally to a mailing list, but is reproduced here essentially unchanged by request of a friend.

I frequently see code (written by others) that is completely double-spaced, heavily commented, loaded with many abbreviated or meaningless variable names, and hundreds of lines long. In order to read the function and understand what it’s doing, poor Tim must wear out a mouse, skip comments, and track the variables on paper. A “smarter” programmer could just load it into his head, I suppose, but not the simpleton who writes this email.

I’m not smart enough to just read it from top to bottom and understand it. Sadly, when I read through and understand what in the heck the thxIniFvt variable is doing, I will forget it by the time I figure out the purpose(es) of pszVbt. I can spend all day, or even a few days to figure out a method, and that’s an admission of feeble-mindedness to be sure. I guess I’m not up to the level of some of the rapid hackers. That’s a limitation I face most days.

I find that I can sometimes understand a method like that only if I just delete all the blank lines and comments first, then reformat to break lines, then inline all methods with seven or more parameters, and then start renaming variables, extracting explanatory variables, and extracting explanatory methods. I may have to break the method into classes even. I guess I’m not one of the smart kids.

I used to be one of the smart kids. I once built a module so complex and fragile that nobody but me could figure out what to do with it. It was all tables and indicators, and stunningly clever. I am so ashamed that I wrote it. It was such a mistake that they eventually disabled it rather than field it in such a state. That was years ago, but so memorable to me. Other programmers said it was like the inside of a swiss watch, all delicate and perfectly balanced, and scary to mess with unless you first knew exactly what each part was doing, and why.

I would like to be faster than I am both mentally and in the sense of quickly producing code. I’d like to be a little less intimidated at the start of a project. .But I would not want those things if it meant building crap that people who are not appreciably more talented than myself would trip over every day. Instead, I sometimes wish I could teach the really fast, smart kids how to dumb down the code for the rest of us morons to read.

The funny thing is that dumbing code to my level doesn’t make it harder for the smart kids to use it, and sometimes allows a compiler to do a better job with it. I guess stupid isn’t so stupid after all.

Not A Task, But An Approach

tottinger — Thu, 02 Aug 2007 22:14:00 -0500

Transitions are tough. It seems that lately I’ve been getting a lot of contact from frustrated people who don’t really have a good handle on the “drive” part of Test Driven Development. A question heard frequently is: “I’ve almost completed the coding, can you help me write the TDD?”

It seems like Test Driven Development is taken backward, that the developers are driven to write tests. The practitioner winces, realizing that he again faces The Great Misunderstanding of TDD.

TDD stands for Test-Driven Development, which is not as clear as TFD (Test-First Development). If the consultant would strive to always say the word “first” in association with testing, most people would more surely grasp the idea. In fact, I’ve begun an experiment in which I will not say the word “test” without the word “first” in close approximation. I’ll let you know how that works out for me.

If the tests are providing nothing more than reassurance on the tail end of a coding phase, then the tests aren’t driving the development. They are like riders instead of drivers. Test-Ridden Development (TRD)[1] would be a better term for such a plan. Even though it is better to have those tail-end tests than to have no automated testing, it misses the point and could not be reasonably be called TDD.

An old mantra for TDD and BDD is “it’s not about testing”. The term BDD was invented largely to get the idea of “testing” out of the way. People tend to associate “test” as a release-preparation activity rather than an active approach to programming. BDD alleviates some of that cognitive dissonance.

In TDD, tests come first. Each unit test is written as it is needed by the programmer. Tests are just-in-time and are active in shaping the code. Acceptance Tests likewise tend to precede programming by some short span of time. [2]

Through months of repetition I have developed the mantra:

TDD isn’t a task. It is not something you do. It is an approach. It is how you write your programs.

I wonder if we shouldn’t resurrect the term Test-First Programming or Test-First Development for simple evocative power. Admittedly there are some who would see that as a phase ordering, but maybe enough people would get the right idea.

Brett Schuchert(with some trivial aid from your humble blogger) has worked up an acronym to help socialize the basic concepts which are somehow being lost in translation to the corporate workplace.

The teaser: Fast, Isolated, Repeatable, Self-validating, and Timely.

As a reader of this blog, you are probably very familiar with all of the terminology and concepts behind TDD. I beg of you, socialize the idea that testing comes first and drives the shape of the code. If we can just get this one simple idea spread into programming dens across our small spheres of influence, then we will have won a very great victory over Test-Ridden Development.

“And there was much rejoicing.”

1 Jeff Langr will refer to this TRD concept as “Test-After-Development”, which he follows with a chuckle and a twinkle, “which is a TAD too late.”

2 Of course, one still needs QC testing as well, however TDD is about driving development, not testing its quality post-facto.

Tuple Madness and STL in C++

tottinger — Thu, 19 Jul 2007 22:21:00 -0500

I have already complained about Tuple Madness with reference to Python and Ruby, and UncleBob has seen the same problem and noted it on the old blog. I have come to realize that the same problems appear in modern C++ code.

I have always looked at the STL as a way of implementing a class, but never as a replacement for “real” classes with named member values. In some ways, I think that templates have made our C++ code worse.

For instance, I need a two-member structure. Say I am going to store Name and Address. Now, those with C background would immediately think along these lines:

    struct name_and_address { char name[34]; struct address address; };
    struct name_and_address get_name_and_address(void);
    // blah blah blah

    struct name_and_address x = get_name_and_address();
    printf("%s", x.name );
    printf("%s", x.address.line1);
    // ... and so on...
    printf("%05d", x.address.zip);

Got to love those old C programmers. There is a nice, primitive struct, maybe with a nice, primitive struct embedded in it. Look at the clear names of those variables. It is even pretty efficient to copy these things. With a Plain Old Data Object, the compiler knows what you’re doing and can do block copies instead of memberwise copies. That can be very efficient, enough that you can often pass structures around by value instead of by reference. That’s kinda cool. That same efficiency also applies to plain old data classes in C++, but a lot of people don’t know it.

Sadly, a modern C++ developer does what? Yeah, he returns a pair. This is annoying because when I call my function, I have to know that the name is now called “first” and the address is called “second”. But that is only for this pair-returning function. In another pair-returning function, “first” means birthday and “second” is account number. Of course the prototype is not always in view, so I have to have more tiles in my window to check my work as I code.

Pair is named well enough for an arbitrary pair of primitive values, but frankly it makes no sense for x,y coordinates or names and address, or for the rolls of two twenty-sided dice. Wait. For the dice, it works just fine. After all, what would you call them? Dice1 and Dice2? But I digress. The point is that for most two-member structures, a two-member structure with named members is probably a better choice.

Now pair is not alone (oh look, a pun). Even when we are dealing with collections, the interface to a vector, map, or set is unlikely to be the right set of names and semantics for our problem domain objects. Maybe our property should be listed, and not push_back()-ed. Maybe we care about a policy, that:

        policy.covers(theCar)

Maybe we’re not so interested in whether list x will:

        x.find(z.first) != x.end()

The fascination with exposing STL containers (combined with unfortunate naming, as given here for effect) is hurting readability. It also is a very unfortunate form of primitive obsession and a violation of the Dependency Inversion Principle.

The people who work on your code with you should not have to reverse engineer your work in order to add new features or correct bugs. The code should be written so clearly that mistaken assumptions will be more obvious. They should not have to chase down typedefs and keep notes on paper to keep track of your structures. They should especially not have to look for data in obliquely named pairs of pairs (x.second.first means “price”? Or was that x.first.second? Or…).

If you write your code with a pair programming partner, you should get plenty of feedback about the readability of your code. Even if you do not work in pairs, you should read your tests when you get back from lunch or a meeting, and you should see if primitive containers, primitive member functions, and template syntax are obscuring their purpose. If your tests don’t speak clearly, then none of the other users of your class can speak clearly. Get the syntax out of their way by giving them reasonable wrapper functions.

You might think that wrapping your containers will cost you performance by adding dispatch here and there, but you might be surprised to find out that not wrapping it may cost you far more by concealing bugs and obscuring functionality than it ever save you in performance. A better policy is to keep your code obvious and simple, and try a little profiling and optimization where it really matters.

Of course, if one was worried about the cost of one dispatch for a named method v. an exposed container, there is always the option to build an inline member function in the header.

I’m mindful that this is not the time for me to rant about C++ programmers who never profile, and who manage performance by rule of thumb and old wives’ tales. There are more of them than us, my dear reader, and I fear their wrath. Maybe some day later, when I’m feeling brave.

My roots run pretty deep into C++, and I appreciate how hard it is to produce a system. I just want the guys who are now running with the ball right now to realize that it is far more valuable to be clear and obvious than they appreciate, and it is far less expensive to do so than they fear.

I guess the points are:

Don’t be afraid to write small classes and structs.
Don’t be afraid to wrap your containers.
Don’t be afraid to use abstraction.
Everything does not have to be written in templates.
Be afraid to write test code that doesn’t tell a story.
Be afraid to give too much access to your data by exposing containers.

Adjust your values accordingly.

Testing Will Challenge Your Conventions

tottinger — Tue, 17 Jul 2007 22:28:00 -0500

If you are doing test-first development, you are likely to find your old coding conventions are no longer valid. There are a few changes you will need to make to your coding standards and practices.

1. Interfaces suddenly seem like a really good idea when you have to start introducing test doubles of various flavors (mocks, etc). You will find yourself creating interfaces in places where you probably would not have created them before. You’ve always known that you shouldn’t depend or derive from concrete classes, and now you are feeling the pain of non-trivial concrete classes. You are forced, more or less, to comply with the Dependency Inversion Principle. Abstraction becomes a way of life.

2. Singletons and static methods no longer seem like a great way to do work (esp in C++) because you can’t easily isolate a module that makes use of them. The exception is when a static method is totally self-contained and testable, and does not use resources like a database, file system, etc. Only the very trivial can be static. Test doubles matter that much. Suddenly substitutability is a primary measure of code goodness.

3. Private makes less sense than it used to. You can’t test anything that’s private. You need to have ways to sense that your tests are working as intended, and you have to be able to test any method that is interesting. That means less private and more accessors. Get used to living in a more public world. If you need to hide something from users, don’t include it in the interface or abstract base class. “Implements”/”public inheritance” is the new “private”.

4. You need to be able to pass a class everything it might need at construction time, so that you can pack it a stubbed logger and a faked database access module. Otherwise, you will not have the isolation you need to write good unit tests. Calling a concrete class constructor inside the body of a method now makes you cringe, because you realize you’ve missed an opportunity for abstraction, and limited the isolation of your method. The fat constructor argument list is a concession to the need for isolation and testing with mocks (and fakes, and stubs).

5. Smaller methods are the norm. It is hard to test a two-hundred line method. It’s far harder than testing a many smaller, equivalent methods. Whatever is easier to test is easier to write. Large is bad. This has always been so, since optimizers love simple functions, but now it’s real to you as well.

6. You hate inheriting code without tests, because you don’t want to reverse-engineer what the other programmer was thinking. In fact, you find yourself looking at tests before you look at code, because you know the tests are better than comments. Testing becomes a kind of “code clarity” mechanism. The profound bit is that code is more clear if it is more testable, even if you have to “pollute” the class to make it testable by moving variables and methods out of the private space or by creating interfaces, or by using fat constructor argument lists. Testability is the new legibility.

7. Hard-to-use class interfaces are now hard for you to use, not just hard for other people. You have to write the tests, so you have to use the class. Usability determines how much you will like your own code, not cleverness.

8. Performance management by old wives tales is dead. You will build for testability, and then use measurement tools to improve performance. This is what the wizened greybeards have been telling us for a lot of years, and with TDD you start listening. You stop avoiding trivial issues like copying PODS and virtual dispatches. You pay attention to more important issues. With optimization, YAGNI applies until it doesn’t, and when it doesn’t you need to know why and where.

9. You care deeply how long it takes to run the tests because you need to be able to run all of the tests all the time, after only a few lines of code are written, and after each refactoring. If it’s taking more than 20 or 30 seconds to run all your unit tests, you start looking for ways to speed it up. You must maintain test performance, because it is key to productivity.

10. Dependency hurts. You can’t afford ‘god classes’ and ‘global hubs’. If test setup becomes a chore, you immediately start reaching for the axe. You have to keep your modules isolated better than ever before. Of course, you should have done this all along, but you never felt it this keenly.

11. “Clever” is dead. Clever is hard to refactor. Clever is hard to isolate, hard to internalize, hard to phrase in tests. One point of “obvious” is worth two hundred points of “clever”.

12. Your IDE is good to the extent that it allows you to do quick write/build/test cycles. You need to do several per minute.

For the most part, TDD forces you to start doing the things you always should have done, though it signals a change in values that should be reflected in your coding standards.

Maybe you should get a highlighter and mark each section of your coding standard or architectural document that encourages any practice that in turn discourages testability. The world has changed, values have shifted, and your style guide needs to change if it is to remain relevant.

What I Don't Like About Python

tottinger — Tue, 17 Jul 2007 21:41:00 -0500

I love python.

I love the simplicity, the nice data types, the strong dynamic typing, the significant indentation, the runtime flexibility, the list comps, the generators, they way that I can get work done in Python. I love the built-in help. I dig the Tao of Python.

There are still a few bits that seem artificial and clumsy. I was told that you don’t really know a language if there are not five things you hate about it. I have more than five, but these will do for now:

Dir is not a very good name for the method that dumps the contents of an object. I don’t like the ruby alternative, where methods and members are separate, but I just don’t like the naming here. A change would be possible. It would have to be grandfathered in, but it could happen.

double-underscores annoy me. I would like to see some other way of denoting operators and other magic functions. I suppose one could grandfather-in a decorator-based trick for defining operators. It would be a real compatibility breaker, but I really feel that __getitem__ is probably not the best naming our guys could come up with.

__name__ has always felt like a hack to me. Try explaining it to a novice programmer, or worse to someone who programs in perl or ruby or java for a living. Comparing the magic variable __name__ to the magic constant ”__main__” feels doubly so. I would rather have a convention such as naming a method “main” or maybe decorating with @script_main now that we have decorators.

Blocks give me of ruby envy or smalltalk envy. I like local functions. Love them. I tolerate lambda. But I really, really would like to see a more rubyesque iterator setup where we pass a callable to the list, and the callable can be defined free-form inline. Python doesn’t really do that and so it’s less of a language lab than I might like.

Properties are unattractive, partly because of blocks being absent. I don’t really want to define a named parameter (with double-underscores, most likely) and then two named functions, and THEN declare a property. That seems like so much work for such a simple situation. It is something I will only do if all other methods fail me, or if all other methods are overriding __setattr__ and __getattr__.

Lack of recognition just kills me. This is a wonderful little language with great libraries and tremendous capability. Google, Eve, Nasa, a great many scientific efforts, a bunch of web sites, and a lot of Linux installers and configuration tools use python. A number of very nice distributed version control tools are written in it. But still people seem to go deaf if anyone mentions python, as if we’d mentioned JCL or something. I don’t get that.

There is a very ugly mutual-inclusion design bug bug. I spent time on it once and was very unhappy.

Self loathing takes over in my functions. I wish that I could refer to member variables without saying “self.” first. Testing would be easier if I could just import all my fixture variables into my local namespace—or better if it were done for me automagically. Typing “self.” doesn’t kill me, but it doesn’t help me. Yes, I have an abbreviation in my editor, but it still bugs me.

Don’t put me down as a complainer. I really, really love working in python. But There’s a lot more love than hate here.

PS: Is it uncommonly sensible that “not” is spelled “not”, “and” is spelled “and” and “or” is spelled “or”? I think so. && I’m ! just kidding || something.

Dependency Broadcaster

tottinger — Mon, 16 Jul 2007 20:34:00 -0500

I’m not sure if there is already a code smell name for this situation. The idea is rather like “Large Class” or “God Class” but isn’t really related to behaviour. It’s just a matter of dependency.

Michael Feathers refers to “horrible include dependencies”, and that’s the right idea.

So what if you have a class file that includes (or forward-declares) a few hundred other classes, and that class is used by almost every other class in the system?

This is perhaps what comes from writing a very class-rich architectural layer as a single class.

Mainly the idea is that the class takes a lot of ugly dependencies and spreads them evenly over the application. I would call that “broadcasting” although the correct agrarian term is “manure spreader.” I will stick with the less evocative “dependency broadcaster.”

Use VIM to do TUT Unit Testing

tottinger — Wed, 11 Jul 2007 09:47:00 -0500

I’m not crazy about TUT, but a customer is using it on a project and wanted to get a little assist from VIM. I don’t blame him. It’s a pain to keep track of your test number and there’s always more testing that we’d like in our test frameworks.

So I scribbled up a little VIM script to handle some of the light housekeeping. It’s not a wonderful script, and I’m betting the readers can give me some pointers (I’ve only written a few other vimscripts ever). I’m sure that I could do better if I reflected and refactored, but I haven’t yet.

Maybe you know a good way to unit test vim scripts?.

My script hijacks your F5 key. You can change that easily enough. Maybe I should have mapped it to \nt for “new test” or something. Anyway, enjoy and comment please:

function! NewTutTest() let s:testNumber = 0 let s:newNumber = 0 " Seek a higher number let s:list = getline("1","$") for s:line in s:list if s:line =~ 'test<' let s:newNumber = 0 + matchstr(s:line, '\d\+') if s:newNumber > s:testNumber let s:testNumber = s:newNumber endif endif endfor "Increment (tests are 1..n) let s:testNumber = s:testNumber + 1 "Output the test values let s:line = line(".") let s:indent = repeat(" ", &shiftwidth) let result = append(s:line, s:indent . "template<>") let s:line = s:line + 1 let result = append(s:line, s:indent . "template<>") let s:line = s:line + 1 let result = append(s:line, s:indent . "void object::test<" . s:testNumber . ">()") let s:line = s:line + 1 let result = append(s:line, s:indent . "{" ) let s:line = s:line + 1 let result = append(s:line, repeat(s:indent,2) ) let s:line = s:line + 1 let result = append(s:line, s:indent . "}" ) call cursor(s:line, (&shiftwidth * 2)) endfunction " Paste a version of this line into .vimrc to assign a keystroke (in this case F5) " Comment-out this version if you do " -------------------------------- map <F5> :call NewTutTest()<CR>Aset_test_name(" " Some helpful macros for command mode. Press \ten and it opens a new line " and types ensure(" so you can fill in the string and the parameters. " Overall, not too shabby. map \tn oensure(" map \te oensure_equals(" map \td oensure_distance(" map \tf ofail(" " Abbreviations to help you in insert mode. Type the " two-letter name, followed by a quote. ab tn ensure( ab te ensure_equals( ab td ensure_distance( ab tf fail(

Revisit: The common subgroups

tottinger — Tue, 03 Jul 2007 10:43:00 -0500

In cleaning up the code, I simplified the algorithm a very little and improved performance considerably. Amazing how that works, how simpler equals faster for so much code. Adding simple data structures, local explanatory functions, and the like often make code much faster.

What I’m hoping is that I will use this in a few different and useful ways.

The first way is to look for interfaces where concrete classes are being used from many other classes. You need to add an interface, but don’t know who needs which part. The goal is to figure out a relatively small number of interfaces that satisfy a number of clients in a module.

The second use would be to look for common clumps of parameters when I’m working in a large code base where the average number of arguments per function call does not remotely approach one. I suspect that there are clumps of similarly-named variables being passed around, and that these are likely “missed classes”. Sometimes these are obvious, but it would be good to see them in a nice list spit out from a nice tool.

So this is a hopeful start on a series of useful tools.

Code follows.

import shelve import sys def find_groups(input): """ Exhaustively searches for grouping of items in a map, such that an input map like this: "first":[1, 2, 3, 4], "second":[1,2,3,5,6], "third":[1,2,5,6] will result in: [1,2,3]: ["first","second"] [1,2]: ["first","second","third"] [5,6]: ["second","third"] Note that the return value dict is a mapping of frozensets to sets, not lists to lists as given above. Also, being a dict, the results are effectively unordered. """ def tupleize(data): "Convert a set or frozenset or list to a tuple with predictable order" return tuple(sorted(set(data))) def append_values(map, key, *values): key=tupleize(key) old_value = map.get(key,[]) new_value = list(old_value) + list(values) new_value = tupleize(new_value) map[key] = new_value return key, new_value result = {} previously_seen = {} for input_identity, signatures in input.iteritems(): input_signatures = set(signatures) for signature_seen, identities_seen in previously_seen.iteritems(): common_signatures = set(signature_seen).intersection(input_signatures) if len(common_signatures) > 1: known_users = list(identities_seen) + [input_identity] append_values(result, common_signatures, *known_users) append_values(previously_seen, signatures, input_identity) return filter(result) def filter(subsets): filtered = {} for key,value in subsets.iteritems(): if (len(key) > 1) and (len(value) > 1): filtered[key] = set(value) return filtered def display_groupings(groupings): "Silly helper function to print groupings" keys = sorted(groupings.keys(), cmp=lambda x,y: cmp(len(x),len(y))) for key in keys: print "\n","-"*40 for item in key: print item for item in sorted(groupings[key]): print " ",item print

Python Subgroup Detection and Optimization

tottinger — Wed, 27 Jun 2007 23:23:00 -0500

I had a moderately interesting customer problem to work on. I got acquainted with a bit of legacy code that is seriously in need of some interface segregation. It’s an entirely concrete class and used from all over the code base. The question is how to segregate, and that depends on what methods are called from which programs. We ran ‘nm’ to extract the link table from our object files, saving me the trouble of parsing C++ (a scary thought) All that remained was for me to compare the method prototypes used by the object files and find the common sets.

Lacking better ideas, I decided to do this exhaustively in a brute-force kind of way. It is only a few hundred files, so it shouldn’t take too long. I was very wrong. It took a long time and eventually failed.

I had TDD-ed the code, so I had tests of correctness, and I relied on these as I added optimizations, but of course the performance problem occurred only under a real load.

I could have run a profiler on it (and probably should have) but instead I simply monitored my computer. I quickly saw that my time was going into memory allocation, which is also the reason it died after many minutes When I have python performance problems, this is usually the reason, and almost never “interpreter drag”.

My friend Norbert (a gentleman of many programming languages, including ruby) suggested that I wasn’t interning my strings, and of course I was not. I switched to interning strings, and noticed a little improvement which meant the program ran longer before failing from memory problems. Well, the tests still passed, so I knew at least that the logic was still good even if the algorithm was primitive and Occam was spinning in his grave.

Next I realized that I am dealing with a lot of small groups of strings, and in a leap of optimism/faith/stupidity I decided to intern groups of strings. That helped very little, but it did help since my program was now running for the better part of an hour (!!!) and then failing. “More of same” without measurement is hardly a good recipe for optimization.

This is when I realized that I shouldn’t be storing sets or frozensets, which are pretty heavyweight data structures. I had chosen them because I was really working with set intersections, but hadn’t counted the in-memory storage cost. I converted the data structure used by the algorithm and added a local function to make tuples out of the sorted sets.

I was very glad to have my tests to catch me when I had some typing mistake or sloppy conversion. My tests had to be edited, but the changes there were very lightweight, and caused me to abstract out some data comparisons that were (admittedly) repeated. It was all good.

When I ran the full program it completed so quickly that I was sure I’d broken it. It was running with sub-second time, including gathering data from various text files (nm output files). I did a few spot-checks, and determined that it was indeed doing the right thing (as far as I know).

The data is moderately interesting, and I will be able to pick out some useful interfaces. Better yet, I have a program that can pick out all the uses of my big, fat class and recommend interfaces to me. This is all good.
Lessons learned:

The tests gave me peace of mind as I worked. I would so hate to have done this “naked”.

Python’s speed is fine (even startling) if you aren’t doing something wasteful and heavy-handed like storing hundreds or thousands of non-interned strings and heavyweight data structures

I could have been done sooner if I’d measured with hotshot instead of guessing. This is very clear to me, and I won’t think I’m too clever or my problem is too simple to do this ever again.

Keep some ruby friends on hand. They come in handy.

I didn’t really need a cooler algorithm. Brute force is sometimes enough.

I want to build a new wrapper for this code to compare parameter lists, to help find unrecognized classes in these same programs. It shouldn’t be too hard, but it will be a larger data set so I will probably need some more optimization or a cooler algorithm later. I think I see some waste in it now, but I guess I’ll have to fix that after blogging.

Of course, I’m not done learning. I am sure there are a lot of ways to improve the core code. I also believe in other people, that I should make it available to criticism and suggestions.

Here it is:

Tests
(Which, embarassingly, could use more refactoring)
import unittest import clumps def keySet(someMap): return set(someMap.keys()) class ClumpFinding(unittest.TestCase): def testNoGroupsForSingleItem(self): input = { "OneGroup": [1,2] } actual = clumps.find_groups(input) self.assertEquals({}, actual) def testNoOverlapMeansNoGroups(self): input = { "first": [1,3], "second": [2,4] } actual = clumps.find_groups(input) self.assertEquals({}, actual) def testIgnoresSingleMatches(self): input = { "first": [1,3], "second": [1,4] } actual = clumps.find_groups(input) self.assertEquals({}, actual) def testTwoInterfaceMatches(self):because of group = (1,2) names = ("first","second") input = dict( [(name,group) for name in names]) actual = clumps.find_groups(input) self.assertEquals(1, len(actual)) [key] = actual.keys() self.contentMatch(group, key) self.contentMatch(input.keys(), actual[key]) def contentMatch(self, left,right): left,right = map( frozenset, [left,right]) self.assertEquals(left,right) def testFindsThreeGroupsMatchingExactly(self): group = [1,3,8] names = "one","two","three" input = dict( [(name,group) for name in names ] ) actual = clumps.find_groups(input) self.assertEquals(1, len(actual)) [clump_found] = actual.keys() self.contentMatch(group, clump_found) self.contentMatch(names, actual[clump_found]) def testFindsPartialMatchInThreeGroups(self): input = { "a":[1,2,3,4,5], "b":[1,4,5,6,8], "c":[0,1,4,5] } target_group = frozenset([1,4,5]) names = input.keys() actual = clumps.find_groups(input) [key] = actual.keys() self.contentMatch(target_group, key) self.contentMatch(names, actual[key]) def testFindsMultipleMatches(self): input = { "a":[1,2,3,4,5], "b":[1,4,5,6,8], "c":[0,1,4,5], "d":[1,2,3], "e":[1,3] } actual = clumps.find_groups(input) keys = actual.keys() self.assertEqual(3, len(actual)) grouping = (1,2,3) referents = set(["a","d"]) self.assert_(grouping in keys, "expect %s in %s" % (grouping,keys) ) self.assertEqual(referents, actual[grouping]) grouping = (1,4,5) referents = set(["a","b","c"]) self.assert_(grouping in keys) self.assertEqual(referents, actual[grouping]) grouping = (1,3) referents = set(["a","d","e"]) self.assert_(grouping in keys) self.assertEqual(referents, actual[grouping]) if __name__ == "__main__": unittest.main()

The Code

import sys def find_groups(named_groups): """ Exhaustively searches for grouping of items in a map, such that an input map like this: "first":[1, 2, 3, 4], "second":[1,2,3,5,6], "third":[1,2,5,6] will result in: [1,2,3]: ["first","second"] [1,2]: ["first","second","third"] [5,6]: ["second","third"] Note that the return value dict is a mapping of frozensets to sets, not lists to lists as given above. Also, being a dict, the results are effectively unordered. """ def tupleize(data): "Convert a set or frozenset or list to a tuple with predictable order" return tuple(sorted(list(data))) result = {} for name, methods_called in named_groups.iteritems(): methods_group = frozenset(methods_called) methods_tuple = tupleize(methods_group) for stored_interface in result.keys(): key_set = frozenset(stored_interface) common_methods = tupleize(key_set.intersection(methods_group)) if common_methods: entry_as_list = list(result.get(common_methods,[])) entry_as_list.append(name) entry_as_list.extend( result[stored_interface] ) result[common_methods] = tupleize(entry_as_list) full_interface_entry = result.setdefault(methods_tuple, []) if name not in full_interface_entry: full_interface_entry.append(name) return filter(result) def filter(subsets): # Apology: I'm betting I can do this in a functional way. filtered = {} for key,value in subsets.iteritems(): if (len(key) > 1) and (len(value) > 1): filtered[key] = set(value) return filtered def display_groupings(groupings): "Silly helper function to print groupings" keys = sorted(groupings.keys(), cmp=lambda x,y: cmp(len(x),len(y))) for key in keys: print "\n","-"*40 for item in key: print item for item in sorted(groupings[key]): print " ",item print

The Things That Pass For Simple I Can't Understand

tottinger — Thu, 14 Jun 2007 00:19:00 -0500

(with apologies to Steely Dan for the nearly-lyrical title)

I’ve noticed that “Do the simplest thing that might possibly work” gets universal agreement in principle, and great divergence in practice.

Is a set of variables related by name prefix simpler than a named class containing the same variables? In the “fewest number of the fewest things” sense, I see a group of variables as a lot more to manage than a class. I can’t understand why some people pass the same group of loose variables to a number of methods. Isn’t a seven-argument method “complex”?

Is a custom exception “fancy” or “trivial”? I’ve had that discussion recently. Some feel one way, some the other. Is it more complex to throw a standard exception type and then try to figure out what it means elsewhere?

Is event-handling simpler than polling? I’ve heard this one too. I don’t know how other people see it. I think events have more complex plumbing, but that polling has a greater run-time complexity because of failure modes (periodicity issues, race conditions, etc). I think that it is simpler if you have fewer problems to look out for.

If you need a fixed set of java instances, is an enum simple or fancy? Here you have a newer language feature, but it exists to simplify the management of a fixed set of instances, so it leaves you less to deal with, no?

Is “more primitive” simpler? How about if you use arrays and integer indices into a string rather than a list of strings? Is that simpler or more complex? Is the Bowling Game “simple” because it uses a primitive array and a separate array counter instead of custom objects?

In C++, is it simpler to have a struct that contains two data members or to use the pair<> template? Is it “simpler” that you have to refer to them as “first” an “second”? Or is that just more obscure? I think that “obvious” is more simple.

I am not sure how other people judge simple. I think “simple to use”, “less bookkeepping”, “harder to mess up”, “less setup to call”, “fewer parameters” are simpler. Other people seem to have contrary thoughts.

Clearly simple involves more than “least thought” or “most primitive data components”. It should have something to do with the lack of effort in using something and the ease with which things can be used correctly. Well, clearly to me. And it should have to do with the least difficulty in reading the resulting code, and the least difficulty in changing it.

Unrelated to this, I find some tools that others find simple to be baffling, and some tools that are complex to others seem pretty easy to use to me. I suppose that’s a kind of “personal taste” but is there a more reliable rule to base “simple” on than personal choice? There should be if the rule is for us all to do the simplest thing that works, and to maintain the simplest design we can afford to build.

But I have unique viewpoints sometimes, and maybe simple to you isn’t simple to me. Personally, some of the things that are simple to you may be all but inscrutable to me. I guess that happens.

Collateral Effort Revisited

tottinger — Mon, 11 Jun 2007 20:36:00 -0500

One of the things I love about TDD is that it takes all the scaffolding and collateral effort for creating a class, all indications of bad coupling, and makes them entirely visible. This is also one thing that makes it very hard to start TDD in legacy system.

When you realize that it’s very hard to test a class out-of-context, the right answer is to start decoupling each class so it can be run out-of-context. When we are successful here, tests run very very fast and are easily self-verifying and isolated. It is a beautiful thing even if it follows many ugly hours of painstaking work.

The wrong answer, I’m convinced, is to build a mega-framework for testing that allows you to test in-context. This type of framework typically creates a totally realistic runtime environment (complete with database, configuration files, directories, etc). This approach allows you to think that you can ignore your dependency nightmare. You really can not Mega-test-frameworks take a such a long time to run that developers stop running all the tests all the time. This is a far worse problem than breaking dependencies because it breaks the whole process. Mega-frameworks don’t solve the problem, they only defer the solution until the problem gets worse.

When you are faced this kind of collateral effort, the answer is to work through it, not to sweep it under a rug. Working through it is work, indeed, but this is about doing the right thing not doing the easiest thing.

The Myth of Learning Compression

tottinger — Thu, 24 May 2007 13:56:00 -0500

There is an odd idea that learning is compressible, that any training course of any length can be squeezed into a very small time box and delivered to any large number of people without significant loss.

As an educator and as a guy who has a pretty big learning load at any time, I know that this is a myth. I can cover any topic much better if I am working with a handful of individuals over the course of a week or two. If you double the number of attendees, then the interactivity is highly degraded, and discussions become time sinks. If you half the time, then interactivity is likewise degraded, and material that might have been covered must be skipped or skimmed.

I have a joke with my coworkers that the ideal environment for some large corporations would be a three-train roller coaster. The participants would buy a ticket, climb on board, sail past the materials at speeds in excess of 60 miles per hour, and would collect their certificate at the other end. As one train is unloading, another is loading, and another is in progress so that we could cover many hundreds or thousands of students per day. Of course, they would learn next to nothing and would not really participate at all.

But I respect my corporate customers, including the fact that they have much to do, and cannot easily afford a one-week work-stop for each dozen team members, all with consultant/trainer fees. Some large organizations would have us on-site for years in a row before all their developers would be trained in any technology. It’s just not reasonable, and they push for a more feasible schedule.

It’s a rock and a hard place, and people on both sides understand that it is so.

The good news is that compressibility is not a total myth. There are techniques and tools that can make the training more effective, so that less time is necessary in class. There is the idea of a jump-start, where a topic is introduced and discussed in a short time and students are given materials on which they can continue via self-training. The compressibility myth drives us to more dense and useful presentation. It is a valuable spur.

It also has the advantage of the time box. If you have only three days to teach someone how to TDD, you have to choose the three most important days-worth of material to teach. This kind of prioritization is the same thing that we demand of our product owners in a scrum, and is perfectly fair and reasonable.

Even though compressibility is a myth, it is a useful myth.

The only mythological part of the compression myth is the idea that it is lossless. It is lossy compression. But as is the case with audio and visual compression, compressed learning can be “good enough.” If that’s what we’re after, it’s all good.

Unit Tests Coverage: Less Is More

tottinger — Mon, 07 May 2007 08:06:00 -0500

In TDD a unit test has to be very small to isolate failures. This does funny things to code coverage as a metric. Each test should have a very small area of effect, and so each unit test should have a negligible effect on the overall code coverage statistic. Bear with me here, see where I’m missing out.

Say you have an existing (legacy) system with no coverage at all. Zero percent. If you start doing TDD today, the overall coverage percentage should barely change at all. If only the new code is test-driven, then the old code is not gaining coverage except where tests are necessary to ensure that the new code is being called. The low coverage per test is a good thing because it shows that the unit tests have good isolation. In such a situation, code coverage is really telling you the ratio of new code to old code. Again, this is so obvious and logical to me that I must be missing some cool subtleties.

If all the code was test-driven from the beginning, you should have a very high coverage number, and writing a new unit test before you add code should not impact that number. You only write enough code to pass the test, so again you aren’t getting much in the way of uncovered code. A lowering of the ratio might indicate a problem with test-to-production-code ratio. This seems pretty simple and logical, so I’m sure I’m missing some interesting corner cases.

OTOH, system tests and integration tests paths through many components at once, and should have a more significant effect on coverage in a previously-untested system, though their job is to prove function points, not to raise the metrics. These non-unit tests are the thing that boost your coverage of old code. That is also a good effect, because it is the goal of the system test to ensure that the parts work together.

I’m not saying that code coverage should be low, only that as we move incrementally, the unit tests we write should individually have negligible effect on our overall code coverage numbers… a thought that intrigues me.

CppUnit and Vim

tottinger — Thu, 03 May 2007 21:54:00 -0500

I’m playing with ways to make CPPUNIT and VIM a bit more agile for my C++ teams.

I’ve been hearing about great lengths people go to, flipping back and forth between .h and cpp for the tests or writing perl scripts to coalesce them together. I’m too simple-minded for that. I figure nobody wants to #include my tests, so I don’t need a .h. I do it all in the .cpp file, all inline. It works fine. This way, everything is in one place. I don’t have to jump up to the header file to add a protoype and then down to the implementation to add the CPPUNIT_TEST line, and then back down to my code to enter it. I have no .h, so no prototype. Inline makes it a bit more like pyunit or junit.

I grokked the CompilerOutputter pretty quick. If you use the quickfix mode (and esp if you use :cw) you will definitely want the CompilerOutputter. I like the way it helps me move from testing error to testing error (err.. if there were ever more than one), just as it does with compile errors (errr. if they happen. They do). I set up my makefiles to run the tests, so there’s no additional step required. That’s handy. I also have the makefile run ctags to make my navigation nice and easy.

Still, adding and removing tests takes some effort. I figure VIM can take up the slack. I have macros to start up a new test file (F2), to add a test class(F11), to register a function(F12), and to delete a function and its CPPUNIT_TEST line (F9) .

I also abbreviate most of the macros that I would otherwise have to type. I am thinking about remapping them maybe to more memorable values, but it helps.

There are probably a lot more elegant ways to do this. I can’t wait to hear about them.

" Add methods to help create CPPUNIT classes function! NewCppUnitFile() call NewCppUnitClass() 0r~/.cppunit_file_template.vim endfunction function! NewCppUnitClass() r~/.cppunit_class_template.vim let name = input("Name your new test class: ") exec "%s/@@@NAMEHERE@@@/" . name . "/g" endfunction nmap <F2> :call NewCppUnitFile()<CR> nmap <F9> ^f(b*ddN0d/{<CR>d% nmap <F11> G:call NewCppUnitClass()<CR> nmap <F12> ebmz"zyw?CPPUNIT_TEST_SUITE_END(<CR>OCPPUNIT_TEST(<C-R>z);<ESC>'z " CPPUnit abbreviations ab cas CPPUNIT_ASSERT ab cam CPPUNIT_ASSERT_MESSAGE ab cfa CPPUNIT_FAIL ab caf CPPUNIT_FAIL ab cae CPPUNIT_ASSERT_EQUAL ab cat CPPUNIT_ASSERT_THROW ab cde CPPUNIT_ASSERT_DOUBLES_EQUAL

The current rhythm is to open a new file and press F2. F2 calls NewCppUnitFile, which sucks in my file header template. It’s pretty bare, but it can be edited to taste. Mostly it helps me remember what to include.

#include <cppunit/TestFixture.h> #include <cppunit/extensions/HelperMacros.h>

Then I give the name of my first test class. I get a starter class. I can then add it to the makefile (note to self: automate the makefile bit). the starter template I have has a dummy test, so I can run it and see if it fails. Failing means it’s being built and run.

class @@@NAMEHERE@@@: public CPPUNIT_NS::TestFixture { CPPUNIT_TEST_SUITE(@@@NAMEHERE@@@); CPPUNIT_TEST(shouldFailAsEvidenceThisMethodIsRegistered); // DELETEME? CPPUNIT_TEST_SUITE_END(); public: void setUp() { } void tearDown() { } protected: void shouldFailAsEvidenceThisMethodIsRegistered() { // DELETEME? CPPUNIT_FAIL("This function is registered and can be deleted"); } }; CPPUNIT_TEST_SUITE_REGISTRATION(@@@NAMEHERE@@@);

Substitution is automagical. the @@NAMEHERE@@ is the magic pattern being replaced with test names.

A nice thing about macros is that they’re atomic. If I hate the name I gave my test, I press ‘u’ for undo, and it vanishes. Back to empty file. :-)

If the ”:mak” works and I get the inevitable failure in the “shouldFail” method, I put my cursor on the “void shouldFail…” line and press F9. In the wink of an eye, the method and its CPPUNIT_TEST line both vanish. Again, macros are atomic, so if I delete the wrong thing, I just press ‘u’ for undo and both the function and its registration reappear.

No flipping between files, no scrolling up and down. We have software to do that now.

I add a method under “protected”. I put the mouse on the method name and press F12 to add the CPPUNIT_TEST for it above. I add some more text. When I get to the part where I need to assert, I use one of the abbreviations. If I’m in insert mode and type “cae(” I get “CPPUNIT_ASSERT_EQUALS(”. I don’t have to remember whether it ends in an S or not. I actually struggle with that, sad thing that I am.

I can press F11 to create more test classes. I like to keep a test class for every circumstance (setup) I am using. I don’t mind at all having multiple tests per .cpp file. I like to group the tests together in files. After all, quickfix knows where the code is. This is actually one of the reasons I like to keep using CppUnit. I’ll have to tell the others someday.

Well, that’s how I do it. You don’t have to, but you could try it. If you find any errors, or glaring omissions, please mail them to me or comment here. In return, I will claim my actual scripts are perfect and blame it on the web master. ;-)

What Cant Who Do?

tottinger — Tue, 01 May 2007 21:55:00 -0500

I have a fresh idea. When someone is supposed to change their workstyle to do TDD, and they say “we can’t do that”, how about we assume they’re telling the truth.

The first word is “we”. It may well be that the speaker and the people he represents might not be able to do the work. They’re not saying that it can’t be done. They’re only saying that they can’t do it.

The next word is “can’t” and that’s an interesting word, too. It might mean that they lack the ability or knowledge. That’s ideal, because such things can be taught and can be led. The other sense is that they are not allowed. If it’s like that, then there is going to be some political work going on. But at least you know what you’re getting into.

The other tell is the word “that”. Maybe they should describe what it is that they can’t do, and you can tell if the thing they can’t do is the same thing that you are wanting them to do. Sometimes it is not.

Coercive Immediacy

tottinger — Tue, 01 May 2007 20:13:00 -0500

I have been thinking a lot about the “short reach” idea, and tryng to reach beyond it to capture the thing that the good agile project teams have and several others are missing. At the same time, I’ve been reading up on some agile tools (including one by a guy I know) in my spare time. In addition, I am on a mailing list for multisite agile development. Finally, I have been pondering how to work this agile stuff when the developers cannot be co-located due to shortage of office space or shortage of common areas.

I realized that the whole “short reach” thing creates a kind of coercive immediacy.

The corkboard with all the story cards in their four columns has that kind of immediacy. I can’t walk past it without assessing how far we are in the current iteration. I can certainly ignore a web page, or a display in some application on my computer. I can’t walk into the common area past that board without thinking.

Likewise, I could ignore an app that holds story status and backlog, but the cork board is where I go to show that I’ve completed a task and to get the next task. If my stories were assigned to me (rather than allowing self-organization) then I could neglect the corkboard too. I would not have to look at it or think about it. I could hold “my” story card at “my” desk until I decided I was done. When we are self-organizing, we are always thinking about who could help us, when we can get done, and what is the best way to truly complete this story and move on.

A partner has coercive immediacy. When you are programming in pairs, you don’t answer the phone unless you have to. IM and mail can wait. Interruptions are less welcome. Having two people on the story pushes one to focus. Also, the pair partner is always there to catch the sloppy shortcut, the poor naming, the ill-considered algorithm.

The customer team member also participates, and both feeds and feels the coercive immediacy of the team. Rather than make a guess, the programmer feels the nearness of the customer, and it’s much easier to ask than guess.

I think that TDD shows this most of all. I can’t choose to ignore the red bar. I write a line of code or two, and now my tests pass. The immediacy would be gone with it, except that I know I have some cheesy, minimal-effort code and I can’t leave it that way. I need a better test. When a test fails in some area where I’m not working, I feel the immediacy.

With distribution, multi-site, tools, and the rest I fear that we lose the coercive immediacy of Agile development. If we don’t have a partner in our way/face, then we lose something. If the status board is a tool somewhere, even a really nice tool, but I have to remember to ask for it to see it, then it is not in my way. It doesn’t inspire me.

My thought now is that an agile management tool won’t help unless it is something we have to live with all the time, during pairing, as part of the test/code/refactor cycle. It has to be in our faces, over our heads, and not just tickling the edge of our awareness like a more polite version of “clippy”.

Short Reach

tottinger — Mon, 23 Apr 2007 15:33:00 -0500

I’m always trying to find newer, better, shorter, more powerful ways to explain what Agile is about. I suppose I’m some kind of obsessive about expressive power and economy.

Finally I decided that Agile, as I understand it today, is about the short reach.

It seems to me that all of the agile practices are about shortening our reach, the distance in time-and-space that one leaves an assumption, decision, or line of code untested and unconfirmed. All the practices seem to follow this one rule.

The customer/analyst is keep in the same room, in the same short reach.

We feed back the iterations to the customer/analyst so that his every decision has a shorter reach.

We do iterations to ensure our planning has short reach.

We keep our teammates very close, in the same room, so that it’s a shorter reach to them.

The test is written first, so that implementation has shorter feedback on correctness.

We compile/test frequently because our code time should have a short reach.

We pair so that our code is instantly vetted through a peer. We don’t pile it up and review it after the tests pass.

Our planning is based on “yesterday’s weather”, data collected a very short time ago.

We don’t plan the team structure and the assignments, we self-organize so that tasks are waiting for the shortest time possible.

We talk face-to-face, not across chat and email and official company documents.

Tell-dont-ask and the Law of Demeter guide us in keeping the reach of our objects very short.

We use unit tests to exercise a class directly, and we isolate with mocks to reduce the reach of our tests through the system.

Shared code ownership means that the guy sitting behind your keyboard has all the permission he needs to do excellent work, even if it impacts existing design.

Test-first development means that the guy who makes a change knows very quickly whether his change is safe or not. He doesn’t have to wait until the week before integration when the “real” tests are run.

Where does Agile run into logistical or operational difficulties? Wherever a long reach is required or imposed. Where an organization chooses to continue in waterfall-style management, where the team is distributed among managers with appointed “point of contact” and “official channels”, and where the developers are not placed in a common work area agility is very difficult.

I’m not saying that agile techniques can’t work for large companies, but that is an area where a lot of experts are trying (maybe succeeding) to extend the agile techniques and where the average “agilist” finds challenges. When it works, it is almost certain it will be because someone has found a way to shorten the reach of the teams so that all they need to know is never more than a few seconds or minutes away.

At least that’s my half-baked observation of the day. Let me know if I’m wrong here. Or if I’m more right than I think I am.

Using VIM

tottinger — Fri, 20 Apr 2007 01:19:00 -0500

It isn’t done yet, but I figured I would post my work in progress in the meantime for feedback. It’s a little article about the proper use of the VIM editor. I tried to mix in some keystroke quick-reference with the philosophy of the tool in a way that might make it a bit more accessible.

When it’s done (Ha!) I will probably find new places to store it, but for now it’s at my personal blog.

Don’t be afraid to offer corrective assistance, or to recommend sections.

The Two Things Agile

tottinger — Mon, 16 Apr 2007 22:07:00 -0500

For Agile, maybe the Two Things are:

We can do better/faster this week.

And this way next week will be better yet.

Code is a Liability

tottinger — Mon, 16 Apr 2007 21:10:00 -0500

I have thought for a long time that code is liability, and that viewing code as a liability has the power to transform our art. I read articles about code-as-a-corporate-asset and most of them are pretty good articles about wringing the value from the code you have, but I still feel that there’s something essentially wrong about it.

Programs are a great means to an end. Don’t get me wrong, I’m in favor of having software and I’m greatly in favor of writing software. I think we would be in pretty poor shape without it. But I can’t call code an asset.

Functionality is clearly an asset. When we can inter-operate with various services (possibly even web services) our businesses produce more business results more quickly. We use software to control machines and to increase public safety in more ways than most of us know (talk to the guys who write the software that controls your car, the over-the-road trucks, forklifts, industrial robots, and aircraft). Software makes our life better and it’s a wonderful thing. I’m all in favor of software functionality and you have to write code to get it.

Our bosses and clients will pay good money to get the functionality they want, and they want it right now! If we could give them what they want without writing a line, it would be a tremendous win. If we could do it with one line or two lines of well-considered code, we would be heroes! Why is doing less so valuable if code is an asset? Clearly less code is better.

Sadly, most companies have to deal with heaping, shaggy mounds of code. Code takes up time and space. It has to be managed. It has to be versioned. It hast to be tracked, and planned. It has to be updated, and packaged, and revised. It needs backup to save us from having to reproduce it by hand. It has to be reviewed (hopefully in an efficient way like pairing). It often drives companies to expand staff and dedicate people to manage it (version control administrators, managers, build czars, consultants, contractors, metric-gathering tool specialists, etc).

Old code gets in the way of new code. Having more code will typically slow development, and will certainly reduce your ability to incorporate new programmers. Of course you’ll need more programmers because you have all this code to deal with. Size has a cost.

The problem doesn’t go away if you artificially reduce the code. Folding a lot of effects into few lines of code makes the code worse. Adding voluminous documentation makes the code worse. Moving it into metadata and models and other forms doesn’t make it any smaller, and often makes it worse. Hand-crafted code is almost always more readable, smaller, more optimal, more focused, more literary in its style than generated code or funky data tables. Since there has to be code, it might as well be the best code we can write. Coding well takes human beings who value minimalism.

It seems that the trick with the functionality/code game is in trying to maintain a positive balance.

We reach a point on some projects where we are producing a positive number of functions with a net loss in lines of code. That’s a wonderful time. We reduce the line count by simplifying the code and eliminating duplication. Sometimes we refactor our code into a more compact form and then only add a very few lines of code to produce a change. If code were an asset, then those moments would be failures instead of triumphs.

Code size reduction is a very good thing. I want the simplest, the smallest, the least code possible, with the least risk of breaking something else in the system. I don’t want big functions that touch everything in sight. I don’t want long complicated blocks of if/else statements. I don’t want monster functions which conglomerate dozens of operations in a single, fat interface. Shallow is good. Short is good. Less code is good. More code is a liability. This isn’t about typing less, it’s about owning less.

This is the point of view that makes test-first (TDD) so important. TDD/BDD has us encode the functionality (the asset) first, and then write minimal code to realize the specified feature. If code is a liability, and function is an asset, this is exactly the right way to do things.

And maybe, just maybe, it might explain why one might open-source less-important useful systems.

It's a Poor Coach Who Slaps The Kids

tottinger — Mon, 16 Apr 2007 21:01:00 -0500

Today I was remembering a bit of advice I received back in the late 80s when I started trying to work with people more. I’d been a programmer for a while, and mainly survived by reading a lot and practicing what I read. I taught myself most of all the programming I knew (I did have some good mentoring peers). I wasn’t much of a people person.

A fellow named Kevin worked on a project with me in the DC area. One day he had the kindness to take me aside. He explained to me how to work with people. He taught me that when someone had an error in their understanding I shouldn’t tell them they were wrong, but rather tell them they were right except for the part where there was an error. He helped me more than he knows. We only had the one project together, and he went on to a different employer and I went on to become me.

He told me that a good coach doesn’t tear down his kids. He doesn’t tell they they stand like a sissy or throw like a girl. Instead, a good coach will let them know what’s right and what’s valuable about what they’re doing, and then explain what to add to their technique to get better results. And he doesn’t tell them everything all at once.

Now I’m a coach. I teach development methods, including agile methods, including TDD. I now work with people more than technology. I also work with a lot of other coaches from time to time. The best ones seem to intuitively understand when to criticize and how, and when to hold their tongues.

In the warm rush of enthusiasm and passion for the art of development, we coaches have to continually remind ourselves that it’s the poor coach indeed who slaps the kids around. We have to instead meet them where they are and look for ways to help them get the results they want.

Kevin’s lesson continues to teach.

Single Ownership in C++ (intro to auto_ptr)

tottinger — Fri, 30 Mar 2007 10:48:00 -0500

I find a lot of C++ programmers experience memory management differently than I do. I used to have the same kinds of problems that they have, but I learned an important rule: every heap object has exactly one owner, and n borrowers.

The way that I realize this is using reference types. Of course, all good C++ programmers write their functions to use the canonical “const T&” parameters whenever possible. Using a reference means that we have an actual object and not a null reference (side effect: no null checks) and we have simple dot notation (side effect: less typing), but I also use this to mean that we are borrowing the object, not owning it. Likewise, if we have to drop the ‘const’, we are still borrowing.
What about passing by pointer? I only do that if I absolutely must accept 0 (null) as a possible value. I always hate to do that. I would rather have an object outright than a pointer that I dereference. If the function is being called by a framework, I don’t get a choice. I might do something like this:
void functionName(const Snorkle* snorkleWithNull) { if (0 == snorkleWithNull) { throw SomeIdiotPassedMeANull(); const Snorkle& snorkle = snorkleWithNull; // ... work with snorkle, ignore snorkleWithNull ... }
However sick that seems (and it does) it reverts back to the const reference case quickly and I’m left with a reference to a real object. It keeps me from forgetting to dereference when I’m in a hurry, and keeps me from casting. I have to work extra-hard to do bad things, and that’s a good thing. On the other hand, in modern TDD style, with refactoring, it can double the length of a function (hence feeling sick).
Either way, if you are a borrower, you have no right to delete the object. You can reference it for the duration of your function, or (if the intent is that you will have a short-lived object) for the duration of the object lifetime, but no ownership is given by pointer or reference. Remember that we still have a risk of the pointer being dangled, so holding a reference/pointer for longer than a method call should be considered fairly dangerous and must be done rather intentionally.

But what about transfer of ownership? That’s what the auto_ptr in <memory> is for. Yes, it’s a template. If I can’t use templates, I’ll make up something new that acts roughly the same. The auto_ptr is the one structure in C++ that undeniably says “owner”. When I receive an object via auto_ptr, I am now the owner. When I return an auto_ptr from a function, I’ve clearly given up ownership.

The recipient of the auto_ptr has to do something with the referenced object, or else it will be deleted when the auto_ptr passes out of scope. If the recipient decided to release the auto_ptr, they can keep a normal pointer or reference in whatever structure pleases them. They can make a copy and delete the original (remember the warning about dangling).

The trick is to remember that 99% of the time, you want to be a borrower, and that any class that is going to hold a borrower’s reference to an object needs an algorithmic guarantee that the referenced object will outlive the reference.

The scary part is for an object to have a large number of long-term borrowers, because there is the constant fear that one might outlive the owner. In that case, you need to hang the auto_ptr on the shelf and go get some bigger magic. You might actually need the smart pointer. I have created reference-counting pointer templates in several projects, which will cause C++ to act more like a garbage-collecting language. I know that there are 3rd-party libraries for garbage-collecting. If I find myself in a situation where there are a number of long-term borrowers, I will definitely move in one of these directions again. It is too important to leave object ownership up to chance or naming convention.

Solution Probleming

tottinger — Fri, 23 Mar 2007 11:07:00 -0500

I’ve been fortunate enough to have spent most of the past three weeks teaching advanced Object-Oriented Design with patterns. The students have been wonderful and helpful and brilliant, so the classes have gone very well.

As I introduced students to Visitors, Adaptors, Chain of Responsibility, and mult-level State patterns, I was reminded of the problem that was rampant in the 90s: Having learned a catalog of patterns, many OO designers could not wait to put them into practice. Rather than using patterns to solve problems, they began to look for ways to apply these pattern solutions.

I call it “solution probleming”, which is of course the opposite of “problem solving.”

It wasn’t just the patterns community, it was software tools and it was frameworks and languages. A company I know used to say “I don’t know what the problem is, but the solution is DB2, CICS, and COBOL.” That’s solution probleming. It’s one of the reason that so many of us strive to not be product-based consultants. We want to solve problems, not just shoehorn prefab solutions into people’s applications.

I think any well-developed software developer’s immune system should be wary of the smell of solution probleming. It’s one of the best ways I know to overcomplicate and overengineer a solution - which is one of the worst ways to develop software.

Many

tottinger — Fri, 23 Mar 2007 11:01:00 -0500

We’ve talked about Zero, and One, so all that’s left of the big Three Numbers is “many”.

Many is more than one. More than one object that must be used in the same way, more than one implementation of an algorithm, more than one implementation of an interface. An API wrapper implemented once for production, and once for testing is Many. One real object and one mock is Many.

Many gives power to the Liskov Substitution Principle. If there are many (more than one) objects to be used in the same way, then it is imperative that an interface is created for them so that they may be used interchangeably (transparently) by the client.

Many objects of the same interface (“type”) may be used one-at-a-time (by reference) or in groups (containers and loops). We don’t write code that calls A and B and C in the same way because that would be “three” and we don’t believe in 3. We believe in Many. It’s a loop over a list. If we’re treating the objects similarly, we should exalt the similarity by putting them in a loop, rather than stress individuality by coding two statements for making the call.

By the way, don’t confuse Many with “two things with similar structure”. Many refers to “more than one thing with similar usage” in the LSP-compliant sense. It doesn’t demand that you create base classes to capture some essential similarities, only that you create interfaces and loops as required.

In a multi-generational GC, the theory is that some things by nature have short lifespans, and others by nature have long lifespans. Most objects are created and deleted, and a few may make it to the third-generation cache where they’re almost never examined (comparatively) for collection. Many has a similar theory. Objects that occur once are singular things, but once they start to multiply they are expected to multiply more in the future. Not only does the “interfaces & loops” keep the code clean, it recognizes the Many things as an extension point in the code. That’s double-benefit if we listen to our code as it goes from one to many.

Zero

tottinger — Thu, 22 Mar 2007 12:23:00 -0500

Clarity of code is a signal-to-noise ratio. We need our code to be as concisely meaningful as possible (not as cramped and crowded as possible, and not unnecessarily spread out over many pages). Whatever size provides the most accurate and quickest reading for trouble-shooting and enhancement is the best size.

On one of my past projects, we found ourselves able to remove a lot of dead code and as we added new features we were also able to simplify a lot of existing code. This is a very good thing, since code is a liability and functionality is an asset (I’ll elaborate more on that at a later date).

However, my manager was secretly reporting our progress in terms of lines of code/week. I was called on the carpet after some very productive weeks of producing unusually clean code. My manager’s problem was that we had been averaging over 200 lines less per week. By making the system cleaner and smaller, we were being reported as making negative progress, and were ordered to stop!

It’s just another “bad metrics” story, but it underscores an unspoken assumption that code should always grow bigger over time.

One non-zero-respecting practice is that of commenting-out code and leaving it. people have to scroll around the code, reading bits above and below, and so it is really just a speed bump for those who are trying to learn. It’s worse if the code is inside some nested level of a function than if an entire function is commented out. This is unnecessary, because a version control system holds the old code and can restore it if you really need it again (if not, consider a new VCS). Commented code is redundant with the VCS, in addition to being a nuisance.

Zero isn’t all really about comments, but about noise in general. Systems often have ‘dead code’, code which is never used but never deleted. When there are insufficient tests, one is unsure of coverage and cannot tell if a piece of code is being used or not. That means that code with zero use may have non-zero occurrences. In such systems, it can be very dangerous to remove code at all. Since the rule of zero is not respected, code accumulates.

Zero also will have us remove abstractions and interfaces when we no longer need them.

Zero tells us to remove tests when they’re no longer valuable.

Zero is a good idea.

I just located an old Jeff Atwood quote that lights up a few of the same synapses.

One

tottinger — Wed, 21 Mar 2007 09:17:00 -0500

I posted earlier about the three numbers of software design (zero, one, many). Zero and Many are numbers we see pretty often, but the rise of TDD has made “one” just a little bit lonelier. I wanted to say that one is the loneliest number, but David Chelimsky beat me to it several months ago, and I would just feel like a copycat.

When we’re test-driving an application into existence, we tend to create mock objects. The presence of one mock object in addition to one real object places is firmly in the design space of “many”, and so interfaces become the rule of the day.

In addition, we have dependency management to deal with. In order to reduce the D metric on a package, we find ourselves often breaking a class into an interface (increasing A) and an implementation that depends upon it (increasing I) in a separate package.

Of course, the “other question” is volatility. If we have one implementation of something small and simple in a domain that is very stable then we don’t need any interface and “one” takes center stage. This is true for whole value objects (fowler) in general. If the class is more complex, or dependent on special hardware or changing algorithms, then we end up creating a mock (driving us back to “many”).

But the number one is becoming more and more rare in our design world.

The Three Numbers

tottinger — Tue, 20 Mar 2007 11:27:00 -0500

There are only three numbers that are truly meaningful in software design. The numbers are the same as the numbers in relational database design: zero, one, and many.

Zero: if a condition, circumstance, requirement does not exist, then no code should exist (commented or uncommented) to deal with it. This is true for requirements we haven’t seen (future proofing), but is also true for requirements which have some and gone. When a test, function, variable, or class has zero uses, it must be deleted. Recognizing the power of zero helps us keep our code clean.

One: A condition or class, event or circumstance, role or behavior exists, and so we must code for it. There is a limit on how much insulation and abstraction is needed. We can keep things very simple. Moving from zero to one (adding a requirement) is the typical operation, but moving from one to zero also happens.

Many: More than one is “many”. You might as well consider adding a list and/or interfaces. In design, some things repeat and some do not. Those that repeat are likely to continue repeating. Therefore, many is represented first by the value of two. Where there are many, there needs to be some insulation. The open-closed principle tells us that we should open the system for extension. The usual case in design is moving from one to many, but sometimes one will move in the opposite direction by removing interfaces or lists.

The transitions in either direction are important design events.

It's All Iterative

tottinger — Fri, 16 Mar 2007 09:38:55 -0500

People say “we don’t do iterative”. Sometimes it is a prideful thing, more often a kind of self-effacing confession. However, you and I know that it’s not true. All software development is iterative, it’s a matter of granularity.

When working on the very first 1.0 release of a product, developers and managers may think that they’re not doing iterative work, but any idiot knows that they’re not writing all the code at once. They couldn’t possibly. They’re not releasing yet, but they’re appending functionality every day.

They try to design all at once, but that’s just for 1.0. Software is like a plan: it never survives contact with the real world unscathed.

After 1.0 do we cancel the project? If so, it’s a failure. After 1.0 is 1.0.1 or 1.1 or 2.0. The project continues and continues to receive new appendages and improvements. Isn’t the whole idea of “next release” an iteration?

Since you’re really iterating anyway, why not capture the data that iterating produces, and why not take advantage of the integration and testing opportunities that iterating will give us? And why not have useful feedback loops? And why not measure our progress as we go? And for that matter, why not go for smaller measured periods so we can judge trends more reliably?

When we say “it isn’t an incremental and iterative effort” we are really saying that we’re denying a basic truth, and choosing not to use the data it produces for us. Not intentionally, but almost certainly.

Outliving The Great Variable Shortage

tottinger — Mon, 26 Feb 2007 23:07:00 -0600

One of the more annoying problems in code, confounding readability and maintainability, frustrating test-writing, is that of the multidomain variable.

I suppose somebody forgot to clue me in to the Great Variable Shortage that is coming. I have seen people recycling variables to mean different things at different times in the program or different states of the containing object. I’ve witnessed magic negative values in variables that normally would contain a count (sometimes as indicators that there is no count, much as a NULL/nil/None would). I might be willing to tolerate this in programming languages where a null value is not present.

Yet I have seen some code spring from multidomain variables that made the code rather less than obvious. I think that it can be a much worse problem than tuple madness.

I have a rule that I stand by in OO and in database design, and that is that a variable should have a single, reasonable domain. It is either a flag or a counter. It is either an indicator or a measurement. It is never both, depending on the state of something else.

It is sort of like applying Curly’s law (Single Responsibility Principle) to variables. A variable should mean one thing, and one thing only. It should not mean one thing in one circumstance, and carry a different value from a different domain some other time. It should not mean two things at once. It must not be both a floor polish and a dessert topping. It should mean One Thing, and should mean it all of the time.

Surely I’ll take a shot from someone citing the value of a reduced footprint, and I won’t argue very long about the value of usinge only as much memory as you must. In C++ I was a happy advocate of bitfields. I haven’t been overly vocal about using various data-packing schemes, but I think that it can be a reasonable choice for compressing many values into a smaller space, but I will maintain that multipurpose variables are a bad idea, and will damage the readability of any body of code.

I suggest, in the case of constrained footprint where there truly is a Great Variable Shortage (GVS) that if (and that’s a big if) the author absolutely MUST repurpose variables on the fly that it is the lot of that programmer to make sure that users of the class/struct never have to know that it is being done. Never. Including those writing unit tests. The class will have to keep data-packing and variable-repurposing as a dirty secret.

Perhaps a strong statement or two in rule-form should be made here:

A variable should have a SINGLE DOMAIN.

One must NEVER GET CAUGHT repurposing a variable.

We don’t have to make do with the smallest number of variable names possible. Try to learn to live in plenty: use all the variables you want. For the sake of readability, consider having a single purpose for every variable not only at a given point in time, but for the entire program you’re writing.

PySweeper: Un-refactoring, Tuple Madness, and Injection

tottinger — Sat, 17 Feb 2007 13:02:00 -0600

I figure that to understand the code before me, I need to start testing. I’ll start with the constructor. The constructor actually is doing a lot of work. In order to test out my refactoring tools, I recklessly broke out two routines without adding any testing. I don’t recommend that on any project that needs to work, but I also don’t recommend using a real product to learn your tools.

Today, the constructor looks something like this:

def __init__( self, rows = 16, cols = 16, mines = 10): # 40 ): """Initialize the playing field. This function creates a playing field of the given size, and randomly places mines within it. rows and cols are the numbers of rows and columns of the playing field, respectively. mines is the number of mines to be placed within the field. """ self._validate_parameters( rows, cols, mines ) self.rows = rows self.cols = cols self.mines = mines self.cleared = 0 self.flags = 0 self.start_time = None minelist = [] self.freecoords = {} for col in range( cols ): self.freecoords[col] = range( rows ) self._place_mines( mines, minelist ) self.board = [] for col in range( cols ): self.board.append( [( -2, 0 )] * rows ) for row in range( rows ): if ( row, col ) in minelist: self.board[col][row] = ( -1, 0 ) def _validate_parameters( self, rows, cols, mines ): for var in ( rows, cols, mines ): if var < 0: raise ValueError, "all arguments must be > 0" if mines >= ( rows * cols ): raise ValueError, "mines must be < (rows * cols)" def _place_mines( self, mines, minelist ): while mines > 0: y = random.choice( self.freecoords.keys() ) x = random.randrange( len( self.freecoords[y] ) ) minelist.append( ( self.freecoords[y][x], y ) ) del self.freecoords[y][x] if not self.freecoords[y]: del self.freecoords[y] mines = mines - 1

I feel a little guilty for breaking out functions without testing them. I start to write tests for _validate_parameters, but I cannot call it in isolation. I have to have an instance of Field to call it, and it is also called in Field’s constructor. I can’t test weird values as easily as I’d like. I need something like a static method (c++/java). I add the @classmethod decorator and I can call it freely from my unit tests.

I remind the reader that the game runs and is playable, so the addition of tests should pretty much be a bit of pedantic formalism in preparation for refactoring. However, look at the code carefully and see if you can see a loophole:

@classmethod def _validate_parameters( self, rows, cols, mines ): for var in ( rows, cols, mines ): if var < 0: raise ValueError, "all arguments must be > 0 if mines >= ( rows * cols ): raise ValueError, "mines must be < (rows * cols)"

A perceptive reader will realize that the code is checking that the value is less than zero, but reporting that it must be greater than zero. I found this with tests before it jumped out at me visually.

I write and run several tests and then get a nag from PyDev. I guess somehow I missed the part where PyDev told me it was not free software. The nag messages tells me I’m in a trial period and I’m expected to eventually buy or quit. Bummer. PyDev is not expensive. I just have to decide if I want to afford it.

I am also playing with Eric3 which is a python-specific IDE with refactoring, and it seems to have better integration with PyUnit. It is not as nice in terms of automatic completion and seems to have trouble with the context menu for refactoring. I immediately miss the cool way that PyDev would stick the word “self” in so I didn’t have to type it.

In the last installment, I extracted a method called _place_mines but it’s a bit hard to test. It’s actually a very poor refactoring, done to test a tool and not really to improve the code. As is, it wants to muck about with an unspoken parameter.

def _place_mines( self, mines, minelist ): while mines > 0: y = random.choice( self.freecoords.keys() ) x = random.randrange( len( self.freecoords[y] ) ) minelist.append( ( self.freecoords[y][x], y ) ) del self.freecoords[y][x] if not self.freecoords[y]: del self.freecoords[y] mines = mines - 1

It’s small, but not obvious. Nor is it tested. It is using freecoords rather heavily. Breaking out that method was a mistake, and I put it back inline. It seems that freecords is a list of all the positions in the grid that do not contain mines. It is only used in the constructor and one other place. In the other place, it’s used and then deleted from the object.

The names of variables mines and minelist aren’t helpful because the names mean the same thing, even though the variables do not. They each mean more than one mine, but one is an integer count and the other is a list of the coordinates of the mines. That should change right away. I rename the integer to numberOfMines.

With the __init__ reconstituted, I notice that I don’t even keep the minelist. It’s discarded at the end of the __init__method. Maybe I can do this more simply.

I realize now that the code suffers from Tuple Madness. The cells are represented by two-tuples. The first and second item are not distinguished by a name, only by subscripts 0 and 1. There can hardly be anything less descriptive of the content. The initial values are (-2,0), but sometimes they are replaced with (-1,0). The code gives no obvious guidance to the significance of the first parameter versus the second parameter, and no real guide to the meaning of the integer-coded values. If one were going to document this code, it would be helpful to explain the use of anonymous data structures. Of course, that’s selfish because my goal is to obviate any comments I find. I just want it to be easy.

I dig in and discover that subscript zero really is the count of adjacent cells with mines. Since you can’t have less than zero mines, the number -2 tells us that we’ve not counted yet. The number -1 is a code telling us that the cell itself contains a mine. I’ll want to split these out later into named attributes.

Tuple Madness is such an easy trap to fall into with Python or Ruby (and I’m told Lisp as well). Each has such great native tuple and list types that we may forget to create real data structures for small data sets. I’ve done it, the PySweeper author fell into it, and so have my coworkers. If we had n-tuples in C, the resulting code would have been so inscrutible that the software practice might never have recovered. Yet two-tuples can be very good things. Just not here. I vow to create a data structure for the cell as soon as I have this body of code sufficiently under test.

As I learn about the two-tuples I write a couple of tests, as if there were more descriptive functions. I write calls to such functions as hasAMine() to check if subscript zero has a -1 value, and cellAt to get a cell at a given location, etc. Explanatory functions are a powerful documentation technique
def cellAt(self, x, y): return self.board[x][y] def isOpened( self, x, y ): return self.cellAt(x, y)[1] == -1 def countAt(self, x,y): result = self.cellAt(x,y)[0] if result < 0: result = None # indicator value return result def hasMine(self, x,y): return self.cellAt(x,y)[0] == -1
I have a small-but-growing body of tests. I should soon have enough confidence to do bigger refactorings.

Note the ugly comment in countAt. This is variable reuse. Rather than have a separate indicator (another element in the tuple maybe), one element is serving double-duty. It is too entrenched right now, so I’ll work on making it obvious and easier to separate later. Soon my body of explanatory methods will compose an opaque interface, and there will be little or no upset when I change the implementation of those methods.

Note also that we’re in violation of the SRP. The class has methods related to the field and also to specific cells in the board. Maybe I will eventually have classes for the cells, the game, and the board.

Time to stop reading and do more testing. I did learn enough to know that the game tries to be “fair”. If there is a mine on the very first square you select, it moves it out of the way. My “gaming the system” gene took over. I should be able to fill a three-by-three grid with eight mines, and then click (open) the center square. That should guarantee me that I have this structure:

mine mine mine

mine empty mine

mine mine mine

The more astute reader knows that this disregards the first rule of testing: Control The Environment. I should take control of the placement of mines rather than testing the system with random mine placement.

Sure enough, I find that the routine that moves the mine to a new spot is not working quite as expected. Now I have a failing test which allows me to make changes to the randomizer. I fix it, and then decide to pull the random chooser into a separate class. It will behave according to the iterator protocol in Python so I can pass a simple array of (col,row] pairs to the constructor as a testing stub/mock/thing.

Now the top of my file has the random cell chooser. There are some named constants, the still-busy __init__ method, the now-tested _validate_parameters method, and then other code we’ve either talked about already or haven’t touched yet.

random.seed() class random_cell_chooser(object): def __init__(self, columns, rows): self.choices = [ (x,y) for x in range(columns) for y in range(rows) ] def next(self): if len(self.choices) == 0: raise StopIteration, "Out of free cells" item = random.choice(self.choices) self.choices.remove(item) return item INITIALLY_EMPTY = ( -2, 0 ) INITIALLY_MINED = ( -1, 0 ) class Field: """Provide a playing field for a Minesweeper game. This class internally represents a Minesweeper playing field, and provides all functions necessary for the basic manipulations used in the game. """ def __init__( self, rows = 16, columns = 16, numberOfMines = 40, chooser=None): """Initialize the playing field. """ self._validate_parameters( rows, columns, numberOfMines ) self.rows = rows self.cols = columns self.mines = numberOfMines self.selector = chooser or random_cell_chooser(columns,rows) self.cleared = 0 self.flags = 0 self.start_time = None self.board = [] for col in range( columns ): self.board.append( [INITIALLY_EMPTY] * rows ) for counter in range(numberOfMines): (col,row) = self.selector.next() self.board[col][row] = INITIALLY_MINED @classmethod def _validate_parameters( self, rows, cols, mines ): for var in ( rows, cols, mines ): if var < 0: raise ValueError, "all arguments must be > 0" if mines >= ( rows * cols ): raise ValueError, "mines must be < (rows * cols)"

An engineer friend of mine is fond of saying “suspenders and belt”, referring to having redundant safety mechanisms. I’m appreciating the value of both a good version control system and a growing set of tests. I am aware of the quality of the code I’m modifying (whether it passes all the tests) and that I can always abandon a change I’m making if it gets messy.

My little test is building that three-by-three grid and playing a very simple game to completion by opening the middle square. I am testing setup and a bit of gameplay. I’ll put a better test in tomorrow.

It is a simple thing to extract the method for placing the mines and pass it to the constructor for Field. Now I can set the mines just where I want them, giving me total control of the field constructor. I make it act like an iterator so my tests can pass a list iterator to the constructor:
class random_cell_chooser(object): def __init__(self, columns, rows): self.choices = [ (x,y) for x in range(columns) for y in range(rows) ] def next(self): if len(self.choices) == 0: raise StopIteration, "Out of free cells" item = random.choice(self.choices) self.choices.remove(item) return item

Because the tests are new and few, I’m also playing (losing) a game of minesweeper every so often so that I know that it still beats me in reasonable ways. I can’t wait until I can depend entirely on automated tests. Maybe in this way, it would have been better to have some basic acceptance tests before working in unit tests. It’s a thought.

PySweeper: Python test collector

tottinger — Sat, 17 Feb 2007 12:00:00 -0600

I wrote a number of separate test files for the pysweeper refactoring exercise, and very quickly got tired of running them all individually. I created a stupid test composite tool and I run this one instead.

Here’s what it is looks like right now.

import os import unittest import sys def collect_tests(directoryName): """Gather all tests matching "test*.py" in the current directory""" alltests = [] for filename in os.listdir(directoryName): if filename.startswith('test') and filename.endswith('.py'): module = __import__(filename.replace(".py","")) for item_name in dir(module): item = getattr(module,item_name) if isinstance(item,type) and issubclass(item,unittest.TestCase): test = unittest.makeSuite(item) alltests.append(test) return alltests def suite(): "Collect all local tests into a monster suite" suite = unittest.TestSuite() here = os.path.dirname(sys.argv[0]) or os.getcwd() alltests = collect_tests(here) suite.addTests(alltests) return suite if __name__ == "__main__": tester = unittest.TextTestRunner() tester.run(suite())

Thre may be easier ways or better ways, but right now I don’t know them. I figure it will be good enough until I learn something cooler. I can think of several ways to extend it, but don’t need them.

Seed-And-Harvest Considered Harmful

tottinger — Thu, 15 Feb 2007 14:28:00 -0600

Quite a while back I complained that a class is not a BASIC program, and more recently I blogged on my personal site about The Unspoken Parameter, but on further consideration I realize that these are both aspects of larger, well-known problem that vexes developers.

The real problem is the idea of seeding a function’s input in various places and then calling the program, and then harvesting the function’s results from the surrounding environment. This indirect access to input and output is troublesome and frustrates maintenance.

Seed-and-harvest programming was common in BASIC “gosub” programming, and I see it in some command line programs too. I see it in classes where methods take no parameters and have no return values and you must seed and harvest the object’s fields to make a successful test. I see it happening in poorly extracted test methods. I see it in java programs where you have to have a host of environment variables and files and command line parameters all set up to be harvested by the program.

Seed-and-harvest is daft. Every program is composed of input, process, output. We try to simplify and focus processing, but still I find code that does not take the same degree of care with input and output.

Since seed-and-harvest are bad, the degree to which the input and output are formal and direct is a goodness. It isn’t the only measure of goodness, but it is a goodness.

If I can call a function, passing everything it needs to know, and it returns a value indicating everything it has done for me, then I have an optimally direct and formal interface. Barring foolish and meaningless naming (or personal inability), I should be able to understand what it does fairly easily and without error. It is also far more likely to work in multithreaded environments, by the way.

If I must set environment variables or manipulate global data stores or even local data structures in order to call a function, then I have diluted the interface by relying on input seeding.

After calling a function, if I must to collect its output from the environment, global data structures, or local data structures, or various fields of the function’s object/class/module then the interface is diluted by harvesting.

An example of good here is a sensor object that emits a reading. The method to get a reading is obvious, clean, and likely to be thread-safe. A less good solution would leave the reading data in data fields of the sensor to be collected by a series of accessor calls. It defocuses the class re: SRP, while creating risk of update anomalies. It requires more effort from its clients than if it returned a reading.

This principle leads to other well-known principles. Dependency injection is a solution to a problem of indirectness. The Law of Demeter makes sense in the sense that it reduces the degree a method must reach to collect its input and to place its output.

In OO programs, static functions often have the most direct and formal interfaces since they can’t rely on instance variables. We know that static functions have problems with regard to substitutability in TDD, so we may make them non-static functions by moving them to a class or passing them by reference (closures and blocks). But we can see how much easier it is to read static functions than members, since we seldom have to read their code to understand what they are doing (though we may open them up to see how).

Objects with fields will exhibit some seed-and-harvest behavior to manage state and change of state, but we can preserve an otherwise direct formal interface with reference to callers of the object’s methods. We should not have to call a function to cause an effect and then call a number of accessors to get the results, as this would indicate we’ve diluted the interface with harvesting. We need to understand and bear in mind that we’re trading off some directness for convenience and encapsulation. If we know that we’re making that tradeoff then we are better equipped to do it well.

In end-user experience, this becomes more important. Plenty has been said by others about “direct manipulation”. They were wiser and earlier on the scene than myself; when it comes to UI I am still but an egg.

This concept is nothing new, even to me. I’m not saying it because I think it is something new. I’m saying it because I think people need to keep hearing it. It is simple but also profound.

Tuple Madness

tottinger — Tue, 13 Feb 2007 12:07:16 -0600

If there is a downside to modern OO languages like Python and Ruby it is the great built-in list types. They’re wonderful, but they’re tempting.

Today I had my fourth or fifth encounter with tuple madness. It seems that the siren song of “free” data structures with unnamed members is just too much to resist. I’ve decided to call it Tuple Madness. An example is: x = (2,3,-1) Could one be less descriptive? Probably not.

I came across some code which uses two-tuples where both values are encoded integer values. It’s a pain to figure out.

In a previous job, a coworker would write his python entirely as filters on n-tuples so that figuring out what his code was doing was really painful. A function would take a sequence of n-tuples, and return a sequence of n-tuples where some of the values were preserved (though reordered) from the input, and the rest were transformations on the input + some local data. It was really tough to read a program where the data structures had unnamed elements and only existed between the return of one function and the invocation of the next.

At least one of my colleagues has blogged about suffering from Tuple Madness as well.

I don’t think that “magic tuples” are always evil, only that they’re no substitute for proper objects and meaningful data structures.

PySweeper: Digging In

tottinger — Sat, 10 Feb 2007 21:44:00 -0600

In the last installment I got the source and started up the development environment.Now I start to play with the tools.

I start to peruse the source. Python programs are written a mix of paradigms, and this is something I love about it. Every variable is a first-class object, but not all methods have to be members of a class. You can have global functions (well, module-scoped) which are themselves first-class objects capable of having multiple methods and member variables, all participating in reflection/introspection. As such, some will describe the use of non-class-bound methods as “procedural”, and I suppose it’s technically a fair cop. Some of the program is written in class-free functions, mostly the bits that are used to initialize the application.

Python also has a number of functional programming features, but I suspect that the 2002 copyright predates list comprehensions and generators, so I don’t see them represented here. I will be using some of those techniques to simplify code that had to be written in loops and nested loops before. It should shorten a number of methods.

The PyGame UI and business logic (guts) of the code are separate modules with a protocol for communicating between the two. I was afraid that I would find them all intermingled, and was delighted to see separate modules. The core of the game itself is only a few hundred lines of code if I include comments, docstrings, and whitespace.

I know that the first order of business is to get some testing started, but I want so badly to to take Eclipse PyDev for a spin that I’m having an impulse control problem. I think the code is formatted reasonably, but I use the Eclipse feature to reformat the code to see if it works. I’m rewarded with equally attractive code. No damage is done!

The outlining feature works fine, and so does completion. I scribble a little and delete it. It’s looking good so far. I notice that the Run As.. menu has options to run Python, Jython, PyUnit, Python coverage and more. It’s looking good.

What of refactoring? Sure enough I find that I can rename, extract and inline variables, and extract and inline functions. Those are the biggies. I scroll into the constructor of the Field class. The Field represents the playing field of the game, the grid of bombs and clues. In the constructor are various paragraphs of code that help set up the class. I pick the section that adds mines, move some of the constants out of the way by extracting local variables, and then extract that portion of the function as _place_mines. In Python no functions are private, so the standard protocol is to use an underscore prefix as a ‘wart’ to tell other authors that it’s not really part of the formal interface, just a convenience function for the class. Love it or hate it, that is just how it’s done.

I do the same with the paragraph that validates the parameters (ie there are more than zero rows, columns, and mines, and that there are not more mines than cells in the grid). This becomes _validate_parameters, and is called from the constructor (in Pythonese the constructor is a function named __init__).

I find eight instance variables in __init__. That’s quite a few. I pledge to look for violations of the Single Responsibility Principle later.

It dawns on me that Eclipse doesn’t know my current version control selection, <a href= “http://en.wikipedia.org/wiki/Bazaar_(vcs)”>bazaar. This distresses me, because all the Team features of Eclipse are lost to me. I won’t be able to do all my checkin/update/diff functions from within the IDE. I vow to look for a plugin soonest. That’s two promises, two items of debt so far. I will keep going today.

For now, I see a function called _get_adjacent. It has a pretty simple function, but it looks like this:

def _get_adjacent(self, x, y): """Provide a list of all tiles adjacent to the given tile. This function takes the x and y coordinates of a tile, and returns a list of a 2-tuples containing the coordinates of all adjacent tiles. x and y are the x and y coordinates of the base tile, respectively. """ adjlist = [(x - 1, y - 1), (x, y - 1), (x + 1, y - 1), (x - 1, y), (x + 1, y), (x - 1, y + 1), (x, y + 1), (x + 1, y + 1)] rmlist = [] for adjpair in adjlist: for value, index in [(-1, 0), (-1, 1), (self.cols, 0), (self.rows, 1)]: if adjpair[index] == value: rmlist.append(adjpair) break for item in rmlist: adjlist.remove(item) return adjlist
It looked pretty heavy to me, for the small amount of work it’s doing. Ah, it had to use nested loops instead of list comprehensions, and it’s adding unnecessary rows only to remove them later. I think I could do better if I had a body of tests to protect me.
I figure it’s time to try PyUnit inside Eclipse. I create a new file called testPysweeper.py. I hate the name, and again go against my better instincts by leaving it that for a while. That would be yet another promise, and I’m running out of patience for procrastination. I rename it testAdjacency.py. I feel an instant sense of relief and go on to write a series of tests of the existing _get_adjacent method. It is

class PySweeperTest(unittest.TestCase): def parentSetup(self, columns, rows, mines, target): target_column, target_row = target self.game = pysweeper.Field(columns,rows,mines) self.target = (target_column,target_row) self.result = self.game._get_adjacent_cells(*self.target) self.result.sort() class CellsAdjacentToCol2Row2(PySweeperTest): """Expecting a circle of squares around the target cell""" def setUp(self): self.parentSetup(5,5,0, (2,2)) def testShouldFindEightCells(self): assert len(self.result) == 8 def testShouldNotIncludeCol2Row2(self): assert self.target not in self.result
These are not highly imaginative, and continue with several different scenarios. I test a normal “middle” cell, one against an edge, the top left and bottom right corners, etc.

You’ll notice the base class PySweeperTest. I didn’t have that to start with but added it to reduce duplication, and I named it badly. These are not general PySweeper tests, but they are specific cell adjacency tests. Naming is easily fixed.

A cool thing happens. I rename it AdjacencyTest and Suddenly all the test names become redundant and silly. I start renaming them, and
class CellsAdjacentToCol0Row0(PysweeperTest):
becomes
class Column0Row0(AdjacencyTest):
On further reflection, this becomes:
class UpperLeftCorner(AdjacencyTest):
It’s amazing how wrong my first instinct can be. The test is so much more obvious now, and the conditions make so much more sense. I smile in mild self-derision, but I’m happy to have cleaned it up quickly. The PyDev refactoring tools feel pretty comfortable to me now. One goal of the exercise is being met.

The tests run for the old routine, and I write a new routine.

I hit my next snag trying to run the python unit test. I have always done python from the command line, and would run my tests as if they were any other python program. I never did them as:

python -m unittest filename.testname
I think that this must be what Eclipse wants to do. I can use the option to Run as.../Python run but I cannot use the option to Run as.../Python unit-test. When I try I get the following error:
Finding files... ['/home/tottinge/Projects/Pysweeper-1.0/testAdjacency.py'] done. Importing test modules... done. ---------------------------------------------------------------------- Ran 0 tests in 0.000s
It found the right test file, but either it didn’t import it, or else it imported it and somehow didn’t recognize any tests. I added a print line to the file so I could see it, and got nothing. Why would it find my file and not import it? This is something that requires a little google..

It doesn’t work when I use the python -m unittest blah either. Maybe the problem isn’t that I’m not used to PyDev, but that I don’t know enough about PyUnit? I put this off, since I have a reasonable workaround for now. This is another bit of debt to work off. I’m unhappy at how fast it piles up.

The old version of _get_adjacent was about 21 lines with loops and local variables. Here is a smaller version using list comprehensions:

def _get_adjacent(self, x, y): """Provide a list of all tiles adjacent to the given tile.""" columns = (x-1,x,x+1); rows = (y-1,y,y+1); adjacent_pairs = [ (col,row) for col in columns for row in rows if 0<=row<self.rows and 0<=col<self.cols] adjacent_pairs.remove( (x,y) ) return adjacent_pairs

A list comp is very simple. The one above returns a list of (row,column) pairs by iterating over columns and rows, and only returns a pair if the row and column are both on the grid. But it doesn’t look that simple above. My spider sense buzzes. I immediately worry that the list comp is too complex. The filtering is too deep in the list comprehension and is too far from the initialization of rows and columns. A quick modification and retest.

def _get_adjacent_cells( self, x, y ): """Provide a list of all tiles adjacent to the given tile.""" columns = [ col for col in ( x-1, x, x+1) if 0<=col<self.cols ] rows = [ row for row in (y-1, y, y+1) if 0<=row<self.rows ] adjacent_pairs = [ ( col, row ) for col in columns for row in rows ] adjacent_pairs.remove( ( x, y ) ) return adjacent_pairs

The tests all pass, and the code is more obvious to me. I have no teammate to correct me, so I leave it as-is.

One more feature to look at today, and that’s the compare with local history. It’s a beautiful thing. Eclipse is becoming a very good friend to me.