I worked through just this smaller problem of converting an expression into tokens using the iterator pattern a few times and eventually got what I thought was a pretty good solution. I was wrong, I got it even tighter by making more appropriate use of the regular expression libraries in Java. (This is an example of where my regex mental model – based primarily on grep, vi and AWK – was inadequate when taking into consideration typical implementations of regex libraries with classes like Pattern and Matcher.)

Anyway, here’s a writeup for that Kata: http://schuchert.wikispaces.com/Katas.ExpressionTokenizer.

My final version turned into something crazy-short. I did not anticipate the final version. Part of it, as mentioned above, had to do with how I previously envisioned using regular expressions. Part of it might have been that it just took a few times for the patterns naturally occurring to become evident to me. In any case, working through this problem three times resulted in the following revelations:

• Initially I though it would be hard, it was not and in fact much less work than what I was doing in the original problem. And it took a lot less time than I expected.
• The second time around I was looking to duplicate the experience, which I did. But I saw quite a bit of improvement.
• The final version was really an a-ah for me. Initially it was just like the second version. Then, once I had the tests in the kata written, I started messing around, looking for additional improvements (and heavily applying the DRY principle). I found them, exploited commonality and finally I got to the point where there was one thing I did not like, and fixing that led to a huge improvement, which led to a few more significant improvements.

Ultimately, I’ll have several useful take-aways from working on the problem several times.

If you’d like to see the “latest final version” of this, ping me. I’ll consider posting it to the blog, but I’d rather not give anything away without a specific request.