Software Calculus - The Missing Abstraction. 324
The problem of infinity plagued mathematicians for millennia. Consider Xeno’s paradox; the one with Achilles and the tortoise. While it was intuitively clear that Achilles would pass the Tortoise quickly, the algebra and logic of the day seemed to suggest that the Tortoise would win every race given a head start. Every time Achilles got to where the tortoise was, the tortoise would have moved on. The ancients had no way to see that an infinite sum could be finite.
Then came Leibnitz and everything changed. Suddenly infinity was tractable. Suddenly you could sum infinite series and write the equations that showed Achilles passing the tortoise. Suddenly a whole range of calculations that had either been impossible or intractable became trivial.
Calculus was a watershed invention for mathematics. It opened up vistas that we have yet to fully plumb. It made possible things like Newtonian mechanics, Maxwell’s equations, special and general relativity and quantum mechanics. It supports the entire framework of our modern world. We need a watershed like that for software.
If you listen to my keynote: Twenty-Five Zeros you’ll hear me go on and on about how even though software has changed a lot in form over the last fifty years, it has changed little in substance. Software is still the organization of sequence, selection, and iteration.
For fifty years we have been inventing new languages, notations, and formulations to manage Sequence, Selection, and Iteration (SSI). Structured Programming is simply a way to organize SSI. Objects are another way to organize SSI. Functional is still another. Indeed, almost all of our software technologies are just different ways of organizing Sequence, Selection, and Iteration.
This is similar to Algebra in the days before calculus. We knew how to solve linear and polynomial equations. We knew how to complete squares and find roots. But in the end it was all just different ways to organize adding. That may sound simplistic, but it’s not. Subtracting is just adding in reverse. Multiplying is just adding repeatedly. Division is just multiplication in reverse. In short, Algebra is an organizational strategy for adding.
Algebra went through many different languages and notations too, just like software has. Think about Roman and Greek numerals. Think how long it took to invent the concept of zero, or the positional exponential notation we use today.
And then one day Newton saw an apple fall, and he changed the way we thought about mathematics. Suddenly it wasn’t about adding anymore. Suddenly it was about infinities and differentials. Mathematical reasoning was raised to a new order of abstraction.
Where is that apple for software (pun intended). Where is the Newton or Leibnitz that will transform everything about the way we think about software. Where is that long-sought new level of abstraction?
For awhile we thought it would be MDA. Bzzzzt, wrong. We thought it would be logic programming like prolog1. Bzzzt. We thought it would be database scripts and 4GLs. Bzzzt. None of those did it. None of those can do it. They are still just various ways of organizing sequence, selection, and iteration.
Some people have set their sights on quantum computing. While I’ll grant you that computations with bits that can be both states simultaneously is interesting, in the end I think this is just another hardware trick to increase throughput as opposed to a whole new way to think about software.
This software transformation, whatever it is, is coming. It must come; because we simply cannot keep piling complexity upon complexity. We need some new organizing principle that revamps the very foundations of the way we think about software and opens up vistas that will take us into the 22nd century.
1 Prolog comes closest to being something more than a simple reorganization of sequence, selection, and iteration. At first look logic programming seems very different. In the end, however, an algorithm expressed in prolog can be translated into any of the other languages, demonstrating the eventual equivalence.
TDD in Clojure 461
OO is a tell-don’t-ask paradigm. Yes, I know people don’t always use it that way, but one of Kay’s original concepts was that objects were like cells in a living creature. The cells in a living creature do not ask any questions. They simply tell each other what to do. Neurons are tellers, not askers. Hormones are tellers not askers. In biological systems, (and in Kay’s original concept for OO) communication was half-duplex.
Clojure is a functional language. Functional languages are ask-dont-tell. Indeed, the whole notion of “tell” is to change the state of the system. In a functional program there is no state to change. So “telling” makes little sense.
When we use TDD to develop a tell-don’t-ask system, we start at the high level and write tests using mocks to make sure we are issuing the correct “tells”. We proceed from the top of the system to the bottom of the system. The last tests we write are for the utilities at the very bottom.
In an ask-don’t-tell system, data starts at the bottom and flows upwards. The operation of each function depends on the data fed to it by the lower level functions. There is no mocking framework. So we write tests that start at the bottom, and we work our way up the the top.
Therein lies the rub.
In a tell-don’t-ask system, the tells at the high level are relatively complex. They branch out into lower subsystems getting simpler, but more numerous as they descend. Testing these tells using mocks is not particularly difficult because we don’t need to depend on the lower level functions being there. The mocks make them irrelevant.
In an ask-don’t-tell system the asks at the low level are simple, but as the data moves upwards it gets grouped and composed into lists, maps, sets, and other complex data structures. At the top the data is in it’s most complex form. Writing tests against that complex data is difficult at best. And there is currently no way to mock out the lower levels1 so all tests written at the high level depend on all the functions below.
The perception of writing tests from the bottom to the top can be horrific at first. Consider, for example, the Orbit program I just wrote. This program simulates N-body gravitation. Imagine that I am writing tests at the top level. I have three bodies at position Pa, Pb, and Pc. They have masses Ma, Mb, and Mc. They have velocity vectors of Va, Vb, Vc. The test I want to write needs to make sure that new positions Pa’, Pb’, Pc’, and new Velocity vectors Va’, Vb’, and Vc’ are computed correctly. How do I do that?
Should I write a test that looks like this?
test-update {
Pa = (1,1)
Ma = 2
Va = (0,0)
Pb = (1,2)
Mb = 3
Vb = (0,0)
Pc = (4,5)
Mc = 4
Vc = (0,0)
update-all
Pa should == (1.096, 4.128)
Va should == (0.096, 3.128)
Pb should == (1.1571348402636772, 0.1571348402636774)
Vb should == (0.15713484026367727, -1.8428651597363226)
Pc should == (3.834148869802242, 4.818148869802242)
Vc should == (-0.16585113019775796, -0.18185113019775795)
}
A test like this is awful. It’s loaded with magic numbers, and secret information. It tells me nothing about how the update-all function is working. It only tells me that it generated certain numbers. Are those numbers correct? How would I know?
But wait! I’m working in a functional language. That means that every function I call with certain inputs will always return the same value; no matter how many times I call it. Functions don’t change state! And that means that I can write my tests quite differently.
How does update-all work? Simple, given a list of objects it performs the following operations (written statefully):
update-all(objects) {
for each object in objects {
accumulate-forces(object, objects)
}
for each object in objects {
accelerate(object)
reposition(object)
}
}
This is written in stateful form to make is easier for our non-functional friends to follow. First we accumulate the force of gravity between all the objects. This amounts to evaluating Newton’s F=Gm1m1/r^2 formula for each pair of objects, and adding up the force vectors.
Then, for each object we accelerate that object by applying the force vector to it’s mass, and adding the resultant delta-v vector to it’s velocity vector.
Then, for each object we reposition that object by applying the velocity vector to it’s current position.
Here’s the clojure code for update-all
(defn update-all [os] (reposition-all (accelerate-all (calculate-forces-on-all os))))
In this code you can clearly see the bottom-to-top flow of the application. First we calculate forces, then we accelerate, and finally we reposition.
Now, what do these -all functions look like? Here they are:
(defn calculate-forces-on-all [os] (map #(accumulate-forces % os) os)) (defn accelerate-all [os] (map accelerate os)) (defn reposition-all [os] (map reposition os))
If you don’t read clojure, don’t worry. the map function simply creates a new list from an old list by applying a function to each element of the old list. So in the case of reposition-all it simply calls reposition on the list of objects (os) producing a new list of objects that have been repositioned.
From this we can determine that the function of update-all is to call the three functions (accumulate-forces, accelerate, and reposition) on each element of the input list, producing a new list.
Notice how similar that is to a statement we might make about a high level method in an OO program. (It’s got to call these three functions on each element of the list). In an OO language we would mock out the three functions and just make sure they’d been called for each element. The calculations would be bypassed as irrelevant.
Oddly, we can make the same statement in clojure. Here’s the test for update-all
(testing "update-all"
(let [
o1 (make-object ...)
o2 (make-object ...)
o3 (make-object ...)
os [o1 o2 o3]
us (update-all os)
]
(is (= (nth us 0) (reposition (accelerate (accumulate-forces os o1)
(is (= (nth us 1) (reposition (accelerate (accumulate-forces os o2)
(is (= (nth us 2) (reposition (accelerate (accumulate-forces os o3)
)
)
If you don’t read clojure don’t worry. All this is saying is that we test the update-all function by calling the appropriate functions for each input object, and then see if the elements in the output list match them.
In an OO program we’d find this dangerous because of side-effects. We couldn’t be sure that the functions could safely be called without changing the state of some object in the system. But in a functional language it doesn’t matter how many times you call a function. So long as you pass in the same data, you will get the same result.
So this test simply checks that the appropriate three functions are getting called on each element of the list. This is exactly the same thing an OO programmer would do with a mock object!
Is TDD necessary in Clojure?
If you follow the code in the Orbit example, you’ll note that I wrote tests for all the computations, but did not write tests for the Swing-Gui. This is typical of the way that I work. I try to test all business rules, but I “fiddle” with the GUI until I like it.
If you look carefully you’ll find that amidst the GUI functions there are some “presentation” functions that could have been tested, but that I neglected to write with TDD[2]. These functions were the worst to get working. I continuously encountered NPEs and Illegal Cast exceptions while trying to get them to work.
My conclusion is that Clojure without TDD is just as much a nightmare as Java or Ruby without TDD.
Summary
In OO we tend to TDD our way from the top to the bottom by using Mocks. In Clojure we tend to TDD our way from the bottom to the top. In either case we can compose our tests in terms of the functions they should call on the lower level objects. In the case of OO we use mocks to tell us if the functions have been called properly. This protects us from side-effects and allows us to decouple our tests from the whole system. In clojure we can rely on the fact that the language is functional, and that no matter how many times you call a function it will return the same value.
1 Brian Marick is working on something that looks a lot like a mocking framework for clojure. If his ideas pan out, we may be able to TDD from the top to the bottom in Clojure.
2 This is an unconscious game we all play with ourselves. When we have a segment of code that we consider to be immune to TDD (like GUI) then we unconsciously move lots of otherwise testable code into that segment. Yes, I heard my green band complain every time I did it; but I ignored it because I was in the GUI. Whoops.
Orbit in Clojure 387
I spent the last two days (in between the usual BS) writing a simple orbital simulator in Clojure using Java interop with Swing. This was a very pleasant experience, and I like the way the code turned out – even the swing code!
You can see the source code here
Those of you who are experienced with Clojure, I’d like your opinion on my use of namespaces and modules and other issues of style.
Those of you who are not experienced with Clojure, should start. You might want to use this application as a tutorial.
And just have fun watching the simulation of the coalescence of an accretion disk around a newly formed star.
Clojure Prime Factors 220
Can anyone create a simpler version of prime factors in Clojure?
(ns primeFactors)
(defn of
([n]
(of [] n 2))
([factors n candidate]
(cond
(= n 1) factors
(= 0 (rem n candidate)) (recur (conj factors candidate) (quot n candidate) candidate)
(> candidate (Math/sqrt n)) (conj factors n)
:else (recur factors n (inc candidate))
)
)
)
Sufficient Design means Damned Good Design. 264
@JoshuaKerievsky wrote a blog entitled “Sufficient Design”.
Josh makes this point:
‘Yet some programmers argue that the software design quality of every last piece of code ought to be as high as possible. “Produce clean code or you are not a software craftsman!”’
He goes on to say:
“Yet ultimately the craftsmanship advice fails to consider simple economics: If you take the time to craft code of little or moderate value to users, you’re wasting time and money.”
Now this sounds like heresy, and I can imagine that software craftsmanship supporters (like me) are ready to storm the halls of Industrial Logic and string the blighter up by his toenails!
But hold on there craftsmen, don’t get the pitchforks out yet. Look at the scenario that Josh describes. It’s quite revealing.
Josh’s example of “not being a craftsman” is his niggling worry over a function that returns a string but in one derivative case ought to return a void.
Horrors! Horrors! He left a bit of evolved cruft in the design! Revoke his craftsman license and sick the dogs on him!
Ah, but wait. He also says that he spent a half-day trying to refactor this into a better shape but eventually had to give up.
The fool! The nincompoop! The anti-craftsman! A pox on him and all his ilk!
OK, enough hyperbole. It seems to me that Josh was behaving exactly as a craftsman should. He was worrying about exactly the kinds of things a craftsman ought to worry about. He was making the kinds of pragmatic decisions a craftsman ought to make. He was not leaving a huge mess in the system and rushing on to the next feature to implement. Instead he was taking care of his code. The fact that he put so much energy, time, and thought into such a small issue as an inconsistent return type speaks volumes for Josh’s integrity as a software craftsman.
So, as far as Josh Kerievsky is concerned “Sufficient Design” is “Pretty Damned Good Design”.
Look, all our software will have little tiny warts. There’s no such thing as a perfect system. Craftsmen like Josh work hard to keep those warts as small as possible, and to sequester them into parts of the system where they can do the least harm.
So the only aspect of Josh’s post that I disagree with is his contention that the “craftsman” message is one of unrelenting perfection. Craftsmen are first and foremost pragmatists. They seek very high quality code; but they are not so consumed with perfection that they make foolish economic tradeoffs.
Certification: Don't Waste Your Time! 394
As I have said before, there’s nothing particularly wrong with the current mania for certification. If you want to be certified at the cost of a 2-day course, by all means get certified. If you want to certify people for attending your 2-day course, by all means hold the course and hand out the certificates. It’s all good. Make money! Be fruitful and multiply!
But be careful not to waste your time.
How could certification be a waste of time? That depends on your motive.
- If you are getting certified in order to impress someone, like a hiring manager, or a recruiter, or your peers, you are wasting your time. Nobody worth impressing is going to find this certification impressive. Indeed, the people you really want to impress are likely to find it a bit mundane.
- If you are getting certified in order to get hired, you are wasting your time. Nobody is going to hire you simply because of that “C”, and nobody worth working for is going to require that “C”.
- If you have decided to hire only certified people, you are wasting your time. The population of certified people is not richer in talent, skill, or knowledge. Indeed, it may be poorer. Remember, those who have talent don’t need certification as much as those who don’t.
What part of certification is not a waste of time?
- The primary benefit you are getting is the instruction; but be careful: There are lots of pretty mediocre instructors out there. Some of the instructors teach pretty good courses, but others are just hoping that all you care about is the certification.
- So do a little research and find the best instructors. You may, in fact, find that some of the best instructors and courses do not offer certification. That shouldn’t stop you from considering them.
But wait, aren’t the instructors certified as trainers?
- Sure. They paid the money and took the course to become a certified trainer.
- But that doesn’t necessarily mean that they:
- are a good instructor.
- know what they are teaching.
- have done what they are teaching.
- are qualified to teach you.
OK, but isn’t there some benefit to the certification itself?
- Sure. A nice piece of paper.
The Tricky Bit 224
I once heard a story about the early days of the Concorde. A british MP was on board for a demonstration flight. As the jet went supersonic he disappointedly commented to one of the designers that it didn’t feel any different at all. The designer beamed and said: “Yes, that was the tricky bit.”
I wonder if the MP would have been happier if the plane had begun to vibrate and shake and make terrifying noises.
While at #openvolcano10, Gojko Adzic and Steve Freeman told the story of a company in which one team, among many, had gone agile. Over time that team managed to get into a nice rhythm, passing it’s continuous builds, adding lots of business value, working normal hours, and keeping their customers happy. Meanwhile other teams at this company were facing delays, defects, frustrated customers, and lots of overtime.
The managers at this company looked at the agile team, working like a well-oiled machine, and looked at all the other teams toiling in vain at the bilge pump, and came to the conclusion that the project that the agile team was working on was too simple to keep them busy. That the other teams’ projects were far more difficult. It didn’t even occur to them that the agile team had figured out the tricky bit.
The R.E.A.L.I.T.Y. principles of Agile Software Certification. 207
As you are probably aware the Scrum Alliance is planning to offer a Certified Scrum Developer program. You can read about it here.
Interestingly enough Microsoft, in collaboration with Ken Schwaber, is offering a Professional Scrum Developer program. You can read about that here (look carefully at the url).
The scrum alliance program seems to be about agile skills. The Microsoft program seems to be about Visual Studio. So all is right with the world.
Is there anything wrong with this? Is this somehow immoral or bad? Not at all. This is just people doing what people do: Selling services and making money. I’m all for it. I expect the courses to be of a quality that you’d expect from professionals; and I’m sure that students will learn useful information. Indeed, from time to time, even I have offered CSM courses taught by people I trust and respect.
There are, however, some claims that need to be dealt with. The most important is that these programs shorten the recruiting process.
I’ve been thinking about offering a Certified Recruiter for Agile Professionals course. This fifty-two week course will teach, and reteach, a number of principles of Agile Software Certification. I call this repertoire of principles: R.E.A.L.I.T.Y.
The first principle is the Redaction of Certification Principle (RCP). The principle states:
Certifications generally certify nothing whatever about experience, knowledge, or skill. Generally they certify that the certificate holder has attended (or at least paid to attend) a course. Perhaps they took (and maybe even passed) an exam based on that course.
Based on this principle, any recruiter taking my Certified Recruiter for Agile Professionals course will not be allowed to complete the course or receive their certification until they have taken the following oath.
As a Cerftified Recruiter for Agile Professionals I solemnly swear before Knuth and all here present:
That before I read a resume, I will find every use of the word “certified” on that document and redact it with whiteout.
That if a candidate uses the word “certified” in an interview, I will ask the candidate to repeat him- or herself without using that word.
That I will pay no attention whatever to any implications of that word, nor will that word in any way influence my opinions regarding that candidate.
This oath is an exemplar of the Certification Nullification Principle (CNP), which is another of the R.E.A.L.I.T.Y. principles of common sense and moderately competent thinking. The principle states:
The word “certified”, when used in the context of Agile Software Development, is a filler word that has no bearing on anything salient or interesting about individuals, skills, knowledge, or anything else. It is a marketing word similar to “new”, “improved”, “natural” and “organic”. It can safely be removed from all documents and discussions without changing their meaning.
This principle leads to the well known Certification Uncertainty Principle (CUP), yet another of the R.E.A.L.I.T.Y. principles. This principle states:
Any acronym that includes the letter ‘C’ standing for the word “Certification” can be safely changed into a similar acronym that eliminates the letter ‘C’ and puts a ’?’ at the end. This transformation of the acronym vastly improves the meaning of the orginal.
That last sentence probably requires some justification. After all, extraordinary claims require extraordinary substantiation. So lets try a few experiments:
Statement Ac'nm CUP Resulting Statement ---------------------------- ----- ---- --------------------- Certified Scrum Master CSM SM? Scrum Master? Certified Scrum Developer CSD SD? Scrum Developer? Certified Scrum Trainer CST ST? Scrum Trainer? Certified Scrum Professional CSP SP? Scrum Professional? Certified Scrum Product Owner CSPO SPO? Scrum Product Owner? Certified Scrum Coach CSC SC? Scrum Coach?
As we can see, the meanings of the statements are indeed clarified. In my course the Certified Recruiter for Agile Professionals is taught that the question mark is the most significant aspect of each of the resultant statements.
In recent weeks we have uncovered a potentially profound new principle which we may find necessary to include in the R.E.A.L.I.T.Y. repertoire. This is the so-called Scrum Exclusion Principle (SEP) which tentatively states:
Wherever the word SCRUM appears in any statement, or is represented within any acronym, it can be safely excluded without changing any meaning.
The following table show just how profound the effects of this principle are:
Statement Ac'nm CUP SEP Resulting Statement ---------------------------- ----- ---- ---- --------------------- Certified Scrum Master CSM SM? M? Master? Certified Scrum Developer CSD SD? D? Developer? Certified Scrum Trainer CST ST? T? Trainer? Certified Scrum Professional CSP SP? P? Professional? Certified Scrum Product Owner CSPO SPO? PO? Product Owner? Certified Scrum Coach CSC SC? C? Coach?
We are not yet willing to say that these transformation are a reliable feature of nature. We can only say that, at this point, there are very troubling implications that support it.
Our final advice to Certified Recruiters for Agile Professionals is based on one final principle: The Smoke and Mirrors principle, (SAM) which states:
Well… You know what it states.
The Mayans are Coming! 149
This one is just a quickie; too long for a tweet.
I gave a lightning keynote (actually a key-rant) today at #accu2010. It was about a bet between John Lakos and Kevlin Henney. Kevlin “won” the bet last night by showing that he could get a C++ program to behave in a way that John said couldn’t be done. My rant was about how both of them were wrong (not technically, but morally) and so they both owed me the sum of the bet. (I’m always happy to insert myself into any available revenue stream.)
The problem they had coded, and for which I presented a far superior solution in Clojure, was that old work-horse of prob/stat: “What’s the probability that out of N people in a room two will have the same birthday?”
Now, just to show you how strange James Bach is (and why his strangeness is in some bizarre way actually valuable); once my talk was done, James got up and said “What if some of the people in the room are ancient Mayans who use a completely different kind of calendar making it impossible to compare their birthdays?”
I’ve long said that testers were deviant souls with twisted minds that delight in reducing helpless software systems into piles of smoking slag. I believe James may be the exemplar.
Sapient Testing: The "Professionalism" meme. 308
James Bach gave a stirring keynote today at ACCU 2010. He described a vision of testing that our industry sorely needs. Towit: Testing requires sapience.
Testing, according to Bach, is not about assuring conformance to requirements; rather it is about understanding the requirements. Even that’s not quite right. It is not sufficient to simply understand and verify the requirements. A good tester uses the behavior of the system and the descriptions in the requirements, (and face-to-face interaction with the authors of both) to understand the motivation behind the system. Ultimately it is the tester’s job to divine the system that the customer imagined; and then to illuminate those parts of the system that are not consistent with that imagination.
It seems to me that James is attempting to define “professionalism” as it applies to testing. A professional tester does not blindly follow a test plan. A professional tester does not simply write test plans that reflect the stated requirements. Rather a professional tester takes responsibility for interpreting the requirements with intelligence. He tests, not only the system, but also (and more importantly) the assumptions of the programmers, and specifiers.
I like this view. I like it a lot. I like the fact that testers are seeking professionalism in the same way that developer are. I like the fact that testing is becoming a craft, and that people like James are passionate about that craft. There may yet be hope for our industry!
There has been a long standing frission between James’ view of testing and the Agile emphasis on TDD and automated tests. Agilists have been very focussed on creating suites of automated tests, and exposing the insanity (and inanity) of huge manual testing suites. This focus can be (and has been) misinterpreted as an anti-tester bias.
It seems to me that professional testers are completely compatible with agile development. No, that’s wrong. I think professional testers are utterly essential to agile development. I don’t want testers who rotely execute brain-dead manual test plans. I want testers using their brains! I want testers to be partners in the effort to create world-class, high-quality software. As a professional developer I want – I need – professional testers helping me find my blind spots, illuminating the naivete of my assumptions, and partnering with me to satisfy the needs of our customers.