Rich Hickey on Testing 18
It was an interesting week at JavaOne, with lots of talks and hallway discussions about new languages on the JVM. One of those languages is Clojure.
Rich Hickey, the creator of Clojure, gave a talk at the Bay Area Clojure User Group Wednesday evening. During the Q&A part, he said that he’s not big on writing tests, although he always runs the tests that other people have written before he commits changes.
Of course, there are many people, including us Object Mentors, who consider TDD to be an essential part of professional software development. Obviously not everyone agrees. James Coplien has been another critic of this view.
We should never accept any dogma. Why is TDD considered important? What does it purport to do? TDD provides two important benefits.
- Driving the design.
- Building a suite of automated regression tests.
So, if you can satisfy both requirements without TDD, then technically you don’t need it. In Rich’s case, he said he spends a lot of time thinking about what he’s going to do before he does it. In this way, he satisfies the first requirement, driving the design. I had a spirited discussion with some Thoughtworkers afterwards and Ola Bini said what a lot of us think, “I do that thinking by writing tests.” I’ll freely admit that when I am really experimenting with ideas, I might just write code, but once I know how to proceed, I return to the beginning and test drive the “production” code.
Rich also made an off-hand comment that if he screws something up, he’s got thousands of users who will let him know! That ex post facto testing, along with the Rich’s own devotion to doing high-quality work, does a good job of handling regressions.
But Rich mentioned something else that is also very important. In a functional language, where values are immutability and mutable state is handled in specific, principled ways, regressions don’t happen nearly as often. Clojure has one of the most deeply thought out approaches for handling state, which is the genius of Clojure.
I asked Rich how long he worked on Clojure before releasing it to the world. He spent about 2 1/2 years, much of that time working exclusively on Clojure (and eating through his savings). When he finally “announced” it, his “marketing” consisted of one email to some friends in the Common Lisp community. The rest was viral, a testament to the justified excitement Clojure has generated.
For me, I’ll probably always do my design thinking through tests, especially when I’m writing code in imperative languages, like Java and Ruby. I’ll continue to encourage my clients to use TDD, because I find that TDD is the most productive way to achieve high quality. I want the safety net of a good test suite. I’m also writing more and more of my code in a functional style, with minimal side effects and mutable data. You should, too.
Not sure if I get it right, I know immutability does help in less screw ups by removing side effects as we can’t change the state but how does it help in designing the software? Oh well… It all depends on how you design. I do using tests other may not do so :)
Not sure if I get it right, I know immutability does help in less screw ups by removing side effects as we can’t change the state but how does it help in designing the software? Oh well… It all depends on how you design. I do using tests other may not do so :)
The comments about “thinking it out ahead” kind of creeped me out too at first, but the next day I read the paper “Out of the Tarpit” that he recommended; it makes a very strong case for informal reasoning being a suitable substitute for test cases (not counting the regression scenario). It also helps that Clojure itself has much less statefulness than your average Clojure project.
Referential transparency is a beautiful thing.
It’s also worth mentioning that clojure-contrib has a nascent community contributed test suite that Rich has said that he does run regularly.
Another important benefit of a good collection of tests is communication.
Clojure is a fine Lisp in many ways. I personally would hesitate to use it for anything in which I had a significant investment given the maintainer’s stand on testing. At least not without a good deal of evidence that Clojure will continue to be maintainable and understood (at the implementation level in particular) by more than one person.
Maybe his approach will work over a long period of time, and for a user community that will rely on Clojure for many heavy-duty, valuable production uses. I cannot say that it won’t.
I can only say that for me this would be a significant reason to hesitate before taking too significant of a plunge.
And that’s saying something because I am a veteran of programming in various Lisps for 29 years, and Lisp generally is my favorite language. I love that Clojure has rejuvinated interest in Lisp.
I had a chance to talk to Rich at QCON earlier this year and he was saying that same thing, that he didn’t see the value of unit testing. When I heard that, it was a bit of a throwback to an earlier day when I knew many people who thought that way and they were really in a very similar situation to his: smallish, single (or few) person project, very intricate, with time and patience to memorize nearly the whole code base. Personally, I wouldn’t do anything like that without unit tests today, but some people did work very well that way as long as they were able to keep the code at a scale in which they could just rewrite the offending parts. Too many projects, however, are runaways. The complexity and size grow to the point where that style of working falls apart.
The thing that annoys me is when someone takes their experience under a set of particular set of rarified conditions and tries to generalize it all the way across development. In this case, saying “Well, I’ve never had a problem on my projects where UTs would help so they are useless for everyone.” I don’t remember Rich ever saying that, but others have.
Oh god someone applied common sense.. better call the internet!
Enough! TDD is fine within a context, so is not using TDD. Wait what did I just say? Context?
Yes context! If people knew about context no one would care about what joel spolsky has to say.
Just look at the code for Clojure and you can see how Rich develops it. There is plenty of scaffolding left in the comments. And it looks like development in any other dynamic language. With repl at your side, the application always running and code always being compiled. The whole cycle of write test, write code, becomes so much shorter that it just does not make sense to call it a formal test. The main point of TDD is to compile and execute your production code right away. This is exactly what repl gives you with shortest cycle possible. Another use of TDD is a way to experiment with API use, or kind of top down design, and this use of TDD is also captured by growing code organically in a repl. I personally try to capture those repl experiments into the tests and then just add them to the unit tests.
Patrick: as for readability/maintainability—I read through the entirety of core.clj (the most fundamental part of Clojure outside of the Java primitives) three or four months after I started with Clojure, and it was very straightforward.
There were only three or four functions (out of ~4000 lines) that didn’t immediately make sense to me, which pleasantly surprised me. Like I said, referential transparency is a great thing. Does wonders for readability.
Phil – the concern I was trying to express goes well beyond readability. But I agree that the clojure code I have seen tends to be v.readable. And having a lisp that takes the applicative style much further than scheme helps and is about time.
The concern I tried to express was for an application developer making a long term, monetary investment in clojure code. Currently it appears to invest in a clojure code base is to put all your eggs in one basket, and for that basket to be Rich Hickey in person.
If I had to choose one person for this, Rich seems to be a good candidate. But I would be unlikely to choose a language so dependent on one person, so early in that languages lifetime.
YMMV
@Patrick,
I should have included tests as communication. Unfortunately, I find that few people are in the habit of using tests this way, either when writing tests or trying to learn a new code base. It’s one of the benefits of TDD that is least appreciated.
JDT tests (http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.jdt.core.tests.model/src/org/eclipse/jdt/core/tests/?sortdir=down ) is an excellent example why tests are so much useful in understanding complex APIs such as JDT AST manipulation.
I am not sure why people (including me) tag this up as a benefit of TDD when it really is tests (TDD or non-TDD) that helps improve communication.
Re tests as communication: I personally find that a test suite is useful for learning a baroque object-oriented API (take, for a Java example, the Apache HTTP library, or POI… or most any Java library for that matter!).
I find the documentation aspect of tests to be of much, much less use in a Lisp, increasingly so as the purity (in a functional sense) of the language increases. You simply don’t need to know which objects to build and when, and which get mutated: so long as you can read a one-line description of a function, you’re good to go. If you still can’t figure it out, you can poke at it in a REPL to find out what it does.
I have not yet encountered a Clojure function or library that I couldn’t understand from a) the function signature, b) the docstring, or as a last resort c) the code itself, which is rarely more than a dozen lines. Most of the docs I need to find are for Java libraries I’m calling from Clojure!
Similarly, the “web of objects” with mutable state are the primary cause of regressions, and thus a big motivation for test suites. I feel much less worried about Clojure’s lack of tests than I would if, say, a Java application server had none.
I intend to contribute to Clojure’s test suite, because I’m a belt and braces kind of guy… but I’m not losing sleep over the lack of one now.
Re tests as communication: I personally find that a test suite is useful for learning a baroque object-oriented API (take, for a Java example, the Apache HTTP library, or POI… or most any Java library for that matter!).
I find the documentation aspect of tests to be of much, much less use in a Lisp, increasingly so as the purity (in a functional sense) of the language increases. You simply don’t need to know which objects to build and when, and which get mutated: so long as you can read a one-line description of a function, you’re good to go. If you still can’t figure it out, you can poke at it in a REPL to find out what it does.
I have not yet encountered a Clojure function or library that I couldn’t understand from a) the function signature, b) the docstring, or as a last resort c) the code itself, which is rarely more than a dozen lines. Most of the docs I need to find are for Java libraries I’m calling from Clojure!
Similarly, the “web of objects” with mutable state are the primary cause of regressions, and thus a big motivation for test suites. I feel much less worried about Clojure’s lack of tests than I would if, say, a Java application server had none.
I intend to contribute to Clojure’s test suite, because I’m a belt and braces kind of guy… but I’m not losing sleep over the lack of one now.
Hmm, this comment system sure makes dupes easy! Sorry about that; please delete one.
I’ve switched to coding in Scheme (another Lisp dialect like Clojure) from Java and Ruby some time ago. And the plain fact is that you don’t need as much testing.
The only reason for TDD is because Java is such a poor language that is ill-suited to concurrent applications with shared state and many pitfalls to trap the unwary. Ruby is possible worse because you lose the protection of static typing although it is largely single threaded so you lose the concurrency problems.
As Rich says by the time you have thought through the code you want to write in Lisp/Scheme – and you really have to do this unlike Java where you can just throw some likely looking code on the screen, and trialled a few things in REPL (interactive evaluation) you’re pretty confident that you code will work for all cases. As this happens at a function level and functions are pure the process is very manageable and repeatable. Compare that with constantly mutating complex object models in Java – hence the need for comprehensive test coverage.
It’s not more testing we need but a better approach i.e. functional programming rather than imperative. There is a reason that studies have shown that Lisp/Scheme is about 8 times more productive than Java and produces faster, more concise and more maintainable code.
That’s not to say that we don’t need thorough integration testing though.
As this happens at a function level and functions are pure the process is very manageable and repeatable.
I come from a C++, Python and Java background and started with Common Lisp more than 2 years ago. For complex logic TDD always helps, even if the functions / objects and methods are purely functional. The benefit of having a set of test cases that signal a regression – even one introduced in the context of the implementation of a new piece of logic or refactoring – persists.
For non-complex functions I agree that the REPL and working and experimenting against a live application are enough in the majority of the cases.
Alas, in big teams a simple function in the midst of several hundred others isn’t such a clear cut and regressions do occur more often than expected. This is something that I think could be covered at the level of integration and acceptance automated tests. I regret that currently in my company we haven’t such a thing… Top of it all, the in most common case the users of your software aren’t motivated and understanding developers who might even send a patch with a test case that exposes the bug, but, paying customers that get very angry when things go wrong.