X Tests are not X Tests 14
Testing is a slippery subject, and it’s reasonably hard to talk about for one simple reason: the nomenclature is chaotic. Years ago, I went to a summit with some testing gurus. I was one of the lone developers there and I asked about the taxonomy of testing. Cem Kaner, Bret Pettichord, Brian Marick, and James Bach went through it for us on a flipchart, and it was a nightmare. You can name tests after their scope: (unit, component, system), their place in the development process (smoke, integration, acceptance, regression), their focus (performance, functional) their visibility (white box, black box), the role of the people writing them (developer, customer).. The list goes on. There are far more than I can remember.
Why is it so confusing? There are are a couple of reasons. One is that different communities have developed different nomenclature over time. But, let’s face it, that’s true in most fields. The thing which makes testing nomenclature worse is that the tests themselves aren’t all that different, or at least, they are often not different enough for us to for us to distinguish them without being told. Yes, we can tell the difference between a unit test and an acceptance test in most systems, but really there is no force which prevents tests of different types from bleeding through into each other. Often the “type” of a test is more like an attribute: “here I have a blackbox smoke test, written by a developer for component integration.” In the end, all we have are tests and each of them can serve purposes beyond the purpose we originally intended.
Earlier today, I read a blog by Stephen Walther: TDD Tests are not Unit Tests. In it, he draws some distinctions between various types of testing. It’s great that he wrote it because it’s nice for us to have mental categories for these things, but we have to remember is that they really are just categories. We get to choose how distinct they will be. When I write code, most of my TDD tests end up being the same as my unit tests. I find value in forcing that overlap, and in general, I think overlapping test purposes are great to the degree that the purposes don’t conflict. You get more for less that way.
I don’t see any remedy for the muddle of test types. We will continue to make up terms to distinguish tests. We’ll just have to remember that the types are labels, not bins.
Thanks for putting this up. I didn’t know how to read Stephen’s article as I don’t think there was a clear enough distinction between the activities he’s describing as unit testing and TDD. It seemed like an odd mixture between unit testing, TDD, acceptance testing and a dash of BDD. There were one or two comments that left me scratching my head.
Anyways, I am happily doing TDD of which unit tests are the by-product. Nothing wrong with that.
...Like tags. As in a test might fulfil more than one purpose or be relevant for more than one point of focus, much like the tags we use on blog posts. (or is that what you meant by label?)
Just read Stephen’s blog… I think he’s trying to get things clear in his own head… (nothing wrong with that… it’s one of the things I use my blog for)...
Beck pre-empts this debate in TDD by example by clarifying that by unit tests he means ‘small-scale tests’ rather than the ‘traditional definition of unit tests’.
Perhaps a better starting point for Stephen might have been “TDD tests are sometimes unit tests and sometimes more”... (Although, I can’t remember the last time I felt the need for a “TDD test” to use concrete collaborators… I find Test-First Acceptance Tests tend to flush out integration oversights adequately).
I think the number of names for the different tests comes from its uses. Developers write unit test. QA run functional tests. They also do smoke tests. But I made developers run the smoke tests prior to moving the build to QA.
So the jargon is out there, and everybody uses it differently. There are some terms that are more clear to most people (I believe) like black-box testing. The more we discuss these it will become clearer, and that’s why I welcome Stephen’s post. Even if I don’t agree with everything he says. But the discussion has its way of strengthening the definition overtime.
Perfect summation – and was (essentially) the point I was raising on his blog. It was a decent discussion until he decided to twist my words for me and then deleted all comments.
Weak, weak maneuver. Complete professional disrespect and I work with the guy. Go figure.
There are many ways to think of tests – Stephen had a valid point to make but he never made it. I kept expecting a “therefore…” and it didn’t come.
/unsub for me and as far as I’m concerned, my conversations with him are over.
“taxonomy of testing” that’s heavenly.
I think of my TDD tests as both unit tests and sample test-runs of the interfaces/classes I build. In the past, when it was much harder to unit test C++, I would write the top-most-level code first, in order to figure out exactly what functions I needed on the objects that code used – that’s exactly how I use TDD, and I think is the primary reason to use TDD. I forget, but I may have a good post about that up on my blog. If not, I have one in the works.
I think if you’re not able to use your TDD tests as unit tests, then you’ve lost some of the power of TDD. Admittedly, it’s not what I consider the most important part of TDD, but it is a very nice aspect. I think I consider Walther’s post to be mostly artificial. There may or may not be something there, but he certainly (as Rob points out) did not make a point. It may be that I’m just defining unit test differently than he is, though.
Reminds me of a comment on Ward’s Wiki
http://c2.com/cgi/wiki?SixThinkingHats
‘They are not classifications for thoughts which have already occurred’
I think the important thing is that a team understand the naming convention. If different teams are calling things differently, then it is not that big a deal. I know I sometimes struggle with the naming – what is an acceptance test, can a functional test be an acceptance test? I like calling tests that are created in an iteration “acceptance tests” but they can be different types of tests – performance, security, functional, etc.
??? ???
http://www.mistergammon.com/he/
Nice post but i got little.
Not just this post, the whole site and the articles posted here are great. You guys are really woking hard and giving most your time to such things.
“Why is it so confusing? There are are a couple of reasons”
- Actualy, it is not so confusing. It’s all depends on your point of view. However, very good post!
Cheers!