Object Mentor Blog: X Tests are not X Tests

X Tests are not X Tests

Michael Feathers — Mon, 13 Apr 2009 11:48:00 -0500

Testing is a slippery subject, and it’s reasonably hard to talk about for one simple reason: the nomenclature is chaotic. Years ago, I went to a summit with some testing gurus. I was one of the lone developers there and I asked about the taxonomy of testing. Cem Kaner, Bret Pettichord, Brian Marick, and James Bach went through it for us on a flipchart, and it was a nightmare. You can name tests after their scope: (unit, component, system), their place in the development process (smoke, integration, acceptance, regression), their focus (performance, functional) their visibility (white box, black box), the role of the people writing them (developer, customer).. The list goes on. There are far more than I can remember.

Why is it so confusing? There are are a couple of reasons. One is that different communities have developed different nomenclature over time. But, let’s face it, that’s true in most fields. The thing which makes testing nomenclature worse is that the tests themselves aren’t all that different, or at least, they are often not different enough for us to for us to distinguish them without being told. Yes, we can tell the difference between a unit test and an acceptance test in most systems, but really there is no force which prevents tests of different types from bleeding through into each other. Often the “type” of a test is more like an attribute: “here I have a blackbox smoke test, written by a developer for component integration.” In the end, all we have are tests and each of them can serve purposes beyond the purpose we originally intended.

Earlier today, I read a blog by Stephen Walther: TDD Tests are not Unit Tests. In it, he draws some distinctions between various types of testing. It’s great that he wrote it because it’s nice for us to have mental categories for these things, but we have to remember is that they really are just categories. We get to choose how distinct they will be. When I write code, most of my TDD tests end up being the same as my unit tests. I find value in forcing that overlap, and in general, I think overlapping test purposes are great to the degree that the purposes don’t conflict. You get more for less that way.

I don’t see any remedy for the muddle of test types. We will continue to make up terms to distinguish tests. We’ll just have to remember that the types are labels, not bins.

"X Tests are not X Tests" by iphone fix

Tue, 09 Mar 2010 21:34:34 -0600

it is really complicated.

"X Tests are not X Tests" by fuel

Fri, 05 Mar 2010 04:55:19 -0600

Yeah same here – outsourcing is the key to all big project and testing, but I always first learn the stuff myself so I would not be screwed hard, and this is the usual happening for people who throw the big money in and find that all the money is spent for nothing! JoHnV

"X Tests are not X Tests" by Energy

Mon, 01 Mar 2010 10:24:40 -0600

I always outsource big testing, I know I could know the stuff myself, but I have chosen the simpler road :)

"X Tests are not X Tests" by Ivonne

Fri, 18 Dec 2009 04:30:40 -0600

“Why is it so confusing? There are are a couple of reasons”

- Actualy, it is not so confusing. It’s all depends on your point of view. However, very good post!

Cheers!

"X Tests are not X Tests" by free casino bonus offers

Mon, 16 Nov 2009 23:07:20 -0600

Not just this post, the whole site and the articles posted here are great. You guys are really woking hard and giving most your time to such things.

"X Tests are not X Tests" by Bingo

Thu, 12 Nov 2009 05:54:52 -0600

Nice post but i got little.

"X Tests are not X Tests" by media.miko@googlemail.com

Tue, 25 Aug 2009 06:22:38 -0500

http://www.mistergammon.com/he/

"X Tests are not X Tests" by ?????

Wed, 08 Jul 2009 02:47:57 -0500

??? ???

"X Tests are not X Tests" by jean

Tue, 21 Apr 2009 23:12:18 -0500

I think the important thing is that a team understand the naming convention. If different teams are calling things differently, then it is not that big a deal. I know I sometimes struggle with the naming – what is an acceptance test, can a functional test be an acceptance test? I like calling tests that are created in an iteration “acceptance tests” but they can be different types of tests – performance, security, functional, etc.

"X Tests are not X Tests" by Pete Wood

Fri, 17 Apr 2009 07:32:18 -0500

Reminds me of a comment on Ward’s Wiki

http://c2.com/cgi/wiki?SixThinkingHats

‘They are not classifications for thoughts which have already occurred’

"X Tests are not X Tests" by Kyle Szklenski

Wed, 15 Apr 2009 08:42:55 -0500

I think of my TDD tests as both unit tests and sample test-runs of the interfaces/classes I build. In the past, when it was much harder to unit test C++, I would write the top-most-level code first, in order to figure out exactly what functions I needed on the objects that code used – that’s exactly how I use TDD, and I think is the primary reason to use TDD. I forget, but I may have a good post about that up on my blog. If not, I have one in the works.

I think if you’re not able to use your TDD tests as unit tests, then you’ve lost some of the power of TDD. Admittedly, it’s not what I consider the most important part of TDD, but it is a very nice aspect. I think I consider Walther’s post to be mostly artificial. There may or may not be something there, but he certainly (as Rob points out) did not make a point. It may be that I’m just defining unit test differently than he is, though.

"X Tests are not X Tests" by Oil

Tue, 14 Apr 2009 11:53:29 -0500

“taxonomy of testing” that’s heavenly.

"X Tests are not X Tests" by Rob Conery

Mon, 13 Apr 2009 14:53:15 -0500

Perfect summation – and was (essentially) the point I was raising on his blog. It was a decent discussion until he decided to twist my words for me and then deleted all comments.

Weak, weak maneuver. Complete professional disrespect and I work with the guy. Go figure.

There are many ways to think of tests – Stephen had a valid point to make but he never made it. I kept expecting a “therefore…” and it didn’t come.

/unsub for me and as far as I’m concerned, my conversations with him are over.

"X Tests are not X Tests" by Gil Zilberfeld

Mon, 13 Apr 2009 13:49:02 -0500

I think the number of names for the different tests comes from its uses. Developers write unit test. QA run functional tests. They also do smoke tests. But I made developers run the smoke tests prior to moving the build to QA.

So the jargon is out there, and everybody uses it differently. There are some terms that are more clear to most people (I believe) like black-box testing. The more we discuss these it will become clearer, and that’s why I welcome Stephen’s post. Even if I don’t agree with everything he says. But the discussion has its way of strengthening the definition overtime.

"X Tests are not X Tests" by Antony Marcano

Mon, 13 Apr 2009 13:38:06 -0500

Just read Stephen’s blog… I think he’s trying to get things clear in his own head… (nothing wrong with that… it’s one of the things I use my blog for)...

Beck pre-empts this debate in TDD by example by clarifying that by unit tests he means ‘small-scale tests’ rather than the ‘traditional definition of unit tests’.

Perhaps a better starting point for Stephen might have been “TDD tests are sometimes unit tests and sometimes more”... (Although, I can’t remember the last time I felt the need for a “TDD test” to use concrete collaborators… I find Test-First Acceptance Tests tend to flush out integration oversights adequately).

"X Tests are not X Tests" by Antony Marcano

Mon, 13 Apr 2009 13:19:31 -0500

...Like tags. As in a test might fulfil more than one purpose or be relevant for more than one point of focus, much like the tags we use on blog posts. (or is that what you meant by label?)

"X Tests are not X Tests" by Nick Swarr

Mon, 13 Apr 2009 13:15:02 -0500

Thanks for putting this up. I didn’t know how to read Stephen’s article as I don’t think there was a clear enough distinction between the activities he’s describing as unit testing and TDD. It seemed like an odd mixture between unit testing, TDD, acceptance testing and a dash of BDD. There were one or two comments that left me scratching my head.

Anyways, I am happily doing TDD of which unit tests are the by-product. Nothing wrong with that.