Object Mentor Blog: Ruining your Test Automation Strategy.

Ruining your Test Automation Strategy.

Uncle Bob — Tue, 29 Sep 2009 15:42:00 -0500

Everybody wants a test automation strategy nowadays. The reason is clear. It take a lot of effort to run a suite of manual tests; and that effort has be be repeated several times per year.

Consider the fate of this poor customer of mine. The picture you see here is simply the Table of Contents of his manual test plan. That plan has tens of thousands of individual manual tests. It costs them millions of dollars to execute, and they must execute it many times each year.

To make matters worse, in these difficult times, management has told the QA manager that he must cut 50% from his manual testing budget. The question he asked me was: “Which 40,000 of these 80,000 tests should I delete?”

So, as you can probably imagine, this poor guy really wished his tests were automated. Runnning automated tests does not cost six figures several times a year. Running automated tests does not need to be cut in half when budgets get tight. Running automated tests are the way to go.

One common strategy to get your tests automated is to outsource the problem. You hire some team of test writers to transform your manual tests into automated tests using some automation tool. These folks execute the manual test plan while setting up the automation tool to record their actions. Then the tool can simply play the actions back for each new release of the system; and make sure the screens don’t change.

Sounds great doesn’t it? Sounds like just the ticket! Sounds like a simple way to convert manual tests into automated tests!

Yeah, and it’s a sure fire way to make sure you utterly ruin your strategy for test automation!...

Why is this so runious?

Automatically testing a system through the GUI couples the tests to the GUI. That’s right, this is a coupling problem! A standard, every-day, good ol’ software design coupling problem. And the problem with coupling is that when two things are coupled, and you change one of them, the other one breaks.

In the case of automated tests, if you change the GUI, the automated tests break. This was a big enough problem when the test were manual. You always had to go back and edit the test plans to take GUI changes into account. Fortunately, since the test were manual, you had human beings in the loop, and you could expect them to use common sense about simple GUI changes. The fact that what they saw on the screen differed from what they test plan said they should see would not confound them so long as they understood how the GUI had changed.

But an automated tool doesn’t have that kind of reasoning power. If the automated tool sees anything different from what it expects, it simply fails.

Now, clearly, automated tools can be made to be clever enough to avoid simple cosmetic issues like the moving of a button, or a change in the spelling of a menu. But you have to work at making the tests tolerant of such changes. Do you think that outsourced team of test writers care much about that?

In any case, no tool can deal with changes to the navigation structure, or gross changes to the structure of screens. Thus, there will always be a class of GUI changes that will cause huge swaths of automated tests to fail. Unfortunately these kinds of changes are all too common. What’s more the cost of re-recording the tests is high, and the re-recording process itself is error-prone.

The net result is that GUI driven automated tests are fragile, and the process of maintaining them is expensive and unreliable.

To drive this point home, consider the fate of an old client of mine who had tens of thousands of automated tests driven through the GUI. Every time anyone changed the GUI a thousand or so tests broke. The burden of maintaining those tests was so great that this customer added one restrictive policy after another in an effort to prevent changes to the GUI. In the end, GUI changes became officially prohibited. (This was a DOS GUI in the era of Windows!)

Or consider another client who spent a great deal of time and effort creating automated tests through the GUI. This client could not constrain the GUI against change, so had to live with the burden of re-recording the tests. But the burden was too great. Trade-offs were made. Many of the tests de-prioritized, and therefore lost. Bit by bit this customer lost his investment in automated tests.

The bottom line is that automated tests through the GUI are inherently unstable, and will drive you to one or the other of those two undesirable states.

It is hopeless?

Not at all. First of all, you need some acceptance tests that go through the GUI. I think that number is on the order of 5%. These are integration tests that make sure that the whole system is wired up correctly.

You also need to test the GUI through the GUI, and this might be another 5-10% of the total body of tests. What does it mean to test the GUI? It means that you aren’t testing any business rules! You test the GUI and only the GUI. My favorite scheme for doing this is to mock out all the business rule code entirely, replacing it with a dummy that simply provides canned answers to all the GUI interfaces. Imagine, for example, replacing all the servlets in a web system with dummy servlets that pay no attention to their inputs, use no database at all, and simply return canned HTML.

By the same token, the business rule tests should not use the GUI. In fact, the GUI shouldn’t even be turned on. Rather, your tests should invoke the same business rule APIs that the GUI invokes. Indeed, the tests become an alternative UI.

Preferably the Business Rule API is well below the GUI layer. In a web system it should be below the servlet layer. In the best designed systems, it can even be outside the container.

Tools like FitNesse, Cucumber, RobotFX, and GreenPepper are designed for just this kind of API testing.

Tests that drive the system through the API are entirely decoupled from the GUI. When the GUI changes, these tests are completely unaffected. There is no huge burden of test maintenance, and you are not at risk for losing your tests or constraining the GUI.

Summary

Automated testing through the GUI is intuitive, seductive, and almost always wrong! A dedicated program of automatically testing your system through the GUI is almost certain to fail. If you want a robust test automation strategy, plan to use a healthy dose of API testing, and as little GUI testing as possible.

"Ruining your Test Automation Strategy." by Michael

Mon, 26 Jul 2010 21:22:57 -0500

But what if your test department will only give you sign-off if they have tests that are based on the gui. Because the system is so critical that they need to gurantee when they click on a button, the correct business logic is executed. They would not simply accept a bunch of green lights coming from a tool such as Fitnesse where the developers assure them they all is functioning fine.

"Ruining your Test Automation Strategy." by Create a Website

Sat, 17 Jul 2010 00:37:22 -0500

The aim of most businesses with a website is to successfully market their website and increase sales.

"Ruining your Test Automation Strategy." by louis vuitton

Tue, 22 Jun 2010 22:58:22 -0500

*Well , the view of the passage is totally correct ,your details is really reasonable and you guy give us valuable informative post, I totally agree the standpoint of upstairs . http://www.precision-mechanical.net I often surfing on this forum when I m free and I find there are so much good information we can learn in this forum!

"Ruining your Test Automation Strategy." by Great Article

Sat, 12 Jun 2010 05:58:00 -0500

Great stuff about gui tests, use with care and common sense. But at the end of your post you emphasize API-testing, mentioning FitNesse (among others) as a tool with no huge burden of test maintenance.

"Ruining your Test Automation Strategy." by Fireworks

Fri, 28 May 2010 02:16:27 -0500

Really a good post!!!!! The author has done a great research on automation strategy. Thanks for the information.

"Ruining your Test Automation Strategy." by PDF to epub Converter

Wed, 12 May 2010 03:57:45 -0500

e found on this topic for me, I would like to appreciate your posting style, it’s really good, keep it up.

"Ruining your Test Automation Strategy." by Printing Services

Fri, 30 Apr 2010 07:43:35 -0500

This information is some of the best I have found on this topic for me, I would like to appreciate your posting style, it’s really good, keep it up.

"Ruining your Test Automation Strategy." by Audio Production

Sun, 25 Apr 2010 05:03:32 -0500

Very Interesting article and really had me thinking will make sure I check your posts more often!

"Ruining your Test Automation Strategy." by Endep

Wed, 17 Feb 2010 07:59:31 -0600

Great research! Thanks.

"Ruining your Test Automation Strategy." by digital camera brisbane

Wed, 27 Jan 2010 01:42:49 -0600

Great post! Thanks for the information

"Ruining your Test Automation Strategy." by Fabrício Lemos

Thu, 19 Nov 2009 13:06:32 -0600

“In the case of automated tests, if you change the GUI, the automated tests break”

And in case of unit tests, if you change the method signature, or move, or remove the method, the automated tests also breaks. Does this mean we shouldn´t do unit tests? I don´t think so.

I already worked on systems with great unit tests coverage and no system tests and also on systems with great system test coverage and no unit tests. And, in my experience, the cost of change was bigger with unit tests. Having to change a lot of tests just because of little changes at the api level was really making our refactoring more painful. And that is a bigger problem than making UI changes more painful.

I still prefer to focus on unit tests, but for a lot of other reasons than the “cost of change”, that could be really minimized on system tests with the use of Page Objects.

"Ruining your Test Automation Strategy." by Dean Cornish

Tue, 06 Oct 2009 20:12:28 -0500

Thanks for your thoughts as always UB :)

I do agree with parts of your thoughts, and disagree with others. I do see great value in testing via the GUI, the issue is how. The pain in updating the automation is the problem..

When you’re using recorder methods, or lots and lots of redundant code- that pain is amplified significantly.

Most automation systems I’ve seen- just took a nudge towards page object patterning to remove alot of this pain eg. dont change 40,000 tests with exactly the same line, extract it out and change it once to affect all 40,000, then extend and extend..

To me, its a fairly natural thing to do, but it isn’t always obvious to some. (eg. non-technicals & managers)

I firmly believe the old school testing approaches (unit -> system/component level testing -> integration testing) should still apply to your automation suite. I believe my views here correlate with yours thoughts on mocking and testing individual components. That’s a practice I agree with, its typically fast and gives great feedback- and doesn’t get killed by crazy integration points that seem to riddle so many projects out there today. Usually when I see Unit Testing -> Integration testing what I see is pain.

Thanks for your thoughts as always UB :)

"Ruining your Test Automation Strategy." by Nigel

Sun, 04 Oct 2009 18:37:42 -0500

If test scripts are well designed and with right tool, GUI automated testing is not only possible, and quite enjoyable experience.

http://www.infoq.com/articles/refactoring-test-scripts

We had problems as you described with GUI automated testing before.

"Ruining your Test Automation Strategy." by Emily Bache

Fri, 02 Oct 2009 14:52:47 -0500

Hi, I don’t think that GUI tests have to be as expensive and difficult to maintain as Bob makes out in this post. I think we should look beyond the popular tools for GUI testing, which I agree have limitations. I wrote my own blog post outlining my point of view:

http://emilybache.blogspot.com/2009/10/testing-through-gui-costs-less-with.html

"Ruining your Test Automation Strategy." by Vijay Nathani

Thu, 01 Oct 2009 19:07:46 -0500

I agree 100% with the contents of this blog. I prefer most tests to be Unit Tests written in xUnit. Integration and System tests are also usually in xUnit. Acceptance testing is in Fitnesse or Concordion. The purpose of GUI testing is only to ensure that values are correctly passed and received. It should never be used to test the business logic.

"Ruining your Test Automation Strategy." by perottobc

Thu, 01 Oct 2009 16:36:13 -0500

But Uncle Bob, seriously: No huge burden of test maintenance?

I think Curtis Cooley hits the nail spot on: fragility is the number one concern with automated testing in general. API-testing is a much better concept than gui testing, but there are no silver bullets.

Philip Schwarz points out that larger projects can easily have hundreds or thousands of acceptance tests. I would like to see this with no huge burden of test maintenance, especially with FitNesse out of the box!

It is hopeless?

Of course not, just like you have to design your system to be testable you have to realize that FitNesse has some missing parts, that your project have to figure out how to solve. Failure to do so will give your project a huge burden of test maintenance.

So, what are the missing parts? For the Java implementation I would suggest:

Maven integration (real programmers use maven)
Version control system integration (real projects use version controll)
A user friendly GUI (real testers deserve a decent gui)

Of course, it is open source, so we could all fix it. Projects like richness and trinidad gives hope, but having FitNesse integrated with your environment is no easy task.

Instead I’m more inspired by the a tool well integrated with my development environment, JUnit. Of course it can’t compete with a web gui for interactive testing, but it can compete on total cost of ownership. For me, putting 100% of the tests in FitNesse sounds like an expensive experiment. Especially since some essential parts are missing, and you seldom get it right the first time. To help out JUnit with it’s communication problem I have developed bdoc (bdoc.googlecode.com), that creates documentation out of tests. It would be interesting to get comments on it, a lightweight approach to automatic acceptance testing.

I’m sorry for my battering on FitNesse, but I feel that it is worth pointing out that any tool needs to be used with care an common sense. A test automation strategy that puts 100% of the tests in FitNesse could be the same as Ruining your Test Automation Strategy, and end up with a huge burden of test maintenance!

"Ruining your Test Automation Strategy." by Chuck vdL

Thu, 01 Oct 2009 14:43:11 -0500

Harold,

If you have a good accessable ‘api’ just under the UI skin, then you can drive most ‘end to end’ behavior at that level, it’s faster, easier to test, and if you are using TDD/DbE/BDD then the developers are creating those tests to ensure that they are building the right code that does the right things. (and that later changes to the code don’t break that stuff)

At the UI level you then need to verify that 1) the proper controls exist on the right pages, in the right places, is visable to the user using the right graphics, etc. This is hard to automate without the test being fragile, as it requires a lot of ‘judgement calls’ 2) CRITICAL. Each and every one of those controls is hooked up to the proper thing just under the skin. (this can usually be automated). If testing as Bob is suggesting above, this is really the weakest link where a bug might creap in and be otherwise undetected. 3) some basic sanity checks that when driven from the UI, everhthing is working, all the buttons and knobs function, etc. (ditto)

The ‘wrongness’ of driving all your automated testing at the GUI level comes in multiple forms. 1) if that’s ALL your automated tests, then you are missing unit tests, which provide tremendous value to developers on so many levels that there’s no way for me to cover it in a simple comment. 2) there’s no fast simple way for a develper that just refactored some code to determine if it’s broken without fully compiling everything and running the exaustive UI tests 3) The UI is slow, so the tests take a lot longer to run. Which means you can’t usually do them as as part of CI, but instead have to do it as some long overnight process. 4) the tests tend to be fragile, and small tweaks to the UI can break a lot of tests, requiring a lot of work to maintain. This ties up testers and QA folks doing test maintance of non-bug test breaks instead of doing other testing that could be finding bugs.

"Ruining your Test Automation Strategy." by Esko Luontola

Thu, 01 Oct 2009 14:38:57 -0500

Fredrik Bertilsson: When writing unit tests my experience is that the time for writing (or modifying) test cases are about the same time, or more, as writing the code itself.

My experience is that writing a test and a code to pass the test, takes almost the same time as thinking about what code to write and then writing the code. Writing a test is all about thinking what you want, before deciding how to do it.

Documenting as a formal executable specification (also known as “a test”) that what the system should do, does not slow you down because you need to type about twice as much text. If it slows you down, it’s because you need to think before acting. Just hacking some code together without that thinking is faster only in the short term (i.e. about 15-60 minutes, until you need to change the code you wrote), and it produces much worse quality.

Fredrik Bertilsson: Regarding the statement that testing the GUI is only 5-10% of the total testing time, I could only be true for software that lacks any substantial GUI.

If testing the GUI takes more than 5-10% of all tests, it could only be true if there is barely anything behind the GUI.

Also keep in mind that the logic of the GUI should not be part of the GUI. It should be a layer which is completely decoupled from the GUI using a suitable design pattern (http://martinfowler.com/eaaDev/ModelViewPresenter.html). In a well designed application, the GUI layer will consist only of GUI library components and simple glue code. If the GUI layer contains complex logic, then the design is wrong.

Harold: And I still don’t understand why automated tests through the GUI are wrong, you’re not giving much evidence for this point (and as a matter of fact, the company I’m working for is doing fine with 100% of our UI extensively tested every night).

So you test the GUI every night? Why is it not tested every time that even a single line of code is changed? If the developers hesitate to run all tests after changing a single line of code, then the tests are too slow (http://agileinaflash.blogspot.com/2009/02/first.html).

Fast feedback is one of the big benefits of TDD and a good unit test suite. You can change a line, run all tests, and know immediately whether your change broke anything. It totally changes the way that you can approach development.

Fear is the path to the dark side. Tests remove the fear. You can change things without fear of breaking things, because you will find out if anything was broken in a few seconds. You can even write some nasty algorithm with nested loops and conditionals by just trying every possible way to write it (if there are only a couple dozen alternatives), until one of them passes all tests, and after that you can digest the algorithm that you wrote and try to understand why it works.

Lately I’ve been dreaming about the near future, when each workstation will have tens or hundreds of CPU cores (after 5-10 years?). Because the available processing power will multiply faster than the number of tests we write per project, sooner or later it will become feasible to run all tests between every keystroke. I wonder how that fast feedback will change the way we write code… :)

"Ruining your Test Automation Strategy." by Marc

Thu, 01 Oct 2009 10:39:38 -0500

Harold:

> Unit tests can pass and yet your product can still have functional bugs.

So what? It sounds like you just discovered a truth known for thousands of years, “To err is human.”

> Not having functional tests for 100% of your functionalities still sounds like a bad idea.

Where did you get the idea that anyone here is advocating not testing 100% of your functionallity? All Uncle Bob has said is that out of the totality of your automated tests (that is, your unit tests, acceptance tests, gui tests, etc) roughly 10-15% should hit the gui. The other 85-90% covers the non-gui functionallity.

> And I still don’t understand why automated tests through the GUI are wrong, you’re not giving much evidence for this point (and as a matter of fact, the company I’m working for is doing fine with 100% of our UI extensively tested every night).

Um, you are commenting in a post encouraging testing the gui; where do you get the idea that anyone things it is “wrong” to do so? All Uncle Bob points out is that there is a cost to doing so, and offers some recomendations to help make it easier and more effective.

"Ruining your Test Automation Strategy." by Fredrik Bertilsson

Thu, 01 Oct 2009 00:01:40 -0500

Most shops I have been working for have a ratio of 1/4 to 1/2 between (manual) testers and programmers. One tester tested the code for two to four programmers. When writing unit tests my experience is that the time for writing (or modifying) test cases are about the same time, or more, as writing the code itself. And you would still need manual testers for stuff like GUI etc, that unit tests cant handle.

Accordning these metrics you would for example need 4+2 members in the manual case compared to 8+ members in the automatic case. If unit tests should save any time, the metrics must be much different from mine, or? Regarding the statement that testing the GUI is only 5-10% of the total testing time, I could only be true for software that lacks any substantial GUI. In normal enterprise application, I could not imagine that the GUI part of the application is so small.

"Ruining your Test Automation Strategy." by Maria Marcano

Wed, 30 Sep 2009 20:35:37 -0500

Very interesting topic, I have the following questions:

About this statement “Indeed, in a truly decoupled system most GUI tests seem absurd.” I understand that we can guarantee that components work well isolated but do you think we can guarantee that all pieces are working well together without having a similar coupling issue?. How would you test integration?. Many code I’ve seen are too coupled to have replaceable pieces to use during tests, how can we live with that?.

Does this makes sense to you?

• Have a test collection (classes and methods) that states how the application must work in a High level (how major business process should work, like stating the requirements)

? Major requirements may use more detailed pieces of the applications. For example, the specific GUI being used will be injected to the test collection that runs and verify different business processes (Test Collection doesn’t know about the details of the GUI)

• Regarding to the GUI it is expressed in objects and provide services to the users (if a particular part of the UI changes, we make the change in only one specific place)

I am going to begin doing more research about the last point (hopefully start blogging about this idea I have), and I am interested in having your opinion.

Thanks,

"Ruining your Test Automation Strategy." by Harold

Wed, 30 Sep 2009 18:12:29 -0500

Unit tests can pass and yet your product can still have functional bugs. Not having functional tests for 100% of your functionalities still sounds like a bad idea.

And I still don’t understand why automated tests through the GUI are wrong, you’re not giving much evidence for this point (and as a matter of fact, the company I’m working for is doing fine with 100% of our UI extensively tested every night).

"Ruining your Test Automation Strategy." by Ian4101

Wed, 30 Sep 2009 17:56:32 -0500

@Harold, That’s not quite what Bob is suggesting by any means. I think you are suggesting a black box approach to the problem and in practice a slightly transparent box approach is warranted. Bob hasn’t accounted for the other 85-90% – which is ok, that’s not the point of the article. I am curious whether there are metrics for business logic tests and persistence logic tests; tests on the db schema for example.

"Ruining your Test Automation Strategy." by Uncle Bob

Wed, 30 Sep 2009 17:02:47 -0500

Harold about 23 hours later:
10% of tests will not be enough to test all the functions that your users will be using, so are you basically recommending to ship software that hasn’t been tested?

No. I’m suggesting you test 5%-10% through the GUI. Test 100% through APIs. Test 100% again with unit tests.

"Ruining your Test Automation Strategy." by Anthony Bailey

Wed, 30 Sep 2009 16:06:59 -0500

About five years ago I worked on a well-tested application that was mostly driven through a direct manipulation user interface; a substantial part of the value of the application was in the way that the direct manipulation of rendered objects meshed tightly to the underlying domain. (It was a 3D medical visualization app; think of it as PhotoShop with lots of meaning to the pictures.)

We tested in a wide variety of ways. My house of testing has many rooms and I would keep some of every flavor of test, but the tests that proved to have the most value were, oh guess what, those that tested the app just beneath the skin, hooking into careful abstractions of input events and representations of graphics on the screen.

Of course testing GUI code can be worthwhile. But I would not test the domain logic through the GUI, nor test the GUI through its results on the domain. I advocate letting the tests get under your skin and crafting a layer primarily for testing purposes. (And being only slightly surprised when in the usual TDDesign fashion that code turns out to have other production clients too, such as the contextual help system, macro recording, etc.)