Ruining your Test Automation Strategy. 28
Everybody wants a test automation strategy nowadays. The reason is clear. It take a lot of effort to run a suite of manual tests; and that effort has be be repeated several times per year.
Consider the fate of this poor customer of mine. The picture you see here is simply the Table of Contents of his manual test plan. That plan has tens of thousands of individual manual tests. It costs them millions of dollars to execute, and they must execute it many times each year.
To make matters worse, in these difficult times, management has told the QA manager that he must cut 50% from his manual testing budget. The question he asked me was: “Which 40,000 of these 80,000 tests should I delete?”
So, as you can probably imagine, this poor guy really wished his tests were automated. Runnning automated tests does not cost six figures several times a year. Running automated tests does not need to be cut in half when budgets get tight. Running automated tests are the way to go.
One common strategy to get your tests automated is to outsource the problem. You hire some team of test writers to transform your manual tests into automated tests using some automation tool. These folks execute the manual test plan while setting up the automation tool to record their actions. Then the tool can simply play the actions back for each new release of the system; and make sure the screens don’t change.
Sounds great doesn’t it? Sounds like just the ticket! Sounds like a simple way to convert manual tests into automated tests!
Yeah, and it’s a sure fire way to make sure you utterly ruin your strategy for test automation!...
Why is this so runious?
Automatically testing a system through the GUI couples the tests to the GUI. That’s right, this is a coupling problem! A standard, every-day, good ol’ software design coupling problem. And the problem with coupling is that when two things are coupled, and you change one of them, the other one breaks.
In the case of automated tests, if you change the GUI, the automated tests break. This was a big enough problem when the test were manual. You always had to go back and edit the test plans to take GUI changes into account. Fortunately, since the test were manual, you had human beings in the loop, and you could expect them to use common sense about simple GUI changes. The fact that what they saw on the screen differed from what they test plan said they should see would not confound them so long as they understood how the GUI had changed.
But an automated tool doesn’t have that kind of reasoning power. If the automated tool sees anything different from what it expects, it simply fails.
Now, clearly, automated tools can be made to be clever enough to avoid simple cosmetic issues like the moving of a button, or a change in the spelling of a menu. But you have to work at making the tests tolerant of such changes. Do you think that outsourced team of test writers care much about that?
In any case, no tool can deal with changes to the navigation structure, or gross changes to the structure of screens. Thus, there will always be a class of GUI changes that will cause huge swaths of automated tests to fail. Unfortunately these kinds of changes are all too common. What’s more the cost of re-recording the tests is high, and the re-recording process itself is error-prone.
The net result is that GUI driven automated tests are fragile, and the process of maintaining them is expensive and unreliable.
To drive this point home, consider the fate of an old client of mine who had tens of thousands of automated tests driven through the GUI. Every time anyone changed the GUI a thousand or so tests broke. The burden of maintaining those tests was so great that this customer added one restrictive policy after another in an effort to prevent changes to the GUI. In the end, GUI changes became officially prohibited. (This was a DOS GUI in the era of Windows!)
Or consider another client who spent a great deal of time and effort creating automated tests through the GUI. This client could not constrain the GUI against change, so had to live with the burden of re-recording the tests. But the burden was too great. Trade-offs were made. Many of the tests de-prioritized, and therefore lost. Bit by bit this customer lost his investment in automated tests.
The bottom line is that automated tests through the GUI are inherently unstable, and will drive you to one or the other of those two undesirable states.
It is hopeless?
Not at all. First of all, you need some acceptance tests that go through the GUI. I think that number is on the order of 5%. These are integration tests that make sure that the whole system is wired up correctly.
You also need to test the GUI through the GUI, and this might be another 5-10% of the total body of tests. What does it mean to test the GUI? It means that you aren’t testing any business rules! You test the GUI and only the GUI. My favorite scheme for doing this is to mock out all the business rule code entirely, replacing it with a dummy that simply provides canned answers to all the GUI interfaces. Imagine, for example, replacing all the servlets in a web system with dummy servlets that pay no attention to their inputs, use no database at all, and simply return canned HTML.
By the same token, the business rule tests should not use the GUI. In fact, the GUI shouldn’t even be turned on. Rather, your tests should invoke the same business rule APIs that the GUI invokes. Indeed, the tests become an alternative UI.
Preferably the Business Rule API is well below the GUI layer. In a web system it should be below the servlet layer. In the best designed systems, it can even be outside the container.
Tools like FitNesse, Cucumber, RobotFX, and GreenPepper are designed for just this kind of API testing.
Tests that drive the system through the API are entirely decoupled from the GUI. When the GUI changes, these tests are completely unaffected. There is no huge burden of test maintenance, and you are not at risk for losing your tests or constraining the GUI.
Summary
Automated testing through the GUI is intuitive, seductive, and almost always wrong! A dedicated program of automatically testing your system through the GUI is almost certain to fail. If you want a robust test automation strategy, plan to use a healthy dose of API testing, and as little GUI testing as possible.
Very interesting, thanks for the info…....
Not that I disagree with you, but what about designing your GUI tests with the same care with which you design your code. When you go to get a certain screen for testing, you go through, say, a factory. All your GUI test code interfacing is done through a set of factories, so when the GUI changes, a thousand tests break, but you change one class and a thousand tests now pass.
Test code must be as well designed as your production code. Just as much care and time and effort and budget should be put towards testing. Otherwise you are putting yourself in a mode where you’re always chasing the rabbit but will never catch it.
In addition to pure GUI tests, the number of system integration tests need to be kept to an absolute minimum. Even if you don’t use record and playback, these kinds of tests will be brittle (and slow). Faster, yes, than manual testing…but still so slow that you wont run them very often.
Effective unit testing is step one in any test automation strategy.
In my opinion GUI tests can hardly be worth their time to construct because of a low ROI. Even with the business layer faked out they can still be pretty fragile, and if you’ve made the dire mistake of running them in CI you can expect CI to be red several times per week even though the code base is 100% defect free and super clean.
I like to view automated GUI tests as not tests, but automated demos. In the long run they don’t provide as much value as the tests at the business layers.
I’d also like to mention JBehave and Concordion to the list of story level specification frameworks that can be used to describe business logic. I’ve really liked Concordion, but it always seems to get snubbed when story specification frameworks are mentioned. ;)
About acceptance tests that drive the system through the API rather than through the GUI: in Test-Driven – Practical TDD and Acceptance TDD for Java Developers, Lasse Koskela says that these tests crawl under the system’s skin (the UI), so he calls their usage ‘crawling under the skin’, or ‘skinning the application’.
On a project that uses the Acceptance TDD (ATDD) cycle
acceptance tests that skin the application are important because they are often up to an order of magnitude faster than tests that don’t bypass the UI. As Lasse says in his book:
Running tests beneath the GUI requires investment in the application code to make this happen. For new applications you can make this investment off the bat. For legacy code, the chances are that the application has not been designed this way.
For legacy applications, I think it is a worthwhile investment to replace manual rote testing with automated tests through the GUI, even though such tests are brittle. You just have to be aware that they are brittle, and when you change the GUI then you change the infrastructure to make it easier to test the logic beneath the GUI and the GUI itself separately. If you keep up with this continual investment then the automated tests will become less brittle over time.
Anthony, wouldn’t it be better to invest all this effort in improving your code base to enable ‘skinning the application’, instead of building elaborate crutches to make GUI tests less brittle? This would not only result in faster tests but also in cleaner, more well-designed production code.
I definitely agree with your opinion, Bob. I think one has to distinguish precisely between acceptance tests (which are for your customer) and developer tests (which help your programmers tracking bugs down to their root cause). I want to add a source, that persuaded me deeply about this necessary difference and how one should cope with the consequences. J. B. Rainsberger spoke about it at Agile 2009. One can watch the video at http://www.infoq.com/presentations/integration-tests-scam. Very insightful for me.
Absolutely! Systems that are designed to be tested are much much easier to test. This makes it much easier to write both API tests, and GUI tests, and reduces the coupling problem.
I think you’ll find, however, that the more you design your systems to be testable, the more you will naturally concentrate on API tests as opposed to GUI tests. This is because it is far easier to understand API tests in a well decoupled system. Indeed, in a truly decoupled system most GUI tests seem absurd. When you’ve got a nicely isolated component with a clear and focussed interface why would you test it through some arcane GUI?
If you follow my suggestions, that’s essentially what you get.
Step 0: Manual rote testing through GUI Step 1: Automated rote testing through GUI Step 2a: Modify feature X which impacts GUI Step 2b: Ensure GUI for feature X cleanly separated from backend (i.e. allow skinning for feature X) Step 2c: Automated rote testing of everything but feature X through GUI Step 2d: Automated rote testing of GUI for feature X independent of logic Step 2e: Automated rote testing of feature X without GUI Step 3: Repeat for feature Y
Eventually your application is entirely skinnable, apart from those features you haven’t touched the GUI for, and in the mean time the automated tests through the GUI provide a safety net when refactoring the app to allow such skinning. You can of course opt to make the remaining features skinnable too, but you don’t have to.
Typically, manual rote testing is too expensive and time consuming for it to provide the necessary safety net for refactoring a non-skinnable app into a skinnable one.
I absolutely agree, but I’ve been bitten by fragile tests, and it seems that fragility is the number one concern with automated testing in general. We have ways of limiting fragility in production code, but I’ve not seen anyone specifically suggest that we use the same techniques, patterns and idioms we use to limit fragility in production code to limit the fragility in test code. So I said it ;)
10% of tests will not be enough to test all the functions that your users will be using, so are you basically recommending to ship software that hasn’t been tested?
This is shocking.
Every single functionality that users will be using needs to have at least one automated test, and if this functionality happens to be driven by a GUI, then you should be testing the GUI.
I don’t understand why GUI’s should be getting a waiver when it comes to automated testing, and I don’t understand why GUI tests are inherently less stable than other kinds of tests. This makes no sense to me.
About five years ago I worked on a well-tested application that was mostly driven through a direct manipulation user interface; a substantial part of the value of the application was in the way that the direct manipulation of rendered objects meshed tightly to the underlying domain. (It was a 3D medical visualization app; think of it as PhotoShop with lots of meaning to the pictures.)
We tested in a wide variety of ways. My house of testing has many rooms and I would keep some of every flavor of test, but the tests that proved to have the most value were, oh guess what, those that tested the app just beneath the skin, hooking into careful abstractions of input events and representations of graphics on the screen.
Of course testing GUI code can be worthwhile. But I would not test the domain logic through the GUI, nor test the GUI through its results on the domain. I advocate letting the tests get under your skin and crafting a layer primarily for testing purposes. (And being only slightly surprised when in the usual TDDesign fashion that code turns out to have other production clients too, such as the contextual help system, macro recording, etc.)
No. I’m suggesting you test 5%-10% through the GUI. Test 100% through APIs. Test 100% again with unit tests.
@Harold, That’s not quite what Bob is suggesting by any means. I think you are suggesting a black box approach to the problem and in practice a slightly transparent box approach is warranted. Bob hasn’t accounted for the other 85-90% – which is ok, that’s not the point of the article. I am curious whether there are metrics for business logic tests and persistence logic tests; tests on the db schema for example.
Unit tests can pass and yet your product can still have functional bugs. Not having functional tests for 100% of your functionalities still sounds like a bad idea.
And I still don’t understand why automated tests through the GUI are wrong, you’re not giving much evidence for this point (and as a matter of fact, the company I’m working for is doing fine with 100% of our UI extensively tested every night).
Very interesting topic, I have the following questions:
About this statement “Indeed, in a truly decoupled system most GUI tests seem absurd.” I understand that we can guarantee that components work well isolated but do you think we can guarantee that all pieces are working well together without having a similar coupling issue?. How would you test integration?. Many code I’ve seen are too coupled to have replaceable pieces to use during tests, how can we live with that?.
Does this makes sense to you?
• Have a test collection (classes and methods) that states how the application must work in a High level (how major business process should work, like stating the requirements)
? Major requirements may use more detailed pieces of the applications. For example, the specific GUI being used will be injected to the test collection that runs and verify different business processes (Test Collection doesn’t know about the details of the GUI)
• Regarding to the GUI it is expressed in objects and provide services to the users (if a particular part of the UI changes, we make the change in only one specific place)
I am going to begin doing more research about the last point (hopefully start blogging about this idea I have), and I am interested in having your opinion.
Thanks,
Most shops I have been working for have a ratio of 1/4 to 1/2 between (manual) testers and programmers. One tester tested the code for two to four programmers. When writing unit tests my experience is that the time for writing (or modifying) test cases are about the same time, or more, as writing the code itself. And you would still need manual testers for stuff like GUI etc, that unit tests cant handle.
Accordning these metrics you would for example need 4+2 members in the manual case compared to 8+ members in the automatic case. If unit tests should save any time, the metrics must be much different from mine, or? Regarding the statement that testing the GUI is only 5-10% of the total testing time, I could only be true for software that lacks any substantial GUI. In normal enterprise application, I could not imagine that the GUI part of the application is so small.
Harold:
> Unit tests can pass and yet your product can still have functional bugs.
So what? It sounds like you just discovered a truth known for thousands of years, “To err is human.”
> Not having functional tests for 100% of your functionalities still sounds like a bad idea.
Where did you get the idea that anyone here is advocating not testing 100% of your functionallity? All Uncle Bob has said is that out of the totality of your automated tests (that is, your unit tests, acceptance tests, gui tests, etc) roughly 10-15% should hit the gui. The other 85-90% covers the non-gui functionallity.
> And I still don’t understand why automated tests through the GUI are wrong, you’re not giving much evidence for this point (and as a matter of fact, the company I’m working for is doing fine with 100% of our UI extensively tested every night).
Um, you are commenting in a post encouraging testing the gui; where do you get the idea that anyone things it is “wrong” to do so? All Uncle Bob points out is that there is a cost to doing so, and offers some recomendations to help make it easier and more effective.
My experience is that writing a test and a code to pass the test, takes almost the same time as thinking about what code to write and then writing the code. Writing a test is all about thinking what you want, before deciding how to do it.
Documenting as a formal executable specification (also known as “a test”) that what the system should do, does not slow you down because you need to type about twice as much text. If it slows you down, it’s because you need to think before acting. Just hacking some code together without that thinking is faster only in the short term (i.e. about 15-60 minutes, until you need to change the code you wrote), and it produces much worse quality.
If testing the GUI takes more than 5-10% of all tests, it could only be true if there is barely anything behind the GUI.
Also keep in mind that the logic of the GUI should not be part of the GUI. It should be a layer which is completely decoupled from the GUI using a suitable design pattern (http://martinfowler.com/eaaDev/ModelViewPresenter.html). In a well designed application, the GUI layer will consist only of GUI library components and simple glue code. If the GUI layer contains complex logic, then the design is wrong.
So you test the GUI every night? Why is it not tested every time that even a single line of code is changed? If the developers hesitate to run all tests after changing a single line of code, then the tests are too slow (http://agileinaflash.blogspot.com/2009/02/first.html).
Fast feedback is one of the big benefits of TDD and a good unit test suite. You can change a line, run all tests, and know immediately whether your change broke anything. It totally changes the way that you can approach development.
Fear is the path to the dark side. Tests remove the fear. You can change things without fear of breaking things, because you will find out if anything was broken in a few seconds. You can even write some nasty algorithm with nested loops and conditionals by just trying every possible way to write it (if there are only a couple dozen alternatives), until one of them passes all tests, and after that you can digest the algorithm that you wrote and try to understand why it works.
Lately I’ve been dreaming about the near future, when each workstation will have tens or hundreds of CPU cores (after 5-10 years?). Because the available processing power will multiply faster than the number of tests we write per project, sooner or later it will become feasible to run all tests between every keystroke. I wonder how that fast feedback will change the way we write code… :)
Harold,
If you have a good accessable ‘api’ just under the UI skin, then you can drive most ‘end to end’ behavior at that level, it’s faster, easier to test, and if you are using TDD/DbE/BDD then the developers are creating those tests to ensure that they are building the right code that does the right things. (and that later changes to the code don’t break that stuff)
At the UI level you then need to verify that 1) the proper controls exist on the right pages, in the right places, is visable to the user using the right graphics, etc. This is hard to automate without the test being fragile, as it requires a lot of ‘judgement calls’ 2) CRITICAL. Each and every one of those controls is hooked up to the proper thing just under the skin. (this can usually be automated). If testing as Bob is suggesting above, this is really the weakest link where a bug might creap in and be otherwise undetected. 3) some basic sanity checks that when driven from the UI, everhthing is working, all the buttons and knobs function, etc. (ditto)
The ‘wrongness’ of driving all your automated testing at the GUI level comes in multiple forms. 1) if that’s ALL your automated tests, then you are missing unit tests, which provide tremendous value to developers on so many levels that there’s no way for me to cover it in a simple comment. 2) there’s no fast simple way for a develper that just refactored some code to determine if it’s broken without fully compiling everything and running the exaustive UI tests 3) The UI is slow, so the tests take a lot longer to run. Which means you can’t usually do them as as part of CI, but instead have to do it as some long overnight process. 4) the tests tend to be fragile, and small tweaks to the UI can break a lot of tests, requiring a lot of work to maintain. This ties up testers and QA folks doing test maintance of non-bug test breaks instead of doing other testing that could be finding bugs.
But Uncle Bob, seriously: No huge burden of test maintenance?
Great stuff about gui tests, use with care and common sense. But at the end of your post you emphasize API-testing, mentioning FitNesse (among others) as a tool with no huge burden of test maintenance.
I think Curtis Cooley hits the nail spot on: fragility is the number one concern with automated testing in general. API-testing is a much better concept than gui testing, but there are no silver bullets.
Philip Schwarz points out that larger projects can easily have hundreds or thousands of acceptance tests. I would like to see this with no huge burden of test maintenance, especially with FitNesse out of the box!
It is hopeless?
Of course not, just like you have to design your system to be testable you have to realize that FitNesse has some missing parts, that your project have to figure out how to solve. Failure to do so will give your project a huge burden of test maintenance.
So, what are the missing parts? For the Java implementation I would suggest:
Of course, it is open source, so we could all fix it. Projects like richness and trinidad gives hope, but having FitNesse integrated with your environment is no easy task.
Instead I’m more inspired by the a tool well integrated with my development environment, JUnit. Of course it can’t compete with a web gui for interactive testing, but it can compete on total cost of ownership. For me, putting 100% of the tests in FitNesse sounds like an expensive experiment. Especially since some essential parts are missing, and you seldom get it right the first time. To help out JUnit with it’s communication problem I have developed bdoc (bdoc.googlecode.com), that creates documentation out of tests. It would be interesting to get comments on it, a lightweight approach to automatic acceptance testing.
I’m sorry for my battering on FitNesse, but I feel that it is worth pointing out that any tool needs to be used with care an common sense. A test automation strategy that puts 100% of the tests in FitNesse could be the same as Ruining your Test Automation Strategy, and end up with a huge burden of test maintenance!
I agree 100% with the contents of this blog. I prefer most tests to be Unit Tests written in xUnit. Integration and System tests are also usually in xUnit. Acceptance testing is in Fitnesse or Concordion. The purpose of GUI testing is only to ensure that values are correctly passed and received. It should never be used to test the business logic.
Hi, I don’t think that GUI tests have to be as expensive and difficult to maintain as Bob makes out in this post. I think we should look beyond the popular tools for GUI testing, which I agree have limitations. I wrote my own blog post outlining my point of view:
http://emilybache.blogspot.com/2009/10/testing-through-gui-costs-less-with.html
If test scripts are well designed and with right tool, GUI automated testing is not only possible, and quite enjoyable experience.
http://www.infoq.com/articles/refactoring-test-scripts
We had problems as you described with GUI automated testing before.
Thanks for your thoughts as always UB :)
I do agree with parts of your thoughts, and disagree with others. I do see great value in testing via the GUI, the issue is how. The pain in updating the automation is the problem..
When you’re using recorder methods, or lots and lots of redundant code- that pain is amplified significantly.
Most automation systems I’ve seen- just took a nudge towards page object patterning to remove alot of this pain eg. dont change 40,000 tests with exactly the same line, extract it out and change it once to affect all 40,000, then extend and extend..
To me, its a fairly natural thing to do, but it isn’t always obvious to some. (eg. non-technicals & managers)
I firmly believe the old school testing approaches (unit -> system/component level testing -> integration testing) should still apply to your automation suite. I believe my views here correlate with yours thoughts on mocking and testing individual components. That’s a practice I agree with, its typically fast and gives great feedback- and doesn’t get killed by crazy integration points that seem to riddle so many projects out there today. Usually when I see Unit Testing -> Integration testing what I see is pain.
Thanks for your thoughts as always UB :)
“In the case of automated tests, if you change the GUI, the automated tests break”
And in case of unit tests, if you change the method signature, or move, or remove the method, the automated tests also breaks. Does this mean we shouldn´t do unit tests? I don´t think so.
I already worked on systems with great unit tests coverage and no system tests and also on systems with great system test coverage and no unit tests. And, in my experience, the cost of change was bigger with unit tests. Having to change a lot of tests just because of little changes at the api level was really making our refactoring more painful. And that is a bigger problem than making UI changes more painful.
I still prefer to focus on unit tests, but for a lot of other reasons than the “cost of change”, that could be really minimized on system tests with the use of Page Objects.
Great post! Thanks for the information