Ruining your Test Automation Strategy. 156
Everybody wants a test automation strategy nowadays. The reason is clear. It take a lot of effort to run a suite of manual tests; and that effort has be be repeated several times per year.
Consider the fate of this poor customer of mine. The picture you see here is simply the Table of Contents of his manual test plan. That plan has tens of thousands of individual manual tests. It costs them millions of dollars to execute, and they must execute it many times each year.
To make matters worse, in these difficult times, management has told the QA manager that he must cut 50% from his manual testing budget. The question he asked me was: “Which 40,000 of these 80,000 tests should I delete?”
So, as you can probably imagine, this poor guy really wished his tests were automated. Runnning automated tests does not cost six figures several times a year. Running automated tests does not need to be cut in half when budgets get tight. Running automated tests are the way to go.
One common strategy to get your tests automated is to outsource the problem. You hire some team of test writers to transform your manual tests into automated tests using some automation tool. These folks execute the manual test plan while setting up the automation tool to record their actions. Then the tool can simply play the actions back for each new release of the system; and make sure the screens don’t change.
Sounds great doesn’t it? Sounds like just the ticket! Sounds like a simple way to convert manual tests into automated tests!
Yeah, and it’s a sure fire way to make sure you utterly ruin your strategy for test automation!...
Why is this so runious?
Automatically testing a system through the GUI couples the tests to the GUI. That’s right, this is a coupling problem! A standard, every-day, good ol’ software design coupling problem. And the problem with coupling is that when two things are coupled, and you change one of them, the other one breaks.
In the case of automated tests, if you change the GUI, the automated tests break. This was a big enough problem when the test were manual. You always had to go back and edit the test plans to take GUI changes into account. Fortunately, since the test were manual, you had human beings in the loop, and you could expect them to use common sense about simple GUI changes. The fact that what they saw on the screen differed from what they test plan said they should see would not confound them so long as they understood how the GUI had changed.
But an automated tool doesn’t have that kind of reasoning power. If the automated tool sees anything different from what it expects, it simply fails.
Now, clearly, automated tools can be made to be clever enough to avoid simple cosmetic issues like the moving of a button, or a change in the spelling of a menu. But you have to work at making the tests tolerant of such changes. Do you think that outsourced team of test writers care much about that?
In any case, no tool can deal with changes to the navigation structure, or gross changes to the structure of screens. Thus, there will always be a class of GUI changes that will cause huge swaths of automated tests to fail. Unfortunately these kinds of changes are all too common. What’s more the cost of re-recording the tests is high, and the re-recording process itself is error-prone.
The net result is that GUI driven automated tests are fragile, and the process of maintaining them is expensive and unreliable.
To drive this point home, consider the fate of an old client of mine who had tens of thousands of automated tests driven through the GUI. Every time anyone changed the GUI a thousand or so tests broke. The burden of maintaining those tests was so great that this customer added one restrictive policy after another in an effort to prevent changes to the GUI. In the end, GUI changes became officially prohibited. (This was a DOS GUI in the era of Windows!)
Or consider another client who spent a great deal of time and effort creating automated tests through the GUI. This client could not constrain the GUI against change, so had to live with the burden of re-recording the tests. But the burden was too great. Trade-offs were made. Many of the tests de-prioritized, and therefore lost. Bit by bit this customer lost his investment in automated tests.
The bottom line is that automated tests through the GUI are inherently unstable, and will drive you to one or the other of those two undesirable states.
It is hopeless?
Not at all. First of all, you need some acceptance tests that go through the GUI. I think that number is on the order of 5%. These are integration tests that make sure that the whole system is wired up correctly.
You also need to test the GUI through the GUI, and this might be another 5-10% of the total body of tests. What does it mean to test the GUI? It means that you aren’t testing any business rules! You test the GUI and only the GUI. My favorite scheme for doing this is to mock out all the business rule code entirely, replacing it with a dummy that simply provides canned answers to all the GUI interfaces. Imagine, for example, replacing all the servlets in a web system with dummy servlets that pay no attention to their inputs, use no database at all, and simply return canned HTML.
By the same token, the business rule tests should not use the GUI. In fact, the GUI shouldn’t even be turned on. Rather, your tests should invoke the same business rule APIs that the GUI invokes. Indeed, the tests become an alternative UI.
Preferably the Business Rule API is well below the GUI layer. In a web system it should be below the servlet layer. In the best designed systems, it can even be outside the container.
Tools like FitNesse, Cucumber, RobotFX, and GreenPepper are designed for just this kind of API testing.
Tests that drive the system through the API are entirely decoupled from the GUI. When the GUI changes, these tests are completely unaffected. There is no huge burden of test maintenance, and you are not at risk for losing your tests or constraining the GUI.
Summary
Automated testing through the GUI is intuitive, seductive, and almost always wrong! A dedicated program of automatically testing your system through the GUI is almost certain to fail. If you want a robust test automation strategy, plan to use a healthy dose of API testing, and as little GUI testing as possible.