Object Mentor Blog: Ruining your Test Automation Strategy.

Ruining your Test Automation Strategy.

Uncle Bob — Tue, 29 Sep 2009 15:42:00 -0500

Everybody wants a test automation strategy nowadays. The reason is clear. It take a lot of effort to run a suite of manual tests; and that effort has be be repeated several times per year.

Consider the fate of this poor customer of mine. The picture you see here is simply the Table of Contents of his manual test plan. That plan has tens of thousands of individual manual tests. It costs them millions of dollars to execute, and they must execute it many times each year.

To make matters worse, in these difficult times, management has told the QA manager that he must cut 50% from his manual testing budget. The question he asked me was: “Which 40,000 of these 80,000 tests should I delete?”

So, as you can probably imagine, this poor guy really wished his tests were automated. Runnning automated tests does not cost six figures several times a year. Running automated tests does not need to be cut in half when budgets get tight. Running automated tests are the way to go.

One common strategy to get your tests automated is to outsource the problem. You hire some team of test writers to transform your manual tests into automated tests using some automation tool. These folks execute the manual test plan while setting up the automation tool to record their actions. Then the tool can simply play the actions back for each new release of the system; and make sure the screens don’t change.

Sounds great doesn’t it? Sounds like just the ticket! Sounds like a simple way to convert manual tests into automated tests!

Yeah, and it’s a sure fire way to make sure you utterly ruin your strategy for test automation!...

Why is this so runious?

Automatically testing a system through the GUI couples the tests to the GUI. That’s right, this is a coupling problem! A standard, every-day, good ol’ software design coupling problem. And the problem with coupling is that when two things are coupled, and you change one of them, the other one breaks.

In the case of automated tests, if you change the GUI, the automated tests break. This was a big enough problem when the test were manual. You always had to go back and edit the test plans to take GUI changes into account. Fortunately, since the test were manual, you had human beings in the loop, and you could expect them to use common sense about simple GUI changes. The fact that what they saw on the screen differed from what they test plan said they should see would not confound them so long as they understood how the GUI had changed.

But an automated tool doesn’t have that kind of reasoning power. If the automated tool sees anything different from what it expects, it simply fails.

Now, clearly, automated tools can be made to be clever enough to avoid simple cosmetic issues like the moving of a button, or a change in the spelling of a menu. But you have to work at making the tests tolerant of such changes. Do you think that outsourced team of test writers care much about that?

In any case, no tool can deal with changes to the navigation structure, or gross changes to the structure of screens. Thus, there will always be a class of GUI changes that will cause huge swaths of automated tests to fail. Unfortunately these kinds of changes are all too common. What’s more the cost of re-recording the tests is high, and the re-recording process itself is error-prone.

The net result is that GUI driven automated tests are fragile, and the process of maintaining them is expensive and unreliable.

To drive this point home, consider the fate of an old client of mine who had tens of thousands of automated tests driven through the GUI. Every time anyone changed the GUI a thousand or so tests broke. The burden of maintaining those tests was so great that this customer added one restrictive policy after another in an effort to prevent changes to the GUI. In the end, GUI changes became officially prohibited. (This was a DOS GUI in the era of Windows!)

Or consider another client who spent a great deal of time and effort creating automated tests through the GUI. This client could not constrain the GUI against change, so had to live with the burden of re-recording the tests. But the burden was too great. Trade-offs were made. Many of the tests de-prioritized, and therefore lost. Bit by bit this customer lost his investment in automated tests.

The bottom line is that automated tests through the GUI are inherently unstable, and will drive you to one or the other of those two undesirable states.

It is hopeless?

Not at all. First of all, you need some acceptance tests that go through the GUI. I think that number is on the order of 5%. These are integration tests that make sure that the whole system is wired up correctly.

You also need to test the GUI through the GUI, and this might be another 5-10% of the total body of tests. What does it mean to test the GUI? It means that you aren’t testing any business rules! You test the GUI and only the GUI. My favorite scheme for doing this is to mock out all the business rule code entirely, replacing it with a dummy that simply provides canned answers to all the GUI interfaces. Imagine, for example, replacing all the servlets in a web system with dummy servlets that pay no attention to their inputs, use no database at all, and simply return canned HTML.

By the same token, the business rule tests should not use the GUI. In fact, the GUI shouldn’t even be turned on. Rather, your tests should invoke the same business rule APIs that the GUI invokes. Indeed, the tests become an alternative UI.

Preferably the Business Rule API is well below the GUI layer. In a web system it should be below the servlet layer. In the best designed systems, it can even be outside the container.

Tools like FitNesse, Cucumber, RobotFX, and GreenPepper are designed for just this kind of API testing.

Tests that drive the system through the API are entirely decoupled from the GUI. When the GUI changes, these tests are completely unaffected. There is no huge burden of test maintenance, and you are not at risk for losing your tests or constraining the GUI.

Summary

Automated testing through the GUI is intuitive, seductive, and almost always wrong! A dedicated program of automatically testing your system through the GUI is almost certain to fail. If you want a robust test automation strategy, plan to use a healthy dose of API testing, and as little GUI testing as possible.

"Ruining your Test Automation Strategy." by Steve Waters Vancouver

Wed, 25 Jan 2012 22:03:15 -0600

It’s random act of kindness week, have you done yours?

"Ruining your Test Automation Strategy." by reclaim care home fees

Mon, 23 Jan 2012 23:19:27 -0600

Congratulations for posting such a useful blog. Your blog isn?t only informative but also extremely artistic too. There usually are extremely couple of individuals who can write not so easy articles that creatively. Keep up the good writing !!

"Ruining your Test Automation Strategy." by winwindeals

Sun, 22 Jan 2012 06:30:20 -0600

thanks for this great blog!! I enjoy reading your articles!!

"Ruining your Test Automation Strategy." by deals

Sun, 22 Jan 2012 06:23:34 -0600

Great post and great blog!! thanks!

"Ruining your Test Automation Strategy." by reclaim care home fees

Mon, 16 Jan 2012 21:58:18 -0600

Substantially, the write-up is truly the very best on this laudable subject. I concur inside your conclusions and will eagerly seem forward to your future updates.

"Ruining your Test Automation Strategy." by Polo Ralph Lauren Pas Cher

Thu, 12 Jan 2012 20:01:44 -0600

Thanks for this beautiful website. I have enjoyed reading through a few of the articles.

"Ruining your Test Automation Strategy." by reclaim care home fees

Tue, 10 Jan 2012 20:10:03 -0600

I genuinely liked reading by means of your post! You’ve great high quality material. I may well advise you to come up with weblog posts much more often.

"Ruining your Test Automation Strategy." by Louis Vuitton sale online

Tue, 10 Jan 2012 02:07:28 -0600

It’s useful to study your blog. The information of your site is precisely excellent, and your blog design is Simple good.

"Ruining your Test Automation Strategy." by good jokes

Sun, 08 Jan 2012 02:09:39 -0600

Great article. Thanks for the detailed post!

"Ruining your Test Automation Strategy." by http://www.monsterbeatsfactory.com

Fri, 30 Dec 2011 06:49:29 -0600

The new white version of the flagship fashion defeated by the Dr. Dre Studio Headphones represents a continuing commitment to the personal audio space design and good technical level is high. Monster Beats Dr. Brenner. Dre, Jimmy Iovine and Monster co-founded in 2008, Dr. monster beats by dr. dre, to provide consumers with a new headset design new level of audio performance and extreme clarity, deep bass and full power which had been lacking in conventional headphones. http://www.monsterbeatsfactory.com

"Ruining your Test Automation Strategy." by mac cosmetics

Sun, 18 Dec 2011 18:52:20 -0600

mac cosmetics oakley active sunglasses wholesale mac cosmetics Oakley Sport Sunglasses mac lipstick Yanggengsheng

"Ruining your Test Automation Strategy." by tv show

Sat, 26 Nov 2011 09:51:32 -0600

very good article. thanks a alot!

Keep up the good work!

"Ruining your Test Automation Strategy." by Sac Gucci

Thu, 24 Nov 2011 20:26:25 -0600

Conception unique de la Sac Gucci est très sympa,Sac Gucci Soldes sont de qualité fiable et à prix raisonnable.Sac Gucci Pas Cher a été considéré comme un Gucci Soldes signe de tendances de la mode et des Gucci Sac a main sélection symbole de statut social,Gucci Soldes online sélectionnez en ligne un Gucci Pas Cher, facile d’avoir un Sac a Main Gucci Soldes 2011 populaire.Oui,désormais tomber en amour avec Sac Gucci Guccissima.

"Ruining your Test Automation Strategy." by pligg

Tue, 22 Nov 2011 13:17:05 -0600

Great post! Thanks for the information…

"Ruining your Test Automation Strategy." by beautyjewelry.org

Wed, 16 Nov 2011 20:44:56 -0600

Thanks for the shopingpandora in this website i find some good Thomas Sabo jewerly.Now I put it introduced to you.Thomas Sabo jewerly

"Ruining your Test Automation Strategy." by beats by dr dre

Tue, 15 Nov 2011 02:24:07 -0600

A university studentbeats by dr dre caught by the enemy, the enemy tied him at the poles,just beats solo headphones purple and then asked him: say, where are you? You do not say it electrocuted! Scheap dr.dre beats studio headphones balck/yellowtudents back to the enemy a word, the result was electrocuted, he said: I am TVU.Hot sale beats by dr dre pro headphones

"Ruining your Test Automation Strategy." by Dr Dre beats Headphones Canada

Sat, 12 Nov 2011 02:18:02 -0600

integrated and easy to utilize. Headset stylish and generous

"Ruining your Test Automation Strategy." by Cheap Beats By Dr.Dre

Tue, 08 Nov 2011 20:31:59 -0600

relative to other headphones model is more suitable for use in portable system

"Ruining your Test Automation Strategy." by a real estate

Sun, 06 Nov 2011 09:05:51 -0600

great post informative and knowledgeable thanks

"Ruining your Test Automation Strategy." by shopping us online

Sun, 06 Nov 2011 09:02:45 -0600

I think you’ll find, however, that the more you design your systems to be testable, the more you will naturally concentrate on API tests as opposed to GUI tests. This is because it is far easier to understand API tests.

The Systems that are designed to be tested are much more comfortable to check. This makes it more easy to write both API tests, and GUI tests, and reduces the coupling issues.

"Ruining your Test Automation Strategy." by Download Android Apps

Wed, 02 Nov 2011 21:04:59 -0500

You can definitely see your expertise in the work you write. The world hopes for more passionate writers like you who aren’t afraid to say how they believe. Always go after your heart.

"Ruining your Test Automation Strategy." by christian louboutin

Mon, 31 Oct 2011 07:30:57 -0500

Hi buddy, Great post, please write more about this, and I like it. I really enjoy reading your blog popular distributed: a good article waiting for you!

"Ruining your Test Automation Strategy." by Pandora Bracelets

Tue, 25 Oct 2011 01:38:54 -0500

But for the similar profession character accessories, it also need to consider beads jewelry industry development ,which may bring competition and impact, and we should adjust industrial structure and product positioning, making our products get continuous development in the market competition.Pandora Bracelets 4Pandora jewelry (charmilia beads) derived from Scandinavia’s latest fashion statement and become fashion world rise

"Ruining your Test Automation Strategy." by credit calculators

Mon, 17 Oct 2011 08:20:03 -0500

great to have this site. thanks for sharing.

"Ruining your Test Automation Strategy." by Flagyl 400mg

Mon, 17 Oct 2011 06:49:03 -0500

I agree in this statement “The number of mocking frameworks has proliferated in recent years”