UI Test Automation Tools are Snake Oil 62
It happens over and over again. I visit a team and I ask about their testing situation. We talk about unit tests, exploratory testing, the works. Then, I ask about automated end-to-end testing and they point at a machine in the corner. That poor machine has an installation of some highly-priced per seat testing tool (or an open source one, it doesn’t matter), and the chair in front of it is empty. We walk over, sweep the dust away from the keyboard, and load up the tool. Then, we glance through their set of test scripts and try to run them. The system falls over a couple of times and then they give me that sheepish grin and say “we tried.” I say, “don’t worry, everyone does.”
It’s a very familiar ‘rabbit hole’ in the industry. It’s sort of like the old days when you’d find a couple of classes generated by a CASE tool in every code base you visited if you looked hard enough. People started with the merry notion that they were going to round-trip code with some CASE tool and they learned (like most lucky teams do) that it just doesn’t pay for itself, it’s not worth the time or the frustration. UI Test Automation tools are in the same category. Personally, I think that in this day and age selling them is irresponsible. Developing them open-source? Well, let your conscience be your guide, but really, even though people can use them responsibly, they hardly ever do because these tools are sold with a dream, a very seductive and dangerous one.
The Dream
Janet comes into work in the morning and she sits down at her super-duper testing console. She presses a button and the testing system springs to life. The application comes up all at once across ten monitors. Cursors move, selections are made (silently) and tests run against the user interface magically, as if some eager set of ghost elves took control, mischievously burrowing through the nooks and crannies of the application, running scripts to completion, and making little notes whenever there is a failure. Janet sits back in her chair, waiting for the elves to report back to her. She stirs her coffee gently.
The Reality
Janet hasn’t gone home yet. It’s 2AM and she has to report completion of all her test cases at a meeting in the morning. She thinks she’s past the last configuration issue but she’s not sure. For the last hour, she’s been trying to make sure that a particular button is pressed at step 14 of her script, but quirky latency on the server is preventing it from happening consistently. Sadly, she has to run the script from the beginning each time. Oh, and five hours ago she discovered UI changes which invalidated 30% of the regression tests. Most of the changes were easy but she still has 12 cases to go and her 9AM meeting looms ahead of her.
This gap between the dream and the reality is not a matter of flawed execution, it’s endemic. Here’s the scoop.
UI Test Automation Tools are Brittle
You might not think this is fair but it is, really. I haven’t seen one of these tools yet which isn’t susceptible to missed events or timing madness. It just happens. The fact of the matter is, it is hard to sit on the outside of an application an instrument it. It’s a very technology sensitive problem. You need to hook into either the OS or the browser or both. Neither are ever really built from the ground up for that sort of access.
UI Based Testing Is Not the Solution That Vendors Imply It Is
This is the big issue, the one which really hurts the industry. The fact of the matter is that UI based testing should be used for UIs: that’s it. You should not be testing your full application end-to-end through a UI Testing tool. First of all, that sort of testing couples some of the most important tests in your system to one of the most volatile parts of it. It’s easy for us to see business logic as volatile, but really, the UI is the thing which twists and ripples in the winds of change. When customers want new features, often those features involve new workflows. When usability experts discover better ways of models of interaction, an agile business seizes upon them and makes the changes—if they can. You’d be surprised at the number of applications which continue to sport out of day user interfaces simply because the development organization is terrified of throwing away all of their regression tests which (by the way) go through the UI. Even if you’re not a consultant like me, visiting teams and seeing their development processes, you can see hints from the outside. Think of every website or shrink-wrap application which has “the same old workflow” and a UI that has become more cluttered over the years. Often it’s because of that lock-in.
UI Based Testing Takes More Staff and Time Than You Expect
This, really, is the most common failure case. It’s the case which explains the dust on the testing box’s keyboard. Someone, usually disconnected from the development organization, decides that “hey, we need to solve the testing problem. We have too many people doing manual testing. It’s taking forever.” So, they do their research, find a vendor with with a good licensing model and a good pitch and then they push it on the development organization. They are, of course, looking to reduce staff so when they realize that translating all of those manual tests to the tool is very labor-intensive, they are taken aback. But, of course, it is just a temporary cost, right? But, then it takes far longer than they expect. Remember Janet’s story? It’s really hard to catch up with a UI-Based testing tool. It’s hard to even stay in place with one. Typically it takes a number of people to do UI-Based automated testing for a development team in sync with an iteration and worse, they’ll always lag behind a bit because you can’t really write UI-based tests ahead of time the way you can with FIT and other beneath-the-UI testing tools. From what I’ve seen UI-based testing, done diligently, takes the effort of about one tester for every two to three developers. That’s what it seems to cost amortized across all of the maintenance of UI-induced test breakage. Oh, and by the way, if think you are going to save labor using record and playback? Nope, you aren’t. It doesn’t work.
Solutions
The fact of the matter is, you can use these tools effectively, but in a very narrow space. It’s nice to be able to test the UI—by itself. However, this sort of thing requires an architectural change.
In general, UIs are too volatile for end-to-end testing. Teams that do it well, typically develop a small task-focused scripting layer and build tests on top if it so that the actual tests don’t touch the UI directly. But, if they happen across that technique, they are lucky. Still, it isn’t an ideal solution. You really want to be below the UI working against an API which exposes the business logic. And, because of that nearly mystical synergy between testability and good design, that API layer is often useful for many things other than testing.
Conclusion/Challenge
I recognize that I’ve been rather vicious in the this blog. If you develop these tools for a living, you might not think it’s fair. But consider this. If you don’t think I’m being fair, take a look at how your tools are marketed. In particular, show me where the product literature discourages end-to-end testing through the tool. Otherwise, well, you know, you are probably developing snake-oil.
Hi,
I don’t think you’re being fair. Well, actually, I think you’re pointing fingers at the wrong people. Automated UI Testing tools are frail, brittle and unreliable because most view “languages” are frail, brittle and unreliable. The easiest example to talk about is HTML and CSS. When performing actions against HTML and CSS, uncertainty is high. It is easy to corrupt a script by duplicating names, having things appear in unexpected ways, etc.
Now, lets take a more strict, more tightly controlled, more reliable view language such as Cappuccino . When you have reliable foundations, you can build reliable tools. This is why I was able to create Barista which is not frail or brittle but is completely reliable.
Having said that, I don’t think that “end-to-end” testing from an automated UI testing tool is a great thing, either. Instead, we should flex the muscle that is MVC. Because MVC decouples these layers, we can test them independently of each other quite easily. I shouldn’t need a view to test my domain logic on a model and I shouldn’t need a database to test my view logic. Use that with integration tests and large-scale UATs and there you have an effective test suite.
I thing that, while everything you’ve said is technically correct, the implication is incorrect.
In the hands of a dedicated QA engineer, automated UI test tools are wonderful. They are just not useful tools for continuous integration testing by a development team.
A good QA engineer who is an expert at baby sitting a particular automated test tool can accomplish an awful lot of important testing in a relatively short time.
Umm – that’s “I think”... Unfortunately, I can’t edit the comment to fix the typo.
Did you choose that title deliberately? This is from 1996: http://www.satisfice.com/articles/test_automation_snake_oil.pdf
Good post , Michael.
I agree with the problem and some of the solution – if you want to use the tools correctly, you need a dedicated person/team working on maintaining and adding scripts.
But that means that there isn’t a real solution for automated testing to most people. Most people do not read this blog, and don’t know what FIT is. Testing the API is something they would like to do, but that’s after they separate the GUI from the rest of the app (the day after they have time). The minority who are experienced with testing can do FIT + unit testing, but – they are the minority.
And that leaves “UI based testing” a recipe for failure. Thanks for the post.
Gil Zilberfeld
I would have agreed with this a couple of years ago – but not any more. If I did unit testing without TDD, I would have said unit tests were snake oil. But UI tests, like unit tests, require design for testability at the UI layer. And until our team learned design for testability at the UI layer, we wrote bad, broken, brittle UI tests.
Can you talk more about this?
At my last job, I used WatiR to script the UI of my company’s flagship .Net ASP web app. I wrote a layer on top of WatiR that, given an HTML ID, a meaningful name, and the form field type, would generate methods for getting/setting the field value, getting all options from a drop-down, clicking a button, etc. I wrote a layer on top of that, where each .aspx had a corresponding ruby class—using the lower layer, you would list each field’s name, HTML ID, and type, and the lower layer would generate methods on that class. You would write your tests against that layer, and they would read pretty much like a test script.
Is that something like what you mean by “a small task-focused scripting layer”?
I left the company soon after writing that, so I didn’t get to see how it turned out in practice. But now, at my current gig, we’re facing a QA bottleneck, and talking about a similar solution. Like you said, we have an expensive, and mostly unused, test automation tool. The QA staff insists that automating tests, and keeping up with functionality changes, is too expensive, but I wonder whether it’s because the tests can’t be factored, like I think they could if they were written in something as flexible and composable as ruby, or even java/c#.
An interesting article that addresses a problem (end-to-end testing), but criticizes the tool for it, which I think is wrong.
Background: Currently, I am one of the testing guys for a company that mostly relies on automated UI end-to-end-testing for their product (big one, millions of customers).
How this started is simple: They didn’t have tests at all. After a new dev team leader came up, he decided that the fastest way to get good coverage done was running tests via the UI (they didn’t have a real API yet). They introduced Selenium for that. And these days it is not unlike the scene described under “The Dream”.
There are some issues with that (which is where I come into play, writing unit tests, api tests and all that), mostly speed and the inability to pinpoint a problem to a certain layer. But the results were impressive nonetheless, because after half a year, bugs disappeared and never reappeared.
The system using selenium is not always easy to use (for complex things – for standard cases, you can do a nice YML style DSL, pretty much like FIT). Not because the UI is fragile, but because the framework using Selenium grew to a certain complexity.
So, here is one example where it works. No snake oil there, tangible results, no placebo effect.
I think the mistake is in here: “It’s nice to be able to test the UI—by itself. However, this sort of thing requires an architectural change”. Right. But where is the difference to other cases of test-driven development?
I’m currently introducing unit tests to several rather old modules of the code. I can tell you: Whether GUI or “normal” code, if the code is not designed to be tested, then every test you write is brittle.
Put differently: “because of that nearly mystical synergy between testability and good design, that API layer…” – why would those paradigms only apply to API design?
That’s not a GUI problem. I think it is just that we programmers usually do it more properly on our code than those GUI guys who usually have more of a web-designer background. That’s why we got the notion that it is easier to do on those low levels we work on.
As a conclusion: I don’t think one way of testing is more brittle than another. All tests do have their uses. All should be applied.
Ideally, one would go unit testing -> API testing -> UI testing, every layer of tests only testing the additional complexity. If you step in later, it may well be the other way around if you’re pressed for results.
Good post.
I have seen a QA team that successfully implemented end-to-end UI testing just like you stated in your “Dream” section. It does require in-sync communication, though, as you state and there was roughly a 1-to-3 QA-to-developer ratio.
The term UI test automation is very general and as such I do agree with you, for a large part it’s painful and can lead to a lot of wasted effort. However, the core problem is the way people write the tests. Too often I see people write something like (ignore syntax obviously):
test_logging_into_UI() { Browser = new IE(); Browser.goto “http://UAT/login.aspx” Browser.textbox “username”, “Admin” Browser.textbox “password”, ”@dmin” Browser.click “Sign in” Assert.Contains “Logged In”, Browser.Body }
They have this over hundreds of tests, without any separation. One day, the way the application needs to login changes and as a result everything breaks.
Alternatively, they have used automated UI recorders which produce unmaintainable code that results in the entire test needing to be re-recorded after each change.
Bad times and long nights are the main outcome from this approach.
I agree with you, the only way for UI testing to work is have a “small task-focused scripting layer”. I’ve found using a framework such as Cucumber + Ruby can create this layer for you based on the frameworks approach to testing.
I try to keep the tests focused on core scenarios and try and abstract away into behaviour of your application (login) instead of how the application works (fill in textbox, click button).
Following this approach does result in successful UI tests.
Jeremy D Miller has a series of blog posts Build your own CAB and is writing a book (together with Martin Fowler), Presentation Patterns, on this very subject of getting the UI patterns right so that unit and integration testing is possible and ultimately less brittle.
I agree with much of what you said for point and click record tools, but I sincerely believe that frameworks like Watij, Watin, Watir and the like are usable to build robust automated UI testing solutions. I actually wrote about the difference between the two approaches here: http://simpleprogrammer.com/2009/12/18/automated-functional-testing-record-or-program/
I am currently using Watij to build a custom framework on top of the UI layer. For example I have screens and processes, the screens and process have functionality like: Fill in customer name, save the page, etc. The screens and processes use the UI layer to map the to the Watij libraries for doing the things. And all of tests use the screens and processes layer. So if something changes on a screen, it just has to be changed in one place and all tests will still run.
I think the real key to winning with automated UI testing is you have to build and invest in your own framework tailored to your application.
Generally disagree. It is labor intensive, but more effective than a team of manual testers because I can run the suite at my desk and make fixes if I change something it depends upon. In particular, I find that many bugs only pop in when an end to end test is run that includes the UI. I am generally an advocate for more integration testing.
Would you recommend doing end-to-end testing at all? If so, how?
I’ve been teaching TDD over web user interface for some time and I see as much benefit here as doing TDD a non-user interface. As with anything in software development sustainability, implementation design delivers more productivity returns than tools do.
The problem isn’t the level of the architecture that tests touch. The problem is trying to make people with insufficient skills productive by arming them with elaborate tools. The brittleness is usually a result of poor abstraction than deficiencies in the architectural layer. Tests over a non-user interface can be just as brittle when the abstractions are inappropriate, just plain wrong, or just plain missing.
Good software developers do good work. Knowing how to do that work makes all the difference. There is less understanding and experience of doing TDD well over UI in the programmer world than doing it over APIs. The ultimate reason is a failure to extract ever more essential principles of TDD and applying them more broadly.
TDD has been around long enough to have become mired in the cognitive constraints its own orthodoxy. It was inevitable. When celebrity programmers become so invested in a schtick, they usually fail to continue pushing for the next cognitive breakthrough. One of these failures is more exploration in TDD over UI and the extrapolation of ever more essential practice principles that show how and when it can be done well, and under what circumstances it can be responsibly undertaken.
A GUI in general is responsible to query the system state and represent this information to user and take user’s input to alter the system state.
1- Unit Tests (No GUI) 2- Integration Tests (Still no GUI) 3- Automated GUI tests (which have no system dependency) 4- AdHoc Manual GUI tests
cheers,
Hy Michael, I do agree that designing the architecture such that the UI testing is easier to write is the best approach to some of the problems you described in the article.
But as some of the comments already state the problem of UI testing has some greater roots of evilness. I was part of a developer group with about ten developers working on the same product. The code foundation was C#/.NET and for the UI part we used DevExpress with Model/Passive View/Presenter approach. Large parts of the business logic could be tested with unit testing approaches (we used TDD). Nonetheless we had serious UI quality problems and a large number of software issues reported by the client which were only related to UI components (grid handling, refreshing problem etc.). So we introduced manual testing which every developer of the team needed to do once in a round robin fashion every sprint.
This manual testing considerable increased the UI stability and over all quality of the software. But there was a serious drawback. It took every day at least one developer about 2 to 3 hours manual testing. Then the customers introduced a automated UI testing framework which was based on quick test pro with a scripting engine above for the customers use cases. But the problem with this approach was, that the core scripts were written in visual basic and the system was pretty unstable.
Then we stumbled over Ranorex. Ranorex allows us to create object oriented repositories of the UI components by using code generation and UI inspection. Ranorex has a really flexible query language in xPath manner which makes the querying of different attributes and component really easy. Upon these automated repositories we built some sort of abstraction layer with several access components. Theses components are written in C# and are reusable. If the UI changes you need only to update the automated repositories without touching the UI tests. And the best is we were able to write the UI tests with xUnit or nUnit and also the tests could be run by our CI server. Ranorex also offers a rich reporting API which allows to make screenshots, html reports etc.
Daniel
I actually have developed four testing frameworks in my career in testing, at two different companies, and I’m confident that in my next job I’ll be able to do the same. I’ve done this in WinRunner, LoadRunner, QuickTest Pro at the big company, and TestComplete at my current employer. I’m going to work at another employer next week, and we’re using Selenium there, and possibly something else in the near future.
The old job was UI in VB and Object Cobol and wire-level performance tests in raw sockets, and the current job is UI testing for validation purposes. The next job is web interfaces, and I can’t wait for the challenges I’ll face there.
The key is to try to detach from your tool a little and store information about objects in some database or tool that you can quickly change out when a new version of a screen strikes. You need to ‘wrap’ the built-in functions because they are never good enough at reporting and fail-safe and fall-back. You have to program exception handling so if your app falls down, it pins it down, etc.
It is not a QA job, it is a programming/engineering job. You can’t record-and-playback and knock wood.
By definition the only thing that cannot be tested any other way than via UI is the UI itself – in a typical web application this means mainly JavaScript. However the good thing with JavaScript is that usually there are not many dependencies between different pages. If you change one page then JavaScript on it may break but at least it’s not likely going to cause bugs on some other totally unexpected places.
For example in recent Grails project we stopped writing UI tests because they just needed to be updated too often and Grails offers very nice integration testing tools where you can easily test your whole system using MVC controllers as the API. In such small 9-man month project we just were not able to stabilize the UI soon enough to make UI test automation beneficial. On the other hand if the project is longer or there just is no API layer suitable for testing then UI tests may be the only way for automating integration tests.
I was shocked when I started reading this article.
I thought it had been written by someone working at my last company.
When I started there they had a legacy code base of over 580K lines of code. There were 80 unit tests, the majority of which tested code that was not in use. The only effective way of validating the product was the GUI-based testing.
We had a dedicated testing team of three intense Russians doing the testing. I knew that something was really broken when one of them would come over with a big smile on their face and say “Guess What”! One person maintained the Scripts, and two people ran them. The initial smoke tests would take 6 to 8 hours with full testing taking at least two days. They were very professional crew, yet the brittleness of the tools and process lowered their effectiveness drastically.
A single unannounced change in the UI could derail days of testing.
I am clear that the sheer slowness of the product was part of the issue, and the testing scripts themselves could have been better designed for independent usage. In the end though, I agree that the UI testing tools should only be used for testing the UI, not the backend functionality.
Elijah Sarver’s recommendations above would make a real difference in the effectiveness of the tool but would still be subject to radical changes in user interface (Though much less sensitive than the current implementation).
The one additional thing I would mention is that the high price of the GUI testing tools combined with how slowly they adapted to new technologies meant that open source GUI solutions were rapidly becoming a more desirable set of tools because while they have the same limitations as the commercial set, they are cheaper and adapt more readily to new technologies.
I agree and disagree.
It’s certainly not easy!
In reality automated UI/E2E/Business tests do work it just takes alot of effort, the right people. A strong vision, sweat tears and the right tools….
All of these are critical, if you developers don’t want to refactor to fit predictable testing in or work with testers it will fail. If you choose crap tools it will fail. If you get some numpty/idiots to write the tests it will fail. If your boss / manager / director starts to cut the hours/funding after only 6 months it will fail. If you don’t have someone with the vision and the drive it will fail.
Get these things right and it will work. But then you need to maintain them :)
When you have the right people the right tools, the vision and dedication. You get hundreds/thousands of tests running daily in a satisfactory fashion and they do expose issues. You can release in tight cycles with a high degree of confidence because you know your tests are running regularly and they pass. Rather than waiting months for testers to re-test everything. That’s the dream and when you put the effort in it pays off. Personally I believe lots of manual testing is for hacks.
But your right don’t believe vendors. Make sure you get the right tools and expect sweat and tears for 6 – 18 months. Then reap the rewards.
One of the reasons the one framework was very robust is that we took the time up front to consider the workflow of moving between versions or builds of an application. We were lucky enough to have radical changes to an app’s UI weeded out of some of our test cycles:
Everything was reviewed by another developer when the change went in to find additional errors that the original developer may have missed. And specifications before the developer got it were checked for technical issues before they were coded. Why more companies don’t do this is beyond my comprehension.
I could read what everyone else said, or just leave my own opinion and walk away. I will do the latter.
Software QA 12 years now and counting. So this is where I am coming from. Games and Business software. Console, PC and web based programs.
Programmers and Developers never seem to create the UI to be tested with Automation tools. Rarely do they have the foresight for it. Sure they may create Unit Tests, and other ways of ensuring the functionality may be intact. But often they do what sales or marketing wants, they make clear buttons, or jpgs text or use 8 different code bases and a crystal report and 4 other esoteric means of accomplishing the same thing.
Sure it works, and it looks pretty. But if we are wasting our time wading through the Pomp and Circumstances of a program then we are undoubtedly going to miss a big kahuna of bugs.
Automated Testing tools and Programmers = QA job security. Well until they machines start coding themselves.
All of the above comments are true - But - Test automation like application development is in fact a development endeavor. If you take the time to properly develop & build up a support structure (framework) for the test automation that addresses both generic (platform specific) & application (application specific) functionality, your automation suite will serve its purpose well (assuming expectations were properly set in the first place).
If in fact you are planning on quickly implementing test automation because your project is behind & could benefit from a quick fix – DON’T. That is not what automation is well suited for. Test Automation can be helpful on several fronts. The gamut can be divided into the following major categories, Smoke, Functional, System & Performance. Each of those categories is intended to address different facets of UI or application testing. None alone is a solution by itself. It is the concatenation & integration of these different testing domains that provides quality assurance.
Smoke testing lets say, can be used primarily as build validation. We can build a simple test that just opens & closes every screen from every possible option; menu, button or combo box selection.
Functional testing takes the testing to the next level. It tests the UI to make sure that all buttons button, all fields accept or impede data from entering the system. Functional testing should exercise the UI functionality exhaustively if possible. This is where the power of automation can alone prove its worth. For example, by developing a function that tests data entry fields, you can generically apply a gamut of tests, to every occurrence of a data entry field in your application; valid, invalid, conditional, length, positive, negative & more. The function can be parameterized to add further flexibility to the automation yielding even greater benefit.
System testing should test the business & use case scenarios intended for the application; for example, taking a customer call, filling out the screen form & generating the follow up letter. Of all the test categories, this one may be the most difficult to wrap the automation around. Smoke & functional testing has numerous but finite paths to test. System testing can sometimes identify a multitude of paths requiring a more measured approach. Perhaps this is where prioritization & continuous evaluation can yield the most benefit to the automation. Test the high priority, most used, used by most functionality first. Review your test suite to minimize redundancy (often introduced when the suite is developed over time by different people) & eliminate obsolete tests.
Performance testing is intended to identify scalability & stress related problems. Planning accurate performance test can some times be guess work at best, particularly on new applications where empirical use data is not available. However, overlooking performance testing as a necessary facet of the overall test automation can be the lynch pin on a project. Your application may work well in a standalone mode & under a normal user capacity but introducing an inordinate number of users or oversized data retrieval such as a transaction history or large document can bring the system to a grinding halt or an unacceptable performance threshold.
Thinking you can create a quick & dirty test automation strategy is like thinking you can build a smooth sphere using square pegs. If you look closely you’ll find sharp points everywhere you look. Test automation needs to be carefully & thoughtfully conceived so that it can address the needs of the development organization. Putting extra time & effort in developing a robust & flexible automation framework upfront will pay big dividends on the back end – at crunch time & during regular regression cycles.
In the end, it may or may not be matter of snake oil vs. silver bullet but more a question of what you do with the tool.
While I agree that testing tools are sold with the dream of an automated test suite with little/no work, the reality is that a UI test suite is no different from any other test suite.
If you write it so it relies on lots of little implementation details with none of the necessary abstraction then of course you will end up with duplication, overly tight coupling and something that is impossible to maintain.
It’s like most coding tasks where you can use tools.
For instance – a GUI builder – yes the code will suck. But you can take the essence as a starting point and refactor as necessary.
Or you can just take the generated code to learn the APIs better.
This is how I use Selenium’s tool. It tells me what the element locators are and I can save the whole thing down as a GroovyTestCase. I then just extract the bits I want into scripts.
So use the tool, don’t let the tool use you.
In general though regardless of tools UI testing is possible. I would recommend the following to help stability -
Choose a (more) stable interface to test – the user intention interface is better than the UI. Then inside your script code bind to UI elements using the least specific expression you can get away with.
Use abstraction and stay DRY.
Choose a decent scripting language (preferably one that integrates with standard test tools like JUnit).
I’m currently using Concordion (open source), Selenium (open source) and Groovy to write my UI tests. Suffice to say when I execute them there are just some great tool APIs in use – albeit used by some clean DRY layered test code.
Here’s an example of what I’m doing (albeit with a small example).
http://bit.ly/4BeuA7
While I do agree that UI test tools are frail and brittle if you’re testing the UI, using automated test tools against the UI in order to test some outcome tends not to be. At least, for me and my teams.
So, the difference is this: are you testing the registration screen(s) to make sure the elements are located correctly? Or are you testing them (and entering data in the correct fields and making sure you get the appropriate error screens and messages wrapped in the appropriate HTML element) to make sure you run through the UI as a user would, then connecting into the DB to make sure it is in the right state AND that the user gets the right confirm screen? For me and my teams, the latter works well (using Selenium driven by executable use cases or BDD or FIT).
So the tools were blamed. Shouldn’t the developers be blamed for not having a good standard to implement UI designs to.
A real and consistent development process can cure the brittle failures.
Only one person has hinted on what I consider the key point: This is not really a technology problem, but a personel problem.
The short of it is that QA is almost ALWAYS junior or very junior engineers. Even when they are more senior QA, and very good at testing, they are still fairly junior engineers. Good developers are not compensated well enough (with money or status) to stay in QA and solve QA problems.
The solution is to put more genuine engineering effort and talent into your automated testing creation. There is no shortcut to this.
You can see from these posts that the basic solution is generally agreed upon: Build an abstraction layer between your tests and your UI implementation. It’s not an astounding insight, it’s good code 101. But actually executing on this ideal takes someone who is senior enough not only to know what needs to be done and how to do it, but also to INSIST on it being done the right way. And if management is going to insist on short term gains over long term planning, you might as well not even bother.
Just like in development, there is no QA silver bullet. There is getting experienced, smart people working on the testing project and giving them the authority to do things well.
Interesting article and I kind of agree with most things you are saying. But doesn’t your arguments apply to all automated test tools?
Attitude may be the issue.
Developers need to take a more active role in the quality and well being of the software. On projects that I am involved with we use HTMLUnit and FlexMonkey for our automated UI testing in dev, and our UI is locked down. Any code changes that adversely affect the UI are caught with the first integration build, well before QA sees the changes. Any issues caught by QA are incorporated into the test suite. It makes sleep restful, and facilitates maintenance.
I have alternative project based on Selenium here: http://code.google.com/p/design-of-selenium-tests-for-asp-net/
I used several levels of abstraction and therefore all tests look like:
This simplicity allows to non-developers to write tests :)
The big software testing tool vendors knew their testing tools won’t work, so that they charge big $$$$$ for one seat. How could a project claim being ‘Agile’, but only have one software testing license? Isn’t agile embracing whole team collaboration and automated testing? The reality says all.
However, there is hope. Our project dumped several expensive (so called enterprise) testing tools after year of frustration. We found iTest2 (renamed TestWise recently, http://testwisely.com) from a small company, to our surprise, It answers virtually all our problems (hard to learn, complex test script syntax, impossible to maintain, expensive and lack of team collaboration) in an extremely simple way. It is now the most used software in our team, on par with Outlook.
I’m with Chris M on the title “Snake Oil”.
I believe GUI can be stable, but not as stable as API tests. I worked on a product that didn’t have an API, so we are used web services. The web services weren’t as rich as the UI, not touching lots of parts of the product. We had both… Actually we have junit tests too.
A good library specific to the application can make maintenance much easier. A ‘submit’ function makes more sense than clicking ‘ok’ buttons, the way the form is submitted is subject to change.
Timing can also be an issue, but that’s possible with any interface (API or web services), especially when background processes and schedulers have to be relied on. Data assumptions are also problems no matter what interface is used.
I make it a point to identify the cause for every script that I have to fix. That way I cam learn from it and teach others.
Dave
Web UI testing is one of the hardest thing to do. Be prepared to account for different browsers, Javascripts, types of test data, etc. In a way, I second Chris on his points and they are all valid.
This all depends on whether you’re testing the browser or the web service. One would think it’s sufficient to test the web service, and you can even eliminate the browser in a majority of test cases. I think the consensus here is to rid ourselves of UI tests when at all possible and concentrate on what we’re trying to test and not the shiny shell.
Using QA tool the whole application can be tested including database. All it needs is a strong framework. It would not be required to babysit automation scripts if proper error handling techniques are implemented. Coming to UI changes, sometimes developers find it easy to replace the old code instead of updating. It is here mostly automation scripts fail, because the object properties would have changed and are not recognized by the tool. Even at this point smart identification in QTP plays a major role. Smart identification some times fails when there is more than one object similar to other, which can also be handled using different property settings.
If the scripts are not used for a while they go into maintanance/update mode… where repairing these scripts would take longer than creating new ones.
I think there are some valid points (despite being 15 years behind), but it’s a bit of a straw man argument. The problems you describe are to do with strategy (both test and architectural), skill (testing, automation and development) and management approaches (unrealistic expectations and delivering expedient solutions rather than good ones). I don’t see it as a tooling problem.
Any test effort involves tradeoffs. If someone can’t or doesn’t articulate when their approach would be a good idea and when it might be a bad idea, I’m going to assume they haven’t thought it through or lack experience.
Testers have seen lots of automation approaches succeed and fail. This includes the approaches you describe.
Spot on.
OK, so it is true that many people buy expensive test automation software and it becomes shelfware after some time. I see that all the time when talking to people who wish to make it work.
Usually, the issue is partly the tool selection – some tool that requires either programming, which has a high skill, error and maintainability factor, or record & playback, which is extremely brittle and high maintenance. Scriptless tools which are not record & playback can solve this part of the problem.
Often, the other part of the issue is lack of commitment to developing the critical mass of automation needed to gain the advantages promised. Management does not develop realistic plans to ensure that sufficient automation is done to have a net positive effect, and this is often due to vendors setting unrealistic expectations.
However, with usage of scriptless tools that enable rapid and easy update when the application under test changes, and keep documentation up to date automatically, and with education of management on the commitment needed, I do see huge successes with test automation. The Dream, as you call it, is realized.
I have personally been involved with numerous projects involving large, complex transactional web sites, POS systems, legacy systems, and traditional client-server systems, where, after about 3-4 months of automation implementation, major time and cost savings for regression testing were realized, and both more, and more complex, defects found. Yes, the applications under test changed all the time, but the tools used were such that rapid and accurate change to the test suite was also possible.
For such projects, I’ve seen many cases where the testing results achieved were not only faster and more cost effective than manual test, but manual test could not possibly have uncovered the issues found, as the complexity of testing, the timing and synchronization required, and the gathering of realtime data to analyze and report whether the system was behaving correctly, were beyond human capabilities.
So, it is true that many automation projects founder. It is also true that with the right tools and realistic planning, tremendous success can be realized via GUI-driven test automation.
“The fact of the matter is, you can use these tools effectively, but in a very narrow space. It’s nice to be able to test the UI—by itself. However, this sort of thing requires an architectural change. ” these tools can be tried on various matters and basis on many things
I like This site! Thank you for your information
Free download Blu-ray to iPad Mac, you can easily convert Blu ray and DVDs to iPad for playing.
Hi!
Would you be interested reviewing our RoutineBot automation software (www.routinebot.com)? It is would be possible to use a backlink to our web-site into the post, then I’ll be interested in paying you via paypal.
Thank you!
Well written about testing. But I cant say that all agree with you.
Well it gives lots of knowledge about … “The fact of the matter is, you can use these tools effectively”
Virtual Receptionist
This post was mentioned on Twitter by lostechies: blogged: UI Automation tools snake oil?: Michael Feathers posted a thoughtful piece describing the general problems
I surfed here through Google! Great job.
Thanks for sharing the information.It is definitely going to help me some time
The noblest search is the search for excellence
interested reviewing our RoutineBot automation software ? It is would be possible to use a backlink to our web-site into the post, then I’ll be interested in paying you via paypal.
This blog contains about the test automation which cover the whole story about the automation, but the main stream line is over crossing the limits so there for I am adding some of the information about the covering article. Now I am talking about the internet, from where we got some more and more info about the automation but I consider that it solutions have a ability to do this.
In post your work up to the task,and beautiful post , Design is very deficalt.thaks you for the sharing .
I am facing problem to test one of my website through Coded UI Testing. I am using MS Studio 2010. And I found problem during testing through Different Browser. Also sometimes facing the problem of Fatal Error and Unexpected Error need to restart studio 2010. Can you please help me out how to resolve this issue.
I am happy to find much useful information in the post, writing sequence is awesome, I always look for quality content, thanks for sharing.
Good article. Excellent written: clear and correct. Link put on Twitter.
Your article was quite intriguing and the information quite useful. Will check your site often to see other great posts you make! Regards call center outsourcingr
You’re doing a great service to your readers by opening up the comments section, it’s a great way to keep in touch with them.
What a bunch of verbose crap, all of it.
As always an excellent posting.The way you write is awesome.Thanks. Adding more information will be more useful.
seattle dui
7yrs of QA exp. I agree with Michael Feathers.I have seen people enhancing/correcting their UI test suites at the critical phases of releases and feel happy that all the issues of UI scripts are fixed now but they miss the effective testing of the AUT in this happiness. And they do it repeatidly in all test cycles (spending days/nights testing the test suite only). I firmly agree with the API testing , you can cover any kind of testing like functional/parameters/integration testing using APIs and remove most of the bugs. Michael rightly said test UI for UI issues only no need to functionaly test using UI.
Me & my neighbour were preparing to do some research about that. We got a good book on that matter from our local library and most books where not as influensive as your information. I am very glad to see such information which I was searching for a long time.
los angeles locksmith
Your article was quite intriguing and the information quite useful. Will check your site often to see other great posts you make! Regards