http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil en-us 40 <p>It happens over and over again. I visit a team and I ask about their testing situation. We talk about unit tests, exploratory testing, the works. Then, I ask about automated end-to-end testing and they point at a machine in the corner. That poor machine has an installation of some highly-priced per seat testing tool (or an open source one, it doesn&#8217;t matter), and the chair in front of it is empty. We walk over, sweep the dust away from the keyboard, and load up the tool. Then, we glance through their set of test scripts and try to run them. The system falls over a couple of times and then they give me that sheepish grin and say &#8220;we tried.&#8221; I say, &#8220;don&#8217;t worry, everyone does.&#8221; <p>It&#8217;s a very familiar &#8216;rabbit hole&#8217; in the industry. It&#8217;s sort of like the old days when you&#8217;d find a couple of classes generated by a <span class="caps">CASE</span> tool in every code base you visited if you looked hard enough. People started with the merry notion that they were going to round-trip code with some <span class="caps">CASE</span> tool and they learned (like most lucky teams do) that it just doesn&#8217;t pay for itself, it&#8217;s not worth the time or the frustration. UI Test Automation tools are in the same category. Personally, I think that in this day and age selling them is irresponsible. Developing them open-source? Well, let your conscience be your guide, but really, even though people can use them responsibly, they hardly ever do because these tools are sold with a dream, a very seductive and dangerous one. <h2>The Dream</h2> <p>Janet comes into work in the morning and she sits down at her super-duper testing console. She presses a button and the testing system springs to life. The application comes up all at once across ten monitors. Cursors move, selections are made (silently) and tests run against the user interface magically, as if some eager set of ghost elves took control, mischievously burrowing through the nooks and crannies of the application, running scripts to completion, and making little notes whenever there is a failure. Janet sits back in her chair, waiting for the elves to report back to her. She stirs her coffee gently. <h2>The Reality</h2> <p>Janet hasn&#8217;t gone home yet. It&#8217;s 2AM and she has to report completion of all her test cases at a meeting in the morning. She thinks she&#8217;s past the last configuration issue but she&#8217;s not sure. For the last hour, she&#8217;s been trying to make sure that a particular button is pressed at step 14 of her script, but quirky latency on the server is preventing it from happening consistently. Sadly, she has to run the script from the beginning each time. Oh, and five hours ago she discovered UI changes which invalidated 30% of the regression tests. Most of the changes were easy but she still has 12 cases to go and her 9AM meeting looms ahead of her. <p>This gap between the dream and the reality is not a matter of flawed execution, it&#8217;s endemic. Here&#8217;s the scoop. <h2>UI Test Automation Tools are Brittle</h2> <p>You might not think this is fair but it is, really. I haven&#8217;t seen one of these tools yet which isn&#8217;t susceptible to missed events or timing madness. It just happens. The fact of the matter is, it is hard to sit on the outside of an application an instrument it. It&#8217;s a very technology sensitive problem. You need to hook into either the OS or the browser or both. Neither are ever really built from the ground up for that sort of access. <h2>UI Based Testing Is Not the Solution That Vendors Imply It Is</h2> <p>This is the big issue, the one which really hurts the industry. The fact of the matter is that UI based testing should be used for UIs: that&#8217;s it. You should not be testing your full application end-to-end through a UI Testing tool. First of all, that sort of testing couples some of the most important tests in your system to one of the most volatile parts of it. It&#8217;s easy for us to see business logic as volatile, but really, the UI is the thing which twists and ripples in the winds of change. When customers want new features, often those features involve new workflows. When usability experts discover better ways of models of interaction, an agile business seizes upon them and makes the changes&#8212;if they can. You&#8217;d be surprised at the number of applications which continue to sport out of day user interfaces simply because the development organization is terrified of throwing away all of their regression tests which (by the way) go through the UI. Even if you&#8217;re not a consultant like me, visiting teams and seeing their development processes, you can see hints from the outside. Think of every website or shrink-wrap application which has &#8220;the same old workflow&#8221; and a UI that has become more cluttered over the years. Often it&#8217;s because of that lock-in. <h2>UI Based Testing Takes More Staff and Time Than You Expect</h2> <p>This, really, is the most common failure case. It&#8217;s the case which explains the dust on the testing box&#8217;s keyboard. Someone, usually disconnected from the development organization, decides that &#8220;hey, we need to solve the testing problem. We have too many people doing manual testing. It&#8217;s taking forever.&#8221; So, they do their research, find a vendor with with a good licensing model and a good pitch and then they push it on the development organization. They are, of course, looking to reduce staff so when they realize that translating all of those manual tests to the tool is very labor-intensive, they are taken aback. But, of course, it is just a temporary cost, right? But, then it takes far longer than they expect. Remember Janet&#8217;s story? It&#8217;s really hard to catch up with a UI-Based testing tool. It&#8217;s hard to even stay in place with one. Typically it takes a number of people to do UI-Based automated testing for a development team in sync with an iteration and worse, they&#8217;ll always lag behind a bit because you can&#8217;t really write UI-based tests ahead of time the way you can with <span class="caps">FIT</span> and other beneath-the-UI testing tools. From what I&#8217;ve seen UI-based testing, done diligently, takes the effort of about one tester for every two to three developers. That&#8217;s what it seems to cost amortized across all of the maintenance of UI-induced test breakage. Oh, and by the way, if think you are going to save labor using record and playback? Nope, you aren&#8217;t. It doesn&#8217;t work. <h2>Solutions</h2> <p>The fact of the matter is, you can use these tools effectively, but in a very narrow space. It&#8217;s nice to be able to test the UI&#8212;by itself. However, this sort of thing <a href=http://www.computer.org/portal/web/csdl/doi/10.1109/MS.2005.110>requires an architectural change</a>. <p> In general, UIs are too volatile for end-to-end testing. Teams that do it well, typically develop a small task-focused scripting layer and build tests on top if it so that the actual tests don&#8217;t touch the UI directly. But, if they happen across that technique, they are lucky. Still, it isn&#8217;t an ideal solution. You really want to be <i>below</i> the UI working against an <span class="caps">API</span> which exposes the business logic. And, because of that nearly mystical synergy between testability and good design, that <span class="caps">API</span> layer is often useful for many things other than testing. <h2>Conclusion/Challenge</h2> <p>I recognize that I&#8217;ve been rather vicious in the this blog. If you develop these tools for a living, you might not think it&#8217;s fair. But consider this. If you don&#8217;t think I&#8217;m being fair, take a look at how your tools are marketed. In particular, show me where the product literature discourages end-to-end testing through the tool. Otherwise, well, you know, you are probably developing snake-oil. Mon, 04 Jan 2010 05:35:00 -0600 urn:uuid:b8d700fd-4c7c-4e3c-8d6f-d91710d990d4 Michael Feathers http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil Michaels Musings <p>Nicely done, sir. I hadn’t expected the Twit-encoding to use real words, even if together they amount to nonsense. A fun toy, in any event.</p> Wed, 08 Feb 2012 02:42:57 -0600 urn:uuid:e6463c18-97b2-4c1d-a89c-6a50a66004f8 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-202300 <p>Nicely done, sir. I hadn’t expected the Twit-encoding to use real words, even if together they amount to nonsense. A fun toy, in any event.</p> Wed, 08 Feb 2012 02:42:00 -0600 urn:uuid:1aeddb81-a3f6-4346-9202-3d4199f49fa1 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-202299 <p>UI Test Automation Tools is very useful for me.Thank you very much for sharing. very much impressive post sa</p> Wed, 08 Feb 2012 02:39:20 -0600 urn:uuid:dc73c587-81b1-4539-bf10-94064c05913d http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-202298 <p><a href="http://www.findermind.com/is-beenverified-com-a-scam/" rel="nofollow">http://www.findermind.com/is-beenverified-com-a-scam/</a></p> Sat, 04 Feb 2012 03:02:10 -0600 urn:uuid:cf87034a-1366-4862-bcc2-55be14a489d5 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-201703 <p>his was written over 2 years ago, any update in your tests or goals yet??!</p> Mon, 23 Jan 2012 16:55:41 -0600 urn:uuid:a325425d-80e1-43b0-b7a1-9490e40dd9d1 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-198952 <p>This was written over 2 years ago, any update in your tests or goals yet?</p> Fri, 20 Jan 2012 13:36:57 -0600 urn:uuid:97cc13b0-24b6-47b8-a48e-c09075c8fb76 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-198708 <p>I should say that you have done a great job and your writing style is awesome. I was searching for this topic and just found your site when I was googling. Your blog can be much better if you put some pictures in it.</p> Sun, 15 Jan 2012 03:57:00 -0600 urn:uuid:fbdab356-be5c-49f2-9e03-bf04d15f36f8 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-197985 <p>Yes. maybe, some of us may ageree with you.</p> Sat, 07 Jan 2012 06:18:18 -0600 urn:uuid:18e64259-f5b7-4305-b667-91691e4d2a4a http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-195900 <p>The truth that your short article here uncovered many of the best realities and on the internet social enhancements about versatile themes like psychometric examine, I believe I will always head to your site. It&#8217;s absolutely worth examining!</p> Fri, 30 Dec 2011 04:18:20 -0600 urn:uuid:2a3a3ade-7a32-4414-9b1c-b9b3f4b48c15 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-193680 <p>good job</p> Thu, 29 Dec 2011 23:24:52 -0600 urn:uuid:64023bc6-857c-4a99-94fb-501f711556a8 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-193569 <p>UI Test Automation Tools is very useful for me.Thank you very much for sharing. very much impressive post</p> Mon, 26 Dec 2011 07:19:27 -0600 urn:uuid:c8f9b966-90f2-4431-87c6-13e267efaeb1 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-192223 <p>These tests are not enough for the essential safety precautions and security!</p> Mon, 26 Dec 2011 01:35:09 -0600 urn:uuid:2a32798b-ee4c-414e-8c00-5b19fa80c609 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-192091 <p>The classical rabbit hole in the industry. I hate to admit it but we are never gonna get away from it. However, there are still things we can do about it. A lot of good comments have pointed them out already!</p> Sun, 25 Dec 2011 16:32:07 -0600 urn:uuid:23ef6630-9764-4633-915a-7e2c60a879b0 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-191440 <p>This article is impressive,I hope that you will continue doing nice article like this.</p> Sat, 24 Dec 2011 07:33:45 -0600 urn:uuid:09d42c47-3e93-43d8-b6f6-ba559c6d7805 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-191273 <p>I am very thankful to the author of this blog because he has given me a very cool thing to read. I have been looking for an article that will satisfy me and I have seen it in this blog..-)</p> Wed, 21 Dec 2011 23:48:21 -0600 urn:uuid:b55e68c1-a734-4294-b593-39b26b8642f8 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-190294 <p>Thank you very much for sharing. very much impressive post.</p> Mon, 12 Dec 2011 14:48:19 -0600 urn:uuid:0caa224a-569d-4001-847d-7e300856f20d http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-185793 <p>Thank you very much for sharing. very much impressive post</p> Tue, 06 Dec 2011 01:53:32 -0600 urn:uuid:728eb42b-c0b4-411e-a8f4-8f4e1409fd3b http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-182738 <p>Thanks!I learn a lot from you.</p> Mon, 05 Dec 2011 23:35:43 -0600 urn:uuid:55ba2112-c7cb-49d6-b922-bb00d9f3485a http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-182651 <p>Keep up the great job. <a href="http://codmw3guide.info/" rel="nofollow">mw3 guide book</a></p> Thu, 01 Dec 2011 23:23:52 -0600 urn:uuid:670e3dd7-2ec9-48ac-946a-e571c4056b9b http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-180882 <p>Thank you very much for sharing. very much impressive post. <a href="http://www.mielevacuumbags.net" rel="nofollow">Miele Vacuum Bags</a></p> Thu, 01 Dec 2011 23:22:27 -0600 urn:uuid:0a1cc9c5-a5c9-478a-a99c-d753d8a609d6 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-180881 <p>I really love reading your comments guys.I learn a lot from you.</p> <p>Thank you so much</p> <p>Regard,</p> <p>Epoyjun</p> Thu, 01 Dec 2011 07:28:49 -0600 urn:uuid:ede84b81-c732-45cb-a507-90bdce10c0b6 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-180731 <p>How to remove wmv drm? Is there a way to <b><a href="http://www.removewmvdrm.com/" rel="nofollow">Remove WMV DRM</a></b> excellently? Yes. Try wmv drm removal software, it is professional yet easy-to-use to remove wmv drm protection even without liscence. It can not only remove drm from wmv files, but various other kinds of formats, such as m4p, m4b, m4v, wma, m4a, aac, aa, etc. It is compatible with Windows 7 / 2000 / 2003 / XP / Vista with 1G Hz CPU or above.</p> <p><b><a href="http://www.removewmvdrm.com/" rel="nofollow">Remove WMV DRM</a></b></p> Mon, 28 Nov 2011 03:13:33 -0600 urn:uuid:0c54474b-7923-4ea9-a736-a45c5afb2722 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-179184 <p>When you use the Beats By Dre In Ear headphones you not only can feel the high definition music just as the AKG but also would let you feel the fashion moment and the culture shock. So, what are you waiting for? Just come here and experience the Shaky feeling. We are looking forward to your orders.</p> Thu, 24 Nov 2011 00:56:17 -0600 urn:uuid:43988714-213d-4837-8f6e-f9809502e59d http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-177363 <p>This <a href="http://www.fmheroes.org/" rel="nofollow">FM TRANSMITTER</a> has been designed to fit and compliment the design of the iPhone, iPod Touch and any iPod with a dock connector (iPod Video, Classic, Mini, Photo, Colour etc). Unlike many other iPod FM transmitters, this one has a sleek and slim line design (see images below) and the sound quality is superb. It has the brilliant feature of being powered by your iPod (and only uses a small amount of power) so you don&#8217;t need to take extra chargers with you. Also, even though it plugs into the bottom of your iPhone or iPod, you can still charge your music player as this FM transmitter has a charge socket underneath it too. If possible you should try to stay away from the <a href="http://www.fmheroes.org/" rel="nofollow">FM TRANSMITTER</a> that also function as a cell phone mount. The reason is that while driving the car’s vibration gets transferred to the mobile and the connector between the fm transmitter and the mobile keeps on breaking up. Once the connect has broken connection the iPhone or iPod will stop playing until you click on it to play again.</p> Wed, 23 Nov 2011 20:44:21 -0600 urn:uuid:f5a420fd-d84c-45ab-beb4-237693c3d226 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-177201 <p>this is the job post of the career brightness. <a href="http://banksjobs.co" rel="nofollow">banks jobs</a></p> Wed, 23 Nov 2011 00:30:25 -0600 urn:uuid:6e77e5d1-0a79-4f91-9c39-c6d01a3db765 http://blog.objectmentor.com/articles/2010/01/04/ui-test-automation-tools-are-snake-oil#comment-176907