Dependency Management: HtmlUnit

Dependency Management: HtmlUnit 112

Posted by Uncle Bob Sun, 11 Feb 2007 18:55:14 GMT

If you are planning on building an API, please, please, think about dependency management. Don’t make me know more about your world view than necessary. Consider what happened to me as I explored HtmlUnit…

I’m using HtmlUnit to parse and interpret HTML web pages. I’ve been very impressed with this library so far. And I appreciate the hard work and dedication of people who give their software away for free. So, although this blog is a complaint, it should not be misconstrued into anything more than constructive criticism. Besides, what I am complaining about here is so universal that it really wouldn’t matter whose software I chose to scrutinize. The HtmlUnit authors just got lucky in this case.

What I want to do with HtmlUnit is quite simple. Given a string containing HTML, I’d like to query that HTML for certain tags and attributes. For example, I’d like to do this:


    HtmlPage page = HTMLParser.parse(htmlString);
    HtmlElement html = page.getDocumentElement();
    HtmlElement listForm = html.getHtmlElementById("list_form");
    assertEquals("/Library/books/manage.do", listForm.getAttributeValue("Action"));

Sweet, simple, uncomplicated. Just create the DOM from an HTML String, and then query that DOM. Unfortunately, HtmlUnit does not appear to be that simple. What you have to do instead looks like this:

    StringWebResponse stringWebResponse = new StringWebResponse(htmlString);
    WebClient webClient = new WebClient();
    webClient.setJavaScriptEnabled(false);
    HtmlPage page = HTMLParser.parse(stringWebResponse, new TopLevelWindow("", webClient));
    HtmlElement html = page.getDocumentElement();
    HtmlElement listForm = html.getHtmlElementById("list_form");
    assertEquals("/Library/books/manage.do", listForm.getAttributeValue("Action"));

The extra stuff in here is apparently due to the fact that the authors wanted to be able to simulate browsers, frames, and javascript. I think their goal was laudable. However, I wish they had done this without forcing those frames, browsers, and script engines down my throat.

Given my simple needs, why do I care about WebClient and Window. Why do I have to turn off the javascript engine? It may seem a small thing, but it bothers me nonetheless. It’s the principle of the matter that gets under my skin. The pragmatic programmers called it The Principle of Least Surprise. I call it, simply, dependency management. Don’t make people depend on more than they need.

The cost, to me, was an hour of rooting around in the documentation, example code, and my own trial-and-error experiments. (The benefit to me was another blog topic ;-) That cost may not seem great; but it must be paid again and again by everyone who wants to use the package in a way that doesn’t quite fit the authors’ world view.

There may, in fact, be a simpler way to do what I want to do with HtmlUnit. If there is, I haven’t been able to find it, and I’d be grateful if anyone out there, including the authors, could guide me in the right direction.

Posted in Uncle Bob's Blatherings
Meta no trackbacks, 112 comments, permalink, rss, atom

Trackbacks

Use the following link to trackback from your own site:
http://blog.objectmentor.com/articles/trackback/159

Comments

Leave a response

Paul King about 6 hours later:

HtmlUnit is streamlined for accessing sites (perhaps the String case is not so well handled). Here is the normal thing you would do – coded in Groovy:

import com.gargoylesoftware.htmlunit.WebClient

def webClient = new WebClient()
def page = webClient.getPage(some_url)
def listForm = page.getFormByName('list_form')
assert '/Library/books/manage.do' == listForm.getAttributeValue("Action")

dtolbert about 1 year later:

I can’t thank you enough, you saved me a couple hours of bumbling around with HtmlUnit. I’ve ran into quite an issue involving a Javascript routine that returns a bit of JSON that I can play a bit with to decode into Html. I then wanted to take that Html and create an HtmlPage out if, which I would then in turn parse.

I think I was on the right path. What I believe I was doing wrong was using my existing WebClient object to create the HtmlPage with a StringWebResponse.

I can’t get enough praise to the HtmlUnit library. It truely is a gem and “just works” in most cases.
Fletch over 3 years later:

Thanks for posting this. It saved me a lot of trouble. Some things are a pain in the ass with HtmlUnit, but generally it’s fantastic for web testing and automation.
uselectit.com over 3 years later:

I think the html unit has been very successful so far and internet operators all over the world are very grateful to these service providers who are providing the software that they have hardly developed all over the years for free. The web client and the window and the java script causes problem for some. Anyway as everything has some sort of disadvantages this software may also have them but the point we have to note here is that how many people are benefiting from this software. I think it definitely needs its admiration. It definitely deserves it! Isn’t it?
sohbet over 3 years later:

The web client and the window and the java script causes problem for some. Anyway as everything has some sort of disadvantages this software may also have them but the point we have to note here is that how many people are
Convert youtube to mp3 over 3 years later:

Really glad I came across this
cheap vps over 4 years later:

Anyway as everything has some sort of disadvantages this software may also have them but the point we have to note here is that how many people arecheap VPS
Chantel Buchbinder over 4 years later:

I like the style of your website, it is beautiful, people feel very free
Ray Cruz over 4 years later:
At the very start, I’d prefer to say thanks to you for this informative article. Second, I’d prefer to interrogate wherever I can find greater data concerning your article. I arrived right here through Ask and i can not find any other corresponding web internet sites connected to this matter. How do I subscribe for your web blog? I’d prefer to bind to your updates as they arrive along! I had a query to interrogate but I forgotten what it absolutely was… anyways, thank you. Author of how to cook beef tenderloin
```
Best wishes, Ray Cruz
```
nike over 4 years later:

Your article is write very well, I like it very much. mason0507
fatmagülün suçu ne izle over 4 years later:

Keep in mind that although we typically know what the effects of two or more drugs have the mechanism
turkce sohbet over 4 years later:

thank you for sharing
jfgj over 4 years later:

’d prefer to say thanks to you for this informative article. Second,
Apartments for rent Bucharest over 4 years later:

Hmm…After trying and failing to get JWebUnit and HttpUnit to work with Jquery (1.2.6), I tried this, but the same problems :( …have reported the issue in detail in the bug tracker.

Chris over 4 years later:

Seems like your approach won’t work anymore. I’m doing the following:

StringWebResponse stringWebResponse = new StringWebResponse(htmlString, new URL("http://fakeurl"));
page = HTMLParser.parseHtml(stringWebResponse, webClient.getCurrentWindow());

Designer Bags over 4 years later:

Awesome article!Thanks for ur nice sharing!!
Beats Headphones over 4 years later:

bravo à vous deux !!! pas mal votre idéé de faire sponsorisé votre mariage !!jje vous souhaite tout le bonheur du monde !!! vous etiez magnifique !!! que de beaux souvenirs pour vous !!! soyez heureux bisous odile de marseille
Beats Headphones over 4 years later:

bravo à vous deux !!! pas mal votre idéé de faire sponsorisé votre mariage !!jje vous souhaite tout le bonheur du monde !!! vous etiez magnifique !!! que de beaux souvenirs pour vous !!! soyez heureux bisous odile de marseille
Electronic Book Readers over 4 years later:

i came across this article and just wanted to say it was a great read. and i will be passing the link on to friends and family. p.s fantastic website youv’e done a wonderful job in creating a informative website on subject’s such as these !
iPad video converter for Mac over 4 years later:

Today I see it. and I think I come to the right place. I find it is very informative and give me a lot of pleasure. A wonderful website that can help people make progress.
Harrishcolin over 4 years later:

Thank you for this nice post

my blog: alpha male | how to run faster
carhartt over 4 years later:

Thank you for the post. The version of the module dependency should be selected according to the following rule: The lowest version providing the functionality required by the module (or bundle). By required functionality we bascially mean provided API
ames over 4 years later:

Dependency Management: HtmlUnit , another master piece of blog post. keep it up
Silicone Molding over 4 years later:

Intertech Machinery Inc. provides the most precise Plastic Injection Mold and Rubber Molds from Taiwan. With applying excellent unscrewing device in molds, Intertech is also very professional for making flip top Cap Molds in the world.
masters in criminal justice over 4 years later:

It is quite interesting and useful. You have done a fantastic job here. Wish to see more updates from you.
Rent A Property over 4 years later:

i came across this article and just wanted to say it was a great read. and i will be passing the link on to friends and family. p.s fantastic website youv’e done a wonderful job in creating a informative website on subject’s such as these.You have done a fantastic job here. Wish to see more updates from you.
Criminal Check over 4 years later:

This is such a good information. Keep posting useful and informative articles.
Criminal Records over 4 years later:

I think I was on the right path. What I believe I was doing wrong was using my existing WebClient object to create the HtmlPage with a StringWebResponse.
Tenant Screening over 4 years later:

Anyway as everything has some sort of disadvantages this software may also have them but the point we have to note here is that how many people are benefiting from this software. I think it definitely needs its admiration.
samsin over 4 years later:

Brown baskrar, här var todaypotensmedelshow LA, visa människor potenspiller minoritet här, var potenspiller anglosaxarna, att vi är här, potenspiller majoritet herepotensmedelstay. Vi potenspiller arbete i denna stad, tar vi carepotenshöjandepotenspiller bortskämda brat barn … Var här i Westwood … potensmedelshow vita anglosaxiska protestantiska LA, potenspiller fewpotenshöjandeyou som är potenspiller, att vi är potenspiller majoritet, och vi hävdar detta land som vårt, dess alltid varit vår, och fortfarande var här, och nonepotenshöjandepotenspiller prata om utvisning.
lv briefcase over 4 years later:

lv briefcase have fairly very rated for rather a few of elements just like pattern durability and ease of use
cable ties over 4 years later:

awesome article. great work.
order cigarettes over 4 years later:

Best BMW offers the latest market news and latest reviews BMW car. Here you can follow the events to show the worlds largest car, discover the latest trends on BMW cars and being in touch with BMWs innovation model.
dswehfhh over 4 years later:

We are the professional jacket manufacturer, jacket supplier, jacket factory, welcome you to custom jacket.
Sunglass over 4 years later:

At fashion-world4u you find Imitation Inspired ,MEN designer Sunglasses Women Replica Sunglass at cheap discount price
clothing manufacturer over 4 years later:

erepotensmedelstay. Vi potenspiller arbete i denna stad

cheap true religion los angeles over 4 years later:

hni

Method of washing and maintenance of True Religion Jeans for cheapjeans 1,

How to get accepted over 4 years later:

If I am not here I don’t know what a great miss I would have done. I think you have researched a lot to give me such kind of writing. I would like to see latest update about this issue. I always skip comments of a blog but cannot resist myself to give you special thanks.
clothing manufacturer over 4 years later:

disadvantages this software may also have them but the point we have to note here is that how many people are
viagra senza ricetta over 4 years later:

and i will be passing the link on to friends and family. p.s fantastic website youv’e done a wonderful job in creating a informative website on subject’s such as these.You have done a fantastic job here. Wish to see more updates from you.
ipad bag over 4 years later:

TopCombine Follow ipad bag the detail tips below, you can increase the laptop battery life of a year or more. Game Controllers first thing you should care about the USB Gadgets END!333
SEO Firm India over 4 years later:

Excellently written article, if only all blogger offered the same content as you, the internet would be a much better place. Please keep it up!
viagra bez recepty over 4 years later:

Given my simple needs, why do I care about WebClient and Window. Why do I have to turn off the javascript engine? It may seem a small thing, but it bothers me nonetheless. It’s the principle of the matter that gets under my skin. The pragmatic programmers called it The Principle of Least Surprise. I call it, simply, dependency management. Don’t make people depend on more than they need.
Monster t shirts over 4 years later:

“One of them certainly will be arming the rebels which requires American Special Forces on the ground to do it right,” said Michael O’Hanlon, of the Brookings Institute.
dory over 4 years later:

The blog was absolutely fantastic! Lots of great information and inspiration, both of which we all need. thank you so much.Keep up the good works. Social Network
bird houses for sale over 4 years later:

Find the best bird houses for sale, which are also quite decorative. We also carry an assortment of bird feeders, bird baths and other accessories
Seobaglyak over 4 years later:

Seobaglyak - Verseny
True Religion Jeans Outlet Online over 4 years later:

it is a useful and wonderful website.thanks for your information.
Nike Sneakers Outlet over 4 years later:

Thank you very good and a healthy writing. I’ll definitely keep track of posts and the occasional visit. Looking forward to reading your next publish.Nike Sneakers Outlet
diddy beats headphones over 4 years later:

diddy beats headphones

beats audio headphones
day over 4 years later:

We also carry an assortment of bird feeders, bird baths and other accessories welcomWood Pellets Fuel
programy over 4 years later:

I am not sure where you are getting your information, but great topic
okey oyunu oyna over 4 years later:

thank you so much :)

Dünyan?n en büyük online okey oyunu bu sitede sizleri bekliyor. Gerçek ki?ilerle sohbet ederek Okey Oyunu Oyna ve internette online oyun oynaman?n zevkini ç?kar
http://www.truereligionjean.net/ over 4 years later:

just like before, i know, i will gain more knowledge when i entry into you website.
http://www.coach-factory-outlet-online.com/ over 4 years later:

from your website,i can learn as more as i can ,just as i need it.
http://www.wayshoes.com/ over 4 years later:

i know,as a social people,i must study as possible as i can,then i can learn a lots from you website.thanks.
funny pictures over 4 years later:

yes agreed with the above statement
ford leveling kit over 4 years later:

Thanks for sharing! It really helpful to me about those information!!
leveling kit ford over 4 years later:

Great blog and very nice post, Thanks for sharing this.
leveling kit f250 over 4 years later:

Thanks for your superb post in this blog!
f350 leveling kit over 4 years later:

Great blog, I am very interested of your blogs Thanks for sharing and God Bless!!!
egr kit over 4 years later:

This is such a good information. Keep posting useful and informative articles. Thank you very much for your providing good information on this blog….
egr delete kit over 4 years later:

very cool blogs I really like it Thanks :-)
ford leveling kit over 4 years later:

I want to read more article from your site Thanks :-)
leveling kit ford over 4 years later:

very cool blogs I really like it Thanks :-)
leveling kit f250 over 4 years later:

I want to read more article from your site Thanks :-)
f350 leveling kit over 4 years later:

Wow nice blog
Futons For You over 4 years later:

Given my simple needs, why do I care about WebClient and Window. Why do I have to turn off the javascript engine? It may seem a small thing, but it bothers me nonetheless. It’s the principle of the matter that gets under my skin.
austin bail bonds over 4 years later:

Really impressed! Everything is very open and very clear explanation of issues. It contains truly information. Your website is very useful. Big buddy in the picture.Thanks for sharing. Looking forward to more!
austin defense lawyers over 4 years later:

I definitely desired to deliver a quick concept to thank you for the nice tips and hints you’re posting on this website. My time-consuming internet appear up has now been rewarded with helpful strategies to exchange with my family and friends. I ?d claim that we readers fact are truly blessed to dwell in a helpful community with incredibly some great individuals with insightful points. I really feel really grateful to get discovered the webpage and appear ahead to a lot of additional entertaining moments reading right here. Thanks a good deal again for a good deal of things.
christian louboutin shoes on sale over 4 years later:

Have the christian louboutin patent leather pumps is a happy thing. Here have the most complete kinds of christian louboutin leather platform pumps.
Sildenafil over 4 years later:

Great article. This article is very educational and interesting for me. Thanks for the info, keep it up!
beats by dr dre headphones over 4 years later:

I found that his foot odors never bring us to death.I never regret buying these beats by dr dre studio for him.
Jewellery over 4 years later:

Online UK costume and fashion jewellery shop with, k
Fruta Planta Diet Pills over 4 years later:

welcome to our store for buy the weight loss, we have the best Diet Plan.
Bilal Shahid over 4 years later:

I have been using htmlunit for some time now but after reading this post, I think I might switch to httpunit.
filtration d'eau over 4 years later:

Can I write this in PHP format? It will take less time
retro football shirts over 4 years later:

Your article is really catch my eyes, and help me get much more knowledge, I really hope I could read your posts in future.
router table over 4 years later:

The web client and the window and the java script causes problem for some. Anyway as everything has some sort of disadvantages this software may also have them but the point we have to
Unique Article Wizard over 4 years later:

very nice article ! quite helpful for a newbie like me !
deadbeat millionaire over 4 years later:

very cool blog post ! i’ve been reading this blog for some time and it have cool content!
septic system over 4 years later:

Thanks for the nice written article.I found this thing so much helpful.
complete websites for sale over 4 years later:

This is excellent post. Its having good description regarding this topic.It is informative and helpful.I have known many information from this. Thanks for shearing.

complete websites for sale
Pandora Rings over 4 years later:

Every woman does have a sole desire to have the perfect kind of fashion lady designer jewellery. In essence, there are more than enough fashion jewellery designs available online. One of them is fabulously known as the Pandora Rings. Generally it is a kind of women antique. It is very lovely and versatile kind of jewellery for the gracious looking ladies in the world today. Most the Pandora bracelets can be available in lots of sizzling styles and designs online. For example, there is a demand of wearing the Pandora Bangles around your neck; you should not hesitate to use it on time at all. The greatest advantage of using the Pandora bracelets is that they would abruptly boost up your natural beauty and grace for long time. Greatly online replica jewellery shop offers you greatest kinds of fashion designer replica jewels online.
Coach Tote over 4 years later:

I am happy to have found this blog, and more happy to have long tail explained so well. Thanks.
monicajoy56 over 4 years later:

Nice Article. Wish to get such article again again. Thanks a lot for sharing this.

Scottsdale Custom Home Builders High End Home Builders
austin remodeler over 5 years later:

Dependency management is really important.Thanks for the nice written article being shared here.Love your concept and making it very clearance indeed.
Jewellery Xy over 5 years later:

Nice work uncle Bob. Saved me some time with your information on html unit. Only genuine and authentic designer brand name handbags at discounted prices. Our brands include gucci, guess, calvin klein, diesel, prada
beats by dr dre over 5 years later:

important.Thanks for the nice written article being shared here.Love yourbeats by dr dre beats by dre sale concept and making it very clearance indeed
carpet cleaning over 5 years later:

This is really stunning work.Thanks for this kind of stuff.I wanna say I am totally impressed.Hope to see some more update work here.I have to say it is quite impressing.
Ashley Bowling over 5 years later:

One computer might easily replace 100 white-collar ‘pencil pushers’,
hair extensions cost over 5 years later:

Also prefer to admire the time and work you place into your weblog as well as in level information and details you offer! i am going to bookmark your weblog and also have my youthful children confirm up right here often.
christian louboutin over 5 years later:

Good artical,I learn something!
christian louboutin over 5 years later:

Good artical,I learn something!
LiMelia over 5 years later:

thanks for ur sharing, I like your blog, content is very rich, allow me to leave a message well, wish you are lucky!!!!! http://www.junyuetrade.com/
where and how over 5 years later:

. Perfect ideas for your story and great food for thought
ithemes coupon code over 5 years later:

thanks for ur sharing, I like your blog, content is very rich, allow me to leave a message well, wish you are lucky!!!!!
iolo coupon over 5 years later:

I like this concept. I visited your blog for the first time and just been your fan. Keep posting as I am gonna come to read it everyday. Thanks for sharing.
custom essay paper over 5 years later:

Academic research papers, buy a research paper, buy essay paper online.
psb over 5 years later:

Good stufff… keep it up..
psk over 5 years later:

There is appreciably a lot to know about this. I’m glad reading this content, I very much like and agree with your thoughts.
writing service over 5 years later:

Essay writing company , college term papers, buy research paper .
coach handbags uk over 5 years later:

Its like you read my mind! You appear to know so much about this, like you wrote the book in it or something.
iphone sms to mac backup over 5 years later:

Most of us will delete the SMS file if the iPhone inbox is full. For some of the very important text file, you would like to save it to Mac hard drive and read it later or you need print them. So, why not export the text message to HDD and save it now?
krzyzowki over 5 years later:

I am not sure where you are getting your information, but great topic
krzyzowki edukacyjne over 5 years later:

Thanks for all the tips i got through this blog
youngbrown over 5 years later:

Thanks for the information, I’ll visit the site again to get update information action figures
wholesale tisa hats over 5 years later:

Your Blog is very good, I like it! Thank you for you sharing!Your blog is really helps for my search and i really like it.
louboutin sales over 5 years later:

Dependency Management: HtmlUnit 108 hoo,good article!!I like the post!10
Authorhouse over 5 years later:

I am a frequent visitor of your blog and usually do not leave comments. Just wanted to let you know that I very much enjoy reading your blog. Thanks. Regards,
Injection mold over 5 years later:

Intertech Machinery Inc. provides the most precise Plastic Injection Mold and Rubber Molds from Taiwan. With applying excellent unscrewing device in molds,

Intertech is also very professional for making flip top Cap Molds in the world. Mold making is the core business of Intertech (Taiwan). With world level technology, Intertech enjoys a very good reputation for making Injection Mold and Plastic Molds for their worldwide customers.
hermes kelly birkin over 5 years later:

When you’re getting ready to attend a special occasion like a corporate event, business convention or evening reception, it is important that you find the perfect tuxedo?

Mentor	twitter id
Uncle Bob	unclebobmartin
Brett Schuchert	schuchert
Michael Feathers	mfeathers
Bob Koss	bob_koss