Object Mentor Blog: Dependency Management: HtmlUnit

Dependency Management: HtmlUnit

Uncle Bob — Sun, 11 Feb 2007 12:55:14 -0600

If you are planning on building an API, please, please, think about dependency management. Don’t make me know more about your world view than necessary. Consider what happened to me as I explored HtmlUnit…

I’m using HtmlUnit to parse and interpret HTML web pages. I’ve been very impressed with this library so far. And I appreciate the hard work and dedication of people who give their software away for free. So, although this blog is a complaint, it should not be misconstrued into anything more than constructive criticism. Besides, what I am complaining about here is so universal that it really wouldn’t matter whose software I chose to scrutinize. The HtmlUnit authors just got lucky in this case.

What I want to do with HtmlUnit is quite simple. Given a string containing HTML, I’d like to query that HTML for certain tags and attributes. For example, I’d like to do this:


    HtmlPage page = HTMLParser.parse(htmlString);
    HtmlElement html = page.getDocumentElement();
    HtmlElement listForm = html.getHtmlElementById("list_form");
    assertEquals("/Library/books/manage.do", listForm.getAttributeValue("Action"));

Sweet, simple, uncomplicated. Just create the DOM from an HTML String, and then query that DOM. Unfortunately, HtmlUnit does not appear to be that simple. What you have to do instead looks like this:

    StringWebResponse stringWebResponse = new StringWebResponse(htmlString);
    WebClient webClient = new WebClient();
    webClient.setJavaScriptEnabled(false);
    HtmlPage page = HTMLParser.parse(stringWebResponse, new TopLevelWindow("", webClient));
    HtmlElement html = page.getDocumentElement();
    HtmlElement listForm = html.getHtmlElementById("list_form");
    assertEquals("/Library/books/manage.do", listForm.getAttributeValue("Action"));

The extra stuff in here is apparently due to the fact that the authors wanted to be able to simulate browsers, frames, and javascript. I think their goal was laudable. However, I wish they had done this without forcing those frames, browsers, and script engines down my throat.

Given my simple needs, why do I care about WebClient and Window. Why do I have to turn off the javascript engine? It may seem a small thing, but it bothers me nonetheless. It’s the principle of the matter that gets under my skin. The pragmatic programmers called it The Principle of Least Surprise. I call it, simply, dependency management. Don’t make people depend on more than they need.

The cost, to me, was an hour of rooting around in the documentation, example code, and my own trial-and-error experiments. (The benefit to me was another blog topic ;-) That cost may not seem great; but it must be paid again and again by everyone who wants to use the package in a way that doesn’t quite fit the authors’ world view.

There may, in fact, be a simpler way to do what I want to do with HtmlUnit. If there is, I haven’t been able to find it, and I’d be grateful if anyone out there, including the authors, could guide me in the right direction.

"Dependency Management: HtmlUnit" by coach handbags uk

Mon, 06 Feb 2012 18:54:17 -0600

Its like you read my mind! You appear to know so much about this, like you wrote the book in it or something.

"Dependency Management: HtmlUnit" by writing service

Mon, 30 Jan 2012 05:22:02 -0600

Essay writing company , college term papers, buy research paper .

"Dependency Management: HtmlUnit" by psk

Thu, 26 Jan 2012 01:10:03 -0600

There is appreciably a lot to know about this. I’m glad reading this content, I very much like and agree with your thoughts.

"Dependency Management: HtmlUnit" by psb

Mon, 23 Jan 2012 23:03:47 -0600

Good stufff… keep it up..

"Dependency Management: HtmlUnit" by custom essay paper

Fri, 20 Jan 2012 02:30:31 -0600

Academic research papers, buy a research paper, buy essay paper online.

"Dependency Management: HtmlUnit" by iolo coupon

Thu, 12 Jan 2012 03:07:10 -0600

I like this concept. I visited your blog for the first time and just been your fan. Keep posting as I am gonna come to read it everyday. Thanks for sharing.

"Dependency Management: HtmlUnit" by ithemes coupon code

Wed, 11 Jan 2012 15:40:02 -0600

thanks for ur sharing, I like your blog, content is very rich, allow me to leave a message well, wish you are lucky!!!!!

"Dependency Management: HtmlUnit" by where and how

Tue, 27 Dec 2011 19:36:53 -0600

. Perfect ideas for your story and great food for thought

"Dependency Management: HtmlUnit" by LiMelia

Sun, 27 Nov 2011 04:33:11 -0600

thanks for ur sharing, I like your blog, content is very rich, allow me to leave a message well, wish you are lucky!!!!! http://www.junyuetrade.com/

"Dependency Management: HtmlUnit" by christian louboutin

Sun, 30 Oct 2011 06:31:15 -0500

Good artical,I learn something!

"Dependency Management: HtmlUnit" by christian louboutin

Sun, 30 Oct 2011 05:20:40 -0500

Good artical,I learn something!

"Dependency Management: HtmlUnit" by hair extensions cost

Tue, 25 Oct 2011 04:16:19 -0500

Also prefer to admire the time and work you place into your weblog as well as in level information and details you offer! i am going to bookmark your weblog and also have my youthful children confirm up right here often.

"Dependency Management: HtmlUnit" by Ashley Bowling

Sun, 16 Oct 2011 05:32:07 -0500

One computer might easily replace 100 white-collar ‘pencil pushers’,

"Dependency Management: HtmlUnit" by carpet cleaning

Thu, 13 Oct 2011 13:08:04 -0500

This is really stunning work.Thanks for this kind of stuff.I wanna say I am totally impressed.Hope to see some more update work here.I have to say it is quite impressing.

"Dependency Management: HtmlUnit" by beats by dr dre

Mon, 29 Aug 2011 21:46:24 -0500

important.Thanks for the nice written article being shared here.Love yourbeats by dr dre beats by dre sale concept and making it very clearance indeed

"Dependency Management: HtmlUnit" by Jewellery Xy

Sun, 21 Aug 2011 14:02:57 -0500

Nice work uncle Bob. Saved me some time with your information on html unit. Only genuine and authentic designer brand name handbags at discounted prices. Our brands include gucci, guess, calvin klein, diesel, prada

"Dependency Management: HtmlUnit" by austin remodeler

Fri, 19 Aug 2011 10:07:39 -0500

Dependency management is really important.Thanks for the nice written article being shared here.Love your concept and making it very clearance indeed.

"Dependency Management: HtmlUnit" by monicajoy56

Sun, 10 Jul 2011 13:20:37 -0500

Nice Article. Wish to get such article again again. Thanks a lot for sharing this.

Scottsdale Custom Home Builders High End Home Builders

"Dependency Management: HtmlUnit" by Coach Tote

Fri, 08 Jul 2011 03:49:56 -0500

I am happy to have found this blog, and more happy to have long tail explained so well. Thanks.

"Dependency Management: HtmlUnit" by Pandora Rings

Fri, 08 Jul 2011 03:35:50 -0500

Every woman does have a sole desire to have the perfect kind of fashion lady designer jewellery. In essence, there are more than enough fashion jewellery designs available online. One of them is fabulously known as the Pandora Rings. Generally it is a kind of women antique. It is very lovely and versatile kind of jewellery for the gracious looking ladies in the world today. Most the Pandora bracelets can be available in lots of sizzling styles and designs online. For example, there is a demand of wearing the Pandora Bangles around your neck; you should not hesitate to use it on time at all. The greatest advantage of using the Pandora bracelets is that they would abruptly boost up your natural beauty and grace for long time. Greatly online replica jewellery shop offers you greatest kinds of fashion designer replica jewels online.

"Dependency Management: HtmlUnit" by complete websites for sale

Thu, 07 Jul 2011 06:05:30 -0500

This is excellent post. Its having good description regarding this topic.It is informative and helpful.I have known many information from this. Thanks for shearing.

complete websites for sale

"Dependency Management: HtmlUnit" by septic system

Sat, 02 Jul 2011 12:12:23 -0500

Thanks for the nice written article.I found this thing so much helpful.

"Dependency Management: HtmlUnit" by deadbeat millionaire

Fri, 17 Jun 2011 15:40:34 -0500

very cool blog post ! i’ve been reading this blog for some time and it have cool content!

"Dependency Management: HtmlUnit" by Unique Article Wizard

Fri, 17 Jun 2011 14:44:23 -0500

very nice article ! quite helpful for a newbie like me !

"Dependency Management: HtmlUnit" by router table

Thu, 16 Jun 2011 05:33:09 -0500

The web client and the window and the java script causes problem for some. Anyway as everything has some sort of disadvantages this software may also have them but the point we have to