Thirteen years ago, while working on my first book, Jim Coplien and I were having a debate on the nature of threads and objects. He made a clarifying statement that has stuck with me since. He said: “An object is an abstraction of function. A thread is an abstraction of schedule.”

It has become the norm, in Java applications, to assume that there is a one-to-one correspondence between a thread, and a unit-of-work. This appears to make sense since every Servlet request has it’s own particular thread. Framework authors have built on this assumption by putting unit-of-work related information (e.g. session) into ThreadLocal variables.

Clearly, the more ThreadLocal variables that hold unit-of-work related information, the more that the thread and the unit-of-work are related. While very convenient, the basic assumption is dead wrong.

A unit-of-work is a task to be performed. There is no rule that says that this task must be single threaded. Nor is there any rule that says that the components of the task must run at the same priority. Indeed, it is not uncommon for a task to have a high-priority event driven part, and a low priority compute-bound part.

Consider, for example, a unit-of-work that makes a request from a third-party-service, and then processes the response with a complex number-crunching algorithm. We want the request to run at a high priority so that it is not waiting for other low priority compute-bound tasks to finish. On the other hand, we want the number-cruncher to run at a low priority so that it yields the processor as soon as possible whenever one of the high priority requests needs a few cycles.

Clearly these two tasks form a complete unit-of-work, but just as clearly they should run as two different threads—probably with a queue in between them in a standard producer-consumer model.

Where are the unit-of-work related variables? They can’t be kept in a ThreadLocal since each part of the task runs in a separate thread. They can’t be kept in static variables since there is more than one thread. The answer is that they have to be passed around between the threads as function arguments on the stack, and recorded in the data structured placed on the queue.

So, though convenient, ThreadLocal variables confuse the issue of separating function from schedule. They tempt us to couple function and schedule together. This is unfortunate since the correspondence of function and schedule is weak and accidental.

What we’d really like is to be able to create UnitOfWorkLocal variables.

In the spring of 2000 Kent Beck called a meeting at the Rogue River Lodge near his home in Medford, Oregon. He called it the Extreme Programming Leadership Conference. In attendance were: myself, Ron Jeffries, Ken Auer, Martin Fowler, and several others who had been instrumental in getting the XP movement off the ground.

In one of the sessions we discussed the creation of an organization to drive and support the adoption of XP. This was an idea that I argued in favor of. It seemed to me that the industry would benefit from a body who promoted the ideas of lightweight methods like XP. However, the idea was not met with enthusiasm amongst many of the other attendees. The session turned into a collection of painful reminiscences about a previous attempt at creating an organization around the Patterns movement. I came away convinced that a different group of people would have to drive the creation of such an organization.

Apparently I was not alone. Martin Fowler had noticed the arguments I had made in the session, and approached me during a break. We briefly discussed the idea of broadening the scope of the proposed organization to all the various “lightweight methods” such as Scrum and Crystal Light. We felt that the folks who espoused these methods would have a lot to add, and could form the core of a comprehensive movement. Martin and I determined to meet a few weeks later to work out the details.

In the meantime I contacted my friend and advisor Dave Thomas of OTI fame. (We call him “Big Dave” to differentiate him from the Pragmatic Programmer of the same name.) I mentioned the notion of a Lightweight Methods Summit. We put a tentative invitation list together. Dave suggested that we meet in February at his winter home in Anguilla.

Later that fall I met with Martin at a cafe in Chicago. We added to the invitation list and outlined the invitation email. The invitation stated the goal of creating a manifesto describing what all the various lightweight methods had in common. I sent the email that evening.

The Lightweight Process Summit invitation drew an enthusiastic response from the invitees. Alistair Cockburn was especially interested. As it happened, he was on the verge of sending out his own invitation for a similar event. His comment to me was: “Your invitation list was better than mine”. The two invitation lists were merged, and the discussion of logistics began.

Alistair became the de-facto organizer. He suggested that we change the venue from Anquilla to Snowbird in Salt Lake City. Almost all the other invitees agreed because the flights would be easier. Alistair and Jim Highsmith did the leg-work to arrange the rooms, meals, and activities. Things came together rapidly.

The meeting was very well attended by 17 out of ~20 invitees. (See www.agilemanifesto.org to see the list of attendees.) Unfortunately neither Grady Booch nor Big Dave Thomas were able to make it. However, their influence was strongly felt in the subsequent discussions.

I kicked off the meeting with a brief introduction and an appeal to the similarities between all the various lightweight processes. I reaffirmed that the goal of our two day meeting was to create a manifesto that outlined those similarities. This was to be a document, addressed to the industry at large, describing our views about the process of creating useful software.

Martin Fowler and Ward Cunningham quickly became the de-facto facilitators of the meeting. With their help we quickly worked out an agenda for the two days, and a method for making decisions.

It was actually quite a thrill to see all these people with disparate ideas working together so well. I don’t think I’ve ever been to a meeting that stayed on point and met it’s goals with such ease and with so little conflict. It was as though the pieces just somehow fell together.

One of the early discussions was about a name. No one liked the term Lightweight. Several other options were suggested, including Lean, and Adaptive. But the name Agile won the day.

The structure of the manifesto was mutually agreed; I recall that Ward played a big part framing the idea of pairs of relative value, but it may have been Martin and Pragmatic Dave Thomas who came up with the idea. Here’s PragDave’s recollection:

Actually, that was Martin and me, noodling on the whiteboard over lunch. I think we came up with the initial three. The group that then formed took it up to five, and that was then pruned down to four. In fact, Ward took the picture of everyone discussing the idea that now adorns his manifesto site.

By the end of the two days in Snowbird, the manifesto was done, the principles were outlined, and the Agile Alliance had been born. Much more would need to be done before there would be a true legal entity with a board and members, but that’s a story for another time.

For another version of this history see Jim Highsmith’s description at http://www.agilemanifesto.org/history.html

I’ve had it. What is the matter with these people? How, after all the experience we’ve had with XSLT, Ant, WSDL, etc., etc., could they create YET ANOTHER XML language. Are they dolts? Are they idiots? What gives.

Look, writing in XML is hideous. It’s wordy, it’s error-prone, it’s arcane, it’s redundant, it’s redundant, it’s redundant, it’s… HIDEOUS! To make matters worse, we have been embedding OTHER languages INSIDE this horrible container. EGAD! YIKES! ZOUNDS! FORSOOTH! This is just plain nuts, stupid, idiotic, retarded, poo-poo-headed, silliness!

Haven’t these people heard of economy of expression? Haven’t they heard of YACC? Don’t they know that domain specific languages SHOULD BE LANGUAGES? Don’t they know that languages have specific GRAMMARS?

Besides, have they ever tried to write an interpreter or compiler that uses XML as it’s source? It’s not any easier than writing a YACC parser! Indeed, it’s a lot, lot, harder.

EMBEDDING THE GRAMMAR OF YOUR DSL IN XML IS STUPID!!! DON’T DO IT!!!!!#$#$!!!

I hereby declare a revolt. From now on anyone who considers themselves to be a serious professional must refuse to write another line of XML. When asked, say NO. Instead, write a little YACC grammar that is nice, and small, and translates into that hideous XML. You’ll save yourself GOBS of time if you do! What’s more, if you sell your parser for $5 per download, you’ll probably be able to buy a new boat! Maybe a fleet!

Of course this means that the members of the team will have to be mature enough to realize that it doesn’t really matter where they put their braces, or how they adorn their member variables. What matters is that they all use the same conventions.

Consistency

My goal for a good coding standard is to eliminate individual styles in favor of a team style. The code produced by a team should look like the team produced it. I don’t want any code recognizable as Bob’s or Bill’s.

This is not some egalitarian fantasy to hide individuality for the sake of the collective. Rather, it is a raw necessity. We’ve all seen products that look like they were designed by a committee. We’ve all used software products where the look and feel changed depending on which part of the application you were using. The result feels messy, clumsy, inefficient.

Individuals, used to their own particular style, will reformat other people’s code when forced to work on it, further shuffling the patchwork of styles. Over time, as each team member touches different parts of the code, and team members come and go from the team, the code begins to look like a jumbled Rubick’s cube of different styles.

Code is a product, in and of itself. The team producing it needs to take pride in the elegance of it’s structure, and the expressiveness of it’s presentation. This kind of pride is infeasible when the code is crisscrossed with a patchwork of individual styles. Without this pride, there is no drive to keep the overall product clean. Without that cleanliness, messes build up at the boundaries. And, as we all know, messes slow us down, and they spread.

Style not Substance

A good coding standard should be about style and form, not about substance. It should not attempt to legislate good design. It should not, for example, proscribe goto or public variables. Those rules are part of the body of knowledge that all software developers should have, and are not a matter of style.

Coding standards should be about the things that don’t matter. They should be about brace placement, naming conventions, the use of blank lines, indentation levels, etc. A coding standard should describe the way code looks, not the substance the code is made from.

It is important to keep style and substance separate because they matter for different reasons. Issues of style matter only for consistency. It does not matter whether your indent depth is 2 or 4, so long as everyone uses the same depth. It does matter if you use public variables inappropriately.

Oral/Code Tradition

Documents that describe coding standards tend to be useless. They often become a bloated battleground for many different competing ideas. My advice is to avoid writing them.

The real document that describes your coding standard is your code. If you want to know how to name a variable, look at how they are named in the code. If you want to know what the standard indent depth is, look in the code. The code is the living document that describes the coding standard.

Oral tradition plays a role as well; especially when communicating issues of substance vs. style. Teams should make use of code reviews and pair programming to communicate with each other about issues of style and substance. New members of the team should have frequent exposure to the more seasoned members, so that the issues of style and substance are inculcated with moral authority. Nothing is quite as persuasive to a young programmer than pairing with the lead programmer and hearing him say: “We don’t do things that way; we do things this way.”

The Tyranny of Tools

I have seen teams attempt to enforce a style through the use of tools. Some tools are benign and helpful. Many IDEs, for example, allow you to specify things like indent level, brace placement, etc. With a single keystroke you can ensure that a batch of code conforms to the team style. I use tools like this, an depend upon them. I make sure that all the team members set their IDEs to use the conventions.

Other tools can be more intrusive. Some tools can act like compilers, generating errors if the style is not adhered to. Such tools might be useful for an occasional scan of the code; but I think you have to be very careful if you put them into the normal build cycle.

Automatic enforcement is power; and power corrupts. We do not want a well-meaning bureaucrat deciding, one day, to enforce the style that every function argument must have a javadoc comment. This leads to comments of the form: //required comment.

This is not to say that tools like findbugs and checkstyle should be avoided. Indeed, I find them very useful. However, I think they should be run occasionally and manually, not as part of every build. The issues that these tools discover should be dealt with on a case-by-case basis.

Many tools like this allow you to insert special meta-comments that override the warnings. If these tools are placed in the build process; then the code will become cluttered with these meta-comments.

I have a pathological distaste for meta-comments.

Conclusion

Coding style is a matter of team pride and team identity. Teams should be free to adopt their own styles, and to change those styles as the spirit moves them. Each member of the team should follow the team style, and work to ensure that the body of code is a consistent statement of that style. If this sounds too artsy-fartsy, keep in mind that pride of workmanship is a powerful motivator. We want teams to be proud of their creations.

There are things in a business that don’t change very often. Gas stations in the U.S., for example, still sell gasoline by the gallon. Restaurants still sell meals from a menu. Dentists still sell cleanings every 6 months. Every business has these aspects that don’t change very frequently. They often represent a huge part of the business. We’ll call these things the _core business functions.

There are other things in a business that change very frequently. Prices, tax rates, catalogs, new products, new marketing campaigns, advertising, new business areas, new customer areas, etc. Indeed, businesses must be able to change, and change quickly, in order to survive. And yet, it is vital that those changes do not adversely affect the core business functions.

Software developers have known for years that software that changes frequently should be decoupled from software that changes infrequently. When applied to individual programs and systems this principle is sometimes called The Common Closure Principle. When it is applied to the information management of an enterprise, it is called SOA.

SOA is the practice of sequestering the core business functions into independent services that don’t change frequently. These services are glorified functions that are called by one or more presentation programs. The presentation programs are volatile bits of software that present data to, and accept data from, various users.

To make this clear, imagine an internet store-front. Customers use a browser to talk to the presentation software that displays the store’s website. The presentation software interprets the gestures of the customer and invokes services that do things like acquiring the data for the current catalog, or registering the customer’s order. Note that the services have no idea they are talking to a website. They could just as well be talking to a thick client, or a 3270 green screen. They simply accept and return data in a standard format that the web system happens to be able to use.

That’s really all there is to it. The rest of SOA is just a matter of details. At the highest level, SOA is nothing more (and nothing less) than separating changeable elements from unchangeable elements. But why is this important?

Consider that internet store-front again. It presents the user with a catalog, allows the user to move items into, and out of a shopping cart, and accepts the eventual order. The presentation of these concepts is very volatile. Marketing people are likely to want to change it frequently. For example, they might want to change from a shopping cart metaphor to scrollable receipt on the sidebar. They may wish to present more or less descriptive data in the product list. They may want to experiment with different colors, font-faces, and layouts. Indeed, it’s feasible that they’ll want to try applets, JStart clients, Ajax, and a myriad of other presentation options. But none of this has anything to do with the core business functions encapsulated by the services. Those services that acquire catalogs and register orders remain unchanged despite all the presentation thrashing. That’s why the separation is important. It protects the information processing assets of the business from the constant jitter and spin of the presentation.

But presentation is not the only thing that jitters and spins. So do the business processes. Again, consider our store-front. Perhaps our business has decided to offer fine wines as one of the products it sells. Selling alcohol requires that the age of the customer be verified. Let us say that we have a service that provides this verification. This service must be called for any order that contains alcohol products. The decision to call this service is neither a presentation decision, nor a service decision. Rather it is part of the business process for a particular kind of order. Business processes are volatile and they breed like rabbits. As businesses evolve they add more and more steps and forks to their business processes. The services being used by those processes don’t change much; but the pathways through the processes do. Therefore we want to separate the business process from the services and from the presentation. Smalltalkers had a name for this separation when it appeared in a single program. They called it Model-View-Controller.

Notice that we have yet to mention even one of the plethora of technologies that are so commonly associated with SOA. That’s because SOA is not about any particular technology. Rather it is a design philosophy that decouples well heeled business functions from volatile processes and presentations. It is the MVC of enterprise software.

In my next blog on this topic, we’ll look at the next level of detail in an attempt to understand HOW services can be constructed, and how the decoupling of presentation, process, and functions can be achieved.

Testing JSP pages.

One of the more annoying aspects of working on a web application is that you have to deploy it in order to test it. This doesn’t apply to everything of course; if you are careful with your design you can test the business rules in plain old java objects. You can test database access, interface layers, and stored procedures without the web server running. But testing the GUI – the HTML produced from the JSP files – is very hard to do unless the system has been deployed.

Lot’s of teams fall back on Selenium, Mercury, or other tools that test GUIs through the web server. However, this leads to very fragile tests that break when the format of a page changes, even though it’s content remains unchanged. Other teams solve the fragility problem by using Cactus, or the somewhat more primitive tools in HtmlUnit and HttpUnit, to inspect the generated HTML delivered to them by the running web application. We’ll talk about those techniques in another blog in this series.

In this article I will demonstrate a simple technique for testing JSP pages using JUnit and HtmlUnit, completely outside the container. The advantages of such a technique should be clear.

• You don’t have to have the container running, or even working. You can test your JSPs before you have even chosen a particular webserver.
• You can spin around the edit/compile/test loop much more quickly because you don’t have to redeploy after every edit.
• You can build your JSPs incrementally using Test Driven Development to guide their construction.

The reason that testing JSPs outside the container isn’t more common is because JSPs are designed to run inside the container. The designers never gave a lot of thought to running them outside the container. So the code generated by the JSP compiler depends on facilities supplied by the container. Even the tools that generate the JSP code expect that you already have a webapp that can be deployed. Therefore, in order to run outside the container, you somehow have to fake these facilities and tools.

Dependency Management Rant.

Compiling JSPs

The first step to testing JSPs is to compile them into servlets. To do that, we first have to translate them from JSP format to JAVA. The apache project provides a tool named Jasper that does this. Invoking Jasper on a JSP named MyPage.jsp creates a JAVA source file named MyPage_jsp.java. This file can then be compiled into a servlet using your favorite IDE. (Mine is IntelliJ).

Unfortunately Jasper was not designed to be run from the command line. Or rather, half of it was, and half of it wasn’t. Jasper does in fact have a main function that takes standard command line arguments. And it is feasible to issue the command java org.apache.jasper.JspC to invoke it. However, Jasper expects the environment it runs in to be consistent with the environment of the container. It expects many the apache JAR files to be in the CLASSPATH. It also expects to see a web.xml file that describes the web application, and it wants to see a WEB-INF directory with all the appropriate web application JAR files and TLD files present. In short, Jasper expects there to be a webapp.

To make matters worse, there appear to be bugs in the version of Jasper that I am using (tomcat 5.5.20) that cause it to generate slightly incorrect code unless it is invoked in a manner that is consistent with the way that TOMCAT invokes it.

The first issue is inconvenient but is just a matter of creating the appropriate directories and files and then invoking Jaser from ANT so that the classpaths are easy to control. The second issue required a fair bit of research and experimentation to get working right. But in the end, the following ant file seems to work quite nicely. Look at the last task to see the JspC invocation.

<project name="Library" default="compile" basedir=".">
  <property environment="env"/>
  <property name="build.home" value="${basedir}/build"/>
  <property name="build.war.home" value="${build.home}/war"/>
  <property name="build.classes.home" value="${build.home}/classes"/>
  <property name="build.jar.home" value="${build.home}/jars"/>
  <property name="catalina.home" value="${env.CATALINA_HOME}"/>
  <property name="dist.home" value="${basedir}/dist"/>
  <property name="web.home" value="${basedir}/web"/>

  <path id="compile.classpath">
    <fileset dir="lib">
      <include name="*.jar"/>
    </fileset>
    <pathelement location="${catalina.home}/common/classes"/>
    <fileset dir="${catalina.home}/common/endorsed">
      <include name="*.jar"/>
    </fileset>
    <fileset dir="${catalina.home}/common/lib">
      <include name="*.jar"/>
    </fileset>
    <pathelement location="${catalina.home}/shared/classes"/>
    <fileset dir="${catalina.home}/shared/lib">
      <include name="*.jar"/>
    </fileset>
  </path>

  <target name="clean">
    <delete dir="${build.home}"/>
    <delete dir="${dist.home}"/>
  </target>

  <target name="compile">
    <mkdir dir="${build.classes.home}"/>
    <javac srcdir="${src.home}" destdir="${build.classes.home}" excludes="**/*Test.java">
      <classpath refid="compile.classpath"/>
    </javac>
  </target>

  <target name="jar" depends="compile">
    <mkdir dir="${build.jar.home}"/>
    <jar jarfile="${build.jar.home}/application.jar" basedir="${build.classes.home}" includes="**/application/**/*.class" />
  </target>

  <target name="dist" depends="jar">
    <copy todir="${build.war.home}">
      <fileset dir="${web.home}"/>
    </copy>

    <copy todir="${build.war.home}/WEB-INF/lib">
      <fileset dir="${build.jar.home}" includes="*.jar"/>
    </copy>

    <mkdir dir="${dist.home}"/>
    <jar jarfile="${dist.home}/${app.name}.war" basedir="${build.war.home}"/>
  </target>

  <target name="jsp" depends="dist">
    <delete dir="${basedir}/testjsp"/>
    <java classname="org.apache.jasper.JspC" fork="true">
      <arg line="-v -d ${basedir}/testjsp -p com.objectmentor.library.jsp -mapped -compile -webapp ${build.war.home}"/>
      <arg line="WEB-INF/pages/patrons/books/loanRecords.jsp"/>
      <classpath>
        <fileset dir="${catalina.home}/common/lib">
          <include name="*.jar"/>
        </fileset>
        <fileset dir="${catalina.home}/server/lib">
          <include name="*.jar"/>
        </fileset>
        <fileset dir="${catalina.home}/bin">
          <include name="*.jar"/>
        </fileset>
        <fileset dir="${build.war.home}/WEB-INF/lib">
          <include name="*.jar"/>
        </fileset>
        <pathelement location="/Developer/Java/Ant/lib/ant.jar"/>
      </classpath>
    </java>
    <jar jarfile="${build.jar.home}/jsp.jar" basedir="${basedir}/testjsp" 
         includes="**/jsp/**/*.class" 
      />
  </target>
</project>

Of course you need all the standard files and directories beneath ${build.war.home} to make this work. If you are using custom tags in your JSPs, make sure you have all the appropriate TLD files in your tld directory.

Notice that the ANT file invokes the JspC command line, rather than using the JspC ant task that comes with TOMCAT. All the docs will tell you to use that ant task. However, I found that it doesn’t work well when you have custom tags. Maybe I’m just dumb, or maybe there is a bug in Jasper; but the only way I found to get Jasper to generate the right code was to invoke it from the command line and explicitly pass the JSP files in as command line argument!. If you depend on either the ant task or the command line to scan the whole web app for all the JSP files, it seems to generate the wrong code. (See: this blog)

Now that we have a JAVA file, let’s take a look at it. First, here’s the JSP file.

<%@ page import="com.objectmentor.library.utils.DateUtil" %>
<%@ page import="com.objectmentor.library.web.controller.patrons.LoanRecord" %>
<%@ page import="java.util.List" %>
<%
  List loanRecords = (List) request.getAttribute("loanRecords");
  if (loanRecords.size() > 0) {
%>
<table class="list" id="loanRecords">
  <tr>
    <th>ID</th>
    <th>Title</th>
    <th>Due date</th>
    <th>Fine</th>
  </tr>
  <%
    for (int i = 0; i < loanRecords.size(); i++) {
      LoanRecord loanRecord = (LoanRecord) loanRecords.get(i);
  %>
  <tr class="<%=i%2==0?"even":"odd"%>">
    <td><%=loanRecord.id%>
    </td>
    <td><%=loanRecord.title%>
    </td>
    <td><%=DateUtil.dateToString(loanRecord.dueDate)%>
    </td>
    <td><%=loanRecord.fine.toString()%>
    </td>
  </tr>
  <%
    }
  %>
</table>
<%
  }
%>

And here is the code that Jasper created from this file.

package com.objectmentor.library.jsp.WEB_002dINF.pages.patrons.books;

import javax.servlet.*;
import javax.servlet.http.*;
import javax.servlet.jsp.*;
import com.objectmentor.library.utils.DateUtil;
import com.objectmentor.library.web.controller.patrons.LoanRecord;
import java.util.List;

public final class loanRecords_jsp extends org.apache.jasper.runtime.HttpJspBase
    implements org.apache.jasper.runtime.JspSourceDependent {

  private static java.util.List _jspx_dependants;

  public Object getDependants() {
    return _jspx_dependants;
  }

  public void _jspService(HttpServletRequest request, HttpServletResponse response)
        throws java.io.IOException, ServletException {

    JspFactory _jspxFactory = null;
    PageContext pageContext = null;
    HttpSession session = null;
    ServletContext application = null;
    ServletConfig config = null;
    JspWriter out = null;
    Object page = this;
    JspWriter _jspx_out = null;
    PageContext _jspx_page_context = null;

    try {
      _jspxFactory = JspFactory.getDefaultFactory();
      response.setContentType("text/html");
      pageContext = _jspxFactory.getPageContext(this, request, response,
                  null, true, 8192, true);
      _jspx_page_context = pageContext;
      application = pageContext.getServletContext();
      config = pageContext.getServletConfig();
      session = pageContext.getSession();
      out = pageContext.getOut();
      _jspx_out = out;

      out.write('\n');
      out.write('\n');
      out.write('\n');

  List loanRecords = (List) request.getAttribute("loanRecords");
  if (loanRecords.size() > 0) {

      out.write("\n");
      out.write("<table class=\"list\" id=\"loanRecords\">\n");
      out.write("  <tr>\n");
      out.write("    <th>ID</th>\n");
      out.write("    <th>Title</th>\n");
      out.write("    <th>Due date</th>\n");
      out.write("    <th>Fine</th>\n");
      out.write("  </tr>\n");
      out.write("  ");

    for (int i = 0; i < loanRecords.size(); i++) {
      LoanRecord loanRecord = (LoanRecord) loanRecords.get(i);

      out.write("\n");
      out.write("  <tr class=\"");
      out.print(i%2==0?"even":"odd");
      out.write("\">\n");
      out.write("    <td>");
      out.print(loanRecord.id);
      out.write("\n");
      out.write("    </td>\n");
      out.write("    <td>");
      out.print(loanRecord.title);
      out.write("\n");
      out.write("    </td>\n");
      out.write("    <td>");
      out.print(DateUtil.dateToString(loanRecord.dueDate));
      out.write("\n");
      out.write("    </td>\n");
      out.write("    <td>");
      out.print(loanRecord.fine.toString());
      out.write("\n");
      out.write("    </td>\n");
      out.write("  </tr>\n");
      out.write("  ");

    }

      out.write("\n");
      out.write("</table>\n");

  }

    } catch (Throwable t) {
      if (!(t instanceof SkipPageException)){
        out = _jspx_out;
        if (out != null && out.getBufferSize() != 0)
          out.clearBuffer();
        if (_jspx_page_context != null) _jspx_page_context.handlePageException(t);
      }
    } finally {
      if (_jspxFactory != null) _jspxFactory.releasePageContext(_jspx_page_context);
    }
  }
}

Rant About Final.

A quick perusal of this code tells us that in order to use an instance of this servlet we need an HttpServletRequest instance, and an HttpServletResponse instance.

Looking a little closer we find that the servlet writes all the HTML to an instance of JspWriter that it gets from something called a PageContext. So, if we could create a mocked up version of JspWriter that saves all this HTML, and a mocked up version of PageContext that delivers the mocked JspWriter, then we could access the written HTML from our tests.

Fortunately the TOMCAT designers decoupled the creation of the JspWriter into a factory named JspFactory. That factory can be overridden! This means that we can get our mocked up JspWriter into the servlet without modifying the servlet. All we need is something like the following code:

  class MockJspFactory extends JspFactory {
    public PageContext getPageContext(Servlet servlet, ServletRequest servletRequest, ServletResponse servletResponse, String string, boolean b, int i, boolean b1) {
      return new MockPageContext(new MockJspWriter());
    }

    public void releasePageContext(PageContext pageContext) {
    }

    public JspEngineInfo getEngineInfo() {
      return null;
    }
  }

Now we need the mocked JspWriter. For the purposes of this demonstration, I used the following:

MockJspWriter

package com.objectmentor.library.web.framework.mocks;

import javax.servlet.jsp.JspWriter;
import java.io.IOException;

public class MockJspWriter extends JspWriter {

  private StringBuffer submittedContent;

  public MockJspWriter(int bufferSize, boolean autoFlush) {
    super(bufferSize, autoFlush);
    submittedContent = new StringBuffer();
  }

  public String getContent() {
    return submittedContent.toString();
  }

  public void print(String arg0) throws IOException {
    submittedContent.append(arg0);
  }

  public void write(char[] arg0, int arg1, int arg2) throws IOException {
    for (int i=0; i<arg2; i++)
      submittedContent.append(String.valueOf(arg0[arg1++]));
  }

  public void write(String content) throws IOException {
    submittedContent.append(content);
  }

  // lots of uninteresting methods elided.  I just gave them
  // degenerate implementations.  (e.g. {})
}

Don’t be concerned about the comment regarding all the unimplemented methods I elided. I’ve taken the attitude that I will only implement those methods that I need to get my tests to work. The others I leave with degenerate implementations.

My IDE was very helpful in creating these mocks. It automatically builds method prototypes and degenerate implementations for all the methods of an interface or abstract class that need implementing.

The MockPageContext, MockHttpServletRequest and MockHttServletResponse classes were created in a similar way.

MockPageContext

package com.objectmentor.library.web.framework.mocks;

import javax.servlet.*;
import javax.servlet.http.*;
import javax.servlet.jsp.*;
import java.io.IOException;
import java.util.Enumeration;

public class MockPageContext extends PageContext {

  private final JspWriter out;
  private HttpServletRequest request;

  public MockPageContext(JspWriter out) {
    this.out = out;
    request = new MockHttpServletRequest();
  }

  public JspWriter getOut() {
    return out;
  }

  public ServletRequest getRequest() {
    return request;
  }
  // lots of degenerate functions elided.
}

MockHttpServletRequest

package com.objectmentor.library.web.framework.mocks;

import javax.servlet.*;
import javax.servlet.http.*;
import java.io.*;
import java.security.Principal;
import java.util.*;

public class MockHttpServletRequest implements HttpServletRequest {

  private String method;
  private String contextPath;
  private String requestURI;
  private HttpSession session = new MockHttpSession();
  private Map parameters = new HashMap();
  private Map attributes = new HashMap();

  public MockHttpServletRequest(String method, String contextPath,
                                String requestURI) {
    super();
    this.method = method;
    this.contextPath = contextPath;
    this.requestURI = requestURI;
  }

  public MockHttpServletRequest() {
    this("GET");
  }

  public MockHttpServletRequest(String method) {
    this(method, "/Library", "/Library/foo/bar.jsp");
  }

  public String getContextPath() {
    return contextPath;
  }

  public String getMethod() {
    return method;
  }

  public String getRequestURI() {
    return requestURI;
  }

  public String getServletPath() {
    return requestURI.substring(getContextPath().length());
  }

  public HttpSession getSession() {
    return session;
  }

  public HttpSession getSession(boolean arg0) {
    return session;
  }

  public Object getAttribute(String arg0) {
    return attributes.get(arg0);
  }

  public String getParameter(String arg0) {
    return (String) parameters.get(arg0);
  }

  public Map getParameterMap() {
    return parameters;
  }

  public Enumeration getParameterNames() {
    return null;
  }

  public void setSession(HttpSession session) {
    this.session = session;
  }

  public void setParameter(String s, String s1) {
    parameters.put(s, s1);
  }

  public void setAttribute(String name, Object value) {
    attributes.put(name, value);
  }

  // Lots of degenerate methods elided.
}

MockHttpServletResponse

package com.objectmentor.library.web.framework.mocks;

import javax.servlet.ServletOutputStream;
import javax.servlet.http.*;
import java.io.*;
import java.util.Locale;

public class MockHttpServletResponse implements HttpServletResponse {
  // all functions are implemented to be degenerate.
}

Given these mock objects, I can now create an instance of my loanRecords_jsp servlet and start calling methods on it! Indeed, my first test case looked something like this:

  public void testSimpleTest() throws Exception {
    MockJspWriter jspWriter = new MockJspWriter();
    MockPageContext pageContext = new MockPageContext(jspWriter);
    JspFactory.setDefaultFactory(new MockJspFactory(pageContext));
    HttpJspBase jspPage = new loanRecords_jsp();
    HttpServletRequest request = new MockHttpServletRequest();
    HttpServletResponse response = new MockHttpServletResponse();

    jspPage._jspInit();
    jspPage._jspService(request, response);

    assertEquals("", jspWriter.getContent());
  }

The test fails, as expected, because the content is a bit more than blank – but not much more. If you look carefully at the JSP file you’ll see that it calls request.getAttribute("loanRecords") and expects a List back. But since our test did not create such an attribute, the code throws an exception that is silently caught.

To get real HTML out of this servlet, we need to load up the attributes. Then we can use HtmlUnit to parse that HTML and write tests against it.

HtmlUnit is very simple to use, especially for testing generated web pages like this. There are a few gotcha’s that I described here

Here is the final test that loads the attributes, inspects the HTML with htmlUnit, and passes appropriately:

package com.objectmentor.library.jspTest.books.patrons.books;

import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.html.*;
import com.objectmentor.library.jsp.WEB_002dINF.pages.patrons.books.loanRecords_jsp;
import com.objectmentor.library.utils.*;
import com.objectmentor.library.web.controller.patrons.LoanRecord;
import com.objectmentor.library.web.framework.mocks.*;
import junit.framework.TestCase;
import org.apache.jasper.runtime.HttpJspBase;

import javax.servlet.*;
import javax.servlet.http.*;
import javax.servlet.jsp.*;
import java.util.*;

public class LoanRecordsJspTest extends TestCase {
  private MockPageContext pageContext;
  private MockJspWriter jspWriter;
  private JspFactory mockFactory;
  private MockHttpServletResponse response;
  private MockHttpServletRequest request;
  private WebClient webClient;
  private TopLevelWindow dummyWindow;

  protected void setUp() throws Exception {
    jspWriter = new MockJspWriter();
    pageContext = new MockPageContext(jspWriter);
    mockFactory = new MockJspFactory(pageContext);

    JspFactory.setDefaultFactory(mockFactory);
    response = new MockHttpServletResponse();
    request = new MockHttpServletRequest();
    webClient = new WebClient();
    webClient.setJavaScriptEnabled(false);
    dummyWindow = new TopLevelWindow("", webClient);
  }

  public void testLoanRecordsPageGeneratesAppropriateTableRows() throws Exception {
    HttpJspBase jspPage = new loanRecords_jsp();
    jspPage._jspInit();

    List<LoanRecord> loanRecords = new ArrayList<LoanRecord>();
    addLoanRecord(loanRecords,
                  "99",
                  "Empire",
                  DateUtil.dateFromString("2/11/2007"),
                  new Money(4200));
    addLoanRecord(loanRecords,
                  "98",
                  "Orbitsville",
                  DateUtil.dateFromString("2/12/2007"),
                  new Money(5200));

    request.setAttribute("loanRecords", loanRecords);

    jspPage._jspService(request, response);

    StringWebResponse stringWebResponse = new StringWebResponse(jspWriter.getContent());
    HtmlPage page = HTMLParser.parse(stringWebResponse, dummyWindow);
    HtmlElement html = page.getDocumentElement();

    HtmlTable table = (HtmlTable) html.getHtmlElementById("loanRecords");
    List<HtmlTableRow> rows = table.getHtmlElementsByTagName("tr");
    assertEquals(3, rows.size());

    assertEquals("even", classOfElement(rows.get(1)));
    assertEquals("odd", classOfElement(rows.get(2)));

    List<HtmlTableDataCell> firstRowCells = rows.get(1).getCells();
    assertEquals(4, firstRowCells.size());

    List<HtmlTableDataCell> secondRowCells = rows.get(2).getCells();
    assertEquals(4, secondRowCells.size());

    assertLoanRecordRowEquals("99", "Empire", "02/11/2007", "$42.00", firstRowCells);
    assertLoanRecordRowEquals("98", "Orbitsville", "02/12/2007", "$52.00", secondRowCells);
  }

  private String classOfElement(HtmlTableRow firstDataRow) {return firstDataRow.getAttributeValue("class");}

  private void assertLoanRecordRowEquals(String id, String title, String dueDate, String fine, List<HtmlTableDataCell> rowCells) {
    assertEquals(id, rowCells.get(0).asText());
    assertEquals(title, rowCells.get(1).asText());
    assertEquals(dueDate, rowCells.get(2).asText());
    assertEquals(fine, rowCells.get(3).asText());
  }

  private void addLoanRecord(List<LoanRecord> loanRecords, String id, String title, Date dueDate, Money fine) {
    LoanRecord loanRecord = new LoanRecord();
    loanRecord.id = id;
    loanRecord.title = title;
    loanRecord.dueDate = dueDate;
    loanRecord.fine = fine;

    loanRecords.add(loanRecord);
  }

  private class MockJspFactory extends JspFactory {
    private PageContext pageContext;
    public MockJspFactory(PageContext pageContext) {
      this.pageContext = pageContext;
    }

    public PageContext getPageContext(Servlet servlet, ServletRequest servletRequest, ServletResponse servletResponse, String string, boolean b, int i, boolean b1) {
      return pageContext;
    }

    public void releasePageContext(PageContext pageContext) {
    }

    public JspEngineInfo getEngineInfo() {
      return null;
    }
  }
}

This test makes sure that the generated HTML has the appropriate content contained within the rows of a table. The test does expect to see a table, and expects the rows to appear in the appropriate order. It also ensures that the rows will have alternating styles. Other than that, all issues of form and syntax are ignored.

Conclusion

The technique described here can be used to test virtually any static web page, or portion thereof outside of a container, and without a webserver running. It is relatively simple to set up; and then very easy to extend. With it, you can spin around the edit/compile/test loop very quickly, and can easily follow the rules of Test Driven Development.