Hiding global methods, a C++ example

Hiding global methods, a C++ example 29

Posted by Brett Schuchert Fri, 12 Jun 2009 15:13:00 GMT

Background

This is a continuation of this discussion. A few of the postings by GBGames have been swallowed so the context is somewhat lost. What follows is an excerpt from an email he sent me and some example C++ code that demonstrates making global methods “testable” in a sense. Or rather, making your code not use global methods during test.

The Offending Method

Here is a method that GBGames has under test:

void SDLHardwareLayer::initializeHardware()
{
    const SDL_version * sdlVersion = SDL_Linked_Version();
    if (sdlVersion->minor < 2 && sdlVersion->major < 2)
    {
        //Error message
    }
    else if (SDL_Init(SDL_INIT_VIDEO |
            SDL_INIT_AUDIO |
            SDL_INIT_NOPARACHUTE) < 0)
    {
        //Error message
    }
    else
    {
        m_screen = SDL_SetVideoMode(m_x, m_y, m_bitDepth, SDL_DOUBLEBUF);
        if (NULL != m_screen)
        {
            SDL_WM_SetCaption(m_title.c_str(), 0);
        }
    }
}

This is how this code is organized:

So What’s Wrong?

One problem he notes is that the test code driving this production code causes windows to pop up during testing. Is this a problem? If it works, then it may be OK. However, you can remove that and also make test-ability a bit easier. You’ll be testing interactions and responses rather than directly using the library through its global functions.

Global Functions Considered Harmful

First off, this code as written directly uses global functions. If you leave this code unchanged, then the only option you have to “fix” the problem of windows popping up is a link-seam (see Working Effective with Legacy Code, page 233-234. In a nutshell, a link seam will link different versions of a library, one for testing, one for actual execution. So you’ll need some “make fu” to get the link seam working in a manner that easily allows building for testing versus building for execution. Since we’re using C++ and we own this code, I chose a different route:

Change this code to use an object that uses the library
That object implements an interface (a base class with all pure virtual methods)
Change this code so that it acquires this object through a factory (I could have simply used constructor injection, but I’ve already written it using a configurable factory [really singleton] – but it would have been simpler if I had injected the interface)
Created a test double that can be used when writing new test doubles making code maintenance a bit easier (mentioned in the other blog posting).
Write test-method specific test doubles

Changing Code to Use Factory + Object w/virtual Methods

Rather than directly calling the various global SDL_* methods, e.g.:

const SDL_version * sdlVersion = SDL_Linked_Version();

I instead use a factory to get an instance and call

const SDL_version * sdlVersion = SdlLayerFactory::getInstance()->SDL_Linked_Version();

Base Interface

To make this work, I created a base interface representing all of the methods needed by the SDLHardwareLayer::initializeHardware:

#pragma once

struct SDL_version;

class SdlLayerInstanceDelegator {
public:
  virtual ~SdlLayerInstanceDelegator(void) = 0;

  virtual const SDL_version *SDL_Linked_Version() = 0;
  virtual int SDL_Init(int bitMask) = 0;
  virtual void *SDL_SetVideoMode(int x, int y, int bitDepth, int mode) = 0;
  virtual void SDL_WM_SetCaption(const char *title, int someIntValue) = 0;

protected:
  SdlLayerInstanceDelegator();

private:
  SdlLayerInstanceDelegator(const SdlLayerInstanceDelegator &rhs);
  SdlLayerInstanceDelegator &operator = (const SdlLayerInstanceDelegator &);
};

Real Calls Global Methods

Getting rid of the direct call to the global method makes this work. However, you still need to call those global methods somewhere. That’s where the “real” version of the interface comes in. The real version of this class simply calls the global methods (code only, not showing header file right now):

const SDL_version *RealSdlLayerInstanceDelegator::SDL_Linked_Version() {
  return ::SDL_Linked_Version();
}

int RealSdlLayerInstanceDelegator::SDL_Init(int bitMask) {
  return ::SDL_Init(bitMask);
}

void *RealSdlLayerInstanceDelegator::SDL_SetVideoMode(int x, int y, int bitDepth, int mode) {
  return ::SDL_SetVideoMode(x, y, bitDepth, mode);
}

void RealSdlLayerInstanceDelegator::SDL_WM_SetCaption(const char *title, int someIntValue) {
  ::SDL_WM_SetCaption(title, someIntValue);
}

The Factory: Inversion of Control

The original code directly used global methods. We need to invert this relationship so that rather than depending directly on those methods, it depends on an interface. Then if it happens that it uses a “real” version, the underlying code will call the actual library. If, however, it is instead using a test double, it will not. It will do whatever the test double dictates.

We cannot leave the decision of which object to talk to in the SdlHardwareLayer class. Instead, I’ve used a configurable factory (singleton). A test can configure the factory, call the method under test, and then reset the factory back after the test.

The SdlLayerFactory allows such use(again, code only, no header file):

SdlLayerInstanceDelegator *SdlLayerFactory::instance = 0;

SdlLayerInstanceDelegator *SdlLayerFactory::getInstance() {
  return instance;
}

SdlLayerInstanceDelegator *SdlLayerFactory::replaceInstance(SdlLayerInstanceDelegator *replacement) {
  SdlLayerInstanceDelegator *original = instance;
  instance = replacement;
  return original;
}

Handling Growing Interfaces

One problem with “interfaces” in any language is that as you add methods to them, their implementations need to be updated. Since the SdlLayerInstanceDelegator class is not complete yet, this will case problems with test doubles. There are three ways to handle this problem: * Deal with it, as you add new methods to the interface, update all existing classes * Use a mocking library – I have not used any for C++, but if this were Java I’d use Mockito and if this were .Net I’d use Moq. * Create a base test double that implements all of the methods. All test doubles derive from that and only implement the methods needed for test.

The last option does not fix the problem, it controls it. When new methods are added, you update one test double class and all other classes still work.

This may not sound like much of an issue, but in general you’ll have many small test doubles rather than a few large test doubles (or at least that’s the standard recommendation). Why? Because a small, focused test double is less likely to be broken. It also better expressed your intention.

Here is that test double (header file excluded again): TestDoubleSdlLayerInstanceDelegator.h

#pragma once
#include "SdlLayerInstanceDelegator.h"

class TestDoubleSdlLayerInstanceDelegator: public SdlLayerInstanceDelegator {
public:
  TestDoubleSdlLayerInstanceDelegator();
  virtual ~TestDoubleSdlLayerInstanceDelegator();

  const SDL_version *SDL_Linked_Version();
  int SDL_Init(int bitMask);
  void *SDL_SetVideoMode(int x, int y, int bitDepth, int mode);
  void SDL_WM_SetCaption(const char *title, int someIntValue);
};

TestDoubleSdlLayerInstanceDelegator.cpp

#include "TestDoubleSdlLayerInstanceDelegator.h"

TestDoubleSdlLayerInstanceDelegator::TestDoubleSdlLayerInstanceDelegator(){}

TestDoubleSdlLayerInstanceDelegator::~TestDoubleSdlLayerInstanceDelegator(){}

const SDL_version *TestDoubleSdlLayerInstanceDelegator::SDL_Linked_Version() {
  return 0;
}

int TestDoubleSdlLayerInstanceDelegator::SDL_Init(int bitMask) {
  return 0;
}

void *TestDoubleSdlLayerInstanceDelegator::SDL_SetVideoMode(int x, int y, int bitDepth, int mode) {
  return 0;
}

void TestDoubleSdlLayerInstanceDelegator::SDL_WM_SetCaption(const char *title, int someIntValue){}

A Test

Finally, we need some tests and some test-specific test-doubles:

#include <CppUTest/TestHarness.h>

#include "sdl.h"
#include "SdlLayerFactory.h"
#include "SdlHardwareLayer.h"
#include "TestDoubleSdlLayerInstanceDelegator.h"

TEST_GROUP(SdlHardwareLayerTest) {
  virtual void setup() {
    original = SdlLayerFactory::getInstance();
    layer = new SdlHardwareLayer;
  }

  virtual void teardown() {
    if(SdlLayerFactory::getInstance() != original)
      delete SdlLayerFactory::getInstance();

    SdlLayerFactory::replaceInstance(original);
    delete layer;
  }

  SdlLayerInstanceDelegator *original;
  SdlHardwareLayer *layer;
};

struct MajorMinorSdlhardwareLayer : public TestDoubleSdlLayerInstanceDelegator {
  struct SDL_version v;

  MajorMinorSdlhardwareLayer() {
    v.major = 0;
    v.minor = 0;
  }

  const SDL_version *SDL_Linked_Version() {
    return &v;
  }
};

TEST(SdlHardwareLayerTest, ItGeneratsErrorWithLowMajorMinorVersionNumber) {
  SdlLayerFactory::replaceInstance(new MajorMinorSdlhardwareLayer);
  try {
    layer->initializeHardware();
    FAIL("Should have thrown int value");
  } catch (int value) {
    LONGS_EQUAL(1, value);
  }
}

struct InitFailingSdlHardwareLayer : public MajorMinorSdlhardwareLayer {
  InitFailingSdlHardwareLayer () {
    v.major = 3;
    v.minor = 3;
  }

  int SDL_Init(int bitMask) {
    return -1;
  }
};

TEST(SdlHardwareLayerTest, ItGeneratesErrorWhenInitReturnsLessThan0) {
  SdlLayerFactory::replaceInstance(new InitFailingSdlHardwareLayer);
  try {
    layer->initializeHardware();
    FAIL("Should have thrown int value");
  } catch (int value) {
    LONGS_EQUAL(2, value);
  }
}

Notice that there are two test doubles. Also notice that I create them as full (implicitly) inlined classes. Virtual methods and inline methods do not interact well. C++ will make a copy of the method bodies in every .o and increase link times. However, these test classes are only used in one place, so this really does not cause any problems.

In lieu of a mocking library, this is how I’d really write this code.

The Final Big Picture

Here’s an image of the completed product:

Summary

So is this overkill? Is using the actual SDL library causing problems? Is it OK for windows to appear during test? I’m neutral on that subject. The question I want to know is: are the tests working?

Do they add value or are they busy work.
Do they allow me to make small, incremental steps towards a complete, working implementation
Do they run fast enough?
Will they run in any environment? (Can I run on a headless system?)
Can I run multiple tests at the same time? (Not generally an issue, but it can be.)

If the SDL library is not causing a problem, then this might be overkill. However, on a long-lived project, this little amount of extra work will pay big dividends.

Oh, how long did this really take? Around 1.5 hours. But that included:

Starting a Windows XP VM
Creating a new projec
Setting up the link paths and include paths
Creating the initial file and the necessary support to get it to compile
Writing the additional classes

In fact, the actual work was probably < 30 minutes total. So while this might look big, in fact it is not. It’s nearly an idiom, so it’s mostly a matter of putting the hooks in place. So is full isolation from SDL worth 30 minutes?

Yes!

All your source files are belong to us

If this looks like a complete example, it is. I have this building and running in Visual Studio 2008. I took GBGame’s original method and got it to compile and link with a minimal set of additional source files. Then I made the structural changes to support test. And here’s all of the source code, file by file:

sdl.h

#pragma once

extern "C" {
  const int SDL_INIT_VIDEO = 1;
  const int SDL_INIT_AUDIO = 2;
  const int SDL_INIT_NOPARACHUTE = 4;
  const int SDL_DOUBLEBUF = 8;

  struct SDL_version {
    int major;
    int minor;
  };

  const SDL_version *SDL_Linked_Version();
  int SDL_Init(int bitMask);
  void *SDL_SetVideoMode(int x, int y, int bitDepth, int mode);
  void SDL_WM_SetCaption(const char *title, int someIntValue);
};

sdl_implementation.cpp

#include "sdl.h"

const SDL_version *SDL_Linked_Version() {
  return 0;
}

int SDL_Init(int bitMask) {
  return  - 1;
}

void *SDL_SetVideoMode(int x, int y, int bitDepth, int mode) {
  return 0;
}

void SDL_WM_SetCaption(const char *title, int someIntValue){}

SdlHardwareLayer.h

#pragma once

#include <string>

class SdlHardwareLayer {
public:
  SdlHardwareLayer();
  virtual ~SdlHardwareLayer();
  void initializeHardware();

private:
  int m_x;
  int m_y;
  int m_bitDepth;
  std::string m_title;
  void *m_screen;
};

SdlHardwarelayer.cpp

#include "SdlHardwareLayer.h"

#include "sdl.h"
#include "SdlLayerFactory.h"
#include "SdlLayerInstanceDelegator.h"

SdlHardwareLayer::SdlHardwareLayer() {
}

SdlHardwareLayer::~SdlHardwareLayer() {
}

void SdlHardwareLayer::initializeHardware() {
    const SDL_version * sdlVersion = SdlLayerFactory::getInstance()->SDL_Linked_Version();
    if (sdlVersion->minor < 2 && sdlVersion->major < 2) {
      throw 1;
    }
    else if (SdlLayerFactory::getInstance()->SDL_Init(SDL_INIT_VIDEO |
            SDL_INIT_AUDIO |
            SDL_INIT_NOPARACHUTE) < 0) {
              throw 2;
    }
    else {
        m_screen = SDL_SetVideoMode(m_x, m_y, m_bitDepth, SDL_DOUBLEBUF);
        if (0 != m_screen) {
            SDL_WM_SetCaption(m_title.c_str(), 0);
        }
    }
}

SdlLayerFactory.h

#pragma once

class SdlLayerInstanceDelegator;

class SdlLayerFactory
{
public:
  static SdlLayerInstanceDelegator *getInstance();
  static SdlLayerInstanceDelegator *replaceInstance(SdlLayerInstanceDelegator *replacement);

private:
  static SdlLayerInstanceDelegator *instance;

  SdlLayerFactory();
  ~SdlLayerFactory();
};

SdlLayerFactory.cpp

#include "SdlLayerFactory.h"

SdlLayerInstanceDelegator *SdlLayerFactory::instance = 0;

SdlLayerFactory::SdlLayerFactory(){}

SdlLayerFactory::~SdlLayerFactory(){}

SdlLayerInstanceDelegator *SdlLayerFactory::getInstance() {
  return instance;
}

SdlLayerInstanceDelegator *SdlLayerFactory::replaceInstance(SdlLayerInstanceDelegator *replacement) {
  SdlLayerInstanceDelegator *original = instance;
  instance = replacement;
  return original;
}

SdlLayerInstanceDelegator.h

#pragma once

struct SDL_version;

class SdlLayerInstanceDelegator {
public:
  virtual ~SdlLayerInstanceDelegator(void) = 0;

  virtual const SDL_version *SDL_Linked_Version() = 0;
  virtual int SDL_Init(int bitMask) = 0;
  virtual void *SDL_SetVideoMode(int x, int y, int bitDepth, int mode) = 0;
  virtual void SDL_WM_SetCaption(const char *title, int someIntValue) = 0;

protected:
  SdlLayerInstanceDelegator();

private:
  SdlLayerInstanceDelegator(const SdlLayerInstanceDelegator &rhs);
  SdlLayerInstanceDelegator &operator = (const SdlLayerInstanceDelegator &);
};

SdlLayerInstanceDelegator.cpp

#include "SdlLayerInstanceDelegator.h"

SdlLayerInstanceDelegator::SdlLayerInstanceDelegator(){}

SdlLayerInstanceDelegator::~SdlLayerInstanceDelegator(){}

TestDoubleSdlLayerInstanceDelegator.h

#pragma once
#include "SdlLayerInstanceDelegator.h"

class TestDoubleSdlLayerInstanceDelegator: public SdlLayerInstanceDelegator {
public:
  TestDoubleSdlLayerInstanceDelegator();
  virtual ~TestDoubleSdlLayerInstanceDelegator();

  const SDL_version *SDL_Linked_Version();
  int SDL_Init(int bitMask);
  void *SDL_SetVideoMode(int x, int y, int bitDepth, int mode);
  void SDL_WM_SetCaption(const char *title, int someIntValue);
};

TestDoubleSdlLayerInstanceDelegator.cpp

#include "TestDoubleSdlLayerInstanceDelegator.h"

TestDoubleSdlLayerInstanceDelegator::TestDoubleSdlLayerInstanceDelegator(){}

TestDoubleSdlLayerInstanceDelegator::~TestDoubleSdlLayerInstanceDelegator(){}

const SDL_version *TestDoubleSdlLayerInstanceDelegator::SDL_Linked_Version() {
  return 0;
}

int TestDoubleSdlLayerInstanceDelegator::SDL_Init(int bitMask) {
  return 0;
}

void *TestDoubleSdlLayerInstanceDelegator::SDL_SetVideoMode(int x, int y, int bitDepth, int mode) {
  return 0;
}

void TestDoubleSdlLayerInstanceDelegator::SDL_WM_SetCaption(const char *title, int someIntValue){}

RealSdlLayerInstanceDelegator.h

#pragma once
#include "SdlLayerInstanceDelegator.h"

class RealSdlLayerInstanceDelegator : public SdlLayerInstanceDelegator
{
public:
  RealSdlLayerInstanceDelegator();
  virtual ~RealSdlLayerInstanceDelegator();

  const SDL_version *SDL_Linked_Version();
  int SDL_Init(int bitMask);
  void *SDL_SetVideoMode(int x, int y, int bitDepth, int mode);
  void SDL_WM_SetCaption(const char *title, int someIntValue);
};

RealSdlLayerInstanceDelegator.cpp

#include "RealSdlLayerInstanceDelegator.h"
#include "sdl.h"

RealSdlLayerInstanceDelegator::RealSdlLayerInstanceDelegator(){}

RealSdlLayerInstanceDelegator::~RealSdlLayerInstanceDelegator(){}

const SDL_version *RealSdlLayerInstanceDelegator::SDL_Linked_Version() {
  return ::SDL_Linked_Version();
}

int RealSdlLayerInstanceDelegator::SDL_Init(int bitMask) {
  return ::SDL_Init(bitMask);
}

void *RealSdlLayerInstanceDelegator::SDL_SetVideoMode(int x, int y, int bitDepth, int mode) {
  return ::SDL_SetVideoMode(x, y, bitDepth, mode);
}

void RealSdlLayerInstanceDelegator::SDL_WM_SetCaption(const char *title, int someIntValue) {
  ::SDL_WM_SetCaption(title, someIntValue);
}

SdlHardwareLayerTest.h

#include <CppUTest/TestHarness.h>

#include "sdl.h"
#include "SdlLayerFactory.h"
#include "SdlHardwareLayer.h"
#include "TestDoubleSdlLayerInstanceDelegator.h"

TEST_GROUP(SdlHardwareLayerTest) {
  virtual void setup() {
    original = SdlLayerFactory::getInstance();
    layer = new SdlHardwareLayer;
  }

  virtual void teardown() {
    if(SdlLayerFactory::getInstance() != original)
      delete SdlLayerFactory::getInstance();

    SdlLayerFactory::replaceInstance(original);
    delete layer;
  }

  SdlLayerInstanceDelegator *original;
  SdlHardwareLayer *layer;
};

struct MajorMinorSdlhardwareLayer : public TestDoubleSdlLayerInstanceDelegator {
  struct SDL_version v;

  MajorMinorSdlhardwareLayer() {
    v.major = 0;
    v.minor = 0;
  }

  const SDL_version *SDL_Linked_Version() {
    return &v;
  }
};

TEST(SdlHardwareLayerTest, ItGeneratsErrorWithLowMajorMinorVersionNumber) {
  SdlLayerFactory::replaceInstance(new MajorMinorSdlhardwareLayer);
  try {
    layer->initializeHardware();
    FAIL("Should have thrown int value");
  } catch (int value) {
    LONGS_EQUAL(1, value);
  }
}

struct InitFailingSdlHardwareLayer : public MajorMinorSdlhardwareLayer {
  InitFailingSdlHardwareLayer () {
    v.major = 3;
    v.minor = 3;
  }

  int SDL_Init(int bitMask) {
    return -1;
  }
};

TEST(SdlHardwareLayerTest, ItGeneratesErrorWhenInitReturnsLessThan0) {
  SdlLayerFactory::replaceInstance(new InitFailingSdlHardwareLayer);
  try {
    layer->initializeHardware();
    FAIL("Should have thrown int value");
  } catch (int value) {
    LONGS_EQUAL(2, value);
  }
}

RunAllTests.cpp

#include <CppUTest/CommandLineTestRunner.h>

int main(int ac, char **av) {
  return CommandLineTestRunner::RunAllTests(ac, av);
}

Posted in Schuchert's Scattered Synapses
Tags c, global, isolation, methods, test, testing, wrapping
Meta 29 comments, permalink, rss, atom

Comments

Leave a response

bruno.uy@gmail.com about 3 hours later:

What about mocking SDL by linking fake SDL_Init instead of the real ones?
Brett L. Schuchert about 4 hours later:
Bruno,
You can do this. That’s the link seam I mentioned. It’s fine if:
- You cannot change the underlying code
- You don’t have virtual methods available
- You’re looking for something no so drastic
And I’d reserve that option.

What I’ve done is essentially the same thing but instead of doing it at link time, I’m doing it at runtime with a configurable factory.

However, the link seam make building more difficult. I’ve seen a lot of places that have issues with building. In fact, often that’s the first thing to get working. Making the build work so you can write tests quickly. And often, in those situations, you’ll use link seams to short circuit the build process.
http://www.jamesladdcode.com about 6 hours later:

Another great post – What tool do you use for the diagrams?
Brett L. Schuchert about 9 hours later:

http://yuml.me
GBGames about 24 hours later:

Whew! That’s quite a bit more complexity than I expected. What surprises me is that out of all of the code written, with new instance delegators and factories, there are only two unit tests. Now, I understand you were simply trying to get my original function to build and run, but if I were doing what you did, I would feel like I needed to have unit tests for the delegator and the factory as well. Is this a bad expectation on my part, did you leave out some of the complete example, or did you take a shortcut?

This post really helps illustrate how to get around global functions. Thanks for taking the time to walk me through all of this!
GBGames 1 day later:

I should add that while it seems more complex than anything I would have thought up on my own, I see what you mean about how idiomatic it is. It’s actually very clean and simple, once I think about it.

To summarize to make sure I understand: If the first thing you think to do is encapsulate the global functions, you need to replace them with something. In this case, the delegator. And if you have a delegator, you need a way to get it, so that where the factory/singleton comes in.

The hardware layer asks the factory for a delegator, and then uses that delegator through its interface, which allows multiple delegators to be created. Simple.

In fact, if I decide to switch from libSDL to Allegro or Clanlib, with a few changes to the design (it wouldn’t be an SDLHardwareLayer anymore if it could be any implementation under it!), this code would be general purpose enough to use whatever similar library I would want. Of course, I’ll worry about that refactoring if I actually care to use other libraries. For now, I just want to see this one project to completion. B-)

In any case, it’s pretty clear what is happening and why. What trips me up are the steps I would take to get to this (or any) solution on my own, but that’s what practice will hopefully teach me. I’ll be participating in a 3 day TDD workshop next week which I hope is pretty hands-on.

I found a store nearby that has Michael Feathers’ book, and I’ll be picking it up today. It seems that while I understood legacy code to mean “any code without unit tests”, it somehow didn’t occur to me that libSDL would apply. And in that sense, a lot of library code would apply. B-)
Ricky about 1 month later:

Perhaps I’m missing something (my C++ is horribly rusty), but is there a reason SDL_SetVideoMode() and SDL_WM_SetCaption() aren’t being called through the factory’s instance in SdlHardwarelayer.cpp?
carmeron about 1 year later:

This may not sound like much of an issue, but in general you’ll have many small test doubles rather than a few large test doubles (or at least that’s the standard recommendation). Why? Because a small, focused test double is less likely to be broken. It also better expressed your intention.
briefcase manufacturer about 1 year later:

What follows is an excerpt from an email he sent me and some example C
Dog bark collar about 1 year later:

Informative article. Thanks for the information. I appreciate your post.
http://www.whiteiphone4transformer.com about 1 year later:

now supplying the latest white iphone 4 conversion kit to make your iphone 4 different from others. As a fashion fans like you will surely love it.
Silicone Molding about 1 year later:

Intertech Machinery Inc. provides the most precise Plastic Injection Mold and Rubber Molds from Taiwan. With applying excellent unscrewing device in molds, Intertech is also very professional for making flip top Cap Molds in the world.
Low Carb Snacks about 1 year later:

Well written post, properly researched and beneficial for me within the future.I am so happy you took the time and effort to create this.
iPhone SMS to Mac Backup about 1 year later:

Would you like to banckup iphone SMS to mac, macBook, macbookPro as .txt files? Now a software iphone SMS to Mac Backup can help you to realize it.
iPad to Mac Transfer about 1 year later:

I really like this essay. Thank you for writing it so seriously. I want to recommend it for my friends strongly. iPad to Mac Transfer can help you transfer music, movie, photo, ePub, PDF, Audiobook, Podcast and TV Show from ipad to mac freely.
Solid State Relay about 1 year later:

Thanks for writing this blog post, it was informative, enjoyable, and most importantly – a good length!
clothing manufacturer about 1 year later:

Informative article. Thanks for the information. I appreciate your post.
okey oyunu oyna about 1 year later:

Thank you very much.

internette görüntülü olarak okey oyunu oyna, gerçek kisilerle tanis, turnuva heyecanini yasa.
http://update-seputar-software.blogspot.com/2011/04/mercedes-benz-mobil-mewah-terbaik.html about 1 year later:

Mercedes-Benz Mobil Mewah Terbaik Indonesia
shoes christian louboutin about 1 year later:

Thankyou. I am glad I could share these words.
christian louboutin shoes on sale about 1 year later:

These christian louboutin patent leather pumps are just the same as selling in the franchise store.Or even better. Quite cheap christian louboutin leather platform pumps.I could say this is the best one I have ever ordered online.
Supra Trainers over 2 years later:

Last but not least, some use a wadded up piece of paper to get a head start while creating the ball.
bagsupplyer over 2 years later:

Designer women Lacoste backpacks free shipping for wholesale
Tips For Bowling over 2 years later:

The Secretary of Defense is not a super General or Admiral. His task is to exercise civilian control over the Department for the Commander-in-Chief and the country. Bowling Tips Tips For Bowling
ysbearing/yisong@1stbearing.com over 2 years later:

Slewing bearing called slewing ring bearings, is a comprehensive load to bear a large bearing, can bear large axial, radial load and overturning moment. http://www.1stbearing.com
Pandora Earrings over 2 years later:

Whether pandora wholesale you have been unable to find a stylish and beautiful shape of the necklace? Then small series with a look at the beauty of these big fake collar necklace has a lovely pandora charm retro small lapel T units for two consecutive quarters as the most popular fashion point. This winter, a small lapel is not limited to fashion, it is more to necklaces pandora necklace discount Pandora Jewelry of the field, the most tide “false collar necklace this year, major fashion brand designers start of the article on the collar, 2011 Milan Fashion Week on, a dazzling white pointed collar Pandora Bracelets collective debut. Chanel’s little black dress with small round white mix is has become a classic, shot in the street this year is everywhere now This stock has fallen a false collar style accessories Pandora Earrings. than the fabric is too low silver pandora chains round neck, collar necklace of fake luxury metal may dazzling land and more. false collar means is a fabric made into a good texture a collar and this collar no pandora necklaces silver bracelets sleeves, no breast, no back, just only a collar, so call it false lead. the most beautiful moment of a woman’s life is put on the holy wedding, into the wedding hall, the love will be forever a Beads Pandora when worn on the finger.
christian louboutin over 2 years later:
The professional design make you foot more comfortable. Even more tantalizing,this pattern make your legs look as long as you can,it will make you looked more attractive.Moveover,it has reasonable price.If you are a popular woman,do not miss it.

Technical details of Christian Louboutin Velours Scrunch Suede Boots Coffee:
```
Color: Coffee
Material: Suede
4(100mm) heel
Signature red sole x
```
Fashion, delicate, luxurious Christian louboutins shoes on sale, one of its series is Christian Louboutin Tall Boots, is urbanism collocation. This Christian louboutins shoes design makes people new and refreshing. Red soles shoes is personality, your charm will be wonderful performance.
Womens Moncler Jackets over 3 years later:

Everyone should have a Mens Moncler Jackets as it is the best outwear against the cold season and is also comfortable and light to wear.
iPhone contacts backup over 3 years later:

If you hide it. will the code run very well? it is worth thinking twice.

Mentor	twitter id
Uncle Bob	unclebobmartin
Brett Schuchert	schuchert
Michael Feathers	mfeathers
Bob Koss	bob_koss