Search | Blog | Contact

CppUTest Recent Experiences 19

Posted by Brett Schuchert Thu, 04 Feb 2010 19:42:00 GMT

Background

Mid last year I ported several exercises from Java to C++. At that time, I used CppUTest 1.x and Boost 1.38. Finally, half a year later, it was time to actually brush the dust off those examples and make sure they still work.

They didn’t. Bit rot. Or user error. Not sure which.

Bit rot: bits decay to the point where things start failing. Compiled programs do have a half-life.

User Error: maybe things were not checked in as clean as I remember. Though I suspect they were, I don’t really have any evidence to prove it, so I have to leave that option available.

To add to the mix, I decided to upgrade to CppUTest 2.x and to the latest version of the Boost library (1.41). I think that broke many several things. But the fixes were simple, once I figured out what I needed to do.

The Fixes

What follows are the three things I needed to do to get CppUTest 2.0, Boost and those exercises playing nicely together.

Header File Include Order

First, I used to have the header file for the CppUTest Test Harness, included first. It seems logical, but it caused all sorts of problems with CppUTest 2. That header file, includes a file, that ultimately includes something that uses macros to redefine new and delete. This is done so the testing framework can do simple memory tracking, which lets you know if your unit tests contains memory leaks.

I like this feature. Sure, it’s simple and light-weight, but it coves a lot of ground for a little hassle. The hassle? Include that header file last, instead of first. Problem Solved. Well at least the code compiles without hundreds of errors.

Boost Shared Pointer

Rather than hold pointers directly, I used the boost shared pointer class for a light-weight way to manage memory allocation. This is something I would do on a real project as well.

Somehow, the updated memory tracking in CppUTest 2.0 found something I had missed when using CppUTest 1.0.

I need to be able to control the date, so I have a simple date factory. By default, the date factory, when asked for the current date, returns the current date. Several unit tests want to simulate different dates. E.g., check out a book on one day, return it 14 days later. To do that, I manipulate the date factory (a form of dependency injection). This works fine, but by default the date factory is allocated using new.

When I replaced the existing date factory, I was not resetting it after the test. It turns out that this did not break anything because I was “lucky”. (Actually unlucky, I like things to fail fast.) CppUTest caught this in the form of not deallocating memory correctly:
  • I want to replace behavior
  • To do so I used polymorphism
  • Polymorphism in C++ requires virtual methods (please don’t correct me by suggesting that overloading is polymorphism, that is an opinion with which I strongly disagree)
  • Methods are only virtually dispatched via references or pointers
  • References cannot be changed, so I must use a pointer if I want a substitutable factory, which I wanted
  • Pointers suggest dynamic memory allocation

To fix this, I updated the setup method to store the original date factory in an attribute and then I updated the teardown method to restore the original date factory from that attribute. That I missed this suggests that my test suite is not adequate. I did not fix this problem for no good reason other than I was porting existing tests, so I left it as is. For the context it will not cause a problem. Pragmatic or lazy? You decide.

One Time Allocation

Here is a simple utility that uses Boost dates and regex: 
ptime DateUtil::dateFromString(const string &dateString) {
  boost::regex e("^(\\d{1,2})/(\\d{1,2})/(\\d{4})$");
  string replace("\\3/\\1/\\2");
  string isoDate = boost::regex_replace(dateString, e, replace, boost::match_default | boost::format_sed);
  return ptime(date(from_string(isoDate)));
}

Now this is somewhat simplistic code. So be it, it serves the purposes of the exercise. I can think of ways to fix this, but there’s an underling issue that exists if you use the regex library from Boost.

When you use the library, it allocates (in this example) 10 blocks of memory. If you read the documentation (I did), it’s making space for its internal state machine for regex evaluation. This is done once and then kept around.

So what’s the problem? Well, when I run my tests, the first test that happens to exercise this block of code reports some memory allocation issues: 
c:\projects\cppppp\dependencyinversionprinciple\dependencyinversionprinciple\pat
rongatewaytest.cpp:34: error: Failure in TEST(PatronGateway, AddAFew)
        Memory leak(s) found.
Leak size: 1120 Allocated at: <unknown> and line: 0. Type: "new" Content: "
?"
Leak size: 16 Allocated at: <unknown> and line: 0. Type: "new" Content: ?a"
Leak size: 20 Allocated at: <unknown> and line: 0. Type: "new" Content: "êà4"
Leak size: 52 Allocated at: <unknown> and line: 0. Type: "new" Content: ä4"
Leak size: 4096 Allocated at: <unknown> and line: 0. Type: "new" Content: ""
Leak size: 52 Allocated at: <unknown> and line: 0. Type: "new" Content: "êâ4"
Leak size: 20 Allocated at: <unknown> and line: 0. Type: "new" Content: ~4"
Leak size: 32 Allocated at: <unknown> and line: 0. Type: "new" Content: "?à4"
Leak size: 32 Allocated at: <unknown> and line: 0. Type: "new" Content: "h~4"
Leak size: 80 Allocated at: <unknown> and line: 0. Type: "new" Content: "êä4"
Total number of leaks:  10

This is a false positive. This is a one-time allocation and a side-effect of C++ memory allocation and static initialization.

There is a way to “fix” this. You use a command line option, -r, to tell the command line test runner to run the tests twice. If the allocation problem happens the first time but not the second time, then the tests are “OK”.

I didn’t want to do this.
  • The tests do take some time to run (30 seconds maybe, but still that doubles the time)
  • The output is ugly
  • It’s off topic for what the exercise is trying to accomplish
I tried a few different options but ultimately I went with simply calling that method before using the command line test runner. So I changed my main from: 
#include <CppUTest/CommandLineTestRunner.h>

int main(int argc, char **argv) {
  return CommandLineTestRunner::RunAllTests(argc, argv);
}
To this: 
#include "DateUtil.h"
#include <CppUTest/CommandLineTestRunner.h>

/** ************************************************************
    The boost regex library allocates several blocks of memory
    for its internal state machine. That memory is listed as a 
    memory leak in the first test that happens to use code that
    uses the boost regext library. To avoid having to run the
    tests twice using the -r option, we instead simply force
    this one-time allocation before starting test execution.
    *********************************************************** **/

void forceBoostRegexOneTimeAllocation() {
  DateUtil::dateFromString("1/1/1980");
}

int main(int argc, char **argv) {
  forceBoostRegexOneTimeAllocation();
  return CommandLineTestRunner::RunAllTests(argc, argv);
}

Since this one-time allocation happens before any of the tests run, it is no longer reported as a problem by CppUTest.

Before I introduced this “fix”, I spent quite a bit of time to verify that each of the 10 allocations were done by one of the three lines dealing with regex code in my DateUtil class. I used a conditional breakpoint and looked at the stack trace. (I know, using the debugger is considered a code smell, but not all smells are bad.)

Conclusion

I still like CppUTest. I’ve used a few C++ unit testing tools but there are several I have not tried. I don’t have enough face-time with C++ for this to be an issue. I am not terribly comfortable with the order of includes sensitivity. I’m not sure if that would scale.

I do appreciate the assistance with memory checking, though dealing with false positives can be a bit of a hassle. There was another technique, that of expressing the number of allocations. But in this case, that simply deferred the reporting of memory leaks to after test execution. In any case, I do like this. I’m not sure how well it would scale so it leaves me a bit uneasy.

If you happen to be using these tools, hope this helps. If not, and you are using C++, what can you say about your experiences with using this or other unit testing tools?

Comments

Leave a response

  1. Avatar
    James about 3 hours later:

    Hi Brett,

    Setting and restoring pointers is quite common, so we added a macro for setting pointers that are automatically restored after teardown(). Use it like this:

    UT_PTR_SET(pointer, new_pointer_value);

    You can put UT_PTR_SET() into setup() or a TEST.

    Re: Lazy initialization There used to be lazy statics in std::string, and well as a few others, in older versions of gcc.

    Any of you using singletons will have the same problem.

    James

  2. Avatar
    Jason Y about 3 hours later:

    I think they should feature C++ programmers on Dirty Jobs because they make civilized, garbage-collected life possible for the rest of us. Thank you!

  3. Avatar
    Matt B about 7 hours later:

    I haven’t used CppUTest, only CppUnit, and it’s worked fine for us so far. Do you prefer one over the other, or is CppUnit one of the ones you haven’t worked too much with?

  4. Avatar
    Arnstein 1 day later:

    I have been using CppUnit in the past, but this is not a good framework for unit testing. The reason for this is the large amount of code needed to write tests. Even the original author Michael Feathers discourages the use of this library.

    During the last year I have tested several C++ unit test frameworks and ended up with Google Test (gtest). After using this for half a year I’m impressed. Minimal code needed, outputs junit compatible XML and it is easy to create type and parameterized permutations of your tests.

  5. Avatar
    Yorgos Pagles 1 day later:

    The problems with C++ unit testing frameworks are both inherent (difficult to support all available platforms, use cases, compiler vendors and C++ never was friendly to create tools for) and also due to the fact that the test driven movement didn’t walk hand in hand with C++. On our subject, I am also fond on the Google testing frameworks (both gtest and gmock), mainly because they are under heavy usage inside Google supporting one of their major products (Chrome) which guarantees that they will be supported and maintained for quite a while.

  6. Avatar
    Brett L. Schuchert 4 days later:

    Matt,

    I think I have used CppUnit. Is that the testing tool with the scripts to build the main? If so, then I’ve used it and it was the first I’ve used.

    I looked at google’s and boost’s offerings, but I have not had enough good reason to delve into them.

    As Yorgos mentioned, C++ does not make it easy to support multiple platforms/compilers, so getting an easy to use tool that works well is not easy.

  7. Avatar
    Mike Long 5 days later:

    Brett, you are thinking of cxxtest, it is a perl or python script that generates the test code from your template. http://cxxtest.sourceforge.net/guide.html

  8. Avatar
    Brett L. Schuchert 7 days later:

    Mike,

    Ah yes! Sorry. It’s been some time and I am especially bad with proper nouns (really).

  9. Avatar
    Ben 9 days later:

    CxxTest is the only solution I’ve used in a C++ environment but a co-developer and I used it on our project for quite some time rather successfully.

    Our biggest problems related to poor coding practices by our project’s original developers (and likely some mistakes of our own) making our software difficult to test and difficulties integrating the tests into our build strategy.

    I’d consider CxxTest a viable contender the next time I’m in a position to select a framework but I’d certainly investigate the competitive solutions as well before moving forward: it’s always possible there’s a better fit out there.

  10. Avatar
    UGG Classic Metallicaz` about 1 month later:

    welcome to http://www.uggboots4buy.com/ l,will have a unexpection.

  11. Avatar
    UGG Classic Metallicaz` about 1 month later:

    Very quietly I take my leave.To seek a dream in http://www.edhardy-buy.com/ starlight.

  12. Avatar
    Kooba Handbags about 1 month later:

    Living without an aim is like sailing without a compass. with a new http://www.handbags4buy.com/ idea is a crank until the idea succeeds.

  13. Avatar
    highkoo ugg boots about 1 month later:

    all products are high quality but low price,welcome to http://www.uggjordanghd.com/.

    >
  14. Avatar
    parça TL kontör 2 months later:

    Very nice art, thank you for this site!

  15. Avatar
    Cloud Hosting 3 months later:

    There are several C++ unit testing tools which can be used for this . It’s a truth that the assistance with memory checking , though dealing with false positives can be a bit of a hassle. There are also another techniques of expressing the number of allocations. I have tried it and it works really well for me. I tried it using the tools mentioned above after reading the article. It simply worked well and helped a lot. I will like to share my experience about using C++ in my next comment.

  16. Avatar
    five finger shoes 3 months later:

    i believe you are good at writing. a good writter need many good topics

  17. Avatar
    m2ts to mkv converter 4 months later:

    M2TS to MKV Converter is the best software for user to convert M2TS to MKV file, With the powerful M2TS to MKV converter,you can convert M2TS to MKV with best quality and convert M2TS to all the video formats.m2ts to mkv converter

  18. Avatar
    immobilier entre particuliers 5 months later:

    thanks for this post i love much

  19. Avatar
    iboots 6 months later:

    Very quietly I take my leave.To seek a dream in

Comments