Testing is an important part of all software development, yet a part that is so often overlooked or skimped on. Why is this? Perhaps it's because testing software is not considered exciting, or perhaps it's because it's not trivial, and if we're honest with ourselves it's impossible to write a set of tests that's perfect, the only way of knowing a test works is if it shows that the software doesn't. Passing all the tests does not mean the software is perfect, it may mean your tests just aren't good enough.
As the title suggests I'm going to look solely at unit testing, as it's currently something I'm focused on, having adopted Extreme Programming (XP) [XP]. So what is unit testing? To us it's testing individual software units to prove that they work as specified. In our case, tests may well form part of that specification, as XP is pro Test First programming, whereby we write our tests before writing the code, and we stop writing the code when all the tests pass. As such tests are both a way of ensuring quality and a way of measuring the status of our development. It's always worth remembering (and reminding the boss) that the sooner problems are found, the cheaper they are to fix. Tests formalise the testing process, they move it from just running the debugger and seeing what happens, to a structured, repeatable process. To encourage regular running of the tests, they should be quick and easy to use, in other words, fully automated. This will help you to get unit testing accepted as part of your personal and company development culture.
However not all code is as simple to test, nor do we want to have to repeat the basic features that all tests share. The ability to report the success or failure of the test is the most obvious feature that we'll be requiring again. We'll also probably want to test the function with more than two inputs, and a massive conditional statement is not clean, maintainable and readable. It would be very nice to know which part of the test failed, and which resulted in an error. It would be useful if we were able to run the tests on application start-up, choosing to run all the tests or a specified subset. Finally, unlike many C++ testing frameworks, we won't assume that the user will only ever test classes.
// . . . code . . .
Test(!isZero(0), __FILE__, __LINE__, results); // fails 0 is zero!
std::cout << "Test failed in file "
<< results[i].file << " on line "
// fails 0 is zero!
<< " failed in file "
<< " on line "
Having used our little bit of macro magic (for those that feel it's voodoo read [KandR88] page 90), it's time to start thinking about how we can scale this up.
We need to know if a test failed.
If a test failed we want the code for the test, the name of the file it is in, and the line it is on.
We want to be able to test multiple conditions in each test, failure of any single condition is a failure for the test.
Failed test(s) should not stop the rest of the test running.
We want a report on the test results, after all tests have run.
We may need to be able to set up some data before a test, and destroy it afterwards.
We should cope with exceptions.
The testing framework must be easy to use, as part of which we will implement our code in the test namespace.
Free/Used memory before and after the test.
To save space I'm not going to detail each function here, as a full description is available in the code [CppUTF].
As each test class may evaluate multiple expressions we're going to need to store more than one result for each test, as such we're going to use the class TestResultCollection to provide the interface for a test's results. It in turn will store a TestResult class for each failure, or error (we don't record passes; they are determined by the absence of a failure).
An error is defined as an exception that escapes the test code, or a failure of the setUp() or tearDown() methods, which are explained later. A failure is an expression that evaluates to false inside an IS_TRUE(), or an expression that evaluates to true within an IS_FALSE(). IS_TRUE and IS_FALSE are explained later.
The constructor requires a name argument, this name will be returned by the member function name() and should be a human friendly name for this test class, as it is only used for reporting.
The four member functions have been designed to allow overriding in order to allow the client to perform appropriate action for each class. setUp() should be used to prepare any data required for the test, the body of the tests should be in run(), and tearDown() should tidy up any resources allocated in setUp(). The function name() returns the name provided to the constructor. Each function has a default implementation provided.
The class also contains the protected member test_out_, intended to allow tests to write out a stream of data during the test. Note however that it is implemented via a std::ostringstream, and as such is not printed to the screen immediately.
Now any class deriving from Testable is automatically registered with the testing framework. The great benefit of this is that it allows us to add a test without changing any of the code already in the test build; all we need to do is add the new test's implementation file to our build list (or makefile). I believe this is important as I've several times seen code go untested as someone has forgotten to add the call to the test suite to the test driver code.
These functions evaluate the result of the test, and ask the test collection to log a failure if they are not true or false, respectively.
The main purpose of the function is to iterate through the list of tests, calling setUp(), run() and tearDown(), for each test in turn, catching any exceptions thrown, and checking that we only run tests that have been successfully set up, or recording tests that have failed to properly tear themselves down.
The final thing the test framework does is call TestCollection::report() which iterates through the TestResultCollection, reporting any passed tests, or failed tests along with any associated failures or errors.
The actual code supplied on my website contains a few more comments than are displayed in this article, and is documented with Doxygen [Doxy], so you can generate HTML, RTF, Latex, or man pages from it.
To demonstrate the use of the unit testing framework, I've chosen to use a modified version of the calculator Bjarne Stroustrup presents in The C++ Programming Language. I've modified it slightly, changing it into a class, and making it perhaps a little more reusable and testable.
std::cerr << "Error setting up test: "
Extend the tests; there is scope for more comprehensive testing.
Extend Calculator to support function calls, for example sin, cos, and tan. Also write suitable tests for each.
Add another class to the project, and a suitable test class.
Sometimes it's just not that easy to test results directly, or we really might just want to log some text during the test, so the framework allows the test to log data. This is provided by the protected member variable test_out_, and all you need to do to record text during the test is treat it exactly as you would std::cout. The text is then reported in the Output section of the tests report.
This article has introduced the design and development of the C++ unit testing framework we use at Crickett Software. It's continually evolving as we need it to, and the latest version will normally be available from our website.
In a future article we will look at how to use unit testing to improve the quality of your code, by catching tests earlier, and automating as much testing as possible. I'll also look at test first design. Any questions on the article are most welcome to <john@crickett.co.uk>.
Jon Jagger (http://www.jaggersoft.com) for feedback on an earlier testing framework and some suggestions that got me started on this incarnation of a C++ unit testing framework. Mark Radford (http://www.twonine.co.uk) for comments on both the framework and this article. Paul Grenyer (http://www.paulgrenyer.co.uk) for comments on several drafts.
[KandR88] The C Programming Language by Kernigan & Richie, 1988.