There are many Unit Testing frameworks for C++, but which one to use? Chris Main shares his experience of some of them.
'Not another one' was the immediate reaction of a work colleague on seeing the article [ FRUCTOSE ] on the FRUCTOSE unit test framework [ Overload07 ] open on my desk. It had also been my first reaction, but as I have always taken an interest in unit testing I made the effort to study the article. I was pleasantly surprised to find that FRUCTOSE has something to offer that other tools did not. The author, Andrew Marlow, also explained in some detail why he had not used any of a number of existing tools. I found this more satisfactory than a previous article [ Overload06 ] on CUTE [ CUTE ] where the author, Peter Sommerlad, only explicitly considered CppUnit [ CPPUnit ] as an alternative, even though Aeryn [ Aeryn ] was referenced in the article. The FRUCTOSE article did not consider Aeryn, so I offer this comparative evaluation. The focus of my evaluation is a narrow one: I am considering frameworks specifically for unit testing C++, which can be run from a command line.
Boost
For the last three and a half years I have been using an in-house framework which is very similar to Boost [ Boost ] but with a richer set of assertion macros and the facility to select tests from the command line. We would have used Boost, except our compiler limited us to the header only libraries. This confirms Andrew Marlow's assertion that there are some environments where the Boost framework cannnot be used. Where the Boost framework can be used, though, it should not be dismissed too hastily. It gives me the impression that it has evolved from something more complex (an impression reinforced by the release notes). The documentation is comprehensive, but could be better organised. I fear that it obscures how very easy the framework is to use if you restrict your tests to standalone functions taking no arguments and returning void. Having worked within this restriction when using our in-house framework, I think this is no problem at all. Test files follow the format in Listing 1.
#include "factorial.hpp"
#include <boost/test/auto_unit_test.hpp>
namespace {
BOOST_AUTO_UNIT_TEST(test_factorial)
{
BOOST_CHECK_EQUAL(factorial(3), 6);
}
}
|
| Listing 1 |
In addition, you need a suite creator which is trivial to implement, as shown below:
#define BOOST_AUTO_TEST_MAIN
#include <boost/test/auto_unit_test.hpp>
You then link together the suite creator, the boost test lib and as many test files as required (from a single test file to all the test files for a library or application). Hence Boost supports running the tests for a single file, but in a way that scales up easily to running a full suite of regression tests.
Boost does not support selecting tests from the command line. This inconvenience can be minimized by ensuring that any long running test is isolated in its own test files.
Boost has a good selection of assertion macros, including allowing tolerances for floating point comparisons. It doesn't have _LT , _LE , _GT , _GE variants which I have come to prefer to boolean assertions where appropriate.
Boost supports two output formats (selectable from the command line) out of the box: human readable and XML. This is a reasonable design. The human readable format is good enough for most purposes, and the XML can be post processed into any conceivable format when it isn't.
In summary, Boost provides a simple to use, fully functional and scaleable unit test framework, so long as you can build it on your platform. It also provides many other features, which you can find in its documentation, but bear in mind that you may not need them.
Aeryn
Aeryn requires a modern C++ compiler, so if you can't build Boost you may not be able to build Aeryn. It has no dependencies upon other libraries. Unlike Boost, Aeryn feels like it has evolved from something simple. Its design is very clean, and this is reflected in the User Guide which is a model of clarity (once you recover from the picture of the leather clad young woman on the title page!). You can use Aeryn in almost exactly the same way as Boost; the difference is that you have to wrap your test functions in TestCase s, as in Listing 2.
#include "factorial.hpp"
#include <aeryn/test_case.hpp>
#include <aeryn/test_registry.hpp>
namespace {
void test_factorial()
{
IS_EQUAL(factorial(3), 6);
}
Aeryn::TestCase numeric_tests[] = {
Aeryn::TestCase( USE_NAME(test_factorial) ),
Aeryn::TestCase() };
REGISTER_TESTS_USE_NAME(numeric_tests)
}
|
| Listing 2 |
This leaves scope for a programmer to write a test function but forget to wrap and register it. It should be possible to write a macro to take a function name, wrap it in a test case and register it all in one to to prevent this happening. Building an executable is also very similar to Boost: link the Aeryn main and core libs with as many test files as required.
Aeryn does support selecting tests from the command line. It has a more limited set of assertion macros (no tolerance for floating point comparisons, for example, you have to implement that yourself).
Aeryn supports selection of output format from the command line. A number of human readable formats are provided out of the box, but not XML or HTML. Aeryn provides an elegant mechanism for building custom output formats into its framework: simply write a class which inherits from Aeryn::IReport and implements its well documented virtual member functions, then add the class to the report factory to make it available from the command line.
In summary, Aeryn provides a simple to use, fully functional and scaleable unit test framework, so long as you can build it on your platform.
Since, in my opinion, Boost and Aeryn are both excellent, mature unit test frameworks I don't understand what CUTE is trying to achieve, given that it requires the same kind of platform as them.
FRUCTOSE
FRUCTOSE does have a clear niche: it's a header file only framework that doesn't require a modern C++ compiler. I think it has the potential to be worthwhile even outside of that niche. It has the richest selection of assertion macros. I have had to insert tracing code to output loop indexes often enough to think that the loop assertion macros are a good idea. It supports selecting tests from the command line. It only supports a single, human readable format. Personally, I think that customisable output is a less important feature than a good set of assertion macros, so this doesn't put me off.
Although I usually work with a framework based upon standalone functions, I find that for a large proportion of my unit tests the standalone function is a wrapper around a test class. This is because there is usually some setup I want to put in a class constructor. I don't, therefore, see FRUCTOSE's requirement to derive test classes from a base class as a great burden. As for incorporating implementation code into the test class by public inheritance, I wouldn't normally consider this good design. However the sheer convenience it brings in this case, and the fact that test classes will not be further derived from nor have instances passed around as objects, leads me to take a pragmatic view of it (just as I pragmatically accept the use of Singleton in the implementation of automatic test registration, even though I normally avoid that pattern).
CxxTest
One weakness of FRUCTOSE when compared to Boost and Aeryn is that its main function has to be hand coded. Possibly this could be remedied using a similar automatic registration mechanism to them, but I think an approach more in keeping with the spirit of FRUCTOSE may be to adopt the design used by CxxTest [ CxxTest ]. This uses a Perl script to generate the main function from the test code. (CxxTest is quite similar to FRUCTOSE in that it has a good set of assertion macros, is distributed as header files, has no other dependencies and will work with older compilers. Its big drawback is that it does not allow selection of tests from the command line; you have to comment out code for a particular test if you wish to exclude it).
By defining a few macros, as shown in Listing 3, so that the test code looks like Listing 4, it becomes very easy to write a parser for the test code.
FRUCTOSE_CLASS(numeric_tests)
{
public:
FRUCTOSE_TEST(test_factorial)
{
...
}
};
|
| Listing 3 |
#define FRUCTOSE_STRUCT(_name_) struct _name_ :
public fructose::test_base<_name_>
#define FRUCTOSE_CLASS(_name_) class _name_ :
public fructose::test_base<_name_>
#define FRUCTOSE_TEST(_name_) void _name_(
const std::string &test_name)
|
| Listing 4 |
The parser only needs to do basic text processing; it does not need to understand the grammar of C++. The parser can then be used to generate the main function automatically, and it is straightforward to write the generator so that it can take any number of input test files. In this way FRUCTOSE, although initially designed for testing a single class, would be made as scaleable as the other frameworks.
The macros have the added advantage, in my opinion, of concealing the Curiously Recurring Template Pattern and of relieving the programmer of the need to remember the correct signature for test member functions. These are simplifications, but some may consider that achieving them by means of macros is a price not worth paying. n
References
[ FRUCTOSE ] https://sourceforge.net/projects/fructose
[ Overload07 ] Overload 77, February 2007
[ Overload06 ] Overload 75, October 2006.
[ CUTE ] http://wiki.hsr.ch/PeterSommerlad/wiki.cgi?CuTeDownload
[ Aeryn ] http://www.aeryn.co.uk
[ CPPUnit ] https://sourceforge.net/projects/cppunit
[ Boost ] http://www.boost.org
[ CxxTest ] http://cxxtest.sourceforge.net
| Checklist | |||||||||||||||||||||||||||||||||||
|
