Unit testing is the process by which we write testing code that fully exercises a function/library/module (base code). This testing code becomes part of the code base that is maintained in source code control. It ideally run as part of a continuous integration process so that every time we modify the base code, the unit tests will be run to ensure that we haven't changed functionality by the modifications to the base code. In some development processes, such as Test Driven Development, the unit tests are actually written first before the base code is written.
There are number of commercial and open source unit testing frameworks for C++ such as Boost Test Library, CPPUnit and Google Test. In this post I develop a simple unit testing framework for C++ that has most of the features needed for successful unit testing. If you are writing code without an IDE that supports a specific unit test framework or are just interested in how a unit testing framework is built, then this post will be of interest to you.
For example, let's say that you have an integer square root function that you have written that you want to test. (See Methods of Computing Square Roots.). The PRECONDITION() macro is from our discussion on Debugging and Logging.
int32_t isqrt(int32_t n) {
PRECONDITION(n >= 0);
if (n == 0)
return 0;
int32_t x = n;
int32_t c = 0;
int32_t d = 1 << 30;
while (d > n)
d >>= 2;
while (d != 0) {
if (x >= c + d) {
x -= c + d;
c = (c >> 1) + d;
} else {
c >>= 1;
}
d >>= 2;
}
return c;
}
Now we need to write a unit test function to exercise all the control paths in the isqrt() function. If it were composed only of simple assignment and arithmetic expressions we would only have one path and would only need one test case. Because the isqrt() function has two IF and two WHILE statements, we will need to have a number of test cases. To test both paths of the first IF we need to test with the argument zero and with a non-zero argument. To test the second IF it is not as clear what values we could pass to isqrt() to exercise both paths. Testing a variety of non-zero values would hopefully end up covering both cases. If your IDE has a "Run with Code Coverage" command we could be sure that both paths are covered. Code coverage commands are also useful for ensuring that both WHILE loops are entered.
The simplest way to test the function is to write a main() function in the compilation unit that has isqrt(). We could write the following to test our isqrt() function. For example...
int main(int argc, const char * argv[]) {
if (isqrt(0) != 0)
cout << "isqrt(0) test case failed!";
if (isqrt(1) != 1)
cout << "isqrt(1) test case failed!";
if (isqrt(4) != 2)
cout << "isqrt(4) test case failed!";
return 0;
}
There are a number of problems with this approach. One is that if you try to use the compilation unit that has isqrt() in another project you will get a conflict with main() function and the main() function of your program. The second is that it contains a lot of redundant boilerplate code. We can solve this by writing a test() function that contains the redundant code. This function would accept a boolean expression which is the test result and a string which is the message to print if the test condition doesn't hold (is false).
Because we want to print the filename and line number where the test condition failed, we use the C++ preprocessor symbols __FILE_NAME__ and __LINE__. Because these take on values based on where they are used in the code, we can not use them inside the test() function. If we did we would always get the filename and line of the test() function and not the filename and line where the test failed. To solve this problem we can easily write some macros that expand to a call to test() using the __FILE_NAME__ and __LINE__ symbols at the site of the macro invocation. We will write one macro that expects the boolean expression to be true and one that expects false.
#define expectTrue(b) UnitTest::test((b), string(#b) + " was not true!", __FILE_NAME__, __LINE__)
#define expectFalse(b) UnitTest::test(!(b), string(#b) + " was not false!", __FILE_NAME__, __LINE__)
namespace UnitTest {
void test(bool b, string msg, char const *filename, int line) {
if (!b) {
stringstream ss;
ss << filename << "(" << line << "):" << msg;
throw runtime_error(ss.str());
}
}
}
This will allow us to write the main() function as follows:
int main(int argc, const char * argv[]) {
expectTrue(isqrt(0) == 0);
expectTrue(isqrt(1) == 1);
expectTrue(isqrt(4) == 2);
expectFalse(isqrt(9) == 3);
return 0;
}
If the individual test succeeds, then nothing is printed. We only expect something to be printed if an individual test fails. The last individual test is intentionally written to fail so that we can see the output when we have a failure. Because test() throws an exception when the condition is false, we will expect an exception to be printed to STDOUT and the program will immediately terminate.
libc++abi: terminating with uncaught exception of type std::runtime_error: main.cpp(137):isqrt(9) == 3 was not false!
Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
Notice that the exception message contains the name of the compilation unit (main.cpp) and the line number where the expectFalse() macro was used. This allows us to easily find the individual test that failed. Also notice that we moved the main() function into its own compilation unit within the project containing isqrt(). This allows us to solve our second problem which was conflicts among multiple main() functions. All your test code can be written in a main.cpp file stored with the project/library that it tests. This file would not be included in other projects that use your project/library under test. Unfortunately, this approach becomes cumbersome for large libraries with a large number of individual tests. It would be better if we could group individual tests into test cases. This way we could have a test case for each individual function or piece of functionality in our project/library under test. It would also be nice if main() could automatically detect all the test cases in the compilation unit without the need to manually enumerate them. For example:
TEST(ISqrtTest) {
expectTrue(isqrt(0) == 0);
expectTrue(isqrt(1) == 1);
expectTrue(isqrt(4) == 2);
expectFalse(isqrt(9) == 3);
}
int main(int argc, const char * argv[]) {
UnitTest::runtests();
return 0;
}
TEST() is a macro that registers the test case so that it can be run by the runtests() call in main(). It also expands to include the function definition for ISqrtTest() using the function body immediately following the TEST() macro. It is a bit of macro magic that makes our code cleaner. You can write as many test cases using TEST() as you want and they will all be run. In addition, runtests() continues to run all the other test cases in a file even if there is an individual test failure. The test case reports that it failed but allows other test cases to run.
Testing ABCTest...Passed Testing ISqrtTest...Failed main.cpp(137):isqrt(9) == 3 was not false! Testing XYZTest...Passed
The code for TEST() actually defines a structure with the name passed to the macro. This is why all test cases defined with TEST() must have unique names in main.cpp.
The structure defines two static members. The first is a static integer member called static_setup. This variable will only be initialized once and it initialization is used to run the registration code that enters the test case function into a standard map containing all the test cases, (i.e. the test suite). The allocation of static_setup is immediately below the structure definition and uses a lambda expression to update the test suite map.
The second member is the testbody() static member function where the test case is going to be defined. It is declared inside the structure but is defined at the end of the macro. Notice that the definition is missing a function body. This is because you will provide the function body in main.cpp immediately after the macro call to TEST().
#define TEST(testname) struct testname{ \
static int static_setup; \
static void testbody(void); \
}; \
int testname::static_setup = [](){UnitTest::getTestSuite().emplace(#testname, testname::testbody);return 0;}(); \
void testname::testbody(void)
Because TEST() records all your test cases in a single test suite map, all that is needed in main() is to enumerate the map, calling each of the test case functions. This is done by calling runtests(). The UnitTest namespace that defines the test suite map and runtests() would look like this.
namespace UnitTest {
// Singleton instances wrapped in functions to avoid the indeterminate
// order of namespace static variable initialization in separate
// compilation units.
map<string, TestFunc> &getTestSuite() {
static map<string, TestFunc> testsuite;
return testsuite;
}
void runtests(map<string, UnitTest::TestFunc> &tests) {
for (auto it = begin(tests); it != end(tests); ++it) {
cout << "Testing " << it->first << "...";
try {
it->second();
cout << "Passed" << endl;
} catch (exception &ex) {
cout << "Failed" << endl;
cout << "\t" << ex.what() << endl;
}
}
}
inline void runtests() {
runtests(getTestSuite());
}
}
It would seems that the testsuite map can just be defined as a static variable in the namespace. The problem with this is that in C++ we don't know the order of initialization of static variables that appear in different compilation units. This means that a compilation unit that uses the TEST() macro might be initialized before the UnitTest.cpp compilation unit that defines the testsuite map causing a compile error. To solve this problem, we write a static function getTestSuite() that includes a static function variable for the testsuite map. It is guaranteed that this static function variable will be initialized before the first call to getTestSuite(). This way if we always call getTestSuite() to access the testsuite map, we are guaranteed that it will be initialized when we need it regardless of the order of compilation unit static variable initialization.
If we modify the UnitTest code to add a second map for disabled tests, we gain the ability to temporarily disable tests. This is useful when we have some long running test cases that we want to skip when debugging other test cases. It is also useful in the situation where you have a test case that is failing but you can't fix it right now for some reason. We want to modify runtests() to print out disabled test cases so that we don't forget about them.
We will also need a new macro that is similar to TEST() but instead adds the test to the map of disabled tests. This macro will be called DISABLED_TEST() to make it easy to switch between enabled and disabled for any particular test case.
#define TEST(testname) struct testname{ \
static int static_setup; \
static void testbody(void); \
}; \
int testname::static_setup = [](){UnitTest::getTestSuite().emplace(#testname, testname::testbody);return 0;}(); \
void testname::testbody(void)
#define DISABLED_TEST(testname) struct testname{ \
static int static_setup; \
static void testbody(void); \
}; \
int testname::static_setup = [](){UnitTest::getDisabledTests().emplace(#testname, testname::testbody);return 0;}(); \
void testname::testbody(void)
namespace UnitTest {
// Singleton instances wrapped in functions to avoid the indeterminate
// order of namespace static variable initialization in separate
// compilation units.
map<string, TestFunc> &getTestSuite() {
static map<string, TestFunc> testsuite;
return testsuite;
}
map<string, TestFunc> &getDisabledTests() {
static map<string, TestFunc> disabled_tests;
return disabled_tests;
}
void runtests(map<string, UnitTest::TestFunc> &tests, bool disabled) {
#ifdef _WIN32
activateVirtualTerminal();
#endif
for (auto it = begin(tests); it != end(tests); ++it) {
cout << "Testing " << it->first << "...";
if (disabled) {
cout << colorize(Colorize::YELLOW) << "Disabled" << colorize(Colorize::NC) << endl;
} else {
try {
it->second();
cout << colorize(Colorize::GREEN) << "Passed" << colorize(Colorize::NC) << endl;
} catch (exception &ex) {
cout << colorize(Colorize::RED) << "Failed" << endl;
cout << "\t" << ex.what() << colorize(Colorize::NC) << endl;
}
}
}
}
}
As a further enhancement to make it easier to distinguish disabled test cases, failed test cases and passed test cases, we can add color to the console output. Console colorize() function was explored in C++ Colorized Console Output.
Testing ABCTest…Passed Testing ISqrtTest2…Failed main.cpp(137):isqrt(9) == 3 was not false! Testing XYZTest…Disabled
The only thing left to do is to define some testing macros beyond expectTrue() and expectFalse() to make it easier to construct different types of test. This will bring our simple UnitTest framework up to par with most other frameworks out there.
UnitTest Testing Macros
- expectTrue(expr)
- Passes if expr is true.
- expectFalse(expr)
- Passes if expr is false.
- expectEqual(expr, expected)
- Passes if
expr == expected. - expectNotEqual(expr, expected)
- Passes if
expr != expected. - expectStringEqual(expr, expected)
- Passes if
expr == expectedafter both are converted tostringusing the insertion operator. - expectGT(expr, expected)
- Passes if
expr > expected. - expectLT(expr, expected)
- Passes if
expr < expected. - expectGE(expr, expected)
- Passes if
expr >= expected. - expectLE(expr, expected)
- Passes if
expr <= expected. - expectApproxEqual(expr, expected, epsilon)
- Passes if
abs(expr - expected) < epsilon. Good for use with floating point values that are close but not exactly equal. - expectException(statement, exceptionType, ...)
- Passes if an exception of type
exceptionTypeis thrown fromstatementwhen executed. The variable arguments at the end are the names of variables needed instatementin order to convertstatementto a lambda expression. - failure(message)
- Immediately causes the test case to fail using
messageas the failure message. - success()
- Immediately causes the test case to succeed causing any remaining code in the test case to remain unexecuted.
The complete source for UnitTest.h and UnitTest.cpp is provided here:
UnitTest.h
//
// Created by Richard Lesh on 10/31/21.
// Copyright © 2021 Richard Lesh. All rights reserved.
//
#ifndef UNITTEST_H
#define UNITTEST_H
#include <functional>
#include <iomanip>
#include <map>
#include <stdexcept>
#include <string>
#include <sstream>
namespace UnitTest {
typedef void (*TestFunc)();
std::map<std::string, TestFunc> &getTestSuite();
std::map<std::string, TestFunc> &getDisabledTests();
void test(bool b, std::string msg, char const *filename, int line);
template <class T, class CALLABLE>
void testThrow(CALLABLE testfunc, std::string funcAsString, std::string exceptionName, char const *filename, int line) {
std::stringstream ss;
ss << filename << "(" << line << "):";
try {
testfunc();
ss << funcAsString << " did not throw " << exceptionName;
} catch (T &ex) {
return;
} catch (std::exception &ex) {
ss << funcAsString << " threw wrong exception " << typeid(ex).name() << ":" << ex.what();
} catch (...) {
ss << funcAsString << " threw unknown exception!";
}
throw std::runtime_error(ss.str());
}
template <class CALLABLE>
long timetask(CALLABLE task) {
auto start = std::chrono::high_resolution_clock::now();
task();
auto stop = std::chrono::high_resolution_clock::now();
return std::chrono::duration_cast<std::chrono::nanoseconds>(stop - start).count();
}
void runtests(std::map<std::string, TestFunc> &tests, bool disabled = false);
inline void runtests() {
runtests(getTestSuite());
runtests(getDisabledTests(), true);
}
}
#define TEST(testname) struct testname{ \
static int static_setup; \
static void testbody(void); \
}; \
int testname::static_setup = [](){UnitTest::getTestSuite().emplace(#testname, testname::testbody);return 0;}(); \
void testname::testbody(void)
#define DISABLED_TEST(testname) struct testname{ \
static int static_setup; \
static void testbody(void); \
}; \
int testname::static_setup = [](){UnitTest::getDisabledTests().emplace(#testname, testname::testbody);return 0;}(); \
void testname::testbody(void)
#define expectTrue(b) UnitTest::test((b), string(#b) + " was not true!", __FILE_NAME__, __LINE__)
#define expectFalse(b) UnitTest::test(!(b), string(#b) + " was not false!", __FILE_NAME__, __LINE__)
#define expectEqual(a, b) { std::stringstream ss; auto A = a; auto B = b; \
ss << #a << " expected equal to " << B << " but was " << A; \
UnitTest::test(A == B, ss.str(), __FILE_NAME__, __LINE__); }
#define expectNotEqual(a, b) { std::stringstream ss; auto A = a; auto B = b; \
ss << #a " expected not equal to " << B << " but was " << A; \
UnitTest::test(A != B, ss.str(), __FILE_NAME__, __LINE__); }
#define expectStringEqual(a, b) { \
std::stringstream ssa; ssa << a; \
std::stringstream ssb; ssb << b; \
std::stringstream ss; auto A = ssa.str(); auto B = ssb.str(); \
ss << #a << " expected equal to " << B << " but was " << A; \
UnitTest::test(A == B, ss.str(), __FILE_NAME__, __LINE__); }
#define expectGT(a, b) { std::stringstream ss; auto A = a; auto B = b; \
ss << #a " expected greater than " << B << " but was " << A; \
UnitTest::test(A > B, ss.str(), __FILE_NAME__, __LINE__); }
#define expectLT(a, b) { std::stringstream ss; auto A = a; auto B = b; \
ss << #a " expected less than " << B << " but was " << A; \
UnitTest::test(A < B, ss.str(), __FILE_NAME__, __LINE__); }
#define expectGE(a, b) { std::stringstream ss; auto A = a; auto B = b; \
ss << #a " expected greater than or equal " << B << " but was " << A; \
UnitTest::test(A >= B, ss.str(), __FILE_NAME__, __LINE__); }
#define expectLE(a, b) { std::stringstream ss; auto A = a; auto B = b; \
ss << #a " expected less than or equal " << B << " but was " << A; \
UnitTest::test(A <= B, ss.str(), __FILE_NAME__, __LINE__); }
#define expectApproxEqual(a, b, epsilon) { std::stringstream ss; auto A = a; auto B = b; \
ss << std::setprecision(17) << #a " expected approx. equal to " << B << " but was " << A; \
UnitTest::test(std::abs(A - B) <= epsilon, ss.str(), __FILE_NAME__, __LINE__); } \
// varargs are the capture parameters to the lambda in statement
#define expectException(statement, exceptionType, ...) \
UnitTest::testThrow<exceptionType>([__VA_ARGS__](){statement;}, #statement, #exceptionType, __FILE_NAME__, __LINE__)
#define expectNull(b) UnitTest::test((b) == nullptr, string(#b) + " was not nullptr!", __FILE_NAME__, __LINE__)
#define expectNotNull(b) UnitTest::test((b) != nullptr, string(#b) + " was was nullptr!", __FILE_NAME__, __LINE__)
#define success() return
#define failure(msg) UnitTest::test(false, string("Test Failure: ") + msg, __FILE_NAME__, __LINE__)
#endif
UnitTest.cpp
//
// Created by Richard Lesh on 10/31/21.
// Copyright © 2021 Richard Lesh. All rights reserved.
//
#include <chrono>
#include <map>
#include "Colorize.h"
#include "UnitTest.h"
using namespace std;
namespace UnitTest {
// Singleton instances wrapped in functions to avoid the indeterminate
// order of namespace static variable initialization in separate
// compilation units.
map<string, TestFunc> &getTestSuite() {
static map<string, TestFunc> testsuite;
return testsuite;
}
map<string, TestFunc> &getDisabledTests() {
static map<string, TestFunc> disabled_tests;
return disabled_tests;
}
void test(bool b, string msg, char const *filename, int line) {
if (!b) {
stringstream ss;
ss << filename << "(" << line << "):" << msg;
throw runtime_error(ss.str());
}
}
void runtests(map<string, UnitTest::TestFunc> &tests, bool disabled) {
#ifdef _WIN32
activateVirtualTerminal();
#endif
for (auto it = begin(tests); it != end(tests); ++it) {
cout << "Testing " << it->first << "...";
if (disabled) {
cout << colorize(Colorize::YELLOW) << "Disabled" << colorize(Colorize::NC) << endl;
} else {
try {
it->second();
cout << colorize(Colorize::GREEN) << "Passed" << colorize(Colorize::NC) << endl;
} catch (exception &ex) {
cout << colorize(Colorize::RED) << "Failed" << endl;
cout << "\t" << ex.what() << colorize(Colorize::NC) << endl;
}
}
}
}
}
I hope that this example is useful to you and helps you to better understand the use of macros, templates and static function variables. Please leave comments or questions that you have.