Speed up your C++ unit tests cycles with CMake/CTest (and the right testing framework)
Waiting for unit tests to compile and run shall not use much time. Unit test shall provide feedback as fast as possible. CMake can help speeding up compilation and execution.
The basic concept
Use a fast testing framework that is easy to integrate
Create multiple, small test binaries
Compile common expensive parts into object libraries
Make binaries that make use of the object libraries
Execute test in parallel
Use a tool that makes this as easy as possible
When the expensive compilation part is already done, only the required minimum will be compiled and this happens faster. Additional, with multiple tests, linking can happen in parallel. This is good.
Some header only C++ testing framework
As written, having a simple to integrate and fast testing framework is good to have.
I do not have a lot of requirements for the unit test framework I use. A testing framework shall be super easy to integrate into a project, compile fast, run fast. A BDD style testing is not required, but a plus.
Here a 3 example I have worked with and from which I know the basic concept works. 2 of them have BDD style testing support.
doctest is a C98/C11 single-header testing framework for unit tests and TDD. It let you define your own main function and comes with support for BDD style tests, what I like this a lot.
This framework will use for the example code of this post.
Catch2 is a well known C++ header only testing framework. There is a documentation available that describes how to write your custom main function. Catch2 has also support for BDD style tests.
I used boost test a lot in the bast but since some time I prefer doctest or catch.
However, since boost test is still used on a lot of places and often what is available, I thought I add it to the list here.
Less known, but boost test can be used as header only testing framework. Very useful, especially on platforms where compiling boost libraries makes not that much fun ;-). Boost test let’s you also implement your custom main function.
A small CMake and doctest example
- This example contains of
2 test files, test1.cpp and test2.cpp, each taking at least once second to execute
The test_main.cpp file, with the custom main implementation
The CMake file, to build and run the tests
I will not put the full code of everything into this blog post focus only on the CMake file.
- The full code example can be seen and downloaded here
How to use cmake and ctest is not the main topic of this post, but the short version is
Tell cmake to enable testing
Compile you test binary as you do with any other executalbe
Tell cmake that a test is available by using the add_test command
CMake object libraries
We have the specialty that we want to compile test_main.cpp into a object file that shall be reusable.
Luckily, CMake supports this out of the box via object libraries.
The resulting object files can be added to a cmake target like source files.
The cmake file for the testing looks like this
cmake_minimum_required(VERSION 3.0) project (testexample) enable_testing() add_library(test_main OBJECT test_main.cpp) add_executable(test1 test1.cpp $<TARGET_OBJECTS:test_main>) add_test(NAME test1 COMMAND test1) add_executable(test2 test2.cpp $<TARGET_OBJECTS:test_main>) add_test(NAME test2 COMMAND test2 )
The - little - specialty in this small cmake snippet is
add_library(test_main OBJECT test_main.cpp).
This create a object library that than can be added to multiple targets.
With this small example, not too much build improvement will be noticeable. But with the more test binaries a project contains, the more noticeable the advantage will become.
Running the tests
Part of faster build/execution loop is being able to run the tests in parallel.
As usual with cmake projects, create a build folder and run
cmake ../ and than
Test can be run with ctest. Like make, the ctest command takes an argument where you can specify the jobs to run.
Parallel test execution via CMake/CTest
To run 2 tests in parallel simply call
- make test
CMake will also provide a
make testtarget in the build folder. This will call ctest. To get parallel test execution we need to forward the jobs arguments to ctest. This can be done via the
make test ARGS=-j2
This will execute the tests in parallel, here the output from my machine.
Running the test parallel
make test ARGS=-j2 Running tests... Test project /home/me/ctestblog/build Start 1: test1 Start 2: test2 1/2 Test #1: test1 ............................ Passed 1.00 sec 2/2 Test #2: test2 ............................ Passed 1.00 sec 100% tests passed, 0 tests failed out of 2 Total Test time (real) = 1.01 sec
The tests run in parallel. So the total testing time will be around 1 second, even if the total execution time is 2 seconds.
And this is what we want.
In reality, with much more tests, jobs should match the number of CPUs on a machine.
Running tasks in parallel can speed up compilation and test execution. We have the CPUs available so we should make use of them.
For nearly all work I do many small test units are the better solution. It makes refactoring more easy and adding new test modules is super simple. Also, from my experience, with the right testing framework and small but concrete modules tests become more meaningful.
Thanks for reading
As usual, if you find any mistakes, in the code or in my spelling, please let
Ideas for improvements, or any other feedback, is also very much welcome.