Speed up your C++ unit tests cycles with CMake/CTest (and the right testing framework)

Waiting for unit tests to compile and run shall not use much time. Unit test shall provide feedback as fast as possible. CMake can help speeding up compilation and execution.

The basic concept

  • Use a fast testing framework that is easy to integrate

  • Create multiple, small test binaries

  • Compile common expensive parts into object libraries

  • Make binaries that make use of the object libraries

  • Execute test in parallel

  • Use a tool that makes this as easy as possible

When the expensive compilation part is already done, only the required minimum will be compiled and this happens faster. Additional, with multiple tests, linking can happen in parallel. This is good.

Some header only C++ testing framework

As written, having a simple to integrate and fast testing framework is good to have.

I do not have a lot of requirements for the unit test framework I use. A testing framework shall be super easy to integrate into a project, compile fast, run fast. A BDD style testing is not required, but a plus.

Here a 3 example I have worked with and from which I know the basic concept works. 2 of them have BDD style testing support.

doctest

doctest is a C98/C11 single-header testing framework for unit tests and TDD. It let you define your own main function and comes with support for BDD style tests, what I like this a lot.

This framework will use for the example code of this post.

Catch2

Catch2 is a well known C++ header only testing framework. There is a documentation available that describes how to write your custom main function. Catch2 has also support for BDD style tests.

Boost.Test

I used boost test a lot in the bast but since some time I prefer doctest or catch. However, since boost test is still used on a lot of places and often what is available, I thought I add it to the list here.
Less known, but boost test can be used as header only testing framework. Very useful, especially on platforms where compiling boost libraries makes not that much fun ;-). Boost test let’s you also implement your custom main function.

A small CMake and doctest example

This example contains of

  • 2 test files, test1.cpp and test2.cpp, each taking at least once second to execute

  • The test_main.cpp file, with the custom main implementation

  • The CMake file, to build and run the tests

I will not put the full code of everything into this blog post focus only on the CMake file.

The full code example can be seen and downloaded here

https://github.com/a4z/cmaketestblogexample1

How to use cmake and ctest is not the main topic of this post, but the short version is

CMake object libraries

We have the specialty that we want to compile test_main.cpp into a object file that shall be reusable.

Luckily, CMake supports this out of the box via object libraries.

The resulting object files can be added to a cmake target like source files.

The cmake file for the testing looks like this

cmake_minimum_required(VERSION 3.0)
project (testexample)

enable_testing()

add_library(test_main OBJECT test_main.cpp)

add_executable(test1 test1.cpp $<TARGET_OBJECTS:test_main>)
add_test(NAME test1 COMMAND test1)

add_executable(test2 test2.cpp $<TARGET_OBJECTS:test_main>)
add_test(NAME test2 COMMAND test2 )

The - little - specialty in this small cmake snippet is add_library(test_main OBJECT test_main.cpp). This create a object library that than can be added to multiple targets.

With this small example, not too much build improvement will be noticeable. But with the more test binaries a project contains, the more noticeable the advantage will become.

Running the tests

Part of faster build/execution loop is being able to run the tests in parallel.

As usual with cmake projects, create a build folder and run cmake ../ and than make.

Test can be run with ctest. Like make, the ctest command takes an argument where you can specify the jobs to run.

Parallel test execution via CMake/CTest

ctest

To run 2 tests in parallel simply call ctest -j2.

make test

CMake will also provide a make test target in the build folder. This will call ctest. To get parallel test execution we need to forward the jobs arguments to ctest. This can be done via the ARGS variable.
Simply run make test ARGS=-j2

This will execute the tests in parallel, here the output from my machine.

Running the test parallel
make test ARGS=-j2
Running tests...
Test project /home/me/ctestblog/build
    Start 1: test1
    Start 2: test2
1/2 Test #1: test1 ............................   Passed    1.00 sec
2/2 Test #2: test2 ............................   Passed    1.00 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   1.01 sec

The tests run in parallel. So the total testing time will be around 1 second, even if the total execution time is 2 seconds.

And this is what we want.

In reality, with much more tests, jobs should match the number of CPUs on a machine.

Summary

Running tasks in parallel can speed up compilation and test execution. We have the CPUs available so we should make use of them.

For nearly all work I do many small test units are the better solution. It makes refactoring more easy and adding new test modules is super simple. Also, from my experience, with the right testing framework and small but concrete modules tests become more meaningful.

Thanks for reading

As usual, if you find any mistakes, in the code or in my spelling, please let me know.
Ideas for improvements, or any other feedback, is also very much welcome.