Unit testing Mbed OS

This document describes how to run and write unit tests for Mbed OS.

Prerequisites

• GNU toolchains installed.
  • GCC 6 or later
  • MinGW-W64 GCC-6.4.0 or MinGW-W64 GCC-7.3.0 (Windows)
• CMake 3.0+ installed.
• Python 2.7.x or >3.5 and pip 10.0 (or newer) installed.
• gcovr >=4.1

Installing dependencies on Debian/Ubuntu

1. sudo apt-get -y install build-essential cmake
2. Install python and pip:

   sudo apt-get -y install python python-setuptools
   sudo easy_install pip

3. Install gcovr.
4. (Optional) Install Mbed CLI.

Installing dependencies on Mac OS

1. Install Homebrew.
2. Install gcc compilers and cmake with: brew install gcc cmake
3. Install python and pip:

   brew install python
   sudo easy_install pip

4. Install gcovr.
5. (Optional) Install Mbed CLI.

Installing dependencies on Windows

1. Download and install MinGW-W64.
2. Download CMake binaries from https://cmake.org/download/ and run the installer.
3. Download Python2.7 or Python3 from https://www.python.org/getit/ and run the installer.
4. Add MinGW, CMake and Python into PATH.
5. Install gcovr.
6. (Optional) Install Mbed CLI.

Installing covr

Install gcovr code coverage tool globally with pip install "gcovr>=4.1" or using virtualenv:

virtualenv

1. Install virtualenv if not installed with pip install virtualenv
2. Install gcovr with:

[Debian/Linux/Mac OS]

virtualenv pyenv
. pyenv/bin/activate
pip install 'gcovr>=4.1'

[Windows]

virtualenv pyenv
pyenv\\Scripts\\activate
pip install "gcovr>=4.1"

Building and running unit tests

In case of running into problems see troubleshooting section.

UNITTESTS/mbed_unittest.py contains testing scripts for Mbed OS unit testing. Mbed CLI supports unit testing through mbed test --unittests command with the same arguments.

Testing with Mbed CLI

mbed test --unittests

A subset of tests can be run by providing -r flag for the tool which runs tests matching a regular expression.

e.g. mbed test --unittests --run -r features-netsocket

Build manually without Python tools

1. Create a build directory e.g. mkdir UNITTESTS/build.
2. Move to the build directory cd UNITTESTS/build.
3. Run CMake with cmake [RELATIVE PATH TO UNITTESTS DIR] [OPTIONAL ARGUMENTS] e.g. cmake ..:
   • Add -g [generator] argument if target other than Unix Makefiles e.g. MinGW -g "MinGW Makefiles"
4. Run make program (make, gmake, mingw32-make, etc).

Run CMake and build Unix Makefiles (GNU make)

cmake ..
make

Run CMake and build MinGW Makefiles (mingw32-make)

cmake -G "MinGW Makefiles" ..
mingw32-make

Custom CMake variables

Usage: cmake [RELATIVE PATH TO UNITTESTS DIR] [OPTIONS]

Keyword variables (usage cmake -D<VARIABLE>(:<TYPE>)=<value>:

Run in terminal

Unit tests can be run separately from each executable or by using ctest test runner. Run ctest with make program using target test. Options can be passed to ctest using ARGS argument. See ctest manual for more information.

Run ctest on test suite level:

{MAKE_PROGRAM} test -C [RELATIVE PATH TO BUILD DIRECTORY]

e.g. make test -C UNITTESTS/build or mingw32-make test -C UNITTESTS/build

Run ctest verbose (show each test case):

{MAKE_PROGRAM} test -C UNITTESTS/build ARGS="-V"

Run ctest dashboard test and create test results:

{MAKE_PROGRAM} test --C UNITTESTS/build ARGS="-D ExperimentalTest"

Run with GUI test runner

1. Build and/or install gtest-runner using the documentation: https://github.com/nholthaus/gtest-runner

2. Run the application, add built test executables into the list and run it.

Get code coverage

Python tools use gcovr to build code coverage reports. Generate html report UNITTESTS/build/coverage/index.html with:

mbed test --unittests --coverage html

To get coverage for a single test suite, run gcovr separately for suite coverage data directory. See gcovr documentation for more information.

e.g. for features/netsocket/InternetSocket coverage:

Debian/Ubuntu/Mac OS:

mkdir UNITTESTS/build
cd UNITTESTS/build
cmake -DCMAKE_BUILD_TYPE=Debug -DCOVERAGE:STRING=html  ..
make
./features-netsocket-InternetSocket
gcovr -r ../.. --html --html-detail -o ./index.html ./CMakeFiles/features-netsocket-InternetSocket.MbedOS.dir/

Windows:

mkdir UNITTESTS/build
cd UNITTESTS/build
cmake -DCMAKE_BUILD_TYPE=Debug -DCOVERAGE:STRING=html -g "MinGW Makefiles" ..
mingw32-make
features-netsocket-InternetSocket.exe
gcovr -r ..\.. --html --html-detail -o .\index.html .\CMakeFiles\features-netsocket-InternetSocket.MbedOS.dir\

The structure of unit tests

The structure of the unit tests directory looks like this:

UNITTESTS
  ├── mbed_unittest.py                       Python tool for unit testing
  ├── unit_test                              Python tool modules
  ├── CMakeLists.txt                         CMake project definition file
  ├── CMakeSettings.json                     CMake configurations for Visual Studio 2017
  ├── README.md
  ├── googletest-CMakeLists.txt.in           CMake project definition file for Google Test
  │
  ├── features
  │   └── netsocket                          Directory tree that mirrors Mbed OS root
  │       ├── NetworkInterface               Name of the class to be tested
  │       │   ├── test_NetworkInterface.cpp
  │       │   └── unittest.cmake             CMake module for unit test
  │       └── Socket
  │
  ├── stubs                                  Shared stubs which can be used for tests.
  ├── target_h                               Shared headers which can be used for tests.
  └── template                               Templates for creating new unittests

Each unit test has an identical directory tree as seen in the Mbed OS root folder. This is not a mandatory requirement but helps to maintain tests. Each class to be tested have their own unittest.cmake which is found by CMakeLists.txt.

Creating a unit test

Each class to be tested requires two files for unit testing:

1. C++ unit test source file (e.g. test_NetworkInterface.cpp)
2. CMake module file for unit test definition (unittest.cmake)

A unit test definition file unittest.cmake requires variables to be set for a test to be configured. File source paths in unittest.cmake files need to be relative to the unit test folder and CMakeLists.txt.

• unittest-includes - Include paths for headers needed to build the tests in addition to the base include paths listed in CMakeLists.txt. Optional.
• unittest-sources - Mbed OS source files and stubs included for the build.
• unittest-test-sources - Unit test source files.

Creating unit tests files with Mbed CLI

mbed test --unittests --new <FILEPATH>

E.g. mbed test --unittests --new rtos/Semaphore.cpp

The generator script only creates the files required for a unit test. It does not write unit tests automatically nor does it handle source dependencies.

Create files manually

For example to create a unit test for rtos/Semaphore.cpp:

1. Create a directory for unit test files in UNITTESTS/rtos/Semaphore.
2. Create a test definition file UNITTESTS/rtos/Semaphore/unittest.cmake with the following content: ``` set(unittest-sources stubs/mbed_assert.c ../rtos/Semaphore.cpp )

set(unittest-test-sources rtos/Semaphore/test_Semaphore.cpp )

3. Create a test source file `UNITTESTS/rtos/Semaphore/test_Semaphore.cpp` with the following content:

#include "gtest/gtest.h"

#include "rtos/Semaphore.h"

static osStatus_t retval = osOK; static uint32_t count = 0;

// Test stubs osStatus_t osSemaphoreAcquire(osSemaphoreId_t semaphore_id, uint32_t timeout) { return retval; } osStatus_t osSemaphoreDelete(osSemaphoreId_t semaphore_id) { return retval; } osStatus_t osSemaphoreRelease(osSemaphoreId_t semaphore_id) { return retval; } uint32_t osSemaphoreGetCount(osSemaphoreId_t semaphore_id) { return count; } osSemaphoreId_t osSemaphoreNew(uint32_t max_count, uint32_t initial_count, const osSemaphoreAttr_t attr) { return (void )&count; // Just a dymmy reference }

class TestSemaphore : public testing::Test { protected: rtos::Semaphore *sem;

virtual  void  SetUp()
{
    sem = new rtos::Semaphore();
}

virtual  void  TearDown()
{
    delete sem;
}

};

TEST_F(TestSemaphore, constructor) { EXPECT_TRUE(sem); } ```

Troubleshooting

Problem: virus protection identifies files generated by CMake as malicious and quarantines the files on Windows.

• Solution: restore the false positive files from the quarantine.