Testing in MITK

The basic idea about unit testing is to write a test for every unit (e.g. function) of your software, which automatically checks if the code still returns the expected result for a defined input. The goal is to isolate units as much as possible, in order to be efficient with fixing errors.

If this is performed on a low level, this is called unit testing. On higher levels, where various circumstances (such as configuration files, internet connection, hardware, or even input/output) may influence the outcome of the test, this is called integration testing.

MITK uses CppUnit, which is a framework for efficient and fast test writing, to support developers in test coding. Furthermore MITK offers currently three types of tests: classic unit tests, rendering tests and interaction tests. Although our unit tests often involve data input (or output), we still consider them to be unit tests.

Warning: Most of our units tests have been written before adopting CppUnit and have not yet been updated. Therefore we do not recommend to look at a random test for an example of coding a test.

This pages gives a general introduction to our test framework. Rendering tests are explained in more detail in the section Automatic Rendering Tests and interaction tests the section Interaction Testing in MITK. The following sections will explain how to write your own tests with MITK and how to run them.

Adding a test to your module

Generally, functional code you want to test using unit tests will be located in a module. For an overview of the directory structure see Create a Folder for your Module and how to add the files comprising your test compare Set up the Test environment

Test and test class/suite

CppUnit and MITK distinguish between the term test class and test, which is important to understand. A test is simply a call of a single function and a comparison against a reference.

The ideal way to write a test is:

result = myGreatFunction(myParameter);

ASSERT_EQUAL(result, reference);

which is just 2 lines of code at the best. Sometimes objects and data may be initialized in a test, but usually this should be done in the setUp() method (see below). If possible, avoid multiple assertions in order to be efficient in error localization and for better readability. For instance, if you instantiate an object and test if its data is not NULL, this is already a complete test and you do not want to repeat this comparison in other tests.

All tests check different things and just one at a time. Remember: The goal is to have isolated unit tests and avoid large, confusing tests which are hard to maintain.

Note: The reference should never be computed and ideally be a distinct and well defined (often hard coded) value. You don't care where you reference comes from, you just want to ensure that your test actually results in the reference value.

A test class (also called suite) is a set of tests and often associated to a single functional class in MITK. A test class refers to a TestSuite in CppUnit. These terms are easily confused, so we suggest to use the term test suite for a collection of tests (which is often in a single file), and the term "test" for a single test (usually many per file).

In order to create a test class you need to create a new class deriving from mitk::TestFixture. While a purist approach would be to create a new mitk::TestFixture for each method of your class (resulting in as many test source files as your class has methods), we usually follow a more pragmatic approach within MITK. In most cases we suggest having one test source file per class. If your class source file is called mitkGreatClass.cpp we suggest the name mitkGreatClassTest.cpp for your test source file. For very complex and long classes splitting this into several tests may be advisable. Additionally, rendering tests and interaction tests should always get their own test class/suite to mark them as obvious integration tests, because the goal is to isolate unit tests from integration tests. The dashboard for continuous integration will automatically show the results of a test suite and summarize the output. If you run your test manually (e.g. with ctest -VV), you will see the detailed output of all tests of the suite.

In order to use CppUnit via MITK you will need to include the corresponding files and register your test:

#include <mitkTestingMacros.h>
#include <mitkTestFixture.h>
…
MITK_TEST_SUITE_REGISTRATION(mitkImageEqual)

How to structure your test

As mentioned before, all test suites derive from mitk::TestFixture. A suggested name for your test suite would be <FUNCTIONALITY_TO_BE_TESTED>TestSuite .

You then create a suite and register your tests. A suggested naming convention for test methods is <METHOD_TO_BE_TESTED>_<INPUT_PARAMETERS/CONDITION>_<EXPECTED_RESULT> . We suggest not to add the word test, because that is most likely the only thing the reader knows anyway about your test, as he is reading a test suite.

An example:

class mitkImageEqualTestSuite : public mitk::TestFixture
{
  CPPUNIT_TEST_SUITE(mitkImageEqualTestSuite);
  MITK_TEST(Equal_CloneAndOriginal_ReturnsTrue);
  MITK_TEST(Equal_InputIsNull_ReturnsFalse);
  MITK_TEST(Equal_DifferentImageGeometry_ReturnsFalse);
  MITK_TEST(Equal_DifferentPixelTypes_ReturnsFalse);
  …
  CPPUNIT_TEST_SUITE_END();
  …
}

You also may provide a setUp() and a tearDown() function. These will be called before/after each test and should be used to make sure that each test works independently and on freshly initialized members and memory to maximize isolation. That way you avoid only testing whether a function works after another function has already been called. For an example test suite including tests see An example

Running your test suite

The build system of MITK generates a test driver which includes all test suites that have been added to the project. Alternatively you can run MITK test suites by using the program ctest. This is the way all MITK tests run on the continuous dart clients of MITK. The results of these runs can be found at https://cdash.mitk.org/.

If you use the test driver, you only need to start the executable. If you start it without parameters, it will then give you an overview of all tests which are included in this test driver and you can choose one by typing a number. Alternatively, you can give your test driver the name of your test suite as parameter. If you want to use ctest instead of the test driver you need to start a command line, go to the binary directory of MITK and call ctest. To avoid errors, check if your path variable contains all relevant paths to start MITK.

Adding parameters to your test

If possible, the setUp() method of the test suite should provide all necessary inputs for the respective tests. MITK provides several helper classes to generate synthetic test data, such as the mitk::ImageGenerator. If you have to load data from the hard disc for your test, you can use the method GetTestDataFilePath(string fileName). For an example of loading data from the MITK_DATA_DIR check the mitkIOUtilTestSuite.

Predefined assertions

MITK and CppUnit offer predefined assertions, i.e. helper methods which will help to compare your data against a certain reference. All basic types are covered by CppUnit assertions, such as CPPUNIT_ASSERT. MITK further offers comparison tools for floating point numbers, vectors, images, surfaces and point sets. A complete list of assertion macros is given in MITK Testing.

An example to compare images:

MITK_ASSERT_EQUAL(image, reference, "Checks if image is equal to a reference");

By default, the method uses an mitk::eps for floating point comparison, but this can be adapted. It can be necessary to write your own assertion for your own data meeting your special requirements. Recommended examples are all equal test suites for basic data types (mitkImageEqualTest, mitkSurfaceEqualTest and mitkPointSetEqualTest).

An example

/*============================================================================
 
The Medical Imaging Interaction Toolkit (MITK)
 
Copyright (c) German Cancer Research Center (DKFZ)
All rights reserved.
 
Use of this source code is governed by a 3-clause BSD license that can be
found in the LICENSE file.
 
============================================================================*/
 
// Testing
#include "mitkTestFixture.h"
#include "mitkTestingMacros.h"
 
// std includes
#include <string>
 
// MITK includes
#include "mitkExampleDataStructure.h"
 
// VTK includes
#include <vtkDebugLeaks.h>
 
class mitkExampleDataStructureTestSuite : public mitk::TestFixture
{
  CPPUNIT_TEST_SUITE(mitkExampleDataStructureTestSuite);
 
  // Test the append method
  MITK_TEST(Append_ExampleString_AddsExampleStringToData);
  CPPUNIT_TEST_SUITE_END();
 
private:
  mitk::ExampleDataStructure::Pointer m_Data;
  std::string m_DefaultDataString;
 
public:
  void setUp() override
  {
    m_DefaultDataString = "This is the example data content\nAnd a second line\n";
    m_Data = mitk::ExampleDataStructure::New();
    m_Data->SetData(m_DefaultDataString);
  }
 
  void tearDown() override
  {
    m_DefaultDataString = "";
    m_Data = nullptr;
  }
 
  void Append_ExampleString_AddsExampleStringToData()
  {
    std::string appendedString = "And a third line\n";
    std::string wholeString = m_DefaultDataString + appendedString;
    m_Data->AppendAString(appendedString);
 
    CPPUNIT_ASSERT_MESSAGE("Checking whether string was correctly appended.", m_Data->GetData() == wholeString);
  }
};
 
MITK_TEST_SUITE_REGISTRATION(mitkExampleDataStructure)

Further information

More examples can be found in the corresponding bugsquashing presentation.

Table of Contents