Medical Imaging Interaction Toolkit  2018.4.99-31d1c60d
Medical Imaging Interaction Toolkit
The Segmentation Plugin
segmentation.svg
Icon of the Segmentation Plugin

Some of the features described below are closed source additions to the open source toolkit MITK and are not available in every application.

Overview

The Segmentation plugin allows you to create segmentations of anatomical and pathological structures in medical images of the human body. The plugin consists of a number of view which can be used for:

  • manual and (semi-)automatic segmentation of organs on CT or MR image volumes via the Segmentation View
  • segmentation postprocessing via the The Segmentation Utilities View
  • clipping of existing segmentations using a resection plane via the The Clipping Plane
QmitkSegmentation_IMGApplication.png
Segmentation Plugin consisting of the Segmentation View the Segmentation Utilities View and the Clipping Plane View

The segmentation plugin offers a number of preferences which can be set via the MITK Workbench application preference dialog:

QmitkSegmentation_IMGPreferences.png
Segmentation Plugin consisting of the Segmentation View the Segmentation Utilities View and the Clipping Plane View

The following preferences can be set:

  • Slim view: Allows you to show or hide the tool button description of the Segmentation View
  • 2D display: Specify whether the segmentation is drawn as outline or as a transparent overlay
  • 3D display: Activate 3D volume rendering for your segmentation
  • Data node selection mode: If activated the segmentation image combo box is always sychronized with the data manager selection.
  • Smoothed surface creation: Set certain smoothing parameters for surface creation

If you wonder what segmentations are good for, we shortly revisit the concept of a segmentation here. A CT or MR image is made up of volume of physical measurements (volume elements are called voxels). In CT images, for example, the gray value of each voxel corresponds to the mass absorbtion coefficient for X-rays in this voxel, which is similar in many parts of the human body. The gray value does not contain any further information, so the computer does not know whether a given voxel is part of the body or the background, nor can it tell a brain from a liver. However, the distinction between a foreground and a background structure is required when:

  • you want to know the volume of a given organ (the computer needs to know which parts of the image belong to this organ)
  • you want to create 3D polygon visualizations (the computer needs to know the surfaces of structures that should be drawn)
  • as a necessary pre-processing step for therapy planning, therapy support, and therapy monitoring

Creating this distinction between foreground and background is called segmentation. The Segmentation perspective of the MITK Workbench uses a voxel based approach to segmentation, i.e. each voxel of an image must be completely assigned to either foreground or background. This is in contrast to some other applications which might use an approach based on contours, where the border of a structure might cut a voxel into two parts.

The remainder of this document will summarize the features of the Segmentation perspective and how they are used.

Technical Issues

The Segmentation perspective makes a number of assumptions. To know what this view can be used for, it will help you to know that:

  • Images must be 2D, 3D, or 3D+t
  • Images must be single-values, i.e. CT, MRI or "normal" ultrasound. Images from color doppler or photographic (RGB) images are not supported
  • Segmentations are handled as binary images of the same extent as the original image

Image Selection

The Segmentation perspective makes use of the Data Manager view to give you an overview of all images and segmentations.

QmitkSegmentation_IMGSelection.png
Data Manager is used for selecting the current segmentation. The reference image is selected in the drop down box of the control area.

To select the reference image (e.g. the original CT/MR image) use the patient image drop down box in the control area of the Segmentation view. The segmentation image selected in the Data Manager is displayed below in the segmentation drop down box. By default the auto selection mode is enabled, which always keeps the selection of the segmentation drop down box in synch with the selection in the data manager. If you disable the auto selection mode the selection of the right segmentation image has to be done via the drop down box. If no segmentation image exists or none is selected create a new segmentation image by using the "New segmentation" button on the right of the Segmentation drop down box. Some items of the graphical user interface might be disabled when no image is selected or the selected image does not fit to the patient image's geoemtry. In any case, the application will give you hints if a selection is needed.

Tool overview

MITK comes with a comprehensive set of segmentation tools. These tools can be differenciated between manual slice-based 2D segmentation tools and (semi-)automated 3D tools. The manual 2D tools require a big amount of user interaction and can only be applied to a single image slice whereas the 3D tools operate on the hole image. The 3D tools usually require a small amount of interaction like placin seedpoints of setting some parameters. You can switch between the different toolsets by switching the 2D/3D tab in the segmentation view.

QmitkSegmentation_ToolOverview.png
An overview of the existing tools in MITK. There are interactive 2D tools as well as (semi-)automated 3D tools

Manual Contouring

With manual contouring you define which voxels are part of the segmentation and which are not. This allows you to create segmentations of any structeres that you may find in an image, even if they are not part of the human body. You might also use manual contouring to correct segmentations that result from sub-optimal automatic methods. The drawback of manual contouring is that you might need to define contours on many 2D slices. However, this is moderated by the interpolation feature, which will make suggestions for a segmentation.

Creating New Segmentations

Unless you want to edit existing segmentations, you have to create a new, empty segmentation before you can edit it. To do so, click the "New manual segmentation" button. Input fields will appear where you can choose a name for the new segmentation and a color for its display. Click the checkmark button to confirm or the X button to cancel the new segmentation. Notice that the input field suggests names once you start typing and that it also suggests colors for known organ names. If you use names that are not yet known to the application, it will automatically remember these names and consider them the next time you create a new segmentation.

Once you created a new segmentation, you can notice a new item with the "binary mask" icon in the Data Manager tree view. This item is automatically selected for you, allowing you to start editing the new segmentation right away.

Selecting Segmentations for Editing

As you might want to have segmentations of multiple structures in a single patient image, the application needs to know which of them to use for editing. You select a segmenation by clicking it in the tree view of Data Manager. Note that segmentations are usually displayed as sub-items of "their" patient image. In the rare case, where you need to edit a segmentation that is not displayed as a a sub-item, you can click both the original image AND the segmentation while holding down CTRL or for macOS the CMD on the keyboard.

When a selection is made, the Segmentation View will hide all but the selected segmentation and the corresponding original image. When there are multiple segmentations, the unselected ones will remain in the Data Manager, you can make them visible at any time by selecting them.

Selecting Editing Tools

If you are familiar with the MITK Workbench, you know that clicking and moving the mouse in any of the 2D render windows will move around the crosshair that defines what part of the image is displayed. This behavior is disabled while any of the manual segmentation tools are active – otherwise you might have a hard time concentrating on the contour you are drawing.

To start using one of the editing tools, click its button the the displayed toolbox. The selected editing tool will be active and its corresponding button will stay pressed until you click the button again. Selecting a different tool also deactivates the previous one.

If you have to delineate a lot of images, you should try using shortcuts to switch tools. Just hit the first letter of each tool to activate it (A for Add, S for Subtract, etc.).

Using Editing Tools

All of the editing tools work by the same principle: you use the mouse (left button) to click anywhere in a 2D window (any of the orientations axial, sagittal, or frontal), move the mouse while holding the mouse button and release to finish the editing action.

Multi-step undo and redo is fully supported by all editing tools. Use the application-wide undo button in the toolbar to revert erroneous actions.

QmitkSegmentation_IMGIconAddSubtract.png
Add and Subtract Tools

Use the left mouse button to draw a closed contour. When releasing the mouse button, the contour will be added (Add tool) to or removed from (Subtract tool) the current segmentation. Hold down the CTRL / CMD key to invert the operation (this will switch tools temporarily to allow for quick corrections).

QmitkSegmentation_IMGIconPaintWipe.png
Paint and Wipe Tools

Use the slider below the toolbox to change the radius of these round paintbrush tools. Move the mouse in any 2D window and press the left button to draw or erase pixels. As the Add/Subtract tools, holding CTRL / CMD while drawing will invert the current tool's behavior.

QmitkSegmentation_IMGIconRegionGrowing.png
Region Growing Tool

Click at one point in a 2D slice widget to add an image region to the segmentation with the region growing tool. Moving up the cursor while holding the left mouse button widens the range for the included grey values; moving it down narrows it. Moving the mouse left and right will shift the range.

Region Growing selects all pixels around the mouse cursor that have a similar gray value as the pixel below the mouse cursor. This enables you to quickly create segmentations of structures that have a good contrast to surrounding tissue, e.g. the lungs. The tool will select more or less pixels (corresponding to a changing gray value interval width) when you move the mouse up or down while holding down the left mouse button.


QmitkSegmentation_IMGIconCorrection.png
Correction Tool

You do not have to draw a closed contour to use the Correction tool and do not need to switch between the Add and Substract tool to perform small corrective changes. The following figure shows the usage of this tool:

  • if the user draws a line which starts and ends outside the segmenation AND it intersects no other segmentation the endpoints of the line are connected and the resulting contour is filled
  • if the user draws a line which starts and ends outside the segmenation a part of it is cut off (left image)
  • if the line is drawn fully inside the segmentation the marked region is added to the segmentation (right image)
QmitkSegmentation_IMGCorrectionActions.png
%Actions of the Correction tool illustrated.

QmitkSegmentation_IMGIconFill.png
Fill Tool

Left-click inside a segmentation with holes to completely fill all holes (left-click outside a segmentation).

QmitkSegmentation_IMGIconErase.png
Erase Tool

This tool removes a connected part of pixels that form a segmentation. You may use it to remove so called islands (see picture) or to clear a whole slice at once (left-click outside a segmentation).

QmitkSegmentation_IMGIconLiveWire.png
LiveWire Tool

The LiveWire Tool acts as a magnetic lasso with a contour snapping to edges of objects.

QmitkSegmentation_IMGLiveWireUsage.PNG
Steps for using LiveWire Tool
  • (1) To start the Tool you have to double click near the edge of the object you want to segment. The initial anchor point will snap to the edge within a 3x3 region.
  • (2) Move the mouse. You don't have trace the edge of the object. The contour will automatically snap to it.
  • (3) To fix a segment you can set anchor points by single left mouse button click.
  • (4) Go on with moving the mouse and setting anchor points.
  • (5) To close the contour double click on the initial anchor point.
  • (6) After closing the contour can be edited by moving, inserting and deleting anchor points.

The contour will be transfered to its binary image representation by deactivating the tool.

QmitkSegmentation_IMG2DFastMarchingUsage.png
2D Fast Marching Tool

Provides a fast marching based 2D interaction segmentation tool. You start with setting seedpoints in an image slice. Via several sliders you can adapt parameters and see the fast marching result instantly.

Interpolation

Creating segmentations for modern CT volumes is very time-consuming, because structures of interest can easily cover a range of 50 or more slices. The Manual Segmentation View offers two helpful features for these cases:

  • 3D Interpolation
  • 2D Interpolation


The 3D interpolation is activated by default when using the manual segmentation tools. That means if you start contouring, from the second contour onwards, the surface of the segmented area will be interpolated based on the given contour information. The interpolation works with all available manual tools. Please note that this is currently a pure mathematical interpolation, i.e. image intensity information is not taken into account. With each further contour the interpolation result will be improved, but the more contours you provide the longer the recalculation will take. To achieve an optimal interpolation result and in this way a most accurate segmentation you should try to describe the surface with sparse contours by segmenting in arbitrary oriented planes. The 3D interpolation is not meant to be used for parallel slice-wise segmentation.

QmitkSegmentation_3DInterpolationWrongRight.png
3D Interpolation HowTo

You can accept the interpolation result by clicking the "Accept" - button below the tool buttons. In this case the 3D interpolation will be deactivated automatically so that the result can be postprocessed without any interpolation running in background. During recalculation the interpolated surface is blinking yellow/white. When the interpolation has finished the surface is shown yellow with a small opacity. Additional to the surface, black contours are shown in the 3D render window. They mark the positions of all the drawn contours which were used for the interpolation. You can navigate between the drawn contours by clicking on the „Position“ - Nodes in the datamanager which are located below the selected segmentation. If you don't want to see these nodes just unckeck the „Show Position Nodes“ Checkbox and these nodes will be hidden. If you want to delete a drawn contour we recommend to use the Erase-Tool since Redo/Undo is not yet working for 3D interpolation. The current state of the 3D interpolation can be saved accross application restart. Therefor just click on save project during the interpolation is active. After restarting the application and load your project you can click on "Reinit Interpolation" within the 3D interpolation GUI area.


The 2D Interpolation creates suggestions for a segmentation whenever you have a slice that

  • has got neighboring slices with segmentations (these do not need to be direct neighbors but could also be a couple of slices away) AND
  • is completely clear of a manual segmentation – i.e. there will be no suggestion if there is even only a single pixel of segmentation in the current slice.

Interpolated suggestions are displayed in a different way than manual segmentations are, until you "accept" them as part of the segmentation. To accept single slices, click the "Accept" button below the toolbox. If you have segmented a whole organ in every-x-slice, you may also review the interpolations and then accept all of them at once by clicking "... all slices".

3D Segmenation tools

The 3D tools operate on the hole image and require usually a small amount of interaction like placing seed-points or specifying certain parameters. All 3D tools provide an immediate segmentation feedback, which is displayed as a transparent green overlay. For accepting a preview you have to press the "Comfirm" button of the selected tool. The following 3D tools are at your disposal:

3D Threshold tool

The Thresholding tool simply applies a 3D threshold to the patient image. All pixels with values equal or above the selected threshold are labeled. You can change the threshold by either moving the slider of setting a certain value in the spinbox.

QmitkSegmentation_3DThresholdTool.png
3D Threshold tool

3D Upper/Lower Threshold tool

The Upper/Lower Thresholding tool works similar to the simple 3D threshold tool but allows you to define an upper and lower threshold. All pixels with values within this threshold intervall will be labeled

QmitkSegmentation_3DULThresholdTool.png
3D Upper/Lower Threshold tool

3D Otsu tool

The 3D Otsu tool provides a more sophisticated thresholding algorithm. It allows you to define a number of regions. Based on the image histogram the pixels will then divided into different regions. There more regions you define the longer will the calculation take.

QmitkSegmentation_3DOtsuTool.png
3D Otsu tool

3D Fast Marching tool

The 3D Fast Marching tools works similar to the 2D pendant but on the hole image. Depending on you image's size the calculation will take some time. You can interactive set the parameters of the algorithm via the tool GUI.

QmitkSegmentation_3DFMTool.png
3D Fast Marching tool

3D Region Growing tool

The 3D Region Growing tool works similar to the 2D pendant. At the beginning you have to place a seedpoint and define a threshold intervall. If you press "Run segmentation" a preview is calculated, if the "3D preview" box is checked you will also see the result in 3D. By moving the "Adapt region growing slider" you can interactively adapt the result to you image.

QmitkSegmentation_3DRGTool.png
3D Region Growing tool

+

3D Watershed tool

This tool provides a watershed based segmentation algorithm.

QmitkSegmentation_3DWatershedTool.png
3D Watershed tool

Picking tool

The Picking tool allows you to select islands within your segmentation. This is especially usefull if e.g. a thresholding delivered your several areas within your image but you are just interested in one special region.

QmitkSegmentation_PickingTool.png
Picking tool

Things you can do with segmentations

As mentioned in the introduction, segmentations are never an end in themselves. Consequently, the Segmentation view adds a couple of "post-processing" actions to the Data Manager. These actions are accessible through the context-menu of segmentations in Data Manager's list view

QmitkSegmentation_IMGDataManagerContextMenu.png
Context menu items for segmentations.
  • Create polygon model applies the marching cubes algorithms to the segmentation. This polygon model can be used for visualization in 3D or other things such as stereolithography (3D printing).
  • Create smoothed polygon model uses smoothing in addition to the marching cubes algorithms, which creates models that do not follow the exact outlines of the segmentation, but look smoother.
  • Autocrop can save memory. Manual segmentations have the same extent as the patient image, even if the segmentation comprises only a small sub-volume. This invisible and meaningless margin is removed by autocropping.

Surface Masking

You can use the surface masking tool to create binary images from a surface which is used used as a mask on an image. This task is demonstrated below:

QmitkSegmentation_FromSurfaceBefore.png
Load an image and a surface.

Select the image and the surface in the corresponding drop-down boxes (both are selected automatically if there is just one image and one surface)

QmitkSegmentation_FromSurfaceAfter.png
Create segmentation from surface

After clicking "Create segmentation from surface" the newly created binary image is inserted in the DataManager and can be used for further processing

Technical Information for Developers

For technical specifications see Technical design of QmitkSegmentation and for information on the extensions of the tools system How to extend the Segmentation view with external tools .