Table of Contents
- Overview
- Quick Reference
- Image and segmentation prerequisites
- Image selection and creating new segmentations
- Groups
- Labels vs. label instances
- Unlocking, changing color of, and hiding label instances
- Label highlighting
- Context menus
- Label search bar
- Label name and color suggestions
- Saving and loading label set presets
- Preferences
- Segmentation tool overview
- Selection tool
- 2D segmentation tools
- 3D segmentation tools
- Additional things you can do with segmentations
- Segmentation of 3D+t images
Overview
Segmentation is the act of separating an image into foreground and background subsets by manual or automated delineation, while the foreground is defined to be part of the segmentation. Such a segmented image subset is also called a label as it typically labels a specific region of interest. A multilabel segmentation may contain multiple labels organized in distinct groups. You can create multiple labels for different regions of interest contained within a single segmentation image. Labels in the same group cannot overlap each other but labels from different groups may overlap.
The MITK Segmentation Plugin allows you to create multilabel segmentations of anatomical and pathological structures in medical images. The plugin consists of three views:
- Segmentation View: Manual and (semi-)automatic segmentation
- Segmentation Utilities: Post-processing of segmentations
- Segmentation Task List: Optimized workflow for batches of segmentation tasks based on a user-defined task list
In this user guide, the features of the Segmentation View are described. For an introduction to the Segmentation Utilities or Segmentation Task List, refer to the respective user guides.
Quick Reference
Keyboard Shortcuts and Tips
| Category | Action | Shortcut / Method |
|---|---|---|
| Preferences | Open preferences | Ctrl+P |
| Label Management | Add new label to active group | Ctrl+L Ctrl+A |
| Add new label instance of active label | Ctrl+L Ctrl+I | |
| Delete active label instance | Ctrl+L Ctrl+D | |
| Toggle active label visibility | Ctrl+H | |
| Toggle to next label as active label | Ctrl+L Ctrl+N | |
| Label Renaming | Rename label | Ctrl+L Ctrl+R |
| Rename label (alternative) | Alt+Double-Click label | |
| Quick select suggestion in the Name/Rename dialog | Double-click suggestion | |
| Label Navigation | Select and center on label | Double-click label |
| Highlight label in views | Hover over label | |
| Show invisible labels | Shift+Hover | |
| Tool Modifiers | Invert tool behavior (Add ↔ Subtract, Paint ↔ Wipe) | Ctrl+Draw |
| Lasso Tool | Start/end segmentation | Double left-click |
| Add anchor point | Ctrl+Left-Click | |
| Delete anchor point | Del (when selected) | |
| Move anchor point | Click+Drag anchor | |
| Region Growing | Widen/narrow gray value range | Mouse Up/Down while drawing |
| Shift gray value range | Mouse Left/Right while drawing | |
| Selection Tool | Show label info popup | Hover over label |
| Select topmost label | Left-Click |
Important Behaviors
| Feature | Behavior |
|---|---|
| Locked labels | Act as "cookie cutters" - prevent override within same group |
| Group overlap | Labels from different groups can overlap (not overriding each other) |
| Tool crosshair | Crosshair movement is disabled when manual tools are active |
| Close tool | Uses region's label, not necessarily the active label |
| Fill/Erase tools | Only work on unlocked labels or active label regions |
| Preset loading | Label presets do not remove labels that are not affected by them. Also it never removes pixel content (except when moving to different group) |
Image and segmentation prerequisites
The Segmentation View has a few prerequisites regarding segmentations and their reference image:
- Images must be two or three-dimensional and may be either static or dynamic, e.g., are time-resolved resp. have different pixel values for different time steps.
- Images must be single-valued, i.e. CT, MRI or ultrasound. Images from color doppler or photographic (RGB) images are only partially supported (please be aware that some tools might not be compatible with this image type).
- Segmentations must be congruent to their reference images, e.g., their voxel grids must match.
Image selection and creating new segmentations
The view automatically selects a reference image from the Data Manager if one is available. If you have multiple images loaded, verify that the correct one is selected in the Image widget at the top of the Data selection section.
Once a reference image is selected, the view automatically selects a matching segmentation if available. When no segmentation exists, only a "Create new segmentation" button is displayed.
Click this button to create a new multilabel segmentation with an initial, empty label (unless configured otherwise in preferences). The new segmentation is added to the Data Manager as a child node of its reference image and is automatically selected for editing.
If multiple segmentations exist for the reference image, verify the correct one is selected. You can change the selection by clicking the Segmentation widget, or create an additional segmentation by clicking the collapsed "Create new segmentation" button next to the segmentation selector.
Groups
Segmentation images consist of at least a single group called "Group 0" in which the first default label is created. More groups can be added and removed but there will always be at least a single group. Labels of the same group cannot overlap each other. Labels of different groups may overlap each other.
For example, you could segment the whole heart as "Heart" label in "Group 0", add "Group 1" and create multiple labels of the anatomical details of the heart in that group. Naturally, all these labels lie within the extents of the "Heart" label of "Group 0" but in principle they are completely independent of "Group 0". Some pixels are now labelled twice, e.g., as "Heart" and "Left ventricle". Since the labels of "Group 1" cannot overlap each other, it is impossible to accidentally label a pixel as both "Left ventricle" and "Right ventricle".
If you would like to segment even more details you could create "Group 2" to have up to three labels per pixel. Nevertheless, groups are technically a flat data structure and cannot contain nested groups. It is all about possibly overlapping labels from distinct groups and spatially exclusive, non-overlapping labels within the same group.
Labels vs. label instances
The Segmentation View supports label instances. That is, segmenting multiple distributed entities of the same thing like metastases for example.
A label, as we used the term before, is already a single instance of itself but it may consist of multiple label instances.
If a label consists of multiple label instances, they each show their own distinct pixel value in square brackets as a clue for distinction and identification.
It is important to understand that this number is not a separate, consecutive index for each label. It is just the plain pixel value of the label instance, which is unique across all label instances of the whole segmentation.
Unlocking, changing color of, and hiding label instances
Label instances are locked by default: label instances from the same group cannot accidentally override pixels from other label instances. Locked label instances behave like cookie cutters for other label instances of the same group. You can unlock label instances to remove that protection from other label instances of the same group. Their pixel contents can then be overridden by other label instances of the same group.
Remember that label instances from distinct groups do not interact with each other. They can always overlap (not override) each other.
You can also change the color of label instances as well as show (default) or hide their pixel contents. The icons at the right side of the rows of the groups and labels widget reflect their state in all these regards.
Renaming of labels and label instances can be found in their content menu as shown further below.
Label highlighting
Especially if you have segmentations with many label instances or the label names are not telling, it can be nontrivial to match the pixel content of a label with its entry in the list of labels (called label inspector) in the Segmentation plugin. To mitigate this problem, MITK uses label highlighting. As long as you hover with the mouse cursor over a group, label or label instance in the segmentation plugin, the pixels of the respective label instances will be highlighted. Highlighted labels will visually pop out by being shown with full opacity (no transparency) while the opacity of all non-highlighted labels of the same segmentation will be reduced, e.g., they become more transparent.
By default, label instances that are set to be invisible are not shown while highlighted. By pressing the shift key, one can enforce also invisible label instances to be shown while highlighting.
Remark: The highlighting is supported in all views that use the label inspector (e.g. also the segmentation utilities).
Hint: You can also use the Selection tool to highlight labels by hovering over their pixels with the mouse cursor.
Context menus
Actions for organization of groups, labels, and label instances (as well as other operations) can be also found in their right-click context menus.
Most actions available in these context menus are self-explanatory or were already described above by other means of access like the tool button bar for adding and removing groups, labels, and label instances.
Note that renaming a label instance will make a separate label out of it, since all instances of the same label share a single common name.
Clear content only clears the pixels of a label instance but won't delete the actual label instance.
Groups can be locked and unlocked as a whole from their context menu, while label instances can be directly locked and unlocked outside the context menu as described further below.
Label search bar
The label search bar is located directly below the list of labels and provides quick access to label instances.
Type any substring into the search field to display a dropdown list showing all label instances containing that text. You can then:
- Press Return to select the first label instance in the list
- Use arrow keys to navigate the list before pressing Return
- Click on any entry with the mouse to select it directly
When a label instance is selected from the search results, it becomes active and the views center on the label's pixel content (equivalent to double-clicking a label).
Label name and color suggestions
Renaming and naming labels
When renaming a label (via CTRL+L CTRL+R, label context menu, or ALT+double left-click on a label instance) or creating a new label with enforced name suggestions in the preferences, a name dialog appears.
The dialog displays the current color and name of the label. Its behavior depends on whether name suggestions are enforced in the preferences:
Enforced suggestions mode:
- You must select a name from the suggestion list; direct name and color editing is disabled
- Use the filter field in the Label suggestion box to search for specific suggestions
Non-enforced mode:
- Type the label name directly and click on the color to change it
- Optionally enable Auto-filter suggestions to filter the suggestion list based on the current name input
- If auto-filter is disabled, use the filter field in the Label suggestion box to search suggestions
Hint: Double-click any suggestion to select it and automatically close the dialog.
Label suggestion file format
Label name and color suggestions are stored in a special version of the MITK Stacked Segmentation Format (JSON). The custom suggestion list must contain an array of objects, each with a mandatory "name" string and an optional "color" string. In addition, suggestions can also define additional information that should be associated with the label as well as the maximum number of instance occurrences allowed (unique, unlimited, max number).
You can configure which suggestion file to use and how suggestions are handled in the preferences (Ctrl+P) > Segmentation. For detailed information about the MITK Stacked Segmentation Format, refer to the format specification.
Since suggestions use the same format as segmentations stored in the MITK Stacked Segmentation format, you may use those as suggestions for other segmentations as well, if desired.
Saving and loading label set presets
Label set presets are useful to share a certain style or scheme between different segmentation sessions or to provide templates for new segmentation sessions.
Creating and applying presets
The properties of all label instances in all groups like their names, colors, and visibilities are saved as a label set preset by clicking on the 'Save label set preset' button. Label set presets are applied to any segmentation session by clicking on the 'Load label set preset' button.
Preset behavior
When loading a preset onto an existing segmentation:
- Label instances not defined in the preset remain unchanged - they are neither removed nor modified
- Label instances defined in the preset are added, if they don't exist, or have their properties updated otherwise (name, color, visibility)
- The pixel contents of any label remain untouched (see exception below)
- Exception: If a label is moved to a different group by the preset, the old label's pixel content will be removed
File format and compatibility
Label set presets are stored in a version of the MITK Stacked Segmentation Format (JSON). For detailed information about this format, refer to the respective in-depth documentation.
Since presets use the same format as segmentations stored in the MITK Stacked Segmentation format, you may use those as a preset for other segmentations as well, if desired.
Default preset
If you work on a repetitive segmentation task, manually loading the same label set preset for each and every new segmentation can be tedious. To streamline your workflow, you can set a default label set preset in the Segmentation preferences (Ctrl+P). When set, this label set preset will be applied to all new segmentations instead of creating the default red "Label 1" label instance.
Preferences
The Segmentation Plugin offers a number of preferences which can be set via the MITK Workbench application preferences (Ctrl+P):
Display options
- Compact view: Hide tool button texts and increase icon size to save screen space
- 2D display: Choose between drawing segmentations as outlines or transparent overlays
- Opacity factor: Adjust the transparency of segmentation overlays (slider from 0-100%)
- 3D display: Activate 3D rendering of segmentations (requires GPU)
- Data node selection mode: Show only selected nodes - ensures only the selected segmentation and its reference image are visible
Note: To control 3D rendering for individual segmentations, right-click the segmentation in the Data Manager and toggle the 3D visualization entry in the context menu.
Label management
- Default label set preset: Specify a label set preset file to use for all new segmentations instead of creating a default label
- Label creation: Choose between automatically assigning default names and colors, or being prompted to specify name and color for each new label
Label suggestions
- Built-in suggestions: Select from available built-in label suggestion sets
- External suggestions: Specify a custom label suggestions file (JSON format)
- Replace standard label suggestions by external: When checked, external suggestions replace built-in ones; when unchecked, they are combined
- Enforce suggestions: Require label names to be valid suggestions - custom label names are not allowed
- Suggest once per segmentation: Each suggestion is only offered once per segmentation to avoid duplicate labels. If this is active, occurrence constraints defined by the suggestions are ignored.
Segmentation tool overview
MITK offers a comprehensive set of slice-based 2D and (semi-)automated 3D segmentation tools. The manual 2D tools require some user interaction and can only be applied to a single image slice whereas the 3D tools operate on the whole image. The 3D tools usually only require a small amount of user interaction, i.e. placing seed points or setting / adjusting parameters. You can switch between the different toolsets by selecting the 2D or 3D tab in the segmentation view.
Selection tool
The Selection tool allows you to identify and select labels directly in the 2D render views. When the tool is active, hovering over labels highlights them and displays a popup near the mouse cursor showing useful information such as name, color, pixel value, and anatomical region (if defined).
If multiple labels from different groups overlap at the mouse position, all of them will be highlighted simultaneously (unless the "Check only active group" option is enabled).
The tool provides a Check only active group option. When enabled, only labels belonging to the same group as the currently active label will be highlighted by the mouse cursor.
Left-click to select the topmost highlighted label instance in the label manager and make it the active label.
2D segmentation tools
With 2D manual contouring you define which voxels are part of the segmentation and which are not. This allows you to create segmentations of any structures of interest in an image. You can also use manual contouring to correct segmentations that result from sub-optimal automatic methods. The drawback of manual contouring is that you might need to define contours on many 2D slices. However, this is mitigated by the interpolation feature, which will make suggestions for a segmentation.
To start using one of the editing tools, click its button from the displayed toolbox. The selected editing tool will be active and its corresponding button will stay pressed until you click the button again. Selecting a different tool also deactivates the previous one.
If you have to delineate a lot of images, shortcuts to switch between tools becomes convenient. For that, just hit the first letter of each tool to activate it (A for Add, S for Subtract, etc.).
All of the editing tools work by the same principle: using the mouse (left button) to click anywhere in a 2D window (any of the orientations axial, sagittal, or coronal), moving the mouse while holding the mouse button and releasing the button to finish the editing action. Multi-step undo and redo is fully supported by all editing tools by using the application-wide undo / redo buttons in the toolbar.
Remark: Clicking and moving the mouse in any of the 2D render windows will move the crosshair that defines what part of the image is displayed. This behavior is disabled as long as any of the manual segmentation tools are active - otherwise you might have a hard time concentrating on the contour you are drawing.
Add and subtract tools
Use the left mouse button to draw a closed contour. When releasing the mouse button, the contour will be added (Add tool) to or removed (Subtract tool) from the current segmentation. Adding and subtracting voxels can be iteratively repeated for the same segmentation. Holding CTRL / CMD while drawing will invert the current tool's behavior (i.e. instead of adding voxels, they will be subtracted).
| Category | Action | Shortcut |
|---|---|---|
| Tool Modifiers | Invert tool behavior (Add ↔ Subtract, Paint ↔ Wipe) | Ctrl+Draw |
Lasso tool
The tool is a more advanced version of the add/subtract tool. It offers you the following features:
- Generating a polygon segmentation (click left mouse button to set anchor point)
- Freehand contouring (like the add tool; press left mouse button while moving the mouse)
- Move anchor points (select an anchor point, press left mouse button while moving the mouse)
- Add new anchor points (press CTRL while click left mouse to add an anchor to the contour)
- Delete an anchor point (press Del while ancor point is selected)
- Segmentation can be added to the label (Add mode) or subtracted (Subtract mode)
To start a segmentation double left click where the first ancor point should be. To end the segmentation double left click where the last ancor point should be. Please note that:
- Features 3-6 are only available, if auto confirm is not activated
- Features 3-5 are not available for freehand contour segments
| Category | Action | Shortcut |
|---|---|---|
| Lasso Tool | Start/end segmentation | Double left-click |
| Add anchor point | Ctrl+Left-Click | |
| Delete anchor point | Del (when selected) | |
| Move anchor point | Click+Drag anchor |
Paint and wipe tools
Use the Size slider to change the radius of the round paintbrush tool. Move the mouse in any 2D window and press the left button to draw or erase pixels. Holding CTRL / CMD while drawing will invert the current tool's behavior (i.e. instead of painting voxels, they will be wiped).
| Category | Action | Shortcut |
|---|---|---|
| Tool Modifiers | Invert tool behavior (Add ↔ Subtract, Paint ↔ Wipe) | Ctrl+Draw |
Region growing tool
Click at one point in a 2D slice widget to add an image region to the segmentation with the region growing tool. Region Growing selects all pixels around the mouse cursor that have a similar gray value as the pixel below the mouse cursor. This allows to quickly create segmentations of structures that have a good contrast to surrounding tissue. The tool operates based on the current level window, so changing the level window to optimize the contrast for the ROI is encouraged. Moving the mouse up / down is different from left / right: Moving up the cursor while holding the left mouse button widens the range for the included grey values; moving it down narrows it. Moving the mouse left and right will shift the range. The tool will select more or less pixels, corresponding to the changing gray value range.
| Category | Action | Shortcut |
|---|---|---|
| Region Growing | Widen/narrow gray value range | Mouse Up/Down while drawing |
| Shift gray value range | Mouse Left/Right while drawing |
Fill tool
Left-click inside a region/segmentation to flood fill all connected pixels that have the same label with the active label. This tool will only work on regions of unlocked labels or on regions that are not labeled at all.
Erase tool
This tool removes a connected part of pixels that form a segmentation. You may use it to remove single segmented regions by left-click on specific segmentation region. This tool will only work and regions of unlocked labels or on regions of the active label.
Close tool
Left-click inside the region/segmentation to fill all "holes" (pixels labelled with another label or no label) inside the region. Therefore this tool behaves like a local closing operation. This tool will not work, when a non-labeled region is selected and holes of locked labels will not be filled.
Remark: This tool always uses the label of the selected region (even if this label is not the active label). Therefore you can use this tool on regions of the active label and of none locked labels (without the need to change the active label).
Live wire tool
The Live Wire Tool acts as a magnetic lasso with a contour snapping to edges of objects.
The tool handling is the same like the Lasso tool (see for more info), except it generates live wire contours instead of straight lines.
2D and 3D Interpolation
Creating segmentations using 2D manual contouring for large image volumes may be very time-consuming, because structures of interest may cover a large range of slices. The segmentation view offers two helpful features to mitigate this drawback:
- 2D Interpolation
- 3D Interpolation
The 2D Interpolation creates suggestions for a segmentation whenever you have a slice that
- has got neighboring slices with segmentations (these do not need to be direct neighbors but could also be a couple of slices away) AND
- is completely clear of a manual segmentation, i.e. there will be no suggestion if there is even only a single pixel of segmentation in the current slice.
Interpolated suggestions are displayed in a different way than manual segmentations are, until you "accept" them as part of the segmentation. To accept single slices, click the "Accept" button below the toolbox. Once you are happy with the interpolations between two neighboring slices, navigate to one of the two parent slices (the ones that you did the manual contouring on) and click the 'Accept all interpolations' button (see above).
The 3D interpolation creates suggestions for 3D segmentations. That means if you start contouring, from the second contour onwards, the surface of the segmented area will be interpolated based on the given contour information. The interpolation works with all available manual tools. Please note that this is currently a pure mathematical interpolation, i.e. image intensity information is not taken into account. With each further contour the interpolation result will be improved, but the more contours you provide the longer the recalculation will take. To achieve an optimal interpolation result and in this way a most accurate segmentation you should try to describe the surface with sparse contours by segmenting in arbitrary oriented planes. The 3D interpolation is not meant to be used for parallel slice-wise segmentation, but rather segmentations in i.e. the axial, coronal and sagittal plane.
You can accept the interpolation result by clicking the Confirm-button below the tool buttons. In this case the 3D interpolation will be deactivated automatically so that the result can be post-processed without any interpolation running in the background.
Additional to the surface, black contours are shown in the 3D render window, which mark all the drawn contours used for the interpolation. You can navigate between the drawn contours by clicking on the corresponding position nodes in the data manager which are stored as sub-nodes of the selected segmentation. If you do not want to see these nodes just uncheck the Show Position Nodes checkbox and these nodes will be hidden.
If you want to delete a drawn contour we recommend to use the Erase-Tool since undo / redo is not yet working for 3D interpolation. The current state of the 3D interpolation can be saved across application restart. For that, just click on save project during the interpolation is active. After restarting the application and load your project you can click on "Reinit Interpolation" within the 3D interpolation GUI area.
3D segmentation tools
The 3D tools operate on the whole image and require usually a small amount of interaction like placing seed-points or specifying certain parameters. All 3D tools provide an immediate segmentation feedback, which is displayed as a transparent green overlay. For accepting a preview you have to press the Confirm button of the selected tool. The following 3D tools are available:
nnInteractive
| Category | Action | Shortcut |
|---|---|---|
| nnInteractive | Switch prompt type (positive ↔ negative) | T |
| Toggle point interaction | P | |
| Toggle box interaction | B | |
| Toggle scribble interaction | S | |
| Toggle lasso interaction | L | |
| Undo last interaction | U | |
| Reset all interactions | R | |
| Confirm segmentation | C | |
| Label management | Add and select new label | CTRL+L, CTRL+A |
These shortcuts also appear in the tool's button labels. You can hide them by disabling Show keyboard shortcuts on the nnInteractive preferences page.
The nnInteractive tool is a state-of-the-art AI solution for fully segmenting organs and other anatomical structures with minimal user interaction - often a single click is sufficient. This capability comes with significant resource requirements, including several gigabytes of disk space and a powerful GPU. Although nnInteractive can run on CPUs, performance is orders of magnitude slower compared to modern GPUs with at least 6 GB of VRAM. A truly interactive experience is only achievable on high-performance GPUs such as an Nvidia GeForce 4090, which can typically deliver response times of roughly one second per interaction.
Installation
Before its first use, nnInteractive must be installed. While it is not installed, the Initialize button reads Install nnInteractive; clicking it asks which kind of installation you want. Choose Full to install nnInteractive together with PyTorch and a model checkpoint; this enables both local inference on your own machine and connecting to a remote server. Choose Client only for a lightweight installation that can connect to a remote nnInteractive server but cannot run inference locally. If you are unsure, pick Full.
After you confirm the mode, an installation dialog appears. In most cases, you can simply confirm it and wait a few minutes for the installation to complete; a full installation requires several gigabytes of disk space. If installation fails, several advanced settings are available to assist with troubleshooting. These options require some technical understanding.
nnInteractive is installed into a private virtual environment that is reused across MITK updates. If the installed version is too old for the current version of MITK, or when a newer compatible release becomes available, MITK offers to update nnInteractive in place. The update is offered automatically when you initialize, and you can also start it at any time with the Check for updates button on the nnInteractive preferences page. An in-place update is applied while nnInteractive is not yet loaded, so it normally takes place at the first initialization after MITK starts. The Uninstall nnInteractive button on the same preferences page removes the environment entirely, for example to switch between Full and Client only mode.
Initialization
nnInteractive must be initialized before it can operate on an image. For a local session, the model is loaded onto the compute device during initialization, which may take several seconds; the very first initialization after installation may take up to a minute because the model checkpoint is downloaded from the internet first. For a remote session, MITK connects to the configured server instead.
Once initialization completes successfully, you can begin using the nnInteractive tool. If you run locally and no supported GPU is detected, a warning informs you that nnInteractive will run slowly and recommends hardware that enables a fully interactive workflow.
The Initialize button is a toggle: while a session is running it is labeled Uninitialize, which you can click to end the session and release its resources without leaving the tool. When nnInteractive is configured to use a remote server, the button reads Initialize (remote server).
Next to Initialize is a Settings button that opens the nnInteractive preferences page, where you configure how and where inference runs.
Under Inference you choose between Local (the default), which runs the model in this application's own Python environment, and Remote server, which offloads inference to an nnInteractive server that provides the model and GPU. For a remote server, enter its Server URL and, if required, an API key; when the key field is left empty, the NN_INTERACTIVE_API_KEY environment variable is used instead. An API key entered here is stored in plain text in the application preferences. A client-only installation can only run remotely, so its inference mode is fixed to Remote server.
Under Model, nnInteractive uses a recommended checkpoint that is downloaded from Hugging Face by default. You can select a different checkpoint from the list or point nnInteractive to a checkpoint folder already present on disk. When you run locally with a downloaded checkpoint and a newer recommended checkpoint becomes available, MITK offers to switch to it once per application run. Once a session is running, the checkpoint's license is shown in the tool, directly below the Initialize button.
Under Computation backend you can switch between Auto (default), CPU, and GPU. When selecting GPU, you may also specify the device number if you are working on a multi-GPU system. On Linux with a CUDA GPU, an additional option, enabled by default, uses torch.compile for faster interactions at the cost of a slightly longer initialization. The Storage backend setting controls how interaction data is held in memory: Auto (default) balances speed and memory, blosc2 minimizes memory use, and tensor maximizes speed. The computation and storage backend settings apply to local sessions only; a remote server provides its own compute device and interaction storage. If a local session fails to initialize, switching the computation backend or choosing a specific device here is a good first troubleshooting step.
The inference mode, server, model, and backend settings define how a session runs, so changing any of them ends a running session; the new setting takes effect the next time you initialize.
Interactions
After a session has been initialized, you can begin interacting by placing points, drawing bounding boxes, outlining contours, or scribbling with simple mouse clicks. Each interaction can be set to positive or negative, thereby adding or removing areas from the proposed segmentation. Interactions are shown on the 2D slice on which you draw them; they are not rendered in the 3D window.
While you work, the proposed segmentation is shown as a live preview in the color of the active label. The Confirm button names the label that the result will be written to and shows its color, for example Confirm Liver, so you always know where the segmentation will go. When you are satisfied with the result - often after only a single point or scribble - click it (or press C) to write the segmented pixels to that label.
The Auto-zoom option, enabled by default, lets the model widen the region it analyzes when a segment reaches the edge, so large structures are not cut off. It improves completeness at the cost of speed and is not available on the CPU.
If the active label already contains a segmentation, nnInteractive uses it as a starting point, so you can refine an existing result instead of starting over. With Auto-refine preexisting label content enabled, the model refines that content immediately; with it disabled, the existing content is kept as-is until your next interaction. Refining preexisting content requires a model or server that supports it.
You can undo the most recent interaction with the Undo button (or press U). nnInteractive supports a single level of undo, so after undoing once nothing more can be undone until you add another interaction. To discard all current interactions and start over, use Reset (or press R).
Within a single session, you can switch the active label at any time to segment multiple structures without reinitializing nnInteractive. Two automation options on the preferences page streamline this workflow: Auto-create next label prepares a new empty label as soon as you confirm one, so you can keep segmenting, and Auto-confirm after single interaction confirms automatically after your first interaction, which is convenient when a single interaction per label is enough.
The tool's Undo and the application-wide undo work at different levels, and both stay available while the tool is active. The tool's Undo (U) reverts only your most recent interaction within the current, unconfirmed result; there is no further interaction history to step back through. Confirming a segmentation is recorded as a single step on the application-wide undo stack, just like the other editing tools, so a confirmed label is reverted with the application-wide Undo in the toolbar (CTRL+Z) rather than with the tool's Undo. While you have an unconfirmed result in progress, the application-wide Undo does not affect your current interactions; it reverts a previously confirmed label or an earlier operation instead. In short, press U to take back your last interaction and CTRL+Z to take back a confirmed label.
TotalSegmentator Tool
This tool provides a GUI to the deep learning-based segmentation algorithm called TotalSegmentator. With it you can predict segmentation masks for a large number of anatomical structures in CT and MR images loaded in MITK. For the list of tasks and supported classes, refer to https://github.com/wasserth/TotalSegmentator
TotalSegmentator runs in a Python virtual environment that MITK creates and manages for you using its own embedded Python; no separate system Python installation is required. For best performance, a CUDA-enabled GPU is recommended.
Workflow
- Select the TotalSegmentator tool in the Segmentation View. If TotalSegmentator is not installed yet, click "Install TotalSegmentator". A dialog shows the installation progress; it downloads several packages and may take a while, but this is a one-time step.
- Once installed, choose a task from the drop-down. The default "total" segments the whole body. For the "total" and "total_mr" tasks, faster lower-resolution variants ("fast", 3 mm, and "fastest", 6 mm) are offered as separate entries. Tasks that require a (free, non-commercial) TotalSegmentator license are marked "License required:" and cannot be run until a license is set (see below).
- Click "Run TotalSegmentator" to compute a preview. On the first run of a task, the required model weights are downloaded automatically. The run executes in a background process, so the application stays responsive and can be cancelled.
- Confirm the preview to transfer it into your segmentation.
Click "Settings" in the tool (or open Preferences (Ctrl+P) > TotalSegmentator) to choose the compute device (a specific GPU, or CPU), enter a TotalSegmentator license to unlock the licensed tasks, clear the downloaded model weights, or uninstall the managed environment.
3D Threshold tool
The thresholding tool simply applies a 3D threshold to the patient image. All pixels with values equal or above the selected threshold are labeled as part of the segmentation. You can change the threshold by either moving the slider of setting a certain value in the spinbox.
3D upper / lower threshold tool
The Upper/Lower Thresholding tool works similar to the simple 3D threshold tool but allows you to define an upper and lower threshold. All pixels with values within this threshold interval will be labeled as part of the segmentation.
3D Otsu tool
The 3D Otsu tool provides a more sophisticated thresholding algorithm. It allows you to define a number of regions. Based on the image histogram the pixels will then be divided into different regions. The more regions you define the longer the calculation will take. You can select afterwards which of these regions you want to confirm as segmentation.
3D GrowCut tool
The 3D GrowCut tool uses previously created segmentation labels (e.g. by the "Add"-tool) stored in the segmentation layer 0. The GrowCut tool will use these segmentation labels to create a seedimage that will serve as input to the algorithm. As an advanced setting option, a Distance penalty can be set, which increases the cohesion in the immediate surroundings of the initial labels. Based on the seedimage and the Distance penalty, a growing is started, which includes all areas that are not initially assigned to a specific label. During this process, initially unassigned areas are assigned to the best fitting labels. After the segmentation process, the user can decide which newly generated labels should be confirmed.
Picking Tool
The Picking tool offers two modes that allow you to manipulate "islands" within your segmentation. This is especially useful if e.g. a thresholding provided you with several areas within your image but you are just interested in one special region.
- Picking mode: Allows you to select certain "islands". When the pick is confirmed, the complete content of the active label will be removed except the pick. This mode is beneficial if you have a lot segmentation noise and want to pick the relevant parts and dismiss the rest. Hint: You can also pick from other labels, but this will only work if these labels are unlocked.
- Relabel mode: Allows you to select certain "islands". When the pick is confirmed, it will be relabeled and added to the active label content. Hint: This mode ignores the locks of other labels, hence you do not need to unlock them explicitly.
Additional things you can do with segmentations
Segmentations are never an end in themselves. Consequently, the segmentation view adds a couple of "post-processing" actions, accessible through the context-menu of the data manager.
- Create polygon model applies the marching cubes algorithm to the segmentation. This polygon model can be used for visualization in 3D or other applications such as stereolithography (3D printing).
- Create smoothed polygon model uses smoothing in addition to the marching cubes algorithm, which creates models that do not follow the exact outlines of the segmentation, but look smoother.
- Autocrop can save memory. Manual segmentations have the same extent as the patient image, even if the segmentation comprises only a small sub-volume. This invisible and meaningless margin is removed by autocropping.
Segmentation of 3D+t images
For segmentation of 3D+t images, some tools give you the option to choose between creating dynamic and static masks.
- Dynamic masks can be created on each time frame individually.
- Static masks will be defined on one time frame and will be the same for all other time frames.
In general, segmentation is applied on the time frame that is selected when execution is performed. If you alter the time frame, the segmentation preview is adapted.
