MITK Tutorial - Step 10: Adding new Interaction

Table of Contents

• Step 01 - How to use an existing DataInteractor in your Module/Plugin/...
• Step 02 - How to modify the behaviour of a DataInteractor
  • How to modify the display interaction behavior
• How to implement a new DataInteractor
• How to create a Config-File
  • Event Description
  • Examples
  • Parameter Description
• HowTo Write a State Machine
  • States
  • Transitions
  • Event Class
  • Actions
  • Conditions
• Integration of the pattern and configuration files
• Implementation of a new mitk::DataInteractor
  • Example Interactor using InternalEvent

Step 01 - How to use an existing DataInteractor in your Module/Plugin/...

MITK provides finished DataInteractors for a variety of tasks, they can be found in Core/Code/Interactions. They can be used with state machine patterns and config files located under Core/Code/Resources/Interactions.

A mitk::DataInteractor consists of four parts. The class describing the functionality and two XML files; one describes the state machine pattern, that is the workflow of an interaction and the second describes the user events which trigger an action. Lastly every mitk::DataInteractor works on a mitk::DataNode in which it stores and manipulates data. To use a mitk::DataInteractor these parts have to be brought together.

TODO add code of mitk::PointSetDataInteractor Plugin ..

This code demonstrates the use of an existing mitk::DataInteractor exemplary for the mitk::PointSetDataInteractor:

First we need a mitk::DataNode in which the PointSets is stored. It has to be added to the mitk::DataStorage.

mitk::DataNode::Pointer dataNode = mitk::DataNode::New();
GetDataStorage()->Add(dataNode.GetPointer());

Then we create an instance of a mitk::PointSetDataInteractor and load a predefined statemachine pattern as well as a configuration for it:

m_CurrentInteractor = mitk::PointSetDataInteractor::New();
m_CurrentInteractor->LoadStateMachine("PointSet.xml");
m_CurrentInteractor->SetEventConfig("PointSetConfig.xml");

Lastly the mitk::DataNode is added to the mitk::DataInteractor

m_CurrentInteractor->SetDataNode(dataNode);

now the mitk::DataInteractor is ready for usage.

Step 02 - How to modify the behaviour of a DataInteractor

The behavior of a mitk::DataInteractor is determined by two aspects. One, the state machine pattern which describes the flow/order of actions that are performed. Secondly the configuration which determines which user interaction triggers the actions.

How to modify the display interaction behavior

Sometimes it may be desirable to change the behaviour of the mitk::DisplayActionEventBroadcast which controls zooming, panning and scrolling, e.g. when a Tool is activated that reacts to the same events. Changing the behavior of the DisplayActionEventBroadcast (or possibly any other EventHandler) can be achieved from anywhere in the code by requesting the InteractionEventObserver and assigning an alternative configuration to it, as demonstrated in this example:

std::ifstream* configStream = new std::ifstream( #path to alternative configuration file# );
mitk::EventConfig newConfig(configStream);
 
// Requesting all registered EventObservers
std::list<mitk::ServiceReference> listEventObserver = GetModuleContext()->GetServiceReferences<InteractionEventObserver>();
 
for (std::list<mitk::ServiceReference>::iterator it = listEventObserver.begin(); it != listEventObserver.end(); ++it)
{
    auto* displayActionEventBroadcast  = dynamic_cast<DisplayActionEventBroadcast*>(GetModuleContext()->GetService<InteractionEventObserver>(*it));
    // filtering: only adjust the displayActionEventBroadcast
    if (nullptr != displayActionEventBroadcast)
    {
        displayActionEventBroadcast->SetEventConfig(newConfig);
    }
}

How to implement a new DataInteractor

This second part of the tutorial step goes beyond the activation of an interactor, that modifies data by user interaction) as shown above. It shows what needs to be implemented to add a new way of interaction within your MITK application. Please see Interaction Concepts as an introduction to the MITK interaction mechanism.

This tutorial is structured as follows: The first section deals with config files, describing all the parameters of events and how to use them in a configuration file. In the second section the basics are described that are needed to write a state machine pattern. The last section deals with bringing configuration, state machine pattern and code together and gives an exemplary implementation of a mitk::DataInteractor.

How to create a Config-File

Event Description

Events are described by their parameters. Each event type has its own set of parameters that can be set in the configuration file. If a parameter is omitted it is set to its default value. All possible parameters are listed and described below. Event parameters are also described in the documentation of the event class itself.

Mandatory for each event description is the event class and the event variant. The parameters of an event are set by attribute tags.

Note

Refer to Event Class for the meaning of event class.

Mouse Buttons
mitk::InteractionEvent::MouseButtons represent the buttons. They can be used for two attributes. First the EventButton attribute which describes the button that triggered the event, this always is a single button. Secondly the ButtonState attribute that describes which buttons were pressed at the moment the event has been generated. For example assume the right mouse button and the middle mouse button are already pressed, now the left mouse button is pressed too and generates a second event, this would be described as follows:

<attribute name="EventButton" value="LeftMouseButton"/>
<attribute name="ButtonState" value="RightMouseButton,MiddleMouseButton"/>

Note: Technically the LeftMouseButton is also pressed and should be listed in the ButtonState, but this is taken care of by the mitk::EventFactory .

Key Events
mitk::InteractionKeyEvent represents a pressed key, which key is pressed is provided with the Key attribute like this

<attribute name="Key" value="A"/>

or

<attribute name="Key" value="Escape"/>

Note

Key Events do not require an explicit configuration, for all key events there exists a predefined event variant with the name 'Std' + value, that is key a is named 'StdA'.

The names for special keys are listed here:

    // Special Keys
    static const std::string KeyEsc;        // = "Escape";
    static const std::string KeyEnter;      // = "Enter";
    static const std::string KeyReturn;     // = "Return";
    static const std::string KeyDelete;     // = "Delete";
    static const std::string KeyArrowUp;    // = "ArrowUp";
    static const std::string KeyArrowDown;  // = "ArrowDown";
    static const std::string KeyArrowLeft;  // = "ArrowLeft";
    static const std::string KeyArrowRight; // = "ArrowRight";
 
    static const std::string KeyF1;  // = "F1";
    static const std::string KeyF2;  // = "F2";
    static const std::string KeyF3;  // = "F3";
    static const std::string KeyF4;  // = "F4";
    static const std::string KeyF5;  // = "F5";
    static const std::string KeyF6;  // = "F6";
    static const std::string KeyF7;  // = "F7";
    static const std::string KeyF8;  // = "F8";
    static const std::string KeyF9;  // = "F9";
    static const std::string KeyF10; // = "F10";
    static const std::string KeyF11; // = "F11";
    static const std::string KeyF12; // = "F12";
 
    static const std::string KeyPos1;     // = "Pos1";
    static const std::string KeyEnd;      // = "End";
    static const std::string KeyInsert;   // = "Insert";
    static const std::string KeyPageUp;   // = "PageUp";
    static const std::string KeyPageDown; // = "PageDown";
    static const std::string KeySpace;    // = "Space";
    // End special keys

Modifier Keys
mitk::InteractionEvent::ModifierKeys represent the combination of pressed modifier keys, several modifier keys pressed at the same time are denoted by listing them all separated by commas.

<!-- shift and control key are pressed -->
<attribute name="Modifiers" value="shift,ctrl"/>

ScrollDirection
This attribute is unique to the mitk::MouseWheelEvent and describes the direction in which the mouse wheel is rotated. In the event description actual only the direction is provided, but the event is generated with the actual value, and this value can be retrieved from the object.

<attribute name="ScrollDirection" value="up"/>
<!-- or -->
<attribute name="ScrollDirection" value="down"/>

Examples

Examples for key events:

<config>
  <!-- Event of key 'a' pressed -->
  <event_variant class="InteractionKeyEvent" name="StdA">
    <attribute name="Key" value="A"/>
  </event_variant>
  <!-- Event of key 'b' pressed  while modifiers ctrl and shift are pressed-->
  <event_variant class="InteractionKeyEvent" name="StdB">
    <attribute name="Key" value="B"/>
    <attribute name="Modifiers" value="shift,ctrl"/>
  </event_variant>
</config>

Examples for MousePress events:

 <!-- Standard left click -->
<config>
  <event_variant class="MousePressEvent" name="StdMousePressPrimaryButton">
    <attribute name="EventButton" value="LeftMouseButton"/>
  </event_variant>
<!-- right click with control key pressed-->
  <event_variant class="MousePressEvent" name="RightWithCTRL">
    <attribute name="EventButton" value="RightMouseButton"/>
    <attribute name="Modifiers" value="ctrl"/>
  </event_variant>
</config>

There exists a standard configuration file for the most common events called GlobalConfig.xml that can be used to as a default and can be extended by a specific definition.

Parameter Description

It is also possible to store parameters in the config file. Those are stored using the param-tag, like this:

<config name="example2">
 <param name="property1" value="yes"/>
 <param name="scrollModus" value="leftright"/>
</config>

Within the application these properties can then be access via a mitk::PropertyList like this:

// sm - state machine loaded with config file example2
 
mitk::PropertyList::Pointer properties = GetAttributes();
std::string prop1;
properties->GetStringProperty("property1",prop1);

HowTo Write a State Machine

A state machine pattern is described in a XML file.

States

States are described using the state-tag. Each state has to have a name. Exactly one state has to be a start state in each state machine to indicate the state in which the state machine is set when it is constructed. So a valid, but rather useless state machine would like like this:

<statemachine>
 <state name="start" startstate="true"/>
</statemachine>

Optionally a state can be assigned a special mode that influences the event distribution. These modes are GRAB_INPUT , PREFER_INPUT and REGULAR (where REGULAR is default and does not need to be indicated). See Event Distribution for a description of these modes. Use the special modes only when necessary as they prevent other DataInteractors to receive events.

<!-- example -->
<state name="someState" startstate="true" state_mode="GRAB_INPUT"/>

Transitions

Transitions are part of a state and describe all possible state switches, and are therefore important for modeling an interaction scheme. Transitions consist a part that describes the event which triggers the transition (event class and event variant) and a target which is state to which the state machine switches after executing a transition. An event class describes the event type (see mitk::InteractionEvent for the different classes) and the event variant is a specification thereof and the exact description is taken from a config file. Together they determine which event can trigger this transition. For example this state machine will switch from state A to state B when the StdMousePressPrimaryButton event (left mouse button is pressed) occurs.

Event Class

The event class description supports the polymorphism of the event classes. Therefore state machine patterns should be written in the most general ways possible. So for a given class hierarchy like this:

in the state machine pattern the mitk::InteractionPositionEvent can be declared as event class to restrict to the events which hold a position information. The actual implementation is then given in the configuration file. In this case it allows to define events of the classes mitk::InteractionPositionEvent itself, or mitk::MousePressEvent, mitk::MouseReleaseEvent, mitk::TouchEvent. This has the advantage that the patterns remain the same no matter what input devices are used, and the state machine patterns can be configured for newly added event classes as long as they match the class hierarchy (this ensures they hold the necessary properties).

<statemachine>
 <state name="A" startstate="true">
   <transition event_class="MousePressEvent" event_variant="StdMousePressPrimaryButton" target="B"/>
 <state/>
 <state name="B" />
</statemachine>

Actions

Actions can be added to transitions and represent functions in the mitk::DataInteractor that are executed on taking a transition. The following simple state machine will listen for left mouse clicks and execute two actions (and actually never stop).

<statemachine>
    <state name="start" startstate="true">
        <transition event_class="MousePressEvent" event_variant="StdMousePressPrimaryButton" target="start">
            <action name="addPoint"/>
            <action name="countClicks"/>
        </transition>
    </state>
</statemachine>

In order to tell the mitk::DataInteractor which function to execute these actions are made known to the mitk::DataInteractor using the CONNECT_FUNCTION macro. This example assumes that there exists an ExampleInteractor which inherits from mitkDataInteractor. This class implements the functions AddPoint and CountClicks. The actions are introduced by implementing the virtual method ConnectActionsAndFunctions():

void mitk::ExampleInteractor::ConnectActionsAndFunctions()
{
  CONNECT_FUNCTION("addPoint", AddPoint);
  CONNECT_FUNCTION("countClicks", CountClicks);
}

Conditions

Conditions can be added to transitions and represent functions in the mitk::DataInteractor that are executed on taking a transition. A condition is used to determine if a following action should be executed or not.

<statemachine>
    <state name="start" startstate="true">
        <transition event_class="MousePressEvent" event_variant="StdMousePressPrimaryButton" target="start">
            <condition name="checkPoint"/>
            <action name="addPoint"/>
            <action name="countClicks"/>
        </transition>
    </state>
</statemachine>

In order to tell the mitk::DataInteractor which function to execute these conditions are made known to the mitk::DataInteractor using the CONNECT_CONDITION macro. The ConnectActionsAndFunctions() method has to be augmented accordingly:

void mitk::ExampleInteractor::ConnectActionsAndFunctions()
{
  CONNECT_CONDITION("checkPoint", CheckPoint);
  CONNECT_FUNCTION("addPoint", AddPoint);
  CONNECT_FUNCTION("countClicks", CountClicks);
}

Integration of the pattern and configuration files

The usage of custom files slightly differs from the existing ones. Custom pattern and config files have to be stored in the /Resources/Interactions directory of the Module that they were designed for. When loading files from a module location into an interactor, the module has to be supplied as a parameter:

m_CurrentInteractor = mitk::CustomDataInteractor::New();
m_CurrentInteractor->LoadStateMachine("CustomStateMachinePattern.xml", us::GetModuleContext()->GetModule());
m_CurrentInteractor->SetEventConfig("CustomConfig.xml", us::GetModuleContext()->GetModule());

See Register statemachine patterns and configuration w/o the build system for a description.

Implementation of a new mitk::DataInteractor

DataInteractors are to inherit from mitk::DataInteractor. Their functionality is implemented in functions that follow this interface: For Actions:

bool SomeFunctionality(StateMachineAction* , InteractionEvent*);

For Conditions:

bool SomeFunctionality(const InteractionEvent*);

Your functions are connected with actions and conditions by implementing the function ConnectActionsAndFunctions(), e.g.

void mitk::ExampleInteractor::ConnectActionsAndFunctions()
{
 CONNECT_CONDITION("checkPoint", CheckPoint);
 CONNECT_FUNCTION("addPoint", AddPoint);
 CONNECT_FUNCTION("enoughPoints", EnoughPoints);
}

Now all that is left is to write a state machine pattern and a config file as is described in the tutorials.

To provide a useful example the mitk::PointSetDataInteractor is annotated with comments that describe the important parts for an implementation of a mitk::DataInteractor.

This step assumes knowledge of the Interaction concept described in Interaction Concepts and some background of the implementation. Please refer to these pages before proceeding.

Now all that is left it to write a state machine pattern and a config file as is described in the tutorials.

Example Interactor using InternalEvent

A useful tool in creating DataInteractors is mitk::InternalEvent which allows the mitk::DataInteractor to send signals on its own. The following will describe how to build a mitk::DataInteractor that allows to add points until a certain number of points is reached. The number of accepted points is provided in the config file as a parameter. So we start by writing a state machine pattern that add points until it receives an mitk::InternalEvent telling it, that enough points have been added.

<statemachine>
    <state name="start" startstate="true" >
        <transition event_class="MousePressEvent" event_variant="AddPointClick" target="start">
            <condition name="checkPoint"/>
            <action name="addPoint"/>
        </transition>
        <transition event_class="InternalEvent" event_variant="enoughPointsAdded" target="final">
            <action name="enoughPoints"/>
        </transition>
    </state>
    <state name="final">
    <--! dead state, nothing happens any more, once we reached this -->
    </state>
</statemachine>

In our config file we set the number of maximal points to 10, and define AddPointClick as a right mouse click with the ctrl button pressed.

<config>
  <param name="NumberOfPoints" value="10">
   <event_variant class="MousePressEvent" name="AddPointClick">
    <attribute name="EventButton" value="RightMouseButton"/>
    <attribute name="Modifiers" value="ctrl"/>
  </event_variant>
</config>

The implementation is described in the following.

See also

Step10.h

Step10.cpp

Implementation of protected functions:

ConnectActionsAndFunctions - Is inherited from mitk::InteractionStateMachine, here action strings from the xml are connected with functions in the mitk::DataInteractor (as seen above). In our example this looks like this:

void mitk::ExampleInteractor::ConnectActionsAndFunctions()
{
  // connect the action and condition names of the state machine pattern with function within
  // this DataInteractor
  CONNECT_CONDITION("checkPoint", CheckPoint);
  CONNECT_FUNCTION("addPoint", AddPoint);
  CONNECT_FUNCTION("enoughPoints", EnoughPoints);
}

ConfigurationChanged - Is called whenever a new configuration file is loaded (by the mitk::InteractionEventHandler super class), this function allows to implement initialization code that depends on configuration values. In our example we want to set the limit of allowed points:

void mitk::ExampleInteractor::ConfigurationChanged()
{
  // read how many points we accept from the config properties
  mitk::PropertyList::Pointer properties = GetPropertyList();
  std::string maxNumber;
  properties->GetStringProperty("NumberOfPoints", maxNumber);
  m_MaximalNumberOfPoints = atoi(maxNumber.c_str());
}

Next the actual functionality of the DataInteractor is implemented, by providing one function per action, following this prototype described before.

  private:
    bool AddPoint(StateMachineAction *, InteractionEvent *); // function to add new points
    bool EnoughPoints(StateMachineAction *,
                      InteractionEvent *); // function changes color of pointset to indicate, it is full
    bool CheckPoint(const InteractionEvent *interactionEvent); // function checks if the clicked point is valid

bool mitk::ExampleInteractor::AddPoint(StateMachineAction *, InteractionEvent *interactionEvent)
{
  // cast InteractionEvent to a position event in order to read out the mouse position
  // we stay here as general as possible so that a different state machine pattern
  // can reuse this code with MouseRelease or MouseMoveEvents.
  InteractionPositionEvent *positionEvent = dynamic_cast<InteractionPositionEvent *>(interactionEvent);
  if (positionEvent != nullptr)
  {
    // query the position of the mouse in the world geometry
    mitk::Point3D point = positionEvent->GetPositionInWorld();
    m_PointSet->InsertPoint(m_NumberOfPoints, point, 0);
    m_NumberOfPoints++;
    GetDataNode()->SetData(m_PointSet);
    GetDataNode()->Modified();
 
    if (m_NumberOfPoints != 0 && m_NumberOfPoints >= m_MaximalNumberOfPoints)
    {
      // create internal event that signal that the maximal number of points is reached
      InternalEvent::Pointer event = InternalEvent::New(nullptr, this, "enoughPointsAdded");
      // add the internal event to the event queue of the Dispatcher
      positionEvent->GetSender()->GetDispatcher()->QueueEvent(event.GetPointer());
    }
    // update the RenderWindow to show new points
    mitk::RenderingManager::GetInstance()->RequestUpdateAll();
    return true;
  }
  else
  {
    return false;
  }
}
 
bool mitk::ExampleInteractor::EnoughPoints(StateMachineAction *, InteractionEvent *)
{
  GetDataNode()->SetProperty("contourcolor", ColorProperty::New(1.0, 1.0, 0.0));
  mitk::RenderingManager::GetInstance()->RequestUpdateAll();
  return true;
} //-

If the conditions returns false the calling transition and the included actions will not be executed. If a condition fails the event is considered as untreated, and will be offered to other Interactors.

bool mitk::ExampleInteractor::CheckPoint(const InteractionEvent *interactionEvent)

Here we see an internal event used to signal that the point set reached the maximal number of allowed points. The event is created and added to the Dispatchers event queue.

      // create internal event that signal that the maximal number of points is reached
      InternalEvent::Pointer event = InternalEvent::New(nullptr, this, "enoughPointsAdded");
      // add the internal event to the event queue of the Dispatcher
      positionEvent->GetSender()->GetDispatcher()->QueueEvent(event.GetPointer());

Note

Internal events do not need any mapping to event variants. Their signal name is equivalent with the event variant.

There are also two documented classes implementing a mitk::DataInteractor and a mitk::InteractionEventObserver which can be looked at for further understanding:

See also

mitk::PointSetDataInteractor

mitk::DisplayActionEventBroadcast

Have fun with creating your own interaction and please think about contributing it to MITK!

If you meet any difficulties during this step, don't hesitate to ask on the MITK mailing list mitk-.nosp@m.user.nosp@m.s@lis.nosp@m.ts.s.nosp@m.ource.nosp@m.forg.nosp@m.e.net! People there are kind and will try to help you.

[Previous step] [Main tutorial page]