Interaction in MITK

Interaction is one of the most important tasks in clinically useful image processing software. Due to that, MITK has a special interaction concept, with which the developer can map the desired interaction. For a simple change in interaction he doesn't have to change the code. All information about the sequence of the interaction is stored in an XML-file that is loaded by the application during startup procedure at runtime. That even allows the storage of different interaction patterns, e.g. an interaction behaviour like in MS PowerPoint, in Adobe Photoshop or like the interaction behaviour on a medical image retrieval system.

Statemachines to implement Interaction

The interaction in MITK is implemented with the concept of state machines (by Mealy). This concept allows to build the steps of interaction with different states, which each have different conditions, very alike the different interactions that may have to be build to develop medical imaging applications. Furthermore state machines can be implemented using object oriented programming (OOP). Due to that we can abstract from the section of code, that implements the interaction and focus on the sequence of interaction. What steps must the user do first before the program can compute a result? For example he has to declare three points in space first and these points are the input of a filter so only after the definition of the points, the filter can produce a result. The according interaction sequence will inform the filter after the third point is set and not before that. Now the filter after an adaption only needs two points as an input. The sequence of the interaction can be easily changed if it is build up as a sequence of objects and not hard implemented in a e.g. switch/case block. Or the user wants to add a point in the scene with the right mouse button instead of the left. Wouldn't it be nice to only change the definition of an interaction sequence rather than having to search through the code and changing every single if/else condition?

State Machine

So a separation of the definition of a sequence in interaction and its implementation is a useful step in the development of an interactive application. To be able to do that, we implemented the concept of state machines with several classes: States, Transitions and Actions define the interaction pattern. The state machine itself adds the handling of events, that are sent to it.

Example A:

A deterministic Mealy state machine has always one current state (here state 1). If an event 1 is sent to the state machine, it searches in its current state for a transition that waits for event 1 (here transition 1). The state machine finds transition 1, changes the current state to state2, as the transition points to it and executes actions 1 and 2. Now state 2 is the current state. The state machine receives an event 2 and searches for an according transition. Transition 2 waits for event 2, and since the transition leads to state 2 the current state is not changed. Action 3 and 4 are executed. Now Event 3 gets send to the state machine but the state machine can't find an according transition in state 2. Only transition 2 , that waits for event 2 and transition 4, that waits for event 4 are defined in that state. So the state machine ignores the event and doesn't change the state or execute an action. Now the state machine receives an event 4 and finds transition 3. So now the current state changes from state 2 to state 1 and actions 5 and 1 are executed.

Several actions can be defined in one transition. The execution of an action is the active part of the state machine. Here is where the state machine can make changes in data, e.g. add a Point into a list.

See mitk::StateMachine, mitk::State, mitk::Event, mitk::Action, mitk::Transition, mitk::Interactor

Guard States

Guard States are a special kind of states. The action, that is executed after the state is set as current state, sends a new event to the state machine, which leads out of the guard state. So the state machine will only stay in a guard state for a short time. This kind of state is used to check different conditions, e.g. if an Object is picked or whether a set of points will be full after the addition of one point.

Example B:

Event 1 is sent to the state machine. This leads the current state from state 1 into state check. The action 1 is executed. This action checks a condition and puts the result into a new event, that is sent and handled by the same (this) state machine. E.g. is the object picked with the received mouse-coordinate? The event, that is generated, will be Yes or No. In case of event No, the state machine sets the current state back to state 1 and executes action 2. In case of event Yes, the state machine changes the state from state check into state 2 and executes action 3, which e.g. can select said object.

Definition of a State machine

Due to the separation of the definition of an interaction sequence and its implementation, the definition has to be archived somewhere, where the application can reach it during startup and build up all the objects (states, transitions and actions) that represent the sequence of a special interaction. In MITK, these informations are defined in an XML-file (i.e. DisplayInteraction.xml or BoundingShapeInteraction.xml)

Note: Please note that since this is a resource which is compiled into the executable, changes you make to this file will only be reflected in application behavior after you recompile your code.

The structure is the following (from Example A:) :

<stateMachine NAME="statemachine1"><!-- defining a new state machine with its name -->
  <state NAME="State1" ID="1" START_STATE="TRUE"><!-- new state tag; start state of the state machine; ID=1 -->
    <transition NAME="transition1" NEXT_STATE_ID="2" EVENT_ID="1"><!-- transition, waits for event ID=1 and leads to state 2 -->
      <action ID="1" /><!-- action ID = 1 shall be executed --> 
      <action ID="2" /><!-- action ID = 2 shall be executed --> 
    </transition><!-- end of transition -->
  </state><!-- end of state ID = 1 -->
  <state NAME="state2" ID="2">
    <transition NAME="transition2" NEXT_STATE_ID="2" EVENT_ID="2">
      <action ID="3" />
      <action ID="4" />
    </transition>
    <transition NAME="transition3" NEXT_STATE_ID="1" EVENT_ID="4" >
      <action ID="5" />
      <action ID="1" />
    </transition>
  </state>
</stateMachine>

The identification numbers (ID) inside a state machine have to be unique. Each state machine has to have one state, that is defined as the start-state of that state machine. This means, initially, the current state of the state machine is the start-state. The Event-Ids seen above are also defined in the statemachine.xml file. They specify a unique number for a combination of input-conditions (key, mouse and so on). See Events for further informations.

The statemachine is compiled into an application at compile time. The definition of one single state machine is called the statemachine-pattern. Since this pattern is build up during startup with objects (states, transitions and actions) and these objects only hold information about what interaction may be done at the current state, we can also reuse the pattern.

Note: You as a developer don't necessarily have to implement your own XML-File! We already have defined some interaction-patterns (e.g. for setting Points in 2D or 3D) which you can use and adapt.

Reuse of Interaction Patterns

If we for example have a pattern called "pointset", which defines how the user can set different points into the scene and there is an instance of a state machine called "PointSetInteractor". This state machine has a pointer pointing to the current state in its assigned state machine pattern. Several events are send to the state machine, which moves the pointer from one state to the next, according to the transitions, and executes the actions, referenced in the transitions. But now a new instance of the class "PointSetInteractor" has to be build. So we reuse the pattern and let the current state pointer of the new object point to the start state of the pattern "pointset". The implementation of the actions is not done inside a class of the pattern (state, transition, action), it is done inside a state machine class (see the reference for mitkStatemachine).

Events

During runtime, events are thrown from e.g. the mouse to the operating system, are then send to your graphical user interface and from there it has to be send to the MITK-object called mitkEventMapper. This class maps the events received with an internal list of all events that can be understood in MITK. The definition of all understandable events is also located in the XML-File the state machines are defined in. If the received event can be found in the list, an internal mitk-eventnumber is added to the event and send to the object mitkGlobalInteraction.

See mitk::Event, mitk::GlobalInteraction

GlobalInteraction

This object administers the transmission of events to registered state machines. There can be two kinds of state machines, the ones that are only listening and ones that also change data. Listening state machines are here called Listeners and state machines that also change data are called Interactors.

Note: The discrimination between Listener and Interactor is only made in mitkGlobalInteraction. As Listener an object derived from class StateMachine can be added and removed from GlobalInteraction and as Interactor an object derived from class Interactor can be added and removed. See the interaction class diagram for further information.

To add or remove a state machine to the list of registered interactors, call AddInteractor or RemoveInteractor of GlobalInteraction or to add or remove a listener call AddListener of RemoveListener. Listeners are always provided with the events. Interactors shall only be provided with an event, if they can handle the event. Because of that the method CanHandleEvent is called, which is implemented in each Interactor. This method analyses the event and returns a value between 0 (can't handle event) and 1 (Best choice to handle the event). Information, that can help to calculate this jurisdiction can be the bounding box of the interacted data and the picked mouse-position stored in the event.

So after the object GlobalInteraction has received an event, it sends this event to all registered Listeners and then asks all registered Interactors through the method CanHandleEvent how good each Interactor can handle this event. The Interactor which can handle the event the best receives the event. Also see the documented code in mitkGlobalInteraction.

To not ask all registered interactors on a new event, the class Interactor also has a mode, which can be one of the following: deselected, subselected (deprecated since HierarchicalInteraction has been removed), selected. These modes are also used for the event mechanism. If an interactor is in a state, where the user builds up a graphical object, it is likely that the following events are also for the build of the object. Here the interactor is in mode selected as long as the interactor couldn't handle an event. Then it changes to mode deselected. The mode changes are done in the actions through operations (described further down) and so declared inside the interaction pattern.

See mitk::GlobalInteraction

Interactors

The class Interactor is the superclass for all state machines, that solve the interaction for a single data-object. An example is the class mitkPointSetInteractor which handles the interaction of the data mitkPointSet. Inside the class mitkPointSetInteractor all actions, defined in the interaction-pattern "pointsetinteractor", are implemented. Inside the implementation of these actions (ExecuteAction(...) ), so called mitkOperations are created, filled with information and send to the mitkUndoController and to mitkOperactionActor (the data, the interaction is handled for).

See mitk::Interactor

Executing Operations

The class mitkOperation and its subclasses basically holds all information needed to execute a certain change of data. This change of data is only done inside the data-class itself, which is derived from the interface mitkOperationActor. Interactors handle the interaction through state-differentiation and combine all informations about the change in a mitkOperation and send this operation-object to the method ExecuteOperation (of data-class). Here the necessary data is extracted and then the change of data is performed.

When the operation-object, here called do-operation, is created inside the method ExecuteAction (in class mitkInteractor), an undo-operation is also created and together with the do-operation stored in an object called OperationEvent. After the Interactor has sent the do-operation to the data, the operation-event-object then is sent to the instance of class mitkUndoController, which administrates the undo-mechanism.

See mitk::Operation, mitk::OperationActor

UndoController

The instance of class mitkUndoController administrates different Undo-Models. Currently implemented is a limited linear Undo. Only one Undo-Model can be activated at a time. The UndoController sends the received operation events further to the current Undo-Model, which then stores it according to the model. If the method Undo() of UndoController is called (e.g. Undo-Button pressed from ) the call is send to the current Undo-Model. Here the undo-operation from the last operation event in list is taken and send to the data, referenced in a pointer which is also stored in the operation-event. A call of the method Redo() is handled accordingly.

See mitk::UndoController, mitk::LimitedLinearUndo

References

[Bin99] Robert V. Binder. Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley, 1999

Table of Contents