Multimodal Architecture and Interfaces

 is an open standard developed by the World Wide Web Consortium since 2005. It was published as a Recommendation of the W3C on October 25, 2012. The document is a technical report specifying a multimodal system architecture and its generic interfaces to facilitate integration and multimodal interaction management in a computer system. It has been developed by the W3C's Multimodal Interaction Working Group.
Description.
The "Multimodal Architecture and Interfaces" recommendation introduces a generic structure and a communication protocol to allow the modules in a multimodal system to communicate with each other.
This specification proposes an event-driven architecture as a general frame of reference focused in the control flow data exchange. It can be used to determine the basic infrastructures needed to command the application's multimodal services.
The architecture is also proposed to facilitate the task of implementing several types of multimodal services providers on multiple devices: mobile devices and cell phones, home appliances, Internet of Things objects, television and home networks, enterprise applications, web applications, "smart" cars or on medical devices and applications.
Logical structure.
"Multimodal Architecture and Interfaces" is the specified description of a larger services infrastructure called The Runtime Framework which provides the main functions that a multimodal system can need. This framework is at a higher level of abstraction than the MMI architecture. The MMI Runtime Framework is the runtime support and communication modules of the multimodal system while MMI Architecture is the description and the specification of its main modules, its interfaces and its communication modes.
The "Multimodal Architecture and Interfaces" specification is based on the MVC design pattern, that proposes to organize the user interface structure in three parts: the "Model", the "View" and the "Controller". This design pattern is also shown by the Data-Flow-Presentation architecture from the Voice Browser Working Group.
A particularity of this architecture is that although the presentation layer represented by the"View" has traditionally been associated with graphical interfaces; this recommendation abstraction generalizes the "View" to the broader context of the multimodal interaction, where the user can use a combination of visual, auditory, biometric and / or tactile modalities.
The MMI Architecture recommendation distinguishes three types of components: the Interaction Manager – IM, the Data Component – DC and the Modality Components – MC. This distinction is similar to the separation between the "Controller", the "Model" and the presentation documents of the "View" in the MVC pattern.
Another characteristic is recursion. The modules are black boxes and it is possible to encapsulate several components in a more complex component, which communicate with an Interaction Manager at a higher level. In this way, the architecture follows the nested dolls principle .
The specification also covers the issues of a distributed implementation on multiple material resources in a network or a centralized implementation, with all the modules installed in a single material support. Information sharing between modules is loose coupled. This promotes low dependence between modules, reducing the impact of changes in one module on other modules, and facilitating the modules' reuse. In this way, the modules have little or no knowledge of the functioning of any other modules and the communication between modules is done through the exchange of messages following a precise communication protocol provided by the architecture's API.
The MMI architecture modules.
The interaction manager.
The interaction manager is a logical component, responsible for all message exchanges between the components of the system and the multimodal Runtime Framework. It is a communication bus and also an event handler.
Each application can configure at least one Interaction Manager to define the required interaction logic. This controller is the core of the multimodal interaction:
The modality components.
The modality components are responsible for specific tasks, including handling inputs and outputs in various ways, such as speech, writing, video, etc..
These are logical entities that handles the input and output of different hardware devices (microphone, graphic tablet, keyboard) and software services (motion detection, biometric changes) associated with the multimodal system. For example, (see figure below), a modality component A can be charged at the same time of the speech recognition and the audio input management. Another modality component B can manage the complementary command inputs on two different devices: a graphics tablet and a microphone. Two modality components C, can manage separately two complementary inputs given by a single device: a camcorder. And finally, a modality component D, can use an external recognition web service and only be responsible for the control of communication exchanges needed for the recognition task.
In all four cases the system has a generic modality component for the detection of a voice command input, despite the differences in implementation. Any modality component can potentially wrap multiple features provided by multiple physical devices but also more than one modality component could be included in a single device. To this extent the modality component is an abstraction of the same kind of input handled and implemented differently in each case.
For this reason, the W3C recommendation currently does not describe in detail the structure or implementation of the modality components. It focuses only on the need of a communication interface with the Interaction Manager and the need of an implementation that follows a specific communication protocol: the Life-Cycle Events.
The data component.
The data component's primary role is to save the public data of the application that may be required by one or several modality components or by other modules (e.g., the Framework's session module).
The data component can be an internal module A or an external module B to the Interaction Manager (see Figure). This depends on the implementation chosen by each application. However, the Interaction Manager is the only module that has direct access to the Data Component and only the Interaction Manager can view and edit the data and communicate with external servers if necessary. As a result, the Modality Components must use the Interaction Manager as an intermediary to access the public data of the multimodal application.
However, for the storage of private data, each Modality Component can implement its own Data Component. This private Data Component can also access external servers B (see Figure) and keep the data that the Modality Component may require, for example, in the speech recognition task. This can be the case of an implementation following the principle of nested dolls given by the MMI architecture recommendation.
Communication protocol between different modules.
In the MMI architecture, the communication protocol is asynchronous, bi-directional and based on the exchange of event notifications that are raised by the system following a user action or some internal activity.
This protocol defines the exchange mode and how to establish and end the communication between modules. In the case of this specification, it is reflected in the Life-Cycle Events. These are six standard control events that are proposed for controlling devices and material services (such as a video player or a sound reproduction device) and two notifications proposed for monitoring the current status of the multimodal system.
The Standard Life-Cycle Events.
The specification recommends eight standard Life-Cycle Events specified as pairs of Request > Response exchanges:
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:newContextRequest requestID="myReq1" source="myPointerMC.php" target="myIM.php" data="myMCStatus.xml" />
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:newContextResponse requestID="myReq1" source="myIM.php" target="myPointerMC.php" context="myContextID1" status="success" />
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:clearContextRequest requestID="myReq2" source="myPointerMC.php" target="myIM.php" context="myContextID1" />
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:clearContextResponse requestID="myReq2" source="myPointerMC.php" target="myIM.php" context="myContextID1" status="success" />
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0" xmlns:svg="http://www.w3.org/2000/svg">
<mmi:prepareRequest requestID="myReq3" source="myIM.php" target="myDisplayMC.php" context="myContextID2">
<mmi:content>
<svg:svg width="100%" height="100%" version="1.1">
<rect width="300" height="100" style="fill:rgb(0,0,255);stroke-width:1; stroke:rgb(0,0,0)"/>
</svg:svg>
</mmi:content>
</mmi:prepareRequest>
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:prepareResponse requestID="myReq3" source="myDisplayMC.php" target="myIM.php" status="success" context="myContextID2">
</mmi:prepareResponse>
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:startRequest requestID="myReq4" source="myIM.php" target="myPlayerMC.php" context="myContextID3" data="myPlayerParams.xml">
<mmi:contentURL href="myAnimation.swf"/>
</mmi:startRequest>
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:startResponse requestID="myReq4" source="myPlayerMC.php" target="myIM.php" status="success" context="myContextID3">
</mmi:startResponse>
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:startResponse requestID="myReq4" source="myPlayerMC.php" target="myIM.php" status="failure" context="myContextID3">
<mmi:statusInfo>
NoContent
</mmi:statusInfo>
</mmi:startResponse>
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:cancelRequest requestID="myReq5" source="myIM.php" target="mySpokerMC.php" context="myContextID4" Immediate="true"/>
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:cancelResponse requestID="myReq5" source="mySpokerMC.php" target="myIM.php" context="myContextID4" status="success" data="myMCStatus.xml"/>
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:pauseRequest requestID="myReq6" source="myIM.php" target="myWriterMC.php" context="myContextID5" immediate="false"/>
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:pauseResponse requestID="myReq6" source="myWriterMC.php" target="myIM.php" context="myContextID5" status="success" data="myMCStatus.xml"/>
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:resumeRequest requestID="myReq7" source="myIM.php" target="myWriterMC.php" context="myContextID5"/>
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:resumeResponse requestID="myReq7" source="myWriterMC.php" target="myIM.php" context="myContextID5" status="success"/>
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:statusRequest requestID="myReq8" source="myRecorderMC.php" target="myIM.php" requestAutomaticUpdate="true" />
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
<mmi:statusResponse requestID="myReq8" source="myIM.php" target="myRecorderMC.php" status="alive" AutomaticUpdate="true" />
</mmi:mmi>
The Notifications.
The specification also recommends two types of notifications, which one, the Extension Notification, can contain control or command data to handle devices or material services. For this reason this notification is regarded like an exception, a standard control event that is not described as a pair Request > Response (in a previous version it was called the Data Event). The two notifications are:
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
 <mmi:extensionNotification requestID="myReq9" source="myPlayerMC.php" target="myIM.php" context="myContextID6" name="playerNavigation">
 <applicationdata>
 <data id="menu" selected="true" />
 </applicationdata>
 </mmi:extensionNotification>
</mmi:mmi>
<mmi:mmi xmlns:mmi="http://www.w3.org/2008/04/mmi-arch" version="1.0">
 <mmi:doneNotification requestID="myReq10" source="myDetectorMC.php" target="myIM.php" context="myContextID7" status="success">
 <mmi:data>
 <data id="detectionList">
 <users>
 <user id="u58" confidence=".85" />
 <user id="u32" confidence=".75" />
 <user id="u87" confidence=".60" />
 </users>
 </data>
 </mmi:data>
 </mmi:doneNotification>
</mmi:mmi>