Representing Multimodal Linguistic Annotated data

The question of interoperability for linguistic annotated resources covers different aspects. First, it requires a representation framework making it possible to compare, and eventually merge, different annotation schema. In this paper, a general description level representing the multimodal linguistic annotations is proposed. It focuses on time representation and on the data content representation: This paper reconsiders and enhances the current and generalized representation of annotations. An XML schema of such annotations is proposed. A Python API is also proposed. This framework is implemented in a multi-platform software and distributed under the terms of the GNU Public License.


Introduction
Multimodal text analysis has become a crucial part of research, teaching and practice for a wide range of academic and practical disciplines. Multimodal annotation is facing the necessity of encoding many different information types, from different domains, with different levels of granularity (Jewitt, 2009). In the Linguistics field, multimodal annotations contain information ranging from general linguistic to domain specific information. Some are annotated with automatic tools, and some are manually annotated. This implies technical and methodological levels to produce high quality multimodal annotations, as proposed in (Blache et al., 2010). Linguistics annotation, especially when dealing with multiple domains, makes use of different tools within a given project. The annotation itself should be done with the most ergonomic and convenient tool for the given annotation task. Then it is up to the person to develop the exploitation tool to create "views" in the very rich and layered object that corresponds to the whole annotation set. This is to make sure that analysts, annotators and end-users never face the intrinsic complexity of a rich multi-level annotation framework. Acoustic analysis, phonetics and prosody annotation are very often done using Praat (Boersma and Weenink, 2011). Gesture annotations, and more generally multimodal studies, now rely on higher level systems such as Anvil (Kipp., 2011) or Elan (Nijmegen: Max Planck Institute for Psycholinguistics, 2011) (see (Rohlfing et al., 2006) for a comparison of multimodal annotation tools). As for orthographic transcriptions, a great number of tools can be used. None of the annotations tools are directly interoperable, each using a native format, some of them on top of XML, some others developing an ad hoc markup language. The heterogeneity of such annotations has been recognized as a key problem limiting the interoperability and re-usability of NLP tools and linguistic data collections. Manipulating linguistic annotated resources requires to be independent from the coding format. This implies two kinds of pre-requisites: • to specify and organize the information to be encoded independently from the constraints or restriction of the format (or the annotation tool); • to encode the information into an exchange format, readable whatever the edition or annotation system.
But schemas and tools multiplication renders the data manipulation very problematic. On the other hand, despite standardization efforts it is unlikely that a unique generic schema or tool emerge. This paper addresses the problem of representing these multimodal annotated speech corpora. Metadata (speaker information, recording condition, etc) are not addressed. This paper focuses on the annotation representation itself: time representation, content representation and data organization.

Related works
Many tools and frameworks are available for handling rich media data. The practice of taking advantage of such a rich tool offer will not change. What can change however is the habit of developing a new language for each new project. Interchange format such as AIF (Bird et al., 2000) constitute an important step for answering this need of common frameworks. Although necessary for bringing some tool interoperability, such interchange formats remained relatively poor with regard to their descriptive power. Indeed the development of these formats was done according to a "tool perspective" as illustrated by the importance of the notions of tiers or tracks. Nowadays, most of linguistic annotation projects require a rich framework in order to deal with all the information levels involved. For example, dialogue act annotation involve multi-dimensional communicative functions (Bunt, 2009) that are organized in a complex taxonomy. Another example concerns discourse units that can be discontinuous which prevent them from treating them a simple sequence of units on a tier. Instead of being answered by richer model, extensibility and flexibility are gained by tricks and ad-hoc solutions to overcome the basic nature of the original model. The other way toward interoperability is the standardization of annotation frameworks. The ISO TC37/4 1 initia-tive is crucial for making converge annotation frameworks. However, ISO standards usually provide precise guidelines for developing interoperable systems but do not resolve the technical issues of this interoperability. Web semantics languages such as RDF (Resource Description Framework) and OWL (Ontology Web Language) also constitute an important path to explore for practical interoperability. Unique Resource Identifier allows to uniquely identified and facilitate sharing schema and resources. Finally, OWL has been developed for providing a standard semantics to the ingredients present in all these resources. These features should allow a real re-usability and interoperability. Moreover the mapping between those welldefined schema should be easier to perform thanks to existing tools and methodologies in the semantic community.
In spite of their benefits, those semantic web languages do not impose themselves easily as standard languages in language annotation projects. This is mainly due to their verbosity and their complexity as illustrated in the short example below. This has two important consequences: (i) they need sophisticated tools to be easily handled while annotation model development early stages is usually not performed with these tools; (ii) it drastically increase data size (although this should not be an issue for other corpora than massive web corpora and some more compact syntax 2 has been proposed).

Linguistics: Annotated data
The annotation of audio-video corpora is concerned by many Linguistics sub-fields as Phonetics, Prosody, Gestures or Discourse. Consequently, organizing such annotations in a hierarchy or simply performing an inventory of all possible annotations is impractical. It ranges from utterances, orthographic transcription, prosodic cues, phonetic segmentation, to gestures and nowadays motion-capture or more physiological data. In annotation tools, annotated data are mainly represented in the form of "tiers" or "tracks" of annotations. Of course, this is not just an artifact of such tools, and clearly, many annotators are using such representation during the annotation procedure. In most tools, tiers are series of intervals defined by: • a time point to represent the beginning of the interval; • a time point to represent the end of the interval; • a label to represent the annotation itself.
In Praat, tiers can be represented by a time point and a label (such tiers are named PointTiers and IntervalTiers). Of course, depending on the annotation tool, the internal data representation and the file formats are different. For example, in Elan, unlabeled intervals are not represented nor saved. On the contrary, in Praat, tiers are made of a succession of consecutive intervals (labeled or un-labeled). Another difference among tools concerns relations between tiers. In Praat for example, no relation can be defined. On the contrary, in Anvil, hierarchies can be established between tiers, by mean of an XML schema describing all annotations and their structures. This XML schema is performed by annotators themselves, before the annotation process. It is then assumed that the user is familiar with XML. Elan proposes an intermediate solution with possibility to connect tiers by a relation named "subdivision". Our proposal of data representation is based on the common set of information all tool are currently sharing: Tiers. Each tier includes a set of annotations. As usual, an annotation is represented by an interval (or a point) and a label. Our proposal focus on the way to represent such information: time and label. Therefore, this paper reconsiders and enhances the current and generalized representation of annotations.

Representation of annotations 4.1. Overview
Time in the various file formats is described in different ways. In the TEI format, all temporal points are grouped in a timeline object and an annotated element can refer to a point using his id. Some other data formats do not use a specific timeline object and directly put the time value inside an element using 'start' and 'end' attributes. Tools are using real numbers to represent time. They are described thank to a timeline: each number corresponds to one point on the line, and each point determines one real number. From the "computer science" point of view, real numbers are approximated and depend on the number of coding digit. Moreover, the time value is expressed with various time units: nanoseconds is the unit used by some tools, milliseconds by some others, and seconds are mainly used (seconds are represented from 4 to 10 digits). These approximation can cause troubles in some cases, as for example by comparing numbers: In this case a == b should be true. Consequently, a the framework representing data, as time in our example, and using real numbers must take into account such phenomena. Furthermore, annotated data are often created or modified manually. Then, the precision of the time value depends on various factors, as: • the human decision (aims of the work, time dedicated to annotate, etc); • with an audio or a video media, the annotation itself suffers of various precision: it can be very precise (about 1ms) while annotating Tones in the prosody domain; and for the gestures annotation, the precision of a time value can't be less than the duration of one picture in a video (often 40ms).
• the graphical user interface of the tool: in the best case, the precision of the "time value" corresponds to 1 pixel. This means that the time value is not a real but a duration, which depends on the zoom level.

Time points
Our proposal is to define a TimePoint X as an imprecise value. It is possible to characterize a point in a space immediately allowing its vagueness by choosing the representation space Midpoint / Radius. A TimePoint is modeled as follows: where M X is the midpoint (center) of the value and R X is the radius (as proposed in (D'Urso and De Giovanni, 2011)) representing the uncertainty of X (see Figure 1). It is to be noted that M X − R X ≥ 0.
and the Radius or half-width is the vagueness of the interval a, which becomes directly accessible: The transition from one representation to the other is achieved thanks to the previous equations, or by using the following: x − = M X − R X and x + = M X + R X . The equality and inequalities between 2 TimePoint instances X and Y are then defined as:

Time intervals
A TimeInterval is defined as a temporal interval-valued datum: where X − and X + denote, respectively, the lower and upper TimePoint bounds of the interval as we proposed in equation 1.

Labels
Each annotation holds at least one label, mainly represented in the form of a string, freely written by the annotator or selected from a list of categories. A general framework, aiming at representing any kind of linguistic annotations, have to allow to assign a score to each label, and to allow a multiple label selection.

XML representation
XML has become a de facto standard for the representation of annotated corpus resources . It is also a widely established practice to produce a particular type of annotation with a special-purpose tool that outputs XML or maps its output onto XML. Moreover, any markup or encoding schema will have to be robust enough to handle existing data. As the framework proposed in this paper calls into consideration the common representation of time and labels, none of the existing annotation schema allows to encode such information; and unfortunately it was not relevant to adapt one. It was then appropriate to develop a new schema, that we name XRA. The physical level of representation of XRA obviously makes use of XML, XML-related standards and stand-off annotation. The format specification as well as the texts encoded in XRA are "reusable, i.e., potentially usable in more than one research project and by more than one research team, and extensible, i.e., capable of further enhancement" (Ide and Brew, 2000).

Main structure
An XRA document is linked to a set of recordings of a corpus. Each document holds a set of tiers, which are directly linked to the primary media. Tiers are used to distribute the annotations of a recording. Each tier consists of a set of independent annotations, that all pertain to one or more Time instances. The definition of a tier proposed in (Brugman, 2003) is then largely recovered: "a tier is a group of annotations that all describe the same type of phenomenon, that all share the same metadata attribute values and that are all subject to the same constraints on annotation structures, on annotation content and on time alignment characteristics.". Then, an annotation is made of: a time reference, and (optionally) a label.
As the whole description of the XRA format is very long, only the annotation-related part is detailed.

Inter-tier relations
As we already said, it is not possible to decide over an exhaustive list of annotation levels for all annotation projects. Each project and each annotator as its own habits and needs. It is not possible to fix a set of "normative" tags such as < u > is an utterance and < w > is a word, and < t > is a tone, and < p > is a phoneme and < gt > is a gesture type, and so on... and then to create a static normative hierarchy between them, at least with the purpose to create a generic framework. The genericity and flexibility of "tiers" is appropriate to represent any multimodal annotated data because it simply maps the annotations on the timeline. Then, dependencies between such tiers must be represented and we propose to extend the sub-division concept, proposed in Elan. These inter-tier relations are managed by establishing alignment or constituency links between 2 tiers: • alignment: annotations of a tier A have only Time instances included in those of annotations of tier B; • constituency: annotations of a tier A have only Time instances included in those of annotations of tier B and labels of B are made of those of A.
Example of alignment: Phonemes is the reference tier and Tokens are alignment tier: Phonemes: | l | S | a | e | l | a | Tokens: | le | chat |est| la | Example of constituency: Phonemes is the reference tier and Syllables are made of Phonemes: Phonemes: | l | S | a | e | l | a | Syllabes: There are several advantages to represent data with this solution. Firstly, the relations can easily be obtained by an automatic analysis of time values, even while importing flatdata. Secondly, an existing tree-structure can be mapped on it and vice-versa.

Description
Given the current use of large corpora in NLP work, automatic conversion into the corpus representation is desired, as well as automatic annotation and exploitation. Several tools were created in the scope of this work, to support resource conversion to .xra and to enable the integration of annotation/analysis tools. An Application Programming Interface (API) is proposed in the programming language Python. We implement behaviors in abstract classes ( Figure 2) and each class is documented using docstrings. The current version allows to import data from Praat, Elan, Transcriber or from CSV files. Moreover, the XRA-schema is freely available in the package. This API is already used in the automatic annotation tool SPPAS (Bigi and Hirst, 2012; and is distributed under the terms of the GNU Public License. Among other functions, it proposes an automatic speech segmentation at the phone and token levels for French, English, Spanish, Italian, Chinese, Taiwanese and Japanese.

Main usage
Open/Read a file of any format (TextGrid, Elan, Transcriber, CSV) and load in a Transcription() instance: Create an annotation made of an interval and a label, and add it into a tier: p1 = TimePoint(1.5, radius=0.01) p2 = TimePoint(1.6, radius=0.01) t = TimeInterval(p1,p2) l = Label("foo") ann = Annotation(t,l) tier.Add(ann) Removing an annotation of a tier: Get an annotation of a tier from its index: Getting all annotations between 2 time values: Then, the next occurrence of such pattern can be retrieved by setting the pos argument to annfirstindex. Get the annotation contents: Add a sub-division link between two tiers: trs.AddSubdivision(reftier,\ alignedtier,type="alignment") trs.AddSubdivision(reftier, \ alignedtier,type="constituency") Remove a sub-division link between two tiers: trs.RemoveSubdivision(reftier,alignedtier)

Conclusion
The development of an expressive and generic framework for linguistic annotations is very beneficial. This paper particularly focused on the problem of representing annotations, and an annotation representation was proposed. The paper reconsidered and enhanced the current and generalized representation of speech annotations. We proposed a framework that can capture the indeterminacy of annotations.
We are currently working on extending this work to the construction of a software for the creation, manipulation, and analysis of annotated corpora (Figure 3). : Example of multimodal annotated data, imported from 3 different annotation tools. Each TimePoint is represented by a vertical black line with a gradient color to refer to the radius value (0ms for prosody, 5ms for phonetics, discourse and syntax and 40ms for gestures in this screenshot). Uncertain labels are underlined and the related information is highlighted when it is clicked on.