<commit_before>// Copyright 2016
// Ubiquitous Knowledge Processing (UKP) Lab and FG Language Technology
// Technische Universität Darmstadt
// 
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
// 
// http://www.apache.org/licenses/LICENSE-2.0
// 
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

[[sect_webannotsv]]
= WebAnno TSV File format

In this section, we will discuss the TSV (Tab Separated Value) WebAnno file format. The format is similar to the CoNNL file formats with specialized additions to the header and column representations. The file format inhabits a header and a body section. The *header* section present information about the different types of annotation layers and features used in the file. While importing the WebAnno TSV file, the specified headers should be first created in to the running WebAnno project. Otherwise, the importing of the file will not be possible.

The *body* section of the TSV file presents the document and all the associated annotations including sentence and token annotations.

== Header Section

WebAnno 3.0 TSV file starts with the following header marker

`#FORMAT=WebAnno TSV 3`

Layers are marked by the *#* character followed by *T_SP=*for `span types` (including `slot features`), *T_CH=*for `chain layers`, and *T_RL=*for `relation layers`. 
Every layer is written in new line, followed by the features in the layer

If all layer type exists, first, all the span layers will be written and then the chain layer and finally the relation layers.

Features are separated by the *|* character and only the short name of the feature is provided.

Example: `#T_SP=webanno.custom.Pred|bestSense|lemmaMapped|senseId|senseMapped`
 
Here the layer name is `webanno.custom.Pred` and the features are named `bestSense`, `lemmaMapped`, `senseId`, `senseMapped`.
Slot features start with a prefix `ROLE_`followed by the name of the role and the link. The role feature name and the link feature name are separated by the *_* character.

The target of the slot feature always follows the role/link name

Example: `#T_SP=webanno.custom.SemPred|ROLE_webanno.custom.SemPred:RoleSet_webanno.custom.SemPredRoleSetLink|uima.tcas.Annotation|aFrame`


Here the name of the role is  `webanno.custom.SemPred:RoleSet` and the name of the role link is `webanno.custom.SemPredRoleSetLink` and the target type is `uima.tcas.Annotation`

Chain layers will have always two features, `referenceType` and `referenceRelation`

Example: 
`#T_CH=de.tudarmstadt.ukp.dkpro.core.api.coref.type.CoreferenceLink|referenceType|referenceRelation`

Relation layers will come at last in the list and the very last entry in the features will be the type of the base (governor or dependent) annotations with a prefix `BT_`.

Example: 
`#T_RL=de.tudarmstadt.ukp.dkpro.core.api.syntax.type.dependency.Dependency|DependencyType|BT_de.tudarmstadt.ukp.dkpro.core.api.lexmorph.type.pos.POS`

Here, the relation type `de.tudarmstadt.ukp.dkpro.core.api.syntax.type.dependency.Dependency` has a feature `DependencyType` and the relation is between a base type of `de.tudarmstadt.ukp.dkpro.core.api.lexmorph.type.pos.POS`.

== Annotations Section

In this section we discuss the different representations of texts and annotation in WebAnno TSV3format

=== Sentence Representation

Sentence annotations are presented following the text marker `#Text=`, before the token annotations.
Example

`#Text=Bell , based in Los Angeles , makes and distributes electronic , computer and building products .`

The sentence annotation do not include offsets and mainly presented as additional information. Re-importing the WebAnno TSV file construct sentences from the token annotations.

=== Token and Sub-token Annotations

Token annotation starts with a `sentence-token` number marker followed by the begin-end offsets and the token itself, separated by a TAB characters. 

Example

`1-2	4-8	Haag`

Here `1` indicates the sentence number, `2` indicates the token number (here, the second token in the first sentence) and `4` is the begin offset of the token and `8` is the end offset of the toke while `Haag` is the token.

Sub-token representations are affixed with a `.` and a number starts from 1 to N. 

Exampel

`1-3	9-14	plays`

`1-3.1	9-13	play`

`1-3.2	13-14	s`

Here, the sub-token `play` is indicated by sentence-token number `1-<PASSWORD>` and the sub-token `s` is indicated by `1-3.2`. 

=== Span Annotations

For every features of a span Annotation, annotation value will be presented in the same row as the token/sub-token annotation, separated by a TAB character. If there is no annotation for the given span layer, a `_` character is placed in the column. If the feature has no/null annotation or if the span layer do not have a feature at all, a `*` character represents the annotation.

Example:

Layers:

`#T_SP=de.tudarmstadt.ukp.dkpro.core.api.lexmorph.type.pos.POS|PosValue`
`#T_SP=webanno.custom.Sentiment|Category|Opinion`

Annotations

`1-9	36-43	unhappy		JJ		abstract	negative`

Here, the first annotation at column 4, `JJ` is avalue for a feature `PosValue` of the layer `de.tudarmstadt.ukp.dkpro.core.api.lexmorph.type.pos.POS`. For the two features of the layer `webanno.custom.Sentiment` (`Category` and `Opinion`), the values `abstract` and `negative` are presented at column 5 and 6 resp.

=== Multiple Span Annotation or Stacked Annotations

Multiple span annotation or stacked annotations on a token/sub-token will have a numbered reference enclosed by brackets as [N] where N refers to the nth annotation on the layer. Stacked annotations are separated by *|* character.

Example:

Layer:

`#T_SP=de.tudarmstadt.ukp.dkpro.core.api.lexmorph.type.pos.POS|PosValue`

`#T_SP=de.tudarmstadt.ukp.dkpro.core.api.ner.type.NamedEntity|value`

Annotation:

`1-1	0-3	Ms.		NNP		PER[1]|PERpart[2]`

`1-2	4-8	Haag	NNP		PER[1]`

Here, `PER[1]` indicates that token `1-1` and `1-2` have the same annotation (multiple token annotations) while `PERpart[2]` is the second (stacked) annotation on token  `1-1` separated by *|* character.

=== Slot features

Slot features and the target annotations are separated by TAB character (first the feature column then the target column follows). In the target column, the `sentence-token` id is recorded where the feature is drawn.

Unlike other span layer features (which are separated by *|* character), multiple annotations for a slot feature are separated by the *;* character.

Example

Layers:

`#T_SP=webanno.custom.Frame|FE|ROLE_webanno.custom.Frame:Roles_webanno.custom.FrameRolesLink|webanno.custom.Lu`

`#T_SP=webanno.custom.Lu|luvalue`


Annotations:


`2-1	27-30	Bob	_	_	_	bob`

`2-2	31-40	auctioned	transaction	seller;goods;buyer	2-1;2-3[4];2-6		_`

`2-3	41-44	the	_	_	_	clock[4]`

`2-4	45-50	clock	_	_	_	clock[4]`

`2-5	52-54	to	_	_	_	_`

`2-6	55-59	John	_	_	_	john`

`2-7	59-60	.	_	_	_	_`

Here, for example, at token `2-2`, we have three slot annotations for feature `Roles` that are `seller`, `goods`, and `buyer`. The targets are on token `2-1 `,`2-3[4]`, and `2-6` resp. which are on annotations of the layer `webanno.custom.Lu` which are `bob`, `clock` and `john`. 

=== special Character Annotations

Special characters are escaped with the *\* character. Special characters includes the following characters.

`\,[,],|,_,->,;,\t,\n,*`<commit_msg>Documentation of the "WebAnno TSV 3" format #193 	Adding chain annotation descriptions<commit_after>// Copyright 2016
// Ubiquitous Knowledge Processing (UKP) Lab and FG Language Technology
// Technische Universität Darmstadt
// 
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
// 
// http://www.apache.org/licenses/LICENSE-2.0
// 
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

[[sect_webannotsv]]
= WebAnno TSV File format

In this section, we will discuss the TSV (Tab Separated Value) WebAnno file format. The format is similar to the CoNNL file formats with specialized additions to the header and column representations. The file format inhabits a header and a body section. The *header* section present information about the different types of annotation layers and features used in the file. While importing the WebAnno TSV file, the specified headers should be first created in to the running WebAnno project. Otherwise, the importing of the file will not be possible.

The *body* section of the TSV file presents the document and all the associated annotations including sentence and token annotations.

== Header Section

WebAnno 3.0 TSV file starts with the following header marker

`#FORMAT=WebAnno TSV 3`

Layers are marked by the *#* character followed by *T_SP=*for `span types` (including `slot features`), *T_CH=*for `chain layers`, and *T_RL=*for `relation layers`. 
Every layer is written in new line, followed by the features in the layer

If all layer type exists, first, all the span layers will be written and then the chain layer and finally the relation layers.

Features are separated by the *|* character and only the short name of the feature is provided.

Example: `#T_SP=webanno.custom.Pred|bestSense|lemmaMapped|senseId|senseMapped`
 
Here the layer name is `webanno.custom.Pred` and the features are named `bestSense`, `lemmaMapped`, `senseId`, `senseMapped`.
Slot features start with a prefix `ROLE_`followed by the name of the role and the link. The role feature name and the link feature name are separated by the *_* character.

The target of the slot feature always follows the role/link name

Example: `#T_SP=webanno.custom.SemPred|ROLE_webanno.custom.SemPred:RoleSet_webanno.custom.SemPredRoleSetLink|uima.tcas.Annotation|aFrame`


Here the name of the role is  `webanno.custom.SemPred:RoleSet` and the name of the role link is `webanno.custom.SemPredRoleSetLink` and the target type is `uima.tcas.Annotation`

Chain layers will have always two features, `referenceType` and `referenceRelation`

Example: 
`#T_CH=de.tudarmstadt.ukp.dkpro.core.api.coref.type.CoreferenceLink|referenceType|referenceRelation`

Relation layers will come at last in the list and the very last entry in the features will be the type of the base (governor or dependent) annotations with a prefix `BT_`.

Example: 
`#T_RL=de.tudarmstadt.ukp.dkpro.core.api.syntax.type.dependency.Dependency|DependencyType|BT_de.tudarmstadt.ukp.dkpro.core.api.lexmorph.type.pos.POS`

Here, the relation type `de.tudarmstadt.ukp.dkpro.core.api.syntax.type.dependency.Dependency` has a feature `DependencyType` and the relation is between a base type of `de.tudarmstadt.ukp.dkpro.core.api.lexmorph.type.pos.POS`.

== Annotations Section

In this section we discuss the different representations of texts and annotation in WebAnno TSV3format

=== Sentence Representation

Sentence annotations are presented following the text marker `#Text=`, before the token annotations.
Example

`#Text=Bell , based in Los Angeles , makes and distributes electronic , computer and building products .`

The sentence annotation do not include offsets and mainly presented as additional information. Re-importing the WebAnno TSV file construct sentences from the token annotations.

=== Token and Sub-token Annotations

Token annotation starts with a `sentence-token` number marker followed by the begin-end offsets and the token itself, separated by a TAB characters. 

Example

`1-2	4-8	Haag`

Here `1` indicates the sentence number, `2` indicates the token number (here, the second token in the first sentence) and `4` is the begin offset of the token and `8` is the end offset of the toke while `Haag` is the token.

Sub-token representations are affixed with a `.` and a number starts from 1 to N. 

Exampel

`1-3	9-14	plays`

`1-3.1	9-13	play`

`1-3.2	13-14	s`

Here, the sub-token `play` is indicated by sentence-token number `1-3.1` and the sub-token `s` is indicated by `1-3.2`. 

=== Span Annotations

For every features of a span Annotation, annotation value will be presented in the same row as the token/sub-token annotation, separated by a TAB character. If there is no annotation for the given span layer, a `_` character is placed in the column. If the feature has no/null annotation or if the span layer do not have a feature at all, a `*` character represents the annotation.

Example:

Layers:

`#T_SP=de.tudarmstadt.ukp.dkpro.core.api.lexmorph.type.pos.POS|PosValue`
`#T_SP=webanno.custom.Sentiment|Category|Opinion`

Annotations

`1-9	36-43	unhappy		JJ		abstract	negative`

Here, the first annotation at column 4, `JJ` is avalue for a feature `PosValue` of the layer `de.tudarmstadt.ukp.dkpro.core.api.lexmorph.type.pos.POS`. For the two features of the layer `webanno.custom.Sentiment` (`Category` and `Opinion`), the values `abstract` and `negative` are presented at column 5 and 6 resp.

=== Multiple Span Annotation or Stacked Annotations

Multiple span annotation or stacked annotations on a token/sub-token will have a numbered reference enclosed by brackets as [N] where N refers to the nth annotation on the layer. Stacked annotations are separated by *|* character.

Example:

Layer:

`#T_SP=de.tudarmstadt.ukp.dkpro.core.api.lexmorph.type.pos.POS|PosValue`

`#T_SP=de.tudarmstadt.ukp.dkpro.core.api.ner.type.NamedEntity|value`

Annotation:

`1-1	0-3	Ms.		NNP		PER[1]|PERpart[2]`

`1-2	4-8	Haag	NNP		PER[1]`

Here, `PER[1]` indicates that token `1-1` and `1-2` have the same annotation (multiple token annotations) while `PERpart[2]` is the second (stacked) annotation on token  `1-1` separated by *|* character.

=== Slot features

Slot features and the target annotations are separated by TAB character (first the feature column then the target column follows). In the target column, the `sentence-token` id is recorded where the feature is drawn.

Unlike other span layer features (which are separated by *|* character), multiple annotations for a slot feature are separated by the *;* character.

Example

Layers:

`#T_SP=webanno.custom.Frame|FE|ROLE_webanno.custom.Frame:Roles_webanno.custom.FrameRolesLink|webanno.custom.Lu`

`#T_SP=webanno.custom.Lu|luvalue`


Annotations:


`2-1	27-30	Bob	_	_	_	bob`

`2-2	31-40	auctioned	transaction	seller;goods;buyer	2-1;2-3[4];2-6		_`

`2-3	41-44	the	_	_	_	clock[4]`

`2-4	45-50	clock	_	_	_	clock[4]`

`2-5	52-54	to	_	_	_	_`

`2-6	55-59	John	_	_	_	john`

`2-7	59-60	.	_	_	_	_`

Here, for example, at token `2-2`, we have three slot annotations for feature `Roles` that are `seller`, `goods`, and `buyer`. The targets are on token `2-1 `,`2-3[4]`, and `2-6` resp. which are on annotations of the layer `webanno.custom.Lu` which are `bob`, `clock` and `john`. 

=== special Character Annotations

Special characters are escaped with the *\* character. Special characters includes the following characters.

`\,[,],|,_,->,;,\t,\n,*`

=== Chain Annotations

In the Chain annotation, two columns (TAB separated) are used to represent the `referenceType` and the `referenceRelation`. A chain number is attached to the `referenceType` to distinguish to which of the coreference chains the annotation belongs. The `referenceRelation` of the chain is represented by the relation value followed by `->` and followed by the `CH-LINK` number where `CH` is the chain number and `LINK` is the link number (the order the chain).

Example

Layer

`#T_CH=de.tudarmstadt.ukp.dkpro.core.api.coref.type.CoreferenceLink|referenceType|referenceRelation`

Annotations

`1-1	0-2	He	pr[1]	coref->1-1`

`1-2	3-7	shot	_	_`

`1-3	8-15	himself		pr[1]	coref->1-2`

`1-4	16-20	with	_	_`

`1-5	21-24	his		pr[1]	*->1-3`	

`1-6	25-33	revolver	_	_`

`1-7	33-34	.	_	_`

In this example, token `1-3` is marked as `pr[1]` which explains that the `referenceType` is `pr` and it is part of coreference chain `1`. The coreference relation value is `coref` and with the `CH-LINK` number `1-2` which means that it belongs to chain `1` and this is the second link in the coreference chain.
{'fraction_non_alphanumeric': 0.08786384411835459, 'fraction_numerical': 0.01930478710608612, 'mean_word_length': 4.4844986807387865, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1560738', 'n_tokens_mistral': 5367, 'n_tokens_neox': 5121, 'n_words': 2131}