<commit_before><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html lang="en-GB" xml:lang="en-GB" xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
    <title>CoNLL-U Format</title>
    <link rel="root" href=""/> <!-- for JS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
    <link rel="stylesheet" type="text/css" href="css/jquery-ui-redmond.css"/>
    <link rel="stylesheet" type="text/css" href="css/style.css"/>
    <link rel="stylesheet" type="text/css" href="css/style-vis.css"/>
    <link rel="stylesheet" type="text/css" href="css/hint.css"/>
    <script type="text/javascript" src="lib/ext/head.load.min.js"></script>
    <script type="text/javascript" src="lib/ext/jquery.timeago.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/3.2.2/anchor.min.js"></script>
    <script>document.addEventListener("DOMContentLoaded", function(event) {anchors.add();});</script>
    <!-- Set up this custom Google search at https://cse.google.com/cse/business/settings?cx=001145188882102106025:dl1mehhcgbo -->
    <script>
      (function() {
        var cx = '001145188882102106025:dl1mehhcgbo';
        var gcse = document.createElement('script');
        gcse.type = 'text/javascript';
        gcse.async = true;
        gcse.src = 'https://cse.google.com/cse.js?cx=' + cx;
        var s = document.getElementsByTagName('script')[0];
        s.parentNode.insertBefore(gcse, s);
      })();
    </script>
<!--     <link rel="shortcut icon" href="favicon.ico"/> -->


  </head>
  <body>
    <div id="main" class="center">

      <div id="hp-header">
        <table width="100%"><tr><td width="50%">
          <span class="header-text"><a href="http://universaldependencies.org/#language-">home</a></span>

          <span class="header-text"><a href="https://github.com/universaldependencies/docs/edit/pages-source/format.md" target="#">edit page</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/issues">issue tracker</a></span>
        </td><td>
          <gcse:search></gcse:search>
        </td></tr></table>
      </div>

      <hr/>

      
      <div class="v2complete">
	This page pertains to UD version 2.
      </div>
      

      <div id="content">
	<noscript>
	  <div id="noscript">
	    It appears that you have Javascript disabled.
	    Please consider enabling Javascript for this page to see the visualizations.
	  </div>
	</noscript>

          <h1 id="conll-u-format">CoNLL-U Format</h1>

<p>Quick links:
[<a href="#words-tokens-and-empty-nodes">Word segmentation</a>]
[<a href="#morphological-annotation">Morphology</a>]
[<a href="#syntactic-annotation">Syntax</a>]
[<a href="#miscellaneous">Miscellaneous</a>]
[<a href="#extensions">Extensions</a>]</p>

<p>We use a revised version of <a href="http://anthology.aclweb.org/W/W06/W06-2920.pdf">the CoNLL-X format</a> called CoNLL-U.
Annotations are encoded in plain text files (UTF-8, using only the LF character as line break, including an LF character at the end of file) with three types of lines:</p>

<ol>
  <li>Word lines containing the annotation of a word/token in 10 fields separated by single tab characters; see below.</li>
  <li>Blank lines marking sentence boundaries.</li>
  <li>Comment lines starting with hash (#).</li>
</ol>

<p>Sentences consist of one or more word lines, and word lines contain the following fields:</p>

<ol>
  <li>ID: Word index, integer starting at 1 for each new sentence; may be a range for multiword tokens; may be a decimal number for empty nodes.</li>
  <li>FORM: Word form or punctuation symbol.</li>
  <li>LEMMA: Lemma or stem of word form.</li>
  <li>UPOS: <a href="u/pos/index.html">Universal part-of-speech tag</a>.</li>
  <li>XPOS: Language-specific part-of-speech tag; underscore if not available.</li>
  <li>FEATS: List of morphological features from the <a href="u/feat/index.html">universal feature inventory</a> or from a defined <a href="ext-feat-index.html">language-specific extension</a>; underscore if not available.</li>
  <li>HEAD: Head of the current word, which is either a value of ID or zero (0).</li>
  <li>DEPREL: <a href="u/dep/index.html">Universal dependency relation</a> to the HEAD (<a href="u-dep/root">root</a> iff HEAD = 0) or a defined language-specific subtype of one.</li>
  <li>DEPS: Enhanced dependency graph in the form of a list of head-deprel pairs.</li>
  <li>MISC: Any other annotation.</li>
</ol>

<p>The fields DEPS and MISC replace the obsolete fields PHEAD and PDEPREL of the CoNLL-X format.
In addition, we have modified the usage of the ID, FORM, LEMMA, XPOS, FEATS and HEAD fields as explained below.</p>

<p>The fields must additionally meet the following constraints:</p>

<ul>
  <li>Fields must not be empty.</li>
  <li>Fields other than FORM and LEMMA must not contain space characters.</li>
  <li>Underscore (_) is used to denote unspecified values in all fields except ID. Note that no format-level distinction is made for the rare cases where the FORM or LEMMA is the literal underscore – processing in such cases is application-dependent. Further, in UD treebanks the UPOS, HEAD, and DEPREL columns are not allowed to be left unspecified.</li>
</ul>

<p>See the <a href="tools.html">tools page</a> for a list of tools that work with the CoNLL-U format. There is also a simple JavaScript-based <a href="conllu_viewer.html">CoNLL-U file viewer</a>.</p>

<h1 id="words-tokens-and-empty-nodes">Words, Tokens and Empty Nodes</h1>

<p>To accommodate the representation of both words and (multiword) tokens (see <a href="u/overview/tokenization.html">Tokenization and word segmentation</a>),
we adopt an extension of the original CoNLL-X token indexing scheme, where words are indexed with integers 1, 2, 3, …,
while (multiword) tokens are indexed with integer ranges like 1-2 or 3-5.
Lines representing such tokens are inserted before the first word in the range.
These ranges must be nonempty and must not overlap.
They have a FORM value – the string that occurs in the sentence – but have an underscore in all the remaining fields except MISC
(because the token represents multiple words, each with its own lemma, part-of-speech tag, syntactic head, and so on).
This is illustrated in the following annotation snippet,
showing only the first three fields for the Spanish sentence <em>vámonos al mar</em> (let’s go to the sea):</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1-2    vámonos   _
1      vamos     ir
2      nos       nosotros
3-4    al        _
3      a         a
4      el        el
5      mar       mar
</code></pre></div></div>

<p>We extract the word sequence by skipping all range IDs:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1      vamos     ir
2      nos       nosotros
3      a         a
4      el        el
5      mar       mar
</code></pre></div></div>

<p>We extract the raw token sequence by skipping all integer IDs that are included in a preceding range ID:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1-2    vámonos   _
3-4    al        _
5      mar       mar
</code></pre></div></div>

<p>To accommodate the use of empty nodes for the analysis of ellipsis in the enhanced dependency representation, we adopt
a further extension of the indexing scheme from v2. It is possible to insert one or more empty nodes indexed <em>i</em>.1, <em>i</em>.2, etc. immediately after a word with index <em>i</em> (where <em>i</em> = 0 for sentence-initial empty nodes). Note that the the numbers after the decimal point must form a sequence starting at 1, i.e. it is not allowed to skip <em>i</em>.1 and use <em>i</em>.2. Here is an example showing the use of an empty node in the analysis of the sentence <em>Sue likes coffee and Bill tea</em>:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1      Sue       Sue
2      likes     like
3      coffee    coffee
4      and       and
5      Bill      Bill
5.1    likes     like
6      tea       tea
</code></pre></div></div>

<p>Empty nodes must have non-empty ID and DEPS fields and empty fields (i.e. underscores) for HEAD and DEPREL, because they are only part of the enhanced dependency graph. All other fields may contain either underscores or values: for example, they can optionally have a word form and lemma as in the example above.</p>

<h1 id="morphological-annotation">Morphological Annotation</h1>

<p>The UPOS field contains a part-of-speech tag from the <a href="u/pos/index.html">universal POS tag</a> set, while the XPOS optionally contains a language-specific part-of-speech tag, normally from a traditional, more fine-grained tagset. If the XPOS field is used, the treebank-specific documentation should define a mapping from XPOS to UPOS values (which may be context-sensitive
and refer to other fields as well). If no language-specific tags are available, the XPOS field should contain an underscore for all words.</p>

<p>The FEATS field contains a list
of morphological features, with vertical bar (|) as list separator and with underscore to represent the empty list.
All features should be represented as attribute-value pairs, with an equals sign (=) separating the attribute from the value. In addition, features should as far as possible be selected from the <a href="u/feat/index.html">universal feature inventory</a> and be sorted alphabetically by attribute names. It is possible to declare that a feature has two or more values for a given word: <code class="highlighter-rouge">Case=Acc,Dat</code>. In this case, the values are sorted alphabetically. In sorting, uppercase letters are considered identical to their lowercase counterparts. Feature names must have the form <code class="highlighter-rouge">[A-Z0-9][A-Z0-9a-z]*(\[[a-z0-9]+\])?</code> and feature values must have the form <code class="highlighter-rouge">[A-Z0-9][a-zA-Z0-9]*</code>.</p>

<p>Here is an example, showing only the first five fields for the<commit_msg>autogenerated by https://github.com/spyysalo/jekyll-hook<commit_after><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html lang="en-GB" xml:lang="en-GB" xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
    <title>CoNLL-U Format</title>
    <link rel="root" href=""/> <!-- for JS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
    <link rel="stylesheet" type="text/css" href="css/jquery-ui-redmond.css"/>
    <link rel="stylesheet" type="text/css" href="css/style.css"/>
    <link rel="stylesheet" type="text/css" href="css/style-vis.css"/>
    <link rel="stylesheet" type="text/css" href="css/hint.css"/>
    <script type="text/javascript" src="lib/ext/head.load.min.js"></script>
    <script type="text/javascript" src="lib/ext/jquery.timeago.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/3.2.2/anchor.min.js"></script>
    <script>document.addEventListener("DOMContentLoaded", function(event) {anchors.add();});</script>
    <!-- Set up this custom Google search at https://cse.google.com/cse/business/settings?cx=001145188882102106025:dl1mehhcgbo -->
    <script>
      (function() {
        var cx = '001145188882102106025:dl1mehhcgbo';
        var gcse = document.createElement('script');
        gcse.type = 'text/javascript';
        gcse.async = true;
        gcse.src = 'https://cse.google.com/cse.js?cx=' + cx;
        var s = document.getElementsByTagName('script')[0];
        s.parentNode.insertBefore(gcse, s);
      })();
    </script>
<!--     <link rel="shortcut icon" href="favicon.ico"/> -->


  </head>
  <body>
    <div id="main" class="center">

      <div id="hp-header">
        <table width="100%"><tr><td width="50%">
          <span class="header-text"><a href="http://universaldependencies.org/#language-">home</a></span>

          <span class="header-text"><a href="https://github.com/universaldependencies/docs/edit/pages-source/format.md" target="#">edit page</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/issues">issue tracker</a></span>
        </td><td>
          <gcse:search></gcse:search>
        </td></tr></table>
      </div>

      <hr/>

      
      <div class="v2complete">
	This page pertains to UD version 2.
      </div>
      

      <div id="content">
	<noscript>
	  <div id="noscript">
	    It appears that you have Javascript disabled.
	    Please consider enabling Javascript for this page to see the visualizations.
	  </div>
	</noscript>

          <h1 id="conll-u-format">CoNLL-U Format</h1>

<p>Quick links:
[<a href="#words-tokens-and-empty-nodes">Word segmentation</a>]
[<a href="#morphological-annotation">Morphology</a>]
[<a href="#syntactic-annotation">Syntax</a>]
[<a href="#miscellaneous">Miscellaneous</a>]
[<a href="#extensions">Extensions</a>]</p>

<p>We use a revised version of <a href="http://anthology.aclweb.org/W/W06/W06-2920.pdf">the CoNLL-X format</a> called CoNLL-U.
Annotations are encoded in plain text files (UTF-8, using only the LF character as line break, including an LF character at the end of file) with three types of lines:</p>

<ol>
  <li>Word lines containing the annotation of a word/token in 10 fields separated by single tab characters; see below.</li>
  <li>Blank lines marking sentence boundaries.</li>
  <li>Comment lines starting with hash (#).</li>
</ol>

<p>Sentences consist of one or more word lines, and word lines contain the following fields:</p>

<ol>
  <li>ID: Word index, integer starting at 1 for each new sentence; may be a range for multiword tokens; may be a decimal number for empty nodes (decimal numbers can be lower than 1 but must be greater than 0).</li>
  <li>FORM: Word form or punctuation symbol.</li>
  <li>LEMMA: Lemma or stem of word form.</li>
  <li>UPOS: <a href="u/pos/index.html">Universal part-of-speech tag</a>.</li>
  <li>XPOS: Language-specific part-of-speech tag; underscore if not available.</li>
  <li>FEATS: List of morphological features from the <a href="u/feat/index.html">universal feature inventory</a> or from a defined <a href="ext-feat-index.html">language-specific extension</a>; underscore if not available.</li>
  <li>HEAD: Head of the current word, which is either a value of ID or zero (0).</li>
  <li>DEPREL: <a href="u/dep/index.html">Universal dependency relation</a> to the HEAD (<a href="u-dep/root">root</a> iff HEAD = 0) or a defined language-specific subtype of one.</li>
  <li>DEPS: Enhanced dependency graph in the form of a list of head-deprel pairs.</li>
  <li>MISC: Any other annotation.</li>
</ol>

<p>The fields DEPS and MISC replace the obsolete fields PHEAD and PDEPREL of the CoNLL-X format.
In addition, we have modified the usage of the ID, FORM, LEMMA, XPOS, FEATS and HEAD fields as explained below.</p>

<p>The fields must additionally meet the following constraints:</p>

<ul>
  <li>Fields must not be empty.</li>
  <li>Fields other than FORM and LEMMA must not contain space characters.</li>
  <li>Underscore (_) is used to denote unspecified values in all fields except ID. Note that no format-level distinction is made for the rare cases where the FORM or LEMMA is the literal underscore – processing in such cases is application-dependent. Further, in UD treebanks the UPOS, HEAD, and DEPREL columns are not allowed to be left unspecified.</li>
</ul>

<p>See the <a href="tools.html">tools page</a> for a list of tools that work with the CoNLL-U format. There is also a simple JavaScript-based <a href="conllu_viewer.html">CoNLL-U file viewer</a>.</p>

<h1 id="words-tokens-and-empty-nodes">Words, Tokens and Empty Nodes</h1>

<p>To accommodate the representation of both words and (multiword) tokens (see <a href="u/overview/tokenization.html">Tokenization and word segmentation</a>),
we adopt an extension of the original CoNLL-X token indexing scheme, where words are indexed with integers 1, 2, 3, …,
while (multiword) tokens are indexed with integer ranges like 1-2 or 3-5.
Lines representing such tokens are inserted before the first word in the range.
These ranges must be nonempty and must not overlap.
They have a FORM value – the string that occurs in the sentence – but have an underscore in all the remaining fields except MISC
(because the token represents multiple words, each with its own lemma, part-of-speech tag, syntactic head, and so on).
This is illustrated in the following annotation snippet,
showing only the first three fields for the Spanish sentence <em>vámonos al mar</em> (let’s go to the sea):</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1-2    vámonos   _
1      vamos     ir
2      nos       nosotros
3-4    al        _
3      a         a
4      el        el
5      mar       mar
</code></pre></div></div>

<p>We extract the word sequence by skipping all range IDs:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1      vamos     ir
2      nos       nosotros
3      a         a
4      el        el
5      mar       mar
</code></pre></div></div>

<p>We extract the raw token sequence by skipping all integer IDs that are included in a preceding range ID:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1-2    vámonos   _
3-4    al        _
5      mar       mar
</code></pre></div></div>

<p>To accommodate the use of empty nodes for the analysis of ellipsis in the enhanced dependency representation, we adopt
a further extension of the indexing scheme from v2. It is possible to insert one or more empty nodes indexed <em>i</em>.1, <em>i</em>.2, etc. immediately after a word with index <em>i</em> (where <em>i</em> = 0 for sentence-initial empty nodes). Note that the the numbers after the decimal point must form a sequence starting at 1, i.e. it is not allowed to skip <em>i</em>.1 and use <em>i</em>.2. In the unlikely case that there are ten or more empty nodes between two real tokens, <em>i</em>.10 comes after <em>i</em>.9, that is, the entire ID is not ordered numerically as a decimal number. Here is an example showing the use of an empty node in the analysis of the sentence <em>Sue likes coffee and Bill tea</em>:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1      Sue       Sue
2      likes     like
3      coffee    coffee
4      and       and
5      Bill      Bill
5.1    likes     like
6      tea       tea
</code></pre></div></div>

<p>Empty nodes must have non-empty ID and DEPS fields and empty fields (i.e. underscores) for HEAD and DEPREL, because they are only part of the enhanced dependency graph. All other fields may contain either underscores or values: for example, they can optionally have a word form and lemma as in the example above.</p>

<h1 id="morphological-annotation">Morphological Annotation</h1>

<p>The UPOS field contains a part-of-speech tag from the <a href="u/pos/index.html">universal POS tag</a> set, while the XPOS optionally contains a language-specific part-of-speech tag, normally from a traditional, more fine-grained tagset. If the XPOS field is used, the treebank-specific documentation should define a mapping from XPOS to UPOS values (which may be context-sensitive
and refer to other fields as well). If no language-specific tags are available, the XPOS field should contain an underscore for all words.</p>

<p>The FEATS field contains a list
of morphological features, with vertical bar (|) as list separator and with underscore to represent the empty list.
All features should be represented as attribute-value pairs, with an equals sign (=) separating the attribute from the value. In addition, features should as far as possible be selected from the <a href="u/feat/index.html">universal feature inventory</a> and be sorted alphabetically by attribute names. It is possible to declare that a feature has two or more values for a given word: <code class="highlighter-rouge">Case=Acc,Dat</code>. In this case, the values are sorted alphabetically. In sorting, uppercase letters are considered identical to their lowercase counterparts. Feature names must have the form <code class="highlighter-rouge">[A-Z0-9][A-Z0-9a-z]*(\[[a-z0-9]+\])?</code> and feature values must have the form <code class="highlighter-rouge">[A-Z0-9][a-zA-Z0-9]*</code>.</p>

<p>Here is an example, showing only the first five fields for the
{'fraction_non_alphanumeric': 0.1277996731712006, 'fraction_numerical': 0.01379409785638758, 'mean_word_length': 3.7613272311212813, 'pattern_counts': {'":': 0, '<': 443, '<?xml version=': 0, '>': 443, 'https://': 13, 'lorem ipsum': 0, 'www.': 4, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '854204', 'n_tokens_mistral': 6599, 'n_tokens_neox': 6275, 'n_words': 2341}