<commit_before><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html lang="en-GB" xml:lang="en-GB" xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
    <title>English UD</title>
    <link rel="root" href=""/> <!-- for JS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
    <link rel="stylesheet" type="text/css" href="../css/jquery-ui-redmond.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style-vis.css"/>
    <link rel="stylesheet" type="text/css" href="../css/hint.css"/>
    <script type="text/javascript" src="../lib/ext/head.load.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/3.2.2/anchor.min.js"></script>
    <script>document.addEventListener("DOMContentLoaded", function(event) {anchors.add();});</script>
    <!-- Set up this custom Google search at https://cse.google.com/cse/business/settings?cx=001145188882102106025:dl1mehhcgbo -->
    <!-- DZ 2021-01-22: I am temporarily hiding the search field to find out whether it slows down loading of the title page.
    <script>
      (function() {
        var cx = '001145188882102106025:dl1mehhcgbo';
        var gcse = document.createElement('script');
        gcse.type = 'text/javascript';
        gcse.async = true;
        gcse.src = 'https://cse.google.com/cse.js?cx=' + cx;
        var s = document.getElementsByTagName('script')[0];
        s.parentNode.insertBefore(gcse, s);
      })();
    </script> -->
<!--     <link rel="shortcut icon" href="favicon.ico"/> -->


  </head>
  <body>
    <div id="main" class="center">

      <div id="hp-header">
        <table width="100%"><tr><td width="50%">
          <span class="header-text"><a href="http://universaldependencies.org/#language-en">home</a></span>

          <span class="header-text"><a href="https://github.com/universaldependencies/docs/edit/pages-source/_en/index.md" target="#">edit page</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/issues">issue tracker</a></span>
        </td><td>
          <gcse:search></gcse:search>
        </td></tr></table>
      </div>

      <hr/>

      
      <div class="v2complete">
        This page pertains to UD version 2.
      </div>
      

      <div id="content">
        <noscript>
          <div id="noscript">
            It appears that you have Javascript disabled.
            Please consider enabling Javascript for this page to see the visualizations.
          </div>
        </noscript>

        <!-- The content may include scripts and styles, hence we must load the shared libraries before the content. -->
        <script type="text/javascript">
            console.time('loading libraries');
            var root = '../'; // filled in by jekyll
            head.js(
                // External libraries
                // DZ: Copied from embedding.html. I don't know which one is needed for what, so I'm currently keeping them all.
                root + 'lib/ext/jquery.min.js',
                root + 'lib/ext/jquery.svg.min.js',
                root + 'lib/ext/jquery.svgdom.min.js',
                root + 'lib/ext/jquery.timeago.js',
                root + 'lib/ext/jquery-ui.min.js',
                root + 'lib/ext/waypoints.min.js',
                root + 'lib/ext/jquery.address.min.js'
            );
        </script>
        <h1 id="ud-for-english-----">UD for English <span class="flagspan"><img class="flag" src="../../flags/svg/GB.svg" /></span> <span class="flagspan" style="padding-left:1em"><img class="flag" src="../../flags/svg/US.svg" /></span> <span class="flagspan" style="padding-left:1em"><img class="flag" src="../../flags/svg/CA.svg" /></span> <span class="flagspan" style="padding-left:1em"><img class="flag" src="../../flags/svg/AU.svg" /></span> <span class="flagspan" style="padding-left:1em"><img class="flag" src="../../flags/svg/NZ.svg" /></span></h1>

<p>UD English contains data from multiple treebanks created by different teams at different times and with often different conversion tools (from gold constituent treebanks, such as the English Web Treebank for English-EWT, or from different gold dependency treeebanks, such as English-GUM). As a result, differences may sometimes be found across treebanks, though an effort is made to harmonize them when issues are identified.</p>

<h2 id="tokenization-and-word-segmentation">Tokenization and Word Segmentation</h2>

<p>Words are generally delimited by whitespace or punctuation. No tokens in any of the UD English corpora currently contain whitespace. Most corpora do not use multiword tokens, since clitics in English can usually be expressed using <code class="language-plaintext highlighter-rouge">SpaceAfter=No</code> (for forms like “don’t” = “do” + “n’t”). The <code class="language-plaintext highlighter-rouge">SpaceAfter</code> annotation also allows the distinction between otherwise identical token sequences, such as “can not” versus “cannot”. An exception regarding complex unit is Eng-ParTUT, which does use multiword tokens.</p>

<p>Units that are generally tokenized apart include:</p>

<ul>
  <li>Clitic auxiliaries (‘ll, ‘m, ‘s, ‘ve …)</li>
  <li>Possessive genitive markers (‘s, ‘)</li>
  <li>Clitic negation (n’t, but also not in cannot)</li>
</ul>

<p>Units that are not tokenized apart include:</p>

<ul>
  <li>Acronyms (FBI, U.S.)</li>
  <li>Abbreviations without spaces (e.g., i.e.)</li>
  <li>Hyphenated words, which are treated as tokens and not as compounds (part-time, [a] follow-up)</li>
</ul>

<h2 id="morphology">Morphology</h2>

<h3 id="tags">Tags</h3>

<p>All corpora use the full range of UPOS tags. The XPOS column uses the Penn Treebank tagset. Note that XPOS does not have a simple mapping to UPOS tags, as UD guidelines enforce complex relations between dependency relations and POS tags: for example, since the relation <code class="language-plaintext highlighter-rouge">advmod</code> must generally have the tag <code class="language-plaintext highlighter-rouge">ADV</code>, UPOS may have <code class="language-plaintext highlighter-rouge">ADV</code> for some non-adverbial XPOS tags, and vice versa.</p>

<p>Closed class auxiliaries (tagged <code class="language-plaintext highlighter-rouge">AUX</code>) include:</p>
<ul>
  <li>auxiliaries for periphrastic tenses, aspects, voices and modalities – the lemmas ‘be’, ‘have’, ‘do’, ‘will’, ‘would’, ‘may’, ‘might’, ‘can’, ‘could’, ‘shall’, ‘should’, ‘must’, ‘get’ when attached with relations <code class="language-plaintext highlighter-rouge">aux</code> or <code class="language-plaintext highlighter-rouge">aux:pass</code></li>
  <li>copula – the lemma ‘be’</li>
</ul>

<h3 id="features">Features</h3>

<p>All treebanks currently contain whitespace information, except for English-ESL. Morphological features are included in all corpora except English-ESL. In some corpora these are added automatically using CoreNLP (EWT, GUM) and in some cases supplemented using information from other annotation layers (e.g. GUM).</p>

<h2 id="syntax">Syntax</h2>

<p>Standard deprels are used, except for <code class="language-plaintext highlighter-rouge">clf</code> which is not used in any treebank. Commonly used custom subtypes include <code class="language-plaintext highlighter-rouge">obl:npmod</code> for oblique nominals (corresponds to Stanford Dependencies <code class="language-plaintext highlighter-rouge">npadvmod</code>), <code class="language-plaintext highlighter-rouge">nmod:tmod</code> and <code class="language-plaintext highlighter-rouge">obl:tmod</code> for temporal nouns used adverbially (e.g. “today”), based on the Stanford Dependencies label <code class="language-plaintext highlighter-rouge">tmod</code>. Additionally, passives are distinguished (<code class="language-plaintext highlighter-rouge">nsubj:pass</code>, <code class="language-plaintext highlighter-rouge">csubj:pass</code>), possessived (<code class="language-plaintext highlighter-rouge">nmod:poss</code>), predeterminers (<code class="language-plaintext highlighter-rouge">det:predet</code> for “both” in “both the children), preconj (<code class="language-plaintext highlighter-rouge">cc:preconj</code> for “either” in “either X or Y”) and a special compound subtype for phrasal verb particles (<code class="language-plaintext highlighter-rouge">compound:prt</code> for “up” in “pick up”).</p>

<p>For more information, see the list of <a href="dep/index.html">English relations</a>.</p>

<h2 id="treebanks">Treebanks</h2>

<p>There are six English UD treebanks:</p>

<ul>
  <li>UD_English-ESL</li>
  <li>UD_English-EWT</li>
  <li>UD_English-GUM</li>
  <li>UD_English-LinES</li>
  <li>UD_English-PUD</li>
  <li>UD_English-ParTUT</li>
</ul>

<p>Comparative statistics for tags in the treebanks are available here:</p>

<p><a href="/treebanks/en-comparison.html">https://universaldependencies.org/treebanks/en-comparison.html</a></p>


      </div>

<!-- support for embedded visualizations -->
<script type="text/javascript">
    var root = '../'; // filled in by jekyll
    head.js(
        // We assume that external libraries such as jquery.min.js have already been loaded outside!
        // (See _layouts/base.html.)

        // brat helper modules
        root + 'lib/brat/configuration.js',
        root + 'lib/brat/util.js',
        root + 'lib/brat/annotation_log.js',
        root + 'lib/ext/webfont.js',
        // brat modules
        root + 'lib/brat/dispatcher.js',
        root + 'lib/brat/url_monitor.js',
        root + 'lib/brat/visualizer.js',

        // embedding configuration
        root + 'lib/local/config.js',
        // project-specific collection data
        root + 'lib/local/collections.js',

        // Annodoc
        root + 'lib/annodoc/annodoc.js',

        // NOTE: non-local libraries
        'https://spyysalo.github.io/conllu.js/conllu.js'
    );

    var webFontURLs = [
//        root + 'static/fonts/Astloch-Bold.ttf',
        root + 'static/fonts/PT_Sans-Caption-Web-Regular.ttf',
        root + 'static/fonts/Liberation_Sans-Regular.ttf'
    ];

    var setupTimeago = function() {
        jQuery("time.timeago").timeago();
    };

    head.ready(function() {
        setupTimeago();

        // mark current collection (filled in by Jekyll)
        Collections.listing['_current'] = 'en';

	// perform all embedding and support functions
	Annodoc.activate(Config.bratCollData, Collections.listing);
    });
</script>


<!-- google analytics -->
<script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-55233688-1', 'auto');
  ga('send', 'pageview');

</script>


      <div id="footer">
          <p class="footer-text">&copy; 2014–2021
            <a href="http://universaldependencies.org/introduction.html#contributors" style="color:gray">Universal Dependencies contributors</a>.
            Site powered by <a href="http://spyysalo.github.io/annodoc" style="color:gray">Annodoc</a> and <a href="http://brat.nlplab.org/" style="color:gray">brat</a></p>.
      </div>
    </div>
  </body>
</html>
<commit_msg>autogenerated by https://github.com/spyysalo/jekyll-hook<commit_after><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html lang="en-GB" xml:lang="en-GB" xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
    <title>English UD</title>
    <link rel="root" href=""/> <!-- for JS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
    <link rel="stylesheet" type="text/css" href="../css/jquery-ui-redmond.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style-vis.css"/>
    <link rel="stylesheet" type="text/css" href="../css/hint.css"/>
    <script type="text/javascript" src="../lib/ext/head.load.min.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/3.2.2/anchor.min.js"></script>
    <script>document.addEventListener("DOMContentLoaded", function(event) {anchors.add();});</script>
    <!-- Set up this custom Google search at https://cse.google.com/cse/business/settings?cx=001145188882102106025:dl1mehhcgbo -->
    <!-- DZ 2021-01-22: I am temporarily hiding the search field to find out whether it slows down loading of the title page.
    <script>
      (function() {
        var cx = '001145188882102106025:dl1mehhcgbo';
        var gcse = document.createElement('script');
        gcse.type = 'text/javascript';
        gcse.async = true;
        gcse.src = 'https://cse.google.com/cse.js?cx=' + cx;
        var s = document.getElementsByTagName('script')[0];
        s.parentNode.insertBefore(gcse, s);
      })();
    </script> -->
<!--     <link rel="shortcut icon" href="favicon.ico"/> -->


  </head>
  <body>
    <div id="main" class="center">

      <div id="hp-header">
        <table width="100%"><tr><td width="50%">
          <span class="header-text"><a href="http://universaldependencies.org/#language-en">home</a></span>

          <span class="header-text"><a href="https://github.com/universaldependencies/docs/edit/pages-source/_en/index.md" target="#">edit page</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/issues">issue tracker</a></span>
        </td><td>
          <gcse:search></gcse:search>
        </td></tr></table>
      </div>

      <hr/>

      
      <div class="v2complete">
        This page pertains to UD version 2.
      </div>
      

      <div id="content">
        <noscript>
          <div id="noscript">
            It appears that you have Javascript disabled.
            Please consider enabling Javascript for this page to see the visualizations.
          </div>
        </noscript>

        <!-- The content may include scripts and styles, hence we must load the shared libraries before the content. -->
        <script type="text/javascript">
            console.time('loading libraries');
            var root = '../'; // filled in by jekyll
            head.js(
                // External libraries
                // DZ: Copied from embedding.html. I don't know which one is needed for what, so I'm currently keeping them all.
                root + 'lib/ext/jquery.min.js',
                root + 'lib/ext/jquery.svg.min.js',
                root + 'lib/ext/jquery.svgdom.min.js',
                root + 'lib/ext/jquery.timeago.js',
                root + 'lib/ext/jquery-ui.min.js',
                root + 'lib/ext/waypoints.min.js',
                root + 'lib/ext/jquery.address.min.js'
            );
        </script>
        <h1 id="ud-for-english-----">UD for English <span class="flagspan"><img class="flag" src="../../flags/svg/GB.svg" /></span> <span class="flagspan" style="padding-left:1em"><img class="flag" src="../../flags/svg/US.svg" /></span> <span class="flagspan" style="padding-left:1em"><img class="flag" src="../../flags/svg/CA.svg" /></span> <span class="flagspan" style="padding-left:1em"><img class="flag" src="../../flags/svg/AU.svg" /></span> <span class="flagspan" style="padding-left:1em"><img class="flag" src="../../flags/svg/NZ.svg" /></span></h1>

<p>UD English contains data from multiple treebanks created by different teams at different times and with often different conversion tools (from gold constituent treebanks, such as the English Web Treebank for English-EWT, or from different gold dependency treeebanks, such as English-GUM). As a result, differences may sometimes be found across treebanks, though an effort is made to harmonize them when issues are identified.</p>

<h2 id="tokenization-and-word-segmentation">Tokenization and Word Segmentation</h2>

<p>Words are generally delimited by whitespace or punctuation. No tokens in any of the UD English corpora currently contain whitespace. Multiword tokens should be used for English clitics, such as <em>‘ll</em> (reduced form of the auxiliary <em>will), _n’t</em> (reduced form of <em>not</em>) and <em>‘s</em> (possessive clitic). For example, <em>don’t</em> = <em>do</em> + <em>n’t</em>). As of mid 2021, multiword tokens are used in the following English corpora: GUM, GUMReddit, and EWT, are partially used in ParTUT (used for forms like <em>ain’t</em> and <em>can’t</em> but not for forms that are concatenative like <em>John’s</em> or <em>she’ll</em>), but are not used in: PUD, LinES, Pronouns, or ESL. If multiword tokens are not present, clitics in English can usually be identified by using the <code class="language-plaintext highlighter-rouge">SpaceAfter=No</code> annotation and it also allows distinguishing between otherwise identical token sequences, such as “can not” versus “cannot”.</p>

<p>Units that should be regarded as separate syntactic words include:</p>

<ul>
  <li>Clitic auxiliaries (‘ll, ‘m, ‘s, ‘ve, ‘d, …)</li>
  <li>Possessive genitive markers (‘s, ‘)</li>
  <li>Clitic negation (n’t, and also not in cannot)</li>
  <li>Most hyphenated terms (search-engine becomes 3 words: search, -, engine)</li>
</ul>

<p>Units that are not tokenized apart include:</p>

<ul>
  <li>Acronyms (FBI, U.S.)</li>
  <li>Abbreviations without spaces (e.g., i.e.)</li>
  <li>Some hyphenated words, with common prefixes or occasionally suffixes, such as <em>e-mail</em> or <em>co-ordinated</em></li>
</ul>

<h2 id="morphology">Morphology</h2>

<h3 id="tags">Tags</h3>

<p>All corpora use the full range of UPOS tags. The XPOS column uses the Penn Treebank tagset (as extended in subsequent LDC corpus releases). Note that XPOS does not have a simple mapping to UPOS tags, as UD guidelines enforce complex relations between dependency relations and POS tags: for example, since the relation <code class="language-plaintext highlighter-rouge">advmod</code> must generally have the tag <code class="language-plaintext highlighter-rouge">ADV</code>, UPOS may have <code class="language-plaintext highlighter-rouge">ADV</code> for some non-adverbial XPOS tags, and vice versa.</p>

<p>Closed class auxiliaries (tagged <code class="language-plaintext highlighter-rouge">AUX</code>) include:</p>
<ul>
  <li>auxiliaries for periphrastic tenses, aspects, voices and modalities – the lemmas ‘be’, ‘have’, ‘do’, ‘will’, ‘would’, ‘may’, ‘might’, ‘can’, ‘could’, ‘shall’, ‘should’, ‘must’, ‘get’ when attached with relations <code class="language-plaintext highlighter-rouge">aux</code> or <code class="language-plaintext highlighter-rouge">aux:pass</code></li>
  <li>copula – the lemma ‘be’</li>
</ul>

<h3 id="features">Features</h3>

<p>All treebanks currently contain whitespace information, except for English-ESL. Morphological features are included in all corpora except English-ESL. In some corpora these are added automatically using CoreNLP (EWT, GUM) and in some cases supplemented using information from other annotation layers (e.g. GUM).</p>

<h2 id="syntax">Syntax</h2>

<p>Standard deprels are used, except for <code class="language-plaintext highlighter-rouge">clf</code> which is not used in any treebank. Commonly used custom subtypes include <code class="language-plaintext highlighter-rouge">obl:npmod</code> for oblique nominals (corresponds to Stanford Dependencies <code class="language-plaintext highlighter-rouge">npadvmod</code>), <code class="language-plaintext highlighter-rouge">nmod:tmod</code> and <code class="language-plaintext highlighter-rouge">obl:tmod</code> for temporal nouns used adverbially (e.g. “today”), based on the Stanford Dependencies label <code class="language-plaintext highlighter-rouge">tmod</code>. Additionally, passives are distinguished (<code class="language-plaintext highlighter-rouge">nsubj:pass</code>, <code class="language-plaintext highlighter-rouge">csubj:pass</code>), pre-nominal possessives (<code class="language-plaintext highlighter-rouge">nmod:poss</code>), predeterminers (<code class="language-plaintext highlighter-rouge">det:predet</code> for “both” in “both the children), preconj (<code class="language-plaintext highlighter-rouge">cc:preconj</code> for “either” in “either X or Y”) and a special compound subtype for phrasal verb particles (<code class="language-plaintext highlighter-rouge">compound:prt</code> for “up” in “pick up”).</p>

<p>For more information, see the list of <a href="dep/index.html">English relations</a>.</p>

<h2 id="treebanks">Treebanks</h2>

<p>There are six English UD treebanks:</p>

<ul>
  <li>UD_English-ESL</li>
  <li>UD_English-EWT</li>
  <li>UD_English-GUM</li>
  <li>UD_English-LinES</li>
  <li>UD_English-PUD</li>
  <li>UD_English-ParTUT</li>
</ul>

<p>Comparative statistics for tags in the treebanks are available here:</p>

<p><a href="/treebanks/en-comparison.html">https://universaldependencies.org/treebanks/en-comparison.html</a></p>


      </div>

<!-- support for embedded visualizations -->
<script type="text/javascript">
    var root = '../'; // filled in by jekyll
    head.js(
        // We assume that external libraries such as jquery.min.js have already been loaded outside!
        // (See _layouts/base.html.)

        // brat helper modules
        root + 'lib/brat/configuration.js',
        root + 'lib/brat/util.js',
        root + 'lib/brat/annotation_log.js',
        root + 'lib/ext/webfont.js',
        // brat modules
        root + 'lib/brat/dispatcher.js',
        root + 'lib/brat/url_monitor.js',
        root + 'lib/brat/visualizer.js',

        // embedding configuration
        root + 'lib/local/config.js',
        // project-specific collection data
        root + 'lib/local/collections.js',

        // Annodoc
        root + 'lib/annodoc/annodoc.js',

        // NOTE: non-local libraries
        'https://spyysalo.github.io/conllu.js/conllu.js'
    );

    var webFontURLs = [
//        root + 'static/fonts/Astloch-Bold.ttf',
        root + 'static/fonts/PT_Sans-Caption-Web-Regular.ttf',
        root + 'static/fonts/Liberation_Sans-Regular.ttf'
    ];

    var setupTimeago = function() {
        jQuery("time.timeago").timeago();
    };

    head.ready(function() {
        setupTimeago();

        // mark current collection (filled in by Jekyll)
        Collections.listing['_current'] = 'en';

	// perform all embedding and support functions
	Annodoc.activate(Config.bratCollData, Collections.listing);
    });
</script>


<!-- google analytics -->
<script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-55233688-1', 'auto');
  ga('send', 'pageview');

</script>


      <div id="footer">
          <p class="footer-text">&copy; 2014–2021
            <a href="http://universaldependencies.org/introduction.html#contributors" style="color:gray">Universal Dependencies contributors</a>.
            Site powered by <a href="http://spyysalo.github.io/annodoc" style="color:gray">Annodoc</a> and <a href="http://brat.nlplab.org/" style="color:gray">brat</a></p>.
      </div>
    </div>
  </body>
</html>

{'fraction_non_alphanumeric': 0.16169854950599116, 'fraction_numerical': 0.00996426319108682, 'mean_word_length': 3.559325282729538, 'pattern_counts': {'":': 0, '<': 461, '<?xml version=': 0, '>': 461, 'https://': 17, 'lorem ipsum': 0, 'www.': 6, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5657885', 'n_tokens_mistral': 7898, 'n_tokens_neox': 7511, 'n_words': 1926}