<commit_before><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html lang="en-GB" xml:lang="en-GB" xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
    <title>POS tags</title>
    <link rel="root" href=""/> <!-- for JS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
    <link rel="stylesheet" type="text/css" href="../css/jquery-ui-redmond.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style-vis.css"/>
    <link rel="stylesheet" type="text/css" href="../css/hint.css"/>
    <script type="text/javascript" src="../lib/ext/head.load.min.js"></script>
    <script type="text/javascript" src="../lib/ext/jquery.timeago.js"></script>
<!--     <link rel="shortcut icon" href="favicon.ico"/> -->


  </head>
  <body>
    <div id="main" class="center">

      <div id="hp-header">
	
          <span class="header-text"><a href="http://universaldependencies.org/#language-v2">home</a></span>
          <span class="header-text"><a href="http://universaldependencies.org/v2/v2">v2/v2</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/edit/pages-source/_v2/postags.md" target="#">edit page</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/issues">issue tracker</a></span>
      </div>

      <hr/>

      
      <div class="v2complete">
	This page pertains to UD version 2.
      </div>
      
      
      <div id="content">
	<noscript>
	  <div id="noscript">
	    It appears that you have Javascript disabled.
	    Please consider enabling Javascript for this page to see the visualizations.
	  </div>
	</noscript>

          <h1 id="part-of-speech-tags-in-ud-v2">Part-of-speech tags in UD v2</h1>

<h2 id="summary">Summary</h2>

<p>For v2, we propose</p>

<ul>
  <li>Rename <code>CONJ</code> to <a href="">u-pos/CCONJ</a>.</li>
  <li>Extend the use of <a href="">u-pos/AUX</a> to copula verbs and nonverbal TAMVE particles.</li>
  <li>Minimize usage of <a href="">u-pos/PART</a> – small language-specific list of words, case must be made for each.</li>
  <li>Loosen borderline between <a href="">u-pos/PRON</a> and <a href="">u-pos/DET</a>: set of recommendations instead of purely functional rule.</li>
  <li>Provide more general recommendations about setting language-specific borderlines between word categories.</li>
</ul>

<p>For related discussion see</p>

<ul>
  <li><a href="../u/overview/morphology.html">Morphology overview page</a> has a section on POS tags</li>
  <li><a href="../u/pos/index.html">Current guidelines for POS tags</a></li>
  <li><a href="../v2/form_vs_function.html">Form vs. function in POS tags</a></li>
  <li><a href="http://github.com/UniversalDependencies/docs/issues/159">Issue 159</a> (<code>PRON</code> vs. <code>DET</code>) and the <a href="../2015-08-23-uppsala/determiners.html">Uppsala report on determiners</a></li>
  <li><a href="http://github.com/UniversalDependencies/docs/issues/178#issuecomment-212822810">Issue 178</a> (question particles)</li>
  <li><a href="http://github.com/UniversalDependencies/docs/issues/237">Issue 237</a> (<code>ADV</code> vs. <code>CONJ</code>)</li>
  <li><a href="http://github.com/UniversalDependencies/docs/issues/275">Issue 275</a> (copula <code>VERB</code> vs. <code>AUX</code>)</li>
  <li><a href="http://github.com/UniversalDependencies/docs/issues?q=is%3Aissue%20is%3Aopen%20label%3APOS">List of open issues with the POS label</a></li>
</ul>

<h2 id="rename-conj-to-cconj">Rename CONJ to CCONJ</h2>

<p>We propose to rename the <code>CONJ</code> tag to <code>CCONJ</code> <em>(coordinating conjunction)</em>
and thus make it analogous to <code>SCONJ</code> <em>(subordinating conjunction).</em>
That will hopefully reduce the confusion that we observed in v1:
people have assumed that the <code>CONJ</code> tag would correspond to the <code>conj</code> relation,
while in fact it usually went with the relation <code>cc</code> and practically never with <code>conj</code>.</p>

<h2 id="keep-propn">Keep PROPN</h2>

<p>We considered removing <code>PROPN</code> as a separate POS tag because
1. the motivation to have it is weak in some languages (no grammatical implications except for capital letters);
2. the possibility to say that a language does not have <code>PROPN</code> does not really solve anything – these languages indeed have the category and typically will want to use it if it exists, to increase cross-linguistic parallelism;
3. disambiguation from <code>NOUN</code> is hard in many languages;
4. it would be better to have phrase-level named entity annotation, which would include non-noun words (adjectives) and nesting.</p>

<p>We decided to keep the tag in v2 because
1. the category has been traditionally distinguished in a significant number of languages and tagsets, and people do not want to lose it there;
2. the reasons above do not seem strong enough to remove it;
3. we are not going to have named entity annotation in UD v2.</p>

<h2 id="keep-part-but-minimize-its-usage">Keep PART but minimize its usage</h2>

<p>We considered removing <code>PART</code> because it is a very small category, which is defined mostly negatively (it is particle if it is not anything else). We decided to keep it because
1. we would have to take the current particles one-by-one in each language and decide where to put them;
2. in some instances it would still be hard to find a suitable category;
3. which may actually lead to creation of several new and even smaller categories.</p>

<p>However, the lists of particles in individual languages should be reviewed anyways. Language-specific documentation must list all particles in the language, and ideally also explain why they are particles.</p>

<p>Note that the current guidelines say that [en] <em>not</em>, [de] <em>nicht</em> etc. are negative particles (but negative determiners like [en] <em>no</em>, or negative auxiliary verbs like [cs] <em>není</em> are not particles). This is the only positive part of the current <code>PART</code> definition at the language-universal level, because such words were traditionally tagged as adverbs in some languages, and they could be adverbs in the other languages too, had we decided to remove <code>PART</code>.</p>

<h2 id="extend-the-use-of-aux">Extend the use of AUX</h2>

<p>The <a href="">u-pos/AUX</a> category is currently used for auxiliary verbs, that is verbs used with the syntactic relation <a href="">u-dep/aux</a> (regardless of whether these verbs can be used as main verbs in other contexts). We propose to extend its use in two ways. First, it can be used for nonverbal particles used to express TAMVE categories, which in v2 will also be analyzed using the <a href="">u-dep/aux</a> relation. Secondly, it will be extended to copula verbs, which perform a grammaticalized function in nominal clauses.</p>

<!--In many languages, (some of) the auxiliary verbs may also be used as main verbs. Their paradigm may be defective but the forms they have usually do not distinguish auxiliary usage from full-verb usage, so the `AUX` tag can only be inferred from the syntactic relation [u-dep/aux](). We believe that each POS category should be able to contribute some additional information rather than just mirror a dependency relation (see also further discussion in the section _Borderline between categories_). Since the `AUX` tag served only to distinguish auxiliary and non-auxiliary usage of a verb, it should be removed; the two usages can be recognized by the `aux` relation.

If it is desirable to keep the morphological distinction in a treebank (e.g. because a verb is only auxiliary in a language, and it differs morphologically from normal verbs), the language-specific feature VerbType=Aux (or Mod or Cop) can be used, but there will be no dedicated POS tag.

Removing the tag means changing it to `VERB` in data, removing all the `AUX` pages from the documentation, and also remove references to it from other parts of the documentation (probably from `VERB`, `aux` and `auxpass`; quite likely also from elsewhere).

(NB: If this proposal is not approved by the core group, we should consider re-classifying copula verbs as auxiliaries.)-->

<h2 id="borderlines-between-categories">Borderlines between categories</h2>

<p>Parts of speech have traditionally been defined using a mix of morphological, syntactic and semantic criteria. Given that UD is concerned with grammatical annotation, we in general want to give less priority to (purely) semantic criteria. Morphological criteria are very useful but are not applicable to all languages, or even to all categories within morphologically rich languages. It follows that, in many cases, we have to rely primarily on syntactic criteria. However, we believe that the part-of-speech classification is most useful if it captures regular, prevailing syntactic behavior and does not reflect sentence-specific exceptional behavior. Therefore, we should avoid completely functional guidelines for part-of-speech tags that make the tag completely predictable from the syntactic function. In addition to the obvious argument that such guidelines make (some) tags uninformative, they also make it harder to find interesting cross-lingual differences.
For example, language X allows words of category A to have syntactic function b, but language Y doesn’t.</p>
<commit_msg>autogenerated by https://github.com/spyysalo/jekyll-hook<commit_after><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html lang="en-GB" xml:lang="en-GB" xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
    <title>POS tags</title>
    <link rel="root" href=""/> <!-- for JS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
    <link rel="stylesheet" type="text/css" href="../css/jquery-ui-redmond.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style-vis.css"/>
    <link rel="stylesheet" type="text/css" href="../css/hint.css"/>
    <script type="text/javascript" src="../lib/ext/head.load.min.js"></script>
    <script type="text/javascript" src="../lib/ext/jquery.timeago.js"></script>
<!--     <link rel="shortcut icon" href="favicon.ico"/> -->


  </head>
  <body>
    <div id="main" class="center">

      <div id="hp-header">
	
          <span class="header-text"><a href="http://universaldependencies.org/#language-v2">home</a></span>
          <span class="header-text"><a href="http://universaldependencies.org/v2/v2">v2/v2</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/edit/pages-source/_v2/postags.md" target="#">edit page</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/issues">issue tracker</a></span>
      </div>

      <hr/>

      
      <div class="v2complete">
	This page pertains to UD version 2.
      </div>
      
      
      <div id="content">
	<noscript>
	  <div id="noscript">
	    It appears that you have Javascript disabled.
	    Please consider enabling Javascript for this page to see the visualizations.
	  </div>
	</noscript>

          <h1 id="part-of-speech-tags-in-ud-v2">Part-of-speech tags in UD v2</h1>

<h2 id="summary">Summary</h2>

<p>For v2, we propose</p>

<ul>
  <li>Rename <code>CONJ</code> to <a href="">u-pos/CCONJ</a>.</li>
  <li>Extend the use of <a href="">u-pos/AUX</a> to copula verbs and nonverbal TAMVE particles.</li>
  <li>Minimize usage of <a href="">u-pos/PART</a> – small language-specific list of words, case must be made for each.</li>
  <li>Loosen borderline between <a href="">u-pos/PRON</a> and <a href="">u-pos/DET</a>: set of recommendations instead of purely functional rule.</li>
  <li>Provide more general recommendations about setting language-specific borderlines between word categories.</li>
</ul>

<p>For related discussion see</p>

<ul>
  <li><a href="../u/overview/morphology.html">Morphology overview page</a> has a section on POS tags</li>
  <li><a href="../u/pos/index.html">Current guidelines for POS tags</a></li>
  <li><a href="../v2/form_vs_function.html">Form vs. function in POS tags</a></li>
  <li><a href="http://github.com/UniversalDependencies/docs/issues/159">Issue 159</a> (<code>PRON</code> vs. <code>DET</code>) and the <a href="../2015-08-23-uppsala/determiners.html">Uppsala report on determiners</a></li>
  <li><a href="http://github.com/UniversalDependencies/docs/issues/178#issuecomment-212822810">Issue 178</a> (question particles)</li>
  <li><a href="http://github.com/UniversalDependencies/docs/issues/237">Issue 237</a> (<code>ADV</code> vs. <code>CONJ</code>)</li>
  <li><a href="http://github.com/UniversalDependencies/docs/issues/275">Issue 275</a> (copula <code>VERB</code> vs. <code>AUX</code>)</li>
  <li><a href="http://github.com/UniversalDependencies/docs/issues?q=is%3Aissue%20is%3Aopen%20label%3APOS">List of open issues with the POS label</a></li>
</ul>

<h2 id="rename-conj-to-cconj">Rename CONJ to CCONJ</h2>

<p>We propose to rename the <code>CONJ</code> tag to <code>CCONJ</code> <em>(coordinating conjunction)</em>
and thus make it analogous to <code>SCONJ</code> <em>(subordinating conjunction).</em>
That will hopefully reduce the confusion that we observed in v1:
people have assumed that the <code>CONJ</code> tag would correspond to the <code>conj</code> relation,
while in fact it usually went with the relation <code>cc</code> and practically never with <code>conj</code>.</p>

<h2 id="keep-propn">Keep PROPN</h2>

<p>We considered removing <code>PROPN</code> as a separate POS tag because
1. the motivation to have it is weak in some languages (no grammatical implications except for capital letters);
2. the possibility to say that a language does not have <code>PROPN</code> does not really solve anything – these languages indeed have the category and typically will want to use it if it exists, to increase cross-linguistic parallelism;
3. disambiguation from <code>NOUN</code> is hard in many languages;
4. it would be better to have phrase-level named entity annotation, which would include non-noun words (adjectives) and nesting.</p>

<p>We decided to keep the tag in v2 because
1. the category has been traditionally distinguished in a significant number of languages and tagsets, and people do not want to lose it there;
2. the reasons above do not seem strong enough to remove it;
3. we are not going to have named entity annotation in UD v2.</p>

<h2 id="keep-part-but-minimize-its-usage">Keep PART but minimize its usage</h2>

<p>We considered removing <code>PART</code> because it is a very small category, which is defined mostly negatively (it is particle if it is not anything else). We decided to keep it because
1. we would have to take the current particles one-by-one in each language and decide where to put them;
2. in some instances it would still be hard to find a suitable category;
3. which may actually lead to creation of several new and even smaller categories.</p>

<p>However, the lists of particles in individual languages should be reviewed anyways. Language-specific documentation must list all particles in the language, and ideally also explain why they are particles.</p>

<p>Note that the current guidelines say that [en] <em>not</em>, [de] <em>nicht</em> etc. are negative particles (but negative determiners like [en] <em>no</em>, or negative auxiliary verbs like [cs] <em>není</em> are not particles). This is the only positive part of the current <code>PART</code> definition at the language-universal level, because such words were traditionally tagged as adverbs in some languages, and they could be adverbs in the other languages too, had we decided to remove <code>PART</code>.</p>

<h2 id="extend-the-use-of-aux">Extend the use of AUX</h2>

<p>The <a href="">u-pos/AUX</a> category is currently used for auxiliary verbs, that is verbs used with the syntactic relation <a href="">u-dep/aux</a> (regardless of whether these verbs can be used as main verbs in other contexts). We propose to extend its use in two ways. First, it can be used for nonverbal particles used to express TAMVE categories, which in v2 will also be analyzed using the <a href="">u-dep/aux</a> relation. Secondly, it will be extended to copula (<a href="">u-dep/cop</a>) verbs, which perform a grammaticalized function in nominal clauses.</p>

<!--In many languages, (some of) the auxiliary verbs may also be used as main verbs. Their paradigm may be defective but the forms they have usually do not distinguish auxiliary usage from full-verb usage, so the `AUX` tag can only be inferred from the syntactic relation [u-dep/aux](). We believe that each POS category should be able to contribute some additional information rather than just mirror a dependency relation (see also further discussion in the section _Borderline between categories_). Since the `AUX` tag served only to distinguish auxiliary and non-auxiliary usage of a verb, it should be removed; the two usages can be recognized by the `aux` relation.

If it is desirable to keep the morphological distinction in a treebank (e.g. because a verb is only auxiliary in a language, and it differs morphologically from normal verbs), the language-specific feature VerbType=Aux (or Mod or Cop) can be used, but there will be no dedicated POS tag.

Removing the tag means changing it to `VERB` in data, removing all the `AUX` pages from the documentation, and also remove references to it from other parts of the documentation (probably from `VERB`, `aux` and `auxpass`; quite likely also from elsewhere).

(NB: If this proposal is not approved by the core group, we should consider re-classifying copula verbs as auxiliaries.)-->

<h2 id="borderlines-between-categories">Borderlines between categories</h2>

<p>Parts of speech have traditionally been defined using a mix of morphological, syntactic and semantic criteria. Given that UD is concerned with grammatical annotation, we in general want to give less priority to (purely) semantic criteria. Morphological criteria are very useful but are not applicable to all languages, or even to all categories within morphologically rich languages. It follows that, in many cases, we have to rely primarily on syntactic criteria. However, we believe that the part-of-speech classification is most useful if it captures regular, prevailing syntactic behavior and does not reflect sentence-specific exceptional behavior. Therefore, we should avoid completely functional guidelines for part-of-speech tags that make the tag completely predictable from the syntactic function. In addition to the obvious argument that such guidelines make (some) tags uninformative, they also make it harder to find interesting cross-lingual differences.
For example, language X allows words of category A to have syntactic function b, but language Y doesn’t.</p>

{'fraction_non_alphanumeric': 0.12109455824539624, 'fraction_numerical': 0.010655907303951996, 'mean_word_length': 4.424523007856341, 'pattern_counts': {'":': 0, '<': 403, '<?xml version=': 0, '>': 403, 'https://': 7, 'lorem ipsum': 0, 'www.': 4, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6151786', 'n_tokens_mistral': 5852, 'n_tokens_neox': 5500, 'n_words': 2260}