<commit_before><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html lang="en-GB" xml:lang="en-GB" xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
    <title>Segmentation in UD v2</title>
    <link rel="root" href=""/> <!-- for JS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
    <link rel="stylesheet" type="text/css" href="../css/jquery-ui-redmond.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style-vis.css"/>
    <link rel="stylesheet" type="text/css" href="../css/hint.css"/>
    <script type="text/javascript" src="../lib/ext/head.load.min.js"></script>
    <script type="text/javascript" src="../lib/ext/jquery.timeago.js.js"></script>
<!--     <link rel="shortcut icon" href="favicon.ico"/> -->


  </head>
  <body>
    <div id="main" class="center">

      <div id="hp-header">
	
          <span class="header-text"><a href="http://universaldependencies.org/#language-2016">home</a></span>
          <span class="header-text"><a href="http://universaldependencies.org/2016/prague">2016/prague</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/edit/pages-source/_2016-09-05--06-prague/segmentation.md" target="#">edit page</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/issues">issue tracker</a></span>
      </div>

      <hr/>

      <div id="content">
	<noscript>
	  <div id="noscript">
	    It appears that you have Javascript disabled.
	    Please consider enabling Javascript for this page to see the visualizations.
	  </div>
	</noscript>

          <h1 id="segmentation-in-ud-v2">Segmentation in UD v2</h1>

<p>The UD scheme makes a distinction between <strong>tokens</strong>, word-like elements that can be identified using simple rules, often relying (only) on whitespace and punctuation, and <strong>words</strong>, which are the linguistically relevant word units needed for morphological and syntactic analysis. In v1, it was assumed that words would never contain spaces, and that “multitoken words” should always be handled using special relations like <a href="">u-dep/mwe</a> and <a href="">u-dep/goeswith</a>. For v2, we propose to relax this assumption in two ways:</p>

<ul>
  <li>Allow words with spaces for languages where spaces mark something else than word boundaries (for example, syllable boundaries as in Vietnamese)</li>
  <li>Allow words with spaces (in any language) for an approved (and restricted) list of exceptions like numbers (“100 000”) and abbreviations (“i. e.”)</li>
</ul>

<p>Throughout the remainder of this text, the symbol <code>⎵</code> will be used to indicate orthographic space.</p>

<h2 id="problems-with-current-treatment-of-word-segmentation">Problems with current treatment of word segmentation</h2>

<p>There are two main problems with the current treatment of word segmentation:</p>

<ul>
  <li>The first is that for some languages (Vietnamese being the prototypical example) spaces do not mark word boundaries, but rather syllable boundaries.</li>
  <li>In other languages, some kind of spacing character is used as a delimiter in numerals, or is optionally used in abbreviations.</li>
</ul>

<h3 id="spaces-as-syllable-delimiters">Spaces as syllable delimiters</h3>

<p>There is pretty much unanimous agreement that words with spaces should be allowed in the Vietnamese treebank, because the alternative of treating all polysyllabic words as multiword expressions would artificially make Vietnamese look very different from all other languages. As far as we know, Vietnamese is the only language where this is necessary, but still all tools will need to be able to support having spaces in CoNLL-U columns. Consider the following example, “Minh is (a) teacher.”, where <i>giáo viên</i> is a bisyllabic word meaning “teacher”. (Currently using underscore, “giáo⎵viên”, because even the tree visualization tool cannot work with word-internal spaces.)</p>

<pre><code class="language-conllu">1	Minh	Minh	PROPN	_	_	3	nsubj	_	_
2	là	là	VERB	_	_	3	cop	_	_
3	giáo⎵viên	giáo⎵viên	NOUN	_	_	0	  root  _ _
4	.	.	PUNCT	_	_	3	punct	_	_
</code></pre>

<h3 id="other-cases">Other cases</h3>

<p>There was a general consensus that for the remainder of the languages, we should maintain the ban on spaces in tokens. However, we propose that for a highly restricted closed class of orthographic phenomena (with prior approval), there may be some exceptions, for example:</p>

<h4 id="spaces-as-numeral-separators">Spaces as numeral separators</h4>

<p>In the existing French treebank, space delimited numerals, e.g. “100 000” are collapsed into a single numeral, “… de 8 500 à 20 000 euros.” becomes:</p>

<pre><code class="language-conllu">1	de	de	ADP	_	_	2	case	_	_
2	8500	8500	NUM	_	_	0	nmod	_	_
3	à	à	ADP	_	_	5	case	_	_
4	20000	20000	NUM	_	_	5	nummod	_	_
5	euros	euro	NOUN	_	Gender=Masc|Number=Plur	0	nmod	_	_
</code></pre>

<p>We do not see that this is an improvement over simply allowing the space, and the alternative (to have each 000 as a 
separate token and use <code>goeswith</code> or <code>mwe</code>) is unwieldy and does not exactly fit, e.g. writing 100 000 is not an 
orthographic error, but rather orthographically normative, and neither is it a fixed expression.</p>

<p>The new tokenisation would be:</p>

<pre><code class="language-sdparse">de 8⎵500 à 20⎵000 euros \n from 8,500 to 20,000 euros
</code></pre>

<h4 id="spaces-in-normalising-abbreviations">Spaces in normalising abbreviations</h4>
<commit_msg>autogenerated by https://github.com/spyysalo/jekyll-hook<commit_after><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html lang="en-GB" xml:lang="en-GB" xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
    <title>Segmentation in UD v2</title>
    <link rel="root" href=""/> <!-- for JS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
    <link rel="stylesheet" type="text/css" href="../css/jquery-ui-redmond.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style-vis.css"/>
    <link rel="stylesheet" type="text/css" href="../css/hint.css"/>
    <script type="text/javascript" src="../lib/ext/head.load.min.js"></script>
    <script type="text/javascript" src="../lib/ext/jquery.timeago.js.js"></script>
<!--     <link rel="shortcut icon" href="favicon.ico"/> -->


  </head>
  <body>
    <div id="main" class="center">

      <div id="hp-header">
	
          <span class="header-text"><a href="http://universaldependencies.org/#language-2016">home</a></span>
          <span class="header-text"><a href="http://universaldependencies.org/2016/prague">2016/prague</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/edit/pages-source/_2016-09-05--06-prague/segmentation.md" target="#">edit page</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/issues">issue tracker</a></span>
      </div>

      <hr/>

      <div id="content">
	<noscript>
	  <div id="noscript">
	    It appears that you have Javascript disabled.
	    Please consider enabling Javascript for this page to see the visualizations.
	  </div>
	</noscript>

          <h1 id="segmentation-in-ud-v2">Segmentation in UD v2</h1>

<p>The UD scheme makes a distinction between <strong>tokens</strong>, word-like elements that can be identified using simple rules, often relying (only) on whitespace and punctuation, and <strong>words</strong>, which are the linguistically relevant word units needed for morphological and syntactic analysis. In v1, it was assumed that words would never contain spaces, and that “multitoken words” should always be handled using special relations like <a href="">u-dep/mwe</a> and <a href="">u-dep/goeswith</a>. For v2, we propose to relax this assumption in two ways:</p>

<ul>
  <li>Allow words with spaces for languages where spaces mark something else than word boundaries (for example, syllable boundaries as in Vietnamese)</li>
  <li>Allow words with spaces (in any language) for an approved (and restricted) list of exceptions like numbers (“100 000”) and abbreviations (“i. e.”)</li>
</ul>

<p>Throughout the remainder of this text, the symbol <code>⎵</code> will be used to indicate orthographic space.</p>

<h2 id="problems-with-current-treatment-of-word-segmentation">Problems with current treatment of word segmentation</h2>

<p>There are two main problems with the current treatment of word segmentation:</p>

<ul>
  <li>The first is that for some languages (Vietnamese being the prototypical example) spaces do not mark word boundaries, but rather syllable boundaries.</li>
  <li>In other languages, some kind of spacing character is used as a delimiter in numerals, or is optionally used in abbreviations.</li>
</ul>

<h3 id="spaces-as-syllable-delimiters">Spaces as syllable delimiters</h3>

<p>There is pretty much unanimous agreement that words with spaces should be allowed in the Vietnamese treebank, because the alternative of treating all polysyllabic words as multiword expressions would artificially make Vietnamese look very different from all other languages. As far as we know, Vietnamese is the only language where this is necessary, but still all tools will need to be able to support having spaces in CoNLL-U columns. Consider the following example, “Minh is (a) teacher.”, where <i>giáo viên</i> is a bisyllabic word meaning “teacher”. (Currently using underscore, “giáo⎵viên”, because even the tree visualization tool cannot work with word-internal spaces.)</p>

<pre><code class="language-conllu">
1	Minh	Minh	PROPN	_	_	3	nsubj	_	_
2	là	là	VERB	_	_	3	cop	_	_
3	giáo⎵viên	giáo⎵viên	NOUN	_	_	0	  root  _ _
4	.	.	PUNCT	_	_	3	punct	_	_

</code></pre>

<h3 id="other-cases">Other cases</h3>

<p>There was a general consensus that for the remainder of the languages, we should maintain the ban on spaces in tokens. However, we propose that for a highly restricted closed class of orthographic phenomena (with prior approval), there may be some exceptions, for example:</p>

<h4 id="spaces-as-numeral-separators">Spaces as numeral separators</h4>

<p>In the existing French treebank, space delimited numerals, e.g. “100 000” are collapsed into a single numeral, “… de 8 500 à 20 000 euros.” becomes:</p>

<pre><code class="language-conllu">1	de	de	ADP	_	_	2	case	_	_
2	8500	8500	NUM	_	_	0	nmod	_	_
3	à	à	ADP	_	_	5	case	_	_
4	20000	20000	NUM	_	_	5	nummod	_	_
5	euros	euro	NOUN	_	Gender=Masc|Number=Plur	0	nmod	_	_
</code></pre>

<p>We do not see that this is an improvement over simply allowing the space, and the alternative (to have each 000 as a 
separate token and use <code>goeswith</code> or <code>mwe</code>) is unwieldy and does not exactly fit, e.g. writing 100 000 is not an 
orthographic error, but rather orthographically normative, and neither is it a fixed expression.</p>

<p>The new tokenisation would be:</p>

<pre><code class="language-sdparse">de 8⎵500 à 20⎵000 euros \n from 8,500 to 20,000 euros
</code></pre>

<h4 id="spaces-in-normalising-abbreviations">Spaces in normalising abbreviations</h4>

{'fraction_non_alphanumeric': 0.12746363714605388, 'fraction_numerical': 0.023754195713916862, 'mean_word_length': 4.222471910112359, 'pattern_counts': {'":': 0, '<': 231, '<?xml version=': 0, '>': 231, 'https://': 7, 'lorem ipsum': 0, 'www.': 4, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '511334', 'n_tokens_mistral': 3992, 'n_tokens_neox': 3682, 'n_words': 1275}