<commit_before><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html lang="en-GB" xml:lang="en-GB" xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
    <title>Working Group on Data Splits</title>
    <link rel="root" href=""/> <!-- for JS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
    <link rel="stylesheet" type="text/css" href="../css/jquery-ui-redmond.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style-vis.css"/>
    <link rel="stylesheet" type="text/css" href="../css/hint.css"/>
    <script type="text/javascript" src="../lib/ext/head.load.min.js"></script>
    <script type="text/javascript" src="../lib/ext/jquery.timeago.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/3.2.2/anchor.min.js"></script>
    <script>document.addEventListener("DOMContentLoaded", function(event) {anchors.add();});</script>
    <!-- Set up this custom Google search at https://cse.google.com/cse/business/settings?cx=001145188882102106025:dl1mehhcgbo -->
    <script>
      (function() {
        var cx = '001145188882102106025:dl1mehhcgbo';
        var gcse = document.createElement('script');
        gcse.type = 'text/javascript';
        gcse.async = true;
        gcse.src = 'https://cse.google.com/cse.js?cx=' + cx;
        var s = document.getElementsByTagName('script')[0];
        s.parentNode.insertBefore(gcse, s);
      })();
    </script>
<!--     <link rel="shortcut icon" href="favicon.ico"/> -->


  </head>
  <body>
    <div id="main" class="center">

      <div id="hp-header">
        <table width="100%"><tr><td width="50%">
          <span class="header-text"><a href="http://universaldependencies.org/#language-">home</a></span>

          <span class="header-text"><a href="https://github.com/universaldependencies/docs/edit/pages-source/workgroups/split.md" target="#">edit page</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/issues">issue tracker</a></span>
        </td><td>
          <gcse:search></gcse:search>
        </td></tr></table>
      </div>

      <hr/>

      
      <div class="v2complete">
	This page pertains to UD version 2.
      </div>
      

      <div id="content">
	<noscript>
	  <div id="noscript">
	    It appears that you have Javascript disabled.
	    Please consider enabling Javascript for this page to see the visualizations.
	  </div>
	</noscript>

          <h1 id="working-group-on-data-splits">Working Group on Data Splits</h1>

<p>Until the CoNLL 2017 shared task, UD did not have clear guidelines how to
split the data to training, development and test parts. The shared task
guidelines were only focused on participation in the task, and were changed
on the fly. This is an attempt to put together a set of recommendations that
will hopefully work for treebanks of various sizes and will be more stable
over time.</p>

<p><strong>Note that at present this is just a proposal under discussion and it may
change without warning! When the discussion is over, the guidelines will be
moved from here to the how-to-contribute pages.</strong></p>

<p>These guidelines are strong recommendations rather than strict rules. In some
cases it may not be even possible to meet all of them with a given dataset.</p>

<p>The general underlying idea is that we do not want to ban tiny datasets from
being released, but we only want to distinguish training and test sets if
the test set will be large enough (the 10K-word threshold was set in the
CoNLL 2017 shared task) and if the data that is left for training is at least
as large as the test set. Making an official split is useful for comparison
of experiments evaluated on the treebank, but with small treebanks people
should do ten-fold cross-validation anyway.</p>

<p>There is one exception to that general idea. If this is the only UD treebank
in the given language, we may set aside a small sample and call it “train” (or “sample”?)
This may be useful if the treebank is used in a shared task where the systems
are supposed to use cross-lingual projection techniques and cannot access the test data,
but the developers should have access at least to a sample of the language
so they can see the annotation, language properties etc.
In other situations, users should still use the entire data for training and
testing via cross-validation.
Providing the sample is completely optional.
Note that it does not make sense to set aside a sample
if a larger treebank of the same language exists and has training data.</p>

<ol>
  <li>If you have less than 20K words:
    <ul>
      <li>Option A: Keep everything as test data. Users will have to do 10-fold cross-validation if they want to train on it.</li>
      <li>Option B: If there are no larger treebanks of this language, set aside a small sample (20 to 50 sentences) and call it “train” (or “sample”?) Consider translating and annotating the 20 examples from the <a href="https://github.com/UniversalDependencies/cairo/blob/master/translations.txt">Cairo Cicling Corpus</a> (CCC) and providing them as the sample.</li>
    </ul><commit_msg>autogenerated by https://github.com/spyysalo/jekyll-hook<commit_after><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html lang="en-GB" xml:lang="en-GB" xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
    <title>Working Group on Data Splits</title>
    <link rel="root" href=""/> <!-- for JS -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css">
    <link rel="stylesheet" type="text/css" href="../css/jquery-ui-redmond.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style.css"/>
    <link rel="stylesheet" type="text/css" href="../css/style-vis.css"/>
    <link rel="stylesheet" type="text/css" href="../css/hint.css"/>
    <script type="text/javascript" src="../lib/ext/head.load.min.js"></script>
    <script type="text/javascript" src="../lib/ext/jquery.timeago.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/3.2.2/anchor.min.js"></script>
    <script>document.addEventListener("DOMContentLoaded", function(event) {anchors.add();});</script>
    <!-- Set up this custom Google search at https://cse.google.com/cse/business/settings?cx=001145188882102106025:dl1mehhcgbo -->
    <script>
      (function() {
        var cx = '001145188882102106025:dl1mehhcgbo';
        var gcse = document.createElement('script');
        gcse.type = 'text/javascript';
        gcse.async = true;
        gcse.src = 'https://cse.google.com/cse.js?cx=' + cx;
        var s = document.getElementsByTagName('script')[0];
        s.parentNode.insertBefore(gcse, s);
      })();
    </script>
<!--     <link rel="shortcut icon" href="favicon.ico"/> -->


  </head>
  <body>
    <div id="main" class="center">

      <div id="hp-header">
        <table width="100%"><tr><td width="50%">
          <span class="header-text"><a href="http://universaldependencies.org/#language-">home</a></span>

          <span class="header-text"><a href="https://github.com/universaldependencies/docs/edit/pages-source/workgroups/split.md" target="#">edit page</a></span>
          <span class="header-text"><a href="https://github.com/universaldependencies/docs/issues">issue tracker</a></span>
        </td><td>
          <gcse:search></gcse:search>
        </td></tr></table>
      </div>

      <hr/>

      
      <div class="v2complete">
	This page pertains to UD version 2.
      </div>
      

      <div id="content">
	<noscript>
	  <div id="noscript">
	    It appears that you have Javascript disabled.
	    Please consider enabling Javascript for this page to see the visualizations.
	  </div>
	</noscript>

          <h1 id="working-group-on-data-splits">Working Group on Data Splits</h1>

<p>Until the CoNLL 2017 shared task, UD did not have clear guidelines how to
split the data to training, development and test parts. The shared task
guidelines were only focused on participation in the task, and were changed
on the fly. This is an attempt to put together a set of recommendations that
will hopefully work for treebanks of various sizes and will be more stable
over time.</p>

<p><strong>Note that at present this is just a proposal under discussion and it may
change without warning! When the discussion is over, the guidelines will be
moved from here to the how-to-contribute pages.</strong></p>

<p>These guidelines are strong recommendations rather than strict rules. In some
cases it may not be even possible to meet all of them with a given dataset.</p>

<p>The general underlying idea is that we do not want to ban tiny datasets from
being released, but we only want to distinguish training and test sets if
the test set will be large enough (the 10K-word threshold was set in the
CoNLL 2017 shared task) and if the data that is left for training is at least
as large as the test set. Making an official split is useful for comparison
of experiments evaluated on the treebank, but with small treebanks people
should do ten-fold cross-validation anyway.</p>

<p>There is one exception to that general idea. If this is the only UD treebank
in the given language, we may set aside a small sample and call it “train” (or “sample”?)
This may be useful if the treebank is used in a shared task where the systems
are supposed to use cross-lingual projection techniques and cannot access the test data,
but the developers should have access at least to a sample of the language
so they can see the annotation, language properties etc.
In other situations, users should still use the entire data for training and
testing via cross-validation.
Providing the sample is completely optional.
Note that it does not make sense to set aside a sample
if a larger treebank of the same language exists and has training data.</p>

<ol>
  <li>If you have less than 20K words:
    <ul>
      <li>Option A: Keep everything as test data. Users will have to do 10-fold cross-validation if they want to train on it.</li>
      <li>Option B: If there are no larger treebanks of this language, keep almost everything as test data but set aside a small sample (20 to 50 sentences) and call it “train” (or “sample”?) Consider translating and annotating the 20 examples from the <a href="https://github.com/UniversalDependencies/cairo/blob/master/translations.txt">Cairo Cicling Corpus</a> (CCC) and providing them as the sample.</li>
    </ul>
{'fraction_non_alphanumeric': 0.1256114443931703, 'fraction_numerical': 0.0169820027688048, 'mean_word_length': 3.626814688300598, 'pattern_counts': {'":': 0, '<': 173, '<?xml version=': 0, '>': 173, 'https://': 15, 'lorem ipsum': 0, 'www.': 4, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '622023', 'n_tokens_mistral': 3342, 'n_tokens_neox': 3138, 'n_words': 1174}