<commit_before>---
layout: base
title:  'Uppsala Group on Tokenisation'
---

# Uppsala Group on Tokenisation

_(<NAME>, <NAME>, <NAME>, <NAME>, <NAME>)_


# >1 token is 1 word

Here we define a token as a space delimited sequence of characters.

In CoNLL-U format, there is a restriction that "Fields must not contain space characters.".[1](http://universaldependencies.github.io/docs/format.html) This is problematic for a number of languages, as a word may consist of more than one token.

## Outcome

We came to the conclusion that the CoNLL-U format should allow multi-token words. The options were to use space itself,
another spacing character (e.g. ZWNJ) or to use some punctuation character (such as #, _, ⎵, etc.) The opinion of the group
was that as CoNLL-U format is tab-separated, it would not be problematic to allow spaces.

## Linguistic examples

### Vietnamese

"As a result of influence from the Chinese writing system, each syllable in Vietnamese is written separately as if it were a word. In the past, syllables in multisyllabic words were concatenated with hyphens, but this practice had died out, and hyphenation is now reserved for foreign borrowings."[2](https://en.wikipedia.org/wiki/Vietnamese_alphabet#Structure)

    Đảng  Cộng⎵sản  Việt⎵Nam
    Party Communist Vietnam

    Danh⎵sách      quốc⎵gia  xã⎵hội⎵chủ⎵nghĩa
    List.of.names  state     socialist

In a wordlist for Vietnamese extracted from Wiktionary,[3](https://svn.code.sf.net/p/apertium/svn/incubator/apertium-vie/dev/vie.speling.txt) over a third of headwords were multisyllabic.

### Tuvan

    Мен келген мен.
    I   come.PAST SG1

    Ол келген.
    He come.PAST SG3

    Мен келген-дыр     мен.
    I   come.PAST.EVID SG1

### Turkish

     Eve   geldi      misin?
     Home  come.PAST  QST.SG2 ?

# 1 token is >1 word


## Question word

   (kaz) Нан бар ма?
   (kir) Нан барбы?
   (tat) Ипи бармы?
   (tyv) Хлеб бар бе? 
   (tur) Ekmek var mı?
   (chv) Çăкăр пур-и?

   "Bread existing [is] QST?", "Is there bread?"

## Copula

   

## Productive "derivations" (-DAGI, -NIKI, -LIK, -LI, -SIZ)



## Causative

<commit_msg>updates<commit_after>---
layout: base
title:  'Uppsala Group on Tokenisation'
---

# Uppsala Group on Tokenisation

_(<NAME>, <NAME>, <NAME>, <NAME>, <NAME>)_


# >1 token is 1 word

Here we define a token as a space delimited sequence of characters.

In CoNLL-U format, there is a restriction that "Fields must not contain space characters.".[1](http://universaldependencies.github.io/docs/format.html) This is problematic for a number of languages, as a word may consist of more than one token.

## Outcome

We came to the conclusion that the CoNLL-U format should allow multi-token words. The options were to use space itself,
another spacing character (e.g. ZWNJ) or to use some punctuation character (such as #, _, ⎵, etc.) The opinion of the group
was that as CoNLL-U format is tab-separated, it would not be problematic to allow spaces.

## Linguistic examples

### Vietnamese

"As a result of influence from the Chinese writing system, each syllable in Vietnamese is written separately as if it were a word. In the past, syllables in multisyllabic words were concatenated with hyphens, but this practice had died out, and hyphenation is now reserved for foreign borrowings."[2](https://en.wikipedia.org/wiki/Vietnamese_alphabet#Structure)

    Đảng  Cộng⎵sản  Việt⎵Nam
    Party Communist Vietnam

    Danh⎵sách      quốc⎵gia  xã⎵hội⎵chủ⎵nghĩa
    List.of.names  state     socialist

In a wordlist for Vietnamese extracted from Wiktionary,[3](https://svn.code.sf.net/p/apertium/svn/incubator/apertium-vie/dev/vie.speling.txt) over a third of headwords were multisyllabic.

### Tuvan

    Мен келген мен.
    I   come.PAST SG1

    Ол келген.
    He come.PAST SG3

    Мен келген-дыр     мен.
    I   come.PAST.EVID SG1

### Turkish

     Eve   geldi      misin?
     Home  come.PAST  QST.SG2 ?

# 1 token is >1 word


## Question word

    (kaz) Нан бар ма?
    (kir) Нан барбы?
    (tat) Ипи бармы?
    (tyv) Хлеб бар бе? 
    (tur) Ekmek var mı?
    (chv) Çăкăр пур-и?

    "Bread existing [is] QST?", "Is there bread?"

## Copula

    Onun tutkusu       spor   arabalardı.
   

## Productive "derivations" (-DAGI, -NIKI, -LIK, -LI, -SIZ)

    <NAME>avi  arabadakileri      gördum.
    I   blue  car.LOC.KI.PL.ACC  see.PAST.SG1

    Mavi arabadakiler        gazete         okuyor.
    Blue car.LOC.KI.PL.NOM   newspaper.ACC  read.PROG.3

    Ben  adamınkini      gördum.
    I    man.GEN.KI.ACC  see.PAST.SG1

    Бир  палаталы     парламент
    One  chamber.LI   parlament

    Ол   хабарсыз  кетеді.
    He   news.SIZ  vanished

    Ben  ikinci  evsizleri         gördum.
    I    second  house.SIZ.PL.ACC  see.PAST.SG1

The idea for these is to allow, for example:
 
    0     Ben             Ben      PRON     NUMBER=SG|PERSON=1|CASE=NOM
    1     mavi            mavi     ADJ      _
    2-3   arabadakileri   _        _        _
    2     _               araba    NOUN     CASE=LOC
    3     _               ki       X        NUMBER=PL|CASE=ACC
    4     gördum          gör      VERB     TENSE=PAST|NUMBER=SG|PERSON=1
    
or:
 
    0     Ben             Ben      PRON     NUMBER=SG|PERSON=1|CASE=NOM
    1     mavi            mavi     ADJ      _
    2-3   arabadakileri   _        _        _
    2     arabada         araba    NOUN     CASE=LOC
    3     kileri          ki       X        NUMBER=PL|CASE=ACC
    4     gördum          gör      VERB     TENSE=PAST|NUMBER=SG|PERSON=1

Depending on the language, recovering anything sensible for the sub-surface forms may be more or less difficult.

## Causative

    Babam           arabayı   Ali ustaya      yaptırmış
    Father.SG1      car.ACC   Ali master.DAT  fix.CAUS.EVID

The thoughts were for this construction to use a separate relation, for example `nmod:caus` or `nsubj:caus` for 
the causative subject (causee) of a causative verb. 


{'fraction_non_alphanumeric': 0.08155012636899747, 'fraction_numerical': 0.007582139848357203, 'mean_word_length': 2.4754098360655736, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 18, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4209853', 'n_tokens_mistral': 2195, 'n_tokens_neox': 2105, 'n_words': 695}