<commit_before>---
layout: base
title:  'Tokenization'
permalink: zh/overview/tokenization.html
udver: '2'
---

# Tokenization

## UD Chinese

[Description not currently available]

## UD Chinese-HK

This treebank (to be released) follows the <a href="http://repository.upenn.edu/cgi/viewcontent.cgi?article=1038&context=ircs_reports">segmentation guidelines</a> for the <a href="https://catalog.ldc.upenn.edu/ldc2016t13">Chinese Treebank</a> with two notable exceptions: (a) V-R resultative compounds (see <a href="http://universaldependencies.org/zh/dep/compound-vv.html">`compound:vv`</a>) and (b) V-得-V and V-不-V compounds (also known as "potential complement" in the literature; see <a href="http://universaldependencies.org/zh/dep/compound-dir.html">`compound:dir`</a> and <a href="http://universaldependencies.org/zh/dep/compound-vv.html">`compound:vv`</a>). While the Chinese Treebank treats them as a single token or as separated tokens depending on varying factors (including semantic compositionality and multisyllabicity), UD Chinese-HK separates all these compounds into separate tokens without exception.<commit_msg>Update tokenization.md<commit_after>---
layout: base
title:  'Tokenization'
permalink: zh/overview/tokenization.html
udver: '2'
---

# Tokenization

## UD Chinese

[Description not currently available]

## UD Chinese-HK

This treebank (to be released) follows the <a href="http://repository.upenn.edu/cgi/viewcontent.cgi?article=1038&context=ircs_reports">segmentation guidelines</a> for the <a href="https://catalog.ldc.upenn.edu/ldc2016t13">Chinese Treebank</a> with two notable exceptions: (a) V-R resultative compounds (see <a href="http://universaldependencies.org/zh/dep/compound-vv.html">`compound:vv`</a>) and (b) V-得-V and V-不-V compounds (also known as "potential complement" in the literature; see <a href="http://universaldependencies.org/zh/dep/compound-dir.html">`compound:dir`</a> and <a href="http://universaldependencies.org/zh/dep/compound-vv.html">`compound:vv`</a>). For both cases, while the Chinese Treebank treats each compound as a single token or as separated tokens depending on varying factors including semantic compositionality and multisyllabicity, UD Chinese-HK separates them all into separate tokens without exception.
{'fraction_non_alphanumeric': 0.1297676457693994, 'fraction_numerical': 0.009644892590968873, 'mean_word_length': 5.711764705882353, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7121602', 'n_tokens_mistral': 730, 'n_tokens_neox': 694, 'n_words': 199}