<commit_before># zh_tokenize

Tokenize Chinese using a Trie (prefix-search) based on work done by @kennycason

ZH gzipped file dictionary can be downloaded from:

https://www.mdbg.net/chinese/dictionary?page=cc-cedict

if this file is not in (do not gunzip)

./dict/cedict_1_0_ts_utf-8_mdbg.txt.gz

(dictionary license http://creativecommons.org/licenses/by-sa/3.0/)

export ZH_DICT=/fully/qualified/path/to/cedict-VERSION.gz

Algorithm:

Walked through sentences character-by-character observing if substrings exist in the dictionary. If so parse as a token. This is self correcting for words not in the dictionary. The output will be utf-8 encoded tokens space separated suitable for using in a search engine.<commit_msg>added example usage and badge<commit_after># zh_tokenize

[![Project status](https://img.shields.io/badge/Project%20Status-Complete-brightgreen.svg)](#status)


Tokenize Chinese using a Trie (prefix-search) based on work done by @kennycason

ZH gzipped file dictionary can be downloaded from:

https://www.mdbg.net/chinese/dictionary?page=cc-cedict

if this file is not in (do not gunzip)

./dict/cedict_1_0_ts_utf-8_mdbg.txt.gz

(dictionary license http://creativecommons.org/licenses/by-sa/3.0/)

export ZH_DICT=/fully/qualified/path/to/cedict-VERSION.gz

Algorithm:

Walk through sentences character-by-character checking if substring exist in the Trie. If so parse as a token. This is self correcting for words not in the dictionary. The output will be utf-8 encoded tokens space separated suitable for using in a search engine.

Example Usage:

```
from tokenizer import ChineseWordTokenizer

tokens = tokenizer.tokenize(u"国家都有自己的政府。政府是税收的主体，可以实现福利的合理利用。")
print len(tokens), tokenizer.printable(tokens)
```

Produces:

```
17 [家都] [有] [自] [己的] [政] [府。] [府是] [税] [收的] [主] [体，] [以实] [现福] [利的] [合] [理利] [用。]
```
{'fraction_non_alphanumeric': 0.10724006532389765, 'fraction_numerical': 0.008709853021230268, 'mean_word_length': 4.853503184713376, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7070703', 'n_tokens_mistral': 656, 'n_tokens_neox': 635, 'n_words': 199}