<commit_before># Deepcut

[![License](https://img.shields.io/badge/license-MIT-blue.svg?style=flat)](https://github.com/rkcosmos/deepcut/blob/master/LICENSE)

A Thai word tokenization library using Deep Neural Network.

## What's new?

* Deepcut JS, try tokenizing Thai text on browser [here](https://rkcosmos.github.io/deepcut/)
* `v0.6` Add stop words, updated weight with semi-supervised learning, custom dictionary
* `v0.5.2` Better weight matrix
* `v0.5.1` Faster tokenization by code refactorization from our new contributor: <NAME>

## Performance

The Convolutional Neural network is trained from 90 % of NECTEC's BEST corpus
(consists of 4 sections, article, news, novel and encyclopedia) and test on the rest 10 %.
It is a binary classification model trying to predict whether a character is the beginning of word or not.
The results calculated from only 'true' class are as follow

* f1 score:  98.1%
* precision score:  97.8%
* recall score:  98.5%

## Installation
<commit_msg>add architecture image to README (#52)<commit_after># Deepcut

[![License](https://img.shields.io/badge/license-MIT-blue.svg?style=flat)](https://github.com/rkcosmos/deepcut/blob/master/LICENSE)

A Thai word tokenization library using Deep Neural Network.

![](https://user-images.githubusercontent.com/1214890/58486992-14c1d880-8191-11e9-9122-8385750e06bd.png)

## What's new?

* Deepcut JS, try tokenizing Thai text on browser [here](https://rkcosmos.github.io/deepcut/)
* `v0.6` Add stop words, updated weight with semi-supervised learning, custom dictionary
* `v0.5.2` Better weight matrix
* `v0.5.1` Faster tokenization by code refactorization from our new contributor: <NAME>

## Performance

The Convolutional Neural network is trained from 90 % of NECTEC's BEST corpus
(consists of 4 sections, article, news, novel and encyclopedia) and test on the rest 10 %.
It is a binary classification model trying to predict whether a character is the beginning of word or not.
The results calculated from only 'true' class are as follow

* f1 score:  98.1%
* precision score:  97.8%
* recall score:  98.5%

## Installation

{'fraction_non_alphanumeric': 0.10516342965419233, 'fraction_numerical': 0.04216011369019422, 'mean_word_length': 4.723577235772358, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5463082', 'n_tokens_mistral': 715, 'n_tokens_neox': 641, 'n_words': 238}