<commit_before>node-twokenize
==============

Node.js port and update of Jason Baldrige's Twokenize with layers for hashtag, @mention, email and urls.

Twokenize was written originally by <NAME>' Connor's twokenizer.py.
This is a port of a port (scala version to be exact) (meta! :P) created by
<NAME>. Additional modifications were made to fix some problems
with edge punctuation and handling other various entities like hashtags,
@username and emails. 

This is not fully intended for production use as there are plenty of other
useful tokenizers and nlp libraries that do this sort of thing better.
Though it does give some insight into some nice regex patterns for
handling various types of common entities and a nice overview on
manual tokenization.

INSTALL
-------

    npm install twokenize


LICENSE
--------
apache license 2.0: 
http://www.apache.org/licenses/LICENSE-2.0.html
Copyright <NAME>, <NAME>, and <NAME>, 2009-2010.


USAGE
-----

    var tokens = require('twokenize').tokenize('rt @username: this is some
    neat stuff #awesome google.com/chrome.');

    console.log(tokens);
<commit_msg>Removing extraneous blurb from readme (see included file in repo)<commit_after>node-twokenize
==============

Node.js port and update of Jason Baldrige's Twokenize with layers for hashtag, @mention, email and urls.

Twokenize was written originally by <NAME>' Connor's twokenizer.py.
This is a port of a port (scala version to be exact) (meta! :P) created by
<NAME>. Additional modifications were made to fix some problems
with edge punctuation and handling other various entities like hashtags,
@username and emails. 

This is not fully intended for production use as there are plenty of other
useful tokenizers and nlp libraries that do this sort of thing better.
Though it does give some insight into some nice regex patterns for
handling various types of common entities and a nice overview on
manual tokenization.

INSTALL
-------

    npm install twokenize


USAGE
-----

    var tokens = require('twokenize').tokenize('rt @username: this is some
    neat stuff #awesome google.com/chrome.');

    console.log(tokens);

{'fraction_non_alphanumeric': 0.08758782201405152, 'fraction_numerical': 0.005620608899297424, 'mean_word_length': 4.097852028639618, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '995563', 'n_tokens_mistral': 598, 'n_tokens_neox': 557, 'n_words': 282}