<commit_before># Lexos
# Summary
Lexos is an integrated workflow of tools to facilitate the
computational analyses of texts. 
Functionality provided includes
the ability to "scrub" texts (remove punctuation, lemmatize, consolidate characters, remove stopwords, etc), cut or segment texts, and a suite of options for analysis, including creating and downloading Document Term Matrices (DTM) of token counts
(both word- and character-ngrams), making dendrograms (cluster analysis), 
rolling-window analyses of word usage,
bubble visualizations, and word clouds. More functionality is being added 
on an ongoing basis.

## Audience
From the start, we have worked hard to build a workflow that is (i) easy to
use and (ii) serves as a platform for teaching and learning the science
and art of text mining, for example, we presently use Lexos in our interdisciplinary
undergraduate digital humanities course 
[Computing for Poets](http://"http://wheatoncollege.edu/lexomics/educational-material/").
If you are already cutting code in R to do your
text mining work, then Lexos is probably not for you. That said, we continued
to be impressed with Python as an alternative to R and our
own research group has made significant discoveries with Lexos 
[references](http://"http://wheatoncollege.edu/lexomics/educational-material/").


## System Architecture (in brief)
Lexos is written primarily in Python 2.7.3 using the Flask microframework,
based on Werkzeug and Jinja 2. A heavy dose of Javascript and CSS
is included on the front-end. We increasingly incorporate the wiz from D3.js
in our visualizations and the power
in the scilearn-kit modules for text and statistical processing. <commit_msg>README.md - added comment about non-Western languages<commit_after># Lexos
# Summary
Lexos is an integrated workflow of tools to facilitate the
computational analyses of texts. 
Functionality provided includes
the ability to "scrub" texts (remove punctuation, lemmatize, consolidate characters, remove stopwords, etc), cut or segment texts, and a suite of options for analysis, including creating and downloading Document Term Matrices (DTM) of token counts
(both word- and character-ngrams), making dendrograms (cluster analysis), 
rolling-window analyses of word usage,
bubble visualizations, and word clouds. More functionality is being added 
on an ongoing basis.

## Audience
From the start, we have worked hard to build a workflow that is (i) easy to
use, (ii) accepts texts in most languages, including non-Western character sets, and  
(iii) serves as a platform for teaching and learning the science
and art of text mining, for example, we presently use Lexos in our interdisciplinary
undergraduate digital humanities course 
[Computing for Poets](http://"http://wheatoncollege.edu/lexomics/educational-material/").
If you are already cutting code in R to do your
text mining work, then Lexos is probably not for you. That said, we continued
to be impressed with Python as an alternative to R and our
own research group has made significant discoveries with Lexos 
[references](http://"http://wheatoncollege.edu/lexomics/educational-material/").


## System Architecture (in brief)
Lexos is written primarily in Python 2.7.3 using the Flask microframework,
based on Werkzeug and Jinja 2. A heavy dose of Javascript and CSS
is included on the front-end. We increasingly incorporate the wiz from D3.js
in our visualizations and the power
in the scilearn-kit modules for text and statistical processing. 
{'fraction_non_alphanumeric': 0.055714285714285716, 'fraction_numerical': 0.002857142857142857, 'mean_word_length': 4.984615384615385, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6754414', 'n_tokens_mistral': 908, 'n_tokens_neox': 865, 'n_words': 485}