<jupyter_start><jupyter_text>Last updated on 01-26-2018.Written by <NAME>Email: [<EMAIL>](mailto:<EMAIL>) --- Example usage of `Preprocessor` class.<jupyter_code>import tm_preprocessor
tm_preprocessor.__version__<jupyter_output><empty_output><jupyter_text>Load example data Example from https://radimrehurek.com/gensim/tut1.htmlcorpus-formats, with some minor modifications.<jupyter_code>documents = ["Human machine interface for lab abc computer applications",
             "A survey of user opinion of computer system response time 2017 [\t]",
             "He went to the gym and swam.",  
             "The EPS user interface management system",
             "System and human system engineering testing of EPS",
             "Relation of user perceived response time to error measurement",
             "The generation of random binary unordered trees",
             "The intersection graph of paths in trees",
             "Graph minors IV Widths of trees, and well quasi ordering",
             "Graph minors: A survey"]<jupyter_output><empty_output><jupyter_text>Create a `Preprocessor` object<jupyter_code>from tm_preprocessor import Preprocessor
help(Preprocessor)
preprocessor = Preprocessor(documents)<jupyter_output><empty_output><jupyter_text>States of the object `preprocessor`  `documents` that store the original documents<jupyter_code>preprocessor.documents<jupyter_output><empty_output><jupyter_text>`punctuations` that are to be removed. Here shows the default values.<jupyter_code>preprocessor.punctuations<jupyter_output><empty_output><jupyter_text>`stopwords` Here shows the default values. Note that one can add more using `preprocessor.add_stopwords` method.<jupyter_code>preprocessor.stopwords<jupyter_output><empty_output><jupyter_text>`corpus` and `vocabulary` These are `None` for vocabulary at the moment because no processing has been done at this moment.<jupyter_code>preprocessor.corpus
preprocessor.vocabulary<jupyter_output><empty_output><jupyter_text>Methods of `preprocessor`<jupyter_code>help(preprocessor.add_stopwords)
help(preprocessor.remove_stopwords)
help(preprocessor.normalize)
help(preprocessor.get_word_ranking)<jupyter_output>Help on method get_word_ranking in module tm_preprocessor.preprocessor:

get_word_ranking() method of tm_preprocessor.preprocessor.Preprocessor instance
    Get the ranking of words (tokens). Note that this should be done a
    
    Returns
    -------
    pd.DataFrame
        Sorted dataframe with columns `word` and corresponding `frequency`.<jupyter_text>---  Preprocessing  Remove punctuations and digits<jupyter_code>preprocessor.remove_digits_punctuactions()
preprocessor.corpus<jupyter_output><empty_output><jupyter_text>Remove stopwords<jupyter_code>preprocessor.remove_stopwords()
preprocessor.corpus<jupyter_output><empty_output><jupyter_text>Tokenize Set `min_freq` to 0.1, `max_freq` to 0.9, and `min_length` to 2<jupyter_code>min_freq = 0
max_freq = 1
min_len = 2<jupyter_output><empty_output><jupyter_text>Use lemmatization.<jupyter_code>import nltk
lemmatizer = nltk.WordNetLemmatizer()
preprocessor.normalize(lemmatizer, min_freq, max_freq, min_len)
preprocessor.corpus<jupyter_output><empty_output><jupyter_text>Serialize the corpus  Save as `Market Matrix format` See http://math.nist.gov/MatrixMarket/formats.html<jupyter_code>preprocessor.serialize(format_='MmCorpus', path='~/Desktop')<jupyter_output><empty_output><jupyter_text>---  Frequencies of words/tokens<jupyter_code>preprocessor.get_word_ranking()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08972544579677329, 'fraction_numerical': 0.005943956977073309, 'mean_word_length': 6.301652892561983, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 59, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '296855', 'license': 'MIT', 'n_tokens_mistral': 1125, 'n_tokens_neox': 1084, 'n_words': 249, 'path': 'quick-start.ipynb', 'repo_name': 'zhiyzuo/python-topic-model-preprocessor', 'size': '3558'}