<jupyter_start><jupyter_text>NLP Tools tutorial The **nlp_tools** subpackage offers classic NLP tools implemented as classes that will be used to preprocess an already cleaned text :- a **Tokenizer class** : to split a sentence-like string into a list of sub-strings (tokens).- a **Phraser class** : to transform common multi-word expressions into single elements (*new york* becomes *new_york*)- an **Embedding class** : to represent of words in a lower dimensional vector space.- a **Stemmer class**: to reduce words to their word stem.  Load data<jupyter_code>from melusine.data.data_loader import load_email_data
df_emails = load_email_data(type="preprocessed")<jupyter_output><empty_output><jupyter_text>The Tokenizer classThe Tokenizer class splits a sentence-like string into a list of sub-strings (tokens).  The arguments of a Tokenizer object are just the input_columns and output_columns.<jupyter_code>from melusine.nlp_tools.tokenizer import Tokenizer
tokenizer = Tokenizer (input_column='clean_body', output_column="tokens")<jupyter_output><empty_output><jupyter_text>Use the **fit_transform** method on a dataframe to create a new ***tokens* column**<jupyter_code>df_emails = tokenizer.fit_transform(df_emails)
print("Base text")
print(df_emails.clean_body[1])
print("\nTokenized text")
print(df_emails.tokens[1])<jupyter_output>Base text
je vous informe que la nouvelle immatriculation est enfin faite. je vous prie de trouver donc la carte grise ainsi que la nouvelle immatriculation. je vous demanderai de faire les changements necessaires concernant lassurance.

Tokenized text
['informe', 'nouvelle', 'immatriculation', 'enfin', 'faite', 'prie', 'trouver', 'donc', 'carte', 'grise', 'ainsi', 'nouvelle', 'immatriculation', 'demanderai', 'faire', 'les', 'changements', 'necessaires', 'concernant', 'lassurance']<jupyter_text>Load / Save a tokenizer<jupyter_code>with TemporaryDirectory() as tmpdir:
    path = f"{tmpdir}/tokenizer.pkl"
    _ = joblib.dump(tokenizer, path, compress=True)
    tokenizer_reload = joblib.load(path)
df_emails = tokenizer_reload.fit_transform(df_emails)
print(df_emails.tokens[1])<jupyter_output><empty_output><jupyter_text>The Phraser class The Phraser class transforms common multi-word expressions into single elements: for example *new york* becomes *new_york*. The arguments of a Phraser object are:- **input_column :** the name of the column of the dataframe that will be used as input for the training of the Phraser.- **common_terms :** list of stopwords to be ignored. The default list is defined in the *conf.json* file.- **threshold :** threshold to select collocations.- **min_count :** minimum count of word to be selected as collocation.<jupyter_code>from melusine.nlp_tools.phraser import Phraser

phraser = Phraser(
    input_column='tokens',
    output_column='phrased_tokens',
    threshold=5,
    min_count=2
)<jupyter_output><empty_output><jupyter_text>Training a phraser The input dataframe must contain a column with a clean text : **a sentence-like string with only lowcase letters and no accents**.<jupyter_code>_ = phraser.fit(df_emails)
df_emails = phraser.transform(df_emails)
print(df_emails['phrased_tokens'].iloc[3])<jupyter_output>['fais', 'suite_a', 'mail', 'envoye', 'bulletin_salaire', 'courrier', 'semblerait', 'receptionne', 'trouverez', 'ci-joint', 'bulletin_salaire']<jupyter_text>Expected result : You should see phrased tokens such as  "bulletin" + "salaire" = "bulletin_salaire"  Load/Save a phraser<jupyter_code>with TemporaryDirectory() as tmpdir:
    path = f"{tmpdir}/phraser.pkl"
    _ = joblib.dump(phraser, path, compress=True)
    phraser_reload = joblib.load(path)
    
print(phraser_reload.transform(df_emails)['phrased_tokens'].iloc[3])<jupyter_output>['fais', 'suite_a', 'mail', 'envoye', 'bulletin_salaire', 'courrier', 'semblerait', 'receptionne', 'trouverez', 'ci-joint', 'bulletin_salaire']<jupyter_text>The Embedding class Word embeddings are abstract representations of words in a lower dimensional vector space. One of the advantages of word embeddings is thus to save computational cost. The Melusine Embedding class uses a **Word2Vec** model. The trained Embedding object will be used in the Models subpackage to train a Neural Network to classify emails. The arguments of an Embedding object are :- **input_column :** the name of the column used as an input for the training.- **workers :** the number of cores used for computation. Default value, 40.- **seed :** seed for the embedding model,- **iter :** number of iterations for the training,- **size :** dimension of the embeddings- **window :** - **min_count :** minimum number of occurences for a word to be taken into account.<jupyter_code>from melusine.nlp_tools.embedding import Embedding

embedding = Embedding(
    tokens_column='tokens',
    size=300,
    workers=4,
    min_count=3
)<jupyter_output><empty_output><jupyter_text>Training embeddings<jupyter_code>embedding.train(df_emails)
embedding.embedding.most_similar("vehicule")<jupyter_output><empty_output><jupyter_text>Warning : The embedding is trained on a very small dataset so the results here are irrelevant  The stemmer class The Stemmer class reduces words (tokens) from a list to their word stem.  The arguments of a Stemmer object are:- **input_column :** the name of the column of the dataframe that will be used as input for the reduction of the Stemmer.- **output_column :** the name of the column of the dataframe that will be used as output for the reduction of the Stemmer.- **language :** language used to stem words, default "french".<jupyter_code>from melusine.nlp_tools.stemmer import Stemmer

stemmer = Stemmer()<jupyter_output><empty_output><jupyter_text>The Stemmer doesn't need any training. You can apply it directly on your dataset.<jupyter_code>df_emails = stemmer.transform(df_emails)
print(df_emails['stemmed_tokens'].iloc[3])<jupyter_output>['fais', 'suit', 'a', 'mail', 'envoy', 'bulletin', 'salair', 'courri', 'sembl', 'reception', 'trouv', 'ci-joint', 'bulletin', 'salair']<jupyter_text>The lemmatizer class The Lemmatizer class changes words from a string by the base or dictionary form of a word, which is known as the lemma.The arguments of a Lemmatizer object are:- **in_col :** the name of the column of the dataframe that will be used as input for lemmatization.- **out_col :** the name of the column of the dataframe that will be used as output for lemmatization.- **engine :** lemmatization engine to load. Choices are 'spacy' and 'Lefff'.- **engine_conf:** Spacy model to load under the hood. For french, choices are 'fr_core_news_sm', 'fr_core_news_md',     'fr_core_news_lg'. If you get the following error:MaxRetryError: HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: /sammous/spacy-lefff-model/releases/latest/download/model.tar.gz (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out'))This may due to the use of proxy.                   Please download locally the model: https://github.com/sammous/spacy-lefff-model/releases/latest/download/model.tar.gz To do so: Go to the repository: /spacy_lefff/data/taggerDownload and unzip the model with        >>wget https://github.com/sammous/spacy-lefff-model/releases/latest/download/model.tar.gz        >>tar xvfz model.tar.gz<jupyter_code>from melusine.nlp_tools.lemmatizer import Lemmatizer
spacy_lemma_sm = Lemmatizer('clean_body', 'lemma_spacy_sm', engine='spacy', engine_conf='fr_core_news_sm')
lefff_lemma = Lemmatizer('clean_body', 'lemma_lefff', engine='Lefff', engine_conf='fr_core_news_sm')
df_emails = spacy_lemma_sm.fit_transform(df_emails)
print(df_emails['lemma_spacy_sm'].iloc[3])
df_emails = lefff_lemma.fit_transform(df_emails)
print(df_emails['lemma_lefff'].iloc[3])<jupyter_output>je faire suite avoir son mail . j' avoir envoye son bulletin de salaire par courrier mais il sembler que vous ne l' ayez pas receptionne . vous trouver ci - joindre son bulletin de salaire .
{'fraction_non_alphanumeric': 0.10481550969355848, 'fraction_numerical': 0.00300187617260788, 'mean_word_length': 6.032541776605101, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 48, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '511600', 'license': 'Apache-2.0', 'n_tokens_mistral': 2552, 'n_tokens_neox': 2507, 'n_words': 817, 'path': 'tutorial/tutorial04_nlp_tools.ipynb', 'repo_name': 'DataFactory-Verlingue/dafa_melusine', 'size': '7995'}