<jupyter_start><jupyter_text>Natural Language ProcessingLets understand the basic process of pattern recognition that is needed to work with text data. This approach can be used for a large number of basic operations on text data like - Document Classification (e.g. topic of an article)- Sequence to Sequence Learning (e.g. translations)- Sentiment AnalysisLets start with seeing how a text sequence can be encoded into numbers and processed to prepare for these tasks  TokenisationLets take the sentence - **The quick brown fox jumped over the lazy dog**We need to first break this sentence in to smaller constituents - called **tokens**. Now there are three ways of creating the tokens can happen:- **Individual character** - create tokens for each- **Individual word** - Create tokens for each word in the sentence- **N-gram** - Create tokens by taking n-grams words in the sentence  Create word tokens **Pre-processing - split, punctuation & case**There is some basic **pre-processing** that has been done in the process of creating word tokens- Split the sentence on **whitespace**- Filter **punctuations**- Change to **lower case** textAfter this pre-processing, we can get each token.<jupyter_code>import numpy as np
import pandas as pd
from keras.preprocessing.text import text_to_word_sequence
sentence = 'The quick brown fox jumped over the lazy dog.'
text_to_word_sequence(sentence)
import spacy
#! python -m spacy download en_core_web_sm
nlp = spacy.load('en_core_web_sm')
sentence = 'The quick brown fox jumped over the lazy dog'
doc = nlp(sentence)
for token in doc:
    print(token)<jupyter_output>The
quick
brown
fox
jumped
over
the
lazy
dog<jupyter_text>VectorisationOnce you have tokens, we need to find a way to represent them as vectors. Let's look at two traditional way of representing them as vectors- Frequency Based    - Binary    - Count     - tfidf    - Co-occurence (Skipgram)- Prediction Based    - Pre-trained Vectors    - Learning Vectors    - Learning vectors with the task  One-Hot Encoding<jupyter_code>from keras.preprocessing.text import one_hot, hashing_trick
# Given a size of vocabulary, do one-hot encoding
one_hot(sentence, n=10)
# Given a size of vocabulary, do hash encoding (to save space)
hashing_trick(sentence, n=100, hash_function="md5")<jupyter_output><empty_output><jupyter_text>** Using the Tokenizer API **<jupyter_code>from keras.preprocessing.text import Tokenizer
# Instantiate the Tokenizer
simple_tokenizer = Tokenizer()
# Fit the Tokenizer
simple_tokenizer.fit_on_texts([sentence])
# See the word vectors
simple_tokenizer.word_index
from helpers import get_word_vectors
get_word_vectors(simple_tokenizer)<jupyter_output><empty_output><jupyter_text>Normally we will be working with a set of text (like sentences), so it is better to use the tokenizer API<jupyter_code>sentences = ['The quick brown fox jumped over the lazy dog', 
             'The dog woke up lazily and barked at the fox',
             'The fox looked back and just ignored the dog']
# Instantiate and Fit
tokenizer = Tokenizer()
tokenizer.fit_on_texts(sentences)
get_word_vectors(tokenizer)
tokenizer.texts_to_sequences(sentences)
02

tokenizer.texts_to_matrix(sentences, mode="binary")
tokenizer.texts_to_matrix??
from helpers import get_sentence_vectors
def get_sentence_vectors(sentences, tokenizer, mode="binary"):
    matrix = tokenizer.texts_to_matrix(sentences, mode=mode)
    df = pd.DataFrame(matrix)
    df.drop(columns=0, inplace=True)
    df.columns = tokenizer.word_index
    return df
get_sentence_vectors(sentences, tokenizer, mode="tfidf")
mode = "tfidf"
alt.Chart(df).mark_rect().encode(
    x = "word",
    y = "sentence:N",
    color = "tfidf",
    tooltip = mode
).properties(
    width=700
).interactive()
df.head()
tokenizer.word_index
one_hot_results = tokenizer.texts_to_matrix(sentence, mode='binary')

word_index = tokenizer.word_index
print('Found %s unique tokens.' % len(word_index))<jupyter_output>Found 8 unique tokens.
{'fraction_non_alphanumeric': 0.07970832285642444, 'fraction_numerical': 0.003268795574553684, 'mean_word_length': 5.138888888888889, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '105451', 'license': 'MIT', 'n_tokens_mistral': 1208, 'n_tokens_neox': 1146, 'n_words': 438, 'path': 'Basics/03-NLP-Basics.ipynb', 'repo_name': 'Nayananga/recommendation', 'size': '3977'}