<jupyter_start><jupyter_text>Text Mining of BBC News Data Part 2: Bag-of-Words Text Vectorization  The Analyzer Object<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
count_vectorizer = CountVectorizer()
count_vectorizer
test_sentence = "C'est l'été au Brésil!"
word_analyzer = CountVectorizer().build_analyzer()
word_analyzer(test_sentence)
word_analyzer = CountVectorizer(strip_accents="unicode", ngram_range=(2, 2)).build_analyzer()
word_analyzer(test_sentence)
word_analyzer = CountVectorizer(strip_accents="unicode", ngram_range=(1, 2)).build_analyzer()
word_analyzer(test_sentence)
word_analyzer = CountVectorizer(ngram_range=(2, 2)).build_analyzer()
word_analyzer(test_sentence)<jupyter_output><empty_output><jupyter_text>Analyzer = Preprocessor + Tokenizer (+ Token Filtering) (+ n-grams extraction)<jupyter_code>vectorizer = CountVectorizer(strip_accents="unicode", lowercase=True)
test_sentence = "C'est l'été au Brésil!"<jupyter_output><empty_output><jupyter_text>Exercises- Type `vectorizer.build_` to see the list of methods of the vectorizer object;- Use the vectorizer to build a preprocessor object and apply it to the test sentence: which transformations do you notice?- Use the vectorizer to build a tokenizer object and apply it to the preprocessed test sentence from the previous step;- Compare the results of the previous two steps with the output of the analyzer applied to the original test sentence.<jupyter_code># %load notebook_solutions/build_preprocessor.py
# %load notebook_solutions/build_tokenizer.py<jupyter_output><empty_output><jupyter_text>Vectorization of a Full Dataset<jupyter_code>from pathlib import Path


bbc_folder_path = Path("bbc")
text_filepaths = sorted(bbc_folder_path.glob("*/*.txt"))<jupyter_output><empty_output><jupyter_text>Instead of decoding the text manually as we did before, we will path the filenames directly to the vectorizer and let it decode using the encoding of our choice and ignore decoding errors (as we did previously):<jupyter_code>%%time

vectorizer = CountVectorizer(encoding="utf-8", input="filename",
                             decode_error="ignore")

vectorizer.fit(text_filepaths)
type(vectorizer.vocabulary_)
len(vectorizer.vocabulary_)
vocabulary_items = sorted(vectorizer.vocabulary_.items())
vocabulary_items[:10]
vocabulary_items[5000:5010]
vocabulary_items[-10:]
vectorized_docs = vectorizer.transform(text_filepaths)
vectorized_docs
first_doc = vectorized_docs[0, :]
first_doc
first_doc.toarray()<jupyter_output><empty_output><jupyter_text>**Question**: why does the vectorizer return a sparse matrix instead of a regular (dense) array?<jupyter_code>first_doc.toarray().shape<jupyter_output><empty_output><jupyter_text>Let's have a look at the indices of the non-zero values in the vectorized first document:<jupyter_code>first_doc.indices<jupyter_output><empty_output><jupyter_text>Each dimension corresponds to a word in the vocabulary. We can retrieve the "feature name" for a specific dimension number as follows:<jupyter_code>feature_names = vectorizer.get_feature_names()
feature_names[:10]
feature_names[5360]<jupyter_output><empty_output><jupyter_text>It's possible to inverse a full vectorized document at once:<jupyter_code>vectorizer.inverse_transform(first_doc)<jupyter_output><empty_output><jupyter_text>**Exercise**Print the text of the original first document. And compare to the "inversed transformed" document (can you find the same words?): **Question**:- Why do you thing the result of the text vectorization procedure is called "Bag-of-Words" representation?- What as the main limitation of the "Bag-of-Words" representation?- Can you come up with a pair of sentence with completely different meanings but that would share the same Bag of Words vector?  TF-IDF Normalization- TF-IDF stands for **"Term Frequency" - "Inverse Document Frequency"**.- **Term Frequency** is the number of times a term or token appears in a given document;- **Document Frequency** is the number of times a term or token appears across all the documents of the corpus.**Note**: depending on the context, frequencies can either be:- **absolute** values (**integer counts**) or- **relative** values (**floating point numbers between 0 and 1** for **ratio** between two quantities).TF-IDF is a normalization procedures that transforms integer term frequency counts into a reweights term frequencies such that very frequent words (such as "the", "a", "and", "it"... in English) have lower importance than rarer words.The general idea, is that if we note:- $n_{t, d}$ the absolute term frequency (the number of times terms $t$ appears in document $d$);- $N_D$ the total number of documents in the dataset $D$;- $n_{t}$ the number of documents in dataset $D$ that contain the term $t$;The **term frequency** component can either be defined as:- $\mathrm{tf} (t,d) = n_{t, d}$or my taking a subliner function of the counts such as:- $\mathrm{tf} (t,d) = log(1 + n_{t, d})$Then the raw **inverse document frequency** is:- $\mathrm{idf} (t) = \frac{N}{n_{t}}$In practice one often takes the log of this quantity and define:- $\mathrm{idf} (t, D) = log(\frac{N_D}{n_{t}})$The final TF-IDF value is computed by taking the product of those two quantities:$$\mathrm{tfidf} (t,d,D) = \mathrm{tf} (t,d)\cdot \mathrm{idf} (t,D)$$There are many variants and the one implemented in scikit-learn is not necessarily the most standard (for historical reasons). The Wikipedia article has the list of variants:https://en.wikipedia.org/wiki/Tf–idfThe scikit-learn documentation has comprehensive information on the vectorizers of scikit-learn and the TF-IDF variant implemented there:https://scikit-learn.org/stable/modules/feature_extraction.html<jupyter_code>from sklearn.feature_extraction.text import TfidfTransformer


tfidf_transformer = TfidfTransformer()
tfidf_docs = tfidf_transformer.fit_transform(vectorized_docs)
tfidf_docs
first_doc.toarray()
first_tfidf_doc = tfidf_docs[0, :]
first_tfidf_doc.toarray()
tfidf_transformer.idf_<jupyter_output><empty_output><jupyter_text>**Exercises**:    Using the `vectorizer.get_feature_names()` vector and the `tfidf_transformer.idf_`, compute the list of the top 10 least and most informative words in the BBC corpus.Hint: you can put the two vectors in a `pandas.DataFrame` and use the [nlargest](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.nlargest.html) and [nsmallest](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.nsmallest.html) methods.<jupyter_code># %load notebook_solutions/top_idf.py<jupyter_output><empty_output><jupyter_text>**Questions**- Why do the top 10 IDF-weighted terms have the same IDF value in your opinion?- What is the problem with using such terms in a text classification scenario?- What do you think about the top 10 lowest? What could we do to improve the computational performance of a text classifier?  Vectorizing and Weighting at Once<jupyter_code>%%time
from sklearn.feature_extraction.text import TfidfVectorizer


tfidf_vectorizer = TfidfVectorizer(encoding="utf-8", input="filename",
                                   decode_error="ignore",
                                   min_df=5, max_df=0.8)

tfidf_docs_trimmed = tfidf_vectorizer.fit_transform(text_filepaths)
tfidf_docs_trimmed<jupyter_output><empty_output><jupyter_text>**Questions**:    - What do you notice about the shape of this new vectorized text corpus (compared to the previous version)?<jupyter_code>import pandas as pd


weighted_terms_trimmed = pd.DataFrame({
    "term": tfidf_vectorizer.get_feature_names(),
    "idf": tfidf_vectorizer.idf_})

print("Most 'informative' terms:")
print(weighted_terms_trimmed.nlargest(10, "idf"), end="\n\n")

print("Least informative terms:")
print(weighted_terms_trimmed.nsmallest(10, "idf"), end="\n\n")<jupyter_output><empty_output><jupyter_text>Document Similarities in TF-IDF Space A common way to measure the semantic similarity of two documents is to compute the cosine similarity of their TF-IDF vectors:$$cosine(x_1, x_2) = \frac{x_1 \cdot x_2}{||x_1|| ||x_2||}$$**Question**- Show that if two vectors have unit norms, then maximizing their cosine similarity is equivalent to minimizing their euclidean distances.<jupyter_code>vectorized_docs.multiply(vectorized_docs)
import numpy as np


def sparse_row_norms(data):
    return np.asarray((data.multiply(data)).sum(axis=1)).ravel()


sparse_row_norms(vectorized_docs)
sparse_row_norms(tfidf_docs)
sparse_row_norms(tfidf_docs_trimmed)<jupyter_output><empty_output><jupyter_text>**Question**- Why do all TF-IDF vectorized documents have unit norm? Hint: look at the default parameters for thoe classes.<jupyter_code>def sparse_dot_products(query_vector, other_vectors):
    dot_products = query_vector.multiply(other_vectors).sum(axis=1)
    return np.asarray(dot_products).ravel()
similarities = sparse_dot_products(tfidf_docs[0, :], tfidf_docs[1:, :])
categories = [path.parent.name for path in text_filepaths]<jupyter_output><empty_output><jupyter_text>**Exercises**:- Find the document id and category for the 15 documents that are most similar to the first document in the corpus.- Print the text of the first document (the query) and the text of the most similar documnt from your computed list.<jupyter_code># %load notebook_solutions/tfidf_similarities.py<jupyter_output><empty_output><jupyter_text>Impact of TF-IDF Weighting on K-NN Accuracy<jupyter_code>%%time
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score

n_neighbors = 5
cv_scores = cross_val_score(KNeighborsClassifier(n_neighbors=n_neighbors),
                            vectorized_docs, categories, cv=10)

print(f"Val. accuracy: {cv_scores.mean():.3f} (+/-{cv_scores.std():.3f})")
%%time
cv_scores = cross_val_score(KNeighborsClassifier(n_neighbors=n_neighbors),
                            tfidf_docs, categories, cv=10)

print(f"Val. accuracy: {cv_scores.mean():.3f} (+/-{cv_scores.std():.3f})")
%%time
cv_scores = cross_val_score(KNeighborsClassifier(n_neighbors=n_neighbors),
                            tfidf_docs_trimmed, categories, cv=10)

print(f"Val. accuracy: {cv_scores.mean():.3f} (+/-{cv_scores.std():.3f})")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10172464191756796, 'fraction_numerical': 0.006820617753093637, 'mean_word_length': 6.2332628611698375, 'pattern_counts': {'":': 2, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '110643', 'license': 'MIT', 'n_tokens_mistral': 3323, 'n_tokens_neox': 3166, 'n_words': 934, 'path': 'notebooks/BBC_News_Dataset_part_2.ipynb', 'repo_name': 'kmbrou/text-mining-class', 'size': '10263'}