<jupyter_start><jupyter_text>Term Frequency Inverse Document Frequency (tf-idf)* Rescale features by how informative we expect them to be* Give weight to any term that appears often in a particular document, but not in many documents* TfidfVectorizer takes the text data and does both the bag-of-words feature extraction and tf-idf transformation<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
df = pd.DataFrame({
    "words": ["Sola runs", "Sola is a dog", "Sola chews toys"]
})

# TfidfVectorizer takes hyperparameters such as:
#  min_df=0.1
#  max_df=0.75
#  max_features=50
vectorizer = TfidfVectorizer()

# X here is a sparse matrix with one row per document and one column per 
# unique word in the corpus (all the words in all the documents)
X = vectorizer.fit_transform(df["words"])

print("Sparse:\n\n", X)

print("\nDense:\n\n", X.todense())<jupyter_output>Sparse:

   (0, 3)	0.8610369959439764
  (0, 4)	0.5085423203783267
  (1, 1)	0.652490884512534
  (1, 2)	0.652490884512534
  (1, 4)	0.3853716274664007
  (2, 5)	0.652490884512534
  (2, 0)	0.652490884512534
  (2, 4)	0.3853716274664007

Dense:

 [[0.         0.         0.         0.861037   0.50854232 0.        ]
 [0.         0.65249088 0.65249088 0.         0.38537163 0.        ]
 [0.65249088 0.         0.         0.         0.38537163 0.65249088]]<jupyter_text>We can see which columns represent which words:<jupyter_code>vectorizer.get_feature_names()
# We can see, for example, that the first row (representing "Sola runs") has values in the "runs" and "Sola" columns.
# Together these let us construct a data frame if we want:
pd.DataFrame(X.todense(), columns=vectorizer.get_feature_names())<jupyter_output><empty_output><jupyter_text>Accessing the vocabularyThe attribute vocabulary_ outputs a dictionary in which all ngrams are the dictionary keys and the respective values are the column positions of each ngram (feature) in the tfidf matrix. [](https://stackoverflow.com/a/54338182/156835)<jupyter_code>vectorizer.vocabulary_
# We can swap the keys and values to map the column number to the word:

vocab = {v:k for k, v in vectorizer.vocabulary_.items()}
vocab<jupyter_output><empty_output><jupyter_text>Accessing document data<jupyter_code>print(X)
# For any given document, we can access the word indices and tf-idt weights:
print(X[0].indices)
# And the data:
print(X[0].data)
# We can zip these together and convert them into a dict:
dict(zip(X[0].indices, X[0].data))<jupyter_output><empty_output><jupyter_text>Filtering a tf-idf vectorThe problem with the tf-idf vector is that it may contain thousands of words depending on the size of the corpus.We don't want to train a model on using all of those features (the words). Instead, we can identify the most top_n most important terms in each document and then filter the tf-idf vector so it only contains the columns for terms that are in the top_n for any of the documents.This code is inspired via DataCamp's Selecting Features for Modeling course:<jupyter_code># This function returns the indeces of the top_n terms in a given row.
def return_indeces_of_top_terms_in_document(vocab, original_vocab, vector, vector_index, top_n):
    zipped = dict(zip(vector[vector_index].indices, vector[vector_index].data))
    zipped_series = pd.Series({vocab[i]:zipped[i] for i in vector[vector_index].indices})
    zipped_index = zipped_series.sort_values(ascending=False)[:top_n].index
    return [original_vocab[i] for i in zipped_index]
# For example:
return_indeces_of_top_terms_in_document(vocab, vectorizer.vocabulary_, X, 0, 1)<jupyter_output><empty_output><jupyter_text>It returns 3 because in the first row (index 0), we're looking at the top 1 weight, which is 0.861037, which is in the fourth column (index 3).If instead we looked at the top 2 terms:<jupyter_code>return_indeces_of_top_terms_in_document(vocab, vectorizer.vocabulary_, X, 0, 2)<jupyter_output><empty_output><jupyter_text>Because the two terms (Sola and runs) are in the fourth and fifth columns (index 3 and 4).The second column has a tie in weights between dog and is (both have a weight of 0.652491) but sort_values winds up putting dog first for some reason, so this returns index 2:<jupyter_code>return_indeces_of_top_terms_in_document(vocab, vectorizer.vocabulary_, X, 1, 1)<jupyter_output><empty_output><jupyter_text>Using this, we can iterate over all of the rows (documents) and determine the top_n words. return_top_term_indeces returns their indeces.<jupyter_code>def return_top_term_indeces(vocab, original_vocab, vector, top_n):
    filter_list = []
    for i in range(0, vector.shape[0]):
        filtered = return_indeces_of_top_terms_in_document(vocab, original_vocab, vector, i, top_n)
        filter_list.extend(filtered)

    return set(filter_list)<jupyter_output><empty_output><jupyter_text>For example, if we specify top_n is 1, then it iterates over each row and grabs the index for the highest weighted term:<jupyter_code>indeces = return_top_term_indeces(vocab, vectorizer.vocabulary_, X, 1)
indeces<jupyter_output><empty_output><jupyter_text>This returns a set of {0, 2, 3} because the in row 1 the highest weighted term is at index 3 (Sola), row 2 at index 2 (dog), row 3 at index 0 (chews).Then we can filter the original vector down to just these important terms:<jupyter_code>X_filtered = X[:, list(indeces)]
print(X_filtered.todense())<jupyter_output>[[0.         0.         0.861037  ]
 [0.         0.65249088 0.        ]
 [0.65249088 0.         0.        ]]<jupyter_text>Putting it all together:<jupyter_code>def return_indeces_of_top_terms_in_document(vocab, original_vocab, vector, vector_index, top_n):
    zipped = dict(zip(vector[vector_index].indices, vector[vector_index].data))    
    zipped_series = pd.Series({vocab[i]:zipped[i] for i in vector[vector_index].indices})
    zipped_index = zipped_series.sort_values(ascending=False)[:top_n].index
    return [original_vocab[i] for i in zipped_index]

def return_top_term_indeces(vocab, original_vocab, vector, top_n):
    filter_list = []
    for i in range(0, vector.shape[0]):    
        filtered = return_indeces_of_top_terms_in_document(vocab, original_vocab, vector, i, top_n)
        filter_list.extend(filtered)
    return set(filter_list)

def select_highest_weighted_terms(vectorizer, X, top_n):
    vocab = {v:k for k, v in vectorizer.vocabulary_.items()}

    indeces = return_top_term_indeces(vocab, vectorizer.vocabulary_, X, 1)
    return X[:, list(indeces)]

df = pd.DataFrame({
    "words": ["Sola runs", "Sola is a dog", "Sola chews toys"]
})

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(df["words"])

print("Data Frame:\n{}\n".format(pd.DataFrame(X.todense(), columns=vectorizer.get_feature_names())))
print("Values:\n{}\n".format(X.todense()))

X_filtered = select_highest_weighted_terms(vectorizer, X, 1)
print("Filtered values:\n{}\n".format(X_filtered.todense()))<jupyter_output>Data Frame:
      chews       dog        is      runs      sola      toys
0  0.000000  0.000000  0.000000  0.861037  0.508542  0.000000
1  0.000000  0.652491  0.652491  0.000000  0.385372  0.000000
2  0.652491  0.000000  0.000000  0.000000  0.385372  0.652491

Values:
[[0.         0.         0.         0.861037   0.50854232 0.        ]
 [0.         0.65249088 0.65249088 0.         0.38537163 0.        ]
 [0.65249088 0.         0.         0.         0.38537163 0.65249088]]

Filtered values:
[[0.         0.         0.861037  ]
 [0.         0.65249088 0.        ]
 [0.65249088 0.         0.        ]]
{'fraction_non_alphanumeric': 0.09923260121725325, 'fraction_numerical': 0.07528446679015613, 'mean_word_length': 4.184499314128944, 'pattern_counts': {'":': 2, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '245783', 'license': 'MIT', 'n_tokens_mistral': 2981, 'n_tokens_neox': 2630, 'n_words': 770, 'path': 'TF-IDF.ipynb', 'repo_name': 'dbl007/python-cheat-sheet', 'size': '7558'}