<jupyter_start><jupyter_text>Tf-idf example<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
texts = [
    "good movie", "not a good movie", "did not like", 
    "i like it", "good one"
]
# using default tokenizer in TfidfVectorizer
tfidf = TfidfVectorizer(min_df=2, max_df=0.5, ngram_range=(1, 2))
features = tfidf.fit_transform(texts)
pd.DataFrame(
    features.todense(),
    columns=tfidf.get_feature_names()
)

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
texts2 = [
    "bad movie", "boring  movie", "liked", 
    "she liked it", "good one", "great one!"
]
# using default tokenizer in TfidfVectorizer
tfidf = TfidfVectorizer(min_df=1, max_df=0.5, ngram_range=(1, 2))
features = tfidf.fit_transform(texts2)
pd.DataFrame(
    features.todense(),
    columns=tfidf.get_feature_names()
)<jupyter_output><empty_output><jupyter_text>TF-IDF (term frequency-inverse document frequency) is a statistical measure that evaluates how relevant a word is to a document in a collection of documents.This is done by multiplying two metrics: how many times a word appears in a document, and the inverse document frequency of the word across a set of documents. 3. Given the below documents.     texts = [        "good movie", "not a good movie", "did not like",         "i like it", "good one"    ]Given the definition of TF and IDF, what is the sum of TF-IDF values for 1-grams in "good movie" text? Enter a math expression as an answer.<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
texts = [
    "good movie", "not a good movie", "did not like", 
    "i like it", "good one"
]
# using default tokenizer in TfidfVectorizer
tfidf = TfidfVectorizer(min_df=1, max_df=0.5, ngram_range=(1, 1))
features = tfidf.fit_transform(texts)
pd.DataFrame(
    features.todense(),
    columns=tfidf.get_feature_names()
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1014344262295082, 'fraction_numerical': 0.00973360655737705, 'mean_word_length': 4.761061946902655, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '185097', 'license': 'MIT', 'n_tokens_mistral': 680, 'n_tokens_neox': 660, 'n_words': 212, 'path': 'in-Class- FIles/3 NLP/tfidf_demo.ipynb', 'repo_name': 'MoronSlayer/Deep-Learning_Neural-Nets-628', 'size': '1952'}