<jupyter_start><jupyter_text>How is TF-IDF implemented? Let us take the example of given documents for calculating TF-IDF:<jupyter_code>documentA = 'The car is driven on the road.'
documentB = 'The truck is driven on the highway.'<jupyter_output><empty_output><jupyter_text>Using the formulas, we get the following results: ![](https://cdn-media-1.freecodecamp.org/images/1*q3qYevXqQOjJf6Pwdlx8Mw.png) From the above table, we can see that TF-IDF of common words was zero, which shows they are not significant. On the other hand, the TF-IDF of “car” , “truck”, “road”, and “highway” are non-zero. These words have more significance.  Calculate TF-IDF in Python from Scratch<jupyter_code>bagOfWordsA = documentA.split(' ')
bagOfWordsB = documentB.split(' ')
uniqueWords = set(bagOfWordsA).union(set(bagOfWordsB))
numOfWordsA = dict.fromkeys(uniqueWords, 0)
for word in bagOfWordsA:
    numOfWordsA[word] += 1
numOfWordsB = dict.fromkeys(uniqueWords, 0)
for word in bagOfWordsB:
    numOfWordsB[word] += 1
from nltk.corpus import stopwords
stopwords.words('english')
#The function computeTF computes the TF score for each word in the corpus, by document.

def computeTF(wordDict, bagOfWords):
    tfDict = {}
    bagOfWordsCount = len(bagOfWords)
    for word, count in wordDict.items():
        tfDict[word] = count / float(bagOfWordsCount)
    return tfDict
tfA = computeTF(numOfWordsA, bagOfWordsA)
tfB = computeTF(numOfWordsB, bagOfWordsB)
tfA
tfB
#The function computeIDF computes the IDF score of every word in the corpus.

def computeIDF(documents):
    import math
    N = len(documents)
    
    idfDict = dict.fromkeys(documents[0].keys(), 0)
    for document in documents:
        for word, val in document.items():
            if val > 0:
                idfDict[word] += 1
    
    for word, val in idfDict.items():
        idfDict[word] = math.log10(N / float(val))
    return idfDict
idfs = computeIDF([numOfWordsA, numOfWordsB])
idfs
#The function computeTFIDF below computes the TF-IDF score for each word, by multiplying the TF and IDF scores.

def computeTFIDF(tfBagOfWords, idfs):
    tfidf = {}
    for word, val in tfBagOfWords.items():
        tfidf[word] = val * idfs[word]
    return tfidf
tfidfA = computeTFIDF(tfA, idfs)
tfidfB = computeTFIDF(tfB, idfs)
df = pd.DataFrame([tfidfA, tfidfB])
df<jupyter_output><empty_output><jupyter_text>The output produced by the above code for the set of documents D1 and D2 is the same as what we manually calculated above in the table.  Implementing TF-IDF using sklearn in Python One thing to notice in the above code is that, instead of just the log of n_samples, 1 has been added to n_samples to calculate the IDF score. This ensures that the words with an IDF score of zero don’t get suppressed entirely.<jupyter_code>tfidf_vectoriser = TfidfVectorizer()
tfidf_feature=tfidf_vectoriser.fit_transform([documentA, documentB])
df =pd.DataFrame(data=tfidf_feature.todense(),columns=tfidf_vectoriser.get_feature_names())
df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09027547295054762, 'fraction_numerical': 0.005974112180550946, 'mean_word_length': 4.5711645101663585, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '444221', 'license': 'MIT', 'n_tokens_mistral': 1049, 'n_tokens_neox': 998, 'n_words': 323, 'path': 'TF IDF/Ritik_NLP_TF-IDF.ipynb', 'repo_name': 'Learn-Write-Repeat/nlp', 'size': '3013'}