tfidf wikipedia https en wikipedia orgwikitfe28093idf tfidf and other word frequency algorithms are often used as a weighting factor in information retrieval and text mining 83 of textbased recommender systems use tfidf for term weighting in layman s terms tfidf is a statistic intended to reflect how important a word is to a document in a corpus a collection of documents here i ve implemented several word frequency algorithms that are commonly used in information retrieval term frequency document frequency and tfidf termfrequencyinversedocumentfrequency are included term frequency is a statistical function that returns a number representing how frequently an expression occurs in a document this indicates how significant a particular term is in a given document document frequency is a statistical function that returns an integer representing the number of documents in a corpus that a term occurs in where the max number returned would be the number of documents in the corpus inverse document frequency is mathematically written as log10ndf where n is the number of documents in your corpus and df is the document frequency if df is 0 a zerodivisionerror will be thrown termfrequencyinversedocumentfrequency is a measure of the originality of a term it is mathematically written as tflog10ndf it compares the number of times a term appears in a document with the number of documents the term appears in if df is 0 a zerodivisionerror will be thrown return the number of times a term occurs within a given document params term the term to search a document for and document the document to search within returns an integer representing the number of times a term is found within the document examples termfrequencyto to be or not to be 2 strip all punctuation and newlines and replace it with calculate the number of documents in a corpus that contain a given term params term the term to search each document for and corpus a collection of documents each document should be separated by a newline returns the number of documents in the corpus that contain the term you are searching for and the number of documents in the corpus examples documentfrequencyfirst this is the first document in the corpus nthis is the second document in the corpus nthis is the third document in the corpus 1 3 return an integer denoting the importance of a word this measure of importance is calculated by log10ndf where n is the number of documents and df is the document frequency params df the document frequency n the number of documents in the corpus and smoothing if true return the idfsmooth returns log10ndf or 1log10n1df examples inversedocumentfrequency3 0 traceback most recent call last valueerror log100 is undefined inversedocumentfrequency1 3 0 477 inversedocumentfrequency0 3 traceback most recent call last zerodivisionerror df must be 0 inversedocumentfrequency0 3 true 1 477 combine the term frequency and inverse document frequency functions to calculate the originality of a term this originality is calculated by multiplying the term frequency and the inverse document frequency tfidf tf idf params tf the term frequency and idf the inverse document frequency examples tfidf2 0 477 0 954 tf idf wikipedia https en wikipedia org wiki tf e2 80 93idf tf idf and other word frequency algorithms are often used as a weighting factor in information retrieval and text mining 83 of text based recommender systems use tf idf for term weighting in layman s terms tf idf is a statistic intended to reflect how important a word is to a document in a corpus a collection of documents here i ve implemented several word frequency algorithms that are commonly used in information retrieval term frequency document frequency and tf idf term frequency inverse document frequency are included term frequency is a statistical function that returns a number representing how frequently an expression occurs in a document this indicates how significant a particular term is in a given document document frequency is a statistical function that returns an integer representing the number of documents in a corpus that a term occurs in where the max number returned would be the number of documents in the corpus inverse document frequency is mathematically written as log10 n df where n is the number of documents in your corpus and df is the document frequency if df is 0 a zerodivisionerror will be thrown term frequency inverse document frequency is a measure of the originality of a term it is mathematically written as tf log10 n df it compares the number of times a term appears in a document with the number of documents the term appears in if df is 0 a zerodivisionerror will be thrown return the number of times a term occurs within a given document params term the term to search a document for and document the document to search within returns an integer representing the number of times a term is found within the document examples term_frequency to to be or not to be 2 strip all punctuation and newlines and replace it with word tokenization calculate the number of documents in a corpus that contain a given term params term the term to search each document for and corpus a collection of documents each document should be separated by a newline returns the number of documents in the corpus that contain the term you are searching for and the number of documents in the corpus examples document_frequency first this is the first document in the corpus nthis is the second document in the corpus nthis is the third document in the corpus 1 3 strip all punctuation and replace it with return an integer denoting the importance of a word this measure of importance is calculated by log10 n df where n is the number of documents and df is the document frequency params df the document frequency n the number of documents in the corpus and smoothing if true return the idf smooth returns log10 n df or 1 log10 n 1 df examples inverse_document_frequency 3 0 traceback most recent call last valueerror log10 0 is undefined inverse_document_frequency 1 3 0 477 inverse_document_frequency 0 3 traceback most recent call last zerodivisionerror df must be 0 inverse_document_frequency 0 3 true 1 477 combine the term frequency and inverse document frequency functions to calculate the originality of a term this originality is calculated by multiplying the term frequency and the inverse document frequency tf idf tf idf params tf the term frequency and idf the inverse document frequency examples tf_idf 2 0 477 0 954
import string
from math import log10


def term_frequency(term: str, document: str) -> int:
    
    
    document_without_punctuation = document.translate(
        str.maketrans("", "", string.punctuation)
    ).replace("\n", "")
    tokenize_document = document_without_punctuation.split(" ")  
    return len([word for word in tokenize_document if word.lower() == term.lower()])


def document_frequency(term: str, corpus: str) -> tuple[int, int]:
    
    corpus_without_punctuation = corpus.lower().translate(
        str.maketrans("", "", string.punctuation)
    )  
    docs = corpus_without_punctuation.split("\n")
    term = term.lower()
    return (len([doc for doc in docs if term in doc]), len(docs))


def inverse_document_frequency(df: int, n: int, smoothing=False) -> float:
    
    if smoothing:
        if n == 0:
            raise ValueError("log10(0) is undefined.")
        return round(1 + log10(n / (1 + df)), 3)

    if df == 0:
        raise ZeroDivisionError("df must be > 0")
    elif n == 0:
        raise ValueError("log10(0) is undefined.")
    return round(log10(n / df), 3)


def tf_idf(tf: int, idf: int) -> float:
    
    return round(tf * idf, 3)