<jupyter_start><jupyter_text>Similarity Functions This notebook describes about the similarity functions that can be used to measure the similarity between two sets.Firstly we import the shingling functions and other helpful functions.<jupyter_code>from src.shingle import *
from math import ceil, floor
import numpy as np<jupyter_output><empty_output><jupyter_text>We will then count how frequent a shingle is in the document. For this I have calculated the frequencies in the document called `data/portuguese/two_ends.txt`. Here we are using portuguese corpus.Then we create a dictionary called `frequencies` which goes like from the word to its frequency.<jupyter_code># Initialize counts
frequencies = {}
text = open("data/portuguese/two_ends.txt", "r+")
for line in text:
    word = line.strip().split(' ')
    frequencies[word[0]] = float(word[1])<jupyter_output><empty_output><jupyter_text>TF - IDF TF-IDF (Term-frequency and Inverse Document Frequency) measures similarity using this:Firstly, we define `tf` using this, which is just the frequency counts in the intersection.<jupyter_code>def tf(intersection, query):
    '''Counts term frequency'''
    tf = [query.count(word) for word in intersection]
    return np.array(tf)<jupyter_output><empty_output><jupyter_text>Afterwards, we compute `idf`, which is inverse document frequency. Here we will make use of the dictionary that we created earlier in order to compute document frequencies.<jupyter_code>def idf(intersection, document, N):
    '''Counts inverse document frequency'''
    idf = np.array([frequencies[word] for word in intersection])
    idf = np.log10(np.divide(N + 1, idf + 0.5))
    return idf<jupyter_output><empty_output><jupyter_text>Finally we simulate the function `tf_idf` which takes the dot product of `tf` and `idf` arrays.<jupyter_code>def tf_idf(query, document, N):
    intersection = [word for word in document if word in query] # intersection
    score = np.dot(tf(intersection, query), idf(intersection, document, N))
    return score<jupyter_output><empty_output><jupyter_text>We can then run the similarity function in the following manner:<jupyter_code>query = two_ends("pizza", 2)
document = two_ends("pizza", 2)
tf_idf(query, document, 50000) # number of documents are around 50000<jupyter_output><empty_output><jupyter_text>BM25 The formula of BM25 is given like this:Here we define the `bm25_tf` in the following manner:<jupyter_code>def bm25_tf(intersection, query, document, k1, b, avgdl, N):
    tf_ = tf(intersection, document)
    numerator = tf_ * (k1 + 1.0)
    denominator = tf_ + k1 * (1.0 - b + b * (len(query) / avgdl))
    bm25_tf = np.divide(numerator, denominator)
    return bm25_tf<jupyter_output><empty_output><jupyter_text>Finally we will take the dot product of `bm25_tf` and `idf` to get this:<jupyter_code>def bm25(query, document, k1 = 1.2, b = 0.75, avgdl = 8.3, N = 50000):
    intersection = [word for word in document if word in query] # intersection
    score = np.dot(bm25_tf(intersection, query, document, k1, b, avgdl, N), idf(intersection, document, N))
    return score<jupyter_output><empty_output><jupyter_text>We can run the function in the following manner:<jupyter_code>query = two_ends("pizza", 2)
document = two_ends("pizza", 2)
bm25(query, document)<jupyter_output><empty_output><jupyter_text>Dirichlet The formula of Dirichlet is given like this:Firstly, we will compute the sum dependent function here in the form of `smooth`.<jupyter_code>shingles = 470751
def smooth(intersection, document, mu):
    smooth = []
    for word in intersection:
        prob = 1.0 + np.divide(document.count(word), mu * frequencies[word] / shingles)
        smooth.append(np.log10(prob))
    smooth = np.array(smooth)
    return smooth<jupyter_output><empty_output><jupyter_text>We will add the sum independent function to `smooth` and take the dot product to `tf`.<jupyter_code>def dirichlet(query, document, mu = 100.0):
    intersection = [word for word in document if word in query] # intersection
    add = len(query) * np.log10(np.divide(mu, mu + len(document)))
    score = np.dot(tf(intersection, query), smooth(intersection , document, mu)) + add
    return score<jupyter_output><empty_output><jupyter_text>We can this function in following manner:<jupyter_code>query = two_ends("pizzzza", 2)
document = two_ends("pizzza", 2)
print(dirichlet(query, document))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10400180872710829, 'fraction_numerical': 0.018087271082975358, 'mean_word_length': 5.458394160583942, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '14495', 'license': 'MIT', 'n_tokens_mistral': 1475, 'n_tokens_neox': 1410, 'n_words': 447, 'path': 'Notebook 3, Similarity Functions.ipynb', 'repo_name': 'pranav-ust/cognates', 'size': '4423'}