<jupyter_start><jupyter_text>Solution 0 (dummy): tf-idf on unprocessed text<jupyter_code>search_terms = '<PASSWORD>'
documents = list(df.text)

doc_vectors = TfidfVectorizer().fit_transform([search_terms] + documents)

cosine_similarities = cosine_similarity(doc_vectors[0:1], doc_vectors).flatten()
document_scores = [item.item() for item in cosine_similarities[1:]]
# 3 stored elements: 'all' 'is' 'love' in the scope of one vector are compared to all the rest vectors
# with cosine similarity metric. This metric demonstrates how far are vectors from each other.
# So, if we compare query to all of our documents, transformed in the same way (vectorized with tf-idf)
# then documents that will be closer to our query will have higher cosine similarity score 
# (cos(0) = 1 -> same direction of vectors). Vectorization gave us 2407 unique elements.
doc_vectors[0:1]<jupyter_output><empty_output><jupyter_text>![cosine sim](https://cs.carleton.edu/cs_comps/0910/netflixprize/final_results/knn/img/knn/cos.png)<jupyter_code>cosine_similarities[0]  # - this what happens when comparing same vectors with this metric
top_df = df.copy()
top_df['similarity_score'] = document_scores
top_df.sort_values('similarity_score', ascending=False).head(5)<jupyter_output><empty_output><jupyter_text>Solution 1: tf-idf on preprocessed text  If we will implement same algorithm on preprocessed text, results should be better, regarding the logic we'll get less sparsed matrix; cleaning stop-words, punctuation and lemmatization of our documents will result in lower quantity of elements in vectors<jupyter_code>word_tokenize('Love, love, mind) Relax - and (float down stream...')  # results of word_tokenize
stop_words = stopwords.words('english')
for word in 'all is love'.split():
    if word in stop_words:
        stop_words.remove(word)
stop_words = set(stop_words)
class LemmaTokenizer:
    ignore_tokens = [',', '.', ';', ':', '"', '``', "''", '`', '(', ')']
    def __init__(self):
        self.wnl = WordNetLemmatizer()
    def __call__(self, doc):
        return [self.wnl.lemmatize(t) for t in word_tokenize(doc) if t not in self.ignore_tokens]

# Lemmatize the stop words
tokenizer=LemmaTokenizer()
token_stop = tokenizer(' '.join(stop_words))

search_terms = 'all is love'
documents = list(df.text)

# Create TF-idf model
vectorizer = TfidfVectorizer(stop_words=token_stop,
                             tokenizer=tokenizer)
doc_vectors = vectorizer.fit_transform([search_terms] + documents)

# Calculate similarity
cosine_similarities = linear_kernel(doc_vectors[0:1], doc_vectors).flatten()
document_scores = [item.item() for item in cosine_similarities[1:]]
lm = LemmaTokenizer()  # Example of how lemmatizer modifies words
lm('cats loves')
top_df = df.copy()
top_df['similarity_score'] = document_scores
top_df.sort_values('similarity_score', ascending=False).head(5)<jupyter_output><empty_output><jupyter_text>As one can see, with this approach, we get higher similarity scores. "Love Me Do" was on the 4th Place of previous ratings, now it's on the 3rd place. "Because" was dropped off top-5  Solution 3: Semantic Similarity with SoftCosineSimilarity metric  Vectorize texts with respect to semantic similarity. "cat" is more similar to "dog" then to "truck".<jupyter_code>stop_words = stopwords.words('english')
for word in 'all is love'.split():
    if word in stop_words:
        stop_words.remove(word)
stop_words = set(stop_words)

def preprocess(doc):
    return [token for token in simple_preprocess(doc, min_len=0, max_len=float("inf")) if token not in stop_words]


glove = api.load("glove-wiki-gigaword-50")    
similarity_index = WordEmbeddingSimilarityIndex(glove)

search_terms = '<PASSWORD>'
documents = list(df.text)

corpus = [preprocess(document) for document in documents]
query = preprocess(search_terms)

dictionary = Dictionary(corpus+[query])
tfidf = TfidfModel(dictionary=dictionary)

similarity_matrix = SparseTermSimilarityMatrix(similarity_index, dictionary, tfidf)
query_tf = tfidf[dictionary.doc2bow(query)]

index = SoftCosineSimilarity(
            tfidf[[dictionary.doc2bow(document) for document in corpus]],
            similarity_matrix)

doc_similarity_scores = index[query_tf]
top_df = df.copy()
top_df['similarity_score'] = doc_similarity_scores
top_df.sort_values('similarity_score', ascending=False).head(5)<jupyter_output><empty_output><jupyter_text>"All_You_Need_Is_Love" is still first, but places 2,3 now belong to Penny_Lane.txt, Within_You_WIthout_You.txt  Solution 4: Semantic Similarity with distilbert and CosineSimilarity<jupyter_code>embedder = SentenceTransformer('distilbert-base-nli-stsb-mean-tokens')

corpus = list(df.text)
corpus_embeddings = embedder.encode(corpus, convert_to_tensor=True)

query = '<NAME> love'

query_embedding = embedder.encode(query, convert_to_tensor=True)
cos_scores = util.pytorch_cos_sim(query_embedding, corpus_embeddings)[0]
cos_scores = cos_scores.cpu()
top_df = df.copy()
top_df['similarity_score'] = cos_scores.numpy()
top_df.sort_values('similarity_score', ascending=False).head(5)<jupyter_output><empty_output><jupyter_text>"All_You_Need_Is_Love" is not even in top-5! Probably "all" and "is" were removed from final corpus  Let's analyse top-3 songs, selected by algorithms  tf-idf selections<jupyter_code>list(df[(df.song == "All_You_Need_Is_Love.txt")].text)
list(df[(df.song == "The_End.txt")].text)
list(df[(df.song == "Because.txt")].text)
list(df[(df.song == "Love_Me_Do.txt")].text)<jupyter_output><empty_output><jupyter_text>GLOVE selections<jupyter_code>list(df[(df.song == "Penny_Lane.txt")].text)
list(df[(df.song == "Within_You_WIthout_You.txt")].text)<jupyter_output><empty_output><jupyter_text>distilBERT selections<jupyter_code>list(df[(df.song == "And_I_Love_Her.txt")].text)
list(df[(df.song == "PS_I_Love_You.txt")].text)
list(df[(df.song == "And_I_Love_Her.txt")].text)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11909998308238877, 'fraction_numerical': 0.006936220605650482, 'mean_word_length': 6.353233830845771, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '324137', 'license': 'MIT', 'n_tokens_mistral': 2088, 'n_tokens_neox': 1962, 'n_words': 466, 'path': 'notebooks/document_search_ranking.ipynb', 'repo_name': 'nazariinyzhnyk/nlp-beatles-lyrics-modeling', 'size': '5913'}