<jupyter_start><jupyter_text>Ok, what we have done is do the query against Solr, then finding the similarity between the descriptions of the query and the results.The idea behind the algorithm would be to reorder the results based on the similarity score, not on BM25. Let's see which one is better.<jupyter_code>df_results.head()
a = None # Sort the df_results by similarity column in descending order
a = a[:10].reset_index(drop=True)
print(f'Most similar document after reranking within retrieved results has description: \n\n{a["description"].iloc[0]}\nWith similarity: {a["Similarity"].iloc[0]}')
print(f'Most similar document before reranking within retrieved results has description: \n\n{df_results["description"].iloc[0]}\nWith similarity: {df_results["Similarity"].iloc[0]}')

print(f'Number of documents that surpass 0.5 similarity threshold: {len(a[a["Similarity"] >= 0.5])}')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11220043572984749, 'fraction_numerical': 0.013071895424836602, 'mean_word_length': 6.722689075630252, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '582149', 'license': 'MIT', 'n_tokens_mistral': 287, 'n_tokens_neox': 262, 'n_words': 97, 'path': '2-ranking/lab5/exercise/reranking.ipynb', 'repo_name': 'axel-sirota/ml-solr-course', 'size': '918'}