<jupyter_start><jupyter_text>Step 11 Similarity for a fresh text ![Unknown.png](attachment:c11882a6-af91-4368-973b-0a913ff65fb0.png) |**[Overview](Overview)** |**[Installation](Installation)||**[Prior-steps](Prior-steps)**|**[How-to-use](How-to-use)**|**[Next-steps](Next-steps)**|**[Postscript](Postscript)**|**[Acknowledgements](Acknowledgments)|  Overview Prior stepsSteps 5 and 9 are essential How-to-use Import Gensim, models and library<jupyter_code>from gensim import similarities, models, corpora
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
#hide
import os
directory= "/Users/lawrence/Documents/GitHub/Data-Model-for-Project-Frameworks/Project-frameworks-by-using-NLP-with-Python-libraries/Interim-results/"
index = similarities.MatrixSimilarity.load(directory+'Index_for_corpus_for_similarities.index')
loaded_TFIDF_model = models.TfidfModel.load(directory+'model-from-input-library.tfidf')
Names_of_documents=(pd.read_csv(directory+'Corpus_names_of_documents.csv'))
from gensim.corpora import Dictionary
loaded_dictionary = Dictionary.load(directory+'Library.dict')<jupyter_output><empty_output><jupyter_text>Enter new textI have done it here by assuming this is a new document.- Run step 1, but from a fresh folder (rather than Nuclear sample folder)- run Step 5, but back into a fresh folder (rather than Interim results)However, this can be scraped from an internet documentOr copied in as a stringOr imported as a pdf with Step 1 and Step 2<jupyter_code>directory= "/Users/lawrence/"
with open(os.path.join(directory,'2020-Invernizzi-Locatelli-Brookes-Characterising-nuclear-decommissioning-projects.txt')) as f:
    Fresh_text=f.read()
    f.close()
Fresh_text[0:1000]<jupyter_output><empty_output><jupyter_text>FINDING VECTOR REPRESENTATION OF A SINGLE NEW DOCUMENTFirst CBOW, then TFIDF<jupyter_code>Fresh_text_bow = loaded_dictionary.doc2bow(Fresh_text.lower().split())
vec_tfidf = loaded_TFIDF_model[Fresh_text_bow]  # convert the query to TFIDF space
print(vec_tfidf[0])<jupyter_output>(0, 0.016459011879328547)<jupyter_text>find cosine similarity with existing libraryIn addition, we will be considering cosine similarity to determine the similarity of two vectors. Cosine similarity is a standard measure in Vector Space Modeling, but wherever the vectors represent probability distributions, different similarity measures may be more appropriate.<jupyter_code>Similarity_result=index[vec_tfidf]
plt.bar(range(0,17),height=Similarity_result[0:17])
plt.show()
for f in range(0,17):  # change length manually 
    print (Similarity_result[f],Names_of_documents.iloc[f,0])<jupyter_output>0.031842582 Commissioning-of-security-systems-and-infrastructure-cns-tast-gd-4.4.pdf.txt
0.086545244 Construction Assurance ns-tast-gd-076.pdf.txt
0.07395536 Control of processes involving nuclear matter ns-tast-gd-023.pdf.txt
0.29614276 Decommissioning ns-tast-gd-026.pdf.txt
0.10139444 Design Safety Assurance ns-tast-gd-057.pdf.txt
0.05637164 Fundamental Principles ns-tast-gd-004.pdf.txt
0.15782963 Guidance on the Demonstration of ALARP ns-tast-gd-005.pdf.txt
0.17723335 Management of Radioactive material ns-tast-gd-024.pdf.txt
0.04973495 nuclear construction sites cns-tast-gd-6.6.pdf.txt
0.06878132 Organisational Change ns-tast-gd-048.pdf.txt
0.02818096 oversight of items or services cns-tast-gd-4.3.pdf.txt
0.3880903 PM_guidance_for_Energy_Projects 19.43.14.txt
0.049440667 Probabilistic Safety Analysis.pdf.txt
0.03757745 Procedure Design and Administrative Controls.pdf.txt
0.027715664 Procurement cns-tast-gd-4.1.pdf.txt
0.021520242 Reliability and resilience of the security system cns[...]
{'fraction_non_alphanumeric': 0.104973893926903, 'fraction_numerical': 0.07062379774663369, 'mean_word_length': 6.599164926931106, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '217501', 'license': 'MIT', 'n_tokens_mistral': 1402, 'n_tokens_neox': 1206, 'n_words': 282, 'path': 'Project-frameworks-by-using-NLP-with-Python-libraries/Jupyter-notebooks/Step-11-Similarity_for_a_fresh_text.ipynb', 'repo_name': 'lawrencerowland/Automation-of-construction-contract-review', 'size': '3639'}