<jupyter_start><jupyter_text>Search Engine NLP: Document Similarity Keys Concepts- Term Document Matrix- Cosine Similarity<jupyter_code>import warnings
from collections import OrderedDict
from pathlib import Path
from random import randint
import numpy as np
import pandas as pd

# Visualization
import matplotlib.pyplot as plt
from matplotlib.ticker import FuncFormatter
import seaborn as sns

# sklearn for feature extraction & modeling
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, TfidfTransformer
# from sklearn.decomposition import TruncatedSVD, PCA
from sklearn.model_selection import train_test_split
# from sklearn.externals import joblib
import joblib

# Iteratively read files
import glob
import os

# For displaying images in ipython
from IPython.display import HTML, display
%matplotlib inline
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = (14.0, 8.7)
warnings.filterwarnings('ignore')
pd.options.display.float_format = '{:,.2f}'.format<jupyter_output><empty_output><jupyter_text>Load data<jupyter_code># User defined function to read and store bbc data from multipe folders
def load_data(folder_names,root_path):
    fileNames = [path + '/' + 'bbc' +'/'+ folder + '/*.txt' for path,folder in zip([root_path]*len(folder_names),
                                                                               folder_names )]
    doc_list = []
    tags = folder_names
    for docs in fileNames:
        print(docs)
        #print(type(docs))
        doc = glob.glob(docs) # glob method iterates through all the text documents in a folder
        for text in doc:
            with open(text, encoding='latin1') as f:
                topic = docs.split('/')[8]

                lines = f.readlines()
                heading = lines[0].strip()
                body = ' '.join([l.strip() for l in lines[1:]])
                doc_list.append([topic, heading, body])
        print("Completed loading data from folder: %s"%topic)
    
    print("Completed Loading entire text")
    
    return doc_list
os.getcwd().split()
folder_names = ['business','entertainment','politics','sport','tech']
docs = load_data(folder_names = folder_names, root_path = os.getcwd())
docs = pd.DataFrame(docs, columns=['Category', 'Heading', 'Article'])
print(docs.head())
print('\nShape of data is {}\n'.format(docs.shape))
# print(docs.info())<jupyter_output>Category                            Heading  \
0  business    UK economy facing 'major risks'   
1  business  Aids and climate top Davos agenda   
2  business   Asian quake hits European shares   
3  business   India power shares jump on debut   
4  business    Lacroix label bought by US firm   

                                             Article  
0   The UK manufacturing sector will continue to ...  
1   Climate change and the fight against Aids are...  
2   Shares in Europe's leading reinsurers and tra...  
3   Shares in India's largest power producer, Nat...  
4   Luxury goods group LVMH has sold its loss-mak...  

Shape of data is (2225, 3)<jupyter_text>Documents SimilarityFrom Documents -- DTM -- Cosine SimilarityHTML("") ImportantThe cosine similarity is the cosine of the angle between two vectors.- Cosine Similarity can take value between 0 to 1.- closer to 0 means dissimilar documents- closer to 1 means similar documents  Find Documents Similar to a New document : First Step to Build Search Engine  Convert Raw text --> Parsed Text --> Document Term Matrix  Why to Use Term Frequency Inverse Document Frequency over Term Frequency- TF*IDF is an information retrieval technique that weighs a term’s frequency (TF) and its inverse document frequency (IDF). Each word or term has its respective TF and IDF score. The product of the TF and IDF scores of a term is called the TF*IDF weight of that term.__Put simply, the higher the TF*IDF score (weight), the rarer the term and vice versa.____TFidf__ - is comprised of following two components- __TF: Term Frequency__, which measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more times in long documents than shorter ones. Thus, the term frequency is often divided by the document length (aka. the total number of terms in the document) as a way of normalization: __TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document).____IDF: Inverse Document Frequency__, which measures how important a term is. While computing TF, all terms are considered equally important. However it is known that certain terms, such as "is", "of", and "that", may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scale up the rare ones, by computing the following: __IDF(t) = log_e(Total number of documents / Number of documents with term t in it).____TFidf__ = TF * IDF**Important**Higher the TFidf , more important the significance of word to that document Important Reading Article on TFidfhttps://www.kdnuggets.com/2018/08/wtf-tf-idf.html<jupyter_code>vectorizer = TfidfVectorizer(stop_words = "english")
vectors = vectorizer.fit_transform(docs["Heading"].values)
print("Shape of tfidf matrix: {}".format(vectors.shape))
from sklearn.metrics.pairwise import cosine_similarity
new_query = ["World facing imminent danger across global war theaters"]
new_query_vector = vectorizer.transform(new_query)
new_query_vector
sim = cosine_similarity(X = vectors, Y = new_query_vector)
sim
# Extract Index of Maximum valued similar document
argmax = np.argmax(sim)
print("Index of maximum valued similar doc: %s"%argmax)
print("Retrieved Document Header: %s"%docs["Heading"][argmax])
# To Extract Top 10 Similar Documents against the new query
ind = np.argsort(sim,axis = 0)[::-1][:10]
for i in ind:
    print(docs["Heading"].values[i])
def retrieve_doc(new_query,raw_docs):
    vectorizer = TfidfVectorizer(stop_words = "english")
    vectors = vectorizer.fit_transform(raw_docs["Article"])
    print("Shape of tfidf matrix: {}".format(vectors.shape))
    new_query = [new_query]
    new_query_vector = vectorizer.transform(new_query)
    sim = cosine_similarity(X = vectors, Y = new_query_vector)
    ind = np.argsort(sim,axis = 0)[::-1][:10]
    for i in ind:
        print(docs["Heading"].values[i])
# newQuery = "gaming week in UK"
newQuery = "england to play australia in next ashes series"
retrieve_doc(new_query= newQuery  , raw_docs= docs)<jupyter_output>Shape of tfidf matrix: (2225, 28980)
['Robinson wants dual code success']
['Wilkinson to lead England']
['Barkley fit for match in Ireland']
['Yachvili savours France comeback']
['Dallaglio his own man to the end']
['X Factor show gets second series']
['A November to remember']
['England given tough Sevens draw']
['Dawson set for new Wasps contract']
["O'Gara revels in Ireland victory"]
{'fraction_non_alphanumeric': 0.08793128548551463, 'fraction_numerical': 0.007861406318241374, 'mean_word_length': 3.9424460431654675, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '595088', 'license': 'MIT', 'n_tokens_mistral': 2006, 'n_tokens_neox': 1915, 'n_words': 771, 'path': '5. Text Mining/Document_Similarity.ipynb', 'repo_name': 'kedar750/DSB7', 'size': '6869'}