<jupyter_start><jupyter_text>Book Recommedation Topic Modeling using LDA LDA - Unsupervised ML modelLDA predicts propotion of topics each document(input)Idea is to make recommedation based on inequality of topic propotions predicted by model<jupyter_code>import pandas as pd
import numpy as np
import json
import nltk
import re
import csv
import matplotlib.pyplot as plt
from tqdm import tqdm<jupyter_output><empty_output><jupyter_text>Load CMU book summary dataset for Training modelhttp://www.cs.cmu.edu/~dbamman/booksummaries.html<jupyter_code>records = []

with open("booksummaries.txt", 'r',encoding="utf8") as f:
    data_reader = csv.reader(f, dialect='excel-tab')
    for row in tqdm(data_reader):
        records.append(row)<jupyter_output>16559it [00:00, 30035.20it/s]<jupyter_text>strip summary info as we only need summary for training<jupyter_code>summary = []

#summary is present at index 6 according to CMU site
for i in tqdm(records):
    summary.append(i[6])

books = pd.DataFrame({'summary': summary})
books.head(2)<jupyter_output>100%|███████████████████████████████████████████████████████████████████████| 16559/16559 [00:00<00:00, 1598505.83it/s]<jupyter_text>remove stopwords and punctuationsand Lemmatize words into base form<jupyter_code>from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer
import string
stop = set(stopwords.words('english'))
exclude = set(string.punctuation)
lemma = WordNetLemmatizer()
def clean(doc):
    stop_free = " ".join([i for i in doc.lower().split() if i not in stop])
    punc_free = ''.join(ch for ch in stop_free if ch not in exclude)
    normalized = " ".join(lemma.lemmatize(word) for word in punc_free.split())
    return normalized
doc_clean = [clean(doc).split() for doc in tqdm(books['summary'])]<jupyter_output>100%|███████████████████████████████████████████████████████████████████████████| 16559/16559 [00:17<00:00, 954.23it/s]<jupyter_text>Building Dictionary<jupyter_code>import gensim

# Creating the term dictionary of our courpus, where every unique term is assigned an index. 
dictionary = gensim.corpora.Dictionary(doc_clean)

# convert our books_summaries into numerical data using doc to bag of words
doc_term_matrix = [dictionary.doc2bow(doc) for doc in tqdm(doc_clean)]<jupyter_output>100%|██████████████████████████████████████████████████████████████████████████| 16559/16559 [00:02<00:00, 6537.76it/s]<jupyter_text>Creation and training of LDA<jupyter_code># Creating the object for LDA model using gensim library
Lda = gensim.models.ldamodel.LdaModel<jupyter_output><empty_output><jupyter_text>Set parameters of LDA<jupyter_code>num_topics = 5
passes = 5
eta = [0.01]*len(dictionary.keys())
alpha = [0.01]*num_topics
ldamodel = Lda(doc_term_matrix, num_topics=num_topics, id2word = dictionary, passes=passes, alpha=alpha,eta=eta)<jupyter_output><empty_output><jupyter_text>Prediction using Trained Model<jupyter_code>def predict(summary):
    clean_summary = clean(summary).split()
    doc_trm_matrix = dictionary.doc2bow(clean_summary)
    return ldamodel[doc_trm_matrix]<jupyter_output><empty_output><jupyter_text>1984 <NAME><jupyter_code>book_1984 = """A man loses his identity while living under a repressive regime. \
            In a story based on George Orwell's classic novel, <NAME> (<NAME>) is a government employee whose job involves the rewriting of history in a manner that casts his fictional country's leaders in a charitable light. \
            His trysts with Julia (<NAME>) provide his only measure of enjoyment, but lawmakers frown on the relationship -- and in this closely monitored society, there is no escape from Big Brother."""
print('for Book : 1984')
print(predict(book_1984))<jupyter_output>for Book : 1984
[(2, 0.2600291), (3, 0.73927027)]<jupyter_text><NAME> and the Sorcerer's Stone <NAME><jupyter_code>book_harry1 = """Adaptation of the first of <NAME>'s popular children's novels about <NAME>, \
                a boy who learns on his eleventh birthday that he is the orphaned son of two powerful wizards\
                and possesses unique magical powers of his own. He is summoned from his life as an unwanted child to \
                become a student at Hogwarts, an English boarding school for wizards. There, \
                he meets several friends who become his closest allies \
                and help him discover the truth about his parents' mysterious deaths."""
print('for Book : <NAME> and the Sorcerer\'s Stone')
print(predict(book_harry1))<jupyter_output>for Book : <NAME> and the Sorcerer's Stone
[(1, 0.3311144), (2, 0.08902878), (3, 0.16001026), (4, 0.41962406)]
{'fraction_non_alphanumeric': 0.1412855910267472, 'fraction_numerical': 0.04184641932700604, 'mean_word_length': 5.440277777777778, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '121447', 'license': 'MIT', 'n_tokens_mistral': 1811, 'n_tokens_neox': 1532, 'n_words': 445, 'path': 'tmp/Topic Modeling.ipynb', 'repo_name': 'MilanVZinzuvadiya/BookHound', 'size': '4698'}