<jupyter_start><jupyter_text>Calculating similar books based on reviews<jupyter_code>import pickle
import pandas as pd


with open('/content/drive/MyDrive/GoodReadsClean/IntermediateData/ReviewVars/review_tf_idf.pkl', 'rb') as handle:
    tidf = pickle.load(handle)

with open('/content/drive/MyDrive/GoodReadsClean/IntermediateData/ReviewVars/review_map_workID_index.pkl', 'rb') as handle:
    rev_bks = pickle.load(handle)

bk_data = pd.read_pickle("/content/drive/MyDrive/GoodReadsClean/IntermediateData/BooksVars/book_metadata.pkl")
bk_data.loc[bk_data.work_id == "", "work_id"] = bk_data[bk_data.work_id == ""].book_id
df = bk_data
id_title = dict(zip(df.work_id, df.title))
work_book_map = {}
book_work_map = {}
for x,y in zip(df.work_id, df.book_id):
  if x not in work_book_map.keys():
    work_book_map[x] = []
  work_book_map[x].append(y)
  if y not in book_work_map.keys():
    book_work_map[y] = []
  book_work_map[y].append(x)

import pickle
def save_object(obj, filename):
    with open(filename, 'wb') as output:  # Overwrites any existing file.
        pickle.dump(obj, output, pickle.HIGHEST_PROTOCOL)
  
#save_object(book_work_map, "/content/drive/MyDrive/GoodReadsClean/IntermediateData/OtherVars/book_work_map.pkl")
rev_rev_bks = {cnt:val for cnt, val in enumerate(rev_bks)}
bk_data[bk_data.title.str.lower().str.contains("mistborn")]

import numpy as np
from sklearn.metrics.pairwise import linear_kernel
def reccomendReview(id, n, verbose = False):
  sparseIndex = rev_bks[id]
  cosine_similarities = linear_kernel(tidf[sparseIndex],tidf)
  closest = np.argsort(cosine_similarities[0])[-(n+1):-1]
  closest_ids = [rev_rev_bks[x] for x in closest]
  if verbose:
    print("Books Closest to:", id_title[str(id)])
    for x in closest_ids[::-1]:
      print(id_title[str(x)])
  return closest_ids[::-1]
%%time
ids = reccomendReview(1180409, 50, True)<jupyter_output>Books Closest to: Estudos Sobre Veneno (As Lendas de Yelena Zaltana #1)
Black Soul: L'ombra della magia (Soulfinders, #1)
Magin (Study, #2)
Dawn Study (Soulfinders, #3)
Night Study (Soulfinders, #2)
Fire Study (Study, #3)
The Poison Study Short Story Collection: Assassin Study, Power Study, Ice Study (Study #1.5, #3.5, #3.6)
Assassin Study (Study, #1.5)
Ice Study (Study, #3.6)
The Study Series Bundle (Study, #1-3)
Letters From Yelena
Storm Glass (Glass, #1)
Soul to Soul: A Black Russian American Family, 1865-1992
Of Blood and Magic
Interview with Valek
In The Crossfire
The Vampire's Daughter (Of Light and Darkness, #1)
An Anthology of Contemporary Russian Women Poets
Sea Glass (Glass, #2)
Black Widow: The Itsy-Bitsy Spider
The Vampire's Hourglass
Glass Series Collection: Storm Glass / Sea Glass / Spy Glass
BELLATRIX (Frank Kurns Stories of the UnknownWorld, #3)
The Dead City Blues
Power Study (Study, #3.5)
Transmetropolitan #20
Shattered Glass (Glass, #3.5)
<NAME>: The Things They[...]<jupyter_text>This takes way too long. 45 seconds per prediction is far too much time. We have 2 options: dimensionality reduction or clustering. Then we can search for cosine similar within clusters. We try both  Dimensionality reduction<jupyter_code>%%time
from sklearn.decomposition import TruncatedSVD
svd = TruncatedSVD(300)
ans = svd.fit_transform(tidf[:])
print(svd.explained_variance_ratio_.sum())<jupyter_output>0.1636637891455356<jupyter_text>Try out a rec!<jupyter_code>def reccomendReview2(id, n, verbose = False):
  sparseIndex = rev_bks[id]
  cosine_similarities = linear_kernel([ans[sparseIndex]],ans)
  closest = np.argsort(cosine_similarities[0])[-(n+1):-1]
  closest_ids = [rev_rev_bks[x] for x in closest]
  if verbose:
    print("Books Closest to:", id_title[str(id)])
    for x in closest_ids[::-1]:
      print(id_title[str(x)])
  return closest_ids[::-1]
%%time
ids = reccomendReview2(14245059, 50, True)<jupyter_output>Books Closest to: The Night Circus
نفر من الجن
كل هذا الهراء
في غرفة العنكبوت
عربة الياسمين
Unprotected: A Campus Psychiatrist Reveals How Political Correctness in Her Profession Endangers Every Student
قوم لا يتطهرون
تحرير المرأة في عصر الرسالة
الجسد والروح
سيزيرين
يا صاحبي السجن
Kafka on the Shore
منيجيل
عشيق المترجم
تاريخ ضائع: التراث الخالد لعلماء الإسلام ومفكريه وفنانيه
حياة في الإدارة
نظام الحكومة النبوية المسمى التراتيب الإدراية
ذئاب يلوستون
وطن الراشدين
A History of Reading
زوج حذاء لعائشة
البوصلة القرآنية
العبيد الجدد
ذكريات ضالة
HWJN
التفكير المستقيم والتفكير الأعوج
خرافة التقدم والتخلف: العرب والحضارة الغربية في مستهل القرن الواحد والعشرين
Superman est arabe
أقوم قيلا
La hora veinticinco
Even Angels Ask: A Journey to Islam in America
آذان الأنعام
أنا
خلاصة اليومية والشذور
اللغز وراء السطور
الله والإنسان
تاريخ شكل تاني
الضوء الأزرق
صندوق الدمى
رسالة في الطريق إلى ثقافتنا
رحلة إلى مكة
الإسلام وأصول الحكم: بحث في الخلافة والحكومة في الإسلام
ألواح و دسر
أوراق شخصية : حملة تفتيش
[...]<jupyter_text>Much faster, but completely different recs. Which ones to trust?<jupyter_code>ans.shape<jupyter_output><empty_output><jupyter_text>Clustering - We try on reduced dims to avoid prohibitively long convergence times<jupyter_code>from sklearn.cluster import MiniBatchKMeans
dists = []

cluster_cands = list(range(2,30))
for k in cluster_cands:
    clf = MiniBatchKMeans(n_clusters = k, batch_size = 1000, verbose = 0, max_iter = 1)
    clf.fit(ans[:])
    print(k,clf.inertia_)
    dists.append(clf.inertia_)
%matplotlib inline
from matplotlib import pyplot as plt
plt.plot(cluster_cands,dists, 'xb-')
plt.title("Clusters vs Loss")
plt.xlabel("# of Clusters")<jupyter_output><empty_output><jupyter_text>Tag-based filtering<jupyter_code>with open('/content/drive/MyDrive/GoodReadsClean/IntermediateData/BooksVars/book_tags_tfidf.pkl', 'rb') as handle:
    tag_tidf = pickle.load(handle)
with open('/content/drive/MyDrive/GoodReadsClean/IntermediateData/BooksVars/book_map_workid_index.pkl', 'rb') as handle:
    bk_map = pickle.load(handle)
rev_bk_map = {v:k for k,v in bk_map.items()}
def reccomendGeneral(id, n, tidf, rev_bks, rev_rev_bks, verbose = False):
  sparseIndex = rev_bks[str(id)]
  cosine_similarities = linear_kernel(tidf[sparseIndex],tidf)
  closest = np.argsort(cosine_similarities[0])[-(n+1):-1]
  closest_ids = [rev_rev_bks[x] for x in closest]
  if verbose:
    print("Books Closest to:", id_title[str(id)])
    for x in closest_ids[::-1]:
      print(id_title[str(x)])
  return closest_ids[::-1]

%%time 
ids = reccomendGeneral(1180409, 50,tag_tidf, bk_map, rev_bk_map, True)
bk_map["14245059"]<jupyter_output><empty_output><jupyter_text>Evaluation<jupyter_code>import pickle
with open(r"/content/drive/MyDrive/GoodReadsClean/IntermediateData/InteractVars/interact.pkl", 'rb') as f:
    int_sparse = pickle.load(f)

int_sparse_rowOG = int_sparse.tocsr().astype(float)

int_sparse_colOG = int_sparse.tocsc().astype(float)
def getTrueSim(int_sparse_col, id1, id2):
  a = int_sparse_col.getcol(id1)
  b = int_sparse_col.getcol(id2)
  adict = dict(zip(a.indices,a.data))
  bdict = dict(zip(b.indices,b.data))
  inter = list(set(adict.keys()) & set(bdict.keys()))
  if len(inter) <= 1:
    return 0
  bro = np.corrcoef(a[inter].toarray().reshape(1,-1)[0], b[inter].toarray().reshape(1,-1)[0])[0,1]
  return bro

for num in 
ids = reccomendGeneral(num, 50,tag_tidf, bk_map, rev_bk_map)
for id in ids:
  print(getTrueSim(int_sparse_colOG, 1180409, id))
np.random<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11100658513640639, 'fraction_numerical': 0.019352237602472785, 'mean_word_length': 5.232830820770519, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '504317', 'license': 'MIT', 'n_tokens_mistral': 3316, 'n_tokens_neox': 3065, 'n_words': 692, 'path': 'ContentFiltering.ipynb', 'repo_name': 'vs1720/GoodReadsRecs', 'size': '7446'}