<jupyter_start><jupyter_text>Latent Semantic Analysis Latent Semantic Analysis analyze relationships between a set of documents and the terms they contain by producing a set of concepts related to the documents and terms. It assumes that words that are close in meaning will occur in similar pieces of text.LSA is a unsupervised method which is helpful for data exploration, dimensionality reduction and for experimentation purposes with supervised models.<jupyter_code>import numpy as np
import pandas as pd
import nltk
from nltk.stem import WordNetLemmatizer
from sklearn.decomposition import TruncatedSVD
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>We will be plotting a very dense graph so let's enable notebook styled graphs. They will allow us to zoom in/out and pan so we'll be able to clearly see data we want.<jupyter_code>%matplotlib notebook
data = open('../data/book_titles/all_book_titles.txt').read()
data = data.split('\n')
data[:10]
stop_words = set(open('../data/general/english-stopwords.txt').read().split('\n'))
lemma = WordNetLemmatizer()<jupyter_output><empty_output><jupyter_text>Data preprocessing is standard for a bag-of-words like structures. Only addition to the previous usecases is that we are removing words containing any numbers in order to simplify our model.<jupyter_code>import re

def preprocess_text(t):
    words = re.split('; |, |\*|\n|/|_', t)
    words = [word.strip(',.:-_').lower() for word in words]
    words = [lemma.lemmatize(word) for word in words]
    words = [word for word in words if word not in stop_words]
    words = [word for word in words if not re.search(r'\d', word)]
    return ' '.join(words)

X = [preprocess_text(r) for r in data]
X = [title for title in X if len(title) > 0]
from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(max_features=1000)
X = vectorizer.fit_transform(X)
X.shape<jupyter_output><empty_output><jupyter_text>This transformer performs linear dimensionality reduction by means of truncated singular value decomposition (SVD).<jupyter_code>svd = TruncatedSVD()
Z = svd.fit_transform(X)<jupyter_output><empty_output><jupyter_text>Plotting out multi-dimensional data converted (simplified) to 2D.<jupyter_code>plt.scatter(Z[:,0], Z[:,1])

i = 0
for name in vectorizer.get_feature_names():
    plt.annotate(s=name, xy=(Z[i,0], Z[i,1]))
    i += 1<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09851607584501236, 'fraction_numerical': 0.005770816158285243, 'mean_word_length': 5.5772357723577235, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '425128', 'license': 'MIT', 'n_tokens_mistral': 773, 'n_tokens_neox': 752, 'n_words': 258, 'path': 'nlp/Latent Sentiment Analysis.ipynb', 'repo_name': 'ta-ja/Machine-Learning-Journey-Notebooks', 'size': '2426'}