<jupyter_start><jupyter_text>Visual Analysis of Document Similarity  About the Data: Hobbies Corpus Baleen corpus contains 448 files in 5 categories. Structured as:- 7,420 paragraphs (16.562 mean paragraphs per file)- 14,251 sentences (1.921 mean sentences per paragraph).Word count of 288,520 with a vocabulary of 23,738 (12.154 lexical diversity).Corpus scan took 14 seconds. Category Breakdown The categories in this corpus have the following properties: Documents and File Size - books: 72 docs (4.1MiB)- cinema: 100 docs (9.2MiB)- cooking: 30 docs (3.0MiB)- gaming: 128 docs (8.8MiB)- sports: 118 docs (15.9MiB) Document Structure - books: 844 paragraphs and 2,030 sentences- cinema: 1,475 paragraphs and 3,047 sentences- cooking: 1,190 paragraphs and 2,425 sentences- gaming: 1,802 paragraphs and 3,373 sentences- sports: 2,109 paragraphs and 3,376 sentences Words and Vocabulary- books: 41,851 words with a vocabulary size of 7,838- cinema: 69,153 words with a vocabulary size of 10,274- cooking: 37,854 words with a vocabulary size of 5,038- gaming: 70,778 words with a vocabulary size of 9,120- sports: 68,884 words with a vocabulary size of 8,028  Task 1: Load the Corpus<jupyter_code>from textDB import load_data
corpus = load_data('hobbies')<jupyter_output><empty_output><jupyter_text>Task 2: Vectorize the Documents<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer()
docs       = vectorizer.fit_transform(corpus.data)
labels     = corpus.target<jupyter_output><empty_output><jupyter_text>Task 3: t-SNE: Clustering Similar Documents with Squared Euclidean Distance And Euclidean Distance<jupyter_code>from yellowbrick.text import TSNEVisualizer

tsne = TSNEVisualizer(size=(600,400))
tsne.fit(docs, labels)
tsne.poof()<jupyter_output><empty_output><jupyter_text><jupyter_code>tsne = TSNEVisualizer(metric="euclidean", size=(600,400))
tsne.fit(docs, labels)
tsne.poof()<jupyter_output><empty_output><jupyter_text>Distance Metrics for Non-Numerical Inputs<jupyter_code># Examples of distance metrics
    distance_functions = [
    "braycurtis", "canberra", "chebyshev", "cityblock", "correlation", "cosine",
    "dice", "euclidean", "hamming", "jaccard", "kulsinski", "mahalanobis",
    "matching", "minkowski", "rogerstanimoto", "russellrao", "seuclidean",
    "sokalmichener", "sokalsneath", "sqeuclidean", "yule"
    ]<jupyter_output><empty_output><jupyter_text>We could try them all, but several of them just don't make sense in the context of text data. Let's go through them individually:    Task 4: Manhattan (aka “Taxicab” or “City Block”) Distance ![Alt-text](https://upload.wikimedia.org/wikipedia/commons/0/08/Manhattan_distance.svg "By User:Psychonaut - Created by User:Psychonaut with XFig, Public Domain, https://commons.wikimedia.org/w/index.php?curid=731390")<jupyter_code>tsne = TSNEVisualizer(metric="cityblock", size=(600,400))
tsne.fit(docs, labels)
tsne.poof()<jupyter_output><empty_output><jupyter_text>Task 5: Bray Curtis Dissimilarity and Canberra Distance<jupyter_code>tsne = TSNEVisualizer(metric="braycurtis", size=(600,400))
tsne.fit(docs, labels)
tsne.poof()

tsne = TSNEVisualizer(metric="canberra", size=(600,400))
tsne.fit(docs, labels)
tsne.poof()<jupyter_output><empty_output><jupyter_text>Task 6: Cosine Distance The angle between two documents:$$\cos(\overline{X},\overline{Y})=\frac{\sum_{i=1}^{d} x_{i}y_{i}}{\sqrt{\sum_{i=1}^{d}x_{i}^2} \sqrt{\sum_{i=1}^{d}y_{i}^2}}$$<jupyter_code>tsne = TSNEVisualizer(metric="cosine")
tsne.fit(docs, labels)
tsne.poof()<jupyter_output><empty_output><jupyter_text>Task 7:What metrics not to use?<jupyter_code>distance_functions = [
    "jaccard", "sokalmichener", "rogerstanimoto", "hamming"
]

for metric in distance_functions:
    tsne = TSNEVisualizer(metric=metric)
    tsne.fit(docs, labels)
    tsne.poof()<jupyter_output><empty_output><jupyter_text>If we omit the target during fit, we can visualize the whole dataset to see if any meaningful patterns are observed.    Task 8: Omitting Class Labels - Using KMeans Clustering<jupyter_code># Don't color points with their classes
tsne = TSNEVisualizer(labels=["documents"])
tsne.fit(docs)
tsne.poof()
# Apply clustering instead of class names.
from sklearn.cluster import KMeans

clusters = KMeans(n_clusters=5)
clusters.fit(docs)

tsne = TSNEVisualizer()
tsne.fit(docs, ["c{}".format(c) for c in clusters.labels_])
tsne.poof()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12157303370786517, 'fraction_numerical': 0.045393258426966294, 'mean_word_length': 6.674137931034482, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '472195', 'license': 'MIT', 'n_tokens_mistral': 1726, 'n_tokens_neox': 1493, 'n_words': 401, 'path': 'Analyze Text Data with Yellowbrick/Document Similarity.ipynb', 'repo_name': 'ElizaLo/NLP', 'size': '4450'}