<jupyter_start><jupyter_text>Homework 4 -- Name(s) here **Due Thursday, November 12 by midnight**. You may submit this assignment in groups of 2. Be sure to put your names above. Also, don't forget to submit your data files. In this homework, we approach the problem of identifying a small subset of a dataset using unsupervised and supervised methods. The dataset we'll use is a set of newsgroup postings from the early days of the internet.In a real world application, we often don’t have labels, and clustering and outlier detection are usually applied in settings that don’t have labels. For the clustering part of this homework, you should work without the ground truth labels as much as possible. Often inspecting and visualizing the data is the only way to understand the result of clustering.However, since we do have ground-truth we could do a post-hoc analysis and determine how well we actually did. You  might explore that after you've done the clustering.For tasks 1-3 you don’t need to split the data or use cross-validation. For task 4, you need to use the standard split methods for supervised learning.<jupyter_code>import pandas as pd
import numpy as np
from sklearn.datasets import fetch_20newsgroups
newsgroups = fetch_20newsgroups()

all_df = pd.DataFrame({'Text': newsgroups.data, 'Category Label': newsgroups.target})
all_df['Category Name'] = all_df['Category Label'].map(lambda idx: newsgroups.target_names[idx])
print(all_df['Category Name'].value_counts())
print(all_df['Text'][0])
all_df<jupyter_output>rec.sport.hockey            600
soc.religion.christian      599
rec.motorcycles             598
rec.sport.baseball          597
sci.crypt                   595
sci.med                     594
rec.autos                   594
sci.space                   593
comp.windows.x              593
comp.os.ms-windows.misc     591
sci.electronics             591
comp.sys.ibm.pc.hardware    590
misc.forsale                585
comp.graphics               584
comp.sys.mac.hardware       578
talk.politics.mideast       564
talk.politics.guns          546
alt.atheism                 480
talk.politics.misc          465
talk.religion.misc          377
Name: Category Name, dtype: int64
From: <EMAIL> (where's my thing)
Subject: WHAT car is this!?
Nntp-Posting-Host: rac3.wam.umd.edu
Organization: University of Maryland, College Park
Lines: 15

 I was wondering if anyone out there could enlighten me on this car I saw
the other day. It was a 2-door sports car, looked to be from the late 60s/
early 7[...]
{'fraction_non_alphanumeric': 0.06534653465346535, 'fraction_numerical': 0.03207920792079208, 'mean_word_length': 3.1006493506493507, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '602698', 'license': 'MIT', 'n_tokens_mistral': 779, 'n_tokens_neox': 670, 'n_words': 308, 'path': 'courses/comp347/f20/hwk4.ipynb', 'repo_name': 'robertutterback/robertutterback.github.io', 'size': '2536'}