<jupyter_start><jupyter_text>Homework 14 (or so): TF-IDF text analysis and clusteringHooray, we kind of figured out how text analysis works! Some of it is still magic, but at least the **TF** and **IDF** parts make a little sense. Kind of. Somewhat.No, just kidding, we're *professionals* now.  Investigating the Congressional RecordThe [Congressional Record](https://en.wikipedia.org/wiki/Congressional_Record) is more or less what happened in Congress every single day. Speeches and all that. A good large source of text data, maybe?Let's pretend it's totally secret but we just got it leaked to us in a data dump, and we need to check it out. It was leaked from [this page here](http://www.cs.cornell.edu/home/llee/data/convote.html).<jupyter_code># If you'd like to download it through the command line...
!curl -O http://www.cs.cornell.edu/home/llee/data/convote/convote_v1.1.tar.gz
# And then extract it through the command line...
!tar -zxf convote_v1.1.tar.gz<jupyter_output><empty_output><jupyter_text>You can explore the files if you'd like, but we're going to get the ones from `convote_v1.1/data_stage_one/development_set/`. It's a bunch of text files.<jupyter_code># glob finds files matching a certain filename pattern
import glob

# Give me all the text files
paths = glob.glob('convote_v1.1/data_stage_one/development_set/*')
paths[:5]
len(paths)<jupyter_output><empty_output><jupyter_text>So great, we have 702 of them. Now let's import them.<jupyter_code>speeches = []
for path in paths:
    with open(path) as speech_file:
        speech = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': speech_file.read()
        }
    speeches.append(speech)
speeches_df = pd.DataFrame(speeches)
speeches_df.head()<jupyter_output><empty_output><jupyter_text>In class we had the `texts` variable. For the homework can just do `speeches_df['content']` to get the same sort of list of stuff.**Take a look at the contents of the first 5 speeches**<jupyter_code>speeches_df['content'].head(5)<jupyter_output><empty_output><jupyter_text>Doing our analysisUse the `sklearn` package and a plain boring `CountVectorizer` to get a list of all of the tokens used in the speeches. If it won't list them all, that's ok! Make a dataframe with those terms as columns.**Be sure to include English-language stopwords**<jupyter_code>count_vectorizer = CountVectorizer(stop_words='english')
X=count_vectorizer.fit_transform(speeches_df['content'])
len(count_vectorizer.get_feature_names())
tokens_df=pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())
tokens_df<jupyter_output><empty_output><jupyter_text>Okay, it's **far** too big to even look at. Let's try to get a list of features from a new `CountVectorizer` that only takes the top 100 words.<jupyter_code>count_vectorizer = CountVectorizer(stop_words='english', max_features=100)
X=count_vectorizer.fit_transform(speeches_df['content'])
len(count_vectorizer.get_feature_names())<jupyter_output><empty_output><jupyter_text>Now let's push all of that into a dataframe with nicely named columns.<jupyter_code>tokens_df=pd.DataFrame(X.toarray(), columns=count_vectorizer.get_feature_names())
tokens_df<jupyter_output><empty_output><jupyter_text>Everyone seems to start their speeches with "mr chairman" - how many speeches are there total, and how many don't mention "chairman" and how many mention neither "mr" nor "chairman"?<jupyter_code># 702 rows means 702 speeches, since each speech is a single string
len(tokens_df)
# if the speech doesnt contain a chairman, the column entry will be 0. so, 250 no-chairmain speeches. granted,
# we have no idea if they stared the speech with chairman or just mentioned him somewhere
len(tokens_df[tokens_df['chairman']==0])
# 76 times no mr or chairman. which means they must call the chairman just 'chairman' a lot. rude!
len(tokens_df[(tokens_df['mr']==0) & (tokens_df['chairman']==0)])<jupyter_output><empty_output><jupyter_text>What is the index of the speech which is the most thankful, a.k.a. includes the word 'thank' the most times?<jupyter_code># so speech index 375
tokens_df[tokens_df['thank']==tokens_df['thank'].max()].index
# lets look at the speech
speeches_df['content'][375]
# wow that was long. but its the most thankful one, so whatever.<jupyter_output><empty_output><jupyter_text>If I'm searching for `China` and `trade`, what are the top 3 speeches to read according to the `CountVectoriser`?<jupyter_code># i sorted by china here, on a lark. lets see if it holds true if i sort by trade
tokens_df[(tokens_df['china']>0) & (tokens_df['trade']>0)].sort_values(by='china', ascending=False)[['china', 'trade']].head(3)
tokens_df[(tokens_df['china']>0) & (tokens_df['trade']>0)].sort_values(by='trade', ascending=False)[['china', 'trade']].head(3)
# kind of! at any rate, speech 294 seems to be the most china and trade related. lets look at it!
speeches_df['content'][294]
# thats another super long speech, but it does seem to mostly be about trade and china.<jupyter_output><empty_output><jupyter_text>Now what if I'm using a `TfidfVectorizer`?<jupyter_code>l2_vectorizer = TfidfVectorizer(stop_words='english', use_idf=True)
X = l2_vectorizer.fit_transform(speeches_df['content'])
tfidf_tokens_df = pd.DataFrame(X.toarray(), columns=l2_vectorizer.get_feature_names())
china_trade_df=pd.DataFrame([tfidf_tokens_df['china'], tfidf_tokens_df['trade'], tfidf_tokens_df['china'] + tfidf_tokens_df['trade']], index=["china", "trade", "china + trade"]).T
china_trade_df[china_trade_df.any(axis=1)].sort_values(by='china + trade', ascending=False).head(3)
# wow, that comes up with a totally different list of speeches. lets look at speech 636
speeches_df['content'][636]
# that one is very short by comparison and this time, its really only about china and trade<jupyter_output><empty_output><jupyter_text>**What's the content of the speeches?** Here's a way to get them:<jupyter_code># index 0 is the first speech, which was the first one imported.
paths[0]
# Pass that into 'cat' using { } which lets you put variables in shell commands
# that way you can pass the path to cat
!cat {paths[0]}
# i guess i probably should have read ahead to this part. oh well, dumping the index of the speeches_df 
# was still mostly readable<jupyter_output><empty_output><jupyter_text>**Now search for something else!** Another two terms that might show up. `elections` and `chaos`? Whatever you thnik might be interesting.<jupyter_code>election_chaos_df=pd.DataFrame([tfidf_tokens_df['election'], tfidf_tokens_df['chaos'], tfidf_tokens_df['election'] + tfidf_tokens_df['chaos']], index=["election", "chaos", "election + chaos"]).T
election_chaos_df[election_chaos_df.any(axis=1)].sort_values(by='chaos', ascending=False).head(10)
# i did the sort that way because i guess they dont talk about chaos much. lets look at that speech
!cat {paths[257]}
# thats pretty weak. i tried to come up with something spicier but i cant tell what years these are from.
# how about this?
clinton_welfare_df=pd.DataFrame([tfidf_tokens_df['clinton'], tfidf_tokens_df['welfare'], tfidf_tokens_df['clinton'] + tfidf_tokens_df['welfare']], index=["clinton", "welfare", "clinton + welfare"]).T
clinton_welfare_df[clinton_welfare_df.any(axis=1)].sort_values(by='clinton + welfare', ascending=False).head(10)
!cat {paths[346]}
# still pretty dull. oh well.<jupyter_output><empty_output><jupyter_text>Enough of this garbage, let's clusterUsing a **simple counting vectorizer**, cluster the documents into **eight categories**, telling me what the top terms are per category.Using a **term frequency vectorizer**, cluster the documents into **eight categories**, telling me what the top terms are per category.Using a **term frequency inverse document frequency vectorizer**, cluster the documents into **eight categories**, telling me what the top terms are per category.<jupyter_code>from sklearn.cluster import KMeans
count_vectorizer = CountVectorizer(stop_words='english')
X=count_vectorizer.fit_transform(speeches_df['content'])
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = count_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
tf_vectorizer = TfidfVectorizer(stop_words='english', use_idf=False)
X = tf_vectorizer.fit_transform(speeches_df['content'])
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = tf_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
tfidf_vectorizer = TfidfVectorizer(stop_words='english', use_idf=True)
X = tfidf_vectorizer.fit_transform(speeches_df['content'])
number_of_clusters = 8
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = tfidf_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))<jupyter_output>Top terms per cluster:
Cluster 0: demand recorded vote mr speaker
Cluster 1: yield gentleman mr chairman minutes
Cluster 2: start head children program amendment
Cluster 3: mr amendment chairman time gentleman
Cluster 4: frivolous lawsuits civil religious federal
Cluster 5: claim consent opposition ask unanimous
Cluster 6: china trade speaker madam cafta
Cluster 7: balance time chairman reserve mr<jupyter_text>**Which one do you think works the best?**<jupyter_code># well, it seems to be between results including wild horses and those including frivolous lawsuits.
# i prefer wild horses but the tfidf is probably the best representation of the actual document<jupyter_output><empty_output><jupyter_text>Harry Potter timeI have a scraped collection of Harry Potter fanfiction at https://github.com/ledeprogram/courses/raw/master/algorithms/data/hp.zip.I want you to read them in, vectorize them and cluster them. Use this process to find out **the two types of Harry Potter fanfiction**. What is your hypothesis?<jupyter_code>paths = glob.glob('hp/hp/*')
paths[:5]
hp_fics = []
for path in paths:
    with open(path) as hp_file:
        hp_fic = {
            'pathname': path,
            'filename': path.split('/')[-1],
            'content': hp_file.read()
        }
    hp_fics.append(hp_fic)
hp_fics_df = pd.DataFrame(hp_fics)
hp_fics_df.head()
tfidf_vectorizer = TfidfVectorizer(stop_words='english', use_idf=True)
X = tfidf_vectorizer.fit_transform(hp_fics_df['content'])
number_of_clusters = 2
km = KMeans(n_clusters=number_of_clusters)
km.fit(X)
print("Top terms per cluster:")
order_centroids = km.cluster_centers_.argsort()[:, ::-1]
terms = tfidf_vectorizer.get_feature_names()
for i in range(number_of_clusters):
    top_ten_words = [terms[ind] for ind in order_centroids[i, :5]]
    print("Cluster {}: {}".format(i, ' '.join(top_ten_words)))
# i would say people are either writing about harry and hermione or lily and james
# ive never read harry potter, but i think that means either the harry generation or their parents generation?
# seems legit<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11326468806783767, 'fraction_numerical': 0.009085402786190187, 'mean_word_length': 6.214731585518102, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 68, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '361821', 'license': 'Artistic-2.0', 'n_tokens_mistral': 3896, 'n_tokens_neox': 3701, 'n_words': 1158, 'path': '14/14 - TF-IDF Homework.ipynb', 'repo_name': 'spm2164/foundations-homework', 'size': '11557'}