<jupyter_start><jupyter_text>Stylistics and Document ClusteringIn "Stylistic Analysis and Authorship Studies" (2004), <NAME> describes stylistics as an "open-ended and exploratory" approach that "aims to bring to light patterns in style which influence readers' perceptions and relate to the disciplinary concerns of literary and linguistic interpretation." Craig's description recognizes two important and interlinking facets of text: disciplinary modes of interpretation and style. I like the attention to style as a concept that refracts in two directions: the text and the reader's perception of the text. Although these notebook are meant to be focused on writing and using code, it is important to remember how disciplinary knowledge shapes interpretation and, therefore, obscures as much as it illuminates.In this notebook, we'll begin to work with a major method in both stylistic analysis and information retrieval: term frequency-inverse document frequency. Tf-idf tries to account for the problem mentioned in the last notebook, which is raw word counts will disproportionately favor words in longer texts. Tf-idf, instead, tries to weighs how important a word is to the document it appears in. Importance is measured by the word's frequency divided by the number of documents that contain the word. The [Wikipedia entry for tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) is quite clear if you'd like a longer explanation. For a more detailed explanation of tf-idf and an alternative Python script, look at <NAME>'s "[Analyzing Documents with TF-IDF](https://programminghistorian.org/en/lessons/analyzing-documents-with-tfidf)" at the Programming Historian.With our text data transformed to numerical representations (tf-idf), we can then use that information to cluster texts and try to discover general patterns of real and fake news. To do this, we'll use principal component analysis. Principal Component Analysis (PCA), which is also explained well on [Wikipedia](https://en.wikipedia.org/wiki/Principal_component_analysis), is a method of reducing data to its most significant dimensions. How is a dimension of data considered significant, though? Dimensions of data measure the variation within that data. If the variation between data values is high, then that dimension could be deemed significant. If the variation is low—that is each value does not differ much from the other values—then could be considered less significant.Although we're starting to wade into more advanced methods, there are two simple takeaways that tf-idf and PCA try to get at. The first is probably intuitive at this point: the words in a document say something about that document, such as its subject matter or style. We can then extrapolate out from one text and compare the style of many texts to learn about latent cultural patterns and trends. Tf-idf happens to be a very good way to accomplish the first task while PCA is good at the second. Goals1. Transform text to numerical information with sklearn's TfidfVectorizer.2. Demonstrate text clustering with principal component analysis (PCA).<jupyter_code>import re
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import PCA

import warnings
warnings.filterwarnings('ignore')

abs_dir = "/Users/williamquinn/Desktop/DH/Python/Teaching/Python-Notebooks/"
data = pd.read_csv(abs_dir + "data/dataframe.csv",
                   sep = ",")

print (data.shape)
data.head()<jupyter_output>(57887, 6)<jupyter_text>Before we transform our text to data, let's first clean some of it. For example, we might be worried that the publication byline ("WASHINGTON (Reuters)") is so repetitive that it will become the key distinction between real and fake news.  Routine Text Normalization and CleaningWe also want to lower our text (.str.lower()) and remove extra whitespace (\s+) so that words with varying capitalization aren't treated as different words.<jupyter_code># We can remove the byline, remove any extra whitespace, and lower case the text.
data['text'] = data['text'] \
    .replace(r"[A-Z/]+ \(\w+\) - ", "", regex=True) \
    .replace(r'\s+', ' ', regex=True) \
    .str.lower() \
    .astype(str)

data.head()<jupyter_output><empty_output><jupyter_text>Text RepresentationTransform text into numerical values (numerical feature vectors).Tf-idf is numerical statistic that indicates the significance of a word within a document. Words that appear frequently in many documents, such as "the," have a low score (close to zero). Words that appear infrequently, such as hapax legomenon, or only in a few documents have a higher score (close to one).<jupyter_code>%%time

# Instantiate a vectorizer.
tfidf = TfidfVectorizer(sublinear_tf=True, 
                        min_df=0.1, 
                        max_df=0.9, 
                        stop_words='english',
                        encoding='latin-1', 
                        ngram_range=(1, 2))

features = tfidf.fit_transform(data['text']).toarray()

# Create labels.
labels = data[str(class_to_predict)]

# Show data size.
features.shape<jupyter_output>CPU times: user 1min 18s, sys: 2.97 s, total: 1min 21s
Wall time: 1min 24s<jupyter_text>You might notice that TfidfVectorizer() looks very similar to the CountVectorizer() from the last notebook. Both have many of the same arguments.```pythonsublinear_tf = True```This argument scales the frequency of terms for a smoother gradation (1 + log(tf)).```pythonmin_df = 0.1, max_df = 0.9```The min and max arguments set the lower and upper bounds of word limits. If a word doesn't appear between these two fractions (10% and 90%) it is ignored.```pythonstop_words='english'```Remove English stopwords.```pythonencoding = 'latin-1'```This argument sets the text string to a specific encoding. When working with texts, you'll find that a string can be encoded in different formats. A common encoding is 'utf-8'. Encoding arguments tend to force a string into a specific encoding format. If there's a character that cannot be changed, it can be ignored.```pythonngram_range = (1, 2)```Ngrams are ranges of words. A single ngram is a single word. A bigram is every sequence of two words. For example, a bigram list of this description would look like:```python[('Ngrams','are'), ('are','ranges'), ('ranges', 'of')]```Bigrams are pairs of words that appear together. Ngrams can refer to any number of ranges. ClusteringClustering is an unsupervised method of grouping data that share similar values. It is unsupervised because we don't provide it labels beforehand. Unsupervised methods try to find groups within data on its own. Of course, how we construct a dataset will have the biggest part in determining what can be grouped within our analysis. But, after corpus construction, an unsupervised method will try to cluster similar entities together.Principal Component Analysis (PCA) is an effective approach when dealing with complex data. Looking at features.shape above, we know that we have ~60,000 documents (rows) each with 202 dimensions (columns)—that's a little over 12 million values. PCA will try to reduce that data to its key dimensions so that we can plot documents into 2D space. As PCA iterates over data, it will try to find the dimensions that express the most variability and, therefore, convey the most distinguishing factors amongst documents.<jupyter_code># Set the number of principal components we want.
pca = PCA(n_components=2)

# Determine principal components in our features matrix.
principalComponents = pca.fit_transform(features)

# Transform principal components to a dataframe format.
principalDf = pd.DataFrame(data = principalComponents,
                           columns = ['PC1', 'PC2'])

# Join principal components with metadata (veracity column in our "data" dataframe).
finalDf = pd.concat([principalDf, data[['veracity']]], axis = 1)

finalDf<jupyter_output><empty_output><jupyter_text>We can now imagine PC1 and PC2 now acting as our x and y coordinates and visualize them on a 2D graph, using color to distinguish between real and fake articles.<jupyter_code>fig = plt.figure(figsize = (8,8))

# Here, we're changing the style, font size, and format of our graph labels and titles.
ax = fig.add_subplot(1, 1, 1) 
ax.set_xlabel('Principal Component 1', fontsize = 15)
ax.set_ylabel('Principal Component 2', fontsize = 15)
ax.set_title('Principal Component Analysis of\nFake/Real News Articles', fontsize = 20)

# Manually set our "targets," the values we want to color.
# Set the colors (red and blue).
targets = ['fake', 'real']
colors = ['r', 'b']

# We'll first iterate through our two lists (targets and colors) binding them together with zip()
for target, color in zip(targets, colors):
#     Then, we'll make a new variable (indicesToKeep), which
    indicesToKeep = finalDf['veracity'] == target
    ax.scatter(finalDf.loc[indicesToKeep, 'PC1'],
               finalDf.loc[indicesToKeep, 'PC2'],
               c = color,
               s = 1)
    
ax.legend(targets)
ax.grid()<jupyter_output><empty_output><jupyter_text>Our visualization suggests that the unsupervised learning has identified a linguistic difference between fake and real news. Again, though, this does not mean we've discovered some inherent difference between real news and fake news. Instead, we're learning something (linguistically) specific about our dataset that we might possible extrapolate to explain larger cultural clues about language and writing. Explained VarianceVisualizations can be misleading, though. To better understand the information reduction of PCA, we can use .explained_variance_ratio. The explained variance tells how much information each component accounts for within the data. Explained variance offers a more precise way of knowing whether the principal components describe a large or small portion of the variation within data.<jupyter_code>print (pca.explained_variance_ratio_.sum())

pca.explained_variance_ratio_<jupyter_output>0.06945930850968585
{'fraction_non_alphanumeric': 0.07041695553134594, 'fraction_numerical': 0.010002971179558284, 'mean_word_length': 4.660313901345291, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '433981', 'license': 'MIT', 'n_tokens_mistral': 2742, 'n_tokens_neox': 2532, 'n_words': 1286, 'path': '08_Stylistics-Clustering.ipynb', 'repo_name': 'billquinn/Teaching-Python', 'size': '10111'}