<jupyter_start><jupyter_text>Counting & VisualizationsIt's easy to consider visualization as the final phase or product of text analysis. But, it's often the first and dominant mode we encounter data. Imagining what visualizations you might make from data helps to determine what that data can and cannot show. Visualization should be considered at every phase of data analysis. Goals1. Learn the basics of visualization libraries of Python (seaborn, matplot, plotly).First, we'll read in our data and make sure everything saved properly from the last notebook.<jupyter_code>import re
import pandas as pd

import seaborn as sns
import matplotlib.pyplot as plt

abs_dir = "/Users/williamquinn/Desktop/DH/Python/Teaching/Python-Notebooks/"
data = pd.read_csv(abs_dir + "data/dataframe.csv",
                   sep = ",")

# We're going to print multiple things here, including a '\n', which is a newline.
print (data['veracity'].unique(), '\n', data.shape)
data.head()<jupyter_output>['real' 'fake'] 
 (57887, 6)<jupyter_text>Simple CountingAs we can see in our dataframe, we don't yet have anything to count. That is, there are no columns that are numerical data. Before we can visualize our data, we need to create a new field (or column). This is where data analysis calls for creativity and practice. Looking at our data, we could imagine counting a few different things. For starters, let's first count how our dataset splits between fake and real news.<jupyter_code>counts = data \
    .groupby(['veracity']) \
    .size() \
    .reset_index()

print (counts)

counts = counts.rename(columns = {"veracity":"veracity", 0:"count"})

counts<jupyter_output>veracity      0
0     fake  36470
1     real  21417<jupyter_text>There are a few steps happening quickly in the cell above. We'll walk through them as they appear. counts = data \This should look mostly familiar. We're creating a new variable called "counts." To create that variable, we're first starting with our dataframe, "data." The perhaps new feature is that slash at the end. The slash allows us to write each function on its own line. This makes writing and reading code more readable. The use of slash and indents allows us to visualize the sequence of functions more clearly. .groupby(['veracity']).groupby() takes specified columns and groups them together so that you can perform functions on the different values that appear within that column. Here, we're taking the "veracity" column only and telling the following functions to distinguish between the "real" and "fake" values within the column. The square brackets within the parenthesis allow you to group with a list. Later, we can take the "veracity" and "date" column together. .size()  The .size function counts the number of times an instance occurs. For example, as we can see in our results, there are 23,481 fake articles and 21,417 real articles. .size() counts the number of times each value of our group appears. .reset_index()Dataframes in pandas retain an index throughout the functions we call. This tends to return dataframes that don't quite work the way we expect. For instance, we might get a dataframe with multiple levels, which can make it difficult to run functions. .reset_index(), as the name implies, resets the index and returns a dataframe that behaves in more familiar ways.```pythonprint (counts)```I've decided to print counts here to show how the column names have changed. Although we've effectively counted the values within each group, we never told pandas what that column's name should be. It returns zero (0) by default.```pythoncounts = counts.rename(columns = {"veracity":"veracity", 0:"count"})```We might recognize that we're overriding "counts" with a modified version of "counts." We're also calling the .rename() function to change the column names. Within the rename function, there is columns = {}. This acts just like a variable. We're telling .rename what the columns should be called. The {} looks like a dictionary with keys and values. We can re-read this dictionary like object as ```python{'old-column-name':'new-column-name',  'another-column-old-name':'another-column-new-name'}.```You can add as many columns as you need. They don't need to be in any sort of order because of the key-value structure.   Simple VisualizationWe have very simple data at the moment and don't necessarily need to use a visualization. But, our 2x2 dataframe will allow us to learn the very basics of Python visualization. We'll start with seaborn and matplot, which we've already import above.<jupyter_code>ax = sns.barplot(data=counts, x="veracity", y="count", hue="veracity")<jupyter_output><empty_output><jupyter_text>The simple bar chart above conveys even less than the table we produced above. But, we can start to break down the essential components of seaborn. sns.barplot()Similar to our use of pd instead of pandas, we have sns to specify which barplot() function we want. There are other charts we could've chosen within seaborn, and it could be useful to browse them online. data = countsHere, we have our first argument, which simply tells seaborn where to find the x, y, and hue arguments that we call later. x = "veracity", y = "count"These two arguments inform seaborn how to organize the graph. "x" is our x-axis, and "y" is our y-axis. Seaborn is very helpful because it recognizes the difference between categorical variables ("fake" and "real") and continuous variables (integers and numbers). We could make the graph horizontal by writing: x = "count", y = "veracity". hue = "veracity"Hue determines how to color each value.Once we tell seaborn what our data is, we then use column names to fill out the rest of the arguments to create our graph.When you're comfortable with these basics, you can start to add more details. For instance, the legend in the top right corner is not ideal. Try searching the documentation (do a quick internet search for "python seaborn barchart") for how to place the legend somewhere else.  More Complicated CountingUsing the basics of simple counting and visualization above, we can start to create graphs that convey more information. For example, we might want to find trends of fake articles over time. In other words, were there certain dates when fake news articles were more common?To answer this question, we can use our grouping method from earlier.<jupyter_code># I'm using .to_frame() to name the new field created by size().
trends = data \
    .groupby(['date', 'veracity']) \
    .size() \
    .to_frame('count') \
    .reset_index()

trends<jupyter_output><empty_output><jupyter_text>At the moment, we have the number of news articles (real and fake) published each day. While this is very specific, it might be too exact. It might be more useful to sum the number of articles published each year.<jupyter_code># We'll first create a new column for the year using .DatetimeIndex() to extract the year.
trends['year'] = pd.DatetimeIndex(trends['date']).year

print (trends.head())

# We'll then aggregrate the count for each year following a similar approach as above.
# By the way, when you write over a dataframe, you might have to restart the notebook over if you make a mistake.
yearly_trends = trends \
    .groupby(['year', 'veracity']) \
    .size() \
    .to_frame('count') \
    .reset_index()

yearly_trends<jupyter_output>date veracity  count  year
0  2015-03-31     fake      8  2015
1  2015-04-01     fake      2  2015
2  2015-04-02     fake      1  2015
3  2015-04-04     fake      5  2015
4  2015-04-05     fake     13  2015<jupyter_text>More Advance VisualizationsWith our data condensed to yearly rates of fake and real news, we can make visualizations that highlight these trends more effectively.<jupyter_code>ax = sns.barplot(data=yearly_trends, x="year", y="count", hue="veracity")<jupyter_output><empty_output><jupyter_text>While this bar graph is just as simple to create as the one above, we can start to see our data differently. We become instantly aware of how our data breaks down. It's worth asking now (before we start text analysis) whether it matters that we don't have any real news in 2015 or 2018. Do we want to limit our data so that fake and real news discuss concurrent events? Or, do we want to use everything?Thinking about visualization early in a text analysis project can help clarify these holistic questions. I find that it's useful imagining how data might appear in visualizations early as a way to think about what questions are more productive.<jupyter_code>ax = sns.scatterplot(data=trends, x="date", y="count", hue="veracity")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0711224371597359, 'fraction_numerical': 0.013205143055716437, 'mean_word_length': 4.47148288973384, 'pattern_counts': {'":': 2, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '69678', 'license': 'MIT', 'n_tokens_mistral': 2385, 'n_tokens_neox': 2218, 'n_words': 1223, 'path': '06_Counting-Visualizations.ipynb', 'repo_name': 'billquinn/Teaching-Python', 'size': '8633'}