<jupyter_start><jupyter_text>Our DataSo, we are looking at journalistic articles scrapped from the internet. I scrapped a bunch of articles, namely all the magazine articles from 2010 to 2018 that are published in The Atlantic.My wish is to eventually analyze several other sites. While many other people have mined Twitter for content, analyzing journalistic sites may allow us to get another perspective of certain trends.After scrapping content of text, we store text in various .txt files. We load them up here. This is done in another notebook.<jupyter_code>import os
import re
from collections import defaultdict

import pandas as pd

data = defaultdict(list)
addresses = []

# When we grab the date from the URL. It is hidden in the form '{year}/{month}`.
date_pattern = re.compile(r'\d{4}/\d{2}')

for file in os.listdir('articles/'):
    with open('articles/{}'.format(file), 'r') as f:
        address = f.readlines()[0]
        if address not in addresses:
            addresses.append(address)
            f.seek(0)
            data['text'].append(' '.join(f.readlines()[2:]))
            try:
                d = date_pattern.search(address).group(0)
                data['date'].append(d)
            except:
                print("There was an error processing the following address: {}".format(address))
                print("This was located in the file: {}".format(f))
                date.append(None)
        else:
            print("Article at {} already loaded. Skipping.".format(address))

df = pd.DataFrame(data=data)

# Add a month and year column.
df['year'] = pd.DatetimeIndex(df['date']).year
df['month'] = pd.DatetimeIndex(df['date']).month<jupyter_output>Article at https://www.theatlantic.com/magazine/archive/2019/03/how-humans-tamed-themselves/580447/
 already loaded. Skipping.
Article at https://www.theatlantic.com/magazine/archive/2019/03/how-humans-tamed-themselves/580447/
 already loaded. Skipping.
Article at https://www.theatlantic.com/magazine/archive/2019/03/how-humans-tamed-themselves/580447/
 already loaded. Skipping.
Article at https://www.theatlantic.com/magazine/archive/2019/03/bryan-singers-accusers-speak-out/580462/
 already loaded. Skipping.
Article at https://www.theatlantic.com/magazine/archive/2019/03/how-humans-tamed-themselves/580447/
 already loaded. Skipping.
Article at https://www.theatlantic.com/magazine/archive/2019/03/how-humans-tamed-themselves/580447/
 already loaded. Skipping.
Article at https://www.theatlantic.com/magazine/archive/2019/03/how-humans-tamed-themselves/580447/
 already loaded. Skipping.
Article at https://www.theatlantic.com/magazine/archive/2019/03/bryan-singers-accusers-speak-out/580462/
 [...]<jupyter_text>Taking a look at the head.<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Year by Year Trends<jupyter_code>import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

from bokeh.plotting import figure, show
from bokeh.io import output_notebook

import plotly.offline as py
py.init_notebook_mode(connected=True)
import plotly.graph_objs as go
import plotly.tools as tls

def plot_frequent_word(count_vec, subtitle, feature_names):
    zipped = list(zip(feature_names, count_vec))
    x, y = (list(x) for x in zip(*sorted(zipped, key=lambda x: x[1], reverse=True)))
    X = np.concatenate([x[0:15], x[-16:-1]])
    Y = np.concatenate([y[0:15], y[-16:-1]])
    # Plotting the Plot.ly plot for the Top 50 word frequencies
    data = [go.Bar(
                x = x[0:50],
                y = y[0:50],
                marker= dict(colorscale='Jet',
                             color = y[0:50]
                            ),
                text='Tfidf Value'
        )]
    layout = go.Layout(
    title='Top 50 Tfidf frequencies (%s)' % (subtitle)
    )

    fig = go.Figure(data=data, layout=layout)

    py.iplot(fig, filename='basic-bar')
text_by_year = dict()

for year in range(2010, 2018 + 1):
    text_by_year[year] = df[df['year']==year]['text'].values
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

tf_vectorizer = dict()
tf = dict()
feature_names = dict()
count_vec = dict()

tf_vectorizer = dict()
tf = dict()
feature_names = dict()
count_vec = dict()

for year in range(2010, 2018 + 1):
    txt = text_by_year[year]

    tf_vectorizer[year] = TfidfVectorizer(max_df=0.95,
                                        min_df=2,
                                        stop_words='english',
                                        decode_error='ignore')

    tf[year] = tf_vectorizer[year].fit_transform(txt)

    feature_names[year] = tf_vectorizer[year].get_feature_names()

    count_vec[year] = np.asarray(tf[year].sum(axis=0)).ravel()
    
    plot_frequent_word(count_vec[year], "Year {}".format(year), feature_names[year])<jupyter_output><empty_output><jupyter_text>For the most part, most of the words seem like just noise, which could be used in almost any context.There are a few interesting notes.    1. ‘Trump’ is the top word used in 2017, which is the only year which the top word is something meaningful. Trump was demoted to the top 7th word in 2018.    2. ‘American’ becomes one of the top words in almost every year. Being an American publication, it makes American affairs makes sense to be a main concern.  Latent Dirichlet AllocationLet’s use LDA to get an idea of what topics are of importance from year to year.<jupyter_code>def print_top_words(model, feature_names, n_top_words):
    for topic_idx, topic in enumerate(model.components_):
        message = "Topic #%d: " % topic_idx
        message += " ".join([feature_names[i]
                             for i in topic.argsort()[:-n_top_words - 1:-1]])
        print(message)
from sklearn.decomposition import NMF, LatentDirichletAllocation

lda = LatentDirichletAllocation(n_components=5, max_iter=5,
                                learning_method='online',
                                learning_offset=50.,
                                random_state=0)

for year in range(2010, 2018 + 1):
    lda.fit(tf[year])

    print("For the year of {}, the top five topics:".format(year))
    print_top_words(lda, feature_names[year], 20)
    print("")<jupyter_output>For the year of 2010, the top five topics:
Topic #0: cradle fair flimsy rejoice blessed gown gilded brass cloudy enchanted mugs tattered swoop icons congregation 1890s salvage scoop bare clutch
Topic #1: like people women time new said just years year says men way life world garrison surf love american percent davies
Topic #2: canal suez egyptians egyptian egypt fee red attacks ships pirates assertions says modern subsidize terrorists gulf numbers sea staged desert
Topic #3: passage arcades warming climate global des dyson freeman greenhouse paris 1998 passages july businesses davies november 2009 shopping contrarian shop
Topic #4: tools donald biographies animals gimbel sheet loyal regenerative clatter awaiting times sonic flavor complaint play stopping grimly plastic unthinkable brush

For the year of 2011, the top five topics:
Topic #0: sanguine bites flock replicas meaningfully gathering vice moines playing pensions hone pizza biologist defends palo restrictive octopus deliberation[...]<jupyter_text>A couple of interesting things.* The number one topic in 2015 is on the Middle East politics. This might be due to a huge escalation of the Syrian civil war in that time. We can see a large increase in ‘Syria’ over time.* Trump is the 1 topic in 2016 and 2017. This was not the case at all in Obama in 2012 and 2013, which we shall show again later.Not all the topics listed are meaningful. One issue I’ve always had with LDA, is it sometimes captures a lot of noise.  Keyword Usage Over Time<jupyter_code>def word_count(year, word):
    ind = feature_names[year].index(word)
    return count_vec[year][ind]
import matplotlib.pyplot as plt

YEARS = range(2010, 2018 + 1)

words = ['trump', 'obama', 'bush', 'president']

x = np.array(YEARS)
word_counts = defaultdict(list)

for word in words:
    for year in YEARS:
        word_counts[word].append(word_count(year, word))

plt.figure(figsize=(16, 8))

for word in words:
    y = np.array(word_counts[word])
    
    plt.plot(x, y)
    
plt.title("Keyword Mentions over Time")
plt.legend(words)<jupyter_output><empty_output><jupyter_text>As mentioned before, Trump is way more of a hot topic in his heyday compare to Trump.<jupyter_code>words = ['syria', 'russia', 'iran', 'iraq']

x = np.array(YEARS)
word_counts = defaultdict(list)

for word in words:
    for year in YEARS:
        word_counts[word].append(word_count(year, word))

plt.figure(figsize=(16, 8))

for word in words:
    y = np.array(word_counts[word])
    
    plt.plot(x, y)
    
plt.title("Keyword Mentions over Time")
plt.legend(words)<jupyter_output><empty_output><jupyter_text>Note that I included Russia, due to their frequent involvement with politics there. It seems to pick up more on probably the recent Russian scandal.<jupyter_code>words = ['facebook', 'twitter', 'microsoft', 'amazon']

x = np.array(YEARS)
word_counts = defaultdict(list)

for word in words:
    for year in YEARS:
        word_counts[word].append(word_count(year, word))

plt.figure(figsize=(16, 8))

for word in words:
    y = np.array(word_counts[word])
    
    plt.plot(x, y)

plt.title("Keyword Mentions over Time")
plt.legend(words)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10264830508474576, 'fraction_numerical': 0.02680084745762712, 'mean_word_length': 3.5564671814671813, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 8, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '281650', 'license': 'MIT', 'n_tokens_mistral': 3149, 'n_tokens_neox': 2860, 'n_words': 923, 'path': 'notebooks/the-atlantic-analysis.ipynb', 'repo_name': 'frankkloster/tdi-capstone-project', 'size': '9440'}