<jupyter_start><jupyter_text>Notes: Date appears to be available in link so I will do a quick fill-in with that<jupyter_code># quick replacement
df.iloc[102,1]='2020-04-27'
# convert date column to date time
df.date = pd.to_datetime(df.date,errors='coerce')
df.dtypes
df.describe(include='all')
#check for duplicates
df[df.article_title.duplicated(False) == True]<jupyter_output><empty_output><jupyter_text>Notes: 45-url no longer goes to valid article so I will drop30/60 are the same article with the same authors (maybe bringing in author is available could also help identify duplicates)available from 2 different publishers (based on the notes the authors are listed with ap so I will keep that one in this case)71/79 are same article with same publisher just variations on the url link so I will keep the one with the earliest date Notes:While I am dropping the duplicated articles in this notebook, there is a larger question on how to handle them. Are we trying to understand unique articles or if an article is listed under more than one publisher do we want to consider it twice given it may have greater reach. Things to think about and discuss with partner.<jupyter_code># drop determined duplicates
df.drop(index=[45,30,71],inplace=True)
df.describe(include="all")
fig, ax = plt.subplots(figsize=(10, 6))
df.groupby(df.date.dt.date).size().plot(style="o")
ax.set_ylabel("count")
ax.set_title("All articles")
plt.show()<jupyter_output><empty_output><jupyter_text>Notes: The current dataset looks very limited to more recent articles<jupyter_code>pzifer=df[df.article_title.str.lower().str.contains('pfizer')]
fig, ax = plt.subplots(figsize=(10, 6))
pzifer.groupby(pzifer.date.dt.date).size().plot(style="o")
ax.set_ylabel("count")
ax.set_title("Articles containing pfizer")
ax.set_ylim(0)
ax.set_yticks([0, 1, 2, 3, 4])
plt.show()
df[df.article_title.str.lower().str.contains('politico')]<jupyter_output><empty_output><jupyter_text>Notes: There are some article titles that contain a heading for the publisher that could be removed<jupyter_code># Import/download nltk packages
from nltk.tokenize import word_tokenize
from nltk.tokenize import sent_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from nltk.stem import PorterStemmer
from collections import Counter
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')
# User-Entered Parameters
textcolumn = "article_title"
lowercase = True 
strip_non_alpha = True
remove_stopwords = True
use_default_stopwords = False
user_defined_stopwords = ["vaccine","to","for","in","of","the","coronavirus","s","on","vaccines","and","covid","is","by","a","as","at","from","with","are","vaccination","its","PoliticoPlaybook","get","says"]
lemmatize_stem_algorithm = "none" #other options are "porter" and "wordnet"
subs = {'<NAME>': 'JohnsonandJohnson',
        'White House': 'WhiteHouse',
        '<NAME>':'BorisJohnson',
        'J&J': 'JohnsonandJohnson',
        'POLITICO Playbook': 'PoliticoPlaybook',
        'West Virginia': 'WestVirginia'}
number_top_words = 50
number_top_bigrams = 15<jupyter_output><empty_output><jupyter_text>Notes: In the first round I noticed that there are certain words that we would want to avoid splitting up which for a quick fix I added to subs. In the future probably want to do some NER to better identify and adjust:(('johnson', 'johnson'), 13), (('boris', 'johnson'), 11), (('j', 'j'), 8), (('politico', 'playbook'), 5), (('white', 'house'), 4),<jupyter_code># list English stopwords
print(stopwords.words('english'))<jupyter_output>['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', '[...]<jupyter_text>Notes:The typical stopwords would remove many verbs which we need to understand sentiment and also we need to consider contractions to understand the not cases of verbs<jupyter_code># Create list of text column values
doc = pd.Series.tolist(df[textcolumn])

# Create sentence and word tokens using NLTK tokenizers
sentences = [sent_tokenize(i) for i in doc]
tokenized_sentences = [word_tokenize(i) for i in doc]
words= []
for s in sentences:
    for from_, to in subs.items():
        s[0] = re.sub(from_, to, s[0], flags=re.IGNORECASE)
        for i in s:
            words += word_tokenize(i)  
        
# Lowercase word tokens
if lowercase == True:
    words = [w.lower() for w in words]
    tokenized_sentences = [[w.lower() for w in s] for s in tokenized_sentences]
    
# Strip non-alphabetic tokens
if strip_non_alpha == True:
    words = [w for w in words if w.isalpha()]
    tokenized_sentences = [[w for w in s if w.isalpha()] for s in tokenized_sentences]

# Remove stopwords from tokens
if use_default_stopwords == True:
    stopwords_all = stopwords.words('english')+user_defined_stopwords
    stopwords_all = [sw for sw in stopwords_all if sw not in ["not","no"]] #modification to keep in negatives
else:
    stopwords_all = user_defined_stopwords    

if remove_stopwords == True:
    words = [w for w in words if w not in stopwords_all]
    tokenized_sentences = [[w for w in s if w not in stopwords_all] for s in tokenized_sentences]
       
# Stem/lemmatize word tokens
if lemmatize_stem_algorithm == "wordnet":
    wordnet_lemmatizer = WordNetLemmatizer() 
    words = [wordnet_lemmatizer.lemmatize(w) for w in words]
    tokenized_sentences = [[wordnet_lemmatizer.lemmatize(w) for w in s] for s in tokenized_sentences]
elif lemmatize_stem_algorithm == "porter":
    porter = PorterStemmer() 
    words = [porter.stem(w) for w in words]
    tokenized_sentences = [[porter.stem(w) for w in s] for s in tokenized_sentences]
else:
    words = words
    
#print("Some tokenized sentences:",tokenized_sentences[:4])
#print("\nSome sentences:",sentences[:4])
#print("\nSome words:",words[:15])
# Top Words List
counts = Counter(words) #count words
counts.most_common(number_top_words) #display most common
# Top Words Cloud
wordcloud = WordCloud(background_color="white",max_words=number_top_words)
wordcloud.fit_words(counts)

plt.figure()
plt.axis("off")
plt.imshow(wordcloud,interpolation='bilinear')
plt.show()
# Top Bigrams
counts = Counter(zip(words, words[1:])) #count words
counts.most_common(number_top_bigrams) #display most common<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15286092162049283, 'fraction_numerical': 0.008213838229152165, 'mean_word_length': 5.714018691588785, 'pattern_counts': {'":': 2, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '504540', 'license': 'MIT', 'n_tokens_mistral': 2483, 'n_tokens_neox': 2351, 'n_words': 748, 'path': 'notebooks/newsheadlinesdata_eda_camille.ipynb', 'repo_name': 'datakind/Mar21-vaccine-uptake', 'size': '7201'}