<jupyter_start><jupyter_text>Final Project: Analyzing Online Popularity of Mashable Articles<NAME>University of Washington - DATA 512  Introduction MotivationThis report analyzes articles published by Mashable in a period of two years. The goal of this exploratory analysis is to determine which types of articles are more popular on online platforms. Through this analysis, I want to understand if there are certain criteria for maximizing the impact of an article in terms of shares. This is an interesting topic because it is known that social media plays a role in changing people's perspectives and has a significant influencing power. It would be interesting to see what types of articles receive more shares. From this analysis, I hope to learn more about the trends in news sharing and whether changing certain aspects of an article can attract more shares.  DataThe dataset I plan to use is the Online News Popularity Data Set from the UCI Machine Learning Repository.  - [Dataset](http://archive.ics.uci.edu/ml/datasets/Online+News+Popularity)This dataset has 61 features which makes it a good multivariate dataset for exploratory analysis. It also has 39797 rows or articles for analysis. There is no license available for the dataset. The dataset does not contain any content from the aricles themselves, but instead only specific statistics related to each article that were separately compiled. The articles themselves are not publicly available, therefore cannot be directly analyzed. The terms of use for this data are explicity explained [here](http://archive.ics.uci.edu/ml/datasets/Online+News+Popularity) and the citation is shown below. - <NAME>, <NAME> and <NAME>. A Proactive Intelligent Decision Support System for Predicting the Popularity of Online News. Proceedings of the 17th EPIA 2015 - Portuguese Conference on Artificial Intelligence, September, Coimbra, Portugal. This dataset contains many features that address my points of interest such as worst keyword, best keyword, day article was published, negative word count, postive word count, total shares. There is a lof of potential to find trends for aricles with high or low shares with these features. It also has a large sample of articles so that the trends found would have statistical value. Some of the ethical considerations in using this dataset are that the word classfication is dependant on the author, therefore it contains a certain level of bias. Also, there are subjective features like best and worst word which are also subject to bias since they are from the author's perspective.  UnknownsThe data is in a downloadable format and easily accessible. It comes in a clean and formatted csv file. In order to carry out the analysis, some data transformations will be required and outlined in the report as needed.  Background/Related WorkOnline news sharing is a common practice as people are more and more addicted to their phones. As shown in the Netflix film, The Social Dilemma, polarized news makes its way into people's news feed through complex algorithms in such a way that cannot be monitored or controlled. This becomes a serious problem in the spreading of fake news. There is a lot of responsibility on the part of news creators to be responsible in their writing so that they do not become pawns of these sophisticated algorigthms. Watching this movie made me curious about what factors influence sharing of articles. I shaped my reseach questions to explore different facets of the articles to determine whether they influence the number of shares. This analysis would reveal how authors can shape their content to attract more audiences while still maintaining a non-polar stance. The Journal of Communication article, "The Social News Gap: How News Reading and News Sharing Diverge" by <NAME> talks about the same issue of polarization in online news [[1]](https://doi.org/10.1111/jcom.12232). He explains how news editors have immense power in shaping the wave of social media based on their approach to news writing and editing [[1]](https://doi.org/10.1111/jcom.12232). My research questions are designed to understand which aspects of articles are within the power of news editors and which are not.   Research questions1. Does the theme of the news article affect the number of shares on an online platform? 2. Which theme is gaining in online popularity? 3. Is there a relationship between the polarity of the article and the number of shares it recieves?4. Does the day an article was published impact the amount of shares it receives?  Methodology 1. Does the theme of the news article affect the number of shares on an online platform? To answer this question, I will conduct an exploratory data analysis on the theme features of the dataset. These features are indicator variables. Aggregating the results for each theme with visual plots will help me understand which themes recieve more shares. 2. Which theme is gaining in online popularity? This question relates to the above question. If the theme does not have significance in number of shares, then this question does not have a definite answer. However, comparing the themes side-by-side can reveal if a particular theme is more effective in gaining online popularity over time. Plotting all the themes together over a common time axis will help  me understand a major aspect of online news popularity. 3. Is there a relationship between the polarity of the article and the number of shares it recieves?To delve into this research question, I will analyze the polarity features in the dataset. Analyzing these features will reveal insights on the role of polarization in news sharing. It will help me understand whether the tone of an article affects how namy times it is shared. 4. Does the day an article was published impact the amount of shares it receives? This dataset contains information on which day of the week an article was shared. To answer this question, I can create a feature that groups articles into publish day based on the seven days of the week. Plotting number of shares based on this feature will help me understand whether publishers can use this feature as a marketing decision for attracting higher number of shares.  Analysis  Question 1: Does the theme of the news article affect the number of shares on an online platform? The intent of this question is to understand whether some themes generate more shares than others. The output of this analysis will be a box plot that shows the spread of articles and the average number of shares per theme. The following modules/packages from python must be imported to re-create the analysis and charts/plots.<jupyter_code># Import all packages for plotting/analysis
import pandas as pd
import seaborn as sns
from matplotlib import pyplot as plt
import numpy as np
from scipy import stats
import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>The dataframe below is a snapshot of the raw data imported from the csv file. This dataframe will be copied into several other dataframes for each question/analysis.<jupyter_code># Import the raw data in a pandas dataframe
df = pd.read_csv("OnlineNewsPopularity.csv")
df<jupyter_output><empty_output><jupyter_text>Question 1 uses the below features from the raw data. df1 is a filtered dataframe with only relevant features and the target variable: shares.<jupyter_code># Filter out irrelevant columns from df and save as df1
df1 = df[[' data_channel_is_lifestyle',' data_channel_is_entertainment', ' data_channel_is_bus',
       ' data_channel_is_socmed', ' data_channel_is_tech',
       ' data_channel_is_world',' shares']]
df1<jupyter_output><empty_output><jupyter_text>In order to aggregate the data, it must be pivoted so that each feature in df1 is condensed into a single column called 'theme.'<jupyter_code># Pivot each feature into a new column in df1 called 'theme'
df1['theme'] = np.where(df1[' data_channel_is_lifestyle']==1.0, 'lifestyle', 'Other')
df1['theme'] = np.where(df1[' data_channel_is_entertainment']==1.0, 'entertainment', df1['theme'])
df1['theme'] = np.where(df1[' data_channel_is_bus']==1.0, 'business', df1['theme'])
df1['theme'] = np.where(df1[' data_channel_is_socmed']==1.0, 'socmed', df1['theme'])
df1['theme'] = np.where(df1[' data_channel_is_tech']==1.0, 'tech', df1['theme'])
df1['theme'] = np.where(df1[' data_channel_is_world']==1.0, 'world', df1['theme'])
df1<jupyter_output><empty_output><jupyter_text>The dataframe is then transformed into groupings by theme, shares, and count of articles. This is done via the groupby method of aggregating sum of shares and article counts.<jupyter_code># Group by theme. Aggregate by sum of shares and count of articles
themes = df1.groupby(['theme'])[' shares'].sum().reset_index()
themesCount = df1.groupby('theme').count().reset_index()
themesCount['count'] = themesCount[' shares']
themesCount = themesCount[['theme', 'count']]
themes = themes.merge(themesCount, on='theme')
themes<jupyter_output><empty_output><jupyter_text>To get the average shares per article for each theme, another column needs to be added that divides shares by total article count.<jupyter_code># Get average shares per article
themes['averageShares'] = themes[' shares']/themes['count']
themes<jupyter_output><empty_output><jupyter_text>There are articles with no theme which were earlier categorized as "Other." In order to get interpretable results, we need to filter out the 'Other' category.<jupyter_code># Filter "Other" category
themes = themes[themes['theme']!='Other']
themes<jupyter_output><empty_output><jupyter_text>Use the seaborn boxplot with showfliers set to 'False' in order to remove outliers and get a clean picture of each theme's results. Save the image in png format.<jupyter_code># Plot and save boxplot.
plt.figure(figsize=(10,10))
sns.boxplot(x=' shares', y='theme', data=df1, showfliers = False)
plt.title("Average Shares per Theme", fontsize=15)
plt.ylabel('Article Theme')
plt.xlabel('Shares')
plt.grid()
plt.savefig('SharesPerTheme.png')<jupyter_output><empty_output><jupyter_text>Question 2: Which theme is gaining in online popularity? The intent of this question is to understand if the same theme's are consistantly popular or if some themes are gaining in popularity over time. The output of this analysis will be a plot showing the peak shares for articles from all themes over a period of 2 years. Below is a filtered version of the raw dataframe df. It contains only columns relevant to this analysis.<jupyter_code># Filter df into df2 with relevant columns
df2 = df[['url',' data_channel_is_lifestyle',' data_channel_is_entertainment', ' data_channel_is_bus',
       ' data_channel_is_socmed', ' data_channel_is_tech',
       ' data_channel_is_world',' shares']]
df2<jupyter_output><empty_output><jupyter_text>Since question 2 requires a time analysis, the date must be parsed from the url column. The code below shows the parsing and formatting of the date column.<jupyter_code># Parse and format date
df2['year'] = df2['url'].str.slice(20,24)
df2['month'] = df2['url'].str.slice(25,27)
df2['day'] = df2['url'].str.slice(28,30)
df2['date'] = df2['year']+df2['month']+df2['day']
df2<jupyter_output><empty_output><jupyter_text>The method below is created to parse the theme from the features in the dataset. It filters the dataframe so that each article has only one theme.<jupyter_code># Define a method to create dataframe for all the articles in each theme
def plotTheme(df0, theme):
    column = ' data_channel_is_'+theme
    final_df = df0[df0[column]==1.0]
    return final_df<jupyter_output><empty_output><jupyter_text>The code below is used to convert the date column into a pandas datetime column type so that it can be plotted on a time axis.<jupyter_code># Convert to datetime
df2['date'] = pd.to_datetime(df2['date'], format='%Y%m%d')
# Use the above method on each theme
lifestyleTheme = plotTheme(df2,'lifestyle')
entertainmentTheme = plotTheme(df2,'entertainment')
busTheme = plotTheme(df2,'bus')
socmedTheme = plotTheme(df2,'socmed')
techTheme = plotTheme(df2,'tech')
worldTheme = plotTheme(df2,'world')<jupyter_output><empty_output><jupyter_text>The plots below show each theme over a common time axis. The density of the plots provide insight into the increasing/decreasing popularity for each theme. The second plot shows peaks of shares for each theme. It can be clearly seen that some themes have larger color prescence in this plot over others indicating their gaining popularity.<jupyter_code># Plot shares over time for each theme.
fig, axs = plt.subplots(6, figsize=(20, 20), sharey=True, sharex=True, constrained_layout=True)
fig.suptitle("Article Shares over Time", fontsize=20)
axs[0].set_ylim([0, 100000])
axs[0].set_title('Lifestyle Articles')
axs[0].plot_date(lifestyleTheme['date'], lifestyleTheme[' shares'], linestyle='-', marker='None', label='Lifestyle', color='red')
axs[1].set_title('Entertainment Articles')
axs[1].plot_date(entertainmentTheme['date'], entertainmentTheme[' shares'], linestyle='-', marker='None', label='Entertainment', color='blue')
axs[2].set_title('Business Articles')
axs[2].plot_date(busTheme['date'], busTheme[' shares'], linestyle='-', marker='None', label='Business', color='orange')
axs[3].set_title('Social Media Articles')
axs[3].plot_date(socmedTheme['date'], socmedTheme[' shares'], linestyle='-', marker='None', label='SocialMedia', color='magenta')
axs[4].set_title('Technology Articles')
axs[4].plot_date(techTheme['date'], techTheme[' shares'], linestyle='-', marker='None', label='Technology', color='green')
axs[5].set_title('World Articles')
axs[5].plot_date(worldTheme['date'], worldTheme[' shares'], linestyle='-', marker='None', label='World', color='purple')
# Plot peaks over time for each theme
plt.figure(figsize=(20,10))
plt.title("Peak Shares over Time", fontsize=20)
plt.xlabel('Date', fontsize=15)
plt.ylabel('Shares', fontsize=15)
plt.plot_date(lifestyleTheme['date'], lifestyleTheme[' shares'], linestyle='-', marker='None', label='Lifestyle')
plt.plot_date(entertainmentTheme['date'], entertainmentTheme[' shares'], linestyle='-', marker='None', label='Entertainment')
plt.plot_date(busTheme['date'], busTheme[' shares'], linestyle='-', marker='None', label='Business')
plt.plot_date(socmedTheme['date'], socmedTheme[' shares'], linestyle='-', marker='None', label='SocialMedia')
plt.plot_date(techTheme['date'], techTheme[' shares'], linestyle='-', marker='None', label='Technology')
plt.plot_date(worldTheme['date'], worldTheme[' shares'], linestyle='-', marker='None', label='World')
plt.legend()
plt.ylim([40000, 100000])
plt.savefig('SharesOverTime.png')<jupyter_output><empty_output><jupyter_text>Question 3: Is there a relationship between the polarity of the article and the number of shares it recieves?The intent for this question is to understand whether the tone of an article impacts whether it recieves more shares. The output for this analysis is a linear regression of positive and negative polarity scores of articles. The below dataframe is a filtered version of the initial dataframe with the columns relevant to this analysis.<jupyter_code># Filter df3
df3 = df[[' avg_positive_polarity',' avg_negative_polarity', ' shares']]<jupyter_output><empty_output><jupyter_text>The rows must be filtered to remove any nuetral polarity scores for the articles. Since the question seeks to understand how postive and negative polarity affects sharing, the dataframe needs to be filtered.<jupyter_code># Remove any articles with nuetral polarity scores of 0.0
polarArticles = df3[df3[' avg_positive_polarity']!=0.0]
polarArticles = df3[df3[' avg_negative_polarity']!=0.0]
polarArticles<jupyter_output><empty_output><jupyter_text>The following cells plot the linear regression analysis for positive and negative polarity. The slope of the line can be interpreted as the polarity having an affect on the number of shares.<jupyter_code># Plot positive polarity regression
slope, intercept, r_value, p_value, std_err = stats.linregress(polarArticles[' avg_positive_polarity'], polarArticles[' shares'])
plt.figure(figsize=(20,10))
p = sns.regplot(polarArticles[' avg_positive_polarity'], polarArticles[' shares'], scatter=False, color='green', label='Positive Polarity')
plt.title("Article Shares with Positive Polarity", fontsize=20)
plt.xlabel('Average Polarity Score', fontsize=15)
plt.ylabel('Shares', fontsize=15)
plt.ylim([0,15000])
plt.legend(fontsize=15)
plt.savefig('PositivePolarity.png')
# Plot negative polarity regression
slope, intercept, r_value, p_value, std_err = stats.linregress(polarArticles[' avg_negative_polarity'], polarArticles[' shares'])
plt.figure(figsize=(20,10))
p = sns.regplot(polarArticles[' avg_negative_polarity'], polarArticles[' shares'], color='red', label='Negative Polarity', scatter=False)
plt.title("Article Shares with Negative Polarity", fontsize=20)
plt.xlabel('Average Negative Polarity Score', fontsize=15)
plt.ylabel('Shares', fontsize=15)
plt.ylim([0,15000])
plt.legend(fontsize=15)
plt.savefig('NegativePolarity.png')<jupyter_output><empty_output><jupyter_text>Question 4: Does the day an article was published impact the amount of shares it receives?The intent behind this question was to understand if there the number of shares varies based on when an article is published. The output of this analysis is a bar chart that shows average shares per articles based on day of publishing. The dataframe below is a filtered version of the original dataframe with columns relevant to this analysis.<jupyter_code># Filter df4
df4 = df[[' weekday_is_monday', ' weekday_is_tuesday', ' weekday_is_wednesday',
       ' weekday_is_thursday', ' weekday_is_friday', ' weekday_is_saturday',
       ' weekday_is_sunday',' shares']]
df4<jupyter_output><empty_output><jupyter_text>The dataframe must be transformed in order to produce results showing aggregated shares per day of the week. A new column called 'dayOfWeek' is created and each article is grouped into a day based on the feature in the dataset.<jupyter_code># Create and populate column for dayOfWeek
df4['dayOfWeek'] = np.where(df4[' weekday_is_monday']==1.0, 'monday', 'Other')
df4['dayOfWeek'] = np.where(df4[' weekday_is_tuesday']==1.0, 'tuesday', df4['dayOfWeek'])
df4['dayOfWeek'] = np.where(df4[' weekday_is_wednesday']==1.0, 'wednesday', df4['dayOfWeek'])
df4['dayOfWeek'] = np.where(df4[' weekday_is_thursday']==1.0, 'thursday', df4['dayOfWeek'])
df4['dayOfWeek'] = np.where(df4[' weekday_is_friday']==1.0, 'friday', df4['dayOfWeek'])
df4['dayOfWeek'] = np.where(df4[' weekday_is_saturday']==1.0, 'saturday', df4['dayOfWeek'])
df4['dayOfWeek'] = np.where(df4[' weekday_is_sunday']==1.0, 'sunday', df4['dayOfWeek'])<jupyter_output><empty_output><jupyter_text>The dataframe is transformed to group the data by dayOfWeek and aggregate the shares and count of articles. Then a calculation is performed on the data to produce a column for average shares per article for each day.<jupyter_code># Group and aggregate df4
weekday = df4.groupby(['dayOfWeek'])[' shares'].sum().reset_index()
weekdayCount = df4.groupby('dayOfWeek').count().reset_index()
weekdayCount['count'] = weekdayCount[' shares']
weekdayCount = weekdayCount[['dayOfWeek', 'count']]
weekday = weekday.merge(weekdayCount, on='dayOfWeek')
weekday
# Add column for averageShares
weekday['averageShares'] = weekday[' shares']/weekday['count']
weekday<jupyter_output><empty_output><jupyter_text>A seaborn bar plot is used to present the results of this analysis. The order of the days is manually sorted with the week starting on Mondays. This keeps the visual intuitive.<jupyter_code># Plot and save bar chart
plt.figure(figsize=(10,5))
sns.barplot(x='dayOfWeek', y='averageShares', data=weekday, palette='viridis', order=["monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday"])
plt.title("Average Shares by Day of Week")
plt.xlabel('Day of Week')
plt.ylabel('Shares')
plt.savefig('SharesPerWeekday.png')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0940072850656155, 'fraction_numerical': 0.013871563295244749, 'mean_word_length': 6.277414669571533, 'pattern_counts': {'":': 0, '<': 86, '<?xml version=': 0, '>': 86, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '598759', 'license': 'MIT', 'n_tokens_mistral': 5783, 'n_tokens_neox': 5467, 'n_words': 2336, 'path': 'A7_AnalyzingOnlineArticleShares.ipynb', 'repo_name': 'pribij/data-512-final', 'size': '20073'}