<jupyter_start><jupyter_text>Exploring and Predicting Characteristics of Japanese Newspaper Headlines    STA208 Final Project (Spring 2017)    <NAME> and <NAME>  [ Back to Summary Notebook ](STA208_Project_Summary.ipynb) Summary Notebook  I. Research QuestionUsing unique dataset of Japanese Newspaper Headlines, this project asks two questions: * Can unsupervised-learning methods identify the major categories of news appeared on headlines? * How good supervised-learning methods are to predict positive/negative (PN) sentiments expressed in news headlines?This summary notebook does not include all descriptions and results of our analysis. Click on corresponding section titles to see detailed contents in each section (if link is provided). If links do not work, all relevant jupyter notebook files are store in the same directory as this summary notebook file (/208-final-project-liu_and_kato\Political_Headlines_Project\notebooks).<!--- * What are the impact of political news on public opinion (e.g., prime minister approval)?--->  II. [Data of Japanese Newspaper Headlines](STA208_Data_Description.ipynb)Detailed introduction of data structure can be seen in [HERE](STA208_Data_Description.ipynb) (R is used in the data construction) or by clicking section title. In summary, following data are used in the analysis: * Full texts of *ALL* first page headlines from two major newspapers in Japan (*Yomiuri Shimbun* and *Asahi Shimbun*). The data collection starts on November 1987 and ends on March 2015. (The data are originally collected by the author) * Hand-coded negative sentiments (1 = negative, 0 = positive/neutral) appeared on randomly sampled 1000 headlines.  * Matrix of word appearance frequency from the headlines. <!--- * Dictionary approach to extract political headlines (Tutorial from [HERE](Headline%20Data%20and%20Text%20Search.ipynb)) * Monthly public opinion polls in Japan of the corresponding period.--->  III. Analytical StrategyWe have two major objectives in the analysis, as follows:  1. Explore major categories of news through unsupervised and machine learning. 2. Predict negative sentiments appeared on headlines through supervised machine learning. First, we are interested in identifying major categories of news contents ([Section IV](STA208_Unsupervised_Learning.ipynb)). We don't have pre-defined set of categories, therefore, we impletement **unsupervised learning methods (K-Means and Agglomerative Clustering)** to explore the major categories appeared in this dataset. We then briefly look into generated categories to assess if each of the method is successful in extracting meaningful categories.Second, we are also interested in coding positive-negative sentiments appeared in each headline ([Section V](STA208_Supervised_Learning.ipynb)). In this section, One of the author sample 1000 headlines from the full dataset, and manually code the dichotomous appearance of negative sentiment. Then, we use **supervised learning methods (K Nearest Neighbors, Logit, Linear Discriminant Analysis, Support Verctor Machine with RBF Kernel, Decision Tree, Bagging, Random Forest and Adaboost)** to learn those negative-sentiment coding. We compare the result at the the last to evaluate how different methods perform differently on the learning.<!---* Time-series analysis to assess the impact of news coverage on public opinion.--->  IV. [Exploring Categories of Newspaper Headlines](STA208_Unsupervised_Learning.ipynb)In this section, we apply **unsupervised learning methods to identify major news categories and word appearance patterns** in headlines. Details are presented in separate notebook file. Click [HERE](STA208_Unsupervised_Learning.ipynb) or section title to see the contents.  V. [Predicting Negative Sentiments in Newspaper Headlines](STA208_Supervised_Learning.ipynb)In this section, we apply various **supervised learning methods to predict negative sentiments** appeared in headlines, and compare their performances. Details are presented in separate notebook file. Click [HERE](STA208_Supervised_Learning.ipynb) or section title to see the contents.  VI. DiscussionIn this section, we discuss the relationship between different variables generated from analyses in previous sections, and suggest potential future development of the project. Click on the following button to turn on off the raw codes for generating tables.<jupyter_code>from IPython.display import HTML

HTML('''<script>
code_show=true; 
function code_toggle() {
 if (code_show){
 $('div.input').hide();
 } else {
 $('div.input').show();
 }
 code_show = !code_show
} 
$( document ).ready(code_toggle);
</script>
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>''')
# Computation Timer
from timeit import default_timer as trec

## Data Mining
import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')<jupyter_output><empty_output><jupyter_text>1. Overview of Generated VariablesTo start with, in the analysis in previous sections major variables of headline characteristics are shown as follows. (Given its highly unbalanced and non-intuitive nature, results from hierarchical clustering is omitted.)** Headline Topic Clusters from Unsupervised Learning** ([Section IV](STA208_Unsupervised_Learning.ipynb))* km_7catr_name: Topical categories geneated from *K-means* with $k=7$ on word reduced data. * km_8cat_name: Topical categories geneated from *K-means* with $k=8$ on word reduced data. * km_9cat_name: Topical categories geneated from *K-means* with $k=9$ on word reduced data. ** Negative Sentiments Probability from Supervised Learning** ([Section V](STA208_Supervised_Learning.ipynb))* rf_pred: Negative sentiment probability generated from *Random Forest*. * logit_pred: Negative sentiment probability generated from *Logistic Regression*.   2. Relationship between VariablesHere we cross-section generated variables to assess their relationships, and discuss if the relationship between those variables show expected patterns.  **Word Clusters by Topics**<jupyter_code>## Import Data
alldata = pd.read_csv("../../data/alldata_codepred_170611.csv", encoding='CP932')<jupyter_output><empty_output><jupyter_text>**Negative Sentiments by Topics**Here, we summarize results from both unsupervised and supervised machine learning by mean comparison. Following tables show **average predicted probabilities of negative sentiments by topical categories** generated from K-means. Higher probability indicates that headlines fall into corresponding topic is more likely to involve negative sentiments on average.<jupyter_code>alldata.groupby(['km_7catr_name']).agg({'logit_pred': "mean",'rf_pred': "mean"}).transpose().round(3)
alldata.groupby(['km_8catr_name']).agg({'logit_pred': "mean",'rf_pred': "mean"}).transpose().round(3)
alldata.groupby(['km_9catr_name']).agg({'logit_pred': "mean",'rf_pred': "mean"}).transpose().round(3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07640960090896179, 'fraction_numerical': 0.013350376366993325, 'mean_word_length': 6.098790322580645, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '430221', 'license': 'MIT', 'n_tokens_mistral': 1906, 'n_tokens_neox': 1710, 'n_words': 802, 'path': 'notebooks/.ipynb_checkpoints/STA208_Project_Summary-checkpoint.ipynb', 'repo_name': 'gentok/Political_Headlines_Project', 'size': '7051'}