<jupyter_start><jupyter_text>Run Naive Bayes Classifier on the Text Data to Predict Share Price Direction- As opposed to using the existing Naive Bayes Classifier trained on a movie review corpus, we attempted to fit the classifier directly to the articles and corresponding daily share price returns<jupyter_code>import csv
from yahoo_historical import Fetcher
import numpy as np
import pandas as pd

# Naive Bayes Classifier 
from textblob import TextBlob
from textblob.classifiers import NaiveBayesClassifier<jupyter_output><empty_output><jupyter_text>1. Downlaod the article csv file- Change the directory to where the csv file is saved- Extract only relevant data - dates, abstract, headline, paragraph and snippet<jupyter_code>with open('news_data.csv') as csvfile:
    
    readCSV = csv.reader(csvfile)
    
    dates = []
    abstracts = []
    headlines = []
    snippets = []
    paragraphs = []
    
    for row in readCSV:
        abstract = row[0]
        date = row[1]
        headline = row[4]
        paragraph = row[8]
        snippet = row[9]
        
        dates.append(date)
        abstracts.append(abstract)
        headlines.append(headline)
        snippets.append(snippet)
        paragraphs.append(paragraph)
        
# Change the news into data frame
news = {'date': dates, 'abstract': abstracts, 'headline': headlines, 'snippet': snippets, 'paragraph': paragraphs}
news_df = pd.DataFrame(news)
news_df = news_df.iloc[1:len(news_df['abstract']),]

# Group the data frame by date and get the aggregate articles per day
#news_df_grouped = news_df.groupby('date')['abstract'].sum()
news_df_grouped = news_df.groupby('date').agg({'abstract':'sum', 'headline':'sum', 'snippet':'sum', 'paragraph':'sum'})
news_df_grouped = pd.DataFrame(news_df_grouped.reset_index())

# Check
news_df_grouped.head()<jupyter_output><empty_output><jupyter_text>2. Download Share Prices from Yahoo and Compute Log Differences<jupyter_code>data = Fetcher("AAPL", [2014,1,1], [2019,4,30])
shareprice_df = data.getHistorical()

# Use Adjusted Close Price
close_price = shareprice_df.iloc[:, [0,5]]

# Covert the close price to np.array to get log differences
np_price = np.array(close_price.iloc[:,[1]])
log_diff_price = np.diff(np.log(np_price).reshape(1,len(np_price))).reshape(len(np_price)-1, 1)

# Create the binary response variable
binary_response = ['pos' if i > 0 else 'neg' for i in log_diff_price]

# Create the data frame of share price returns (log differences)
log_diff = close_price.iloc[1:len(np_price),[0]]
log_diff.insert(1, 'log_diff_price', log_diff_price)
log_diff.insert(2, 'binary_response', binary_response)
log_diff.rename(columns={'Date':'date', 'log_diff_price':'log_diff', 'binary_response':'binary_response'}, inplace=True)

# Check
log_diff.head()<jupyter_output><empty_output><jupyter_text>3. Create the join table - Since dates are different between the two data sets, we will use inner join to extract the data where we have both information<jupyter_code>news_sp = pd.merge(log_diff, news_df_grouped, on = 'date', how='inner')
news_sp.head()<jupyter_output><empty_output><jupyter_text>4. Create a list  of training and test sets for Naive Bayes Classifier- There are 785 data points in the data set- We will use 700 data points as a training set which is ~90%- We will use the rest as a test set - Choose which text to fit the classifier- A classifier based on the Naive Bayes algorithm, as implemented in NLTK.<jupyter_code>def nbc(text = 'abstract'):
    
    if text == 'abstract':
        no = 3
    elif text == 'headline':
        no = 4
    elif text == 'snippet':
        no = 5
    elif text == 'paragraph':
        no = 6
    
    df_list = [(news_sp.values[i,no], news_sp.values[i,2]) for i in range(len(news_sp))]
    train = df_list[0:700]
    test = df_list[701:len(df_list)+1]
    
    cl = NaiveBayesClassifier(train)
    accuracy = cl.accuracy(test)

    return [cl, accuracy]<jupyter_output><empty_output><jupyter_text>**Use the function to create classifier**<jupyter_code>cl_abstract = nbc('abstract')<jupyter_output><empty_output><jupyter_text>**Use show_informative_features**<jupyter_code>cl_abstract[0].show_informative_features(20)<jupyter_output>Most Informative Features
            contains(we) = True              neg : pos    =      7.6 : 1.0
         contains(worth) = True              pos : neg    =      7.1 : 1.0
           contains(yet) = True              neg : pos    =      6.9 : 1.0
        contains(showed) = True              neg : pos    =      6.1 : 1.0
         contains(women) = True              neg : pos    =      5.4 : 1.0
  contains(improvements) = True              neg : pos    =      5.4 : 1.0
        contains(appeal) = True              neg : pos    =      5.4 : 1.0
        contains(Street) = True              pos : neg    =      5.2 : 1.0
       contains(lawyers) = True              neg : pos    =      4.7 : 1.0
      contains(pressure) = True              neg : pos    =      4.7 : 1.0
          contains(sign) = True              neg : pos    =      4.7 : 1.0
       contains(improve) = True              neg : pos    =      4.7 : 1.0
      contains(internet) = True              neg : pos    =      4.7 : 1.0[...]<jupyter_text>**Use classify function to classify the text**<jupyter_code>cl_abstract[0].classify('women')<jupyter_output><empty_output><jupyter_text>**Compute Accuracy Rate on Test Set**<jupyter_code>cl_abstract[1]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11143695014662756, 'fraction_numerical': 0.020527859237536656, 'mean_word_length': 3.246692607003891, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '185195', 'license': 'MIT', 'n_tokens_mistral': 1838, 'n_tokens_neox': 1751, 'n_words': 476, 'path': 'codes/TextBlob - Naive Bayes Classifier.ipynb', 'repo_name': 'asy02006/ahsungyang-STAT418-final-project', 'size': '5456'}