<jupyter_start><jupyter_text>Trying to perdict the price of a certain stock has always been a challenging problem. This kernel tries to preict the behavior of a stock from news headlines. The stock price and related news for any trading company can be taken from [FINVIZ](http://finviz.com). I have used [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) for the analysis. 'Beautiful Soup is a python library for pulling data out of HTML and XML files.'<jupyter_code># Import libraries
from bs4 import BeautifulSoup
import pandas as pd
import os

html_pages = {}

# The dataset includes webpages from different days for different companies.
for page_name in os.listdir('../input'):
    page_path = f'../input/{page_name}'
    page_file = open(page_path, 'r')
    html_data = BeautifulSoup(page_file)
    # The .html file contains a 'news-table' section that holds all the news headlines
    html_page = html_data.find(id="news-table")
    # Add the page to our dictionary
    html_pages[page_name] = html_page<jupyter_output><empty_output><jupyter_text>You can display the html pages that we have extracted so far.<jupyter_code>html_pages<jupyter_output><empty_output><jupyter_text>Now we can explore the material of these headlines. What's inside them!You may want to check these links for a better understanding of the elements of a HTML page.[a: The Anchor element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a)[td: The Table Data Cell element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/td)<jupyter_code>amzn = html_pages['amzn_24jul.html']
# All the headlines in the .html file are stored as table rows and are tagged as <tr>
amzn_tr = amzn.findAll('tr') #Remember this is a beautiful soup now. So, we can directly find <tr> tag.

for i, tr in enumerate(amzn_tr):
    # <a> in these .html pages contain the news headlines.
    hyperlinks = tr.a.get_text()
    # <td> in these .html page contain the time stamp.
    headlines = tr.td.get_text()
    print(f'{i}:')
    print(hyperlinks)
    print(headlines)
    
    #if i == 5:
    #    break<jupyter_output>0:
<NAME>: Disney Earnings Could Nearly Double In Five Years
Jul-24-19 02:30PM  
1:
Alibaba Opens Its Doors to U.S. Merchants to Widen Its Moat
02:15PM  
2:
UPS stock soars toward best gain in a decade after profit beat, outlook held firm
02:04PM  
3:
EBay to launch warehousing and shipping services next year
01:39PM  
4:
UPDATE 1-EBay to launch warehousing and shipping services next year
01:34PM  
5:
Facebook and Other Big Tech Stocks Are Barely Moving on the DoJs New Probe. Heres Why.
01:34PM  
6:
Antitrust Probe Likely to Hit These Tech ETFs
01:00PM  
7:
Feds Bite Apple, FANG Stocks, But They Have A Lot To Chew On
12:52PM  
8:
The most intimate data youre giving up may not be to Facebook, Amazon or Apple
12:52PM  
9:
Facebook (FB) Earnings After The Bell: Can The Growth Continue?
12:19PM  
10:
EBay to launch warehousing and shipping service next year
11:58AM  
11:
GRAPHIC-Taking stock: Swiss franc intervention seeps far and wide
11:39AM  
12:
Mnuchin lays into Amazon, says i[...]<jupyter_text>Let's try the same with all the webpages in the dataset.<jupyter_code>news = []

for page_name, news_table in html_pages.items():
    for x in news_table.findAll('tr'):
        text = x.get_text()
        headline = x.a.get_text()
        # Split date-time, because some elements just have time, while others have time and date as well.
        date_td = x.td.text.split()
        # If only time
        if len(date_td) == 1:
            time = date_td[0]
        # If both time and Date
        else:
            date = date_td[0]
            time = date_td[1]
        # Separate the stock symbol    
        stock_symbol = page_name.split('_')[0]
        # Join everything
        news.append([stock_symbol, date, time, headline])
        
news[:5] # print first five news headlines<jupyter_output><empty_output><jupyter_text>I have used VADER to perform sentiment analysis, because it doesn't require any training data. We can add new words to the VADER Lexicon text file which are specific to our problem. In our case, since we are working with stock market, adding words like 'falls', 'trouble', etc. with an associated positive or negative score should be good practice.<jupyter_code>from nltk.sentiment.vader import SentimentIntensityAnalyzer

new_words = {
    'crushes': 10,
    'beats': 5,
    'misses': -5,
    'trouble': -10,
    'falls': -100
}

analyser = SentimentIntensityAnalyzer()

analyser.lexicon.update(new_words)<jupyter_output>/opt/conda/lib/python3.6/site-packages/nltk/twitter/__init__.py:20: UserWarning: The twython library has not been installed. Some functionality from the twitter package will not be available.
  warnings.warn("The twython library has not been installed. "<jupyter_text>Now we have updated the lexicons in the VADER model, we can try to evaluate the polarity of the news headlines.<jupyter_code>columns = ['stock_symbol', 'date', 'time', 'headline']
# Load list of the news into a pandas DataFrame
news_headlines = pd.DataFrame(news, columns = columns)
# Go through each headline and evaluate it's polarity. 
senti_scores = [analyser.polarity_scores(headline) for headline in news_headlines.headline.values]
# Convert it into a DataFrame for easy joining.
sentiment = pd.DataFrame(senti_scores)
# Join headline and its sentiment score.
news_headlines = pd.concat([news_headlines, sentiment], axis = 1)
# Date column is converted from string to datetime
news_headlines['date'] = pd.to_datetime(news_headlines.date).dt.date
news_headlines.head(10)<jupyter_output><empty_output><jupyter_text>Try to plot the sentiment scores<jupyter_code>import matplotlib.pyplot as plt
plt.style.use("fivethirtyeight")
%matplotlib inline
# Group by Date and Company
mean_c = news_headlines.groupby(['date', 'stock_symbol']).mean()

mean_c = mean_c.unstack(level=1)
mean_c = mean_c.xs('compound', axis=1)

mean_c.plot.bar();<jupyter_output><empty_output><jupyter_text>Seems like Tesla had some trouble on November 22. Further exploration reveals that there were only 5 headlines on that day. And 2 of them were almost similar.We can further check and try to remove similar headlines from the dataset.<jupyter_code># Number of news headlines
dirty_news = news_headlines.headline.count()
# Drop duplicates headlines
cleaned_news = news_headlines.drop_duplicates(['stock_symbol', 'headline'])
# Number of headlines after dropping 
clean_news = cleaned_news.headline.count()

print(dirty_news)
print(clean_news)<jupyter_output>900
871<jupyter_text>To finish the project, lets try to explore just one stock on any given day (just a single day), cause this is what one should be interested in.And don't forget to use 'cleaned_news'.<jupyter_code>single_day = cleaned_news.set_index(['stock_symbol', 'date'])
single_day = single_day.loc['amzn']

# Set day to January of 2019
single_day = single_day.loc['2019-07-24']

# Convert the datetime string to just the time since it is just one day.
single_day['time'] = pd.to_datetime(single_day['time'])
single_day['time'] = single_day.time.dt.time

# Set the index to time and sort by it
single_day.set_index('time', inplace=True)
single_day=single_day.sort_index(ascending=True)
single_day.head()<jupyter_output><empty_output><jupyter_text>We can now visualize the sentiment scores using a plot.<jupyter_code># Drop the columns which are useless
plot_day = single_day.drop(['headline', 'compound'], axis=1)
# Give names to the sentiments
plot_day.columns = ['negative', 'positive', 'neutral']
# Plot a stacked bar chart
plot_day.plot.bar(stacked = True, 
                  figsize=(10, 6), 
                  title = "Sentiment Analysis for AMAZON on 2019-07-24", 
                  color = ["red", "blue", "green"])
plt.legend(bbox_to_anchor=(1.2, 0.5))
plt.ylabel("scores");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09307331898189186, 'fraction_numerical': 0.017221729770799037, 'mean_word_length': 4.380108991825613, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '621246', 'license': 'MIT', 'n_tokens_mistral': 2617, 'n_tokens_neox': 2457, 'n_words': 876, 'path': 'Predicting Stock Price from News/stock-behavior-from-business-news.ipynb', 'repo_name': 'navpreet-ml/Machine-Learning-Projects', 'size': '7905'}