<jupyter_start><jupyter_text>**<NAME>**Task - Create a hybrid model for stock price/performance predictionusing numerical analysis of historical stock prices, and sentimental analysis ofnews headlines<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

import datetime
import pandas_datareader.data as web
from pandas import Series, DataFrame
#web data reader is extension of pandas library to communicate with frequently updating data
start = datetime.datetime(2000, 1, 1)
end = datetime.datetime(2019, 12, 31)

df = web.DataReader("MSFT", 'yahoo', start, end)
df.tail()<jupyter_output><empty_output><jupyter_text>We have the data of Microsoft stocks of 19 years from January 2000 till December 2019. We analyse stocks using two measurements- Rolling mean and Return rate. Rolling Mean - Rolling is a very useful operation for time series data. Rolling means creating a rolling window with a specified size and perform calculations on the data in this window which, of course, rolls through the data.In pandas Moving Average- technical analysis tool that smooths out price data by creating a constantly updated average price.<jupyter_code>close_px = df['Adj Close']
mavg = close_px.rolling(window = 100).mean()
mavg.head(50)
%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import style

#adjust the size of matplotlib
import matplotlib as npl
npl.rc('figure', figsize=(10,10))
npl.__version__

#adjust the style of matplotlib
style.use('ggplot')

close_px.plot(label = "MSFT")
mavg.plot(label = 'mavg')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>**Return deviation - determine risk and return**Expected return measures the expected value/mean of investment returns.Expected return is calculated by multiplying the weights of each asset by expected and adding value value for each investment.<jupyter_code>rets = close_px / close_px.shift(1)  - 1
rets.plot(label = 'return')
plt.show()<jupyter_output><empty_output><jupyter_text>Analyze stocks of various IT giants like Microsoft, Apple, Amazon, Google ,IBM<jupyter_code>dfcomp = web.DataReader(['MSFT' , 'AAPL' , 'AMZN' , 'GOOG' , 'IBM'], 'yahoo', start=start, end=end)['Adj Close']
dfcomp.head()<jupyter_output><empty_output><jupyter_text>Correlation analysis using correlation function in pandas<jupyter_code>retscomp = dfcomp.pct_change()
corr = retscomp.corr()
retscomp.head(10)<jupyter_output><empty_output><jupyter_text>using heat maps to visualize correlation range of various stocks<jupyter_code>plt.imshow(corr, cmap = 'hot', interpolation='none')
plt.colorbar()
plt.xticks(range(len(corr)), corr.columns)
plt.yticks(range(len(corr)), corr.columns);<jupyter_output><empty_output><jupyter_text>Stocks Return rate and Risk<jupyter_code>plt.scatter(retscomp.mean(), retscomp.std())
plt.xlabel('Expected returns')
plt.ylabel('Risk')
for label, x, y in zip(retscomp.columns, retscomp.mean(), retscomp.std()):
  plt.annotate(label, xy = (x, y), xytext = (20, -20),
               textcoords = 'offset points', ha = 'right', va = 'bottom',
               bbox = dict(boxstyle = 'round, pad = 0.5', fc  = 'yellow',
                           alpha = 0.5), arrowprops = dict(arrowstyle = '->',
                                                           connectionstyle = 'arc3,rad = 0'))<jupyter_output><empty_output><jupyter_text>Importing News-Headlines files for numerical analysis<jupyter_code>df1 = pd.read_csv('/content/india-news-headlines.csv')
df1.head(10)
df1.tail(10)
df1.describe()
df1.max()
df1.min()
df1['headline_category'].unique()
#case of null values
df1.isna().any()
df1.corr()
sns.set_palette('viridis')
sns.pairplot(df1)
plt.show()
df1['headline_text'].value_counts()
df1['headline_category'].value_counts()<jupyter_output><empty_output><jupyter_text>EDA (Exploratry data analysis) using NLP and NLTK tools<jupyter_code>df1['headline_text'].str.len().hist()
plt.show()<jupyter_output><empty_output><jupyter_text>Graph shows that news headlines range from 10 to 120 characters generally.<jupyter_code>df1['headline_text'].str.split().\
  apply(lambda x : [len(i) for i in x]). \
  map(lambda x: np.mean(x)).hist()
plt.show()<jupyter_output><empty_output><jupyter_text>The average word length ranges from 1 to 15<jupyter_code>import nltk
from nltk.corpus import stopwords
nltk.download('stopwords')
stop=set(stopwords.words('english'))
corpus=[]
new= df1['headline_text'].str.split()
new=new.values.tolist()
corpus=[word for i in new for word in i]

from collections import defaultdict
dic=defaultdict(int)
for word in corpus:
    if word in stop:
        dic[word]+=1
from collections import Counter
counter=Counter(corpus)
most=counter.most_common()
x, y= [], []
for word,count in most[:40]:
    if (word not in stop):
        x.append(word)
        y.append(count)
        
sns.barplot(x=y,y=x)
plt.show()<jupyter_output><empty_output><jupyter_text>Ngram exploration<jupyter_code>from nltk.util import ngrams
list(ngrams(['I' ,'went','on','a','world','Tour'],2))
from sklearn.feature_extraction.text import CountVectorizer
def get_top_ngram(corpus, n=None):
    vec = CountVectorizer(ngram_range=(n, n)).fit(corpus)
    bag_of_words = vec.transform(corpus)
    sum_words = bag_of_words.sum(axis=0) 
    words_freq = [(word, sum_words[0, idx]) 
                  for word, idx in vec.vocabulary_.items()]
    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)
    return words_freq[:10]
top_n_bigrams=get_top_ngram(df1['headline_text'],2)[:10]
x,y=map(list,zip(*top_n_bigrams))
sns.barplot(x=y,y=x)
plt.show()<jupyter_output><empty_output><jupyter_text>Using Textblob which is a python library for NLTK<jupyter_code>from textblob import TextBlob
TextBlob('100 people killed in road accident').sentiment
def polarity(text):
    return TextBlob(text).sentiment.polarity
df1['polarity_score']=df1['headline_text'].\
   apply(lambda x : polarity(x))
df1['polarity_score'].hist()
plt.show()<jupyter_output><empty_output><jupyter_text>![download (1).png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAf8AAAGbCAYAAADKlJnyAAAABHNCSV<KEY>4AwBgGOIPAIBhiD8AAIYh/gAAGIb4AwBgGOIPAIBhiD8AAIYh/gAAGIb4AwBgGOIPAIBhiD8AAIYh/gAAGIb4AwBgGOIPAIBhiD8AAIaJj/QBAIht996+LN<KEY>) The graph indicates that majority of news headlines are neutral as polarity ranges between mainly 0.00 to 0.20<jupyter_code>def sentiment(x):
    if x<0:
        return 'neg'
    elif x==0:
        return 'neu'
    else:
        return 'pos'
    
df1['polarity']=df1['polarity_score'].\
   map(lambda x: sentiment(x))
plt.bar(df1.polarity.value_counts().index,
        df1.polarity.value_counts())
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12330198537095088, 'fraction_numerical': 0.0180623973727422, 'mean_word_length': 5.594488188976378, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '135464', 'license': 'MIT', 'n_tokens_mistral': 2428, 'n_tokens_neox': 2285, 'n_words': 552, 'path': 'TSF_Task#7.ipynb', 'repo_name': 'jayanthd001/TSF_Data-Science-and-Business-Analytics', 'size': '18799'}