<jupyter_start><jupyter_text>**Are financial indicators and stock values related? ** According to historical records, the average annual return of S&P 500 is approximately 10%–11% since its inception in 1926. However, is it possible to build up a machine learning model that can predict a specific stock's price will rise or fall in the next year based on last year's financial performance? In addition, is there a method which is able to predict stock price changes in the following year? In this research, I will use more than 200 financial indicators of US stocks from 2014 to 2018 to predict the price variance and the stock price trend in the following year. This dataset is from Kaggle: https://www.kaggle.com/cnic92/200-financial-indicators-of-us-stocks-20142018. In this dataset, The last column of the dataset represent the class of each stock, where: if the value of a stock increases during next year, then class=1; if the value of a stock decreases during next year, then class=0. The second-to-last column, PRICE VAR [%], lists the percent price variation of each stock for the next year. For example, percent price variation for the year 2016 (meaning from the first trading day on Jan 2016 to the last trading day on Dec 2016).The columns PRICE VAR [%] and class make possible to use the datasets for both classification and regression tasks. Data Exploration *Load Dataset*<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
pd.options.display.float_format = '{:,.0f}'.format
df2014 = pd.read_csv('/content/drive/My Drive/Data/2014_Financial_Data.csv')
df2015 = pd.read_csv('/content/drive/My Drive/Data/2015_Financial_Data.csv')
df2016 = pd.read_csv('/content/drive/My Drive/Data/2016_Financial_Data.csv')
df2017 = pd.read_csv('/content/drive/My Drive/Data/2017_Financial_Data.csv')
df2018 = pd.read_csv('/content/drive/My Drive/Data/2018_Financial_Data.csv')
df2014.shape,df2015.shape,df2016.shape,df2017.shape,df2018.shape
df_all = pd.concat([df2014,df2015,df2016,df2017,df2018], axis=0, sort=False).drop_duplicates()
df_all.shape
df = df_all.drop(['2015 PRICE VAR [%]','2016 PRICE VAR [%]','2017 PRICE VAR [%]','2018 PRICE VAR [%]','2019 PRICE VAR [%]'], axis=1)
df.shape
df.head()<jupyter_output><empty_output><jupyter_text>*Peek at one dataset*<jupyter_code>df2018.head()
df2018.describe()
df2018.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 4392 entries, 0 to 4391
Columns: 225 entries, Unnamed: 0 to Class
dtypes: float64(222), int64(1), object(2)
memory usage: 7.5+ MB<jupyter_text>Takeaways:4392 observations225 columns222 numeric(financial indicators)1 int(class column)1 object(sector column)Now, let's take a look at the distribution of the class values, and the distribution of the categorical variable Sector.Datasets of other years can be explored in the same way.  *Check number of companies with good increase or negative increase.*<jupyter_code>low_rate = 0
high_rate =100
bottom = df2018[df2018['2019 PRICE VAR [%]'] < low_rate]
top = df2018[df2018['2019 PRICE VAR [%]'] > high_rate]

print(f'{len(bottom)} companies got negative increase in 2019.')
print(f'{len(top)} companies got more than {high_rate}% increase in 2019.')<jupyter_output>1346 companies got negative increase in 2019.
210 companies got more than 100% increase in 2019.<jupyter_text>Visualize the distribution of stock price increase in the next year(get rid of outliers)<jupyter_code>df_outlier = df2018.copy()
df_outlier.drop(df_outlier[df_outlier['2019 PRICE VAR [%]'] > 500].index, inplace = True) 
sns.distplot(df_outlier['2019 PRICE VAR [%]'],bins=100,rug=True);
df2018_class = pd.DataFrame(df2018['Class'].value_counts().reset_index())
df2018_class.rename(columns={'index':'Class','Class':'Quantity'},inplace=True)
df2018_class<jupyter_output><empty_output><jupyter_text>Visualize the distribution of class.<jupyter_code>sns.set()
df2018_class.set_index('Class').T.plot(kind='bar', stacked=True)
plt.title('Distribution of Class')
plt.show();<jupyter_output><empty_output><jupyter_text>*Check sector distribution of these companies.*<jupyter_code>df_sector = pd.DataFrame(df2018['Sector'].value_counts().reset_index())
df_sector.rename(columns={'index':'Sector','Sector':'Quantity'},inplace=True)
df_sector
colors = ["#E13F29", "#4a53d4","#D69A80","#6bad42","#e6e622", "#AE5552", "#de33d8", "#EB8076", "#96624E","#7da832"]
plt.figure(figsize=(15,8))
plt.pie(
    df_sector['Quantity'],
    labels=df_sector['Sector'],
    shadow=False,
    colors=colors,
)
plt.title('Sector Distribution of Companies in 2018')
plt.tight_layout()
plt.show()
df2018.isnull().sum()
#Based on different sector, replace nan-values with median value of column
df2018_sector = df2018.groupby(['Sector']).transform(lambda x: x.fillna(x.median()))
df2018_sector.isnull().sum()
# Plot correlation matrix of output dataset
fig, ax = plt.subplots(figsize=(15,15)) 
sns.heatmap(df2018_sector.corr(), annot=False, cmap='BuPu', vmin=-1, vmax=1, center=0, ax=ax)
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Financial Analysis on one company. Analysis on other companies can be done in the same manner.<jupyter_code># Take Apple as an example
df2014_AAPL = df2014[df2014['Unnamed: 0'] =='AAPL' ]
df2015_AAPL = df2015[df2015['Unnamed: 0'] =='AAPL' ]
df2016_AAPL = df2016[df2016['Unnamed: 0'] =='AAPL' ]
df2017_AAPL = df2017[df2017['Unnamed: 0'] =='AAPL' ]
df2018_AAPL = df2018[df2018['Unnamed: 0'] =='AAPL' ]
df2014_AAPL
df_AAPL = df2014_AAPL.append([df2015_AAPL,df2016_AAPL,df2017_AAPL,df2018_AAPL])
df_AAPL.index = [2014,2015,2016,2017,2018]
df_AAPL.replace(np.nan, 0)
df_AAPL.columns.tolist()
from matplotlib import rcParams
rcParams['figure.figsize'] = 10, 8
plt.style.use('fivethirtyeight')
plt.title('Asset and Liabilities of APPLE')
plt.grid(True, linestyle=':')
plt.xticks([2014,2015,2016,2017,2018])
plt.xlabel('Year')
plt.ylabel('Amount')
plt.plot(df_AAPL['Total current assets'],label='Total current assets')
plt.plot(df_AAPL['Total non-current assets'],label='Total non-current assets')
plt.plot(df_AAPL['Total current liabilities'],label='Total current liabilities')
plt.plot(df_AAPL['Total non-current liabilities'],label='Total non-current liabilities')
plt.legend(loc=2);
plt.style.use('fivethirtyeight')
rcParams['figure.figsize'] = 10, 8
pos = np.arange(len(df_AAPL))
year = [2014,2015,2016,2017,2018]
plt.bar(pos,df_AAPL['Revenue'],width = 0.3, label='Revenue')
plt.bar(pos+0.3,df_AAPL['Gross Profit'],width = 0.3,label='Gross Profit')
plt.bar(pos+0.6,df_AAPL['Net Profit Margin'],width = 0.3,label='Net Profit Margin')
plt.xlabel('Year')
plt.ylabel('Amount')
plt.xticks(pos,year)
plt.title('Revenue & Profit in Apple')

plt.legend();

df_AAPL['Net Profit Margin']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13578792341678939, 'fraction_numerical': 0.07893961708394698, 'mean_word_length': 6.961313012895662, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '436120', 'license': 'MIT', 'n_tokens_mistral': 2763, 'n_tokens_neox': 2286, 'n_words': 613, 'path': 'YuanjinRen_Unit2_Project.ipynb', 'repo_name': 'yuanjinren/DS-Unit-2-Applied-Modeling', 'size': '6790'}