<jupyter_start><jupyter_text>This note provides basic idea to check the normality and understand the statistics of the return data. It can help you to link to our lectures on the lognormal and normal properties of stock price and returns. Load libraries<jupyter_code>import numpy as np  # array operations
import pandas as pd  # dataframe
import scipy.stats as scs
import statsmodels.api as sm
from pylab import plt
plt.style.use('ggplot')
# put all plots in the notebook itself
%matplotlib inline<jupyter_output><empty_output><jupyter_text>1. A stock Download some stock data from the Yahoo Finance as we did in the first tutor.<jupyter_code>from pandas_datareader import data
TSM = data.DataReader("TSM", start='2010-1-1', end='2019-12-31', data_source='yahoo')
TSM.head()
TSM['Close'].plot(figsize=(16, 10), grid=True, title='TSMC Close Price')
TSM['Return'] = np.log(TSM['Close'] / TSM['Close'].shift(1))
TSM

TSM[['Close', 'Return']].hist(bins=50, figsize=(16, 10))
TSM[['Close', 'Return']].describe()
scs.describe(TSM['Return'].dropna()) # to see skewness & kurtosis<jupyter_output><empty_output><jupyter_text>Another good skim for the normality data is to check the Q-Q plot:<jupyter_code>sm.qqplot(TSM['Return'].dropna(), line='s')
plt.grid(True)
plt.xlabel('theoretical quantiles')
plt.ylabel('sample quantiles')<jupyter_output><empty_output><jupyter_text>Lastly, we do the normality test. `scipy` gives us several functions to do the test:- [`skewtest`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.skewtest.htmlscipy.stats.skewtest)- [`kurtosistest`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kurtosistest.htmlscipy.stats.kurtosistest)- [`normaltest`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.normaltest.htmlscipy.stats.normaltest)<jupyter_code>def normality_tests(arr):
    ''' Tests for normality distribution of given data set.
    
    Parameters
    ==========
    array: ndarray
        object to generate statistics on
    '''
    print("Skew of data set  %14.3f" % scs.skew(arr))
    print("Skew test p-value %14.3f" % scs.skewtest(arr)[1])
    print("Kurt of data set  %14.3f" % scs.kurtosis(arr))
    print("Kurt test p-value %14.3f" % scs.kurtosistest(arr)[1])
    print("Norm test p-value %14.3f" % scs.normaltest(arr)[1])
    
normality_tests(TSM['Return'].dropna())<jupyter_output>Skew of data set          -0.101
Skew test p-value          0.039
Kurt of data set           1.804
Kurt test p-value          0.000
Norm test p-value          0.000<jupyter_text>Now, please read the results and conclude.  2. The market indexWhile a stock return may has some bias due to firm-specific risk, we may find normality from the market returns, which is the value-weighted returns from many stocks. Let's take the market returns from Fama-French database.<jupyter_code>Factors5 = data.DataReader('F-F_Research_Data_5_Factors_2x3','famafrench', start='1925-01-01')
Factors5[0].head()
Factors5[0]['Mkt'] = Factors5[0]['Mkt-RF'] + Factors5[0]['RF'] # get market returns from excess returns
Factors5[0][['Mkt', 'Mkt-RF']].hist(bins=30, figsize=(16, 10))
# another way
import seaborn as sns
sns.distplot(Factors5[0][['Mkt']], hist=True, kde=True, 
             bins=int(180/5), color = 'darkblue', 
             hist_kws={'edgecolor':'black'},
             kde_kws={'linewidth': 4})
sns.distplot(Factors5[0][['Mkt-RF']], hist=True, kde=True, 
             bins=int(180/5), color = 'darkblue', 
             hist_kws={'edgecolor':'black'},
             kde_kws={'linewidth': 4})<jupyter_output><empty_output><jupyter_text>Let's check some statistics. We'll rely on the [`scipy.stats`](https://docs.scipy.org/doc/scipy/reference/stats.html) package.<jupyter_code>Mkt = Factors5[0]['Mkt']
Mkt.describe()
scs.describe(Mkt)
scs.scoreatpercentile(Mkt, 5), scs.scoreatpercentile(Mkt, 95)
normality_tests(Mkt)<jupyter_output>Skew of data set          -0.546
Skew test p-value          0.000
Kurt of data set           1.949
Kurt test p-value          0.000
Norm test p-value          0.000
{'fraction_non_alphanumeric': 0.14169541646131098, 'fraction_numerical': 0.032528339083292265, 'mean_word_length': 4.433734939759036, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '497264', 'license': 'MIT', 'n_tokens_mistral': 1524, 'n_tokens_neox': 1425, 'n_words': 342, 'path': '07_priceReturnDist.ipynb', 'repo_name': 'diengiau/py18plus', 'size': '4058'}