<jupyter_start><jupyter_text>Investigate free data sets at their providers' pages1. Based on the descriptions at the provider pages, explain what each data set means.2. Select two data sets, and compare their changes at time points when something happened (or nothing happened) in the world.3. Note: the word "spread" means difference.| Source | Name, Link to page | Type | Further information || :--- | :--| :--- | :-- || [Yahoo Finance](https://finance.yahoo.com) | [Walmart stock price](https://finance.yahoo.com/quote/WMT/history) | Equity | Retail, Big capitalization || &nbsp; | [Amazon](https://finance.yahoo.com/quote/AMZN/history) | Equity | IT, Big cap || &nbsp; | [Tesla](https://finance.yahoo.com/quote/TSLA/history) | Equity | New technology || &nbsp; | [BJ's Restaurants](https://finance.yahoo.com/quote/BJRI/history) | Equity | Catering industry, Small cap || &nbsp; | [Bitcoin](https://finance.yahoo.com/quote/BTC-USD/history) | FX |  Meant for payments | | &nbsp; | [Ethereum](https://finance.yahoo.com/quote/ETH-USD/history) | FX | More infrastructural | | [FRED](https://fred.stlouisfed.org) | Employment: [Not seasonally adjusted](https://fred.stlouisfed.org/series/PAYNSA)and [Seasonally adjusted](https://fred.stlouisfed.org/series/PAYEMS) | Macroeconomy | Total non-farm employees || &nbsp; | [S&P500 stock market index](https://finance.yahoo.com/quote/%5EGSPC/history) | Equity index |  Large cap stocks || &nbsp; | [USD 1Y swap rate ICE](https://fred.stlouisfed.org/series/SP500) | Rates | [ICE methodology](https://www.theice.com/publicdocs/ICE_Swap_Rate_Full_Calculation_Methodology.pdf) || &nbsp; | [Ounce of gold in USD](https://fred.stlouisfed.org/series/GOLDPMGBD228NLBM) | Commodity | Gold: bullion || &nbsp; | [Moody's AAA 10Y credit spread](https://fred.stlouisfed.org/series/AAA10Y) | Bonds vs 10Y Treasury  | Spread to 10Y T-bond || &nbsp; | [YEN / USD exchange rate](https://fred.stlouisfed.org/series/DEXJPUS) | FX | &nbsp; | | &nbsp; | [Wilshire US Real Estate Securities Price Index](https://fred.stlouisfed.org/series/WILLRESIPR)  | Real estate index | [Wilshire's description](https://www.wilshire.com/indexes/wilshire-real-estate-family/wilshire-us-resi) || [ECB](https://sdw.ecb.europa.eu) | [USD / EUR exchange rate](https://sdw.ecb.europa.eu/quickview.do?SERIES_KEY=120.EXR.D.USD.EUR.SP00.A) | FX | ECB reference rate || [Bp Stock Exch](https://bse.hu/pages/data-download) | OTP | Equity | Banking, Regional || &nbsp; | Richter | Equity | Pharma || &nbsp; | Telekom HU | Equity | Telecom |  Investigate the data locally Download the data filesFor each row in the above list of data sets, please click the link pointing to the download page of the given data set,and then download the data file to the `data` subfolder of your current directory.1. On the (historical data) download page of a Yahoo data set, select "Max" Time Period, click Apply, and then click "Download".2. On the download page of a FRED data set, click "Download" in the top right corner, and select "CSV".3. When downloading the EUR/USD data from the ECB, click "Data Download" (top right), select "Excel (csv)",and save the file with this name: `120.EXR.D.USD.EUR.SP00.A.csv`. You may need to add the .csv extension manually. 4. To download data from the Budapest Stock Exchange, do the following at https://bse.hu/pages/data-download:  Settings    . Time range: From day to day    . From day to day: 2014-01-01 to 2019-12-31    . Data format: Comma separated (.csv)    . Data type: Detailed (values, averages, volumes)  Under "Primary Market" select    . Category: Equities Prime    . Instrument: OTP  Click "Download"    . Save the downloaded file with the name `OTP.csv` to the `data` subfolder of your current directory    . Similarly download into `separate` files the data for `RICHTER` and `MTELEKOM` Questions1. For Yahoo and FRED data sets, explain the meaning of each column.2. For each Yahoo data set calculate the median / maximum ratio of the daily Volume shown in the last column. Which data set has the lowest ratio ?<jupyter_code>import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import os
data_dir = 'data'
file_ext = 'csv'
df = {}  # data frames by data set code

# Time series from: Yahoo Finance, Federal Reserve Economic Data, European Central Bank, Portfolio.hu
yahooCodes = ['WMT','AMZN','TSLA','BJRI','BTC-USD','ETH-USD']
fred_codes = ['PAYEMS','PAYNSA','SP500','AAA10Y','DEXJPUS','GOLDPMGBD228NLBM','ICERATES1100USD1Y','SP500','WILLRESIPR']
ecb_codes = ['120.EXR.D.USD.EUR.SP00.A']
bse_codes = ['MTELEKOM','OTP','RICHTER']
all_codes = yahooCodes + fred_codes + ecb_codes + bse_codes

# Investigate each data frame
for code in all_codes:
    df[code] = pd.read_csv(os.path.join(data_dir, code + '.' + file_ext))
#     display(code, df[code])  # display the data frame for the given code
#     display(code, df[code].describe())  # display its description<jupyter_output><empty_output><jupyter_text>Plot value. Plot Daily and Monthly log return.1. Noting that the vertical scale is logarithmic, which stocks have had long periods of exponential growth ?2. In which year did WMT (Walmart) have bigger changes relative to itself: 1975 or 2005 ?<jupyter_code>import matplotlib.pyplot as plt
import datetime
import numpy as np


def last_date_in_each_month(businessDays):
    '''Get last date in each month of a time series'''
    dateRange = []  
    tempYear = None  
    dictYears = businessDays.groupby(businessDays.year)
    for yr in dictYears.keys():
        tempYear = pd.DatetimeIndex(dictYears[yr]).groupby(pd.DatetimeIndex(dictYears[yr]).month)
        for m in tempYear.keys():
            dateRange.append(max(tempYear[m]))
    return dateRange


def prepare_data_and_plot_time_series():
    # set dataframe index to datetime
    for code in yahooCodes:
        df[code].index = pd.to_datetime( df[code]['Date'] )

    # create dataframe of monthly returns
    dfm = dict() # dict to save monthly close data by data set key
    for code in yahooCodes:    
        all_dates = df[code].index
        month_last_dates = last_date_in_each_month(all_dates)
        dfm[code] = pd.DataFrame(df[code], index=month_last_dates)

    # daily and monthly log return
    for code in yahooCodes:
        df[code]['LogReturn']  = np.log(df[code]['Close']) -  np.log(df[code]['Close'].shift())
        dfm[code]['LogReturn'] = np.log(dfm[code]['Close']) - np.log(dfm[code]['Close'].shift())

    # parameters for drawing
    xlims=[datetime.date(1971,12,31),datetime.date(2020,6,30)] # horizontal axis limits
    ylims=[-.45,.45]
    removeOutlierBelowMinusOne = True # whether we should remove the log daily return outlier
    yahooColors = ['black','blue','#a0a0ff','salmon','limegreen','darkgreen']
    fontsize=12
    marker='.'

    # plot daily values
    plt.subplot(311)
    for code,color in zip(yahooCodes,yahooColors):
        plt.plot(df[code]['Close'], c=color, marker=marker, label=code, lw=0)
    plt.legend(bbox_to_anchor=(0.01, .98), loc=2, borderaxespad=0., fontsize=fontsize)
    plt.yscale('log')
    plt.xlabel('Time [year]',fontsize=fontsize)
    plt.ylabel('Value of 1 Unit on log scale',fontsize=fontsize)
    plt.xlim(xlims)

    # plot logarithmic daily returns
    plt.subplot(312)
    for code,color in zip(yahooCodes,yahooColors):
        s = df[code]['LogReturn']
        if removeOutlierBelowMinusOne:
            s = s[s>-1]
        plt.plot(s, c=color, marker='.', ms=1, label=code, lw=0)
    plt.yscale('linear')
    plt.xlabel('Time [year]', fontsize=fontsize)
    plt.ylabel('Business Day Log Return', fontsize=fontsize)
    plt.xlim(xlims)
    #plt.ylim(ylims)

    # plot logarithmic monthly returns
    plt.subplot(313)
    normalization_factor = 1.0 # / np.sqrt(number_of_business_days_per_month)
    number_of_business_days_per_month = 21
    for code,color in zip(yahooCodes,yahooColors):
        s = dfm[code]['LogReturn']
        plt.plot(s * normalization_factor, c=color, marker='.', ms=2, label=code, lw=0)
    plt.yscale('linear')
    plt.xlabel('Time [year]', fontsize=fontsize)
    #plt.ylabel('Log Monthly Return / ' + r'$\sqrt{' + str(number_of_business_days_per_month) + r'}$', fontsize=fontsize)
    plt.ylabel('Log Monthly Return', fontsize=fontsize)
    plt.xlim(xlims)
    #plt.ylim(ylims)

    fig = plt.gcf()
    fig.set_size_inches([16, 15])
    plt.show()

    return df, dfm, xlims

df, dfm, xlims = prepare_data_and_plot_time_series()<jupyter_output><empty_output><jupyter_text>Log return distribution vs Normal1. On which time scale is BTC closer to normal: daily log returns or monthly log returns ?2. Can you find any data errors, for example, cutoff around zero ?<jupyter_code># selected data set for plotting
selectedCode = 'BTC-USD'

# import modules
import matplotlib.pyplot as plt
import numpy as np
from scipy.optimize import curve_fit
from scipy.special import erf
import pandas as pd
# select color for the data set
yahooColors = ['black','blue','#a0a0ff','salmon','limegreen','darkgreen']
code2num = { yahooCodes[num]:num for num in np.arange(len(yahooColors)) }
selectedColor = yahooColors[code2num[selectedCode]]
fontsize=12

# helper functions
def cdf(series):
    '''Calculate CDF (cumulated density function)'''
    series_dropna = series.dropna()
    series_dropna_sorted = np.sort(series_dropna)
    n = series_dropna.size
    values = np.arange(1, n + 1) / n
    return(series_dropna_sorted, values)

def func_normal_cdf(x, mu, sigma): 
    '''CDF of normal distribution with parameters mu,sigma'''
    return 0.5 * ( 1.0 + erf((x-mu)/(sigma*np.sqrt(2.0))) ) 

# plot DAILY and MONTHLY
for which_period in (["DAILY", "MONTHLY"]):
    dfsel = dict()
    if which_period == "DAILY":
        for key in df:
            dfsel[key] = df[key].copy()
    else:
        for key in dfm:
            dfsel[key] = dfm[key].copy()
            
    # Left: one selected time series as an example
    cdfx, cdfy = cdf(dfsel[selectedCode]['LogReturn']) # CDF of daily log returns
    popt, pcov = curve_fit(func_normal_cdf, cdfx, cdfy) # fit normal's CDF to observed CDF
    cdfy_fit = func_normal_cdf(cdfx, *popt) # CDF fit points
    plt.subplot(121)
    plt.xlabel(which_period + " log return", fontsize=fontsize)
    plt.ylabel("Cumulated density function  (CDF)", fontsize=fontsize)
    plt.title(selectedCode + " : Observed CDF and Normal Fit CDF", fontsize=fontsize)
    plt.plot(cdfx, cdfy, c=selectedColor, marker='o', label=selectedCode, markersize=1, lw=1)
    plt.plot(cdfx, cdfy_fit, c='k', ls=':', label='Normal fit',lw=1)
    plt.legend(bbox_to_anchor=(.02, .93), loc=2, borderaxespad=0., fontsize=fontsize)
    plt.axhline(0, c='k', ls=':', lw=.3)
    plt.axhline(1, c='k', ls=':', lw=.3)
    plt.axvline(0, c='k', ls=':', lw=.3)

    # Right panel: Plot only selected or Plot all
    plt.subplot(122)
    for code,color in zip(yahooCodes,yahooColors):
        #if True:
        if code == selectedCode: # plot the CDF-CDF only for the selected data set, use True to plot for all
            cdfx, cdfy = cdf(dfsel[code]['LogReturn']) # CDF of daily log returns
            popt, pcov = curve_fit(func_normal_cdf, cdfx, cdfy) # fit normal's CDF to observed CDF
            cdfy_fit = func_normal_cdf(cdfx, *popt) # CDF fit points
            plt.plot(cdfy_fit,cdfy,c=color, marker='.', label=code, markersize=1, lw=1)
    plt.title("Slope > 1 means : observed PDF > normal PDF", fontsize=fontsize)
    plt.xlabel("Normal fit CDF", fontsize=fontsize)
    plt.ylabel("Observed " + which_period + " log returns CDF", fontsize=fontsize)
    plt.plot([0,1],[0,1],"k:",lw=1,label="Slope=1")
    plt.legend(bbox_to_anchor=(0.02, .98), loc=2, borderaxespad=0., fontsize=fontsize)
    fig = plt.gcf()
    fig.set_size_inches([14, 4])
    plt.show()<jupyter_output><empty_output><jupyter_text>Log return and Abs value of log return1. The number beside each symbol shows 1-step autocorrelation, for example, WMT (0.055). Which ticker's log return has negative autocorrelation ?2. When we switch from log return to the abs value of log return, how does the autocorrelation change ?<jupyter_code>import matplotlib.pyplot as plt
ylims=[-.45,.45]
abs_ylims=[-.02,.45]
fontsize=12
marker='o'
markersize=2

# daily log return
plt.subplot(211)
for code,color in zip(yahooCodes,yahooColors):
    s = df[code]['LogReturn']
    autocorr = '%.3f' % s.autocorr()
    plt.plot(s, c=color, marker=marker, ms=markersize, label = code + " (" + str(autocorr) + ")", lw=0)
plt.legend(bbox_to_anchor=(1.01, .98), loc=2, borderaxespad=0., fontsize=fontsize)
plt.yscale('linear')
plt.xlabel('Time [year]', fontsize=fontsize)
plt.ylabel('Business Day Log Return', fontsize=fontsize)
plt.xlim(xlims)
plt.ylim(ylims)

# absolute value of log return
plt.subplot(212)
for code,color in zip(yahooCodes,yahooColors):
    s = np.absolute(df[code]['LogReturn'])
    autocorr = '%.3f' % s.autocorr()
    plt.plot(s, c=color, marker=marker, ms=markersize, label = code + " (" + str(autocorr) + ")", lw=0)
plt.legend(bbox_to_anchor=(1.01, .98), loc=2, borderaxespad=0., fontsize=fontsize)
plt.yscale('linear')
plt.xlabel('Time [year]', fontsize=fontsize)
plt.ylabel('Absolute value of Log Return', fontsize=fontsize)
plt.xlim(xlims)
plt.ylim(abs_ylims)

fig = plt.gcf()
fig.set_size_inches([12, 10])
plt.show()<jupyter_output><empty_output><jupyter_text>Autocorr of log return and abs log returnThese plots show autocorrelation vs time difference. 1. Which daily log return has significantly nonzero autocorrelation ?2. Which abs daily log return has the highest and lowest autocorrelation after long time ?<jupyter_code># main parameters
autocorr_len = 126  # check autocorrelation up to this number of business days
xmargin_of_plot = 3
autocorr_shifts = np.arange( 1 , autocorr_len + 1 )

# imports and other parameters
import matplotlib.pyplot as plt
fontsize =14
marker = 'o'
markersize = 4   
xlims = ( 1 - xmargin_of_plot, autocorr_len + xmargin_of_plot)
ylims = ( -.2, .35 )
axhline_width = 0.5

# daily log return
plt.subplot(121)
for code,color in zip(yahooCodes,yahooColors):
    s = df[code]['LogReturn']
    autocorr = [ float( '%.3f' % s.autocorr(shift) ) for shift in autocorr_shifts ]
    plt.plot(autocorr_shifts, autocorr, c=color, marker=marker, ms=markersize, label=code, lw=0)
plt.legend(bbox_to_anchor=(.05, .98), loc=2, borderaxespad=0., fontsize=fontsize)
plt.title("Autocorrelation of daily log return", fontsize=fontsize)
plt.yscale('linear')
plt.xlabel('Shift [business days]', fontsize=fontsize)
plt.ylabel('Autocorrelation with selected shift', fontsize=fontsize)
plt.axhline(0, c='k', ls=':', lw=axhline_width)
plt.axvline(0, c='k', ls=':', lw=axhline_width)
plt.xlim(xlims)
plt.ylim(ylims)

# daily log return
plt.subplot(122)
for code,color in zip(yahooCodes,yahooColors):
    s = np.absolute(df[code]['LogReturn'])
    autocorr = [ float( '%.3f' % s.autocorr(shift) ) for shift in autocorr_shifts ]
    plt.plot(autocorr_shifts, autocorr, c=color, marker=marker, ms=markersize, lw=0)
plt.title("Autocorr. of the abs. value of the daily log return", fontsize=fontsize)
plt.yscale('linear')
plt.xlabel('Shift [business days]', fontsize=fontsize)
plt.axhline(0, c='k', ls=':', lw=axhline_width)
plt.axvline(0, c='k', ls=':', lw=axhline_width)
plt.xlim(xlims)
plt.ylim(ylims)

fig = plt.gcf()
fig.set_size_inches([16, 8])
plt.show()<jupyter_output><empty_output><jupyter_text>Volume vs log Return1. What do you conclude from daily log return vs traded volume plotted for each day ?2. What do you conclude when points are binned by log return ?3. Do your conclusions change if you select a different time window ( not 2014 - 2019 ) ?<jupyter_code># the Budapest Stock Exchange (bse) time series contain trading volume
# it is assumed here that the data sets are already imported
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
bse_colors = ['magenta','limegreen','blue']
markersize=2
xmin, xmax = (-0.15, 0.15)  # note: this tuple assignment is the same as "xmin, xmax = -0.15, 0.15"
xbins = np.linspace(xmin, xmax, 100)
ytics_left = [0.01, 0.1, 1, 10]
ytics_right = [1, 3, 10, 30]
axvline_width = 0.5
marker='o'

# set index to datetime, set closing value, log return, and traded volume
for code in bse_codes:
    df[code].index = pd.to_datetime(df[code]['Date'])
    df[code]['Close'] = df[code]['Close price']
    df[code]['LogReturn'] = np.log(df[code]['Close price']) - np.log(df[code]['Close price'].shift())
    df[code]['Volume'] = df[code]['Volume (HUF value)']

# plot daily values
plt.subplot(121)
for code,color in zip(bse_codes, bse_colors):
    if True: # plot all data sets
    #if 'RICHTER' == code: # plot selected data set
        plt.plot(df[code]['LogReturn'], df[code]['Volume'] / 1e+9, 
                 c=color, marker=marker, label=code, lw=0, markersize=markersize)
plt.xlim(xmin, xmax)
plt.title("Daily traded volume vs log return")
# plt.legend(bbox_to_anchor=(.02, .02), loc=3, borderaxespad=0., fontsize=fontsize)
plt.yscale('log')
plt.xlabel('Daily log return', fontsize=fontsize)
plt.ylabel('Traded volume (billion HUF)', fontsize=fontsize)
plt.yticks(ytics_left, [str(_) for _ in ytics_left])
plt.axvline(0, c='k', ls=':', lw=axvline_width)

plt.subplot(122)
for code,color in zip(bse_codes, bse_colors):
    if True: # plot all data sets
    #if 'RICHTER' == code: # plot only this data set
        groups = df[code].groupby(pd.cut(df[code]['LogReturn'], xbins))
        plot_centers = (xbins[:-1] + xbins[1:]) / 2
        plot_values = groups['Volume'].mean() / 1e+9
        plt.plot(plot_centers, plot_values, c=color, marker=marker, label=code, lw=0, markersize=2 * markersize)
plt.xlim(xmin, xmax)
plt.legend(bbox_to_anchor=(1.02, 1), loc=2, borderaxespad=0., fontsize=fontsize)
plt.title("Traded volume is averaged in bins of log return")
plt.yscale('log')
plt.xlabel('Daily log return', fontsize=fontsize)
plt.ylabel('Traded volume (billion HUF)', fontsize=fontsize)
plt.yticks(ytics_right, [str(_) for _ in ytics_right])
plt.axvline(0, c='k', ls=':', lw=axvline_width)

fig = plt.gcf()
fig.set_size_inches([13, 5])
plt.show()<jupyter_output><empty_output><jupyter_text>Volume vs Volatility of daily close1. Based on the below scatter plot what do you conclude for the relationship between daily log(volume) and log(volatility) ?2. Based on the roughly even distribution of the daily points in the plot what is your chance of having a high volume day ?<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
bse_colors = ['magenta','limegreen','blue']
markersize = 2


def calculate_monthly_volatility_and_average_traded_volume(data):
    '''For each month calculate the volatility of the daily close and the average daily traded volume.'''
    monthly_data = pd.DataFrame(columns=['volatility','average_volume'])
    dates = data.index    
    yearly_dates = dates.groupby(dates.year)
    for year in yearly_dates.keys():
        monthly_dates = pd.DatetimeIndex(yearly_dates[year]).groupby(pd.DatetimeIndex(yearly_dates[year]).month)
        for month in monthly_dates.keys():
            date_all = monthly_dates[month]
            date_first = min( date_all )
            close_daily_all = [ data.loc[date]['Close'] for date in date_all ]
            volume_daily_all = [ data.loc[date]['Volume'] for date in date_all ]
            volatility = np.std( close_daily_all )
            volume_daily_average = np.average( volume_daily_all )
            monthly_data.loc[date_first] = [volatility, volume_daily_average]

    return monthly_data


# volume vs volatility
monthly_data = dict()
for code, color in zip(bse_codes, bse_colors):
    if True: # all data sets
    #if 'RICHTER' == code: # only selected data set
        monthly_data[code] = calculate_monthly_volatility_and_average_traded_volume( df[code] )
        plt.plot(monthly_data[code]['volatility'], monthly_data[code]['average_volume'] / 1e+9,
                 c=color, marker=marker, label=code, lw=0, markersize=markersize)
plt.legend(bbox_to_anchor=(.98, .02), loc=4, borderaxespad=0., fontsize=fontsize)
plt.title("Monthly data: Average daily volume vs Volatility of daily close")
plt.xscale('log')
plt.yscale('log')
plt.xlabel('Volatility of daily close in a month (HUF)', fontsize=fontsize)
plt.ylabel('Average daily volume (billion HUF)', fontsize=fontsize)
plt.xticks([30, 100, 300, 1000],['30','100','300','1000'])
plt.yticks([1, 3, 10, 30],['1','3','10','30'])

fig = plt.gcf()
fig.set_size_inches([8, 6])
plt.show()<jupyter_output><empty_output><jupyter_text>Skewness of log returns distribution 1. What does the sum of the highest and the lowest value tell about a distribution ?2. Does the negative skew of SP500 mean that stock prices respond faster to negative news than to positive news ?| Name | Symbol and Calculation ||:-----|:------------|| Random variable | $X$ || Mean | $\mu = E\left[ \,X \,\right]$ || Variance | ${\displaystyle \sigma^{\,2} = E\left[ \, \left( \, X - \mu \, \right)^{\,2} \, \right] }$ || Volatility = Std.dev. | $\sigma$ || Skewness | ${\displaystyle E\left[\,\left(\frac{X-\mu}{\sigma}\,\right)^{\,3} \, \right]}$|<jupyter_code>import numpy as np
import pandas as pd
from scipy import stats
import datetime
import matplotlib.pyplot as plt
FONTSIZE = 14

# select data sets to be analyzed, set their display names, set colors for plotting them
fred_selected_codes = {'AAA10Y':'AAA10Y', 'GOLDPMGBD228NLBM':'GOLD', 'DEXJPUS':'JPYUSD', 
    'ICERATES1100USD1Y': 'US1YSW', 'SP500': 'S&P500', 'WILLRESIPR':'WILLSH'}
fred_colors = ['black','blue','green','black','red','red']
fred_markers = ['o','^','v','x','o','x']
fred_fill = ['none','none','none','none','none','none']
axhline_width = 0.5 
markersize = 6
markeredgewidth = 1
display_len = 32 # display this number of points
linthreshy = 0.002 # threshold for the simlog y scaling


# read fred data sets again without the . lines, calculate log return
for code in fred_selected_codes:
    df[code] = pd.read_csv(data_dir + os.sep + code + "." + file_ext, na_values='.')
    df[code]['LogReturn'] = np.log(df[code][code]) - np.log(df[code][code]).shift()


# write skewness and plot differences
print("Skew\tName\tLong Name of Data Set")
for code, color, marker, fill in zip(fred_selected_codes, fred_colors, fred_markers, fred_fill):
    if True: # plot all data
    #if code.startswith('AA'): # plot selected
        log_returns = df[code]['LogReturn']
        log_returns_num = log_returns[ (log_returns>-1e+6) & (log_returns<1e+6) ] # select numbers
        sorted_log_returns = pd.Series.sort_values( log_returns_num ).tolist() # sort into ascending order
        sum_reversed = np.add( sorted_log_returns, sorted_log_returns[::-1] ) # add list to itself reversed
        sum_reversed = sum_reversed[:display_len:] # keep only the requested number of items from the start
        display_name = fred_selected_codes[code]
        print("%+.2f\t%s\t%s" % (stats.skew(sorted_log_returns), display_name, code))
        is_first = False
        plt.plot(1 + np.arange(len(sum_reversed)) , sum_reversed,
                 c=color, marker=marker, label=display_name, lw=0, fillstyle=fill, 
                 markersize=markersize, markeredgewidth=markeredgewidth)
plt.title('Sum of $n^{th}$ lowest and $n^{th}$ highest daily log returns', fontsize=FONTSIZE)
plt.legend(bbox_to_anchor=(1.02, 1), loc=2, borderaxespad=0., fontsize=FONTSIZE)
plt.xscale('log')
plt.yscale('symlog', linthreshy=linthreshy)
plt.xlabel('Index of sorted daily log returns ($n$)', fontsize=FONTSIZE)
plt.xticks([1,3,10,30],['1','3','10','30'])
plt.yticks([-0.01,-0.001,0,0.001,0.01,0.1],['$-\,0.01$','$-\,0.001$','','0.001','0.01','0.1'])
plt.axhline(0, c='k', ls='--', lw=axhline_width)

fig = plt.gcf()
fig.set_size_inches([7, 5])
plt.show()<jupyter_output>Skew	Name	Long Name of Data Set
+0.90	AAA10Y	AAA10Y
-0.06	GOLD	GOLDPMGBD228NLBM
-0.42	JPYUSD	DEXJPUS
-0.88	US1YSW	ICERATES1100USD1Y
-1.19	S&P500	SP500
-0.44	WILLSH	WILLRESIPR<jupyter_text>Task: yearly mean of daily log returns and yearly volatilityFor each year separately, calculate (1) the mean of the daily log return of WMT, and (2) the volatility of the daily log returns of WMT.  Answer<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import os
FONTSIZE = 12

# data processing
df = pd.read_csv(os.path.join('data', 'WMT.csv'))
df['Year'] = pd.to_datetime(df.Date).dt.year
df['LogReturnDaily'] = np.log(df['Close']) - np.log(df['Close'].shift())
daily_log_ret_yearly_mean = df.groupby(df.Year).LogReturnDaily.agg('mean')
daily_log_ret_vol = df.groupby(df.Year).LogReturnDaily.agg('std')

# plotting
fig, axes = plt.subplots(figsize=(7, 5))
plt.plot(daily_log_ret_yearly_mean, markersize=10, marker='.', linestyle=':', linewidth=1, color='g')
plt.plot(daily_log_ret_vol, markersize=10, marker='.', linestyle=':', linewidth=1, color='b')
plt.title('Yearly means (green) and volatility (blue) of the daily log returns of WMT', fontsize=FONTSIZE)
plt.xlabel('Y e a r', fontsize=FONTSIZE)
plt.ylabel('Daily log returns: avg. and std.dev', fontsize=FONTSIZE)
plt.axhline(0, c='k', ls=':', lw=.3)
plt.show()

# investigate the average returns in sorted order
daily_log_ret_yearly_mean.sort_values(inplace=True)
display('Yearly mean of daily log returns in sorted order', daily_log_ret_yearly_mean)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14158977998580555, 'fraction_numerical': 0.02204084851352417, 'mean_word_length': 4.538982310548155, 'pattern_counts': {'":': 1, '<': 37, '<?xml version=': 0, '>': 40, 'https://': 22, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '648850', 'license': 'Apache-2.0', 'n_tokens_mistral': 9403, 'n_tokens_neox': 8740, 'n_words': 2343, 'path': '.ipynb_checkpoints/07 Observed Distributions-checkpoint.ipynb', 'repo_name': 'fij/btup', 'size': '25362'}