<jupyter_start><jupyter_text>** Project - Testing for the difference in means between three different exchange traded funds that follow the ESG framwework ** 1. Prior to the period of greatest volatility. Is there a significant difference in the means of daily returns between the three different Exchange Traded Funds?2. During the period of greatest volatility. Is there a significant difference in the means of daily returns between the three different Exchange Traded Funds?3. If there is any significant difference, how large was this difference and  in which period? |Ticker Name                       |              Meaning                ||----------------------------------|-------------------------------------||     ESGE ~ ESG MSCI EM ETF       | Emerging Markets ESG                ||     ESGD ~ ESG MSCI EAFE ETF     | Europe, Australia, Far East ESG     ||     ESGU ~ ESG MSCI USA ETF      | U.S.A. ESG                          | |     ^VIX                         | Volatility Index                    |<jupyter_code># Setting up the enviroment
%matplotlib inline
import pandas as pd
import numpy as np
from scipy import stats
import seaborn as sns
import matplotlib.pyplot as plt
import datetime
import matplotlib.dates as mdates
import statsmodels.api as sm
import pylab
from statsmodels.formula.api import ols
from statsmodels.stats.anova import AnovaRM
from statsmodels.stats.multicomp import pairwise_tukeyhsd
from statsmodels.graphics.gofplots import qqplot
sns.set(style="ticks", context="talk")
plt.style.use("dark_background")
# Bringing the datasets into the notebook
esge = pd.read_csv('Datasets/ESGE.csv')
esgd = pd.read_csv('Datasets/ESGD.csv')
esgu = pd.read_csv('Datasets/ESGU.csv')
vix = pd.read_csv('Datasets/^vix.csv')

print('ESGE shape: ', esge.shape)
print('ESGD shape: ', esgd.shape)
print('ESGU shape: ', esgu.shape)
print('vix shape: ', vix.shape)
# Exploring the first rows of the ESGE dataset
esge.head(3)
# Exploring the first rows of the ESGD dataset
esgd.head(3)
# Exploring the first rows of the ESGU dataset
esgu.head(3)
# Exploring the first rows of the VIX dataset
vix.head(3)
# Converting the Date columns into date type
esge['Date'] = pd.to_datetime(esge['Date'])
esgd['Date'] = pd.to_datetime(esgd['Date'])
esgu['Date'] = pd.to_datetime(esgu['Date'])
vix['Date'] = pd.to_datetime(vix['Date'])
# Setting the date column as the index 
esge.set_index(esge['Date'], inplace=True)
esgd.set_index(esgd['Date'], inplace=True)
esgu.set_index(esgu['Date'], inplace=True)
vix.set_index(vix['Date'], inplace=True)
# Creating a new column named "Daily Returns" with the calculated daily returns using the open and adjusted close price.
esge['Daily Returns ESGE'] = esge['Adj Close'] / esge['Adj Close'].shift(1) - 1
esgd['Daily Returns ESGD'] = esgd['Adj Close'] / esgd['Adj Close'].shift(1) - 1
esgu['Daily Returns ESGU'] = esgu['Adj Close'] / esgu['Adj Close'].shift(1) - 1
vix['Daily Change VIX'] = vix['Adj Close'] / vix['Adj Close'].shift(1) - 1
# Combinig the four dailiy return columns into one dataframe:
esge_daily = esge['Daily Returns ESGE']
esgd_daily = esgd['Daily Returns ESGD']
esgu_daily = esgu['Daily Returns ESGU']
vix_daily = vix['Daily Change VIX']

daily_returns_df = pd.concat([esge_daily, esgd_daily, esgu_daily, vix_daily], axis=1)
daily_returns_df.head()
# Checking for the total number of NaN values in the dataframe:
daily_returns_df.isna().sum()
# Replacing NaN values with zeros instead of deleting them:
daily_returns_df.fillna(0, axis=1, inplace=True)
daily_returns_df.head()
# Creating three different series with the average daily price for each ETF
esge['Avg Daily Price'] = (esge['High'] + esge['Low']) / 2
esgd['Avg Daily Price'] = (esgd['High'] + esgd['Low']) / 2
esgu['Avg Daily Price'] = (esgu['High'] + esgu['Low']) / 2
# Plotting the different average dailiy prices for each ETF:
plt.figure(figsize=(20,8))
sns.lineplot(x = esge.index, y = esge['Avg Daily Price'])
sns.lineplot(x = esgd.index, y = esgd['Avg Daily Price'])
sns.lineplot(x = esgu.index, y = esgu['Avg Daily Price'])
plt.legend(labels=['ESGE Average Daily Price', 'ESGD Average Daily Price', 'ESGU Average Daily Price'])
plt.title('Average Daily Prices for ech ETF')
plt.savefig('prices')
plt.show()
# From the previous chart we can see a common trend between the three ETF´s, which suggests a correlation between them.
# Checking for the correlations between the four different returns:
corr_matrix = daily_returns_df.corr()
corr_matrix.style.background_gradient(cmap='coolwarm')<jupyter_output><empty_output><jupyter_text>From the correlation plot we can observe the following two points:1. The three different ETF's are strongly positively correlated 2. The three different ETF's are strongly negatively correlated with the daily change of the VIX  The next step in the analysis is to evaluate the period where the VIX has increased the most in order to split the dataset into two different periods, one prior to the high volatility period and another during this period of high volatility.<jupyter_code># Creating a new column in the VIX dataframe that contains the average between the High and Low price for each day
vix['Avg Daily Price'] = (vix['High'] + vix['Low']) / 2
vix['Avg Daily Price'].head()
# Plotting the volatility index (VIX) average price to look for the period with the greatest volatility
plt.figure(figsize=(20,5))
sns.lineplot(x = vix.index, y = vix['Avg Daily Price'])
plt.legend(labels = ['Average Daily Price VIX'])
plt.axvspan(*mdates.datestr2num(['2020-01-31', '2020-08-06']), color='red', alpha=0.5)
plt.savefig('Volatility')
plt.show()<jupyter_output><empty_output><jupyter_text>From this chart we can conclude that the period with the greatest volatility was from the beginning of Febraury 2020 to the end of our observations, August 2020.Next step is to divide each dataset into two different periods: -One from the first observation to Febrary 2020.  -Another from February 2020 to the last observation.<jupyter_code># This will be the first period dataframe
first_period_df = daily_returns_df.loc['2016-08-08':'2020-01-31']
first_period_df.shape
first_period_df.head(1)
# This will be the second period dataframe
second_period_df = daily_returns_df.loc['2020-02-01':'2020-08-06']
second_period_df.shape<jupyter_output><empty_output><jupyter_text>The next step in the analysis is to check the daily return distribution for each instrument in the first period.For this we will plot each ditribution as well as stats.describe.<jupyter_code># Checking for normality of distrubution; first period - ESGE:
plt.figure(figsize=(10,4))
sns.distplot(first_period_df['Daily Returns ESGE'], kde=True, bins=20)
plt.title('ESGE ETF Distribution of Daily Returns')
plt.show()
print('ESGE ETF Daily Returns Statistics: ', stats.describe(first_period_df['Daily Returns ESGE']))
# Checking for normality of distrubution; first period - ESGD:
plt.figure(figsize=(10,4))
sns.distplot(first_period_df['Daily Returns ESGD'], kde=True, bins=20)
plt.title('ESGD ETF Distribution of Daily Returns')
plt.xlim(-0.02, 0.02)
plt.show()
print('ESGD ETF Daily Returns Statistics: ', stats.describe(first_period_df['Daily Returns ESGD']))
# Checking for normality of distrubution; first period - ESGU:
plt.figure(figsize=(10,4))
sns.distplot(first_period_df['Daily Returns ESGU'], kde=True, bins=50)
plt.title('ESGU ETF Distribution of Daily Returns')
plt.xlim(-0.02, 0.02)
plt.show()
print('ESGU ETF Daily Returns Statistics: ', stats.describe(first_period_df['Daily Returns ESGU']))<jupyter_output><empty_output><jupyter_text>From the different distribution plots we can observe the following:1. The kurtosis and skewness of the ESGE ETF distribution seems to follow a normal distribution.2. The skewness of the ESGD ETF distribution also seems to follow a normal one, although the kurtosis of 3.37 it's a little bit high.3. The kurtosis of the ESGU ETF seems to be a little vit high, further exploration will be needed.Next step is to create qq plots and run a shapiro test for each ETF daily return for the first period in order to further analyze the normality of returns.<jupyter_code># Creating a qqplot; first period - ESGE
plt.figure(figsize=(10,4))
stats.probplot(first_period_df['Daily Returns ESGE'], dist='norm', plot=plt)
plt.title('ESGE Daily Return QQ Plot')
plt.show()
# Creating a qqplot; first period - ESGD
plt.figure(figsize=(10,4))
stats.probplot(first_period_df['Daily Returns ESGD'], dist='norm', plot=plt)
plt.title('ESGD Daily Return QQ Plot')
plt.show()
# Creating a qqplot; first period - ESGU
plt.figure(figsize=(10,4))
stats.probplot(first_period_df['Daily Returns ESGU'], dist='norm', plot=plt)
plt.title('ESGU Daily Return QQ Plot')
plt.show()<jupyter_output><empty_output><jupyter_text>From the QQ plots we can conclude that the three ETF's  different distributions of daily returns for the first period approximate a normal distribution.The next step is to repeat the previous analysis, this time for the second period.<jupyter_code># Checking for normality of distrubution; second period - ESGE:
plt.figure(figsize=(10,4))
sns.distplot(second_period_df['Daily Returns ESGE'], kde=True, bins=20)
plt.title('ESGE ETF Distribution of Daily Returns')
plt.xlim(-0.10,0.10)
plt.show()
print('ESGE ETF Daily Returns Statistics: ', stats.describe(second_period_df['Daily Returns ESGE']))
# Checking for normality of distrubution; second period - ESGD:
plt.figure(figsize=(10,4))
sns.distplot(second_period_df['Daily Returns ESGD'], kde=True, bins=20)
plt.title('ESGD ETF Distribution of Daily Returns')
plt.xlim(-0.10,0.10)
plt.show()
print('ESGD ETF Daily Returns Statistics: ', stats.describe(second_period_df['Daily Returns ESGD']))
# Checking for normality of distrubution; second period - ESGU:
plt.figure(figsize=(10,4))
sns.distplot(second_period_df['Daily Returns ESGU'], kde=True, bins=20)
plt.title('ESGU ETF Distribution of Daily Returns')
plt.xlim(-0.10,0.10)
plt.show()
print('ESGU ETF Daily Returns Statistics: ', stats.describe(second_period_df['Daily Returns ESGU']))<jupyter_output><empty_output><jupyter_text>From the different distribution plots we can observe the following:1. The kurtosis of the three different distributions seems to be high. Which suggests a leptokurtic distribution for each one in the second period.Next step is to create qq plots and run shapiro tests for each ETF daily return for the first period in order to further analyze the normality of returns.<jupyter_code># Creating a qqplot; second period - ESGE
plt.figure(figsize=(10,4))
stats.probplot(second_period_df['Daily Returns ESGE'], dist='norm', plot=plt)
plt.title('ESGE Daily Return QQ Plot')
plt.show()
# Creating a qqplot; second period - ESGD
plt.figure(figsize=(10,4))
stats.probplot(second_period_df['Daily Returns ESGD'], dist='norm', plot=plt)
plt.title('ESGD Daily Return QQ Plot')
plt.show()
# Creating a qqplot; second period - ESGU
plt.figure(figsize=(10,4))
stats.probplot(second_period_df['Daily Returns ESGU'], dist='norm', plot=plt)
plt.title('ESGU Daily Return QQ Plot')
plt.show()<jupyter_output><empty_output><jupyter_text>From the QQ plots we can conclude that the three ETF's  different distributions of daily returns for the second period does approximate a normal distribution. The next step of the analysis is to perform a Repeated Measures ANOVA test for dependent variables in order to answer the following questions about our data:1. Prior to the period of greatest volatility. Is there a significant difference in the means of daily returns between the three different Exchange Traded Funds?2. During the period of greatest volatility. Is there a significant difference in the means of daily returns between the three different Exchange Traded Funds?3. If there is any significant difference, how large was this difference and  in which period?<jupyter_code># The first step is to apply a melt function to the first period dataframe:
first_period_df = first_period_df.drop(['Daily Change VIX'], axis=1)
first_period_df['Date'] = first_period_df.index
first_melt = pd.melt(first_period_df, id_vars='Date', var_name="Index", value_name="Return")
first_melt.head(5)
# Performing the repeated measures ANOVA test for the first period:
anova = AnovaRM(data = first_melt, depvar = 'Return', subject = 'Index', within = ['Date']).fit()
anova.summary()<jupyter_output><empty_output><jupyter_text>According to the P Value of 0:We see evidence that at least one of the daily return means of the three different ETF's in the first period is significantly different.<jupyter_code># Applying the melt function to the second period dataframe:
second_period_df = second_period_df.drop(['Daily Change VIX'], axis=1)
second_period_df['Date'] = second_period_df.index
second_melt = pd.melt(second_period_df, id_vars='Date', var_name="Index", value_name="Return")
second_melt.head(5)
# Performing the repeated measures ANOVA test for the second period:
anova = AnovaRM(data = second_melt, depvar = 'Return', subject = 'Index', within = ['Date']).fit()
anova.summary()<jupyter_output><empty_output><jupyter_text>Interpretation of results: According to the P Value of 0:We see evidence that at least one of the daily return means of the three different ETF's in the second period is significantly different.Next step is to analyze the mean and standard deviation for each ETF in each period<jupyter_code># Analyzing the mean and standard deviation for ESGE in the first period:
mean_esge_1p = first_period_df['Daily Returns ESGE'].mean()
std_esge_1p = first_period_df['Daily Returns ESGE'].std()
# Analyzing the mean and standard deviation for ESGD in the first period:
mean_esgd_1p = first_period_df['Daily Returns ESGD'].mean()
std_esgd_1p = first_period_df['Daily Returns ESGD'].std()
# Analyzing the mean and standard deviation for ESGU in the first period:
mean_esgu_1p = first_period_df['Daily Returns ESGU'].mean()
std_esgu_1p = first_period_df['Daily Returns ESGU'].std()
# Creating a dataframe with the different means values:
means_first_period = pd.DataFrame({'ETFs':['Mean of ESGE', 'Mean of ESGD', 'Mean of ESGU'], 'Daily Return Means': [mean_esge_1p, mean_esgd_1p, mean_esgu_1p]})
means_first_period
# Creating a dataframe with the different standard deviation values:
std_first_period = pd.DataFrame({'ETFs':['Std of ESGE', 'Std of ESGD', 'Std of ESGU'], 'Daily Return Standard Deviations': [std_esge_1p, std_esgd_1p, std_esgu_1p]})
std_first_period
# Creating a bar chart for each variable in the first period:
ax1 = means_first_period.plot.bar(x = 'ETFs', y = 'Daily Return Means', figsize=(11,7), color = 'lavender')
ax2 = std_first_period.plot.bar(x = 'ETFs', y = 'Daily Return Standard Deviations', figsize=(11,7), color = 'lightsteelblue', alpha = 0.7)<jupyter_output><empty_output><jupyter_text>From this graphs we can observe the following:The ESGU ETF daily return is the one that is significantly different from the others, and for the same measure of risk it seems that it outperforms both the ESGE and the ESGD ETFs (as measured by the daily mean return).Next step is to repeat the same analysis for the second period:<jupyter_code># Analyzing the mean and standard deviation for ESGE in the second period:
mean_esge_2p = second_period_df['Daily Returns ESGE'].mean()
std_esge_2p = second_period_df['Daily Returns ESGE'].std()
# Analyzing the mean and standard deviation for ESGD in the second period:
mean_esgd_2p = second_period_df['Daily Returns ESGD'].mean()
std_esgd_2p = second_period_df['Daily Returns ESGD'].std()
# Analyzing the mean and standard deviation for ESGU in the second period:
mean_esgu_2p = second_period_df['Daily Returns ESGU'].mean()
std_esgu_2p = second_period_df['Daily Returns ESGU'].std()
# Creating a dataframe with the different means values:
means_second_period = pd.DataFrame({'ETFs':['Mean of ESGE', 'Mean of ESGD', 'Mean of ESGU'], 'Daily Return Means': [mean_esge_2p, mean_esgd_2p, mean_esgu_2p]})
means_second_period
# Creating a dataframe with the different standard deviation values:
std_second_period = pd.DataFrame({'ETFs':['Std of ESGE', 'Std of ESGD', 'Std of ESGU'], 'Daily Return Standard Deviations': [std_esge_2p, std_esgd_2p, std_esgu_2p]})
std_second_period
# Creating a bar chart for each variable in the second period:
ax1 = means_second_period.plot.bar(x = 'ETFs', y = 'Daily Return Means', figsize=(11,7), color = 'lavender')
ax2 = std_second_period.plot.bar(x = 'ETFs', y = 'Daily Return Standard Deviations', figsize=(11,7), color = 'lightsteelblue', alpha = 0.7)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1069638219484206, 'fraction_numerical': 0.014253789937790662, 'mean_word_length': 5.5369127516778525, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '188264', 'license': 'MIT', 'n_tokens_mistral': 5460, 'n_tokens_neox': 5021, 'n_words': 1838, 'path': 'statistical_analysis_esg_etf.ipynb', 'repo_name': 'FelipeGalindo/Statistical-Analysis-Exchange-Traded-Funds', 'size': '16557'}