<jupyter_start><jupyter_text><NAME> Note: Many formulas are recycled from homeworks and workbooks to complete homeworks in collaboration with team members. You may notice some similarities in formulas with them.<jupyter_code>import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
import statsmodels.api as sm
import matplotlib.pyplot as plt
import seaborn as sns
from dataclasses import dataclass
import scipy.stats
from scipy.stats import norm
from scipy.stats import ttest_ind
import itertools
from arch import arch_model
from arch.univariate import GARCH, EWMAVariance 
from sklearn.linear_model import LinearRegression
import warnings
sns.set()
# Load Files

# # HW1
# #assume excel sheet in same folder as jupyter notebook
path_to_data_file = r'D:\Documents\Education\University of Chicago\Q1\Portfolio Theory\Data\multi_asset_etf_data.xlsx'
df_etf = pd.read_excel(path_to_data_file, sheet_name="excess returns") # This is already excess return so tilde_r
# #convert dataset to returns only with date as index
df_etf = df_etf.set_index('Date')

# # HW2
path = r'D:\Documents\Education\University of Chicago\Q1\Portfolio Theory\Data\proshares_analysis_data.xlsx'
df_hedge_fund_series = pd.read_excel(path, sheet_name='hedge_fund_series').set_index('date')
df_merrill_facs = pd.read_excel(path, sheet_name='merrill_factors').set_index('date')
df_other_asset = pd.read_excel(path, sheet_name='other_data').set_index('date')
df_hedge_fund_series['SPY US Equity'] = df_merrill_facs['SPY US Equity']
# df_merrill_facs.head()

# HW3
df_bsa = pd.read_excel(r'D:\Documents\Education\University of Chicago\Q1\Portfolio Theory\Data\barnstable_analysis_data.xlsx',sheet_name='data').set_index('date')
#df3['tilde_r'] = df3['SPX'] - df3['TB1M'] # Excess return on the market<jupyter_output><empty_output><jupyter_text>True/False and Short Answer 1. False. Mean Variance optimization attempts to optimize the portfolio based on minimizing the covariance matrix. With regards to Sharpe Ratios, the optimization can end up going long a high sharpe ratio asset and short a very similar asset which has a slightly less high sharpe ratio. Just because a sharpe ratio is low does not mean it will have a low weighting in the mean-variance optimization. 2. False. It ultimately depends on your strategy, but with respect to replicating the levelred returns of a given index, then the LETF will make more sense in the short term. On a daily basis, the LETF tracks the leverage of the index close to perfect if not perfect. The issue with LETFs is that the returns are reset on a daily basis in order to account for subscriptions and redemptions to the LETF. Due to this, one cannot realize compounded gains that one would have if invested directly in the given index or portfolio, at whatever levered rate is desired. The answer to this would be True if the investor had no alternative, i.e. is not able to invest in the underlying asset of the LETF in the long term. 3. Include an intercept! Given we would have only an year of data, we would not be confident in the mean return of our data. Typically we would want to have a couple decades of data before drawing any conclusions with regards to a mean. By including an intercept, we are allowing the regression to focus on the variation of the data, wheras if we didn't include the intercept, the regression would try to fit SPY and IEF to explain both trend (mean) and variation. In other words, the regression will have more explanatory power with regards to the betas if we include an intercept. 4. HDG is effective in tracking HFRI in sample and out of sample, to an extent. HDG is an actual fund, while the HFRI is a survey and not actually tradable. Therefore, there are costs associated with actualizing a theoretical return. Although HDG will not be able to ever perform as well as the HFRI given market frictions, it is able to replicate reasonably well with regards to Sharpe and Treynore ratios. The HDG is not able to capture alpha, and we can see in HW2 and the analysis of it in class that the alphas and information ratio of HDG are negative. 5. The most typical reason this can happen is that the hedge fund is using a different benchmark, and most likely that benchmark is a simple, broad index like SPX (or the dividend-including SPY). When compared to a broad index, the activities of the hedge fund are most likely not attributable to the beta of the index, which results in any "over-performance" to go into alpha. However when compared to Merrill's six factors, much of the returns can be explained by the factors, resulting in a negative alpha. Context is important in stating these performance figures.  Allocation  Q1<jupyter_code>df_merrill_facs
df_merrill_facs_sx = df_merrill_facs.subtract(df_merrill_facs['USGG3M Index'], axis=0).drop(columns='USGG3M Index')
###Assumes data is made up of monthly returns!!!
def annualize(d):
    return d * 12

def annualizeMuSigSharpe(d):
    d_a = d * 12
    return pd.DataFrame({'Mean':d_a.mean(),'Vol':d_a.std()/np.sqrt(12),'Sharpe': d.mean() / d.std() * np.sqrt(12)})
# Mean-Variance Frontier (MV)

# Compute and display the weights, mu and Sigma of the tangency portfolio:
# This function takes ANNUAL returns data
def tangent_portfolioAnnRet(d,diag=False):
    sigma = d.cov()
    N = sigma.shape[0] # N is the number of assets
    Sigma_adj = sigma.copy()
    if diag:
        Sigma_adj.loc[:,:] = np.diag(np.diag(Sigma_adj))
    
    mu_tilde = d.mean()
    Sigma_inv = np.linalg.inv(Sigma_adj) # Compute convariance inverse
    weights = Sigma_inv @ mu_tilde / (np.ones(N) @ Sigma_inv @ mu_tilde) # Tangency weight, (Lecture 1, Slide 40)
    omega_tangency = pd.Series(weights, index=mu_tilde.index)
    return omega_tangency, mu_tilde, sigma

# Tangency portfolios (Assume the data is annualized)
omega_tangency, mu_tilde, Sigma = tangent_portfolioAnnRet(annualize(df_merrill_facs_sx)) 
omega_tangency.to_frame('MV Tangency Weights')<jupyter_output><empty_output><jupyter_text>Q2<jupyter_code># Compute and display the weights of MV portfolios, target must be ANNUAL!!!!!!!
def Target_MV_Port(df_tilde, target_return = 0.01*12, diagonalize_Sigma = False):
    
    # Compute efficient portfolio weights first (omega_tangency)
    omega_tangency, mu_tilde, Sigma = tangent_portfolioAnnRet(df_tilde, diag=diagonalize_Sigma)
    Sigma_adj = Sigma.copy()
    if diagonalize_Sigma:
        Sigma_adj.loc[:,:] = np.diag(np.diag(Sigma_adj))
        
    Sigma_inv = np.linalg.inv(Sigma_adj) # Compute covariance inverse if applicable
    N = Sigma_adj.shape[0]
    # Compute allocation delta (Lecture 1, Slide 50)
    delta_tilde = ((np.ones(N) @ Sigma_inv @ mu_tilde)/(mu_tilde @ Sigma_inv @ mu_tilde)) * target_return
    # Compute tangency weight wrt portfolio target return (Lecture 1, Slide 50)
    omega_star = delta_tilde * omega_tangency
    return omega_star, mu_tilde, Sigma_adj

# Tangency portfolios weight allocated wrt given portfolio excess mean return
mu_p = .02
mu_p_ann = mu_p * 12
omega_star, mu_tilde, sigma = Target_MV_Port(annualize(df_merrill_facs_sx), target_return = mu_p_ann)
omega_star_df = omega_star.to_frame('Target mean return MV Portfolio Weights')
omega_star_df
omega_star_df.sum()<jupyter_output><empty_output><jupyter_text>Given the target mean return MV Portfolio Weights do not sum to 1, the optimal portfolio will invest in the risk free rate. Specifically, given the weights of 1.1575, we would borrow 15.75% of the portfolio starting capital in the risk free rate.  Q3<jupyter_code># Function 4

# Compute the mean, volatility and Sharpe ratio for the tangency portfolio
# Can be used in general for tangency, assuming weight, mean, covariance matrix and annual factor is computed
def portfolio_stats(omega, mu_tilde, Sigma, annualize_fac=1):
    mean = (mu_tilde @ omega) # Mean -> NOTE: mu_tilde = df_tilde.mean()
    vol = np.sqrt(omega @ Sigma @ omega) / np.sqrt(annualize_fac) # Volatility -> NOTE: Sigma = df_tilde.cov()
    sharpe_ratio = mean / vol # Sharpe ratio
    return round(pd.DataFrame(data = [mean, vol, sharpe_ratio], 
                              index = ['Mean', 'Volatility', 'Sharpe'], 
                              columns = ['Portfolio Stats']), 4)

# Pass the computed omega_tan, mu_tilde and Sigma, with annualization to get mean, volatility and sharpe ratio
portfolio_stats(omega_star, mu_tilde, sigma, 12)<jupyter_output><empty_output><jupyter_text>Q4<jupyter_code>merrillp1 = df_merrill_facs_sx.loc[:'2018',:]
omega_starp1, mu_tildep1, sigmap1 = Target_MV_Port(annualize(merrillp1), target_return = mu_p_ann)
omega_star_dfp1 = omega_star.to_frame('Target mean return MV Portfolio Weights')
omega_star_dfp1
### assumes that returns data is annualized!!!
def RiskOnlyPortfMomentsAnnRet(omega_table,mu_tilde,Sigma):
    mu_tilde_p = omega_table.T @ mu_tilde
    vol_p = np.sqrt(omega_table.T @ Sigma @ omega_table) / np.sqrt(12)
    Sharpe_p = mu_tilde_p / vol_p
    Cov_p = Sigma @ omega_table
    return mu_tilde_p, vol_p, Sharpe_p, Cov_p

###Function for in sample weights vs Out of Sample portf
###assumes ANNUALIZED data
def ISweightsOSAnnualdataSharpe(isweights,osdata):
    ###pull out sample data properties
    osmean = tangent_portfolioAnnRet(osdata)[1]
    ossigma = tangent_portfolioAnnRet(osdata)[2]
    ###Apply in sample weights. Commented section is the manual way.
    """os_mean_p = isweights @ osmean
    os_vol_p = np.sqrt(isweights.T @ ossigma @ isweights) / np.sqrt(12)
    os_Sharpe = os_mean_p / os_vol_p"""
    os_Sharpe = RiskOnlyPortfMomentsAnnRet(isweights,osmean,ossigma)[2]
    return os_Sharpe

###Function for in sample weights vs Out of Sample portf
###assumes ANNUALIZED data
def ISweightsOSAnnualdataMeanVolSharpe(isweights,osdata):
    ###pull out sample data properties
    osmean = tangent_portfolioAnnRet(osdata)[1]
    ossigma = tangent_portfolioAnnRet(osdata)[2]
    ###Apply in sample weights. Commented section is the manual way.
    """os_mean_p = isweights @ osmean
    os_vol_p = np.sqrt(isweights.T @ ossigma @ isweights) / np.sqrt(12)
    os_Sharpe = os_mean_p / os_vol_p"""
    os_Sharpe = portfolio_stats(isweights, osmean, ossigma, annualize_fac=12)
    return os_Sharpe

#ISweightsOSAnnualdataSharpe(omega_starp1,annualize(df_merrill_facs_sx.loc['2019':,:]))

ISweightsOSAnnualdataMeanVolSharpe(omega_starp1,annualize(df_merrill_facs_sx.loc['2019':,:]))<jupyter_output><empty_output><jupyter_text>Q5 OOS fragility would be even worse! Commodities are much more volatile in general, and have spurious means across time, as they are highly impacted by anything that goes on in the world (weather (too much or too little rain), governments (OPEC), politics, technology (Semiconductor or solar polar demand), diseases (swine flu, bird flu) etc.) The covariance between the assets can change for various reasons, and even a slight change in the expected mean will drastically change the optimal weights, let alone mean/vol/sharpe.  Hedging and Replication  Q1<jupyter_code>df_merrill_facs_sx
# Function 12

# Run regression using no intercept
# Arguments: y dataframe, x dataframe
# Returns: betas, R-squared, tracking error = epsilon of the residual
def OLSNOIntercept(y,x):
    myx = x
    OLS = sm.OLS(y,myx,missing='drop').fit()
    TE = OLS.resid.std() * np.sqrt(12)
    return ('Beta:',(OLS.params[:])),('R-Squared: {:.5f} Tracking Error: {:.5f}'.format(OLS.rsquared,TE))
OLSNOIntercept(df_merrill_facs_sx['EEM US Equity'],df_merrill_facs['SPY US Equity'])<jupyter_output><empty_output><jupyter_text>For every dollar we invest in EEM, we would want 92 cents of SPY.  Q2<jupyter_code># Function 13

# Returns: The mean of the fitted value W/O intercept
def ANNUALOLSNOInterceptMean(y,x):
    OLS = sm.OLS(y,x,missing='drop').fit()
    return print("The Annualized mean of the fitted value is {:.4f}".format((OLS.params[:][0] * x ).mean() * 12 ))# print("The Annualized mean of the fitted value is {:.4f}".format(()))#.sum(axis=1).mean() * 12))
ANNUALOLSNOInterceptMean(df_merrill_facs_sx['EEM US Equity'],df_merrill_facs['SPY US Equity'])
# Function 14

# Calculate the mean of the fitted value r
# Returns: The mean of the fitted value W intercept
def ANNUALOLSInterceptVol(y,x):
    OLS = sm.OLS(y,x,missing='drop').fit()
    return print("The Annualized vol of the {} is {:.4f}".format(y.name,(OLS.params[:][0] * x).std() * np.sqrt(12)))
    return (OLS.params[:][0] * x).std() * np.sqrt(12)
ANNUALOLSInterceptVol(df_merrill_facs_sx['EEM US Equity'],df_merrill_facs['SPY US Equity'])
print('The Annualized Sharpe Ratio of the EEM US Equity is {}'.format(.0136/.1247))<jupyter_output>The Annualized Sharpe Ratio of the EEM US Equity is 0.10906174819566959<jupyter_text>Q3<jupyter_code>df_merrill_facs_sx['EEM US Equity'].mean() * 12<jupyter_output><empty_output><jupyter_text>The hedge ratio calculation regression formula yields a very different mean from the actual one of EEM. This is because we did not include an intercept. By not including an intercept, the regression does not try to fit trend (the mean), and therefore the hedge ratio is accounting for only the variation in SPY with respect to EEM. Choosing not to include an intercept can result in very different means, and this example makes it clear.  Q4 With respect to emerging markets, both SPY and the Russell index, which focus on the US market, may make it difficult to show a purely independent hedging relationship. Although SPY and the Russell are different (russell includes small caps), there may be some multicollinearity with respect to emerging markets. The hedge as a whole may work, but the allocation between the two indices may not always make sense.  Modeling Risk  Q1<jupyter_code>drisk =.drop(columns=['USGG3M Index','EEM US Equity','IWM US Equity','EUO US Equity'])
drisk
# meanvoltable: Compiles a table with mean/vol stats using linear or log returns.
# REQUIRES risk free column to be at beginning. 

# Arguments: d= dataframe, r = ARRAY of year ranges, ann_factor = annualization component, compound = linear(default) vs log

def meanvoltable(d,r,riskfree=-1,ann_factor=1,compound=False):
    if compound==True:
            d = np.log(d+1)
    melt = pd.melt(d,ignore_index=False).sort_index()
    arrlist = []

    
    for start,end in r:
        arr = pd.pivot_table(melt[start:end], index="variable", values="value",aggfunc=[np.mean, np.std])
        arr[("mean","value")] *= ann_factor
        arr[("std","value")] *= np.sqrt(ann_factor)
        arr.columns = pd.MultiIndex.from_arrays([[f"{start}-{end}",f"{start}-{end}"],["mean","vol"]],names=["year","factor"])
        arrlist.append(arr)
    
    output = pd.concat(arrlist,axis=1)
    
    #in case risk free returns are at the end of df, change d.columns[0] to d.columns[-1]
    for i in range(len(output.columns)):
        if output.columns[i][1] == 'mean':
            output.loc['Excess Returns',output.columns[i]] = output.loc[d.columns[0],(output.columns[i][0], output.columns[i][1])]\
                            - output.loc[d.columns[1],(output.columns[i][0], output.columns[i][1])]
        elif output.columns[i][1] == 'vol':
            output.loc['Excess Returns',output.columns[i]] = (d.loc["{}".format(year_mask[i//2][0]):"{}".format(year_mask[i//2][1]),d.columns[0]]\
                                                            - d.loc["{}".format(year_mask[i//2][0]):"{}".format(year_mask[i//2][1]),d.columns[1]]).std() * np.sqrt(ann_factor)
    return output

year_mask=[("2011","2021")]

output_log = meanvoltable(drisk,year_mask,ann_factor=12,compound=True)
output_log
# 2a Calculate (1) using the subsample 1965-1999.

table4 = pd.DataFrame(columns=['h', 'tilde_mu_hat'])
table4['h'] = [10] # Time frame of predicting out
table4 = table4.set_index('h')

# Calculating Normal distribution (Probability)
# Arguments h = array of years, tilde_mu = average return, tilde_sigma = vol
def shortfall_p(h, tilde_mu, tilde_sigma):
    x = - np.sqrt(h) * tilde_mu / tilde_sigma
    val = scipy.stats.norm.cdf(x)
    return val

tilde_mu = output_log.loc["Excess Returns",('2011-2021','mean')]
tilde_sigma = output_log.loc["Excess Returns",('2011-2021','vol')]

table4['tilde_mu_hat'] = shortfall_p(table4.index, tilde_mu=tilde_mu, tilde_sigma=tilde_sigma)
table4.T<jupyter_output><empty_output><jupyter_text>We are extremely confident that SPY will outperform EFA over the next 10 years. Above, we calculated the probability of shortfall of SPY with respect to EFA. The probability is negligible.  Q2<jupyter_code># Calculate all applicable variance models here. var_models is separate from VaR
# returns explanding, rolling, GARCH, and IGARCH for given data
# arguments: d = data (daily RETURNS), theta = unit root operator for IGARCH, sig = starting vol, expanding = expanding startpoint, rolling = roll window
def var_models(d,theta,sig,expanding = 60, rolling = 60):
    var = (d**2).shift().dropna().expanding(expanding).mean().to_frame("Expanding Window")
    var['Rolling Window'] = (d**2).shift().dropna().rolling(rolling).mean()
    sig2 = sig ** 2
    var[['GARCH','IGARCH']] = None
    GARCH = arch_model(d.dropna(), vol='Garch', p = 1, o = 0, q = 1, dist='Normal',rescale=False)
    Garch_model = GARCH.fit(disp='off')
    print(Garch_model.summary())
    var['GARCH'].iat[0] = sig **2
    var['IGARCH'].iat[0] = sig **2
    for i in range(1,len(d.shift().dropna())):
        avgret = (d.shift().dropna().iloc[i]**2)
        sig2 = sig2 * (theta) + (avgret) * (1 - theta)
        var['IGARCH'].iloc[i] = sig2
        var['GARCH'].iloc[i] = Garch_model.params['omega'] + Garch_model.params['beta[1]'] * var['GARCH'].iloc[i-1] + Garch_model.params['alpha[1]'] * (d.shift().dropna().iloc[i] **2)
    return var

var = var_models(df_merrill_facs['EFA US Equity'],0.97,0.15)
var
z_thresh = scipy.stats.norm().ppf(.01) 
VaRSep = z_thresh * np.sqrt(.001755)
VaRSep<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10545129579982127, 'fraction_numerical': 0.01524798927613941, 'mean_word_length': 4.757234726688103, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '169425', 'license': 'MIT', 'n_tokens_mistral': 6111, 'n_tokens_neox': 5718, 'n_words': 1861, 'path': 'solutions/mid1/submissions/frankebergerrichmond_138674_6241586_Midterm 2021_Richmond_Frankeberger.ipynb', 'repo_name': 'tulyu96/finm-portfolio-2021', 'size': '17919'}