<jupyter_start><jupyter_text>1. Short Answer1. True, the efficient half of the MV frontline has the highest possible sharpe ratio nad the inefficient half have the lowest2. True, LETF is typically more stable than single stock, therefore there is less reasons to constantly buy and sell based on short-term current situation3. With intercept, because sample is small, mean returns might be estimated inaccurately so we need to include alpha to elimate means and focus on explaining variation4. It's pretty good at tracking in-sample but not so much for out-of-sample because they use a lot of factors and therefore is at risk of over-fitting5. A high alpha can either be from a good portfolio performance or from a bad model, in this case it looks like it's due to bad model<jupyter_code>import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
import statsmodels.api as sm
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.stats<jupyter_output><empty_output><jupyter_text>2. Allocation<jupyter_code>import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
import statsmodels.api as sm
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.stats

df = pd.read_excel('proshares_analysis_data.xlsx',
                   sheet_name = 'merrill_factors').set_index('date')

df['rf'] = df['USGG3M Index']
rf_mean_annualized = df['rf'].mean() * 12
risky_assets = df.columns[0].join(df.columns[2:5])

df_tilde = df.loc[:,risky_assets]
df_tilde = df_tilde.subtract(df['rf'],axis = 0)
df_tilde_annual = 12 * df_tilde
mu_tilde = df_tilde_annual.mean()
sigma = df_tilde_annual.std()/np.sqrt(12)
# Code used for grading to evaluate the input
df = pd.read_excel('../proshares_analysis_data.xlsx',
                   sheet_name = 'merrill_factors').set_index('date')

df['rf'] = df['USGG3M Index']
rf_mean_annualized = df['rf'].mean() * 12
#risky_assets = df.columns[0].join(df.columns[2:5])
risky_assets = df.columns.tolist()
risky_assets.remove('USGG3M Index')
risky_assets.remove('rf')
risky_assets
df_tilde = df.loc[:,risky_assets]
df_tilde = df_tilde.subtract(df['rf'],axis = 0)
df_tilde_annual = 12 * df_tilde
mu_tilde = df_tilde_annual.mean()
sigma = df_tilde_annual.std()/np.sqrt(12)
## 1. What are the weights of the tangency portfolio?
def compute_tangency(df_tilde, diagonalize_Sigma=False):
    Sigma = df_tilde.cov()
    N = Sigma.shape[0]
    Sigma_adj = Sigma.copy()
    if diagonalize_Sigma:
        Sigma_adj.loc[:,:] = np.diag(np.diag(Sigma_adj))
    mu_tilde = df_tilde.mean()
    Sigma_inv = np.linalg.inv(Sigma_adj)
    weights = Sigma_inv @ mu_tilde / (np.ones(N) @ Sigma_inv @ mu_tilde)
    omega_tangency = pd.Series(weights, index=mu_tilde.index)
    return omega_tangency, mu_tilde, Sigma

omega_tangency, mu_tilde, Sigma = compute_tangency(df_tilde_annual)
omega_tangency
## 2. What are the weights of the optimal portfolio, w∗, with a targeted excess mean return of.02 per month?Is the optimal portfolio, w∗, invested in the risk-free rate?
def target_mv_portfolio(df_tilde, target_return=0.02, diagonalize_Sigma=False):

    omega_tangency, mu_tilde, Sigma = compute_tangency(df_tilde, diagonalize_Sigma=diagonalize_Sigma)

    Sigma_adj = Sigma.copy()

    if diagonalize_Sigma:

        Sigma_adj.loc[:,:] = np.diag(np.diag(Sigma_adj))

    Sigma_inv = np.linalg.inv(Sigma_adj)

    N = Sigma_adj.shape[0]

    delta_tilde = ((np.ones(N) @ Sigma_inv @ mu_tilde)/(mu_tilde @ Sigma_inv @ mu_tilde)) * target_return

    omega_star = delta_tilde * omega_tangency

    return omega_star, mu_tilde, Sigma_adj

omega_star, mu_tilde, Sigma = target_mv_portfolio(df_ex)

omega_star_df = omega_star.to_frame('MV Portfolio Weights')

omega_star_df

# the code doesn't run, non-defined variable
# grading section 
omega_star, mu_tilde, Sigma = target_mv_portfolio(df_tilde)

omega_star_df = omega_star.to_frame('MV Portfolio Weights')

omega_star_df
omega_star.sum()<jupyter_output><empty_output><jupyter_text>It is invested in the risk-free rate<jupyter_code>## 3.  Report the mean, volatility, and Sharpe ratio of the optimized portfolio. Annualize all three statistics.
def portfolio_stats(omega, mu_tilde, Sigma, annualize_fac):
    # Mean
    mean = (mu_tilde @ omega) * annualize_fac

    # Volatility
    vol = np.sqrt(omega @ Sigma @ omega) * np.sqrt(annualize_fac)


    # Sharpe ratio
    sharpe_ratio = mean / vol

    return round(pd.DataFrame(data = [mean, vol, sharpe_ratio], 
                              index = ['Mean', 'Volatility', 'Sharpe'], 
                              columns = ['Portfolio Stats']), 4)

tangency_stats(omega_star, mu_tilde, Sigma, 12)

# the code doesn't run, call the wrong function 
# grading section
portfolio_stats(omega_star, mu_tilde, Sigma, 12)
## 3.  Report the mean, volatility, and Sharpe ratio of the optimized portfolio. Annualize all three statistics.
def portfolio_stats(omega, mu_tilde, Sigma, annualize_fac):
    # Mean
    mean = (mu_tilde @ omega) * annualize_fac

    # Volatility
    vol = np.sqrt(omega @ Sigma @ omega) * np.sqrt(annualize_fac)


    # Sharpe ratio
    sharpe_ratio = mean / vol

    return round(pd.DataFrame(data = [mean, vol, sharpe_ratio], 
                              index = ['Mean', 'Volatility', 'Sharpe'], 
                              columns = ['Portfolio Stats']), 4)

tangency_stats(omega_star, mu_tilde, Sigma, 12)
## 4. Re-calculate the optimal portfolio, w∗ with target excess mean of .02 per month. But this time only use data through 2018 in doing the calculation. Calculate the return in 2019-2021 based on those optimal weights. Report the mean, volatility, and Sharpe ratio of the 2019-2021 performance
df_ex_2018 = df.loc[:'2018']
omega_star, mu_tilde, Sigma = target_mv_portfolio(df_ex_2018)

omega_star_df = omega_star.to_frame('MV Portfolio Weights')

omega_star_df

df_af_2019 = df.loc['2019':]
omega_2019, mu_tilde_2019, Sigma_2019 = compute_tangency(df_ex_2018)
portfolio_stats(omega_2019, mu_tilde_2019, Sigma_2019, 12)
## 4. Re-calculate the optimal portfolio, w∗ with target excess mean of .02 per month. But this time only use data through 2018 in doing the calculation. Calculate the return in 2019-2021 based on those optimal weights. Report the mean, volatility, and Sharpe ratio of the 2019-2021 performance
df_ex_2018 = df.loc[:'2018'] # wrong input 
omega_star, mu_tilde, Sigma = target_mv_portfolio(df_ex_2018)

omega_star_df = omega_star.to_frame('MV Portfolio Weights')

omega_star_df

df_af_2019 = df.loc['2019':]
omega_2019, mu_tilde_2019, Sigma_2019 = compute_tangency(df_ex_2018)
portfolio_stats(omega_2019, mu_tilde_2019, Sigma_2019, 12)<jupyter_output><empty_output><jupyter_text>5. Suppose that instead of optimizing these 5 risky assets, we optimized 5 commodity futures: oil, coffee, cocoa, lumber, cattle, and gold. Do you think the out-of-sample fragility problem would be better or worse than what we have seen optimizing equities? No calculation is needed for this question–we just want a conceptual (though specific) answer.They would be better because commodities tend to have more stable volatility than risky assets, and therefore the model that we get from the samples are more applicable to out-of-sample data  3. Hedging & ReplicationSuppose we want to invest in EEM, but hedge out SPY. Do this by estimating a regression of EEMon SPY.<jupyter_code># 1. What is the optimal hedge ratio over the full sample of data? That is, for every dollar invested in EEM, what would you invest in SPY?

from dataclasses import dataclass

@dataclass
class RegressionsOutput:
    excess_ret_stats: pd.DataFrame
    params: pd.DataFrame
    residuals: pd.DataFrame
    tstats: pd.DataFrame
    other: pd.DataFrame
    df: pd.DataFrame
        
        

def lfm_time_series_regression(df, portfolio_names, factors, annualize_factor=12):
    excess_ret_stats = pd.DataFrame(index=factors, columns=['average', 'std'], dtype=float)
    for factor in factors:
        excess_ret_stats.loc[factor, 'average'] = annualize_factor * df[factor].mean()
        excess_ret_stats.loc[factor, 'std'] = np.sqrt(annualize_factor) * df[factor].std()
        excess_ret_stats.loc[factor, 'sharpe_ratio'] = \
            excess_ret_stats.loc[factor, 'average'] / excess_ret_stats.loc[factor, 'std']
        # Here I'll just report the unscaled skewness
        excess_ret_stats.loc[factor, 'skewness'] = df[factor].skew()
        # excess_ret_stats.loc[factor, 'skewness'] = annualize_factor * df[factor].skew()

    _temp_excess_ret_stats = excess_ret_stats.copy()
    _temp_excess_ret_stats.loc['const', :] = 0

    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        rhs = sm.add_constant(df[factors])
    df_params = pd.DataFrame(columns=portfolio_names)
    df_other = pd.DataFrame(columns=portfolio_names)
    df_residuals = pd.DataFrame(columns=portfolio_names)
    df_tstats = pd.DataFrame(columns=portfolio_names)
    for portfolio in portfolio_names:
        lhs = df[portfolio]
        res = sm.OLS(lhs, rhs, missing='drop').fit()
        df_params[portfolio] = res.params
        df_params.loc['const', portfolio] = annualize_factor * res.params['const']
        df_other.loc['r_squared', portfolio] = res.rsquared
        df_other.loc['model_implied_excess_ret', portfolio] = df_params[portfolio] @ _temp_excess_ret_stats['average']
        df_other.loc['ave_excess_ret', portfolio] = \
            annualize_factor * df[portfolio].mean()
        df_other.loc['std_excess_ret', portfolio] = \
            np.sqrt(annualize_factor) * df[portfolio].std()
        df_other.loc['skewness_excess_ret', portfolio] = \
            annualize_factor * df[portfolio].skew()
        df_other.loc['sharpe_ratio', portfolio] = \
            df_other.loc['ave_excess_ret', portfolio] / df_other.loc['std_excess_ret', portfolio]
        df_residuals[portfolio] = res.resid
        df_tstats[portfolio] = res.tvalues

    regression_outputs = RegressionsOutput(
        excess_ret_stats.T,
        df_params.T,
        df_residuals,
        df_tstats.T,
        df_other.T,
        df)

    return regression_outputs

hedge_rep_portfolio_names = ['EEM US Equity']
hedge_rep_regs = lfm_time_series_regression(df=df_hedge_rep_excess,portfolio_names=hedge_rep_portfolio_names,factors=['SPY US Equity'])



# 2. What is the mean, volatility, and Sharpe ratio of the hedged position, had we applied that hedge throughout the full sample? Annualize the statistics.
hedge_rep_regs = lfm_time_series_regression(
    df=df_hedge_rep_excess,
    portfolio_names=hedge_rep_portfolio_names,
    factors=['SPY US Equity'])

def create_table1(regs_object):
    table1 = pd.DataFrame(regs_object.other[['ave_excess_ret', 'std_excess_ret', 'sharpe_ratio']])
    return table1

table1 = create_table1(hedge_rep_regs)

# 3. Does it have the same mean as EEM? Why or why not?
#     no, because SPY and EEM's correlation is not 0 and therefore by hedging out SPY, we hedge out a part of what influences EEM as well

# 4.  Suppose we estimated a multifactor regression where in addition to SPY, we had IWM as a regressor. Why might this regression be difficult to use for attribution or even hedging?
#     because they are not idd and the ratio might not be able to be linearly estimated<jupyter_output><empty_output><jupyter_text>3. Does it have the same mean as EEM? Why or why not?         no, because SPY and EEM's correlation is not 0 and therefore by hedging out SPY, we hedge out a part of what influences EEM as well4.  Suppose we estimated a multifactor regression where in addition to SPY, we had IWM as a regressor. Why might this regression be difficult to use for attribution or even hedging?        because they are not idd and might have a complicated influence on each other  4. Modeling Risk Continue to use the same data file used in the previous problem. But for this problem use the totalreturns of SPY and EFA. That is, use the returns as given in the spreadsheet–without subtractingUSGG3M Index.1. SPY and EFA are highly correlated, yet SPY has had a much higher return. Howconfident are we that SPY will overperform EFA over the next 10 years?To answer the question,• use statistical estimates of the total returns of SPY and EFA over the full sample.• Assume that log returns for both assets are normally distributed.<jupyter_code>def p(h,mu=0.5,sigma=0.01):
    L = -np.sqrt(h)*mu/sigma
    return scipy.stats.norm.cdf(L)

tilde_r_M = df['SPY US Equity']-df['EFA US Equity']
tilde_r_M_mean = tilde_r_M.mean() * 12
tilde_r_M_std = tilde_r_M.std() * (12**0.5)
p(10,tilde_r_M_mean,tilde_r_M_std)<jupyter_output><empty_output><jupyter_text>2. Calculate the 60-month rolling volatility of EFA.Use the latest estimate of the volatility (Sep 2021), along with the normality formula, to calculatea Sep 2021 estimate of the 1-month, 1% VaR. In using the VaR formula, assume that the meanis zero<jupyter_code>import itertools
from arch import arch_model

def calculate_vol(x):
    return ((x**2).sum()/len(x))**0.5

vol_rolling = df['EFA US Equity'].shift(1).dropna().rolling(60).apply(lambda x: calculate_vol(x)).dropna()
vol = vol_rolling[-1]

historic_var = df['tilde_r_M'].shift(1).expanding(60).quantile(.01).dropna()
plt.plot(historic_var)
plt.ylabel('VaR')
plt.title("Historical VaR")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09275101305718145, 'fraction_numerical': 0.01936064835659613, 'mean_word_length': 4.345768150822303, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '626157', 'license': 'MIT', 'n_tokens_mistral': 4617, 'n_tokens_neox': 4262, 'n_words': 1296, 'path': 'solutions/mid1/submissions/caoholly_170697_6241690_Midterm_HollyCao.ipynb', 'repo_name': 'tulyu96/finm-portfolio-2021', 'size': '13326'}