<jupyter_start><jupyter_text>Section 2 Mean-Variance Optimization  Data-preprocessing<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
from google.colab import drive
drive.mount('/content/drive')

path_to_data_file = '/content/drive/My Drive/A11/multi_asset_etf_data.xlsx'
df = pd.read_excel(path_to_data_file, sheet_name='excess returns')
df.isnull().values.any() #checking for NaN in dataframe.
df = df.set_index('Date')<jupyter_output><empty_output><jupyter_text>Problem 1: Summary statistics (a)Calculate and display the mean and volatility of each asset’s excess return<jupyter_code>mu_tilde = df.mean() * 12
sigma = df.std() * np.sqrt(12)
table1 = pd.DataFrame({'Mean':mu_tilde, 'Vol':sigma, 'Sharpe':mu_tilde/sigma})

table1<jupyter_output><empty_output><jupyter_text>(b)Which assets have the best and worst Sharpe ratios?Ans: The SPY has the best Sharpe ratio of 1.18; the DBC(Invesco DB Commodity Index Tracking Fund) has the worse Sharpe ratio of 0.06. Problem 2: Descriptive Analysis (a)Calculate the correlation matrix of the returns. Which pair has the highest correlation? And the lowest?<jupyter_code>corr = df.corr()

corr
sns.heatmap(df.corr())<jupyter_output><empty_output><jupyter_text>Ans: PSP(private equity) and EFA(foreign equity) has the highest correlation of 0.908; IEF(domestic bonds) and SPY has the lowest correlation of -0.420. (b)How well have TIPS done in our sample? Have they outperformed domestic bonds? Foreign bonds?Ans: See table below. The TIPS outperformed both the domestic bonds and the foreign bonds, regarding to the annualized excess return and Sharpe ratio.<jupyter_code>table1.loc[['TIP','IEF','BWX']]<jupyter_output><empty_output><jupyter_text>(c)Based on the data, do TIPS seem to expand the investment opportunity set, implying that Harvard should consider them as a separate asset?Ans: Yes. The Harvard Mgmt Company consider three aspects for choosing assets: the expected return, volatility and correlation with other assets. Therefore, the TIPS, which represents the Inflation-IndexediSharesTIPSBond has higher expected returns than other two bonds. Meanwhile, it has highest Sharpe ratio among the three treasury bonds. And according to the correlation heatmap, the TIPS is relatively less correlated with other 10 assets. Below is a measurement we use to give another perspective other than the heatmap regarding to the correlations. We took the absolute value of the mean of TIP's correlations with all other assets and do so for IEF and BWX. We can see that TIP has the lowest value, which is another way to say that TIPS has lower correlations with other asset classes.<jupyter_code>corr_measure_tip = round(corr["TIP"].abs().mean(),2)
corr_measure_ief = round(corr["IEF"].abs().mean(),2)
corr_measure_bwx = round(corr["BWX"].abs().mean(),2)

print({"TIP:":[corr_measure_tip], "IEF:":[corr_measure_ief],"BWX:":[corr_measure_bwx]})<jupyter_output>{'TIP:': [0.34], 'IEF:': [0.39], 'BWX:': [0.52]}<jupyter_text>Problem 3: The MV frontier (a)  Compute and display the weights of the tangency portfolios: w_tan.(b)  Compute the mean, volatility, and Sharpe ratio for the tangency portfolio corresponding to w_tan.<jupyter_code># (a) Compute and display the weights of the tangency portfolio:

def compute_tangency(df_tilde, diagonalize_Sigma=False):

    """Compute tangency portfolio given a set of excess returns.
    """

    Sigma = df_tilde.cov()

    # N is the number of assets

    N = Sigma.shape[0]

    Sigma_adj = Sigma.copy()

    if diagonalize_Sigma:

        Sigma_adj.loc[:,:] = np.diag(np.diag(Sigma_adj))



    mu_tilde = df_tilde.mean()

    Sigma_inv = np.linalg.inv(Sigma_adj)

    weights = Sigma_inv @ mu_tilde / (np.ones(N) @ Sigma_inv @ mu_tilde)

    # For convenience, I'll wrap the solution back into a pandas.Series object.

    omega_tangency = pd.Series(weights, index=mu_tilde.index)

    return omega_tangency, mu_tilde, Sigma

df_tilde_annual = df * 12

omega_tangency, mu_tilde, Sigma = compute_tangency(df_tilde_annual)

omega_tangency
# (b) Compute the mean, volatility, and Sharpe ratio for the tangency...

# Mean

mean = mu_tilde @ omega_tangency

# Volatlity

vol = np.sqrt(omega_tangency @ Sigma @ omega_tangency)/np.sqrt(12)


# Sharpe ratio

sharpe_ratio = mean/vol

print("Mean: ",mean,", vol: ",vol,", sharpe_ratio: ",sharpe_ratio)<jupyter_output>Mean:  0.23776716339549395 , vol:  0.10480197847812693 , sharpe_ratio:  2.2687278126635557<jupyter_text>Problem 4: The allocation (a)  Compute and display the weights of MV portfolios with target returns of ̃μ_p=.01(b)  What is the mean, volatility, and Sharpe ratio for w_p?(c)  Discuss the allocation. In which assets is the portfolio most long? And short?(d)  Does this line up with which assets have the strongest Sharpe ratios?<jupyter_code># (a) Compute and display the weights of MV portfolios with target returns...

def target_mv_portfolio(df_tilde, target_return=0.01, diagonalize_Sigma=False):

    """Compute MV optimal portfolio, given target return and set of excess returns.
    """

    omega_tangency, mu_tilde, Sigma = compute_tangency(df_tilde, diagonalize_Sigma=diagonalize_Sigma)

    Sigma_adj = Sigma.copy()

    if diagonalize_Sigma:

        Sigma_adj.loc[:,:] = np.diag(np.diag(Sigma_adj))

    Sigma_inv = np.linalg.inv(Sigma_adj)

    N = Sigma_adj.shape[0]

    delta_tilde = ((np.ones(N) @ Sigma_inv @ mu_tilde)/(mu_tilde @ Sigma_inv @ mu_tilde)) * target_return

    omega_star = delta_tilde * omega_tangency

    return omega_star



# TODO: Annualized target return in footnote is not up to date

df_tilde_annual = df * 12

omega_star = target_mv_portfolio(df_tilde_annual, target_return=0.01 * 12)

omega_star

# (b) What is the mean, volatility, and Sharpe ratio for w_p?


# Mean

mean = mu_tilde @ omega_star


# Volatlity

vol = np.sqrt(omega_star @ Sigma @ omega_star)/np.sqrt(12)


# Sharpe ratio

sharpe_ratio = mean/vol

print("Mean: ",mean,", vol: ",vol,", sharpe_ratio: ",sharpe_ratio)
# Fraction in risky assets

omega_star.sum()
omega_star<jupyter_output><empty_output><jupyter_text>(c)  Discuss the allocation. In which assets is the portfolio most long? And short?Ans: 50.47% in risky assets and other is invested on risk-free asset. For the risky assets' weights, see the omega_star. The portfolio is most long in IEF: Domestic Bonds iShares 7-10 Year Treasury and most short in QAI: Absolute Return IQ Multi-Strat (d)  Does this line up with which assets have the strongest Sharpe ratios?Ans: Not exactly, see the sorted chart below. Additionally,  when we refer back to the correlation heat map that shows IEF has a remarkably low correlation with other assets and that is why it gets a very high asset allocation in MV portfolio.<jupyter_code>table1['omega_star'] = omega_star
table1.sort_values('Sharpe', ascending=False)
sns.heatmap(df.corr())<jupyter_output><empty_output><jupyter_text>Problem 5: Simple portfolios (a)  Calculate the performance of the equally-weighted portfolio over the sample. Rescale the entire weighting vector to have target mean mu_tilde=.01.   Report its mean, volatility, and Sharpe ratio.<jupyter_code>Sigma = df_tilde_annual.cov()

omega_eql = np.repeat(1/Sigma.shape[0], Sigma.shape[0])

mean = mu_tilde @ omega_eql

rescale_factor = 0.01 * 12 / mean

omega = omega_eql * rescale_factor

vol = np.sqrt(omega @ Sigma @ omega)/np.sqrt(12)

sharpe_ratio = mean / vol

print("Mean: ",mean * rescale_factor,", vol: ",vol,", sharpe_ratio: ",sharpe_ratio)<jupyter_output>Mean:  0.12 , vol:  0.14355613150835045 , sharpe_ratio:  0.5714137805381279<jupyter_text>(b)  Calculate the performance of the “risk-parity” portfolio over the sample.  Risk-parity is a term used in a variety of ways, but here we have in mind setting the weight of the portfolioto be proportional to the inverse of its full-sample volatility estimate.wi=1/σi. This will give the weight vector,w, but you will need to rescale it to have a target mean of ̃μ=.01.<jupyter_code>omega_rp = 1 / (df_tilde_annual.std() / np.sqrt(12))

mean = mu_tilde @ omega_rp

rescale = 0.01 * 12 / mean

omega_rs = omega_rp * rescale
mu_tilde @ omega_rs
Sigma = df_tilde_annual.cov()

mean = mu_tilde @ omega_rs

vol = np.sqrt(omega_rs @ Sigma @ omega_rs)/np.sqrt(12)

sharpe_ratio = mean / vol

print("Mean: ",mean,", vol: ",vol,", sharpe_ratio: ",sharpe_ratio)<jupyter_output>Mean:  0.12000000000000001 , vol:  0.12835250527091432 , sharpe_ratio:  0.9349252649702112<jupyter_text>(c)  How does these compare to the MV portfolio from problem 2.4?Ans: The volatility is higher here compared with problem 2.4. With the same target return, the Sharpe ratio here(0.93) is therefore lower than before(2.27). Problem 6: Out-of-sample performance (a)  Using only data through the end of 2020,  compute w_p for  μ_tilde_p =.01,  allocating to all 11 assets.<jupyter_code># (a) Using only data through the end of 2020, compute...

df_temp = df_tilde_annual.loc[:'2020', :]

omega_tangency, mu_tilde, Sigma = compute_tangency(df_temp)

omega_star = target_mv_portfolio(df_temp, target_return=0.01*12)

omega_star<jupyter_output><empty_output><jupyter_text>(b)  Using those weights, calculate the portfolio’s Sharpe ratio within that sample, through the end of 2020.<jupyter_code># (b) Calculate the portfolio's Sharpe ratio within that sample...

mean = omega_star @ mu_tilde

vol = np.sqrt(omega_star @ Sigma @ omega_star)/np.sqrt(12)

sharpe_ratio_in_sample = mean/vol

sharpe_ratio_in_sample<jupyter_output><empty_output><jupyter_text>(c)  Again  using  those  weights,  (derived  using  data  through  2020,)  calculate  the  portfolio’s Sharpe ratio based on performance in 2021.<jupyter_code>df_oos = df_tilde_annual.loc['2021':,:]

Sigma = df_oos.cov()

mu_tilde_oos = df_oos.mean()

mean = omega_star @ mu_tilde_oos

vol = np.sqrt(omega_star @ Sigma @ omega_star)/np.sqrt(12)

sharpe_ratio_oos = mean / vol

sharpe_ratio_oos<jupyter_output><empty_output><jupyter_text>Ans: The out-of-sample Sharpe ratio is less than that of the in-sample Sharpe. This tells us that using the in-the-sample covariance matrix Sigma and then apply the MV optimization can give us a omega_star that might not be so good in the out-of-sample data. In other words, the solution of the MV is sensitive to the covariance matrix Sigma. Section 3: Extensions 1. Re-do the analysis for total returns. (That is, do not use excess returns.) You will need to drop the column with the risk-free rate, (SHV), to again have our 11 assets.<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
from google.colab import drive
drive.mount('/content/drive')

path_to_data_file = '/content/drive/My Drive/A11/multi_asset_etf_data.xlsx'
df_tt = pd.read_excel(path_to_data_file, sheet_name='total returns')
df_tt = df_tt.drop(columns='SHV')
df_tt = df_tt.set_index('Date')
df_tt
df_tt.isnull().values.any()
df_annual = df_tt * 12

# mean

mean = df_annual.mean()

# volatility

sigma = df_annual.std() / np.sqrt(12)

# Sharpe ratio

sharpe_ratio = mean / sigma

table2 = pd.DataFrame({'Mean':mean, 'Vol':sigma, 'Sharpe':sharpe_ratio})

table2.sort_values('Sharpe', ascending=False) # analysis results of the total return
table1.sort_values('Sharpe', ascending=False) # analysis results of the excess return<jupyter_output><empty_output><jupyter_text>Ans: Compared with the analysis of the excess return, the Sharpe ratios in the analysis of the total returns are different. And the rankings are also slightly different. 2. Robustness (a)Reset to the unadjusted data, and 11-asset problem. This time, make one change: in building w_tan, do not use Σ as given in the formulas in the lecture. Rather, use a diaganalized ΣD, which zeroes out all non-diagonal elements of the full covariance matrix, Σ. How does the allocation look now?<jupyter_code>omega_tangency_, mu_tilde_, Sigma_ = compute_tangency(df_tilde_annual,diagonalize_Sigma=True)

omega_tangency_
omega_tangency, mu_tilde, Sigma = compute_tangency(df_tilde_annual)

omega_tangency<jupyter_output><empty_output><jupyter_text>Ans: If we change the covariance matrix, the allocation looks totally different. This means that the MV optimization results are sensitive to the Sigma, and the results are not so robust on Sigma. (b) What does this suggest about the sensitivity of the solution to estimated means and estimated covariances? Ans: Extremely sensitive to the estimated covariance. (c) HMC deals with this sensitivity by using explicit constraints on the allocation vector. Conceptually, what are the pros/cons of doing that versus modifying the formula with ΣD? Ans: Pros: It's not that reasonable to set modify the cov matrix to be null for all the off-diagnal elements, since we know the assets are correlated with each other. Making them non-correlated with each other doesn't make sense. Also, using explicit constraints on the allocation vector avoids extreme long or short on some assets which are highly correlated.Con:? (d)Re-do the previous problem on Out-of-Sample Performance, but this time using the diagonalized covariance matrix for estimating the weight vector in 2009-2020, and assessing it in 2021.<jupyter_code># (a) Using only data through the end of 2020, compute...

df_temp = df_tilde_annual.loc[:'2020', :]

omega_tangency_diag_, mu_tilde_diag_, Sigma_diag_ = compute_tangency(df_temp, diagonalize_Sigma=True)

omega_star_diag_ = target_mv_portfolio(df_temp, target_return=0.01*12, diagonalize_Sigma=True)

omega_star_diag_
# (b) Calculate the portfolio's Sharpe ratio within that sample...

mean = omega_star_diag_ @ mu_tilde_diag_

vol = np.sqrt(omega_star_diag_ @ Sigma_diag_ @ omega_star_diag_)/np.sqrt(12)

sharpe_ratio_in_sample_diag = mean/vol

sharpe_ratio_in_sample_diag
# (c) Calculate the portfolio's Sharpe ratio out-of-sample...
df_oos = df_tilde_annual.loc['2021':,:]

Sigma_oos_diag_ = df_oos.cov()

mu_tilde_oos = df_oos.mean()

mean = omega_star_diag_ @ mu_tilde_oos

vol = np.sqrt(omega_star_diag_ @ Sigma_oos_diag_ @ omega_star_diag_)/np.sqrt(12)

sharpe_ratio_oos_diag_ = mean / vol

sharpe_ratio_oos_diag_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0899950435459888, 'fraction_numerical': 0.024215818168944275, 'mean_word_length': 5.274544646823634, 'pattern_counts': {'":': 3, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '600200', 'license': 'MIT', 'n_tokens_mistral': 5011, 'n_tokens_neox': 4548, 'n_words': 1539, 'path': 'solutions/hw1/hw1_A11.ipynb', 'repo_name': 'tulyu96/finm-portfolio-2021', 'size': '14123'}