<jupyter_start><jupyter_text>Reference: The code in this notebook is modified from the HW2_2020_solution.ipynb notebook on GitHub.  1. Summary Statistics<jupyter_code>df_hedge_fund=pd.read_excel(path,sheet_name='hedge_fund_series').set_index('date')
df_merrill_factors=pd.read_excel(path,sheet_name='merrill_factors').set_index('date')
df_hedge_fund['SPY US Equity']=df_merrill_factors['SPY US Equity']
df_hedge_fund.head()
mean_hf=df_hedge_fund.mean()*12
std_hf=df_hedge_fund.std()*np.sqrt(12)
sharpe_ratio_hf=mean_hf/std_hf
df_stats=pd.DataFrame({'Mean':mean_hf, 'Volatility':std_hf, 'Sharpe Ratio':sharpe_ratio_hf})
df_stats<jupyter_output><empty_output><jupyter_text>2. Summary Tail-Risk Statistics Reference: The calculate_MDD_with_Dates function is modified from https://www.quantstart.com/articles/Event-Driven-Backtesting-with-Python-Part-VII/<jupyter_code>portfolio_names=df_hedge_fund.columns
df_tail_risk_hf = pd.DataFrame(columns=portfolio_names)
def calculate_MDD_with_Dates(data):
    # Calculate the cumulative returns curve 
    # and set up the High Water Mark
    # Then create the drawdown and duration series
    hwm = [0]
    idx = data.index
    drawdown = pd.Series(index = idx)
    duration = pd.Series(index = idx)

    # Loop over the index range
    for t in range(1, len(idx)):
        cur_hwm = max(hwm[t-1], data[t])
        hwm.append(cur_hwm)
        drawdown[t]= hwm[t] - data[t]
        duration[t]= 0 if drawdown[t] == 0 else duration[t-1] + 1
    max_date=drawdown[drawdown==drawdown.max()].index[0]
    
    min_date=max_date+timedelta(days=duration.max())
    
        
    
    return drawdown.max(),max_date,min_date,duration.max()

def calculate_VaR(data, percentile):
    VaR=np.percentile(data,percentile)
    return VaR

def calculate_CVaR(data,VaR):
    CVaR=data[data<=VaR].mean()
    return CVaR

for portfolio in portfolio_names:
    df_tail_risk_hf.loc['Skewness', portfolio] = df_hedge_fund[portfolio].skew()
    df_tail_risk_hf.loc['Excess Kurtosis', portfolio] = df_hedge_fund[portfolio].kurtosis()-3
    VaR_5=  calculate_VaR(df_hedge_fund[portfolio],5)
    df_tail_risk_hf.loc['VaR(.05)', portfolio] = VaR_5
    df_tail_risk_hf.loc['CVaR(.05)', portfolio] = calculate_CVaR(df_hedge_fund[portfolio],VaR_5)
    df_tail_risk_hf.loc['Maximum Drawdown', portfolio] =calculate_MDD_with_Dates(df_hedge_fund[portfolio])[0]
    df_tail_risk_hf.loc['MDD Max Date', portfolio]=calculate_MDD_with_Dates(df_hedge_fund[portfolio])[1]
    df_tail_risk_hf.loc['MDD Min Date', portfolio]=calculate_MDD_with_Dates(df_hedge_fund[portfolio])[2]
    df_tail_risk_hf.loc['MDD Duration', portfolio]=calculate_MDD_with_Dates(df_hedge_fund[portfolio])[3]


df_tail_risk_hf

factor='SPY US Equity'
df_tail_risk_hf.loc['Skewness', factor] = df_merrill_factors[factor].skew()
df_tail_risk_hf.loc['Excess Kurtosis', factor] = df_merrill_factors[factor].kurtosis()-3
VaR_5=  calculate_VaR(df_merrill_factors[factor],5)
df_tail_risk_hf.loc['VaR(.05)', factor] = VaR_5
df_tail_risk_hf.loc['CVaR(.05)', factor] = calculate_CVaR(df_merrill_factors[factor],VaR_5)
df_tail_risk_hf.loc['Maximum Drawdown', factor] =calculate_MDD_with_Dates(df_merrill_factors[factor])[0]
df_tail_risk_hf.loc['MDD Max Date', factor]=calculate_MDD_with_Dates(df_merrill_factors[factor])[1]
df_tail_risk_hf.loc['MDD Min Date', factor]=calculate_MDD_with_Dates(df_merrill_factors[factor])[2]
df_tail_risk_hf.loc['MDD Duration', factor]=calculate_MDD_with_Dates(df_merrill_factors[factor])[3]<jupyter_output><empty_output><jupyter_text>3. Summary Regression Statistics<jupyter_code>#df_merrill_factors=pd.read_excel(path,sheet_name='merrill_factors').set_index('date')
df_hedge_fund_excess=df_hedge_fund.subtract(df_merrill_factors['USGG3M Index'],axis=0)

rhs=sm.add_constant(df_hedge_fund_excess['SPY US Equity'])
df_params=pd.DataFrame(columns=portfolio_names)
for portfolio in portfolio_names:
    lhs=df_hedge_fund_excess[portfolio]
    res=sm.OLS(lhs,rhs,missing='drop').fit()
    
    df_params.loc['const', portfolio]=12*res.params['const']
    df_params.loc['Market Beta', portfolio]=res.params['SPY US Equity']
    df_params.loc['Treynor Ratio', portfolio]=12*df_hedge_fund_excess[portfolio].mean()/res.params['SPY US Equity']
    df_params.loc['Information Ratio', portfolio]=12*res.params['const']/(res.resid.std()*np.sqrt(12))

    

df_params.T<jupyter_output><empty_output><jupyter_text>4. Relative Performance<jupyter_code>df_stats<jupyter_output><empty_output><jupyter_text>(a) SPY vs. Hedge-Fund Series SPY has the largest mean return, volatility, and Sharpe Ratio, and also the highest Treynor Ratio and Information Ratio among all the securities in the hedge fund series.  (b) HDG vs. QAI HDG has higher mean, higher volatility, and lower Sharpe Ratio than QAI, so it only outperforms QAI in mean return, but QAI outperforms HDG in Sharpe Ratio.  (c) HDG, ML, and HFRI HFRI has a much higher mean return and Sharpe Ratio than the ML indexes and HDG, and HFRI also has a slightly higher volatility than the ML series and HDG. Overall HDG and the ML seriies capture the high mean, high volatility features of HFRI, and HDG and the ML series also have similar beta that are slightly lower than HFRI.  5. Correlation Matrix  (a) Heatmap<jupyter_code>corrmat=df_hedge_fund_excess.corr()
# ignore self-corr
corrmat[corrmat==1] = None
sns.heatmap(corrmat,annot=True)<jupyter_output><empty_output><jupyter_text>(b) Highest and Lowest Correlations<jupyter_code>corr_rank = corrmat.unstack().sort_values().dropna()
pair_max = corr_rank.index[-1]
pair_min = corr_rank.index[0]

print(f'MIN Correlation pair is {pair_min}')
print(f'MAX Correlation pair is {pair_max}')<jupyter_output>MIN Correlation pair is ('HDG US Equity', 'QAI US Equity')
MAX Correlation pair is ('MLEIFCTR Index', 'MLEIFCTX Index')<jupyter_text>6. Replicate HFRI<jupyter_code>merrill_factors=df_merrill_factors.columns
df_merrill_factors['HFRIFWI Index']=df_hedge_fund['HFRIFWI Index']
portfolio_names=['HFRIFWI Index']
rhs=sm.add_constant(df_merrill_factors[merrill_factors])
df_params=pd.DataFrame(columns=portfolio_names)
df_other=pd.DataFrame(columns=portfolio_names)
df_fitted=pd.DataFrame(columns=portfolio_names)
df_residuals=pd.DataFrame(columns=portfolio_names)
for portfolio in portfolio_names:
    lhs=df_merrill_factors[portfolio]
    res=sm.OLS(lhs,rhs,missing='drop').fit()
    df_params[portfolio]=res.params
    df_params.loc['const', portfolio]=12*res.params['const']
    df_params.loc['Mean Fitted Values', portfolio]=np.mean(res.fittedvalues)
    df_fitted[portfolio]=res.fittedvalues
    df_other.loc['R-Squared',portfolio]=res.rsquared
    df_residuals[portfolio]=res.resid
    
df_fitted_with_intercept=df_fitted<jupyter_output><empty_output><jupyter_text>(a) Intercept and Betas<jupyter_code>df_params_with_intercept=df_params.T
df_params_with_intercept<jupyter_output><empty_output><jupyter_text>(b) Betas The betas require a relatively huge short position in USGG3M index, but the sizes are realistic to use in replicating portfolios.   (c) R-squared<jupyter_code>print('\nR-squared = {:.5f}'.format(np.array(df_other.T['R-Squared'])[0]))<jupyter_output>R-squared = 0.85569<jupyter_text>(d) Tracing Error<jupyter_code>print('Tracking error = {:.5f}'.format(np.array(df_residuals.std() * np.sqrt(12))[0]))<jupyter_output>Tracking error = 0.02337<jupyter_text>7. Replication Out-of-sample<jupyter_code>def oos_reg(df,factors,t):
    date_range =df.index[t:]
    oos_fitted = pd.Series(index=date_range, name='OOS_fit')
    for date in date_range:
        date_month_prior = pd.DatetimeIndex([date]).shift(periods=-1, freq='M')[0]
        df_subset = df[:date_month_prior]
        rhs = sm.add_constant(df_subset[factors])
        lhs = df_subset['HFRIFWI Index']
        res = sm.OLS(lhs, rhs, drop="missing").fit()
        alpha = res.params['const']
        beta = res.params.drop(index='const')
        x_t = df.loc[date, factors]
        predicted_next_value = alpha + x_t @ beta
        oos_fitted[date] = predicted_next_value
    output=(pd.DataFrame([oos_fitted, df_merrill_factors.loc[df_merrill_factors.index[61]:,'HFRIFWI Index']])).T.corr()
    '''
    oos_fitted[df.index[t]:].plot(figsize=(14,3))
    df.loc[df.index[t]:,'HFRIFWI Index'].plot()
    plt.legend()
    plt.show()
    None

    (oos_fitted[df.index[t]:] + 1).cumprod().plot(figsize=(14,3))
    (df.loc[df.index[t]:,'HFRIFWI Index'] + 1).cumprod().plot()
    plt.legend()
    plt.show()
    None
    '''
    return output
oos_reg(df_merrill_factors,merrill_factors,61)
oos_reg(df_merrill_factors,merrill_factors,62)

months=len(df_merrill_factors)
pd.set_option('display.max_rows', None)
df_list=[]
for m in range(61,months):
    df=oos_reg(df_merrill_factors,merrill_factors,m)
    
    df_list.append(df)
pd.concat(df_list)<jupyter_output><empty_output><jupyter_text>While we may worry that the strong replication results were dependent on being in-sample, for t=61 and t=62, the OOS results show almost the same level of replicability - roughly  93%  correlation between the replicating portfolio and the HFRI. The rolling 60-month regression result also shows high correlation for most of the time values, but the correlation decreases to 70%-80% for the last 10 time values.  8. Replication without an Intercept<jupyter_code>df_merrill_factors=pd.read_excel(path,sheet_name='merrill_factors').set_index('date')
merrill_factors=df_merrill_factors.columns
df_merrill_factors['HFRIFWI Index']=df_hedge_fund['HFRIFWI Index']
portfolio_names=['HFRIFWI Index']
rhs=df_merrill_factors[merrill_factors]
df_params=pd.DataFrame(columns=portfolio_names)
df_other=pd.DataFrame(columns=portfolio_names)
df_fitted=pd.DataFrame(columns=portfolio_names)
df_residuals=pd.DataFrame(columns=portfolio_names)
for portfolio in portfolio_names:
    lhs=df_merrill_factors[portfolio]
    res=sm.OLS(lhs,rhs,missing='drop').fit()
    df_params[portfolio]=res.params
    df_params.loc['Mean Fitted Values', portfolio]=np.mean(res.fittedvalues)
    df_fitted[portfolio]=res.fittedvalues
    df_other.loc['R-Squared',portfolio]=res.rsquared
    df_residuals[portfolio]=res.resid<jupyter_output><empty_output><jupyter_text>(a) (b) Regression Beta and Mean Fitted Values<jupyter_code>df_params_no_intercept=df_params.T
df_params_no_intercept
df_compare_reg=pd.concat([df_params_with_intercept,df_params_no_intercept],sort=True)
df_compare_reg<jupyter_output><empty_output><jupyter_text>The replication without intercept has a lower mean fitted values and smaller beta for all US Equity series except SPY, and USGG2M has a large positive beta in the no-intercept replication, while the beta for USGG2M with intercept is a large negative number. Also, both of the regressions have mean of fitted values for HFRIFWI smaller than the mean of HFRIFWI (0.0508), and the replication with intercept has a closer value.   (c) Correlations of Fitted value to the HFRI<jupyter_code>df_fitted_no_intercept=df_fitted

df_fitted_no_intercept['HFRIFWI Index'].corr(df_hedge_fund['HFRIFWI Index'])
df_fitted_with_intercept['HFRIFWI Index'].corr(df_hedge_fund['HFRIFWI Index'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11841278391237993, 'fraction_numerical': 0.011132058533081964, 'mean_word_length': 6.7630662020905925, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 54, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '504031', 'license': 'MIT', 'n_tokens_mistral': 4319, 'n_tokens_neox': 4078, 'n_words': 717, 'path': 'solutions/hw2/HW2-1.ipynb', 'repo_name': 'tulyu96/finm-portfolio-2021', 'size': '11139'}