<jupyter_start><jupyter_text>2 Analyzing the Data<jupyter_code>import numpy as np
import pandas as pd
import scipy.stats as st
import statsmodels.api as sm
import seaborn as sns
from statsmodels.regression.rolling import RollingOLS
import warnings
warnings.filterwarnings('ignore')
df = pd.read_excel("proshares_analysis_data.xlsx",sheet_name="hedge_fund_series",index_col="date")
df.head()<jupyter_output><empty_output><jupyter_text>1. For the series in the “hedge fund series” tab, report the following summary statistics:(a) mean (b) volatility (c) Sharpe ratio Annualize these statistics.<jupyter_code>def summary(df):
    '''
    Given a pandas DataFrame of time series with monthly data, returns annualized mean, volatility and Sharpe Ratio
    '''
    mean = df.mean()*12
    volatility = df.std()*np.sqrt(12)
    out = pd.DataFrame({'mean':mean,'volatility':volatility})
    out['SharpeRatio'] = out['mean']/out['volatility']
    return out
summary(df)<jupyter_output><empty_output><jupyter_text>2. For the series in the “hedge fund series” tab, , calculate the following statistics related to tailrisk. (a) Skewness(b) Excess Kurtosis (in excess of 3)(c) VaR (.05) - the fifth quantile of historic returns(d) CVaR (.05) - the mean of the returns at or below the fifth quantile(e) Maximum drawdown - include the dates of the max/min/recovery within the max drawdown period.There is no need to annualize any of these statistics<jupyter_code>def tailRisks(df):
    '''
    Given a pandas DataFrame of time series, returns Skewness, Excess Kurtosis, 5% VaR & CVar, Maximum Drawdown
    '''
        
    sk = df.skew()
    #kt = st.kurtosis(df,fisher=True)
    kt = df.kurtosis()
    VaR = df.quantile(0.05)
    CVaR = df[df < VaR].mean()

    levels = (1+df).cumprod()
    high = levels.cummax()
    drawdown = (levels - high)/high
    maxDrawdown = drawdown.min()
    
    bottom = pd.Series(pd.to_datetime([drawdown[col].idxmin() for col in drawdown]),index=df.columns)
    peak = pd.Series(pd.to_datetime([(levels[col][:bottom[col]].idxmax()) for col in levels]),index=df.columns)
    
    peakLevels = pd.Series([levels[col].loc[peak.loc[col]] for col in levels],index=df.columns)
    

    recovered = []
    for col in levels:
        for lev in levels[col][bottom[col]:]:
            if lev >= peakLevels[col]:
                recovered.append(levels.index[levels[col] == lev][0])
                break

    recovered = pd.Series(recovered,index=df.columns)
    
    drawdown.plot(figsize=(9,6), title = 'Drawdown')    
    

    return pd.DataFrame({'Skewness':sk,'Excess Kurtosis':kt,'5% VaR':VaR,'5% CVaR':CVaR,'Max Drawdown':maxDrawdown,\
                        'Peak':peak,'Bottom':bottom,'Recovered':recovered})
   
    
tailRisks(df)<jupyter_output><empty_output><jupyter_text>3. For the series in the “hedge fund series” tab, run a regression of each against SPY (found in the “merrill factors” tab.) Include an intercept. Report the following regression-based statistics:(a) Market Beta(b) Treynor Ratio(c) Information ratioAnnualize these three statistics as appropriate.<jupyter_code>merrill_factors = pd.read_excel("proshares_analysis_data.xlsx",sheet_name= "merrill_factors",index_col="date")
spy = merrill_factors['SPY US Equity']
merrill_factors.head()
def performanceMeasures(seriesY,seriesX):
    
    mean =seriesY.mean()*12
    sharpe = mean/(seriesY.std()*(12**0.5))
    model = sm.OLS(seriesY,sm.add_constant(seriesX)).fit()
    rsq = model.rsquared
    beta = model.params[1]
    treynor = mean/beta
    information = model.params[0]/(model.resid.std()*np.sqrt(12))
    
    return pd.DataFrame({'Mean Return':mean,'Sharpe Ratio':sharpe,'R Squared':rsq, 'Beta':beta,\
                         'Treynor Ratio':treynor, 'Information Ratio':information},index= [seriesY.name])
frames = []
for col in df:
    p = performanceMeasures(df[col],spy)
    frames.append(p) 
pd.concat(frames)<jupyter_output><empty_output><jupyter_text>4. Relative PerformanceDiscuss the previous statistics, and what they tell us about... (a) the differences between SPY and the hedge-fund series?The large part of most Hedge Fund series can be explained using SPY as the R Squared from Regrssening Hedge Fund series on SPY high (greater than 70% for each series.  (b) which performs better between HDG and QAI.HDG has higher Mean Returns than QAI but has lower Sharpe Ratio, Treynor Ratio and Information Ratios. Thus higher returns have come with higher volaility in HDG. From a mean variance optimized standpoint, QAI has performed better. (c) whether HDG and the ML series capture the most notable properties of HFRI.We note (from question 5) that both HDG and ML are highly correlated with HFRI with a correlation of about 90%. However, HFRI's excess kurtosis of 6.73 is substantially higher than the excess kurtosis of HDG and ML (about 2.4-2.5). Thus HDG and ML would fail to capture the very high tail-risk of HFRI. Furhter, HDG and ML series are highly correlated (~98% correlation). Regressing HFRI on HDG and ML together would thus suffer from a high multi-collinearity.  5. Report the correlation matrix for these assets.(a) Show the correlations as a heat map.(b) Which series have the highest and lowest correlations?<jupyter_code>correlations = df.corr()
correlations
sns.heatmap(correlations)
def findHighestLowestCorrelations(correlationsMatrix):
    high, low = [],[]
    for col in correlationsMatrix.columns:

        val_high = correlationsMatrix[col][(correlationsMatrix[col] == correlationsMatrix[col].nlargest(2).iloc[1])]
        high.append((val_high.index[0],val_high.name,val_high[0]))

        val_low = correlationsMatrix[col][(correlationsMatrix[col] == min(correlationsMatrix[col]))]
        low.append((val_low.index[0],val_low.name,val_low[0]))

    print("Highest correlation pair = ",max(high,key = lambda x:x[2]))
    print("Lowest correlation pair = ",min(low,key = lambda x:x[2]))
    
findHighestLowestCorrelations(correlations)<jupyter_output>Highest correlation pair =  ('MLEIFCTX Index', 'MLEIFCTR Index', 0.9999391791142366)
Lowest correlation pair =  ('QAI US Equity', 'HDG US Equity', 0.8475965688236916)<jupyter_text>6. Replicate HFRI with the six factors listed on the “merrill factors” tab. Include a constant, and run the unrestricted regression,  \begin{align}  r^{hfri}_{t} = \alpha^{merr} + x^{merr}_{t} \beta^{merr} + \epsilon^{merr}_{t} (1)  \end{align}  \begin{align}  \hat r^{hfri}_{t} = \hat\alpha^{merr} + x^{merr}_{t} \hat \beta^{merr} (2)  \end{align}Note that the second equation is just our notation for the fitted replication.(a) Report the intercept and betas.(b) Are the betas realistic position sizes, or do they require huge long-short positions?(c) Report the R-squared.(d) Report the volatility of $\epsilon^{merr}$, (the tracking error.)<jupyter_code>model = sm.OLS(df['HFRIFWI Index'],sm.add_constant(merrill_factors)).fit()
model.params<jupyter_output><empty_output><jupyter_text>The Betas are realistic position sizes except for USGG3M Index<jupyter_code>print("R-squared of the Model =",round(model.rsquared,6))
print("Volatility of Error term (annualized) =",round(model.resid.std()*(12**0.5),6))<jupyter_output>Volatility of Error term (annualized) = 0.023365<jupyter_text>7. Let’s examine the replication out-of-sample.Starting with t = 61 month of the sample, do the following:• Use the previous 60 months of data to estimate the regression equation, (1). This gives time-t estimates of the regression parameters, $\tilde \alpha^{merr}_{t}$ and $\tilde \beta^{merr}_{t}$• Use the estimated regression parameters, along with the time-t regressor values, $x^{merr}_{t}$, to calculate the time-t replication value that is, with respect to the regression estimate, built\begin{align}\tilde r^{merr}_{t} \equiv \tilde \alpha^{merr}_{t} + (x^{merr}_{t})\prime \tilde \beta^{merr}_{t}\end{align}• Step forward to t = 62, and now use t = 2 through t = 61 for the estimation. Re-run thesteps above, and continue this process throughout the data series. Thus, we are running arolling, 60-month regression for each point-in-time.How well does the out-of-sample replication perform with respect to the target?<jupyter_code>def get_est_coef(Y, X):

    X_add_const = sm.add_constant(X)

    model = sm.OLS(Y, X_add_const)

    res = model.fit()

    return res.params[0], res.params[1:]



fitted_values = []
window_size = 60



for i in range(df.shape[0] - window_size - 1):

    alpha, beta = get_est_coef(df['HFRIFWI Index'][i : i+window_size], merrill_factors[i : i+window_size])

    fitted_values.append((alpha + np.dot(merrill_factors[i+window_size+1 : i+window_size+2], beta).item()))
    
table_7 = pd.DataFrame({'True Values': df['HFRIFWI Index'][61:], 'Fitted Values': fitted_values})

table_7.plot(figsize=(13,8))

table_7.corr()<jupyter_output><empty_output><jupyter_text>We observe that the predicted value is 94.34% correlated with actual performance. Thus out-of-sample replication performs very well. The same can also be visualized in the chart below:  8. We estimated the replications using an intercept. Try the full-sample estimation, but this time without an intercept. \begin{align}  r^{hfri}_{t} = x^{merr}_{t} \beta^{merr} + \epsilon^{merr}_{t}  \end{align}  \begin{align}  \check r^{hfri}_{t} = x^{merr}_{t} \check \beta^{merr}  \end{align}Report(a) the regression beta. How does it compare to the estimated beta with an intercept, $\hat \beta^{merr}$?<jupyter_code>model_without_intercept = sm.OLS(df['HFRIFWI Index'],merrill_factors).fit()
regressionBetas = pd.DataFrame({"With Intercept":model.params,"Without Intercept":model_without_intercept.params})
regressionBetas<jupyter_output><empty_output><jupyter_text>The Regression Betas are mostly similar in both the regressions except for USGG3M Index Beta. In the absence of intercept, there is a much higher weightage of USGG3M which could be thought of as a proxy of Risk Free Interest Rates (b) the mean of the fitted value, $\check r^{hfri}_{t}$. How does it compare to the mean of the HFRI?<jupyter_code>pd.Series([df['HFRIFWI Index'].mean()*12,model_without_intercept.predict(merrill_factors).mean()*12],\
          index=['Mean of HFRI','Mean of Fitted Values'])<jupyter_output><empty_output><jupyter_text>The mean of the Fitted Values is lower than the mean of HFRI (c) the correlations of the fitted values, $\check r^{hfri}_{t}$ to the HFRI. How does the correlation compare to that of the fitted values with an intercept, $\hat r^{hfri}_{t}$?<jupyter_code>HFRIpredictions = pd.DataFrame({'HFRI Index':df['HFRIFWI Index'],\
                                'Predict with intercept':model.predict(sm.add_constant(merrill_factors)),\
                               'Predict without intercept':model_without_intercept.predict(merrill_factors)})
HFRIpredictions.corr()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12021002210759027, 'fraction_numerical': 0.013633014001473839, 'mean_word_length': 4.787313432835821, 'pattern_counts': {'":': 2, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '445531', 'license': 'MIT', 'n_tokens_mistral': 3663, 'n_tokens_neox': 3410, 'n_words': 1043, 'path': 'solutions/hw2/FINM36700_HW2_GroupA11v3.ipynb', 'repo_name': 'tulyu96/finm-portfolio-2021', 'size': '10856'}