<jupyter_start><jupyter_text>A Whale off the Port(folio) --- In this assignment, you'll get to use what you've learned this week to evaluate the performance among various algorithmic, hedge, and mutual fund portfolios and compare them against the S&P 500 Index.<jupyter_code># Initial imports
import pandas as pd
import numpy as np
import datetime as dt
from pathlib import Path
%matplotlib inline
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Data CleaningIn this section, you will need to read the CSV files into DataFrames and perform any necessary data cleaning steps. After cleaning, combine all DataFrames into a single DataFrame.Files:* `whale_returns.csv`: Contains returns of some famous "whale" investors' portfolios.* `algo_returns.csv`: Contains returns from the in-house trading algorithms from Harold's company.* `sp500_history.csv`: Contains historical closing prices of the S&P 500 Index.  Whale ReturnsRead the Whale Portfolio daily returns and clean the data<jupyter_code># Reading whale returns
print(f"{Path.cwd()}")
whale_filepath=("./Resources/whale_returns.csv")
#Reading csv file into dataframe
whale = pd.read_csv(whale_filepath, index_col='Date', parse_dates=True, infer_datetime_format=True)

whale.head()
# Count nulls
whale.isnull().sum()
# Drop nulls
whale.dropna(inplace=True)<jupyter_output><empty_output><jupyter_text>Algorithmic Daily ReturnsRead the algorithmic daily returns and clean the data<jupyter_code># Reading algorithmic returns
algo_filepath=("./Resources/algo_returns.csv")
#Reading csv file into dataframe
algo=pd.read_csv(algo_filepath, index_col='Date', parse_dates=True, infer_datetime_format=True)
algo.head()
# Count nulls
algo.isnull().sum()
# Drop nulls
algo.dropna(inplace=True)
algo.head()<jupyter_output><empty_output><jupyter_text>S&P 500 ReturnsRead the S&P 500 historic closing prices and create a new daily returns DataFrame from the data.<jupyter_code># Reading S&P 500 Closing Prices
sp5500_filepath=("./Resources/sp500_history.csv")
sp500=pd.read_csv(sp5500_filepath, index_col='Date', parse_dates=True, infer_datetime_format=True)
# data cleaning; remove $ sign
clean_close=[]
for x in sp500["Close"][:]:
    x=str.replace(x,"$","")
    clean_close.append(x)
sp500["Close"] =clean_close
 
# Check Data Types
sp500.dtypes
# Fix Data Types
#Convert str to float
sp500["Close"]= sp500["Close"].astype('float')
# Calculate Daily Returns
#Cacluate daily return method1:
#sp500_dailyreturn = (sp500["Close"]-sp500["Close"].shift(1))/sp500["Close"].shift(1)
#sp500_dailyreturn

#Cacluate daily return method2:
sp500_dailyreturn = sp500.pct_change()
sp500_dailyreturn.head()
# Drop nulls
sp500_dailyreturn.dropna(inplace=True)
sp500_dailyreturn
# Rename `Close` Column to be specific to this portfolio.
sp500_dailyreturn = sp500_dailyreturn.rename(columns = {"Date":"Date","Close":"SP500_Daily_Return"} )
sp500_dailyreturn.head(20)<jupyter_output><empty_output><jupyter_text>Combine Whale, Algorithmic, and S&P 500 Returns<jupyter_code># Join Whale Returns, Algorithmic Returns, and the S&P 500 Returns into a single DataFrame with columns for each portfolio's returns.

# Concat returns into one DataFrame
all_portfolios_returns = pd.concat([whale, algo,sp500_dailyreturn], axis='columns', join='inner')
all_portfolios_returns.head()<jupyter_output><empty_output><jupyter_text>---  Conduct Quantitative AnalysisIn this section, you will calculate and visualize performance and risk metrics for the portfolios.  Performance Anlysis Calculate and Plot the daily returns.<jupyter_code># Plot daily returns of all portfolios

whale_plot= np.random.normal(scale=1.5, size=10000)
algo_plot = np.random.normal(scale=1.5, size=10000)
sp500_dailyreturn_plot = np.random.normal(scale=1.5, size=10000)

all_portfolio = pd.DataFrame({
    "whale": whale_plot,
    "algo": algo_plot,
    "sp500_dailyreturn": sp500_dailyreturn_plot
})

all_portfolio.plot(figsize=(10,8))<jupyter_output><empty_output><jupyter_text>Calculate and Plot cumulative returns.<jupyter_code># Calculate cumulative returns of all portfolios
# Calculate the cumulative returns using the 'cumprod()' function
cumulative_returns = (1 + all_portfolios_returns).cumprod()
cumulative_returns.head()

# Plot cumulative returns
cumulative_returns.plot(figsize=(10,8))<jupyter_output><empty_output><jupyter_text>---  Risk AnalysisDetermine the _risk_ of each portfolio:1. Create a box plot for each portfolio. 2. Calculate the standard deviation for all portfolios4. Determine which portfolios are riskier than the S&P 5005. Calculate the Annualized Standard Deviation  Create a box plot for each portfolio<jupyter_code># Box plot to visually show risk
all_portfolios_returns.plot.box(figsize=(20,10))<jupyter_output><empty_output><jupyter_text>Calculate Standard Deviations<jupyter_code># Calculate the daily standard deviations of all portfolios
all_portfolios_returns.dropna(inplace=True)
all_portfolio_std = all_portfolios_returns.std()
all_portfolio_std<jupyter_output><empty_output><jupyter_text>Determine which portfolios are riskier than the S&P 500<jupyter_code># Calculate  the daily standard deviation of S&P 500
SP500_Daily_Return_std = sp500_dailyreturn.std()
SP500_Daily_Return_std
# Determine which portfolios are riskier than the S&P 500
# Identify the stock with the most risk
all_portfolio_std = all_portfolio_std.sort_values(ascending=False)
all_portfolio_std[:2]<jupyter_output><empty_output><jupyter_text>Calculate the Annualized Standard Deviation<jupyter_code># Calculate the annualized standard deviation (252 trading days)
all_portfolio_annual_std = all_portfolio_std * np.sqrt(252)
all_portfolio_annual_std<jupyter_output><empty_output><jupyter_text>---  Rolling StatisticsRisk changes over time. Analyze the rolling statistics for Risk and Beta. 1. Calculate and plot the rolling standard deviation for all portfolios using a 21-day window2. Calculate the correlation between each stock to determine which portfolios may mimick the S&P 5003. Choose one portfolio, then calculate and plot the 60-day rolling beta between it and the S&P 500  Calculate and plot rolling `std` for all portfolios with 21-day window<jupyter_code># Calculate the rolling standard deviation for all portfolios using a 21-day window
rolling_21 = all_portfolios_returns.rolling(window=21).std()
rolling_21
# Plot the rolling standard deviation
rolling_21.plot(figsize=(20,8))<jupyter_output><empty_output><jupyter_text>Calculate and plot the correlation<jupyter_code># Calculate the correlation
correlation=all_portfolios_returns.corr()
correlation
# Display de correlation matrix
#use heatmap to display correlation
sns.heatmap(correlation, vmin=-1, vmax=1)<jupyter_output><empty_output><jupyter_text>Calculate and Plot Beta for a chosen portfolio and the S&P 500<jupyter_code># Calculate covariance of a single portfolio
BK_cov= all_portfolios_returns['BERKSHIRE HATHAWAY INC'].cov(all_portfolios_returns['SP500_Daily_Return'])
BK_cov

# Calculate variance of S&P 500
SP_var = all_portfolios_returns['SP500_Daily_Return'].var()
SP_var
# Computing beta
BK_beta= BK_cov/SP_var
BK_beta

# Plot beta trend
#rolling beta trend
rolling_BK_cov = all_portfolios_returns['BERKSHIRE HATHAWAY INC'].rolling(window=30).cov(all_portfolios_returns['SP500_Daily_Return'])
rolling_SP_var= all_portfolios_returns['SP500_Daily_Return'].rolling(window=30).var()
rolling_BK_beta = rolling_BK_cov/rolling_SP_var
rolling_BK_beta.plot(figsize=(20,7)).legend(["BERKSHIRE HATHAWAY INC Rolling Beta Trend"])<jupyter_output><empty_output><jupyter_text>Rolling Statistics Challenge: Exponentially Weighted Average An alternative way to calculate a rolling window is to take the exponentially weighted moving average. This is like a moving window average, but it assigns greater importance to more recent observations. Try calculating the [`ewm`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.ewm.html) with a 21-day half-life.<jupyter_code># Use `ewm` to calculate the rolling window
BK_ewm = rolling_BK_beta.ewm(halflife='21 days', times = pd.DatetimeIndex(rolling_BK_beta)).mean()
BK_ewm.plot()<jupyter_output><empty_output><jupyter_text>---  Sharpe RatiosIn reality, investment managers and thier institutional investors look at the ratio of return-to-risk, and not just returns alone. After all, if you could invest in one of two portfolios, and each offered the same 10% return, yet one offered lower risk, you'd take that one, right? Using the daily returns, calculate and visualize the Sharpe ratios using a bar plot<jupyter_code># Annualized Sharpe Ratios

sharpe_ratios = ((all_portfolios_returns.mean()) * 252) / (all_portfolios_returns.std() * np.sqrt(252))
sharpe_ratios
# Visualize the sharpe ratios as a bar plot
sharpe_ratios.plot(kind='bar', title ='Annualized Sharp Ratio', figsize=(10,7))<jupyter_output><empty_output><jupyter_text>Determine whether the algorithmic strategies outperform both the market (S&P 500) and the whales portfolios.Write your answer here! Algorithmic strategy Algo 1 has the highest sharp ratio, which indicates has the best return, and outperformed both S&P 500 and whales portfolios.Algorithmic strategy Algo 2 has the third highest sharp ratio, slighlty less than Whale portofolio's 'BERKSHIRE HATHAWAY INC'. However 'BERKSHIRE HATHAWAY INC' has the highest standard deviation, risker than S&P500, in which case indicates that overall, Algo is better than 'BERKSHIRE HATHAWAY INC' and S&P500.Overall, yes the algorithmic strategies Algo1 and 2 outperform both the market and the whales portofolios.  Create Custom PortfolioIn this section, you will build your own portfolio of stocks, calculate the returns, and compare the results to the Whale Portfolios and the S&P 500. 1. Choose 3-5 custom stocks with at last 1 year's worth of historic prices and create a DataFrame of the closing prices and dates for each stock.2. Calculate the weighted returns for the portfolio assuming an equal number of shares for each stock3. Join your portfolio returns to the DataFrame that contains all of the portfolio returns4. Re-run the performance and risk analysis with your portfolio to see how it compares to the others5. Include correlation analysis to determine which stocks (if any) are correlated  Choose 3-5 custom stocks with at last 1 year's worth of historic prices and create a DataFrame of the closing prices and dates for each stock.For this demo solution, we fetch data from three companies listes in the S&P 500 index.* `GOOG` - [Google, LLC](https://en.wikipedia.org/wiki/Google)* `AAPL` - [Apple Inc.](https://en.wikipedia.org/wiki/Apple_Inc.)* `COST` - [Costco Wholesale Corporation](https://en.wikipedia.org/wiki/Costco)<jupyter_code># Reading data from 1st stock
print(f"current directory:{Path.cwd()}")
google_path=Path("./Resources/goog_historical.csv")
google_df = pd.read_csv(google_path,index_col="Trade DATE", parse_dates=True, infer_datetime_format = True)
google_df

# Reading data from 2nd stock
apple_path=Path("./Resources/aapl_historical.csv")
apple_df = pd.read_csv(apple_path,index_col="Trade DATE", parse_dates=True, infer_datetime_format = True)
apple_df
# Reading data from 3rd stock
cost_path=Path("./Resources/cost_historical.csv")
cost_df = pd.read_csv(cost_path,index_col="Trade DATE", parse_dates=True, infer_datetime_format = True)
cost_df
# Combine all stocks in a single DataFrame
combined_df = pd.concat([google_df,apple_df,cost_df],axis="columns",join="inner")

# Reset Date index
combined_df.sort_index(inplace=True)
combined_df
# Reorganize portfolio data by having a column per symbol
# Rename 'NOCP' column name with ticker
combined_df.columns =['GOOG_symbol','GOOG','AAPL_symbol','AAPL','COST_symbol','COST']
combined_df.drop(columns=['GOOG_symbol','AAPL_symbol','COST_symbol'],inplace=True)
combined_df.head()
# Calculate daily returns
daily_return=combined_df.pct_change()
# Drop NAs
daily_return.dropna(inplace=True)
# Display sample data
daily_return.head()<jupyter_output><empty_output><jupyter_text>Calculate the weighted returns for the portfolio assuming an equal number of shares for each stock<jupyter_code># Set weights
weights = [1/3, 1/3, 1/3]

# Calculate portfolio return
weighted_return = daily_return.dot(weights)

# Display sample data
weighted_return.head()<jupyter_output><empty_output><jupyter_text>Join your portfolio returns to the DataFrame that contains all of the portfolio returns<jupyter_code># Join your returns DataFrame to the original returns DataFrame
All_portfolio = pd.concat([all_portfolios_returns,daily_return],axis="columns", join="inner")
All_portfolio.head()
# Only compare dates where return data exists for all the stocks (drop NaNs)
All_portfolio.dropna(inplace=True)<jupyter_output><empty_output><jupyter_text>Re-run the risk analysis with your portfolio to see how it compares to the others  Calculate the Annualized Standard Deviation<jupyter_code># Calculate the annualized `std`
All_portfolio_std = All_portfolio.std() * np.sqrt(252)
All_portfolio_std<jupyter_output><empty_output><jupyter_text>Calculate and plot rolling `std` with 21-day window<jupyter_code># Calculate rolling standard deviation
All_portfolio_rolling = All_portfolio.rolling(window=21).std()

# Plot rolling standard deviation
All_portfolio_rolling.plot(figsize=(10,7))<jupyter_output><empty_output><jupyter_text>Calculate and plot the correlation<jupyter_code># Calculate and plot the correlation
All_portfolio_corr=All_portfolio.corr()
sns.heatmap(All_portfolio_corr,vmin=-1,vmax=1)<jupyter_output><empty_output><jupyter_text>Calculate and Plot Rolling 60-day Beta for Your Portfolio compared to the S&P 500<jupyter_code># Calculate and plot Beta
    # Calculate covariance of a single portfolio
GOOG_rolling_cov= All_portfolio['GOOG'].rolling(window=60).cov(All_portfolio['SP500_Daily_Return'])
AAPL_rolling_cov=All_portfolio['AAPL'].rolling(window=60).cov(All_portfolio['SP500_Daily_Return'])
COST_rolling_cov=All_portfolio['COST'].rolling(window=60).cov(All_portfolio['SP500_Daily_Return'])

# Calculate variance of S&P 500
SP_rolling_var = All_portfolio['SP500_Daily_Return'].rolling(window=60).var()
SP_rolling_var
#Computing rolling beta trend
GOOG_rolling_beta = GOOG_rolling_cov/SP_rolling_var 
AAPL_rolling_beta = AAPL_rolling_cov/SP_rolling_var 
COST_rolling_beta = COST_rolling_cov/SP_rolling_var 

#Plot three lines in one diagram
ax=GOOG_rolling_beta.plot(figsize=(20,7),title="Rolling 60-day Beta compared to S&P 500")
#add the other two lines into the same diagram 'ax'
AAPL_rolling_beta.plot(ax=ax)
COST_rolling_beta.plot(ax=ax)
ax.legend(["Google", "Apple","Costco"])<jupyter_output><empty_output><jupyter_text>Using the daily returns, calculate and visualize the Sharpe ratios using a bar plot<jupyter_code># Calculate Annualzied Sharpe Ratios
# Annualized Sharpe Ratios
All_cus_sharpe_ratios = ((All_portfolio.mean()) * 252) / (All_portfolio.std() * np.sqrt(252))
All_cus_sharpe_ratios

# Visualize the sharpe ratios as a bar plot
All_cus_sharpe_ratios.plot(kind='bar', figsize=(10,7), title='Sharp Ratio')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08936142024377318, 'fraction_numerical': 0.022919978802331743, 'mean_word_length': 7.368625277161862, 'pattern_counts': {'":': 5, '<': 97, '<?xml version=': 0, '>': 97, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '563508', 'license': 'ADSL', 'n_tokens_mistral': 5120, 'n_tokens_neox': 4690, 'n_words': 1397, 'path': 'whale_analysis.ipynb', 'repo_name': 'linshuishui/Pandas_hw_03', 'size': '15096'}