<jupyter_start><jupyter_text>PCA Basics - Coding ExercisesIn this notebook you will apply what you learned about PCA in the previous lesson to real stock data.  Install Packages<jupyter_code>import sys
!{sys.executable} -m pip install -r requirements.txt
!{sys.executable} -m pip install SQLAlchemy --upgrade<jupyter_output>Requirement already satisfied: zipline===1.3.0 in /opt/conda/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (1.3.0)
Requirement already satisfied: scipy>=0.17.1 in /opt/conda/lib/python3.6/site-packages (from zipline===1.3.0->-r requirements.txt (line 1)) (1.2.1)
Requirement already satisfied: numpy>=1.11.1 in /opt/conda/lib/python3.6/site-packages (from zipline===1.3.0->-r requirements.txt (line 1)) (1.12.1)
Requirement already satisfied: cyordereddict>=0.2.2 in /opt/conda/lib/python3.6/site-packages (from zipline===1.3.0->-r requirements.txt (line 1)) (1.0.0)
Requirement already satisfied: sortedcontainers>=1.4.4 in /opt/conda/lib/python3.6/site-packages (from zipline===1.3.0->-r requirements.txt (line 1)) (2.4.0)
Requirement already satisfied: networkx<2.0,>=1.9.1 in /opt/conda/lib/python3.6/site-packages (from zipline===1.3.0->-r requirements.txt (line 1)) (1.11)
Requirement already satisfied: tables>=3.3.0 in /opt/conda/lib/python3.6/site-packages (from zipline===[...]<jupyter_text>Get ReturnsIn the previous lesson we used 2-dimensional randomly correlated data to see how we can use PCA to for dimensionality reduction. In this notebook, we will do the same but for 490-dimensional data of stock returns. We will get the stock returns using Zipline and data from Quotemedia, just as we learned in previous lessons. The function `get_returns(start_date, end_date)` in the `utils` module, gets the data from the Quotemedia data bundle and produces the stock returns for the given `start_date` and `end_date`. You are welcome to take a look at the `utils` module to see how this is done.In the code below, we use `utils.get_returns` funtion to get the returns for stock data between `2011-01-05` and `2016-01-05`. You can change the start and end dates, but if you do, you have to make sure the dates are valid trading dates.<jupyter_code>import utils

# Get the returns for the fiven start and end date. Both dates must be valid trading dates
returns = utils.get_returns(start_date='2011-01-05', end_date='2016-01-05')

# Display the first rows of the returns
returns.head()<jupyter_output><empty_output><jupyter_text>Visualizing the DataAs we san see above, the `returns` dataframe, contains the stock returns for 490 assets. Eventhough we can't make 490-dimensional plots, we can plot the data for two assets at a time. This plot willl then show us visually how correlated the stock returns are for a pair of stocks.In the code below, we use the `.plot.scatter(x, y)` method to make a scatter plot of the returns of column `x` and column `y`. The `x` and `y` parameters are both integers and idicate the number of the columns we want to plot. For example, if we want to see how correlated the stock of `AAL` and `AAPL` are, we can choose `x=1` and `y=3`, since we can see from the dataframe above that stock `AAL` corresponds to column number `1`, and stock `AAPL` corresponds to column number `3`. You are encouraged to plot different pairs of stocks to see how correlated they are.<jupyter_code>%matplotlib inline

import matplotlib.pyplot as plt

# Set the default figure size
plt.rcParams['figure.figsize'] = [10.0, 6.0]

# Make scatter plot
ax = returns.plot.scatter(x = 1, y = 3, grid = True, color = 'white', alpha = 0.5, linewidth = 0)
ax.set_facecolor('lightslategray')<jupyter_output><empty_output><jupyter_text>Correlation of ReturnsApart from visualizing the correlation between stocks as we did above, we can also create a correlation dataframe that gives the correlation between every stock. In the code below, we can accomplish this using the `.corr()` method to calculate the correlation between all the paris of stocks in our `returns` dataframe.<jupyter_code># Display the correlation between all stock pairs in the returns dataframe
returns.corr(method = 'pearson').head()<jupyter_output><empty_output><jupyter_text>As we can see, this is a better way to see how correlated the stock returns are than through visulaization. By looking at the table we can easily spot which pairs of stock have the highest correlation.  TODO:In the code below, make a scatter of equity `A` and equity `XOM`.<jupyter_code>%matplotlib inline
import operator
import matplotlib.pyplot as plt

# Set the default figure size
plt.rcParams['figure.figsize'] = [10.0, 6.0]

# Make scatter plot

a_col_idx = [idx for (idx, col) in enumerate(returns.columns) if col.symbol == "A"]
b_col_idx = [idx for (idx, col) in enumerate(returns.columns) if col.symbol == "XOM"]


ax = returns.plot.scatter(x = a_col_idx, y = b_col_idx, grid = True, color = 'white', alpha = 0.5, linewidth = 0)
ax.set_facecolor('lightslategray')<jupyter_output><empty_output><jupyter_text>TODO:In the code below, write a function `get_num_components(df, var_ret)` that takes a dataframe, `df`, and a value for the desired amount of variance you want to retain from the `df` dataframe,`var_ret`. In this case, the parameter `df` should be the `returns` dataframe obtained above. The parameter  `var_ret` must be anumber between 0 and 1. The function should return the number of principal components you need to retain that amount of variance. To do this, use Scikit-Learn's PCA() class and its `.explained_variance_ratio_`. The function should also print the total amount of variance retained.<jupyter_code># import resources
from sklearn.decomposition import PCA


def get_num_components(df, var_ret):
    
    print(df.shape[1])
    print(var_ret)
    
    # Implement Function
    
    pca = PCA(n_components = df.shape[1])
    
    pca.fit(df)
    
    needed_components = 0
    
    var_sum = 0
    
    for i in range(0, df.shape[1]):

        if var_sum >= var_ret:
            print('Total Variance Retained: ', pca.explained_variance_ratio_[0:needed_components].sum())
            return needed_components
        else:
            needed_components += 1            
            var_sum += pca.explained_variance_ratio_[i]
    
    return needed_components
        
            
num_components = get_num_components(returns, 0.9)

print('\nNumber of Principal Components Needed: ', num_components)<jupyter_output>490
0.9
Total Variance Retained:  0.900236862496

Number of Principal Components Needed:  179<jupyter_text>TODO:In the previous section you calculated the number of principal compenents needed to retain a given amount of variance. As you might notice you can greatly reduce the dimensions of the data even if you retain a high level of variance (`var_ret` > 0.9). In the code below, use the number of components needed calculated in the last section, `num_components` to calculate by the percentage of dimensionality reduction. For example, if the original data was 100-dimensional, and the amount of components needed to retian a certain amount of variance is 70, then we are able to reduce the data by 30%.<jupyter_code># Calculate the percentage of dimensionality reduction
red_per = ((returns.shape[1] - num_components) / returns.shape[1]) * 100

print('We were able to reduce the dimenionality of the data by:', red_per, 'percent')<jupyter_output>We were able to reduce the dimenionality of the data by: 63.469387755102034 percent
{'fraction_non_alphanumeric': 0.08903743315508021, 'fraction_numerical': 0.027673796791443852, 'mean_word_length': 4.460583941605839, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '132253', 'license': 'Unlicense', 'n_tokens_mistral': 2356, 'n_tokens_neox': 2219, 'n_words': 923, 'path': 'quizes/pca-coding-exercise/pca_basics.ipynb', 'repo_name': 'babeal/udacity-ai-for-trading', 'size': '7480'}