<jupyter_start><jupyter_text>Predicting the NBA MVP with PCA-RNNThis project is aimed to build a PCA-RNN model to predict the winner of The National Basketball Association’s (NBA) the Most Valuable Player (MVP) for 2017-18 regular season. Using both NBA players individual statistics and MVP voting history from 2000-01 to 2016-17 season, we will develop a recurrent neural network (RNN) model in order to estimate who can receive the highest MVP votes in 2017-18 season. All the datasets will firstly be scaled using principal component analysis (PCA). The results show that PCA can significantly ascertain appropriate input parameters for our RNN model and the RNN model is a cogent predicator of NBA’s MVP.   What is Principal Component Analysis?Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. If there are n observations with p variables, then the number of distinct principal components is min(n-1, p).This transformation is defined in such a way that the first principal component has the largest possible variance (that is, accounts for as much of the variability in the data as possible), and each succeeding component in turn has the highest variance possible under the constraint that it is orthogonal to the preceding components. The resulting vectors are an uncorrelated orthogonal basis set. PCA is sensitive to the relative scaling of the original variables.from __[Principal component analysis - Wikipedia)](https://en.wikipedia.org/wiki/Principal_component_analysis)__Objectives of PCA    * PCA reduces attribure space from a larger number of variables to a small number of factors and as such is a "non-dependent" procedure.    * PCA is a dimensionality reduction or data compression method. The goal is dimensions are interpretable    * To select a subset of variables from a larger set, based on which original variables have the highest correlations with the pricipal component. What is Recurrent Neural Network?![RNN](images/RNN.png)A recurrent neural network (RNN) is a class of artificial neural network where connections between units form a directed graph along a sequence. This allows it to exhibit dynamic temporal behavior for a time sequence. from __[Recurrent neural network - Wikipedia)](https://en.wikipedia.org/wiki/Recurrent_neural_network)__ Importing and preparing the dataWe have two main datasets which were collected from [Kaggle](https://www.kaggle.com) and [NBA Reference](https://www.basketball-reference.com). The training and testing dataset contains 10221 minutes of data ranging from 2000 to 2017 NBA seasons and predicting dataset contains 664 minutes of data of 2017-2018 NBA season.<jupyter_code># Import
%matplotlib inline
from collections import Counter
import random

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
from sklearn.cluster import KMeans, DBSCAN
from scipy.stats import multivariate_normal
import tensorflow as tf
from sklearn.preprocessing import MinMaxScaler
#Make the plot larger
plt.rcParams['figure.figsize'] = (16, 9)
plt.style.use('ggplot')<jupyter_output><empty_output><jupyter_text>**data_traintest:** used to train and test model**data_result:** used to joint with predicting results(voting rates) and display specific player stats**data_pca:** used to joint with 'data_traintest' and implement pca function(because we need extract same columns from training and predicting data)<jupyter_code>#Load datasets
#Training and testing dataset
data_traintest = pd.read_csv('00-17stats.csv')
#This dataset is used to display specific player stats
data_result = pd.read_csv('17-18stats.csv')
#This dataset is used to combine with ‘data_traintest’ and implement pca function
data_pca = pd.read_csv('17-18statsForPCA.csv')
print(data_traintest.shape)
data_traintest.head()<jupyter_output>(10221, 52)<jupyter_text>Stat Glossary*We use voting rates as the standard stat to evaluate possibility of wining MVP.*G - Games Played&emsp;&emsp;&emsp;FG - Field GoalsGS - Games Started&emsp;&emsp;&emsp;FGA - Field Goal AttemptsMP - Minutes Played&emsp;&emsp;&emsp;FG% - Field Goal PercentagePER - Player Efficiency Rating&emsp;&emsp;&emsp;3P - 3-Point Field GoalsTS% - True Shooting Percentage&emsp;&emsp;&emsp;3PA - 3-Point Field Goal Attempts3PAr - 3-Point Attempt Rate&emsp;&emsp;&emsp;2P - 2-Point Field GoalsORB% - Offensive Rebound Percentage&emsp;&emsp;&emsp;2PA - 2-Point Field Goal AttemptsDRB% - Defensive Rebounds Percentage&emsp;&emsp;&emsp;2P% - 2-Point Field Goal PercentageTRB% - Total Rebound Percentage&emsp;&emsp;&emsp;eFG% - Effective Field Goal PercentageAST% - Assist Percentage&emsp;&emsp;&emsp;FT - Free ThrowsSTL% - Steal Percentage&emsp;&emsp;&emsp;FTA - Free Throw AttemptsBLK% - Block Percentage&emsp;&emsp;&emsp;FT% - Free Throw PercentageTOV% - Turnovers Perventage&emsp;&emsp;&emsp;ORB - Offensive ReboundsUSG% - Usage Percentage&emsp;&emsp;&emsp;DRB - Defensive ReboundsOWS - Offensive Win Shares&emsp;&emsp;&emsp;TRB - otal ReboundsDWS - Defensive Win Shares&emsp;&emsp;&emsp;AST - AssistsWS - Win Shares&emsp;&emsp;&emsp;STL - StealsWS/48 - Win Shares Per 48 Minutes&emsp;&emsp;&emsp;BLK - BlocksOBPM - Offensive Box Plus/Minus&emsp;&emsp;&emsp;TOV - TurnoversDBPM - Defensive Box Plus/Minus&emsp;&emsp;&emsp;PF - Personal FoulsBPM - Box Plus/Minus&emsp;&emsp;&emsp;PTS - PointsVORP - Value Over Replacement Player&emsp;&emsp;&emsp;Voting - Voting Rate          Data CleaningRemove missing data by replacing them with mean values of their own columns.Drop column 'Rk' because it is meaningless and three columns where values are of 'string' type for future computing. *After predicting voting rates of new season's players, we could get these full information by jointing predicting results with data_result.*<jupyter_code>#Data cleaning
#Fill NA values
data_traintest.fillna(data_traintest.mean(),inplace=True)
data_result.fillna(data_result.mean(),inplace=True)
data_pca.fillna(data_pca.mean(),inplace=True)
#Drop 'string' columns and useless columns
data_traintest.drop('Player',axis=1, inplace=True)
data_traintest.drop('Tm',axis=1, inplace=True)
data_traintest.drop('Pos',axis=1, inplace=True)
data_traintest.drop('Rk',axis=1, inplace=True)
data_pca.drop('Player',axis=1, inplace=True)
data_pca.drop('Tm',axis=1, inplace=True)
data_pca.drop('Pos',axis=1, inplace=True)
data_pca.drop('Rk',axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Threshold and RequirementsAfter counting and analyzing all MVP data of past seasons, we find that MVP winners have to be all-around productive and follow many common rules. For example, every MVP in the modern era (since 1979-80) put up a PER of at least 18.5 in the season prior; Individual advanced data is key and 84 percent of MVPs averaged at least 0.20 WS/48 during the season; All MVPs played at least 1500 minutes in that season. So we set Thresholds as follows and only players who meet these requirements can be taken into consideration.*Every player must put up a PER of at least 18.5 in the season.**Every player need to get an average of at least 0.20 WS/48 in the season.**Every player plays  at least 1500 minutes in the season.*<jupyter_code>data_traintest = data_traintest[data_traintest.MP >= 1500]
data_traintest = data_traintest[data_traintest.PER >= 18.5]
data_traintest = data_traintest[data_traintest.WS/48 > 0.2]
data_result = data_result[data_result.MP >= 1500]
data_result = data_result[data_result.PER >= 18.5]
data_result = data_result[data_result.WS/48 > 0.2]
data_pca = data_pca[data_pca.MP >= 1500]
data_pca = data_pca[data_pca.PER >= 18.5]
data_pca = data_pca[data_pca.WS/48 > 0.2]<jupyter_output><empty_output><jupyter_text>*Because our target value, voting rates, is a decimal, we raise it by 1000 times to make the results more obvious.*<jupyter_code>data_traintest['Voting'] = data_traintest['Voting']*1000<jupyter_output><empty_output><jupyter_text>Joint data_traintest with data_pca and get the final dataset<jupyter_code>#Get training, testing and predicting dataset of PCA-RNN model
data_pca = data_traintest.append(data_pca)
data_pca
# Get dimensions of dataset
n = data_traintest.shape[0]
p = data_traintest.shape[1]
end = data_pca.shape[0]
# Make data a np.array
data_traintest = data_traintest.values
data_result = data_result.values
data_pca = data_pca.values<jupyter_output><empty_output><jupyter_text>Implement PCA functionWe first implemented PCA function on our source data.We will ask the model to return 15 components—that is, a 15-dimensional representation of the data.<jupyter_code>from sklearn.decomposition import PCA
pca = PCA(n_components=15)
principalComponents = pca.fit_transform(data_pca)
principalComponents<jupyter_output><empty_output><jupyter_text>Start Creating RNN model and predict MVPWe first split our dataset into training, testing and predicting parts through index.Then we are specifying our variable placeholders.We use two placeholders in order to fit our model: X contains the network's inputs and Y the network's outputs. Weights and biases are represented as variables in order to adapt during training. Variables need to be initialized, prior to model training. Our model consists of five hidden layers and we define the amount in every layers from 2048 to 128 in gradient. The number of neurons in each subsequent layer reduces the amount of information that the network identifies in the previous layer.We initialize the type of activation function we want. Leaky ReLU stands for “Leaky Rectified Linear Units”. The default activation function is ReLU. But for whatever reason, the output of a ReLU is consistently 0 (for example, if the ReLU has a large negative bias), then the gradient through it will consistently be 0. In instead leaky ReLUs are used, the gradient is never 0, and this problem is avoided.The choice of optimizer for a deep learning model can mean the difference between good results. And the Adam optimizer is an extension to stochastic gradient descent that has recently seen broader adoption for deep learning applications in computer vision and natural language processing.<jupyter_code>#Get training and testing data
#Set start index and end index
#Index of Train data
train_start = 0
train_end = int(np.floor(0.8*n))
#Index of Test Data
test_start = train_end + 1
test_end = n-1
#Index of Predict Data
predict_start = test_end + 1
predict_end = end
#data
data_train = principalComponents[np.arange(train_start, train_end), :]
data_test = principalComponents[np.arange(test_start, test_end), :]
data_predict = principalComponents[np.arange(predict_start, predict_end), :]
#Build X and y
X_train = data_train[:, 1:]
y_train = data_train[:, 0]
X_test = data_test[:, 1:]
y_test = data_test[:, 0]
X_predict = data_predict[:, 1:]
#Start creating RNN model and seting variables

# Number of player in training data
n_player = X_train.shape[1]

# Neurons
n_neurons_1 = 2048
n_neurons_2 = 1024
n_neurons_3 = 512
n_neurons_4 = 256
n_neurons_5 = 128<jupyter_output><empty_output><jupyter_text>Introduction to TensorflowTensorFlow is a deep neural network, so it learns to perform a task through positive reinforcement and works through layers of data (nodes) to help it determine the correct outcome.The user defines an abstract representation of the model (neural network) through placeholders and variables. Afterwards, the placeholders get "filled" with real data and the actual computations take place.<jupyter_code># Session
net = tf.InteractiveSession()
# Placeholder
X = tf.placeholder(dtype=tf.float32, shape=[None, n_player])
Y = tf.placeholder(dtype=tf.float32, shape=[None])
# Initializers--tf.zeros_initializer
# Initializers are used to initialize the network’s variables before training.
sigma = 1
weight_initializer = tf.variance_scaling_initializer(mode="fan_avg", distribution="uniform", scale=sigma)
bias_initializer = tf.zeros_initializer()
# Hidden weights
W_hidden_1 = tf.Variable(weight_initializer([n_player, n_neurons_1]))
bias_hidden_1 = tf.Variable(bias_initializer([n_neurons_1]))
W_hidden_2 = tf.Variable(weight_initializer([n_neurons_1, n_neurons_2]))
bias_hidden_2 = tf.Variable(bias_initializer([n_neurons_2]))
W_hidden_3 = tf.Variable(weight_initializer([n_neurons_2, n_neurons_3]))
bias_hidden_3 = tf.Variable(bias_initializer([n_neurons_3]))
W_hidden_4 = tf.Variable(weight_initializer([n_neurons_3, n_neurons_4]))
bias_hidden_4 = tf.Variable(bias_initializer([n_neurons_4]))
W_hidden_5 = tf.Variable(weight_initializer([n_neurons_4, n_neurons_5]))
bias_hidden_5 = tf.Variable(bias_initializer([n_neurons_5]))
# Output weights
W_out = tf.Variable(weight_initializer([n_neurons_5, 1]))
bias_out = tf.Variable(bias_initializer([1]))
# Hidden layer(leaky_relu)
hidden_1 = tf.nn.leaky_relu(tf.add(tf.matmul(X, W_hidden_1), bias_hidden_1))# 
hidden_2 = tf.nn.leaky_relu(tf.add(tf.matmul(hidden_1, W_hidden_2), bias_hidden_2))
hidden_3 = tf.nn.leaky_relu(tf.add(tf.matmul(hidden_2, W_hidden_3), bias_hidden_3))
hidden_4 = tf.nn.leaky_relu(tf.add(tf.matmul(hidden_3, W_hidden_4), bias_hidden_4))
hidden_5 = tf.nn.leaky_relu(tf.add(tf.matmul(hidden_4, W_hidden_5), bias_hidden_5))
# Output layer (transpose!)
out = tf.transpose(tf.add(tf.matmul(hidden_5, W_out), bias_out))
# Cost function--user defined
# MSE computes the average squared deviation between predictions and targets
mse = tf.reduce_mean(tf.squared_difference(out, Y))
# Optimizer--Adam
# Used to compute and adapt weights and biases
opt = tf.train.AdamOptimizer().minimize(mse)
#init
init = tf.global_variables_initializer()<jupyter_output><empty_output><jupyter_text>A quick look at our actual testing resultBecause our data has been implemented by PCA, the scaling has changed.<jupyter_code># Setup plot
plt.ion()
fig = plt.figure()
ax1 = fig.add_subplot(111)
line1, = ax1.plot(y_test)
plt.show()
#Make a plot to view basic situation of MVP Voting data<jupyter_output><empty_output><jupyter_text>Fitting the neural networkAfter setting variables of our RNN model, we run the model and predict testing result and MVP in 17-18 seasom.We set the number of epochs as 4000. Because we have test for many times and 4000 can be a relatively superior choice.<jupyter_code># Number of iterations or training cycles
epochs = 4000
#Run the model
#Display the changing process of mse every 500 epochs
with tf.Session() as sess:
    init.run()
    for e in range(epochs):
        sess.run(opt, feed_dict={X: X_train, Y: y_train})
        if e % 500 == 0:
            loss = mse.eval(feed_dict={X: X_train, Y: y_train})
            print(e, "\tMSE:", loss)
            
    y_pred = sess.run(out, feed_dict={X: X_test})
    # Predict MVP of new season
    MVP = sess.run(out, feed_dict={X: X_predict})<jupyter_output>0 	MSE: 379564.0
500 	MSE: 0.00187836
1000 	MSE: 4.41761e-08
1500 	MSE: 2.67115e-09
2000 	MSE: 2.87407e-09
2500 	MSE: 94.346
3000 	MSE: 0.0872788
3500 	MSE: 3.32703e-05<jupyter_text>**Copyright 2017 <NAME>**Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:    The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.<jupyter_code>#First observe the accuracy of the predicting results and actual values of test data.
#Make the plot
plt.title("Pridiction VS Actual", fontsize=14)
plt.plot(pd.Series(np.ravel(y_test)), "bo", markersize = 5, label="Actual")
plt.plot(pd.Series(np.ravel(y_pred)), "r.", markersize = 5, label="Pridicting")
plt.legend(loc="upper left")
plt.xlabel("Players")

plt.show()<jupyter_output><empty_output><jupyter_text>Predicting MVP in 17-18 season<jupyter_code>#Because the shape of MVP value array and Player array is different, transpose it.
MVP = MVP.T
#Combine the arrays and match the predicting results to the players.
Players = np.concatenate((data_result,MVP),axis=1)
#Sort players in desending order for MVP value
Players = Players[Players[:,51].argsort()][::-1]
Players<jupyter_output><empty_output><jupyter_text>Show specific players intuitively<jupyter_code>l = Players[:,2]
y = Players[:,51]

#Make the plot of predicting results of new season data
plt.title("Pridiction MVP in 17-18 season", fontsize=14)
plt.scatter(l, y, marker='o')  
plt.ylabel("Predicting MVP PTS", fontsize=14)
#Remove x label
plt.xticks([])
#Set labels
for a, b in zip(l, y):  
    plt.text(a, b, a, ha='center', va='bottom', fontsize=15)  
    
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08443013966319904, 'fraction_numerical': 0.024139318351629403, 'mean_word_length': 5.861198738170347, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 65, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '130967', 'license': 'MIT', 'n_tokens_mistral': 5649, 'n_tokens_neox': 5213, 'n_words': 1918, 'path': 'NEU_ADS_Student_Project_Portfolio_Examples/NBA MVP Prediction with Principal Component Analysis/Project/Protfolio Blog.ipynb', 'repo_name': 'Chau-Xochitl/INFO_7390', 'size': '17408'}