<jupyter_start><jupyter_text>**NBA Career Prediction****Predict 5-Year Career Longevity for NBA Rookies**  WEEK 1 - Data PreparationIt will start by read data that used in the project, then check dependent varaible distribution**The steps are:**1. read data2. explore dataset 3. Prepare data  1. Read data **[1.1]** Import all modules needed<jupyter_code>import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from joblib import dump
from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>**[1.2]** Download train.csv and test.csv into data/raw folder then read data in notebook<jupyter_code># Set index_col = 0 to get rid of ID column
data = pd.read_csv("../data/raw/train.csv", index_col=0)
test = pd.read_csv("../data/raw/test.csv", index_col=0)<jupyter_output><empty_output><jupyter_text>2. Explore Dataset **[2.1]** Display first 5 rows of train and test dataframe<jupyter_code>data.head()<jupyter_output><empty_output><jupyter_text>**Note:** There are negative values in columns 3P Made, 3PA, but these values should not be negative.<jupyter_code>test.head()<jupyter_output><empty_output><jupyter_text>**Note:** There are negative values in column BLK but these values should not be negative. **[2.2]** Find all columns containing negative values in data and test<jupyter_code>(data < 0).any()
(test < 0).any()<jupyter_output><empty_output><jupyter_text>**Note:** Columns containing Negative values in both data and test: 3P Made, 3PA, 3P%, BLK Columns containing negative values in data only: GP, FT% **[2.3]** Further investigate negative values<jupyter_code>data[data["GP"] < 0].index
data[data["FT%"] < 0].index
name_list = ["3P Made", "3PA", "3P%", "BLK"]
for name in name_list:
    print(f"No. of Negative values in column '{name}' in (data, test): ({len(data[data[name] < 0].index)}, {len(test[test[name] < 0].index)})")<jupyter_output>No. of Negative values in column '3P Made' in (data, test): (1629, 775)
No. of Negative values in column '3PA' in (data, test): (1658, 773)
No. of Negative values in column '3P%' in (data, test): (878, 435)
No. of Negative values in column 'BLK' in (data, test): (1048, 456)<jupyter_text>Data cleaning to be made:    Drop columns: 3P Made, 3PA, 3P%, BLK - too many negative values in both train and test csv.         Reason: non of these values should be negative, considered as invalid variables.    Drop rows: 1756, 7478, 11294        Reason: only 3 negative values in GP and FT% found in the train set and 0 in test set, considered as input mistakes.    **[2.4]** Remove rows and columns<jupyter_code>data_row_cleaned = data.drop(data[data["FT%"] < 0].index)
data_row_cleaned = data_row_cleaned.drop(data_row_cleaned[data_row_cleaned["GP"] < 0].index)
data_column_cleaned = data_row_cleaned.drop(['3P Made', '3PA', '3P%', 'BLK'], axis = 1)
test_column_cleaned = test.drop(['3P Made', '3PA', '3P%', 'BLK'], axis = 1)<jupyter_output><empty_output><jupyter_text>**[2.5]** Display dimensions of train and test dataframe<jupyter_code>data_column_cleaned.shape
test_column_cleaned.shape<jupyter_output><empty_output><jupyter_text>**[2.6]** Display summary of train and test dataframe<jupyter_code>data_column_cleaned.info()
test_column_cleaned.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 3799 entries, 1 to 8183
Data columns (total 15 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   GP      3799 non-null   int64  
 1   MIN     3799 non-null   float64
 2   PTS     3799 non-null   float64
 3   FGM     3799 non-null   float64
 4   FGA     3799 non-null   float64
 5   FG%     3799 non-null   float64
 6   FTM     3799 non-null   float64
 7   FTA     3799 non-null   float64
 8   FT%     3799 non-null   float64
 9   OREB    3799 non-null   float64
 10  DREB    3799 non-null   float64
 11  REB     3799 non-null   float64
 12  AST     3799 non-null   float64
 13  STL     3799 non-null   float64
 14  TOV     3799 non-null   float64
dtypes: float64(14), int64(1)
memory usage: 474.9 KB<jupyter_text>**[2.7]** Display descriptive data of train and test dataframe<jupyter_code>data_column_cleaned.describe()
test_column_cleaned.describe()<jupyter_output><empty_output><jupyter_text>**[2.8]** Check distribution of target variable<jupyter_code>data_column_cleaned['TARGET_5Yrs'].value_counts()/len(data)<jupyter_output><empty_output><jupyter_text>Imbalanced data found, should consider SMOTE, Oversampling or other methods  3. Prepare data **[3.1]** Create a copy of train and save it into a variable called train_cleaned<jupyter_code>data_cleaned = data_column_cleaned.copy()<jupyter_output><empty_output><jupyter_text>**[3.2]** Remove leading and trailing space from the column names<jupyter_code>data_cleaned.columns = data_cleaned.columns.str.strip()<jupyter_output><empty_output><jupyter_text>**[3.3]** Extract the column `TARGET_5Yrs` and save it into variable called target<jupyter_code>target = data_cleaned.pop('TARGET_5Yrs')<jupyter_output><empty_output><jupyter_text>**[3.4]** Split the data (80%) randomly with random_state=8 into 2 different sets: training (80%) and validation (20%)<jupyter_code>X_train, X_val, y_train, y_val = train_test_split(data_cleaned, target, test_size=0.2, random_state=0, stratify=target)<jupyter_output><empty_output><jupyter_text>**[3.5]** Instantiate the StandardScaler<jupyter_code>scaler = StandardScaler()<jupyter_output><empty_output><jupyter_text>**[3.6]** Fit and apply the scaling on data and test<jupyter_code>X_train = scaler.fit_transform(X_train)
X_val = scaler.transform(X_val)
test = scaler.transform(test_column_cleaned)<jupyter_output><empty_output><jupyter_text>**[3.6]** Save the scaler into the folder models and call the file scaler.joblib<jupyter_code>dump(scaler, '../models/scaler.joblib')<jupyter_output><empty_output><jupyter_text>**[3.7]** Save the different sets in the folder `data/processed`<jupyter_code>np.save('../data/processed/data_cleaned', data_cleaned)
np.save('../data/processed/target', target)

np.save('../data/processed/X_train', X_train)
np.save('../data/processed/y_train', y_train)

np.save('../data/processed/X_val', X_val)
np.save('../data/processed/y_val', y_val)

np.save('../data/processed/test', test)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12898114403422595, 'fraction_numerical': 0.04246553636507685, 'mean_word_length': 5.4081218274111675, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '353783', 'license': 'MIT', 'n_tokens_mistral': 2388, 'n_tokens_neox': 2198, 'n_words': 590, 'path': 'notebooks/wu_wenying-14007025-week1_datapreparation.ipynb', 'repo_name': 'nuwanprabhath/kaggle-nba-career-prediction', 'size': '6311'}