<jupyter_start><jupyter_text>Load & Analyse Data<jupyter_code>import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>Import csv to data frame<jupyter_code>df=pd.read_csv('../data/raw/train.csv')<jupyter_output><empty_output><jupyter_text>Print first 5 rows of datafram to check<jupyter_code>df.head(5)<jupyter_output><empty_output><jupyter_text>Data load was successful. Check the shape of the DF<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>Dataframe has 8000 rows and 22 columns. Deep dive in to the data type and null / not null values<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 8000 entries, 0 to 7999
Data columns (total 22 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   Id_old       8000 non-null   int64  
 1   Id           8000 non-null   int64  
 2   GP           8000 non-null   int64  
 3   MIN          8000 non-null   float64
 4   PTS          8000 non-null   float64
 5   FGM          8000 non-null   float64
 6   FGA          8000 non-null   float64
 7   FG%          8000 non-null   float64
 8   3P Made      8000 non-null   float64
 9   3PA          8000 non-null   float64
 10  3P%          8000 non-null   float64
 11  FTM          8000 non-null   float64
 12  FTA          8000 non-null   float64
 13  FT%          8000 non-null   float64
 14  OREB         8000 non-null   float64
 15  DREB         8000 non-null   float64
 16  REB          8000 non-null   float64
 17  AST          8000 non-null   float64
 18  STL          8000 non-null   float64
 19  BLK   [...]<jupyter_text>There are no null values in the dataframe. Check the Stats to determine validitiy of data<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Describe shows a few columns with negative values like GP which realistially may not be negative. Investigating further...<jupyter_code>df.columns[(df < 0).any()]<jupyter_output><empty_output><jupyter_text>There quite a few columns with values less than zero.  Check the size of dataframe if all the rows with nevative values are removed<jupyter_code>df[(df > 0).all(1)].shape<jupyter_output><empty_output><jupyter_text>Roughly 3500 rows have valid data. Removing rows will not be meaningful in this case. Investigate further to evaluate removal of features vs rows<jupyter_code>for cols in df.columns:
    chk_rows = df[df[cols]<0].shape[0]
    if chk_rows > 0 :
        print(f'Column Name {cols},\tRows with Negative Value {chk_rows},\tPercentage {chk_rows/len(df)*100}')<jupyter_output>Column Name GP,	Rows with Negative Value 2,	Percentage 0.025
Column Name 3P Made,	Rows with Negative Value 1629,	Percentage 20.3625
Column Name 3PA,	Rows with Negative Value 1658,	Percentage 20.724999999999998
Column Name 3P%,	Rows with Negative Value 878,	Percentage 10.975
Column Name FT%,	Rows with Negative Value 1,	Percentage 0.0125
Column Name BLK,	Rows with Negative Value 1048,	Percentage 13.100000000000001<jupyter_text>Based on the above result, we can  - remove the Feature columns 3P Made , 3PA, 3P%, BLK as the rows impacted are > 10%  - remove the Rows with negateive values for GP and FT%Taking 10% as threshhold to retain max available good quality data for train and test Double checking any NA value.<jupyter_code>df.isna().sum()* 100 / len(df)<jupyter_output><empty_output><jupyter_text>All Good, no changes required Checking Duplicates<jupyter_code>df[df.duplicated()]<jupyter_output><empty_output><jupyter_text>All Good. No change required Preparing Data for Training Removing IDs as they will not be required further. And removing other columns based on decision above<jupyter_code>df_cleaned = df.drop(['Id_old','Id', '3P Made', '3PA','3P%', 'BLK' ],axis=1)
df_cleaned = df_cleaned[df_cleaned['GP']>0]
df_cleaned = df_cleaned[df_cleaned['FT%']>0]
print("Original Dataframe Shape",df.shape)
print("Cleansed Dataframe Shape",df_cleaned.shape)<jupyter_output>Original Dataframe Shape (8000, 22)
Cleansed Dataframe Shape (7997, 16)<jupyter_text>Split Feature and Target Column for Training<jupyter_code>x=df_cleaned.drop('TARGET_5Yrs',axis=1)
y=df_cleaned['TARGET_5Yrs']
print("Original Dataframe Shape",x.shape)
print("Cleansed Dataframe Shape",y.shape)<jupyter_output>Original Dataframe Shape (7997, 15)
Cleansed Dataframe Shape (7997,)<jupyter_text>Saving the Column Name for use in checking feature importance<jupyter_code>x_col_names=x.columns.tolist()
y_col_names =['TARGET_5Yrs']
x_col_names , y_col_names<jupyter_output><empty_output><jupyter_text>Feature Scaling<jupyter_code>from sklearn.preprocessing import Normalizer
x = Normalizer().fit_transform(x)<jupyter_output><empty_output><jupyter_text>*Feature Scaling will not change outcome of Tree based algo. However, performing it anyways for good measure and best practice. Also will try to create a generic module out of these steps in future*  Checking if data is balanced or not<jupyter_code>y.value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Clearly it is not balanced.  Split the data into train, validate and test sets<jupyter_code>from sklearn.model_selection import train_test_split
x_data , x_test ,y_data,  y_test = train_test_split(x, y, test_size=0.2, random_state = 8, stratify=y)
x_train , x_val , y_train, y_val = train_test_split(x_data, y_data, test_size=0.2, random_state = 8, stratify=y_data)<jupyter_output><empty_output><jupyter_text>Double check the results from Stratified Split<jupyter_code>y_train.value_counts(normalize=True)
y_val.value_counts(normalize=True)
y_test.value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>All good.  Save the processed dataset for use in model training<jupyter_code>from joblib import dump
#for cross validation training
np.save('../data/processed/sp_x', x)
np.save('../data/processed/sp_y', y)
#for normal train/validate/test 
np.save('../data/processed/sp_x_train', x_train)
np.save('../data/processed/sp_x_val',   x_val)
np.save('../data/processed/sp_x_test',  x_test)
np.save('../data/processed/sp_y_train', y_train)
np.save('../data/processed/sp_y_val',   y_val)
np.save('../data/processed/sp_y_test',  y_test)

np.save('../data/processed/sp_x_col_name',  x_col_names)
np.save('../data/processed/sp_y_col_name',  y_col_names)<jupyter_output><empty_output><jupyter_text>Summary of Observations- There are no NULL/Missing Values- There are no Duplicate Values- Invalid Value was observered in columns '3P Made', '3PA','3P%', 'BLK', 'GP' and 'FT%'- Removed Features '3P Made', '3PA', '3P%' and 'BLK' as there were number of rows impacted. Removing them will reduce the data for training- Removed rows with invalid values for the column 'GP' and'FT%' as there were only minimal rows impacted- Data is highly imbalanced. If any fix is required it will be taken care during the model training - Data is split into Training, Validation and Test sets using train_test_split using stratify on target column- Datasets are saved for use in training Scope for Future Enhancements- Create a module to perform automated basic analysis- Create a module to perform split, perform feature sacling and save data   Visualization<jupyter_code>import matplotlib.pyplot as plt
for cols in df_cleaned.columns:
    if cols != 'TARGET_5Yrs' :
        df_cleaned.plot(x=cols,y='TARGET_5Yrs',kind='scatter')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09501894964807796, 'fraction_numerical': 0.0418245804006497, 'mean_word_length': 4.7146171693735495, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 83, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '228868', 'license': 'MIT', 'n_tokens_mistral': 2708, 'n_tokens_neox': 2434, 'n_words': 733, 'path': 'notebooks/Pitchandi_Sampath_Week1_01-data-exploration.ipynb', 'repo_name': 'ronvoluted/kaggle-nba', 'size': '7388'}