<jupyter_start><jupyter_text>Prepare storage and get Data<jupyter_code>from google.colab import drive
drive.mount('/gdrive')
!wget -O data.zip http://hck.re/TT3Xkb
!unzip data.zip<jupyter_output>Archive:  data.zip
   creating: DataSet/
  inflating: DataSet/Train.csv       
  inflating: DataSet/Sample_submission.csv  
  inflating: DataSet/Test.csv<jupyter_text>Understanding the Problem statement Referring to the competition description ``The definition that is used for the target variable is every 30+ or X+ days delinquent twice in forward  12 months.``If I understand correctly the target variable predicts the number of times that the particular customer will become delinquent (i.e). Fails to pay his dues in the period mentioned. Import Libraries<jupyter_code>import pandas as pd
from tqdm import tqdm_notebook
from pathlib import Path<jupyter_output><empty_output><jupyter_text>Read data<jupyter_code>DATA_PATH = Path('DataSet')
train_data = pd.read_csv(DATA_PATH/'Train.csv')
train_data.head()<jupyter_output>/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py:2718: DtypeWarning: Columns (746,835) have mixed types. Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)<jupyter_text>Check data<jupyter_code>train_data.shape<jupyter_output><empty_output><jupyter_text>This is a huge number of columns, We have been warned by pandas that **Col746** and **Col835** has mixed dtypes, Let's check them.<jupyter_code>train_data.dtypes.value_counts().plot.bar()<jupyter_output><empty_output><jupyter_text>Let's check the object type columns since that is quite low in the dataset<jupyter_code>object_cols = train_data.select_dtypes(['object'])
object_cols.head()<jupyter_output><empty_output><jupyter_text>Let's check the number of nan values<jupyter_code>object_cols.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 17521 entries, 0 to 17520
Data columns (total 3 columns):
Col1      17521 non-null object
Col747    626 non-null object
Col836    624 non-null object
dtypes: object(3)
memory usage: 410.7+ KB<jupyter_text>Ignoring **Col1** since it is an ID column and highly unlikely to have any information, we can see that it has too many **NaN** values, let's check the values present in the columns<jupyter_code>object_cols.loc[:, 'Col747'].value_counts()
object_cols.loc[:, 'Col836'].value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09514087382605145, 'fraction_numerical': 0.026541445487954268, 'mean_word_length': 6.65625, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '193949', 'license': 'MIT', 'n_tokens_mistral': 842, 'n_tokens_neox': 796, 'n_words': 225, 'path': 'HDFC Banking scorecard/HDFC_Banking_scorecard_EDA.ipynb', 'repo_name': 'mani2106/Competition-Notebooks', 'size': '2449'}