<jupyter_start><jupyter_text>Data cleaning is a major part of data science, missing values can be deeply frustrating.  For this let's see how to handle missing values and messy data<jupyter_code>import pandas as pd
import numpy as np
## input all the data
building_permits = pd.read_csv("Building_Permits.csv")
np.random.seed(0)
building_permits.head(5)<jupyter_output><empty_output><jupyter_text>Let's figure how much percent data we are actually missing<jupyter_code>missing_values_count = building_permits.isnull().sum()

## Have a look at some missing values in the column
missing_values_count[0:10]
total_cells = np.product(building_permits.shape) ##count of total cells
total_missing = missing_values_count.sum()

missing_percent = (total_missing/total_cells) * 100
print("Total percent missing :",missing_percent)<jupyter_output>Total percent missing : 26.26002315058403<jupyter_text>Let's have a look at the dataset on why we have such high missing value percent We might have some missing coz they don't exist or if they weren't recorded<jupyter_code>## Street number suffix and street suffix has missing values<jupyter_output><empty_output><jupyter_text>Drop the missing values ( This option is not recommended coz sometimes losing a feature might account for less data intutition and may lead to incorrect predictions, so have a look at the doumentation)<jupyter_code>## Let's remove all the rows which contain a missing value
building_permits.dropna()
## Let's remove the columns with atleast one missing value
cols_drop_na = building_permits.dropna(axis=1)
cols_drop_na.head()
## Now let's have a look at the actual data we lost
print("Columns present in the original dataset:",building_permits.shape[1])
print("Columns with dropped nulls:",cols_drop_na.shape[1])<jupyter_output>Columns present in the original dataset: 43
Columns with dropped nulls: 12<jupyter_text>Automatically fill in the missing values There are three popular approaches in this domain [1].  Drop columns with Missing values which we have seen above and not a recommended choice [2]. Imputation - Fill in the missing values with some number, for example we can just rather fill in the null values with the mean of the values in the column [3]. Extension to imputation - In this we approach the missing value cases same as before but additionally we have a new column which shows the location of the imputed entries (this might be a very important step in the right direction or not a right step at all)<jupyter_code>from sklearn.impute import SimpleImputer
## first approach
reduced_cols = building_permits.dropna(axis=1)
reduced_cols.head(10)
building_permits.fillna(0) ## second apprach
building_permits.fillna(method='bfill',axis=0).fillna(0)
## You can also fill a specific column with a mean value
## Taking a subset for example
subset = building_permits['Revised Cost']
print("Missing values in col:",subset.isnull().sum())
subset.fillna(value=subset.mean(), inplace=True)
print("Updated Column")
subset.head(20)<jupyter_output>Missing values in col: 6066
Updated Column
{'fraction_non_alphanumeric': 0.0715452466514211, 'fraction_numerical': 0.015027768703038223, 'mean_word_length': 6.3253588516746415, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '302678', 'license': 'MIT', 'n_tokens_mistral': 894, 'n_tokens_neox': 842, 'n_words': 359, 'path': 'Handling_missing_values.ipynb', 'repo_name': 'kushantp58/Feature_Engineering', 'size': '3061'}