<jupyter_start><jupyter_text>Dealing with Missing Data We often find the missing data in dataset provided to us or we download. And Machine learning algorithms do not give optimal output when trained on such dataset with missing rows or columns. So, in most of the cases, we have to handle this kind of situation.There are two appraches to deal with missing data Eliminate the training examples or features with missing data Imputation Before we implement techniques, lets look how to find training examples with missing values<jupyter_code>#import libraries
import pandas as pd
import numpy as np
from io import StringIO
csv_data=\
'''
P,Q,R,S
11,12,15,13
5.9,9.8,,8.8
12,34,32,
'''
df=pd.read_csv(StringIO(csv_data))
print(df)
# isnull() method is our tool to find missing data in dataframe .
df.isnull()
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>We find that there are 1 Null or NaN values in column R and S  Technique 1 - Eliminating training examples or features with missing values<jupyter_code># dropna() is our tool to eliminate missing data
df.dropna(axis=0)<jupyter_output><empty_output><jupyter_text>Here axis=0 means that where there is a NaN values in row, just drop them<jupyter_code>df.dropna(axis=1)<jupyter_output><empty_output><jupyter_text>Here axis=1 means that , drop the columns where there are NaN values<jupyter_code>df.dropna(thresh=2)<jupyter_output><empty_output><jupyter_text>Here thresh=2 means that , drop those rows where there are fewer than 2 real values https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html &nbsp; is link for more use of parameters of dropna  Technique 2 - Imputation Removing training examples or features with few missing values , we can lose the meaningful information or insights. So,it is not feasible method to implement.  In practical appraches, we do imputation , we just fill the missing data with other values.<jupyter_code>df.fillna(df.mean())
from sklearn.impute import SimpleImputer
simpleImputer = SimpleImputer( strategy='median' , missing_values=np.nan)
simpleImputer = simpleImputer.fit(df)
imputed_data = simpleImputer.transform(df)
print(imputed_data)<jupyter_output>[[11.  12.  15.  13. ]
 [ 5.9  9.8 23.5  8.8]
 [12.  34.  32.  10.9]]<jupyter_text>strategy='mean' means that we fill missing data with median values There are 2 more values for strategy i.e  mean most_frequent<jupyter_code>simpleImputer_mean = SimpleImputer(strategy='mean')
simpleImputer_mean = simpleImputer_mean.fit(df)
imputedMean_data = simpleImputer_mean.transform(df)
print(imputedMean_data)<jupyter_output>[[11.  12.  15.  13. ]
 [ 5.9  9.8 23.5  8.8]
 [12.  34.  32.  10.9]]<jupyter_text>Check Out more tutorials on building good training data<jupyter_code>%load_ext watermark
 %watermark -a '<NAME>' -u -d<jupyter_output>Author: <NAME>

Last updated: 2021-01-14
{'fraction_non_alphanumeric': 0.08732099196646874, 'fraction_numerical': 0.03178484107579462, 'mean_word_length': 5.553775743707094, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '215257', 'license': 'Apache-2.0', 'n_tokens_mistral': 992, 'n_tokens_neox': 930, 'n_words': 319, 'path': '3 Building Good Training Dataset/1. Dealing with categorical data.ipynb', 'repo_name': 'ankurwasnik/LearnML', 'size': '2875'}