<jupyter_start><jupyter_text>Data Cleaning-----Most of the dataset are not cleaned and usually have messy data so we must clean the data before use that. Data cleaning involves dealing with* Missing data* Duplicated data* Outliers in the data* Extra data that might not be needed* Inconsistent data* Converting data into a standard format so that it is easy to work on  Missing Value----During data collection and entry it is possible that some values are missed or data was not available. Pandas writes `NaN(Not a number)` when it found a missing value. It does not include NaN in any calculation like sum, mean etc. we can detect missing value by using `isnull` function. whenever it found a missing value, return `True` otherwisde `False`.<jupyter_code>df = pd.read_csv('/kaggle/input/california-housing-prices/housing.csv')

print('missing value in every column :', df.isnull().sum())
print('total missing value of all the columns : ', df.isnull().sum().sum())

#show all the rows of missing value 
display(df[df['total_bedrooms'].isnull()])<jupyter_output><empty_output><jupyter_text>Filtering missing values-------when encounter missing value we can either remove those values or fill with suitable value<jupyter_code>print('shape of original dataframe : ', df.shape)

# remove missing rows 
new_df = df.dropna()
print('shape of filtered dataframe :', new_df.shape)

# remove the whole column (bad idea)
new_df = df.dropna(axis=1)
print('shape of filtered dataframe : ', new_df.shape)<jupyter_output><empty_output><jupyter_text>Filling numerical missing values----Removing missing missing value is bad idea, the best way to deal with missing value is filling those missing value with suitable value. The strategy behind filling missing values can vary from problem to problem. In some cases, filling in with the **mean** or **median** value works very well, while in some cases **interpolation** works better. Interpolation is a mathematical technique of constructing new points from a range of points.<jupyter_code>print("Total missing values : ",df.isnull().sum().sum())

#fill value with median value 
value = df['total_bedrooms'].median()
new_df = df['total_bedrooms'].fillna(value)

#fill missing value with interpolation 
new_df = df.interpolate('linear')

print("Total missing values : ",new_df.isnull().sum().sum())

#print(new_df.loc[290, 'total_bedrooms'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09908409658617819, 'fraction_numerical': 0.0016652789342214821, 'mean_word_length': 5.459677419354839, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '593944', 'license': 'MIT', 'n_tokens_mistral': 669, 'n_tokens_neox': 655, 'n_words': 282, 'path': 'Jupyter-Notebook/Intro to Data Science/Intro to Data Science_ Data Cleaning - 01.ipynb', 'repo_name': 'shohan4556/machine-learning-course-notes', 'size': '2402'}