<jupyter_start><jupyter_text><jupyter_code>import pandas as pd
df = pd.read_csv('https://raw.githubusercontent.com/arindamdeyofficial/houseprediction/main/housing.csv')

df.shape
df.head()
df.columns
df.groupby('ocean_proximity').count()
df.isna().any()
df.count()
df[df.columns[df.isna().any()]]
tbOut = df[df.isnull().any(axis=1)]
tbOut
len(tbOut.index)/len(df.index)*100<jupyter_output><empty_output><jupyter_text>So total_bedrooms 207 rows are null values and less than 5%. Then we can't. reject these records. Then we have to impute.<jupyter_code>df.groupby('total_bedrooms').head()<jupyter_output><empty_output><jupyter_text>1923 distinct values of total_bedrooms. So this is continuous value. The impute with mean / median.<jupyter_code>df['total_bedrooms'].fillna(df['total_bedrooms'].mean(), inplace = True)
df[df.isnull().any(axis=1)]<jupyter_output><empty_output><jupyter_text>Now need to do normalisation.<jupyter_code>dNearbay = df.loc[df['ocean_proximity'] == 'NEAR BAY']
dNearbay
dNearbay.describe()<jupyter_output><empty_output><jupyter_text>Boxplot to understand data distribution<jupyter_code>import matplotlib.pyplot as plt
#fig = plt.figure(figsize =(10, 7))
#ax = fig.add_axes([0, 0, 1, 1])
plt.boxplot(dNearbay['median_house_value'])
dNearbay.loc[dNearbay['median_house_value'] >= 500001]<jupyter_output><empty_output><jupyter_text>So there are no outliers for median_house_value as max value has 194 records<jupyter_code>plt.boxplot(dNearbay['housing_median_age'])
dNearbay['housing_median_age'].describe()
dNearbay.loc[dNearbay['housing_median_age'] < 10].count()
dNearbay.plot.scatter('housing_median_age', 'median_house_value')
dNearbay.loc[dNearbay['median_house_value'] < 80000].plot.scatter('housing_median_age', 'median_house_value')
dNearbay.loc[dNearbay['median_house_value'] < 80000].count()/dNearbay['median_house_value'].count()*100<jupyter_output><empty_output><jupyter_text>Now we have identified these less than 80000 as outliers need to remove<jupyter_code>dNearbay.loc[dNearbay['median_house_value'] < 80000].index
dNearbay.drop(index_names, inplace = True)
dNearbay.loc[dNearbay['median_house_value'] < 80000]['longitude'].unique()
plt.boxplot(dNearbay['total_rooms'])
dNearbay.loc[dNearbay['total_rooms'] > 7500].count()/dNearbay.count()*100
plt.boxplot(dNearbay['median_income'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1488095238095238, 'fraction_numerical': 0.02806122448979592, 'mean_word_length': 9.893518518518519, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 31, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '249536', 'license': 'MIT', 'n_tokens_mistral': 998, 'n_tokens_neox': 868, 'n_words': 127, 'path': 'houseprediction.ipynb', 'repo_name': 'arindamdeyofficial/houseprediction', 'size': '2352'}