<jupyter_start><jupyter_text>Data Science Regression Project: Predicting Home Prices in Banglore Dataset source >>> https://www.kaggle.com/amitabhajoy/bengaluru-house-price-data  Importing Libraries<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sns
from matplotlib import pyplot as plt
%matplotlib inline
import matplotlib 
matplotlib.rcParams["figure.figsize"] = (15,8) # setting figsize<jupyter_output><empty_output><jupyter_text>Loading Dataset<jupyter_code>df = pd.read_csv("blr_house_dataset.csv")
df.head(10)<jupyter_output><empty_output><jupyter_text>Checking the count of data and dtypes of features<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 13320 entries, 0 to 13319
Data columns (total 9 columns):
 #   Column        Non-Null Count  Dtype  
---  ------        --------------  -----  
 0   area_type     13320 non-null  object 
 1   availability  13320 non-null  object 
 2   location      13319 non-null  object 
 3   size          13304 non-null  object 
 4   society       7818 non-null   object 
 5   total_sqft    13320 non-null  object 
 6   bath          13247 non-null  float64
 7   balcony       12711 non-null  float64
 8   price         13320 non-null  float64
dtypes: float64(3), object(6)
memory usage: 936.7+ KB<jupyter_text>Statistical Description of Data<jupyter_code>df['price'].describe()<jupyter_output><empty_output><jupyter_text>Dimensions of Original Dataset<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>Columns of the datasets<jupyter_code>df.columns<jupyter_output><empty_output><jupyter_text>Checking unique types of area<jupyter_code>df['area_type'].unique()<jupyter_output><empty_output><jupyter_text>Count of each type of area of house<jupyter_code>df['area_type'].value_counts()<jupyter_output><empty_output><jupyter_text>Drop features that are not required or not useful to build our model<jupyter_code>df_drop_useless = df.drop(['area_type','society','balcony','availability'],axis='columns')
df_drop_useless.shape<jupyter_output><empty_output><jupyter_text>Handling Null Values<jupyter_code>df_drop_useless.isnull().sum()<jupyter_output><empty_output><jupyter_text>Dimensions of Dataset after dropping some features<jupyter_code>df_drop_useless.shape<jupyter_output><empty_output><jupyter_text>Dropping Null Values<jupyter_code>df_drop_nan = df_drop_useless.dropna()
df_drop_nan.isnull().sum()<jupyter_output><empty_output><jupyter_text>Dimensions of Dataset after dropping some features and dropping null values<jupyter_code>df_drop_nan.shape<jupyter_output><empty_output><jupyter_text>Knowing the no. of records in each feature and dtypes of features<jupyter_code>df_drop_nan.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 13246 entries, 0 to 13319
Data columns (total 5 columns):
 #   Column      Non-Null Count  Dtype  
---  ------      --------------  -----  
 0   location    13246 non-null  object 
 1   size        13246 non-null  object 
 2   total_sqft  13246 non-null  object 
 3   bath        13246 non-null  float64
 4   price       13246 non-null  float64
dtypes: float64(2), object(3)
memory usage: 620.9+ KB<jupyter_text>Feature Engineering  Add new feature(integer) for bhk (Bedrooms Hall Kitchen)<jupyter_code>df_drop_nan['bhk'] = df_drop_nan['size'].apply(lambda x: int(x.split(' ')[0])) # accessing first digit 
df_drop_nan['bhk'].unique()<jupyter_output><ipython-input-240-51d5d32febc0>:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_drop_nan['bhk'] = df_drop_nan['size'].apply(lambda x: int(x.split(' ')[0])) # accessing first digit<jupyter_text>Converting and exploring total_sqft feature<jupyter_code>def is_float(n):
    '''to check that the total_sqft is float'''
    try:
        float(n)
    except:
        return False
    return True
df_drop_nan[~df_drop_nan['total_sqft'].apply(is_float)].head()<jupyter_output><empty_output><jupyter_text>The total_sqft  is a object and not numbers; actually it is the range of total_sqft within those two values as min and max value. Taking average of both the values will give approximately equivalent value.   As simple interpretation, we will treat this feature first and will try to get the float values at the place of that object values of current feature.<jupyter_code>def sqft_to_float(n):
    area = n.split('-')
    if len(area) == 2:
        return (float(area[0])+float(area[1]))/2
    try:
        return float(n)
    except:
        return None   
df_clean_copy = df_drop_nan.copy()
df_clean_copy['total_sqft'] = df_clean_copy['total_sqft'].apply(sqft_to_float)
df_clean_copy = df_clean_copy[df_clean_copy['total_sqft'].notnull()]
df_clean_copy.head(5)<jupyter_output><empty_output><jupyter_text>Feature Engineering  Add new feature called price per square feet<jupyter_code>df_clean = df_clean_copy.copy()
df_clean['price_per_sqft'] = df_clean['price']*100000/df_clean['total_sqft']
df_clean.head()
df_clean_describe = df_clean['price_per_sqft'].describe()
df_clean_describe
df_clean.to_csv("fea_eng_blr_data.csv",index=False)<jupyter_output><empty_output><jupyter_text>Dimentionality Reduction  Getting count of number of houses at each location and using dimensionality reduction technique to minimise number of locations<jupyter_code>df_clean['location'] = df_clean['location'].apply(lambda x: x.strip())
location_count = df_clean['location'].value_counts(ascending=False)
location_count<jupyter_output><empty_output><jupyter_text>Count less than 10 for any location will be labeled as "discard" location. The toal number of categories can be reduced at greater extent.<jupyter_code>location_less_10 = location_count[location_count<=10]
location_less_10
df_clean['location'] = df_clean['location'].apply(lambda x: 'discard' if x in location_less_10 else x)
len(df_clean.location.unique())
df_clean.head(10)<jupyter_output><empty_output><jupyter_text>Outlier Removal Using Business Logic  business manager or domain expert, will explain that generally square ft per bedroom is 300. We will remove outliers by keeping our minimum thresold per bhk to be 300 sqft<jupyter_code>df_clean[df_clean['total_sqft']/df_clean['bhk']<300].head()<jupyter_output><empty_output><jupyter_text>The above metrics tell us that, some of the total_sqft and size is not practically matching. We will clearly maintain the data will remove the errors in it.<jupyter_code>df_clean.shape<jupyter_output><empty_output><jupyter_text>Let's the dimension of data after setting threshold per bhk to be 300 sqft<jupyter_code>df_min_thresh = df_clean[~(df_clean['total_sqft']/df_clean['bhk']<300)]
df_min_thresh<jupyter_output><empty_output><jupyter_text>Outlier Removal Using Standard Deviation and Mean<jupyter_code>df_min_thresh['price_per_sqft'].describe()<jupyter_output><empty_output><jupyter_text>As our data is widely spread, we can take look on minimum price per sqft is 267 rs/sqft and comparing to that maximum is 12000000 rs/sqft. Using mean and one standard deviation we will remove ouliers per location.<jupyter_code>def treat_outliers(df):
    df_new = pd.DataFrame()
    for key, df_group in df.groupby('location'):
        std = np.std(df_group['price_per_sqft'])
        mu = np.mean(df_group['price_per_sqft'])
        df_optimized = df_group[(df_group['price_per_sqft'] > (mu-std)) & (df_group['price_per_sqft'] <= (mu+std))]
        df_new = pd.concat([df_new,df_optimized], ignore_index=True)
    return df_new
df_plot_bhk = treat_outliers(df_min_thresh)
df_plot_bhk.shape<jupyter_output><empty_output><jupyter_text>**Let's check if for a given location how does the 2 BHK and 3 BHK property prices look like**<jupyter_code>def plot_scatter_chart(df,location):
    df_2_bhk = df[(df['location']==location) & (df['bhk']==2)]
    df_3_bhk = df[(df['location']==location) & (df['bhk']==3)]
    matplotlib.rcParams['figure.figsize'] = (20,10)
    plt.scatter(df_2_bhk['total_sqft'],df_2_bhk['price'],color='magenta',label='2 BHK', s=50)
    plt.scatter(df_3_bhk['total_sqft'],df_3_bhk['price'], color='seagreen',label='3 BHK', s=50)
    plt.xlabel("Total Square Feet Area")
    plt.ylabel("Price (Lakh Indian Rupees)")
    plt.title(location)
    plt.legend()
plot_scatter_chart(df_plot_bhk,"Thanisandra")
plot_scatter_chart(df_plot_bhk,"Sarjapur  Road")<jupyter_output><empty_output><jupyter_text>Some data points in above visualizations says that, the price of 3 BHK is less than 2 BHK with around equivalent sqft area. We should remove such data points as for location.nWhat we will do is for a given location, we will build a dictionary of stats per bhk, i.e. Also, Now we can remove those 2 BHK apartments whose price_per_sqft is less than mean price_per_sqft of 1 BHK apartment**```{'1' : {'mean': 4000,'std: 2000,'count': 34},'2' : {'mean': 4300,'std: 2300,'count': 22},    }```<jupyter_code>def treat_bhk_error(df):
    index_to_ignore = np.array([])
    for location, location_df in df.groupby('location'):
        bhk_collection = {}
        for bhk, df_bhk in location_df.groupby('bhk'):
            bhk_collection[bhk] = {
                'mean': np.mean(df_bhk['price_per_sqft']),
                'std': np.std(df_bhk['price_per_sqft']),
                'count': df_bhk.shape[0]
            }
        for bhk, df_bhk in location_df.groupby('bhk'):
            collection = bhk_collection.get(bhk-1)
            if collection and collection['count']>5:
                index_to_ignore = np.append(index_to_ignore, df_bhk[df_bhk['price_per_sqft']<(collection['mean'])].index.values)
    return df.drop(index_to_ignore,axis='index')
df_bhk_error_removed = treat_bhk_error(df_plot_bhk)
df8 = df_bhk_error_removed.copy()
df8.shape<jupyter_output><empty_output><jupyter_text>**Plot same scatter chart again to visualize price_per_sqft for 2 BHK and 3 BHK properties**<jupyter_code>plot_scatter_chart(df_bhk_error_removed,"Thanisandra")
plt.show()
plot_scatter_chart(df_bhk_error_removed,"Sarjapur  Road")
plt.show()
import matplotlib
matplotlib.rcParams["figure.figsize"] = (15,8)
plt.hist(df_bhk_error_removed['price_per_sqft'],rwidth=0.8)
plt.xlabel("Price Per Square Feet")
plt.ylabel("Count")
plt.show()<jupyter_output><empty_output><jupyter_text>Outlier Removal Using Bathrooms Feature<jupyter_code>df_bhk_error_removed['bath'].unique()<jupyter_output><empty_output><jupyter_text>the above array has wide spread of values of bathroom feature. We will first look at the spread of values and existence of them in the dataset and then we will decide that which values to be skipped to get the optimised dataset.<jupyter_code>sns.distplot(df_bhk_error_removed['bath'],kde=False,color='seagreen', bins=20)
plt.xlabel("Number of bathrooms")
plt.ylabel("Count")<jupyter_output><empty_output><jupyter_text>After observing the histogram, we can remove the records having bathroom feature more than 10.<jupyter_code>df_bhk_error_removed[df_bhk_error_removed['bath']>10]<jupyter_output><empty_output><jupyter_text>It is unusual to have 2 more bathrooms than number of bedrooms in a home<jupyter_code>df_bhk_error_removed[df_bhk_error_removed['bath']>df_bhk_error_removed['bhk']+2]<jupyter_output><empty_output><jupyter_text>Again the business manager told that if you have 4 bedroom home and even if you have bathroom in all 4 rooms plus one guest bathroom, you will have total bath = total bed + 1 max. Anything above that is an outlier or a data error and can be removed<jupyter_code>df_optimised_data = df_bhk_error_removed[df_bhk_error_removed['bath']<df_bhk_error_removed['bhk']+2]
df_optimised_data.shape
df_optimised_data.head(2)
df_useful_features = df_optimised_data.drop(['size','price_per_sqft'], axis='columns')
df_useful_features.head(5)<jupyter_output><empty_output><jupyter_text>One Hot Encoding For Location  It will help us with having fewer dummy columns<jupyter_code>dummies = pd.get_dummies(df_useful_features.location)
dummies.head(10)
df_integrated_with_location = pd.concat([df_useful_features,dummies.drop('discard',axis='columns')],axis='columns')
df_integrated_with_location.head()
df_integrated = df_integrated_with_location.drop('location',axis='columns')
df_integrated.head(15)<jupyter_output><empty_output><jupyter_text>Model Building<jupyter_code>df_integrated.shape
X = df_integrated.drop(['price'],axis='columns')
X.head()
X.shape
y = df_integrated.price
y.head()
len(y)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2,random_state=10)
from sklearn.linear_model import LinearRegression
lr_clf = LinearRegression()
lr_clf.fit(X_train,y_train)
lr_clf.score(X_test,y_test)<jupyter_output><empty_output><jupyter_text>Cross Validating (K Fold) to check the accuracy of Linear Regression Model<jupyter_code>from sklearn.model_selection import ShuffleSplit
from sklearn.model_selection import cross_val_score

cv = ShuffleSplit(n_splits=5, test_size=0.2, random_state=0)

cross_val_score(LinearRegression(), X, y, cv=cv)<jupyter_output><empty_output><jupyter_text>We got a score above 80% in all the iterations. This result is good and even some regression models get more accuracy than this.  Quantify which model is best using GridSearchCV<jupyter_code>from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeRegressor

def best_gridsearchcv(X,y):
    '''The function for finding which model is best using GridSearchCV'''
    models = {'linear_regression' : {'model': LinearRegression(),'params': {'normalize': [True, False]}},
        'decision_tree': {'model': DecisionTreeRegressor(),'params': {'criterion' : ['mse','friedman_mse'],
                                                                      'splitter': ['best','random']}}
             }
    
    result = []
    
    coeff = ShuffleSplit(n_splits=5, test_size=0.2, random_state=0)
    for model_name, config in models.items():
        GridSea =GridSearchCV(config['model'],  config['params'],cv=coeff, return_train_score=False)
        GridSea.fit(X,y)
        result.append({'model_used': model_name,'result_score': GridSea.best_score_,'final_best_params': GridSea.best_params_})

    return pd.DataFrame(result,columns=['model_used','result_score','final_best_params'])

best_gridsearchcv(X,y)<jupyter_output><empty_output><jupyter_text>The above metrics tells us that the LinearRegression ahs the best score amongst two, with compared with DecisionTree. Hence, We will use LinearRegression   Testing the model<jupyter_code>def prediction(location,sqft,bath,bhk):    
    loc_index = np.where(X.columns==location)[0][0]

    x = np.zeros(len(X.columns))
    x[0] = sqft
    x[1] = bath
    x[2] = bhk
    if loc_index >= 0:
        x[loc_index] = 1

    return lr_clf.predict([x])[0]
prediction('Vittasandra',1000, 2, 2)
prediction("Sarjapur  Road",1000, 3, 3)
prediction('2nd Stage Nagarbhavi',1000, 2, 3)
prediction("Vijayanagar",1000, 2, 3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11208660637665853, 'fraction_numerical': 0.021387550333355338, 'mean_word_length': 5.81205035971223, 'pattern_counts': {'":': 0, '<': 159, '<?xml version=': 0, '>': 161, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '38', 'dir': 'jupyter-structured-clean-dedup', 'id': '438356', 'license': 'MIT', 'n_tokens_mistral': 5468, 'n_tokens_neox': 5154, 'n_words': 1159, 'path': 'Bangalore House Price Prediction - Regression Project.ipynb', 'repo_name': 'Sarthak-GitHub/Regression-Project-Bangalore-House-Price-Prediction', 'size': '15149'}