<jupyter_start><jupyter_text>1. NOTES ON OPERATION OF THE NOTEBOOKIn this notebook, we will employed supervised learning (regression model) to predict the prices of house from Bengaluru, India house prices.It is quite important to follow through on the steps to achieve the aim of learning how to build the model.Dataset can be gotten from: https://www.kaggle.com/amitabhajoy/bengaluru-house-price-data  2a. IMPORT DEPENDENCIES AND PACKAGES<jupyter_code>%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
matplotlib.rcParams['figure.figsize'] = (20, 10)<jupyter_output><empty_output><jupyter_text>2b. IMPORT DATASET<jupyter_code>house_df = pd.read_csv(r'C:\Users\User\Documents\Coding\Datasets\Bengaluru_House_Data.csv')
house_df.head()<jupyter_output><empty_output><jupyter_text>3. GENERAL VIEW OF DATASETS<jupyter_code># display the last five rows
house_df.tail()
house_df.groupby('area_type')['area_type'].agg('count')
# display quick stat view of  
house_df.describe()
# display uniqque values of each columns
house_df.nunique()
# display the number of where 'na' is empty
house_df.isnull().sum()<jupyter_output><empty_output><jupyter_text>`GENERAL EXPLORE OF COLUMNS`<jupyter_code># df.drop() will drop columns OR rows,
# dropna() will drop any row with empty row
house_df1 = house_df.drop(['availability', 'society', 'balcony', 'area_type'], axis='columns')
house_df1<jupyter_output><empty_output><jupyter_text>`EXPLORE THE SIZE COLUMN`<jupyter_code># display the 'size' columns and transform the valuesso that are of same unit
house_df1['size'].unique()
# apply the replace extract method
house_df1['room-size'] = house_df1['size'].str.extract('(\d+)', expand=False)
house_df1.head()
# drop the 'size' column
house_df1.drop('size', axis=1, inplace=True)
house_df1
house_df1['room-size'].unique()
house_df1['room-size'] = house_df1['room-size'].apply(pd.to_numeric)
print(house_df1['room-size'].dtype)<jupyter_output>float64<jupyter_text>`EXPLORE THE TOTAL_SQFT COLUMN`<jupyter_code>house_df1['total_sqft'].unique()
# define a function to determine if the value in total_sqft are float?
def is_float(a):
    try:
        float(a)
    except:
        return False
    return True
# display the values that are float
house_df1[house_df1['total_sqft'].apply(is_float)].head()
# dislays the values that are not float
house_df1[~house_df1['total_sqft'].apply(is_float)].head()
# write a fucntion that converts the values of the 'total_sqft' 
# that are not floats into becoming floats

def convert_sqft_to_num(a):
    value = a.split("-")
    if len(value) ==2:
        return(float(value[0]) + float(value[1]))/2
    try:
        return float(a)
    except:
         return None
# test out the function above
print(convert_sqft_to_num('1056'))
print(convert_sqft_to_num('3010 - 3410'))
print(convert_sqft_to_num('34.456Sq. Meter'))
# create a new dataframe to apply the function
house_df2 = house_df1.copy()
house_df2['total_sqft'] = house_df2['total_sqft'].apply(convert_sqft_to_num)
house_df2.head(2)
# preview a rw using the df.loc function
house_df2.loc[30]<jupyter_output><empty_output><jupyter_text>FEATURE ENGINEERING  `Per per Sqaure Foot`<jupyter_code>house_df3 = house_df2.copy()
# create a new column 'price_per_sqft' this is important to define unit price
house_df3['price_per_sqft'] = (house_df3['price']*100000/house_df3['total_sqft'])
house_df3.head()
len(house_df3['location'].unique())
# create a  function to determine the importance of each
# location on the overall location
location_stats = house_df3.groupby('location')['location'].agg('count').sort_values(ascending=False)
location_stats
# check the number of location that has less than ten houses
len(location_stats[location_stats<= 10])
location_stats_less_than_10 = location_stats[location_stats<= 10]
location_stats_less_than_10
len(house_df3['location'].unique())
house_df3['location'] = house_df3['location'].apply(lambda x: 'other_location' if x in location_stats_less_than_10 else x)
len(house_df3['location'].unique())
house_df3.head()<jupyter_output><empty_output><jupyter_text>OUTLIER DETECTIONS AND REMOVALSOutliers if not well taken care of causes instabilty to our model. It may be important to look at extreme values before doing them  `OUTLIER DETECTION : ROOM SIZE`<jupyter_code># lets take the threshold of 300 
house_df4 = house_df3.copy()
# to use a threshold to determine the outliers
house_df4[house_df4['total_sqft']/house_df4['room-size']<300].head()
house_df4.shape
# the neagtion of the outliers are
house_df5 = house_df4[~(house_df4['total_sqft']/house_df4['room-size']<300)]
house_df5.shape<jupyter_output><empty_output><jupyter_text>`REMOVE OUTLIERS : PRICE_PER_SQUARE_FOOT`<jupyter_code># removing outliers for price columns
house_df5['price_per_sqft'].describe()
# a function to remove outliers in the price_per_sqft columns
def ppsqft_outliers(house_df):
    outlier_df = pd.DataFrame()
    for key, place_df in house_df.groupby('location'):
        m = np.mean(place_df['price_per_sqft'])
        std = np.std(place_df['price_per_sqft'])
        reduced_df = place_df[(place_df['price_per_sqft']>(m - std)) & (place_df['price_per_sqft']<(m + std))]
        outlier_df = pd.concat([outlier_df, reduced_df], ignore_index=True)
    return outlier_df


house_df6 = ppsqft_outliers(house_df5)
house_df6.shape<jupyter_output><empty_output><jupyter_text>`REMOVE OUTLIERS: WITH PRICE ANALYSIS`<jupyter_code># create a function to describe the location diff in prices btw 2 & 3 bedroom

def plot_scatter_plot(house_df, location):
    two_bedroom = house_df[(house_df['location'] == location) & (house_df['room-size'] ==2)]
    three_bedroom = house_df[(house_df['location'] == location) & (house_df['room-size'] ==3)]
    plt.scatter(two_bedroom['total_sqft'], two_bedroom['price'], color='orange', label='2-BD', s= 50)
    plt.scatter(three_bedroom['total_sqft'], three_bedroom['price'], marker='+', color='green', label='3-BD', s= 50)
    plt.xlabel('Total Square Feet Area')
    plt.ylabel('price')
    plt.title(location)
    plt.legend()
plot_scatter_plot(house_df6, '<NAME>')
plot_scatter_plot(house_df6, 'Margondanahalli')<jupyter_output><empty_output><jupyter_text>Further Removal of OutliersWe would remove properties where for the same location, the price of (for example) 3-bedroom apartment is less thna the 2-bedroom apartment (with same sq_ft_area). what we will do iis for a given location, we would build a dictionary of stats per room size{    '1':{        'mean': 4000,        'std': 2000,        'count': 30    }    '2':{        'mean': 4500,        'std': 2500,        'count': 20    }}<jupyter_code># function to  remove 2-bedroom apartment whose price is less
# than mean price_per_sqft of 1-bedroom apartment.
def room_size_outliers(house_df):
    excluded_rooms = np.array([])
    for location, location_df in house_df.groupby('location'):
        room_stats = {}
        for room_size, room_size_df in location_df.groupby('room-size'):
            room_stats[room_size] = {
                'mean': np.mean(room_size_df['price_per_sqft']),
                'std': np.std(room_size_df['price_per_sqft']),
                'count': room_size_df.shape[0]
            }
        for room_size, room_size_df in location_df.groupby('room-size'):
            normal_room = room_stats.get(room_size - 1)
            if normal_room and normal_room['count']>5:
                excluded_rooms = np.append(excluded_rooms, room_size_df[room_size_df['price_per_sqft']<(normal_room['mean'])].index.values)
    return house_df.drop(excluded_rooms, axis='index')

house_df7 = room_size_outliers(house_df6)
house_df7.shape
# visulise the new plot after cleaning
plot_scatter_plot(house_df7, '<NAME>')<jupyter_output><empty_output><jupyter_text>`VISUALISE THE OUTCOMES OF THE ABOVE`<jupyter_code>plt.hist(house_df7['price_per_sqft'], rwidth=0.8)
plt.xlabel('Price Per Square Feet')
plt.ylabel('Count')
# explore the bathroom columns before visualising
house_df7['bath'].unique()
house_df7[house_df7['bath']>10]
plt.hist(house_df7['bath'], rwidth =0.75)
plt.xlabel('Number of Bathroom')
plt.ylabel('count')<jupyter_output>C:\Users\User\anaconda3\lib\site-packages\numpy\lib\histograms.py:839: RuntimeWarning: invalid value encountered in greater_equal
  keep = (tmp_a >= first_edge)
C:\Users\User\anaconda3\lib\site-packages\numpy\lib\histograms.py:840: RuntimeWarning: invalid value encountered in less_equal
  keep &= (tmp_a <= last_edge)<jupyter_text>Further visualisation into bathroom per room shows that if the bathroom is more +2 the number of room-size then its is an outlier<jupyter_code>house_df7[house_df7['bath']>house_df7['room-size']+2]
#remove this unnecessary outliers
house_df8 = house_df7[house_df7['bath']<house_df7['room-size']+2]
house_df8.shape
#drop the unecessary data columns
house_df8.drop('price_per_sqft', axis=1, inplace=True)
house_df8.head()
house_df9 = house_df8.copy()
house_df9.head(3)<jupyter_output><empty_output><jupyter_text>MODEL BUILDING   `USE ONE HOT-ENCODINGTO TRANSFORM THE DIFFERENT LOCATIONS`<jupyter_code># use one hot encoding to transform the diffferent locations
house_df10 = pd.get_dummies(house_df9['location'])
house_df10.head()
house_df11 = pd.concat([house_df9, house_df10.drop('other_location', axis=1)], axis=1)
house_df11.head(2)
# drop the location columns from the maindata frame
house_df12 = house_df11.drop('location', axis=1)
house_df12.head(2)<jupyter_output><empty_output><jupyter_text>`MODEL BUIILDING`<jupyter_code># check the shape of thre dataframe
house_df12.shape

X = house_df12.drop('price', axis=1)
Y = house_df12['price']<jupyter_output><empty_output><jupyter_text>'IMPORT THE TRAIN_TEST_SPLIT<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.25, random_state=10)<jupyter_output><empty_output><jupyter_text>`IMPORT THE LINEAR MODEL REGRESSION`<jupyter_code>from sklearn.linear_model import LinearRegression
lr = LinearRegression()
lr.fit(X_train, Y_train)
lr.score(X_test, Y_test)
from sklearn.model_selection import ShuffleSplit
from sklearn.model_selection import cross_val_score

cv = ShuffleSplit(n_splits=5, test_size=0.25, random_state=10)

cross_val_score(LinearRegression(), X, Y, cv=cv)<jupyter_output><empty_output><jupyter_text>`USING GRID SEARCH CV`<jupyter_code>from sklearn.model_selection import GridSearchCV

from sklearn.linear_model import Lasso
from sklearn.tree import DecisionTreeRegressor

def the_best_model_using_gridsearchcv(X,Y):
    algos ={
        'linear_regression': {
            'model': LinearRegression(),
            'params':{
                    'normalize': [True, False]
            }
        },
        
        'lasso':{
            'model': Lasso(),
            'params':{
                'alpha': [1, 2],
                'selection': ['random', 'cyclic']
            }
        },
        
        'decision_tree': {
            'model': DecisionTreeRegressor(),
            'params':{
                'criterion': ['mse', 'friedman_mse'],
                'splitter':  ['best', 'random']
            }
        }
    }
    
    
    scores = []
    cv = ShuffleSplit(n_splits=5, test_size=0.25, random_state=10)
    for alogs, config in algos.items():
        grader = GridSearchCV(config['model'], config['params'], cv=cv, return_train_score=False)
        grader.fit(X,Y)
        scores.append({
            'model': algos,
            'best_score': grader.best_score_,
            'best_parmas': grader.best_params_
        })
    return pd.DataFrame(scores, columns=['model', 'best_socre', 'best_params'])

the_best_model_using_gridsearchcv(X, Y)
X.columns
np.where(X.columns=='1st Block Jayanagar')[0][0]
# this function 
def predict_price(location, sqft, bath, room_size):
    loc_index = np.where(X.columns==location)[0][0]
    
    x = np.zeros(len(X.columns))
    x[0] = sqft
    x[1] = bath
    x[2] = room_size
    if loc_index >=0:
        x[loc_index] = 1
        
    return lr.predict([x])[0]
predict_price('1st Phase JP Nagar', 1000, 2, 2)
predict_price('1st Phase JP Nagar', 2000, 2, 2)
predict_price('Indira Nagar', 1000, 2, 2)
predict_price('Indira Nagar', 2000, 2, 2)<jupyter_output><empty_output><jupyter_text>`EXPORT MODEL TO PICKLE`<jupyter_code>import pickle
with open('bengluru_home_price_model.pickle', 'wb') as f:
    pickle.dump(lr, f)<jupyter_output><empty_output><jupyter_text>`EXPORT COLUMNS VALUES TO JSON`<jupyter_code>import json
columns = {
    'data_columns': [col.lower() for col in X.columns]
}
with open('bengluru_columns.json', 'w') as f:
    f.write(json.dumps(columns))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12400785854616896, 'fraction_numerical': 0.02200392927308448, 'mean_word_length': 5.118269230769231, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 87, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '332822', 'license': 'zlib-acknowledgement', 'n_tokens_mistral': 4681, 'n_tokens_neox': 4459, 'n_words': 966, 'path': 'DataScience_complete_LINEAR_REGRESSION/model_package/ds_linear_regression_house_prediction.ipynb', 'repo_name': 'chisomloius/small-project', 'size': '12737'}