<jupyter_start><jupyter_text>**Drop features that are not required to build our model**<jupyter_code>df2 = df1.drop(['area_type','society','balcony','availability'],axis='columns')
df2.shape
#Data Cleaning: Handle NA values
df2.isnull().sum()
df2.shape
df3 = df2.dropna()
df3.isnull().sum()
df3.shape<jupyter_output><empty_output><jupyter_text>**Add new feature(integer) for bhk (Bedrooms Hall Kitchen)**<jupyter_code>#Feature Engineering
df3['bhk'] = df3['size'].apply(lambda x: int(x.split(' ')[0]))
df3.bhk.unique()<jupyter_output>C:\Users\Nitish\Anaconda3\lib\site-packages\ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy<jupyter_text>**Explore total_sqft feature**<jupyter_code>def is_float(x):
    try:
        float(x)
    except:
        return False
    return True
2+3
df3[~df3['total_sqft'].apply(is_float)].head(10)<jupyter_output><empty_output><jupyter_text>**Above shows that total_sqft can be a range (e.g. 2100-2850). For such case we can just take average of min and max value in the range. There are other cases such as 34.46Sq. Meter which one can convert to square ft using unit conversion. I am going to just drop such corner cases to keep things simple**<jupyter_code>def convert_sqft_to_num(x):
    tokens = x.split('-')
    if len(tokens) == 2:
        return (float(tokens[0])+float(tokens[1]))/2
    try:
        return float(x)
    except:
        return None   
df4 = df3.copy()
df4.total_sqft = df4.total_sqft.apply(convert_sqft_to_num)
df4 = df4[df4.total_sqft.notnull()]
df4.head(2)<jupyter_output><empty_output><jupyter_text>**For below row, it shows total_sqft as 2475 which is an average of the range 2100-2850**<jupyter_code>df4.loc[30]
(2100+2850)/2<jupyter_output><empty_output><jupyter_text>**Add new feature called price per square feet**<jupyter_code>df5 = df4.copy()
df5['price_per_sqft'] = df5['price']*100000/df5['total_sqft']
df5.head()
df5_stats = df5['price_per_sqft'].describe()
df5_stats
df5.to_csv("bhp.csv",index=False)<jupyter_output><empty_output><jupyter_text>**Examine locations which is a categorical variable. We need to apply dimensionality reduction technique here to reduce number of locations**<jupyter_code>df5.location = df5.location.apply(lambda x: x.strip())
location_stats = df5['location'].value_counts(ascending=False)
location_stats
location_stats.values.sum()
len(location_stats[location_stats>10])
len(location_stats)
len(location_stats[location_stats<=10])

#Dimensionality Reduction<jupyter_output><empty_output><jupyter_text>**Any location having less than 10 data points should be tagged as "other" location. This way number of categories can be reduced by huge amount. Later on when we do one hot encoding, it will help us with having fewer dummy columns**<jupyter_code>location_stats_less_than_10 = location_stats[location_stats<=10]
location_stats_less_than_10
len(df5.location.unique())
df5.location = df5.location.apply(lambda x: 'other' if x in location_stats_less_than_10 else x)
len(df5.location.unique())
df5.head(10)<jupyter_output><empty_output><jupyter_text>**Outlier Removal Using Business Logic****.                                                                                                             .****As a data scientist when you have a conversation with your business manager (who has expertise in real estate), he will tell you that normally square ft per bedroom is 300 (i.e. 2 bhk apartment is minimum 600 sqft. If you have for example 400 sqft apartment with 2 bhk than that seems suspicious and can be removed as an outlier. We will remove such outliers by keeping our minimum thresold per bhk to be 300 sqft**<jupyter_code>df5[df5.total_sqft/df5.bhk<300].head()<jupyter_output><empty_output><jupyter_text>**Check above data points. We have 6 bhk apartment with 1020 sqft. Another one is 8 bhk and total sqft is 600. These are clear data errors that can be removed safely**<jupyter_code>df5.shape
df6 = df5[~(df5.total_sqft/df5.bhk<300)]
df6.shape<jupyter_output><empty_output><jupyter_text>Outlier Removal Using Standard Deviation and Mean<jupyter_code>df6.price_per_sqft.describe()<jupyter_output><empty_output><jupyter_text>**Here we find that min price per sqft is 267 rs/sqft whereas max is 12000000, this shows a wide variation in property prices. We should remove outliers per location using mean and one standard deviation**<jupyter_code>def remove_pps_outliers(df):
    df_out = pd.DataFrame()
    for key, subdf in df.groupby('location'):
        m = np.mean(subdf.price_per_sqft)
        st = np.std(subdf.price_per_sqft)
        reduced_df = subdf[(subdf.price_per_sqft>(m-st)) & (subdf.price_per_sqft<=(m+st))]
        df_out = pd.concat([df_out,reduced_df],ignore_index=True)
    return df_out
df7 = remove_pps_outliers(df6)
df7.shape<jupyter_output><empty_output><jupyter_text>**Let's check if for a given location how does the 2 BHK and 3 BHK property prices look like**<jupyter_code>Let's check if for a given location how does the 2 BHK and 3 BHK property prices look like

def plot_scatter_chart(df,location):
    bhk2 = df[(df.location==location) & (df.bhk==2)]
    bhk3 = df[(df.location==location) & (df.bhk==3)]
    matplotlib.rcParams['figure.figsize'] = (15,10)
    plt.scatter(bhk2.total_sqft,bhk2.price,color='blue',label='2 BHK', s=50)
    plt.scatter(bhk3.total_sqft,bhk3.price,marker='+', color='green',label='3 BHK', s=50)
    plt.xlabel("Total Square Feet Area")
    plt.ylabel("Price (Lakh Indian Rupees)")
    plt.title(location)
    plt.legend()
    
plot_scatter_chart(df7,"<NAME>")
plot_scatter_chart(df7,"Hebbal")<jupyter_output><empty_output><jupyter_text>**We should also remove properties where for same location, the price of (for example) 3 bedroom apartment is less than 2 bedroom apartment (with same square ft area). What we will do is for a given location, we will build a dictionary of stats per bhk, i.e.**```{    '1' : {        'mean': 4000,        'std: 2000,        'count': 34    },    '2' : {        'mean': 4300,        'std: 2300,        'count': 22    },    }```**Now we can remove those 2 BHK apartments whose price_per_sqft is less than mean price_per_sqft of 1 BHK apartment**<jupyter_code>def remove_bhk_outliers(df):
    exclude_indices = np.array([])
    for location, location_df in df.groupby('location'):
        bhk_stats = {}
        for bhk, bhk_df in location_df.groupby('bhk'):
            bhk_stats[bhk] = {
                'mean': np.mean(bhk_df.price_per_sqft),
                'std': np.std(bhk_df.price_per_sqft),
                'count': bhk_df.shape[0]
            }
        for bhk, bhk_df in location_df.groupby('bhk'):
            stats = bhk_stats.get(bhk-1)
            if stats and stats['count']>5:
                exclude_indices = np.append(exclude_indices, bhk_df[bhk_df.price_per_sqft<(stats['mean'])].index.values)
    return df.drop(exclude_indices,axis='index')
df8 = remove_bhk_outliers(df7)
# df8 = df7.copy()
df8.shape<jupyter_output><empty_output><jupyter_text>**Plot same scatter chart again to visualize price_per_sqft for 2 BHK and 3 BHK properties**<jupyter_code>plot_scatter_chart(df8,"<NAME>")
plot_scatter_chart(df8,"Hebbal")
import matplotlib
matplotlib.rcParams["figure.figsize"] = (20,10)
plt.hist(df8.price_per_sqft,rwidth=0.8)
plt.xlabel("Price Per Square Feet")
plt.ylabel("Count")<jupyter_output><empty_output><jupyter_text>Outlier Removal Using Bathrooms Feature<jupyter_code>df8.bath.unique()
plt.hist(df8.bath,rwidth=0.8)
plt.xlabel("Number of bathrooms")
plt.ylabel("Count")
df8[df8.bath>10]<jupyter_output><empty_output><jupyter_text>**It is unusual to have 2 more bathrooms than number of bedrooms in a home**<jupyter_code>df8[df8.bath>df8.bhk+2]<jupyter_output><empty_output><jupyter_text>**Again the business manager has a conversation with you (i.e. a data scientist) that if you have 4 bedroom home and even if you have bathroom in all 4 rooms plus one guest bathroom, you will have total bath = total bed + 1 max. Anything above that is an outlier or a data error and can be removed**<jupyter_code>df9 = df8[df8.bath<df8.bhk+2]
df9.shape
df9.head(2)
df10 = df9.drop(['size','price_per_sqft'],axis='columns')
df10.head(3)<jupyter_output><empty_output><jupyter_text>Use One Hot Encoding For Location<jupyter_code>#pandas dummies method
dummies = pd.get_dummies(df10.location)
dummies.head(10)
df11 = pd.concat([df10,dummies.drop('other',axis='columns')],axis='columns')
df11.head()
df12 = df11.drop('location',axis='columns')
df12.head(2)<jupyter_output><empty_output><jupyter_text>Build a Model Now...<jupyter_code>df12.shape
X = df12.drop(['price'],axis='columns')
X.head(3)
X.shape
Y = df12.price
Y.head()
len(Y)
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X,Y,test_size=0.2,random_state=10)
from sklearn.linear_model import LinearRegression
lr_clf = LinearRegression()
lr_clf.fit(X_train,Y_train)

lr_clf.score(X_test,Y_test)<jupyter_output><empty_output><jupyter_text>Use K Fold cross validation to measure accuracy of our LinearRegression model<jupyter_code>from sklearn.model_selection import ShuffleSplit
from sklearn.model_selection import cross_val_score
cv = ShuffleSplit(n_splits=5, test_size=0.2, random_state=0)
cross_val_score(LinearRegression(), X, Y, cv=cv)<jupyter_output><empty_output><jupyter_text>**We can see that in 5 iterations we get a score above 80% all the time. This is pretty good but we want to test few other algorithms for regression to see if we can get even better score. We will use GridSearchCV for this purpose** Find best model using GridSearchCV<jupyter_code>from sklearn.model_selection import GridSearchCV

from sklearn.linear_model import Lasso
from sklearn.tree import DecisionTreeRegressor


def find_best_model_using_gridsearchcv(X,y):
    algos = {
        'linear_regression' : {
            'model': LinearRegression(),
            'params': {
                'normalize': [True, False]
            }
        },
        'lasso': {
            'model': Lasso(),
            'params': {
                'alpha': [1,2],
                'selection': ['random', 'cyclic']
            }
        },
        'decision_tree': {
            'model': DecisionTreeRegressor(),
            'params': {
                'criterion' : ['mse','friedman_mse'],
                'splitter': ['best','random']
            }
        }
    }
    scores = []
    cv = ShuffleSplit(n_splits=5, test_size=0.2, random_state=0)
    for algo_name, config in algos.items():
        gs =  GridSearchCV(config['model'], config['params'], cv=cv, return_train_score=False)
        gs.fit(X,y)
        scores.append({
            'model': algo_name,
            'best_score': gs.best_score_,
            'best_params': gs.best_params_
        })

    return pd.DataFrame(scores,columns=['model','best_score','best_params'])


find_best_model_using_gridsearchcv(X,Y)<jupyter_output><empty_output><jupyter_text>**Based on above results we can say that LinearRegression gives the best score. Hence we will use that.**<jupyter_code>X.columns<jupyter_output><empty_output><jupyter_text>Test the model for few properties<jupyter_code>def predict_price(location,sqft,bath,bhk):    
    loc_index = np.where(X.columns==location)[0][0]

    x = np.zeros(len(X.columns))
    x[0] = sqft
    x[1] = bath
    x[2] = bhk
    if loc_index >= 0:
        x[loc_index] = 1

    return lr_clf.predict([x])[0]
predict_price('1st Phase JP Nagar',1000, 2, 2)
predict_price('1st Phase JP Nagar',1000, 3, 3)
predict_price('Indira Nagar',1000, 2, 2)
predict_price('Indira Nagar',1000, 3, 3)<jupyter_output><empty_output><jupyter_text>Export the tested model to a pickle file<jupyter_code>import pickle
with open('banglore_home_prices_model.pickle','wb') as f:
    pickle.dump(lr_clf,f)<jupyter_output><empty_output><jupyter_text>Export location and column information to a file that will be useful later on in our prediction application<jupyter_code>import json
columns = {
    'data_columns' : [col.upper() for col in X.columns]
}
with open("columns.json","w") as f:
    f.write(json.dumps(columns))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11996766370250607, 'fraction_numerical': 0.027243330638641875, 'mean_word_length': 4.567506750675068, 'pattern_counts': {'":': 0, '<': 113, '<?xml version=': 0, '>': 112, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '63018', 'license': 'MIT', 'n_tokens_mistral': 4512, 'n_tokens_neox': 4242, 'n_words': 1034, 'path': 'House-Price-Prediction/code/Another_file/banglore_home_prices_final.ipynb', 'repo_name': 'nitish705702/House-Price-Prediction', 'size': '12382'}