<jupyter_start><jupyter_text>Regression ChallengePredicting the selling price of a residential property depends on a number of factors, including the property age, availability of local amenities, and location.In this challenge, you will use a dataset of real estate sales transactions to predict the price-per-unit of a property based on its features. The price-per-unit in this data is based on a unit measurement of 3.3 square meters.> **Citation**: The data used in this exercise originates from the following study:>> *<NAME>., & <NAME>. (2018). Building real estate valuation models with comparative approach through case-based reasoning. Applied Soft Computing, 65, 260-271.*>> It was obtained from the UCI dataset repository (<NAME>. and <NAME>. (2019). [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml). Irvine, CA: University of California, School of Information and Computer Science). Review the dataRun the following cell to load the data and view the first few rows.<jupyter_code>import pandas as pd

# load the training dataset
data = pd.read_csv('data/real_estate.csv')
data.head()<jupyter_output><empty_output><jupyter_text>The data consists of the following variables:- **transaction_date** - the transaction date (for example, 2013.250=2013 March, 2013.500=2013 June, etc.)- **house_age** - the house age (in years)- **transit_distance** - the distance to the nearest light rail station (in meters)- **local_convenience_stores** - the number of convenience stores within walking distance- **latitude** - the geographic coordinate, latitude- **longitude** - the geographic coordinate, longitude- **price_per_unit** house price of unit area (3.3 square meters) Train a Regression ModelYour challenge is to explore and prepare the data, identify predictive features that will help predict the **price_per_unit** label, and train a regression model that achieves the lowest Root Mean Square Error (RMSE) you can achieve (which must be less than **7**) when evaluated against a test subset of data.Add markdown and code cells as required to create your solution.> **Note**: There is no single "correct" solution. A sample solution is provided in [02 - Real Estate Regression Solution.ipynb](02%20-%20Real%20Estate%20Regression%20Solution.ipynb).  Dealing with Null values<jupyter_code>data.isnull().sum()
# So no null values
# checking outliers 
import matplotlib.pyplot as plt 
data.hist(figsize=(10,8))
plt.show()
data.describe()
# removing the outliers 
# removing price outliers 
data= data[data.price_per_unit> data.price_per_unit.quantile(0.01)]
data= data[data.price_per_unit< data.price_per_unit.quantile(0.99)]
data.price_per_unit.hist()
# boxplot to detect outliers and understanding the data 
import seaborn as sns
for col in data.columns:
    sns.boxplot(data=data,x=data[col])
    plt.show()
data.price_per_unit.describe(),data.price_per_unit.quantile(0.01)
sns.heatmap(data.corr(),annot=True)
for col in data.columns[:-1]:
    plt.figure(figsize=(10,6))
    sns.scatterplot(data=data,x=data.columns[-1],y=col)
    plt.show()

# as we can see that transaction date has very less correlation with label so droping it 
data=data.drop(columns=['transaction_date'])<jupyter_output><empty_output><jupyter_text>Use the Trained ModelSave your trained model, and then use it to predict the price-per-unit for the following real estate transactions:| transaction_date | house_age | transit_distance | local_convenience_stores | latitude | longitude || ---------------- | --------- | ---------------- | ------------------------ | -------- | --------- ||2013.167|16.2|289.3248|5|24.98203|121.54348||2013.000|13.6|4082.015|0|24.94155|121.50381|<jupyter_code># Your code to use the trained model
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestRegressor
from sklearn.compose import ColumnTransformer
import numpy as np
x_train,x_test,y_train,y_test= train_test_split(data[data.columns[:-1]].values,data[data.columns[-1]].values,test_size=0.30)
# y_train= y_train.reshape((y_train.shape[0],1))
# y_test= y_test.reshape((y_test.shape[0],1))
x_train.shape,x_test.shape,y_train.shape,y_test.shape
pipeline1= Pipeline(steps=[('standscaler', StandardScaler()),
                          ('Regressor', LinearRegression())])



model1=pipeline1.fit(x_train,y_train)
print(model1)
#mse error 
y_pred=model1.predict(x_test)
print(f'MSE: {mean_squared_error(y_test,y_pred)} \n RMSE: {np.sqrt(mean_squared_error(y_test,y_pred))} \n R2: {r2_score(y_test,y_pred)}')
plt.scatter(y_pred,y_test)
numeric_features= [0,1,3,4]
numeric_transformer=Pipeline(steps=[('stdscaler', StandardScaler())])

preprocessor=ColumnTransformer(transformers=
                               [('transform',numeric_transformer,
                                 numeric_features)])

pipeline2= Pipeline(steps=[('prepro', preprocessor),
                          ('Regressor', RandomForestRegressor())])
model2=pipeline2.fit(x_train,y_train)
print(model2)
y_pred=model2.predict(x_test)
print(f'MSE: {mean_squared_error(y_test,y_pred)} \n RMSE: {np.sqrt(mean_squared_error(y_test,y_pred))} \n R2: {r2_score(y_test,y_pred)}')
plt.scatter(y_pred,y_test)
import joblib
filename = './real_estate_model.pkl'
joblib.dump(model2, filename)
loaded_model= joblib.load(filename)
X_new = np.array([[16.2,289.3248,5,24.98203,121.54348],
                  [13.6,4082.015,0,24.94155,121.5038]])
y_pred=loaded_model.predict(X_new)
y_pred<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12574850299401197, 'fraction_numerical': 0.038746037337090526, 'mean_word_length': 5.086816720257235, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '258636', 'license': 'MIT', 'n_tokens_mistral': 1922, 'n_tokens_neox': 1757, 'n_words': 494, 'path': 'challenges/02 - Real Estate Regression Challenge Solution.ipynb', 'repo_name': 'bksaini078/ml-basics', 'size': '5686'}