<jupyter_start><jupyter_text>Data Exploration<jupyter_code># Read data from csv file into a dataframe

df_train = pd.read_csv('train.csv')
df_train.describe()
df_train.shape
df_train.head()
df_train['SalePrice'].describe()
%matplotlib inline

plt.scatter(df_train['GrLivArea'],df_train['SalePrice'])
plt.xlabel('Area')
plt.ylabel('Price')
plt.show()
plt.scatter(df_train['GrLivArea'],df_train['SalePrice'])
plt.xlabel('Area')
plt.ylabel('Price')
plt.show()
plt.scatter(df_train['TotalBsmtSF'],df_train['SalePrice'])
plt.xlabel('Basement Area')
plt.ylabel('Price')
plt.show()
cm = df_train.corr()
f, ax = plt.subplots(figsize=(15,15))
sns.heatmap(cm,cmap="Blues")
n = 8 
col = cm.nlargest(n, 'SalePrice')['SalePrice'].index
f, ax = plt.subplots(figsize=(14,10))
sns.heatmap(df_train[col].corr(), vmax=0.8, cmap="Blues",annot=True)
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>From above heatmap we can see that only certain features are affecting the target variable 'SalePrice'. We will now extract feature list with which gives the percentage with which these vallues are affecting 'SalePrice'<jupyter_code>columns = ['SalePrice','OverallQual','GrLivArea','TotalBsmtSF','GarageArea']
sns.pairplot(df_train[columns])

feature_list = abs(cm.iloc[:-1,-1]).sort_values()
feature_list
feature_list.plot(kind='barh', figsize=(30,15))
plt.title('Feature Importance Chart',fontsize=20)
plt.xlabel('Correlation', fontsize=20)
plt.ylabel('variable', fontsize=20)<jupyter_output><empty_output><jupyter_text>Data Analysis Concclusions: From above analysis we can see that 'TotalBsmtSF','OverallQual', 'GrLivArea', 'GarageCars','1stFlrSF' are the most imporant features in relation with target feature 'SalePrice'. We will be using only these features to create our model as our target variable is most affected by these.<jupyter_code># checking for missing data
count = df_train[['TotalBsmtSF','OverallQual', 'GrLivArea', 'GarageCars','1stFlrSF']].isnull().sum().sort_values(ascending=False)
missing_data = pd.DataFrame(count, columns=['count'])
missing_data = missing_data[missing_data['count']>0]
missing_data<jupyter_output><empty_output><jupyter_text>There are no null values in the requuired data column. Now we will use Sklearn to create train test split and data preprocessing<jupyter_code>#SalePrice is target column so we will be dropping it from training input and store it in y variable
X = df_train[['TotalBsmtSF','OverallQual', 'GrLivArea', 'GarageCars','1stFlrSF']]
y = df_train['SalePrice']
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.1,random_state=42)
print(X_train.shape,y_train.shape,X_test.shape,y_test.shape)
y_train =y_train.to_numpy()
y_test = y_test.to_numpy()
y_train = y_train.reshape(-1,1)
y_test = y_test.reshape(-1,1)
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train_scaled = sc.fit_transform(X_train)
X_test_scaled = sc.transform(X_test)

y_train_scaled = sc.fit_transform(y_train)
y_test_scaled = sc.transform(y_test)
from sklearn.linear_model import LinearRegression
Reg = LinearRegression()
Reg.fit(X_train,y_train)
y_pred = Reg.predict(X_train)
from sklearn.metrics import mean_squared_error

rmse = np.sqrt(mean_squared_error(y_pred, y_train))
mpe  = np.mean((y_train - y_pred)/y_train*100)
mape = np.mean(abs(y_train - y_pred)/y_train*100)
me = np.mean((y_train - y_pred))
print('RMSE = {}'.format(rmse))
print('MPE = {}'.format(mpe))
print('MAPE = {}'.format(mape))
print('ME = {}'.format(me))
from sklearn.model_selection import cross_val_score
#print(f'The mean cross validation score for this model is {round(cross_val_score(Reg, X_train, y_train).mean(),4)}.')
print(f'The training score for this model is {round(Reg.score(X_train, y_train), 4)}.')

y_pred1 = Reg.predict(X_test)
from sklearn.metrics import mean_squared_error

rmse = np.sqrt(mean_squared_error(y_pred1, y_test))
mpe  = np.mean((y_test - y_pred1)/y_test*100)
mape = np.mean(abs(y_test - y_pred1)/y_test*100)
me = np.mean((y_test - y_pred1))
print('RMSE = {}'.format(rmse))
print('MPE = {}'.format(mpe))
print('MAPE = {}'.format(mape))
print('ME = {}'.format(me))
print(f'The testing score for this model is {round(Reg.score(X_test, y_test), 4)}.')
from sklearn.metrics import r2_score

r2_score(y_test, y_pred1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1454462032576279, 'fraction_numerical': 0.013305804083505391, 'mean_word_length': 7.916155419222904, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '658423', 'license': 'MIT', 'n_tokens_mistral': 1678, 'n_tokens_neox': 1611, 'n_words': 329, 'path': 'Housing_price_pred.ipynb', 'repo_name': 'tejashrigadre/housing-price-prediction', 'size': '4359'}