<jupyter_start><jupyter_text>Prediction Question: How accurately Can I predict the price of houses given Values of all variable Inference: How accurately can I estimate the prices of the houses getting effected by each variable.?<jupyter_code>df = pd.read_csv("House_Price.csv", header= 0)
df.head()
df.shape
df.info() ## direct<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 506 entries, 0 to 505
Data columns (total 19 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   price        506 non-null    float64
 1   crime_rate   506 non-null    float64
 2   resid_area   506 non-null    float64
 3   air_qual     506 non-null    float64
 4   room_num     506 non-null    float64
 5   age          506 non-null    float64
 6   dist1        506 non-null    float64
 7   dist2        506 non-null    float64
 8   dist3        506 non-null    float64
 9   dist4        506 non-null    float64
 10  teachers     506 non-null    float64
 11  poor_prop    506 non-null    float64
 12  airport      506 non-null    object 
 13  n_hos_beds   498 non-null    float64
 14  n_hot_rooms  506 non-null    float64
 15  waterbody    506 non-null    object 
 16  rainfall     506 non-null    int64  
 17  bus_ter      506 non-null    object 
 18  parks        506 non-null    float64
dtypes: float[...]<jupyter_text>Hospital beds have probably Null values<jupyter_code>df.describe()   ## Extrapolatory Data Description
## Hospital Beds have only 498 values read
## 8 Missing Values

## Crime rate is negatively skewed
## outliers in n_hot_rooms and Rainfall

sns.jointplot(x = 'n_hot_rooms', y = 'price', data= df)
sns.jointplot(x = 'rainfall', y = 'price', data= df)
## Rainfall and n_hot_rooms have 1 and 2 outliers respectively
## To be treated
## Categorical Vars
df.head()
## Cat var = airport, waterbody, bus_ter
sns.countplot('airport', data= df)
sns.countplot('waterbody', data= df)
sns.countplot('bus_ter', data= df)
## bus_ter takes only Yes values<jupyter_output><empty_output><jupyter_text>Outlier Treatment USing Capping and Flooring<jupyter_code>uv = np.percentile(df.n_hot_rooms,[99])[0]       ## Since this will give an array
df[(df.n_hot_rooms > uv)]
df.n_hot_rooms[(df.n_hot_rooms > 3* uv)] = 3*uv
lv = np.percentile(df.rainfall,[1])[0]
df[(df.rainfall < lv)]
df.rainfall[(df.rainfall < 0.3* lv)] = 0.3*lv
0.3*lv
## Checking for outlier treatment
df.describe()
## Rainfall and n_hot_rooms are now outlier free
## Crime rate
df.head()
sns.jointplot(x = 'crime_rate', y = 'price', data= df)

df.n_hos_beds = df.n_hos_beds.fillna(df.n_hos_beds.mean())
df.info()
## for skewed data in crime rate
sns.jointplot(x = 'crime_rate', y= 'price', data= df)
## the scatter plot shows a log relation
## we would take log(crime_rate) to get a linear relationshiph betwn Y and x
## crime_rate has a dense scatter cloud at 0 so we take log(1 + crime_rate) since log(0) = -inf
df.crime_rate = np.log(1 + df.crime_rate)
sns.jointplot(x= 'crime_rate', y='price', data=df)
## Graphically more linear than before

## dist (1-4) represent same kind of info and hence we should take an avg of all 4 into 1 var and use it in our model
df['avg_dist'] = (df.dist1+df.dist2+df.dist3+df.dist4)/4
df = df.drop(['dist1', 'dist2', 'dist3', 'dist4'], axis =1)
## we can take min dist or max dist
df.describe()
df.info()
## bus_ter doesnt provide usefull info hence del
del df['bus_ter']
df.head()
## categorical data: airport, waterbody
df = pd.get_dummies(df)
df.head()
## we dont need airport_no/yes 
## aslo waterbody_none : so Drop

df = df.drop(['airport_NO', 'waterbody_None'], axis= 1)
#df.head()<jupyter_output><empty_output><jupyter_text>Now once the data is in some proper form and cleaned. We next run a correlation analysis to check for correlation and Multicollinearity.<jupyter_code>df.corr()
## First Looking at all var with price
## then considering others.<jupyter_output><empty_output><jupyter_text>Park and air_qual have high correlation of 0.9. So this may lead to multicollinearity.Corr(price , Park) = -0.3915Corr(price, air_qual) = -0.429So I will keep air_qual<jupyter_code>del df['parks']
df.head()<jupyter_output><empty_output><jupyter_text>Linear Regression Model<jupyter_code>import statsmodels.api as sn
X = sn.add_constant(df['room_num'])
lm = sn.OLS(df['price'], X).fit()
lm.summary()
from sklearn.linear_model import LinearRegression
y = df['price']
x = df[['room_num']]
lm1 = LinearRegression()
lm1.fit(x,y)
print(lm1.intercept_, lm1.coef_)
## help(lm1)
lm2 = lm1.predict(x)
sns.jointplot(x= df['room_num'], y= df['price'], data=df, kind= 'reg')
## Multiple Regression

x_multi = df.drop('price', axis= 1)
Y_multi = df['price']
x_multi.head()
x_multi_const = sn.add_constant(x_multi)
x_multi_const.head()
rm = sn.OLS(Y_multi, x_multi_const).fit()
rm.summary()
rm1 = LinearRegression()
#rm1.fit(x_multi, Y_multi)
rm1.fit(x_multi,Y_multi)
print(rm1.intercept_, rm1.coef_)<jupyter_output>-6.498625198419479 [ 9.70998193e-03 -4.08746495e-02 -1.58973999e+01  4.01901676e+00
 -5.71475069e-03  1.00700068e+00 -5.77271243e-01  3.29221139e-01
  9.18675603e-02  1.61185504e-02 -1.21863952e+00  1.13151586e+00
  2.64086064e-01 -6.87555889e-01 -2.91318712e-01]<jupyter_text>Test Train split:<jupyter_code>from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x_multi, Y_multi, test_size = 0.2, random_state = 0)
#x_train.head()
lm_a = LinearRegression()
lm_a.fit(x_train, y_train)
y_test_a = lm_a.predict(x_test)
y_train_a = lm_a.predict(x_train)
from sklearn.metrics import r2_score
r2_score(y_test, y_test_a)
r2_score(y_train, y_train_a)
## Model R^2 is greater in test set than in train set denoting proper fit.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1215863628457123, 'fraction_numerical': 0.06627239519916507, 'mean_word_length': 4.213055303717135, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '98964', 'license': 'MIT', 'n_tokens_mistral': 2381, 'n_tokens_neox': 2127, 'n_words': 586, 'path': 'Linear-Regression/House_Price.ipynb', 'repo_name': 'KinjollDey/House_data_regression_modeling', 'size': '5749'}