<jupyter_start><jupyter_text>summary of the boston dataset<jupyter_code>bos.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 506 entries, 0 to 505
Data columns (total 13 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   0       506 non-null    float64
 1   1       506 non-null    float64
 2   2       506 non-null    float64
 3   3       506 non-null    float64
 4   4       506 non-null    float64
 5   5       506 non-null    float64
 6   6       506 non-null    float64
 7   7       506 non-null    float64
 8   8       506 non-null    float64
 9   9       506 non-null    float64
 10  10      506 non-null    float64
 11  11      506 non-null    float64
 12  12      506 non-null    float64
dtypes: float64(13)
memory usage: 51.5 KB<jupyter_text>check for the missing values<jupyter_code>bos.isna().sum()<jupyter_output><empty_output><jupyter_text>No missing values are present in the dataset Get the feature names from boston dataset and rename with those in dataset<jupyter_code>boston.feature_names
print(boston.DESCR)
bos.columns = boston.feature_names
bos.head()<jupyter_output><empty_output><jupyter_text>look for target column<jupyter_code>boston.target<jupyter_output><empty_output><jupyter_text>The feature 'MEDEV' is renamed as 'PRICE' which will be served as target<jupyter_code>bos['PRICE'] = boston.target
bos.head()<jupyter_output><empty_output><jupyter_text>Summarize the central tendency, dispersion and shape of a dataset’s distribution,<jupyter_code>bos.describe()<jupyter_output><empty_output><jupyter_text>Exploratory data analysis<jupyter_code>import seaborn as sns
fig, ax = plt.subplots(ncols=7, nrows=2, figsize=(20, 10))
index = 0
ax = ax.flatten()
for k,v in bos.items():
    sns.boxplot(y=k, data=bos, ax=ax[index])
    index += 1
plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=5.0)<jupyter_output><empty_output><jupyter_text>Analysis: features such as CRIM,ZN,RM,CHAS,DIS,B,PTRATIO,LSTAT,PRICE have the outliers. Let's see the percentage of outliers<jupyter_code>for key,val in bos.items():
    q1 = val.quantile(0.25)
    q3 = val.quantile(0.75)
    irq = q3 - q1
    v_col = val[(val <= q1 - 1.5 * irq) | (val >= q3 + 1.5 * irq)]
    perc = np.shape(v_col)[0] * 100.0 / np.shape(bos)[0]
    print("Column %s outliers = %.2f%%" % (key, perc))
#Let's remove PRICE outliers (PRICE = 50.0)
bos = bos[~(bos['PRICE'] >= 50.0)]
bos.shape
# let's see how data is distributed for every column
plt.figure(figsize=(20,25), facecolor='white')
plotnumber = 1
column_cust = ['CRIM', 'ZN', 'INDUS','NOX', 'RM', 'AGE', 'DIS', 'RAD','TAX', 'PTRATIO', 'B', 'LSTAT']
for column in column_cust:
    if plotnumber<=16 :
        ax = plt.subplot(4,4,plotnumber)
        sns.distplot(bos[column])
        plt.xlabel(column,fontsize=20)
    plotnumber+=1
plt.tight_layout()<jupyter_output><empty_output><jupyter_text>The histogram also shows that columns CRIM, ZN, B has highly skewed distributions. Also MEDV looks to have a normal distribution (the predictions) and other colums seem to have norma or bimodel ditribution of data except CHAS (which is a discrete variable).<jupyter_code>plt.figure(figsize=(20, 10))
sns.heatmap(bos.corr().abs(),  annot=True)<jupyter_output><empty_output><jupyter_text>From correlation matrix, we see TAX and RAD are highly correlated features. The columns LSTAT, INDUS, RM, TAX, NOX, PTRATIO has a correlation score above 0.5 with PRICE which is a good indication of using as predictors. Let's plot these columns against PRICE.<jupyter_code>from sklearn import preprocessing
# Let's scale the columns before plotting them against PRICE
min_max_scaler = preprocessing.MinMaxScaler()
column_sels = ['LSTAT', 'INDUS', 'NOX', 'PTRATIO', 'RM', 'TAX', 'DIS', 'AGE']
x = bos.loc[:,column_sels]
y = bos['PRICE']
x = pd.DataFrame(data=min_max_scaler.fit_transform(x), columns=column_sels)
fig, axs = plt.subplots(ncols=4, nrows=2, figsize=(20, 10))
index = 0
axs = axs.flatten()
for i, k in enumerate(column_sels):
    sns.regplot(y=y, x=x[k], ax=axs[i])
plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=5.0)
#Let's try to remove the skewness of the data trough log transformation.
y =  np.log1p(y)
for col in x.columns:
    if np.abs(x[col].skew()) > 0.3:
        x[col] = np.log1p(x[col])
        
#after log transformation
x
from sklearn.preprocessing import StandardScaler 
scaler =StandardScaler()

X_scaled = scaler.fit_transform(x)
X_scaled
from statsmodels.stats.outliers_influence import variance_inflation_factor
variables = X_scaled

# we create a new data frame which will include all the VIFs
# note that each variable has its own variance inflation factor as this measure is variable specific (not model specific)
# we do not include categorical values for mulitcollinearity as they do not provide much information as numerical ones do
vif = pd.DataFrame()

# here we make use of the variance_inflation_factor, which will basically output the respective VIFs 
vif["VIF"] = [variance_inflation_factor(variables, i) for i in range(variables.shape[1])]
# Finally, I like to include names so it is easier to explore the result
vif["Features"] = x.columns
vif<jupyter_output><empty_output><jupyter_text>Here, we have the correlation values for all the features. As a thumb rule, a VIF value greater than 5 means a very severe multicollinearity. We don't any VIF greater than 5 , For NOX, vif is 5.2 so we will consider as predictor.Great. Let's go ahead and use linear regression and see how good it fits our data. But first. let's split our data in train and test<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size = 0.2, random_state = 366)
print('X_train: ',X_train.shape,'\nX_test: ',X_test.shape,'\nY_train: ',y_train.shape,'\nY_test: ',y_test.shape)
from sklearn.linear_model import LinearRegression
regression = LinearRegression()

regression.fit(X_train,y_train)
regression.score(X_train,y_train)
def adj_r2(x,y):
    r2 = regression.score(x,y)
    n = x.shape[0]
    p = x.shape[1]
    adjusted_r2 = 1-(1-r2)*(n-1)/(n-p-1)
    return adjusted_r2
adj_r2(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Our r2 score is 76.03% and adj r2 is 75.53% for our training et., so looks like we are not being penalized by use of any feature.Let's check how well model fits the test data.Now let's check if our model is overfitting our data using regularization<jupyter_code>regression.score(X_test,y_test)
adj_r2(X_test,y_test)<jupyter_output><empty_output><jupyter_text>So it looks like our model r2 score is less on the test data.Let's see if our model is overfitting our training data.<jupyter_code>from sklearn.linear_model  import Ridge,Lasso,RidgeCV, LassoCV, ElasticNet, ElasticNetCV
# Lasso Regularization
# LassoCV will return best alpha and coefficients after performing 10 cross validations
lasscv = LassoCV(alphas = None,cv =10, max_iter = 100000, normalize = True)
lasscv.fit(X_train, y_train)
# best alpha parameter
alpha = lasscv.alpha_
alpha
#now that we have best parameter, let's use Lasso regression and see how well our data has fitted before

lasso_reg = Lasso(alpha)
lasso_reg.fit(X_train, y_train)
lasso_reg.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>our r2_score for test data (73.18%) comes same as before using regularization. So, it is fair to say our model did not overfit the data.<jupyter_code># Using Ridge regression model
# RidgeCV will return best alpha and coefficients after performing 10 cross validations. 
# We will pass an array of random numbers for ridgeCV to select best alpha from them

alphas = np.random.uniform(low=0, high=10, size=(50,))
ridgecv = RidgeCV(alphas = alphas,cv=10,normalize = True)
ridgecv.fit(X_train, y_train)
ridgecv.alpha_
ridge_model = Ridge(alpha=ridgecv.alpha_)
ridge_model.fit(X_train, y_train)
ridge_model.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>we got the same r2 square using Ridge regression as well. So, it's safe to say there is no overfitting.<jupyter_code># Elastic net

elasticCV = ElasticNetCV(alphas = None, cv =10)

elasticCV.fit(X_train, y_train)
elasticCV.alpha_
elasticnet_reg = ElasticNet(alpha = elasticCV.alpha_,l1_ratio=0.5)
elasticnet_reg.fit(X_train, y_train)
elasticnet_reg.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>So, we can see by using different type of regularization, we still are getting the same r2 score. That means our model has been well trained over the training data and there is no overfitting.<jupyter_code>predictions = regression.predict(X_test)
from sklearn import metrics

print('MAE:', metrics.mean_absolute_error(y_test, predictions))
print('MSE:', metrics.mean_squared_error(y_test, predictions))
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, predictions)))
plt.scatter(y_test,predictions)
plt.xlabel('Y Test')
plt.ylabel('Predicted Y')
sns.distplot((y_test-predictions),bins=50);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10650821856952465, 'fraction_numerical': 0.028653931585961794, 'mean_word_length': 5.007338225483656, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 68, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '308733', 'license': 'MIT', 'n_tokens_mistral': 3335, 'n_tokens_neox': 3092, 'n_words': 929, 'path': 'Linear Regression.ipynb', 'repo_name': 'someshkr/Linear-Regression-Assignment', 'size': '9004'}