<jupyter_start><jupyter_text>--- 
 
    Regression 


--- In the previous Notebook, we have used **image** and **political** datasets to predict binary and multiclass outcomes. But what if your problem requires a continuous outcome? Regression is best suited to solving such problems. You will learn about fundamental concepts in regression and apply them to predict the life expectancy in a given country using Gapminder data  The basics of linear regression- Regression mechanics    - $y = ax + b$        - $y$ = target        - $x$ = single feature        - $a, b$ = parameters of model    - Define an error functions for any given line        - Choose the line that minimizes the error function- The loss function    - Ordinary least squares (OLD) : Minimize sum of squares of residuals- Linear regression in higher dimensions$$ y = a_1 x_1 + a_2 x_2 + b $$    - To fit a linear regression model here:        - Need to specify 3 variables    - In higher dimensions:        - Must specify coefficient for each feature and the variable b           $ y = a_1x_1 + a_2x_2 + a_3x_3 + \dots + a_nx_n + b $<jupyter_code>! git clone https://github.com/mohd-faizy/CAREER-TRACK-Data-Scientist-with-Python.git
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import os

#plt.style.use('ggplot')
sns.set_theme(style='whitegrid')
%matplotlib inline
os.chdir('/content/CAREER-TRACK-Data-Scientist-with-Python/25_Supervised Learning with scikit-learn/_dataset')
cwd = os.getcwd()
print('Curent working directory is ', cwd)
ls<jupyter_output>automobile_miles_per_gallon.csv  gm_2008_region.csv
boston_housing.csv               house-votes-84.csv
diabetes.csv                     house-votes-84_missing_data.csv
gapminder_2008_region.csv        PIMA_Indians.csv
gapminder_all.csv                white-wine.csv
gapminder_drop_region.csv        winequality-red.csv<jupyter_text>__Introduction to Regression__  Importing data for supervised learning<jupyter_code># Read the CSV file into a DataFrame: df
df = pd.read_csv('gm_2008_region.csv')
df.drop(labels=['Region'], axis='columns', inplace=True)

# Create arrays for features and target variable
y = df['life'].values
X = df['fertility'].values

# Print the dimensions of X and y before reshaping
print("Dimensions of y before reshaping: {}".format(y.shape))
print("Dimensions of X before reshaping: {}".format(X.shape))

# Reshaping the arrays by using the `.reshape()` method and passing in -1 and 1.
'''It simply means that you are not sure about what number of rows or columns 
   you can give and you are asking numpy to suggest number of  column or rows 
   to get reshaped in.
'''
y = y.reshape(-1, 1)
X = X.reshape(-1, 1)

# Print the dimensions of X and y after reshaping
print("Dimensions of y after reshaping: {}".format(y.shape))
print("Dimensions of X after reshaping: {}".format(X.shape))<jupyter_output>Dimensions of y before reshaping: (139,)
Dimensions of X before reshaping: (139,)
Dimensions of y after reshaping: (139, 1)
Dimensions of X after reshaping: (139, 1)<jupyter_text>__EDA the Gapminder data__<jupyter_code>df.head()
df.columns
df.isna().sum()
#cmap = "Paired"
#cmap = "icefire"
#cmap = "viridis"
#cmap = "mako"
#cmap = "rocket"
plt.figure(figsize=(9, 9))
sns.heatmap(df.corr(), square=True, cmap='viridis')
df.describe()
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 139 entries, 0 to 138
Data columns (total 9 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   population       139 non-null    float64
 1   fertility        139 non-null    float64
 2   HIV              139 non-null    float64
 3   CO2              139 non-null    float64
 4   BMI_male         139 non-null    float64
 5   GDP              139 non-null    float64
 6   BMI_female       139 non-null    float64
 7   life             139 non-null    float64
 8   child_mortality  139 non-null    float64
dtypes: float64(9)
memory usage: 9.9 KB<jupyter_text>Fit & predict for regression<jupyter_code>X_fertility = df['fertility'].values.reshape(-1, 1)
y = df['life'].values.reshape(-1, 1)
plt.figure(figsize=(12, 7), facecolor='lightgrey')
sns.scatterplot(x='fertility', y='life', data=df)
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

sns.set_theme(style="dark")

# Draw a combo histogram and scatterplot with density contours
f, ax = plt.subplots(figsize=(12, 7), facecolor='lightgrey')
sns.scatterplot(x='fertility', y='life', data=df)
sns.kdeplot(x='fertility', y='life', data=df, levels=5, color="tab:red", linewidths=1)<jupyter_output><empty_output><jupyter_text>**Strongly Negative Correlation**
 
As you can see, there is a **Strongly Negative Correlation**, so a linear regression should be able to capture this trend. Your job is to fit a linear regression and then predict the life expectancy, overlaying these predicted values on the plot to generate a regression line. You will also compute and print the $R^2$ score using sckit-learn's ```.score()``` method.<jupyter_code>from sklearn.linear_model import LinearRegression

# Create the regressor: reg
reg = LinearRegression()

# Create th prediction space
'''Reshape your data either X.reshape(-1, 1) if your data has a single 
   feature/column and X.reshape(1, -1) if it contains a single sample.
'''
prediction_space = np.linspace(min(X_fertility), max(X_fertility)).reshape(-1, 1)

# Fit the model to the data
reg.fit(X_fertility, y)

# compute predictions over the prediction space: y_pred
y_pred = reg.predict(prediction_space)

# Print $R^2$
print(reg.score(X_fertility, y))

# Plot regression line on scatter plot
plt.figure(figsize=(12, 7), facecolor='lightgrey')
sns.scatterplot(x='fertility', y='life', data=df)
plt.plot(prediction_space, y_pred, color='tab:red', linewidth=3, alpha=0.5)<jupyter_output>0.6192442167740035<jupyter_text>Train/test split for regressionTrain and test sets are vital to ensure that your supervised learning model is able to generalize well to new data. This was true for classification models, and is equally true for linear regression models.In this exercise, you will split the Gapminder dataset into training and testing sets, and then fit and predict a linear regression over all features. In addition to computing the $R^2$ score, you will also compute the Root Mean Squared Error (RMSE), which is another commonly used metric to evaluate regression models.<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

# Create training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Create the regressor: reg_all
reg_all = LinearRegression()

# Fit the regressor to the training data
reg_all.fit(X_train, y_train)

# Predict on the test data: y_pred
y_pred = reg_all.predict(X_test)

# compute and print R^2 and RMSE
print("R^2: {}".format(reg_all.score(X_test, y_test)))
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
print("Root Mean Squared Error: {}".format(rmse))<jupyter_output>R^2: 0.7298987360907494
Root Mean Squared Error: 4.194027914110243<jupyter_text>Cross-validation - Cross-validation motivation    - Model performance is dependent on way the data is split    - Not representative of the model's ability to generalize    - Solution : **Cross-validation**!- k-fold Cross-validation  5-fold cross-validationCross-validation is a vital step in evaluating a model. It maximizes the amount of data that is used to train the model, as during the course of training, the model is not only trained, but also tested on all of the available data.In this exercise, you will practice 5-fold cross validation on the Gapminder data. By default, scikit-learn's ```cross_val_score()``` function uses $R^2$ as the metric of choice for regression. Since you are performing 5-fold cross-validation, the function will return 5 scores. Your job is to compute these 5 scores and then take their average.<jupyter_code>from sklearn.model_selection import cross_val_score

# Create a linear regression object: reg
reg = LinearRegression()

# Compute 5-fold cross-validation scores: cv_scores
cv_scores = cross_val_score(reg, X, y, cv=5)

# Print the 5-fold cross-validation scores
print(cv_scores)

print("Average 5-Fold CV Score: {}".format(np.mean(cv_scores)))<jupyter_output>[0.71001079 0.75007717 0.55271526 0.547501   0.52410561]
Average 5-Fold CV Score: 0.6168819644425119<jupyter_text>K-Fold CV comparisonCross validation is essential but do not forget that the more folds you use, the more computationally expensive cross-validation becomes. In this exercise, you will explore this for yourself. Your job is to perform 3-fold cross-validation and then 10-fold cross-validation on the Gapminder dataset.In the IPython Shell, you can use ```%timeit``` to see how long each 3-fold CV takes compared to 10-fold CV by executing the following ```cv=3``` and ```cv=10```:```python%timeit cross_val_score(reg, X, y, cv = ____)```<jupyter_code># Create a linear regression object: reg
reg = LinearRegression()

# Perform 3-fold CV
%timeit cross_val_score(reg, X, y, cv=3)
cvscores_3 = cross_val_score(reg, X, y, cv=3)
print(np.mean(cvscores_3))

# Perform 10-fold CV
%timeit cross_val_score(reg, X, y, cv=10)
cvscores_10 = cross_val_score(reg, X, y, cv=10)
print(np.mean(cvscores_10))<jupyter_output>100 loops, best of 5: 3.33 ms per loop
0.6294715754653507
100 loops, best of 5: 10.4 ms per loop
0.5883937741571185<jupyter_text>**Regularized regression**- Why regularize?    - Recall: Linear regression minimizes a loss function    - It chooses a coefficient for each feature variable    - Large coefficient can lead to overfitting    - Penalizing large coefficients : **Regularization**- Ridge regression    - Loss function = $ \text{OLS loss function} + \alpha \sum^{n}_{i=1}a_i^2 $    - Alpha : Parameter we need to choose (Hyperparameter or $\lambda$)        - Picking alpha is similar to picking k in k-NN    - Alpha controls model complexity        - Alpha = 0: get back OLS (Can lead to overfitting)        - Very high alpha: Can lead to underfitting- Lasso regression    - Loss function = $ \text{OLS loss function} + \alpha \sum^{n}_{i=1}|a_i| $    - Can be used to select import features of a dataset    - Shrinks the coefficients of less important features to exactly 0  **Regularization I: Lasso**In the video, you saw how Lasso selected out the ```'RM'``` feature as being the most important for predicting Boston house prices, while shrinking the coefficients of certain other features to 0. Its ability to perform feature selection in this way becomes even more useful when you are dealing with data involving thousands of features.In this exercise, you will fit a lasso regression to the Gapminder data you have been working with and plot the coefficients. Just as with the Boston data, you will find that the coefficients of some features are shrunk to 0, with only the most important ones remaining.<jupyter_code>X = df.drop('life', axis='columns').values
y = df['life'].values
# Import Lasso
from sklearn.linear_model import Lasso

# Instantiate a lasso regressor: lasso
lasso = Lasso(alpha=0.4, normalize=True)

# Fit the regressor to the data
lasso.fit(X, y)

# Compute and print the coefficients
lasso_coef = lasso.coef_
print(lasso_coef)

# Plot the coefficients
df_columns = df.columns[:-1]
plt.figure(figsize=(12, 7), facecolor='lightgrey')
plt.plot(range(len(df_columns)), lasso_coef)
plt.xticks(range(len(df_columns)), df_columns.values, rotation=60)
plt.margins(0.02)<jupyter_output>[-0.         -0.         -0.          0.          0.          0.
 -0.         -0.07087587]<jupyter_text>**Regularization II: Ridge**Lasso is great for feature selection, but when building regression models, Ridge regression should be your first choice.Recall that lasso performs regularization by adding to the loss function a penalty term of the absolute value of each coefficient multiplied by some alpha. This is also known as L1 regularization because the regularization term is the L1 norm of the coefficients. This is not the only way to regularize, however.If instead you took the sum of the squared values of the coefficients multiplied by some alpha - like in Ridge regression - you would be computing the L2 norm. In this exercise, you will practice fitting ridge regression models over a range of different alphas, and plot cross-validated R2 scores for each.<jupyter_code>def display_plot(cv_scores, cv_scores_std):
    fig = plt.figure(figsize=(12,7), facecolor='lightgrey')
    ax = fig.add_subplot(1,1,1)
    ax.plot(alpha_space, cv_scores, color='tab:red', linewidth=3,)

    std_error = cv_scores_std / np.sqrt(10)

    ax.fill_between(alpha_space, cv_scores + std_error, cv_scores - std_error, alpha=0.2)
    ax.set_ylabel('CV Score +/- Std Error')
    ax.set_xlabel('Alpha')
    ax.axhline(np.max(cv_scores), linestyle='--', color='.5')
    ax.set_xlim([alpha_space[0], alpha_space[-1]])
    ax.set_xscale('log')
from sklearn.linear_model import Ridge

# Setup the array of alphas and lists to store scores
alpha_space = np.logspace(-4, 0, 50)
ridge_scores = []
ridge_scores_std = []

# Create a ridge regressor: ridge
ridge = Ridge(normalize=True)

# Compute scores over range of alphas
for alpha in alpha_space:
    
    # Specify the alpha value to use: ridge.alhpa
    ridge.alpha = alpha
    
    # Perform 10-fold CV: ridge_cv_scores
    ridge_cv_scores = cross_val_score(ridge, X, y, cv=10)
    
    # Append the mean of ridge_cv_scores to ridge_scores
    ridge_scores.append(np.mean(ridge_cv_scores))
    
    # Append the std of ridge_cv_scores to ridge_scores_std
    ridge_scores_std.append(np.std(ridge_cv_scores))
    
# Display the plot
display_plot(ridge_scores, ridge_scores_std)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09389570332113908, 'fraction_numerical': 0.02912273151136934, 'mean_word_length': 4.245297215951844, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '525337', 'license': 'MIT', 'n_tokens_mistral': 4699, 'n_tokens_neox': 4256, 'n_words': 1569, 'path': '25_Supervised Learning with scikit-learn/02_Regression_Supervised_Learning.ipynb', 'repo_name': 'mohd-faizy/DataScience-With-Python', 'size': '13941'}