<jupyter_start><jupyter_text>Linear Regression Final Challenge Import of the necessary libraries and load the dataset from CSV file<jupyter_code>import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

import pandas as pd

df = pd.read_csv('./assets/Life_Expectancy_Data.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Filter the columns needed for this exercise<jupyter_code>le = df[['Country', 'Year', 'Life expectancy ']]
le = le.rename(columns={'Life expectancy ':'Life_expectancy'})
le.describe(include='all')<jupyter_output><empty_output><jupyter_text>Identify null values and remove them<jupyter_code>df_col_nulls = le.isnull().sum()
df_col_nulls = df_col_nulls[df_col_nulls > 0]
print(df_col_nulls)
le.dropna(axis=0, how="any", thresh=None, subset=None, inplace=True)
le.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 2928 entries, 0 to 2937
Data columns (total 3 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   Country          2928 non-null   object 
 1   Year             2928 non-null   int64  
 2   Life_expectancy  2928 non-null   float64
dtypes: float64(1), int64(1), object(1)
memory usage: 91.5+ KB<jupyter_text>Filter the five countries selected to build the regression models<jupyter_code>filtered = le.query('Country == ["El Salvador", "Germany", "Sweden", "United States of America", "United Kingdom of Great Britain and Northern Ireland"]')
filtered.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 80 entries, 818 to 2809
Data columns (total 3 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   Country          80 non-null     object 
 1   Year             80 non-null     int64  
 2   Life_expectancy  80 non-null     float64
dtypes: float64(1), int64(1), object(1)
memory usage: 2.5+ KB<jupyter_text>Preview of the linear correlation of the dataset, not strong linear correlation between year and life expectancy variables in general<jupyter_code>plt.figure(figsize=(16, 6))
heatmap = sns.heatmap(filtered.corr(), vmin=-1, vmax=1, annot=True)<jupyter_output><empty_output><jupyter_text>El Salvador<jupyter_code>SV = filtered[filtered['Country'] == 'El Salvador']
x = SV['Year'].to_numpy()
y = SV['Life_expectancy'].to_numpy()
x = x.reshape(-1, 1)
y = y.reshape(-1, 1)<jupyter_output><empty_output><jupyter_text>Plot of life expectancy vs year<jupyter_code>plt.scatter(x, y)
plt.title("El Salvador")
plt.xlabel("Year")
plt.ylabel("Life expectancy")
plt.show()<jupyter_output><empty_output><jupyter_text>Split of data into training and testing sets<jupyter_code>from sklearn.model_selection import train_test_split 
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)<jupyter_output><empty_output><jupyter_text>Traning of the Linear regression model using training set, model will normalize inputs automatically<jupyter_code>from sklearn.linear_model import LinearRegression
regressor = LinearRegression(normalize=True)
regressor.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Calculated adjusted R-squared<jupyter_code>import statsmodels.formula.api as sm
model = sm.ols(formula='Life_expectancy ~ Year', data=SV[['Year','Life_expectancy']])
fitted1 = model.fit()
fitted1.summary()<jupyter_output>/Users/ferrycosv/opt/anaconda3/envs/becode/lib/python3.7/site-packages/scipy/stats/stats.py:1604: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=16
  "anyway, n=%i" % int(n))<jupyter_text>Calculated R-squared value for the training set<jupyter_code>regressor.score(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Calculated R-squared value for the testing set<jupyter_code>regressor.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Plotting the regression line of the model previously trained<jupyter_code>plt.scatter(x, y)
plt.plot(x, regressor.predict(x),c='r')
plt.title("El Salvador")
plt.xlabel("Year")
plt.ylabel("Life expectancy")
plt.show()<jupyter_output><empty_output><jupyter_text>Predicting the life expectancy value for El Salvador in the year 2100<jupyter_code>regressor.predict(np.array([[2100]]))[0][0]<jupyter_output><empty_output><jupyter_text>The predicted value is within a valid range of what human beings can live, if the life expectancy indicators keep improving at the same pace, this result might represent a real prediction of what would be the life expectancy in year 2100  Germany<jupyter_code>DE = filtered[filtered['Country'] == 'Germany']
x = DE['Year'].to_numpy()
y = DE['Life_expectancy'].to_numpy()
x = x.reshape(-1, 1)
y = y.reshape(-1, 1)<jupyter_output><empty_output><jupyter_text>Plot of life expectancy vs year<jupyter_code>plt.scatter(x, y)
plt.title("Germany")
plt.xlabel("Year")
plt.ylabel("Life expectancy")
plt.show()<jupyter_output><empty_output><jupyter_text>Split of data into training and testing sets<jupyter_code>from sklearn.model_selection import train_test_split 
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)<jupyter_output><empty_output><jupyter_text>Traning of the Linear regression model using training set, model will normalize inputs automatically<jupyter_code>from sklearn.linear_model import LinearRegression
regressor = LinearRegression(normalize=True)
regressor.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Calculated adjusted R-squared<jupyter_code>import statsmodels.formula.api as sm
model = sm.ols(formula='Life_expectancy ~ Year', data=DE[['Year','Life_expectancy']])
fitted1 = model.fit()
fitted1.summary()<jupyter_output>/Users/ferrycosv/opt/anaconda3/envs/becode/lib/python3.7/site-packages/scipy/stats/stats.py:1604: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=16
  "anyway, n=%i" % int(n))<jupyter_text>Calculated R-squared value for the training set<jupyter_code>regressor.score(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Calculated R-squared value for the testing set<jupyter_code>regressor.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Plotting the regression line of the model previously trained<jupyter_code>plt.scatter(x, y)
plt.plot(x, regressor.predict(x),c='r')
plt.title("Germany")
plt.xlabel("Year")
plt.ylabel("Life expectancy")
plt.show()<jupyter_output><empty_output><jupyter_text>Predicting the life expectancy value for Germany in the year 2100<jupyter_code>regressor.predict(np.array([[2100]]))[0][0]<jupyter_output><empty_output><jupyter_text>The predicted value is not inside a valid range of what human beings can live, this result might not represent a real prediction of what would be the life expectancy in year 2100 for Germany  Sweden<jupyter_code>SE = filtered[filtered['Country'] == 'Sweden']
x = SE['Year'].to_numpy()
y = SE['Life_expectancy'].to_numpy()
x = x.reshape(-1, 1)
y = y.reshape(-1, 1)<jupyter_output><empty_output><jupyter_text>Plot of life expectancy vs year<jupyter_code>plt.scatter(x, y)
plt.title("Sweden")
plt.xlabel("Year")
plt.ylabel("Life expectancy")
plt.show()<jupyter_output><empty_output><jupyter_text>Split of data into training and testing sets<jupyter_code>from sklearn.model_selection import train_test_split 
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)<jupyter_output><empty_output><jupyter_text>Traning of the Linear regression model using training set, model will normalize inputs automatically<jupyter_code>from sklearn.linear_model import LinearRegression
regressor = LinearRegression(normalize=True)
regressor.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Calculated adjusted R-squared<jupyter_code>import statsmodels.formula.api as sm
model = sm.ols(formula='Life_expectancy ~ Year', data=DE[['Year','Life_expectancy']])
fitted1 = model.fit()
fitted1.summary()<jupyter_output>/Users/ferrycosv/opt/anaconda3/envs/becode/lib/python3.7/site-packages/scipy/stats/stats.py:1604: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=16
  "anyway, n=%i" % int(n))<jupyter_text>Calculated R-squared value for the training set<jupyter_code>regressor.score(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Calculated R-squared value for the testing set<jupyter_code>regressor.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Plotting the regression line of the model previously trained<jupyter_code>plt.scatter(x, y)
plt.plot(x, regressor.predict(x),c='r')
plt.title("Sweden")
plt.xlabel("Year")
plt.ylabel("Life expectancy")
plt.show()<jupyter_output><empty_output><jupyter_text>Predicting the life expectancy value for Sweden in the year 2100<jupyter_code>regressor.predict(np.array([[2100]]))[0][0]<jupyter_output><empty_output><jupyter_text>The predicted value is inside a valid range of what human beings can live, but this result might not represent a real prediction of what would be the life expectancy in year 2100 for Sweden, the model quality is questionable  United States of America<jupyter_code>US = filtered[filtered['Country'] == 'United States of America']
x = US['Year'].to_numpy()
y = US['Life_expectancy'].to_numpy()
x = x.reshape(-1, 1)
y = y.reshape(-1, 1)<jupyter_output><empty_output><jupyter_text>Plot of life expectancy vs year<jupyter_code>plt.scatter(x, y)
plt.title("United States of America")
plt.xlabel("Year")
plt.ylabel("Life expectancy")
plt.show()<jupyter_output><empty_output><jupyter_text>Split of data into training and testing sets<jupyter_code>from sklearn.model_selection import train_test_split 
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)<jupyter_output><empty_output><jupyter_text>Traning of the Linear regression model using training set, model will normalize inputs automatically<jupyter_code>from sklearn.linear_model import LinearRegression
regressor = LinearRegression(normalize=True)
regressor.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Calculated adjusted R-squared<jupyter_code>import statsmodels.formula.api as sm
model = sm.ols(formula='Life_expectancy ~ Year', data=US[['Year','Life_expectancy']])
fitted1 = model.fit()
fitted1.summary()<jupyter_output>/Users/ferrycosv/opt/anaconda3/envs/becode/lib/python3.7/site-packages/scipy/stats/stats.py:1604: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=16
  "anyway, n=%i" % int(n))<jupyter_text>Calculated R-squared value for the training set<jupyter_code>regressor.score(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Calculated R-squared value for the testing set<jupyter_code>regressor.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Plotting the regression line of the model previously trained<jupyter_code>plt.scatter(x, y)
plt.plot(x, regressor.predict(x),c='r')
plt.title("United States of America")
plt.xlabel("Year")
plt.ylabel("Life expectancy")
plt.show()<jupyter_output><empty_output><jupyter_text>Predicting the life expectancy value for United States of America in the year 2100<jupyter_code>regressor.predict(np.array([[2100]]))[0][0]<jupyter_output><empty_output><jupyter_text>The predicted value is within a valid range of what human beings can live, if the life expectancy indicators keep improving at the same pace, this result might represent a real prediction of what would be the life expectancy in year 2100  United Kingdom of Great Britain and Northern Ireland<jupyter_code>UK = filtered[filtered['Country'] == 'United Kingdom of Great Britain and Northern Ireland']
x = UK['Year'].to_numpy()
y = UK['Life_expectancy'].to_numpy()
x = x.reshape(-1, 1)
y = y.reshape(-1, 1)<jupyter_output><empty_output><jupyter_text>Plot of life expectancy vs year<jupyter_code>plt.scatter(x, y)
plt.title("United Kingdom of Great Britain and Northern Ireland")
plt.xlabel("Year")
plt.ylabel("Life expectancy")
plt.show()<jupyter_output><empty_output><jupyter_text>Split of data into training and testing sets<jupyter_code>from sklearn.model_selection import train_test_split 
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)<jupyter_output><empty_output><jupyter_text>Traning of the Linear regression model using training set, model will normalize inputs automatically<jupyter_code>from sklearn.linear_model import LinearRegression
regressor = LinearRegression(normalize=True)
regressor.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Calculated adjusted R-squared<jupyter_code>import statsmodels.formula.api as sm
model = sm.ols(formula='Life_expectancy ~ Year', data=US[['Year','Life_expectancy']])
fitted1 = model.fit()
fitted1.summary()<jupyter_output>/Users/ferrycosv/opt/anaconda3/envs/becode/lib/python3.7/site-packages/scipy/stats/stats.py:1604: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=16
  "anyway, n=%i" % int(n))<jupyter_text>Calculated R-squared value for the training set<jupyter_code>regressor.score(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Calculated R-squared value for the testing set<jupyter_code>regressor.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Plotting the regression line of the model previously trained<jupyter_code>plt.scatter(x, y)
plt.plot(x, regressor.predict(x),c='r')
plt.title("United Kingdom of Great Britain and Northern Ireland")
plt.xlabel("Year")
plt.ylabel("Life expectancy")
plt.show()<jupyter_output><empty_output><jupyter_text>Predicting the life expectancy value for United Kingdom of Great Britain and Northern Ireland in the year 2100<jupyter_code>regressor.predict(np.array([[2100]]))[0][0]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12047490713089082, 'fraction_numerical': 0.018063952217932843, 'mean_word_length': 7.402692778457772, 'pattern_counts': {'":': 0, '<': 196, '<?xml version=': 0, '>': 202, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '50', 'dir': 'jupyter-structured-clean-dedup', 'id': '139324', 'license': 'MIT', 'n_tokens_mistral': 5051, 'n_tokens_neox': 4660, 'n_words': 1031, 'path': 'challenge.ipynb', 'repo_name': 'ferrycosv/regression-final_challenge', 'size': '13729'}