<jupyter_start><jupyter_text>Preprocessing<jupyter_code>raw_data.describe(include='all')
data = raw_data.drop(['Model'],axis=1)
data.describe(include='all')
# dropped the Model category because it is irrelevant to increasing the accuracy of my model.
data.isnull().sum()
#this shows the missing values in my dataset
data_no_mv = data.dropna(axis=0)
#I drop the missing values here, which is acceptable because it is less than 5% of the observations.
data_no_mv.describe(include='all')<jupyter_output><empty_output><jupyter_text>PDFs Here I check the Probability Distribution Functions (PDF) of the Independent Variables Price, Year, Mileage, and Engine Volume to identify and weed out the Outliers. They can adversely affect the accuracy of my Regression model because a Regression attempts to draw a line closest to all the data; including the Outliers might inflate/deflate my model.<jupyter_code>sns.distplot(data_no_mv['Price'])
q = data_no_mv['Price'].quantile(0.99)
data_1 = data_no_mv[data_no_mv['Price']<q]
data_1.describe(include='all')
# I decided to exclude the observations in the 99th percentile and above to get rid of the Outliers.
sns.distplot(data_1['Price'])
# Now the Price variable only includes observations up to the 98th percentile and has much fewer Outliers.
sns.distplot(data_no_mv['Mileage'])
q = data_1['Mileage'].quantile(0.99)
data_2 = data_1[data_1['Mileage']<q]
# Similar to the Price variable, I decided to exclude the observations in the 99th percentile and beyond to remove the Outliers.
sns.distplot(data_2['Mileage'])
sns.distplot(data_no_mv['EngineV'])
# The PDF looks unusual compared to the previous two.
data_3 = data_2[data_2['EngineV']<6.6]
# After research, I found out that the normal interval of the Engine Volume falls between 06. to 6.5.
# The observations beyond 6.5 are mostly 99.99 - a variable that was used in the past to label missing values. It is a bad idea to label missing values in this manner now.
# I decided to remove such observations as they are Outliers.
sns.distplot(data_3['EngineV'])
sns.distplot(data_no_mv['Year'])
# Most cars are newer but there are a few vintage cars in the variable.
q = data_3['Year'].quantile(0.01)
data_4 = data_3[data_3['Year']>q]
# I decided to remove the 1st percentile and keep the rest
sns.distplot(data_4['Year'])
data_cleaned = data_4.reset_index(drop=True)
#I reset the index to completely forget the old index.
data_cleaned.describe(include='all')
# This excludes ~250 problematic observations that could've hindered the accuracy of my model if left unchecked.<jupyter_output><empty_output><jupyter_text>Checking the OLS assumptions  Distribution<jupyter_code>f, (ax1, ax2, ax3) = plt.subplots(1, 3, sharey=True, figsize =(15,3))
ax1.scatter(data_cleaned['Year'],data_cleaned['Price'])
ax1.set_title('Price and Year')
ax2.scatter(data_cleaned['EngineV'],data_cleaned['Price'])
ax2.set_title('Price and EngineV')
ax3.scatter(data_cleaned['Mileage'],data_cleaned['Price'])
ax3.set_title('Price and Mileage')
plt.show()
# These are not linear regressions and shows that I should first transform one or more variables to run the Regression.
sns.distplot(data_cleaned['Price'])
#Here I check the distribution of the dependent variable Price.
log_price = np.log(data_cleaned['Price'])
# Here I used the log transformation to fix heteroscedasticity and remove outliers from the variable Price.
data_cleaned['log_price'] = log_price
data_cleaned
f, (ax1, ax2, ax3) = plt.subplots(1, 3, sharey=True, figsize =(15,3))
ax1.scatter(data_cleaned['Year'],data_cleaned['log_price'])
ax1.set_title('Log Price and Year')
ax2.scatter(data_cleaned['EngineV'],data_cleaned['log_price'])
ax2.set_title('Log Price and EngineV')
ax3.scatter(data_cleaned['Mileage'],data_cleaned['log_price'])
ax3.set_title('Log Price and Mileage')
plt.show()
# After performing the log transformation on Price, the PDFs now show a linear regression line.
data_cleaned = data_cleaned.drop(['Price'],axis=1)
# Here I dropped the variable Price and replaced it with log_Price because the former has no statistical significance to my model.<jupyter_output><empty_output><jupyter_text>Multicollinearity<jupyter_code>data_cleaned.columns.values
from statsmodels.stats.outliers_influence import variance_inflation_factor
variables = data_cleaned[['Mileage','Year','EngineV']]
vif = pd.DataFrame()
vif["VIF"] = [variance_inflation_factor(variables.values, i) for i in range(variables.shape[1])]
vif["features"] = variables.columns
# Through Statsmodels, I used the Variance Inflation Factor here to check for multicollinearity in my variables.
# While I expect multicollinearity in my data, I wanted to check the variables the introduce unnacceptable correlation to my model; they have high VIFs.
vif
data_no_multicollinearity = data_cleaned.drop(['Year'],axis=1)
# Dropped 'Year' because it has an unacceptably high VIF and is therefore a feature that introduces correlation in my data
data_with_dummies = pd.get_dummies(data_no_multicollinearity, drop_first=True)
# This identifies categorical variables and creates dummies automatically to avoid multicollinearity in my Model
data_with_dummies.head()
data_with_dummies.columns.values
cols = ['log_price', 'Mileage', 'EngineV', 'Brand_BMW',
       'Brand_Mercedes-Benz', 'Brand_Mitsubishi', 'Brand_Renault',
       'Brand_Toyota', 'Brand_Volkswagen', 'Body_hatch', 'Body_other',
       'Body_sedan', 'Body_vagon', 'Body_van', 'Engine Type_Gas',
       'Engine Type_Other', 'Engine Type_Petrol', 'Registration_yes']
data_preprocessed = data_with_dummies[cols]
data_preprocessed.head()
# Here I arranged the data into a table.<jupyter_output><empty_output><jupyter_text>Training my Regression Model<jupyter_code>targets = data_preprocessed['log_price']
inputs = data_preprocessed.drop(['log_price'], axis=1)
# I removed log_price in the inputs to exclude the transformed dependent variable from my inputs.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler ()
scaler.fit(inputs)
inputs_scaled = scaler.transform(inputs)
# This standardizes my inputs; in other words, it subtractrs the mean and divide by the standard deviation from each observation.
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(inputs_scaled, targets, test_size=0.2, random_state=365)
# I did this to avoid overfitting my model to my data.
# The default setting of the train-test split is 75-25, but here I chose 80-20.
# I used 'random_state' to ensure that I get the same random shuffle every time I split my data.

reg = LinearRegression()
reg.fit(x_train, y_train)
y_hat = reg.predict(x_train)
plt.scatter(y_train, y_hat)
plt.xlabel('Targets(y_train)', size=20)
plt.ylabel('Predictions(y_hat)', size=20)
plt.xlim(6,13)
plt.ylim(6,13)
plt.show()<jupyter_output><empty_output><jupyter_text>Using Residuals to check the Model<jupyter_code>sns.distplot(y_train-y_hat)
plt.title("Residuals PDF", size=20)
#to check whether the Residuals is normally distributed and the variability of the outcome
reg.score(x_train, y_train)
reg.intercept_
# The intercept or bias calibrates the model: without it, each feature will be off the mark.
reg.coef_
reg_summary=pd.DataFrame(inputs.columns.values, columns=['Features'])
reg_summary['Weights']=reg.coef_
reg_summary

# A feature with a coefficient of 0 means that it has no significance to the model.
#to know the categorical variables of my features
data_cleaned['Brand'].unique()
data_cleaned['Body'].unique()
data_cleaned['Engine Type'].unique()
data_cleaned['Registration'].unique()<jupyter_output><empty_output><jupyter_text>Testing my Model<jupyter_code>y_hat_test = reg.predict(x_test)
plt.scatter(y_test, y_hat_test, alpha=0.2)
plt.xlabel('Targets(y_test)', size=20)
plt.ylabel('Predictions(y_hat_test)', size=20)
plt.xlim(6, 13)
plt.ylim(6, 13)
plt.show()
df_pf = pd.DataFrame(np.exp(y_hat_test), columns=['Prediction'])
#this returns the exponential of Y Hat Test and removes the log. 
df_pf.head()
y_test = y_test.reset_index(drop=True)
y_test.head()
df_pf['Target'] = np.exp(y_test)
df_pf
df_pf['Residual'] = df_pf['Target'] - df_pf['Prediction']
df_pf['Difference%'] = np.absolute(df_pf['Residual']/df_pf['Target']*100)
df_pf
pd.options.display.max_rows = 999
pd.set_option('display.float_format', lambda x: '%2f' % x)
df_pf.sort_values(by=['Difference%'])
# This table shows the difference in percetage of the prediction and the target using the test data.
# I included the Residuals because examining them as the same as examining the heart of the alogirthm.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10563052582596556, 'fraction_numerical': 0.015006979990693345, 'mean_word_length': 6.731115107913669, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '1961', 'license': 'Apache-2.0', 'n_tokens_mistral': 2900, 'n_tokens_neox': 2764, 'n_words': 858, 'path': 'Linear Regression w sklearn Case Exercise.ipynb', 'repo_name': 'jovemmanuelre/Practical-Case-Example-Regression-with-sklearn', 'size': '8596'}