<jupyter_start><jupyter_text>1- Data loading and first overview  1.1- Loading the Data<jupyter_code># importing the necessary libraries
import numpy as np
import pandas as pd
from pandas import Series,DataFrame
import matplotlib.pyplot as plt
import seaborn as sns

# setting white grid background
sns.set_style('whitegrid')
#%matplotlib notebook
%matplotlib inline<jupyter_output><empty_output><jupyter_text>1.2- Overview of the Data<jupyter_code>from google.colab import drive
drive.mount('/content/drive')

data = pd.read_csv(r'/content/drive/MyDrive/ML_exercise/EX 02/ML4Eng_I_dataset_energy_measurement.txt')
# statistical analysis of the dataset
data.describe()<jupyter_output><empty_output><jupyter_text>2- Visualisation >Data Visualization has the power to tell data-driven stories while allowing people to see patterns and relationships found in data. We can figure out what is the ralation between features and output. Is it linear or not? Then, we can decide what type of Machine Learning approach to use on our dataset. Additionally, in some cases we can figure it out that out data set has normal distribution or it's bias ot the outliers.<jupyter_code># plotting the target variable "Energy requirement"
%matplotlib inline
plt.figure(figsize=(10,7))
plt.hist(data.Energy_Requirement,bins=20, range = (data.Energy_Requirement.min(), data.Energy_Requirement.max()))
plt.xlabel('Energy requirement [kJ]')
plt.ylabel('Total quantity')
# plotting the attribute "Path"
%matplotlib inline
plt.figure(figsize=(10,7))
plt.hist(data.Path,bins = 100, range = (data.Path.min(), data.Path.max()))
plt.xlabel('Path [mm]')
plt.ylabel('Total quantity')
# plotting the attribute "Axis"
%matplotlib inline
plt.figure(figsize=(10,7))
plt.hist(data.Axis,bins =9, range = (data.Axis.min(), data.Axis.max()),color='G')
plt.xlabel('Axis feature')
plt.ylabel('Total quantity')
# plotting the attribue "Feed"

%matplotlib inline
plt.figure(figsize=(10,7))
plt.hist(data.Feed,bins =20, range = (data.Feed.min(), data.Feed.max()),color='r')
plt.xlabel('Feed feature')
plt.ylabel('Total quantity')<jupyter_output><empty_output><jupyter_text>3- Missing Values and Outliers  3.1- Handling missing values<jupyter_code># Before we deal with missing values we visualize the first 10 instances of the data set.
# Missing values can be recognized here as NaN.
data.head(10)<jupyter_output><empty_output><jupyter_text>> The `dropna()` function removes all rows with missing values.<jupyter_code>data=data.dropna(axis=0)

# After the removal of missing values we visualize the first 10 instances of the data set again.
data.head(10)<jupyter_output><empty_output><jupyter_text>3.2- Handling outliers<jupyter_code># Before removing the ouliers we analize our data set with the desrcibe() method.
data.describe()<jupyter_output><empty_output><jupyter_text>> Removing all outliers for the attribute "Feed".<jupyter_code># Values of features outside the range of the known to us are treated as 'Outliers' and removed
# We only include those values of the feature that lie in the particular ranges of the feature
data = data.loc[(data.Axis >= 1) & (data.Axis <= 3) & 
            (data.Path >= -60) & (data.Path <= 60) & 
            (data.Feed >= 500) & (data.Feed <= 3000)        
           ]
# After removing the ouliers we analyze the data set again.
data.describe()<jupyter_output><empty_output><jupyter_text>> We can see that the amount of min and max in all features changed. It means the outliers, which exceed the maximum and was under minimum, ommited.   4- Splitting of the data set   4.1 Seperation features and target variable<jupyter_code># We separate the features (axis, feed, distance) and store it in 'X_multi' then we store our target varible (energy) in 'Y_target' to train our model.
X_multi = data.drop('Energy_Requirement', 1)

# energy requirement is our target variable
Y_target = data.Energy_Requirement<jupyter_output><empty_output><jupyter_text>4.2 Splitting the data into training and test sets<jupyter_code># The given dataset is divided into training and test datasets.
import sklearn
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = sklearn.model_selection.train_test_split(X_multi, Y_target, random_state=42)
# Checking the shapes of the datasets so that we dont wrongly fit the data
print(X_train.shape, Y_train.shape, X_test.shape, Y_test.shape)<jupyter_output>(163, 3) (163,) (55, 3) (55,)<jupyter_text>> The `train_test_split()` allowed inputs are lists, numpy arrays, scipy-sparse matrices or pandas dataframes. The following arguments could be given:- test_size: as default it's None, means it is 0.25 percent of the whole data set.- train_size: as default it's None, the value is automatically set to the complement of the test size. - random_state: as default it's None, use the global random state instance from numpy.random. Calling the function multiple times will reuse the same instance, and will produce different results.- Shuffle: as default it's True, means shuffle data set before spllitting.- stratify: as default it's None, means each set contains approximately the same percentage of samples of each target class as the complete set.>In case of splitting data in train and test set with the portion of 50/50 (same size of the data sets):- disadvantages: It may cause underfitting. underfitting happens whenever the model couldn't figure out the relation between feature and output correctly and we may see low performance on both test and train sets, Specially in small data sets.- advantages: In large data sets, we can have faster training procedure.>splitting data in train and test set with the portion of 20/80:- disadvantages: it may cause overfitting, specially in large data sets. We can find it regarding the good performance on training data set, but a poor result on the test set.- advantages: Avoid underfitting. We can make sure that the amount of data for learning statistical realtion between data points is enough.  5- Linear regression  5.1- Import the library<jupyter_code>from sklearn.linear_model import LinearRegression<jupyter_output><empty_output><jupyter_text>5.2- Training of the model<jupyter_code>lreg = LinearRegression()
lreg.fit(X_train, Y_train)<jupyter_output><empty_output><jupyter_text>5.3- Prediction<jupyter_code>pred_train = lreg.predict(X_train) # prediction of the training data
pred_test = lreg.predict(X_test) # prediction of unseen data<jupyter_output><empty_output><jupyter_text>5.4- Calculate the different losses (Least Square Error, Mean Square Error)<jupyter_code>from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_error

# We calulate the error for the training and test datasets

# Training data
MSE_linear_Train_Data = mean_squared_error(Y_train, pred_train)
MAE_linear_Train_Data = mean_absolute_error(Y_train, pred_train)

print("The Mean Square Error on the training data is:", MSE_linear_Train_Data)
print("The Mean Absolute Error on the training data is:", MAE_linear_Train_Data)


# Test data / unseen data
MSE_linear_Test_Data = mean_squared_error(Y_test, pred_test)
MAE_linear_Test_Data = mean_absolute_error(Y_test, pred_test)

print("The Mean Square Error on the Test data is:", MSE_linear_Test_Data)
print("The Mean Absolute Error on the Test data is:", MAE_linear_Test_Data)<jupyter_output>The Mean Square Error on the training data is: 0.011678924858811793
The Mean Absolute Error on the training data is: 0.09121547964727235
The Mean Square Error on the Test data is: 0.013044389500115314
The Mean Absolute Error on the Test data is: 0.09051149946610133<jupyter_text>> To evaluate Regression models' performance, we use Mean Absolute Error (MAE) and Mean Square Error (MSE): - **MAE**: is a measure of errors between paired observations expressing the same phenomenon.- **MSE**: measures the average of the squares of the errors—that is, the average squared difference between the estimated values and the actual value.>So, we can say MAE is the exact difference between the grandtruth and the predicted value of a data point, but MSE is the average of square difference between the real and the predicted output.>MSE and MAE of our regression model indicates that the error is low in both training and test.Additionally, we don't see huge difference the train and test errors. As a result, our model doesn't overfit or underfit.  5.5 Residual plots for Linear Regression<jupyter_code># We want the data points to be scattered around the horizontal line
%matplotlib inline
plt.figure(figsize=(10,7))
train = plt.scatter(pred_train, (pred_train-Y_train), c='b', alpha=0.5)
test = plt.scatter(pred_test, (pred_test-Y_test), c='r', alpha=0.5)
plt.hlines(y=0, xmin=-0.5, xmax=0.5)
plt.legend((train, test), ('Training', 'Test'),loc='lower left')
plt.title('Residual plot for Linear Regressor')
plt.xlabel("Energy_Requirement - Target variable")
plt.ylabel("Residual Value")<jupyter_output><empty_output><jupyter_text>6- Random Forest Regressor  6.1- Import the library<jupyter_code># Import Random Forest Regressor from Sklearn
from sklearn.ensemble import RandomForestRegressor
# Instantiate model with 1000 decision trees
rf = RandomForestRegressor(n_estimators = 1000, random_state = 42)<jupyter_output><empty_output><jupyter_text>6.2- Training of the model<jupyter_code># Train the model on training data
rf.fit(X_train, Y_train)<jupyter_output><empty_output><jupyter_text>6.3- Prediction<jupyter_code># Use the forest's predict method on the test data
rf_pred_train = rf.predict(X_train)
rf_pred_test = rf.predict(X_test)<jupyter_output><empty_output><jupyter_text>6.4- Calculate the different losses (Mean Absolute Error, Mean Square Error)<jupyter_code>from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_error


# We calulate the errors for the training and test datasets.
# We save the errors according to the Linear Regression ("MSE_rf_Test_Data", "MAE_rf_Test_Data").


# Training data
MSE_rf_Train_Data = mean_squared_error(Y_train, rf_pred_train)
MAE_rf_Train_Data = mean_absolute_error(Y_train, rf_pred_train)

print("Mean Square Error on the training data is:", MSE_rf_Train_Data)
print("Mean Absolute Error on the training data is:", MAE_rf_Train_Data)

# Test data / unseen data
MSE_rf_Test_Data = mean_squared_error(Y_test, rf_pred_test)
MAE_rf_Test_Data = mean_absolute_error(Y_test, rf_pred_test)

print("\n""Mean Square Error on the test data is:", MSE_rf_Test_Data)
print("Mean Absolute Error on the test data is:", MAE_rf_Test_Data)<jupyter_output>Mean Square Error on the training data is: 6.316144649036996e-05
Mean Absolute Error on the training data is: 0.0026688087607363385

Mean Square Error on the test data is: 0.00017261970814991598
Mean Absolute Error on the test data is: 0.005879109009090978<jupyter_text>6.5- Residual plots for Random Forest Regressor<jupyter_code># We want the data points to be scattered around the horizontal line
%matplotlib inline
plt.figure(figsize=(10,7))
train = plt.scatter(rf_pred_train, (rf_pred_train-Y_train), c='b', alpha=0.5)
test = plt.scatter(rf_pred_test, (rf_pred_test-Y_test), c='r', alpha=0.5)
plt.hlines(y=0, xmin=-0.5, xmax=0.5)
plt.legend((train, test), ('Training', 'Test'),loc='lower left')

plt.title('Residual plot for Random Forest Regressor')
plt.xlabel("Energy_Requirement - Target variable")
plt.ylabel("Residual Value")<jupyter_output><empty_output><jupyter_text>7- Support Vector Regression (SVR)  7.1- Import librariies<jupyter_code># Import SVR from Sklearn

from sklearn.svm import SVR
svr_rbf = SVR(kernel="rbf")
svr_poly = SVR(kernel="poly")
# try different types of SVR to figure out which one has higher performance<jupyter_output><empty_output><jupyter_text>7.2- Training model<jupyter_code># initialize the model and train it
svr_poly.fit(X_train,Y_train)
svr_rbf.fit(X_train,Y_train)<jupyter_output><empty_output><jupyter_text>7.3- Prediction<jupyter_code># predict the values for the training and test dataset

svr_rbf_pred_train = svr_rbf.predict(X_train) # prediction of the training data (rbf)
svr_rbf_pred_test = svr_rbf.predict(X_test) # prediction of unseen data (rbf)

svr_poly_pred_train = svr_poly.predict(X_train) # prediction of the training data (poly)
svr_poly_pred_test = svr_poly.predict(X_test) # prediction of unseen data (poly)<jupyter_output><empty_output><jupyter_text>7.4- Calculate the different losses (Mean Absolute Error, Mean Square Error)<jupyter_code># load the libraries and calculate the losses.
# save the errors according to the Linear Regression ("MSE_svr_Test_Data", "MAE_svr_Test_Data", etc.).

from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_error

print("############### rbf ###############") 
# Training data
MSE_rbf_svr_Train_Data = mean_squared_error(Y_train, svr_rbf_pred_train)
MAE_rbf_svr_Train_Data = mean_absolute_error(Y_train, svr_rbf_pred_train)

print("Mean Square Error on the training data is:", MSE_rbf_svr_Train_Data)
print("Mean Absolute Error on the training data is:", MAE_rbf_svr_Train_Data)

# Test data / unseen data
MSE_rbf_svr_Test_Data = mean_squared_error(Y_test, svr_rbf_pred_test)
MAE_rbf_svr_Test_Data = mean_absolute_error(Y_test, svr_rbf_pred_test)
        
print("\n""Mean Square Error on the test data is:", MSE_rbf_svr_Test_Data)
print("Mean Absolute Error on the test data is:", MAE_rbf_svr_Test_Data)
print("\n")

print("############### poly ###############") 
MSE_poly_svr_Train_Data = mean_squared_error(Y_train, svr_poly_pred_train)
MAE_poly_svr_Train_Data = mean_absolute_error(Y_train, svr_poly_pred_train)

print("Mean Square Error on the training data is:", MSE_poly_svr_Train_Data)
print("Mean Absolute Error on the training data is:", MAE_poly_svr_Train_Data)

# Test data / unseen data
MSE_poly_svr_Test_Data = mean_squared_error(Y_test, svr_poly_pred_test)
MAE_poly_svr_Test_Data = mean_absolute_error(Y_test, svr_poly_pred_test)
        
print("\n""Mean Square Error on the test data is:", MSE_poly_svr_Test_Data)
print("Mean Absolute Error on the test data is:", MAE_poly_svr_Test_Data)<jupyter_output>############### rbf ###############
Mean Square Error on the training data is: 0.01358848330204856
Mean Absolute Error on the training data is: 0.08171893308578909

Mean Square Error on the test data is: 0.017124636842031218
Mean Absolute Error on the test data is: 0.09279743047965765


############### poly ###############
Mean Square Error on the training data is: 0.013190656018289707
Mean Absolute Error on the training data is: 0.08290647064719127

Mean Square Error on the test data is: 0.016133220188448198
Mean Absolute Error on the test data is: 0.09230687914075869<jupyter_text>7.5- Residual plots for Support Vector Regressor<jupyter_code>#  visualize the residual plot

############################# rbf
'''
As there is not much difference between 'poly' and 'rbf', poly just showed.
%matplotlib inline
plt.figure(figsize=(10,7))
train = plt.scatter(svr_rbf_pred_train, (svr_rbf_pred_train-Y_train), c='b', alpha=0.5)
test = plt.scatter(svr_rbf_pred_test, (svr_rbf_pred_test-Y_test), c='r', alpha=0.5)
plt.hlines(y=0, xmin=-0.5, xmax=0.5)
plt.legend((train, test), ('Training', 'Test'),loc='lower left')

plt.title('Residual plot for Support Vector Regressor (rbf)')
plt.xlabel("Energy_Requirement - Target variable")
plt.ylabel("Residual Value")
'''
############################# poly
%matplotlib inline
plt.figure(figsize=(10,7))
train = plt.scatter(svr_poly_pred_train, (svr_poly_pred_train-Y_train), c='b', alpha=0.5)
test = plt.scatter(svr_poly_pred_test, (svr_poly_pred_test-Y_test), c='r', alpha=0.5)
plt.hlines(y=0, xmin=-0.5, xmax=0.5)
plt.legend((train, test), ('Training', 'Test'),loc='lower left')

plt.title('Residual plot for Support Vector Regressor (poly)')
plt.xlabel("Energy_Requirement - Target variable")
plt.ylabel("Residual Value")<jupyter_output><empty_output><jupyter_text>8- Comparison and results<jupyter_code># visualisation of the results
%matplotlib inline 
plt.figure(figsize=(10,7))
plt.bar(['MSE_LR'],[MSE_linear_Test_Data], color=['#4DBEEE'], label="Mean Square Error on Linear Regressor")
plt.bar(['MSE_SVR'],[MSE_poly_svr_Test_Data], color=['#A2142F'], label="Mean Square Error on SVR")
plt.bar(['MSE_RF'],[MSE_rf_Test_Data], color=['#0072BD'], label="Mean Square Error on Random Forest")

plt.bar(['MAE_LR'],[MAE_linear_Test_Data], color=['#D95319'], label="Mean Absolute Error on Linear Regressor")
plt.bar(['MAE_SVR'],[MAE_poly_svr_Test_Data], color=['#EDB120'], label="Mean Absolute Error on SVR")
plt.bar(['MAE_RF'],[MAE_rf_Test_Data], color=['#77AC30'], label="Mean Absolute Error on Random Forest")

plt.xlabel('Error')
plt.ylabel('Error values')
plt.title('Performance of different regression models on the test data')
plt.legend(loc="upper left")
plt.show()<jupyter_output><empty_output><jupyter_text>>**Result:** The above table indicates that the Random Forest approach has higher performance as its MSE and MAE is the least among the other models (SVR and LR). So we choose Random Forest as the best model.   9- Deployment of the model>Now we want to predict the energy requirement for the following settings of production parameters with the best model. 1. Setting 1: axis = 2, feed = 800 [mm/min], distance = 60 [mm] 2. Setting 2: axis = 3, feed = 2000 [mm/min], distance = 40 [mm] 3. Setting 3: axis = 1, feed = 1200 [mm/min], distance = -20 [mm]<jupyter_code># Deployment of the best model for the production settings.

print("1. Setting 1: axis = 2, feed = 800 [mm/min], distance = 60 [mm]:")
Y_1=svr_poly.predict([[2,800,60]])
print("Energy requirement predicted: ", Y_1,'\n')

print("2. Setting 2: axis = 3, feed = 2000 [mm/min], distance = 40 [mm]:")
Y_2=svr_poly.predict([[3,2000,40]])
print("Energy requirement predicted: ", Y_2,'\n')

print("3. Setting 3: axis = 1, feed = 1200 [mm/min], distance = -20 [mm]:")
Y_3=svr_poly.predict([[1,1200,-20]])
print("Energy requirement predicted: ", Y_3,'\n')<jupyter_output>1. Setting 1: axis = 2, feed = 800 [mm/min], distance = 60 [mm]:
Energy requirement predicted:  [0.04642924] 

2. Setting 2: axis = 3, feed = 2000 [mm/min], distance = 40 [mm]:
Energy requirement predicted:  [0.09202889] 

3. Setting 3: axis = 1, feed = 1200 [mm/min], distance = -20 [mm]:
Energy requirement predicted:  [0.02779883]
{'fraction_non_alphanumeric': 0.10809484873262469, 'fraction_numerical': 0.03455982556554919, 'mean_word_length': 6.24565560821485, 'pattern_counts': {'":': 0, '<': 103, '<?xml version=': 0, '>': 115, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '269175', 'license': 'MIT', 'n_tokens_mistral': 6548, 'n_tokens_neox': 5900, 'n_words': 1928, 'path': 'ex1_Energy Prediction_Regression/ML_ex1_Regression for Energy Prediction.ipynb', 'repo_name': 'YasamanMoradifard/Machine_Learning_1_EX', 'size': '18345'}