<jupyter_start><jupyter_text>Supervised Machine Learning in Python  Supervised Machine Learning<jupyter_code># library for basic program to deploy data and manipulation data
import pandas as pd
import numpy as np 

# library for splot the data and metric analysis
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score 

# library for preparing machine learning 
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, GridSearchCV

# Machine Learning Library 
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.neighbors import KNeighborsRegressor
from sklearn.model_selection import RandomizedSearchCV
from sklearn.preprocessing import StandardScaler<jupyter_output><empty_output><jupyter_text>Import Data<jupyter_code>df = pd.read_csv('reg_data.csv')
# for quick analysis data
df.head()
df.columns<jupyter_output><empty_output><jupyter_text>1. Simpled Supervised Macine Learning<jupyter_code>y = df[['Price']]
x = df[['Rooms', 'Bedroom2', 'Bathroom', 'Landsize']]
y.shape
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.25, random_state = 10)<jupyter_output><empty_output><jupyter_text>Decision Tree Regression<jupyter_code>dt = DecisionTreeRegressor()
dt.fit(x_train, y_train)
### cara melakukan prediksi ke data baru
dt.predict(x_test.head())
y_pred = dt.predict(x_test)
print('RMSE:', np.sqrt(mean_squared_error(y_test, y_pred)))
print('R2 Score:', r2_score(y_test, y_pred))<jupyter_output>RMSE: 681502.4516959714
R2 Score: 0.0051562387048949<jupyter_text>Random Forest Regression<jupyter_code>rf = RandomForestRegressor()
rf.fit(x_train, y_train.values.ravel())
y_pred = rf.predict(x_test)

print('RMSE:', np.sqrt(mean_squared_error(y_test, y_pred)))
print('R2 Score:', r2_score(y_test, y_pred))<jupyter_output>RMSE: 600403.7945485302
R2 Score: 0.22784072516907183<jupyter_text>K-Nearest-Neighbor Regression<jupyter_code>knn = KNeighborsRegressor()
knn.fit(x_train, y_train.values.ravel())
y_pred = knn.predict(x_test)

print('RMSE:', np.sqrt(mean_squared_error(y_test, y_pred)))
print('R2 Score:', r2_score(y_test, y_pred))<jupyter_output>RMSE: 621528.3788105733
R2 Score: 0.17254961548116332<jupyter_text>In Simple Supervised Regression that we know `Random Forest Regression` have high accuracy, so this model is the best model.But as we know `R2 Score` must have value between 0-1, that means more closer to value 1 its the better match between prediction and real value data.   2. Intermediate Supervised Regression  1. Extra Model with Encoder<jupyter_code># split to chategorical and numeric coloumn
categorical = ['Type', 'CouncilArea', 'Regionname']
numeric = ['Rooms', 'Distance', 'Bedroom2', 'Bathroom', 'Car', 'Landsize', 'BuildingArea', 
      'YearBuilt', 'Lattitude', 'Longtitude', 'Propertycount']
x = df[['Type', 'Postcode', 'CouncilArea', 'Regionname','Rooms', 'Distance', 'Bedroom2', 'Bathroom', 'Car', 'Landsize', 'BuildingArea', 
      'YearBuilt', 'Lattitude', 'Longtitude', 'Propertycount']]

y = df[['Price']]

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.25, random_state = 6)
x_test
cat_transform = OneHotEncoder(handle_unknown = 'ignore', sparse=False)

preprocessor = ColumnTransformer(
    transformers = [
        ("categorical_encoder", cat_transform, categorical)
    ]
)

dt_reg = Pipeline(
    steps=[("preprocessor", preprocessor), ("regressor", DecisionTreeRegressor(random_state = 6))]
)

rf_reg = Pipeline(
    steps=[("preprocessor", preprocessor), ("regressor", RandomForestRegressor(random_state = 6))]
)

kn_reg = Pipeline(
    steps=[("preprocessor", preprocessor), ("regressor", KNeighborsRegressor())]
)

def fit_evaluate(model):
    model.fit(x_train, y_train)
    y_pred = model.predict(x_test)
    print('RMSE:', np.sqrt(mean_squared_error(y_test, y_pred)))
    print('R2 Score:',  r2_score(y_test, y_pred))
fit_evaluate(dt_reg)
fit_evaluate(rf_reg)
fit_evaluate(kn_reg)<jupyter_output>RMSE: 457710.5033009125
R2 Score: 0.4662280551910072<jupyter_text>In this regression that we know `Random Forest Regression` have high accuracy, so this model is the best model just like previous regression without Encoder.But we can re-process the data to be even better by giving a scale to every features in the data, this process to get more accuracy and get more predictable data.  2. Extra Model with Scalling<jupyter_code>cat_transform = OneHotEncoder(handle_unknown = 'ignore', sparse=False)

preprocessor = ColumnTransformer(
    transformers = [
        ("categorical_encoder", cat_transform, categorical),
        ("minmax_scaler", StandardScaler(), numeric)
    ]
)

dt_reg = Pipeline(
    steps=[("preprocessor", preprocessor), ("regressor", DecisionTreeRegressor())]
)

rf_reg = Pipeline(
    steps=[("preprocessor", preprocessor), ("regressor", RandomForestRegressor())]
)

kn_reg = Pipeline(
    steps=[("preprocessor", preprocessor), ("regressor", KNeighborsRegressor())]
)
fit_evaluate(dt_reg)
fit_evaluate(rf_reg)
fit_evaluate(kn_reg)<jupyter_output>RMSE: 326097.54408160446
R2 Score: 0.7290625478229362<jupyter_text>And again `Random Forest Regression` have high accuracy, so this model is the best model just like previous process.We must get more estimator, so with thah estimator we can get more detail from the data and process.  Hyperparameter Tuning  Random Forest<jupyter_code>RandomForestRegressor().get_params()
# Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 200, stop = 700, num = 50)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4]
# Method of selecting samples for training each tree
bootstrap = [True, False]
# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
               'bootstrap': bootstrap}
rf = RandomForestRegressor(n_jobs = -1, random_state = 10)
rf_random = RandomizedSearchCV(estimator = rf, 
                              param_distributions = random_grid,
                              n_iter = 10,
                              cv = 5,
                              random_state = 10,
                              n_jobs = -1)

rf_reg = Pipeline(
    steps=[("preprocessor", preprocessor), ("regressor", rf_random)]
)

rf_reg.fit(x_train, y_train.values.ravel())
rf_random.best_params_
final_rf = RandomForestRegressor(n_estimators = 485, min_samples_split = 2, min_samples_leaf = 2,
                                max_features = 'sqrt', max_depth = 80, bootstrap = False,
                                n_jobs = -1,random_state = 10)

final_reg = Pipeline(
    steps=[("preprocessor", preprocessor), ("regressor", final_rf)]
)
final_reg.fit(x_train, y_train.values.ravel())
y_pred = final_reg.predict(x_test)
print('RMSE:', np.sqrt(mean_squared_error(y_test, y_pred)))
print('R2 Score:', r2_score(y_test, y_pred))<jupyter_output>RMSE: 240961.3410698212
R2 Score: 0.8520657515904142<jupyter_text>Decision Tree<jupyter_code>DecisionTreeRegressor().get_params()
# Number of features to consider at every split
criterion = ['mse']
max_features = ['auto', 'sqrt']
max_leaf_nodes = [1,2,4]
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4]

# Method of selecting samples for training each tree
# Create the random grid
random_grid = {'max_features': max_features,
               'criterion': criterion,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
               'max_leaf_nodes': max_leaf_nodes
               }
dt = DecisionTreeRegressor(random_state = 10)
dt_random = RandomizedSearchCV(estimator = dt, 
                              param_distributions = random_grid,
                              n_iter = 10,
                              cv = 5,
                              random_state = 10,
                              n_jobs = -1)

dt_reg = Pipeline(
    steps=[("preprocessor", preprocessor), ("regressor", dt_random)]
)

dt_reg.fit(x_train, y_train.values.ravel())
dt_random.best_params_
final_dt = DecisionTreeRegressor(min_samples_split= 2,min_samples_leaf= 4,max_leaf_nodes= 4,max_features='sqrt',
                                 max_depth=70,criterion='mse',random_state = 10)

final_reg = Pipeline(
    steps=[("preprocessor", preprocessor), ("regressor", final_dt)]
)
final_reg.fit(x_train, y_train.values.ravel())
y_pred = final_reg.predict(x_test)
print('RMSE:', np.sqrt(mean_squared_error(y_test, y_pred)))
print('R2 Score:', r2_score(y_test, y_pred))<jupyter_output>RMSE: 491364.481717025
R2 Score: 0.38484932199195787<jupyter_text>K-Nearest Neighbor<jupyter_code>KNeighborsRegressor().get_params()
# Number of features to consider at every split
algorithm = ['auto']
leaf_size = [ 2, 5, 30 ]
metric = ['minkowski']
metric_params = [None]
n_jobs = [-1,1,10]
n_neighbors = [1,5,10]
p = [2,4,10]
weights = ['uniform', 'distance']
# Create the random grid
random_grid = {'algorithm': algorithm,
               'leaf_size': leaf_size,
               'metric': metric,
               'metric_params': metric_params,
               'n_jobs': n_jobs,
               'n_neighbors': n_neighbors,
               'p' : p,
               'weights' : weights
               }
kn = KNeighborsRegressor()
kn_random = RandomizedSearchCV(estimator = kn, 
                              param_distributions = random_grid,
                              n_iter = 10,
                              cv = 5,
                              random_state = 10
                              )

kn_reg = Pipeline(
    steps=[("preprocessor", preprocessor), ("regressor", kn_random)]
)

kn_reg.fit(x_train, y_train.values.ravel())
kn_random.best_params_
final_kn = KNeighborsRegressor(weights = 'distance',p=2, n_neighbors=5,n_jobs = 10,metric_params=None,
                                 metric = 'minkowski',leaf_size = 5, algorithm='auto')

final_reg = Pipeline(
    steps=[("preprocessor", preprocessor), ("regressor", final_kn)]
)
final_reg.fit(x_train, y_train.values.ravel())
y_pred = final_reg.predict(x_test)
print('RMSE:', np.sqrt(mean_squared_error(y_test, y_pred)))
print('R2 Score:', r2_score(y_test, y_pred))<jupyter_output>RMSE: 324223.94597873895
R2 Score: 0.7321669539364548
{'fraction_non_alphanumeric': 0.11035835264753076, 'fraction_numerical': 0.036726689249420574, 'mean_word_length': 4.162908421537046, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '82799', 'license': 'MIT', 'n_tokens_mistral': 4035, 'n_tokens_neox': 3660, 'n_words': 879, 'path': 'Machine Learning Python.ipynb', 'repo_name': 'AriefRM/Supervised-Machine-Learning', 'size': '11218'}