<jupyter_start><jupyter_text>Chapter 2. Training Machine Learning Algorithms for Classication.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sbn<jupyter_output><empty_output><jupyter_text>![Iris](http://www.canadianflowerdelivery.com/assets/images/content/iris_195x195.jpg) We will use `Perceptron` algorithm for binary classification problem, based on Iris dataset.<jupyter_code>df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data', header=None)
df.tail()
df.describe()<jupyter_output><empty_output><jupyter_text>Dataset Descripion* **Column 0** - x0 sepal length * **Column 2** - x1 petal lenght* **Column 4** - y class of flower (label)The feature vector `x` would contain of only two features: `sepal length` and `petal length`.  Dataset Preparation and VisualizationLet's plot features on 2D graph.<jupyter_code># prepare labels array
y = df.iloc[0:100, 4].values
# transform label array from text to [-1, 1] values
y_train = np.where(y == 'Iris-setosa', -1, 1)
# prepare feature matrix
X_train = df.iloc[0:100, [0, 2]].values
# prepare visualization
plt.scatter(X_train[:50, 0], X_train[:50, 1], color='red', marker='o', label='setosa')
plt.scatter(X_train[50:100, 0], X_train[50:100, 1], color='blue', marker='x', label='versicolor')
plt.xlabel('petal length')
plt.ylabel('sepal length')
plt.legend(loc='upper left')
plt.show()<jupyter_output><empty_output><jupyter_text>From the plot we can see, that classes are *linearly separable*, so we can apply **Perceptron** model.  Perceptron Let's implement simple percepton model.<jupyter_code>def activation_function(z):
    return np.where(z >= 0, 1, -1)

def add_bias_column(X):
    (m, n) = X.shape
    X = np.concatenate((np.ones((m, 1)), X), 1)
    return X
    
def perceptron(X, w):
    X = add_bias_column(X)
    z = np.dot(X, w)
            
    return activation_function(z)<jupyter_output><empty_output><jupyter_text>Try it with a randomly generated weights:<jupyter_code>def random_weights(features_dimention):
    return np.random.rand(1 + features_dimention) / 32

def zero_weights(features_dimention):
    return np.zeros(1 + features_dimention)

w_random = random_weights(2)

print('random small weights', w_random)<jupyter_output>random small weights [ 0.01829631  0.03121418  0.02712658]<jupyter_text>Run perceptron and check it's current predictions:<jupyter_code>prediction = perceptron(X_train, w_random)
print(prediction)<jupyter_output>[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]<jupyter_text>As you can see, current classifier predicts everything as `virginica`, which is not true. Let's introduce some functions, we can measure errors and accuracy of classifirer:<jupyter_code>def error(predicted, expected):
    return np.subtract(predicted, expected)

def error_count(predicted, expected):
    return np.count_nonzero(error(predicted, expected))

def minimal_square_error(predicted, expected):
    m = predicted.size
    return (np.sum(np.power(np.subtract(predicted, expected), 2)) / 2 * (m))

def classifier_accuracy(predicted, expected):
    m = predicted.size
    return error_count(predicted, expected) / m<jupyter_output><empty_output><jupyter_text>Misclassified samples<jupyter_code>print(error_count(prediction, y_train))<jupyter_output>50<jupyter_text>Minimal squares error<jupyter_code>print(minimal_square_error(prediction, y_train))<jupyter_output>10000.0<jupyter_text>Classifier accuracy<jupyter_code>print(classifier_accuracy(prediction, y_train))<jupyter_output>0.5<jupyter_text>Perceptron Training As we can see from figures above, the performance on given model is very low. Our goal, is to adjust model parameters `w`, for better classification results. The learning algorithm is based on adjustion of `weights` after training session, based on error given by classifier. \begin{align}&\omega_j := \omega_j  + \Delta\omega_j\end{align}\begin{align}&\Delta\omega_j := \alpha (y^i - \hat{y}^i) x_j^j\end{align} Where $\alpha$ - is a constant, called learning rate. We run our training routine until we see good enought prediction results. Each cycle of training is called `epoch`.<jupyter_code>def run_perceptron(X, y, w):
    predicted = perceptron(X, w)  
    return (
        w,
        error(y, predicted),
        error_count(y, predicted)
    )

def adjust_weigths(alpha, X, w, error):
    X = add_bias_column(X)
    delta_w = np.dot(np.dot(error, X), alpha)
    
    return np.add(w, delta_w)

def select_model(train_results):
    for result in train_results:
        if result[1] == 0: 
            return result[0]

def train(alpha, epochs_count, X, y):
    results = []

    w_local = random_weights(X.shape[1])
    
    for epoch in range(epochs_count):
        result = run_perceptron(X, y, w_local)
        # adjust the model, based on currents results and start next epoch..
        w_local = adjust_weigths(alpha, X, w_local, result[1])            
        results.append(result)
    
    return results
def plot_training_results(results, dim, label):
    errors = list(map(lambda r: r[dim], results))
    
    plt.plot(range(1, len(errors) + 1), errors, marker='o')
    plt.xlabel('Epochs')
    plt.ylabel(label)
    plt.show()<jupyter_output><empty_output><jupyter_text>Alpha = 0.01, Epochs = 10<jupyter_code>results_1 = train(0.01, 10, X_train, y_train);
model_1 = select_model(results_1)
print(model_1)

plot_training_results(results_1, 2, 'Misclassification Errors')<jupyter_output><empty_output><jupyter_text>Alpha = 0.03, Epochs = 10<jupyter_code>results_2 = train(0.03, 10, X_train, y_train)
plot_training_results(results_2, 2, 'Misclassification Errors')<jupyter_output><empty_output><jupyter_text>Alpha = 0.1, Epochs = 10<jupyter_code>results_3 = train(0.1, 10, X_train, y_train)
plot_training_results(results_3, 2, 'Misclassification Errors')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11621755662092909, 'fraction_numerical': 0.040006612663250125, 'mean_word_length': 4.902439024390244, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 53, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '450596', 'license': 'MIT', 'n_tokens_mistral': 2340, 'n_tokens_neox': 2110, 'n_words': 584, 'path': 'notebooks/.ipynb_checkpoints/Chapter2_Train_Machine_Learning_Algorithms_For_Classification-checkpoint.ipynb', 'repo_name': 'alexbeletsky/ml-python', 'size': '6049'}