<jupyter_start><jupyter_text>Ensemble methods. Boosting AdaBoostAdaBoost consists of following steps:* initialize weights to $\frac{1}{N}$, where $N$ is the number of datapoints,* loop until   $\varepsilon_{t}<\frac{1}{2}$  or maximum number of iteration is reached,* train classifier on ${S,w^{(t)}}$ and get a hypothesis $h_{t}(x_{n})$ for datapoints $x_{n}$,* compute error $\varepsilon_{t}=\sum_{n=1}^{N}w_{n}^{(t)}I(y_{n}\neq h_{t}(x_{n}))$,       * set $\alpha_{t}=\log(\frac{1-\varepsilon_{t}}{\varepsilon_{t}})$.  * update weights $w_{n}^{(t+1)}=\frac{w_{n}^{(t)}\exp{\alpha_{t}I(y_{n}\neq h_{t}(x_{n}))}}{Z_{t}}$,  where $Z_{t}$ is a normalization constant,* output $f(X)=\text{sign}(\sum_{t=1}^{T}\alpha_{t}h_{t}(x))$.  Example taken from Marsland, Machine Learning: https://seat.massey.ac.nz/personal/s.r.marsland/MLBook.html.First, we need to import libraries:<jupyter_code>import numpy as np
from sklearn.tree import DecisionTreeClassifier<jupyter_output><empty_output><jupyter_text>Variables that are used by the classifier:<jupyter_code>from sklearn.datasets import load_iris
import numpy as np
from sklearn.model_selection import train_test_split


labels = 2
dimension = 2

iris = load_iris()
data_set = iris.data
labels = iris.target

train_set, test_set, train_labels, test_labels = train_test_split(data_set, labels, 
                                                                            test_size=0.33, random_state=42)<jupyter_output><empty_output><jupyter_text>Weights initialization:<jupyter_code>number_of_iterations = 10
weights = np.ones((len(test_set),)) / len(test_set)<jupyter_output><empty_output><jupyter_text>The data set looks as:<jupyter_code>import matplotlib.pyplot as plt

sizes = np.ones(len(test_set))*5

plt.scatter(test_set[:, 1], test_set[:, 2], c=test_labels, s=sizes, cmap='autumn')
plt.show()
def train_model(classifier, weights):
    return classifier.fit(X=test_set, y=test_labels, sample_weight=weights)<jupyter_output><empty_output><jupyter_text>Accuracy vector calculation for the weights: 0 - don't change the weight, 1 - change it.<jupyter_code>def calculate_accuracy_vector(predicted, labels):
    result = []
    for i in range(len(predicted)):
        if predicted[i] == labels[i]:
            result.append(0)
        else:
            result.append(1)
    return result<jupyter_output><empty_output><jupyter_text>Calculate the error rate $\varepsilon_{t}=\sum_{n=1}^{N}w_{n}^{(t)}I(y_{n}\neq h_{t}(x_{n}))$:<jupyter_code>def calculate_error(weights, model):
    predicted = model.predict(test_set)
    return np.dot(weights,calculate_accuracy_vector(predicted, test_labels))<jupyter_output><empty_output><jupyter_text>Calculate the $\alpha_{t}=\log(\frac{1-\varepsilon_{t}}{\varepsilon_{t}})$:<jupyter_code>def set_alpha(error_rate):
    return np.log((1-error_rate)/error_rate)<jupyter_output><empty_output><jupyter_text>Calculate the new weights $w_{n}^{(t+1)}=\frac{w_{n}^{(t)}\exp{\alpha_{t}I(y_{n}\neq h_{t}(x_{n}))}}{Z_{t}}$:<jupyter_code>def set_new_weights(old_weights, alpha, model):
    new_weights = old_weights * np.exp(np.multiply(alpha,calculate_accuracy_vector(model.predict(test_set), test_labels)))
    Zt = np.sum(new_weights)
    return new_weights / Zt<jupyter_output><empty_output><jupyter_text>Now, it's time to run the code and check the weights:<jupyter_code>classifier = DecisionTreeClassifier(max_depth=1, random_state=1)
classifier.fit(X=train_set, y=train_labels)
alphas = []
classifiers = []
for iteration in range(number_of_iterations):
    model = train_model(classifier, weights)
    error_rate = calculate_error(weights, model)
    alpha = set_alpha(error_rate)
    weights = set_new_weights(weights, alpha, model)
    alphas.append(alpha)
    classifiers.append(model)

print(weights)
weights = (weights/np.min(weights))*2
plt.scatter(test_set[:, 1], test_set[:, 2], c=test_labels, s=weights, cmap='autumn')


plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15552178318135765, 'fraction_numerical': 0.008105369807497468, 'mean_word_length': 5.989380530973452, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '612926', 'license': 'MIT', 'n_tokens_mistral': 1411, 'n_tokens_neox': 1368, 'n_words': 249, 'path': 'ML1/ensemble/073Ensemble_Boosting.ipynb', 'repo_name': 'DevilWillReign/ML2022', 'size': '3948'}