<jupyter_start><jupyter_text>Training and Testing Data===================================== To evaluate how well our supervised models generalize, we can split our data into a training and a test set:<jupyter_code>from sklearn.datasets import load_iris
from sklearn.neighbors import KNeighborsClassifier

iris = load_iris()
X, y = iris.data, iris.target

classifier = KNeighborsClassifier()<jupyter_output><empty_output><jupyter_text>* Thinking about how machine learning is normally performed, the idea of a train/test split makes sense. * Real world systems train on the data they have, and as other data comes in (from customers, sensors, or other sources) the classifier that was trained must predict on fundamentally *new* data. * We can simulate this during training using a train/test split - the test data is a simulation of "future data" which will come into the system during production.  Specifically for iris, the labels in iris are sorted, which means that if we split the data using a proportional split, we will get all of specific labels (0 and 1) and very little of another (2). We want to split as illustrated above, but *after* the data has been randomly shuffled.<jupyter_code>y<jupyter_output><empty_output><jupyter_text>To get an accurate simulation of the real world, we will shuffle our data then split.<jupyter_code>import numpy as np
rng = np.random.RandomState(0)

permutation = rng.permutation(len(X))
X, y = X[permutation], y[permutation]
print(y)<jupyter_output>[2 1 0 2 0 2 0 1 1 1 2 1 1 1 1 0 1 1 0 0 2 1 0 0 2 0 0 1 1 0 2 1 0 2 2 1 0
 1 1 1 2 0 2 0 0 1 2 2 2 2 1 2 1 1 2 2 2 2 1 2 1 0 2 1 1 1 1 2 0 0 2 1 0 0
 1 0 2 1 0 1 2 1 0 2 2 2 2 0 0 2 2 0 2 0 2 2 0 0 2 0 0 0 1 2 2 0 0 0 1 1 0
 0 1 0 2 1 2 1 0 2 0 2 0 0 2 0 2 1 1 1 2 2 1 1 0 1 2 2 0 1 1 1 1 0 0 0 2 1
 2 0]<jupyter_text>* Now we need to split the data into training and testing. * Luckily, this is a common pattern in machine learning and scikit-learn has a prebuilt function to split data into training and testing for you. * Here we use 50% of the data as training, and 50% testing. 80% and 20% is another common split, but there are no hard and fast rules. * The most important thing is to fairly evaluate your system on data it *has not* seen during training!<jupyter_code>from sklearn.model_selection import train_test_split
train_X, test_X, train_y, test_y = train_test_split(X, y, test_size=0.5, random_state=1999)
print("Labels for training and testing data")
print(train_y)
print(test_y)<jupyter_output>Labels for training and testing data
[0 1 1 2 2 2 0 2 2 1 1 2 2 1 1 1 1 1 0 2 2 2 1 0 2 1 1 2 1 0 2 2 0 2 2 0 2
 1 0 2 0 2 2 2 0 2 0 0 1 0 1 0 2 1 1 0 1 2 1 0 0 0 1 0 1 0 1 0 0 0 0 0 1 1
 2]
[1 0 1 2 2 1 1 0 1 0 0 2 2 2 1 0 2 0 2 1 0 0 0 2 1 1 1 1 1 0 2 2 2 2 0 1 0
 1 2 1 2 2 1 2 2 1 0 2 0 2 1 2 0 0 2 0 2 1 2 1 0 1 0 1 0 2 2 0 0 0 1 0 1 0
 1]<jupyter_text>By evaluating our classifier performance on data that has been seen during training, we could get false confidence in the power of our system. This might lead to putting a system into production which *fails* at predicting new data! It is much better to use a train/test split in order to properly see how your trained model is doing on new data.<jupyter_code>classifier.fit(train_X, train_y)
pred_y = classifier.predict(test_X)
print("Fraction Correct")
print(np.sum(pred_y == test_y) / float(len(test_y)))<jupyter_output>Fraction Correct
0.96<jupyter_text>We can also visualize the correct and failed predictions<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
correct_idx = np.where(pred_y == test_y)[0]
print(correct_idx)
incorrect_idx = np.where(pred_y != test_y)[0]
print(incorrect_idx)
# Plot two dimensions
from itertools import combinations
colors = ["darkblue", "darkgreen", "gray"]

for x, y in combinations(range(4), 2):
    for n, color in enumerate(colors):
        idx = np.where(test_y == n)[0]
        plt.scatter(test_X[idx, x], test_X[idx, y], color=color, 
                    label="Class %s" % str(n))
    plt.scatter(test_X[incorrect_idx, x], test_X[incorrect_idx, y], 
                color="darkred")
    # Make xlim larger to accommodate legend
    plt.xlim(3, 9)
    plt.legend(loc='best')
    plt.title("Iris Classification results")
    plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07740585774058577, 'fraction_numerical': 0.07624360762436076, 'mean_word_length': 3.754696132596685, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 48, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '22983', 'license': 'MIT', 'n_tokens_mistral': 1743, 'n_tokens_neox': 1400, 'n_words': 743, 'path': '3_mage/2.1.1 Training and Testing Data.ipynb', 'repo_name': 'leriomaggio/develer-data-science', 'size': '4302'}