<jupyter_start><jupyter_text>Introduction to Supervised Learning Version 0.1 ***By <NAME> (Northwestern/CIERA)  03 March 2022 Using the [ten hundred most common words](https://xkcd.com/simplewriter/) in the English language,$^\dagger$ I would describe supervised learning as> building an information map to name different groups $^\dagger$ See [Up-goer five](https://xkcd.com/1133/). More technically, in supervised learning we utilize algorithms to construct a map between features and the target classes. The ultimate goal is to provide the model with unclassified sources, and predict their classification. (Hopefully with a high degree of accuracy and small uncertainty).  Adam's 1 slide summary of supervised machine learning:   I. Training Sets The single most important thing worthy of your attention as you become machine learning practicioners is your training set. The algorithm *only* learns what is in the training set. "Garbage in = garbage out" Sample selection bias: How does one build a training set? Need labels. Labels can be acquired via... | Method | Challenge || :---:    | :---:       || Ground Truth | Impossible(?) in astronomy || Spectroscopy | Expensive & limited to bright end|| Expert Annotation | Expensive & like to avoid true negatives|| Cross-match to reliable catalogs | Unknown error rate || Crowdsourcing (e.g., Zooniverse) | Non-experts | Best methods are expensive (fiscally or time), and likely biased.Before we get bogged down in algorithms and other details of the machine learning pipelines, I implore you to... **Worry about the data**  II. Learning AlgorithmsSimilar to the previous lecture on unsupervised learning, there are far too many algorithms to provide a comprehensive review of every supervised learning method. We will focus on two common methods and their variants today.    A) $k$-nearest neighbors $k$-nearest neighbors is relatively simple:    specify a number, k    identify the $k$ closest neighbors    plurality of neighbors determines classification    B) Decision TreeDecision trees are used for classification and regression. In a decision tree the training set is recursively split based on feature tests (e.g., width $>$ 4 goes here, width $<$ 4 goes there), until a stopping criteria is reached. New sources are then propagated through the tree to assign a final classification.  (credit: <NAME>, Friedman 09)   C) Random ForestAggregates results from a collection of multiple decision trees  Use bagging (bootstrap w/ replacement) for each tree  Select only a random subset of features for split at each node  Average of de-correlated trees reduces variance vs. 1 tree  (credit: http://kazoo04.hatenablog.com/entry/2013/12/04/175402)   D) BoostingPrior to the advent of deep neural networks, methods that utilized boosting provided the best classification results across a variety of problems (the [XGBoost](https://xgboost.readthedocs.io/en/stable/) library remains the go-to method for non-deep learning problems to this day). Similar to random forest, boosting methods utilize multiple learners/decision trees. During training, samples that are misclassified are given more weight. It has been found that boosting provides relatively low generalization errors.  III) Building an ML pipeline  (credit: Real World Machine Learning, <NAME> (2016))   Data Preparation$~~$Query, observe, simulate, etc. - collect data to be modeled  $~~$  $~~$  $~~$  $~~$  $~~$  $~~$    Worry About the Data  Data Preparation$~~$Query, observe, simulate, etc. - collect data to be modeled  $~~$Select features to use in the model  $~~$  $~~$  $~~$  $~~$  $~~$    Data Preparation$~~$Query, observe, simulate, etc. - collect data to be modeled  $~~$Select features to use in the model  $~~$Determine “ground truth” or labels for the training set  $~~$  $~~$  $~~$  $~~$    Data Preparation$~~$Query, observe, simulate, etc. - collect data to be modeled  $~~$Select features to use in the model  $~~$Determine “ground truth” or labels for the training set  $~~$Convert categorical features  $~~$  $~~$  $~~$    (credit: Real World Machine Learning, Brink, Richards, Fetherolf (2016))   Data Preparation$~~$Query, observe, simulate, etc. - collect data to be modeled  $~~$Select features to use in the model  $~~$Determine “ground truth” or labels for the training set  $~~$Convert categorical features  $~~$Impute (or throw out?) missing data  $~~$  $~~$  Strategies for missing dataDoes the missing data have meaning?  $~~$**Yes** – replace with numerical value (-999) or new categorical variable  $~~$**No** – *IF* data set is large with few missing values:    $~~~~~~$remove objects with missing data  $~~~~$*ELSE IF* dataset is large and temporal:  $~~~~~~$replace missing values with preceding value or interpolate  $~~~~$*ELSE IF* dataset has simple distribution:  $~~~~~~$replace missing values with mean or median   $~~~~$*ELSE*  $~~~~~~$build separate ML model to impute (predict) missing data    Data Preparation$~~$Query, observe, simulate, etc. - collect data to be modeled  $~~$Select features to use in the model  $~~$Determine “ground truth” or labels for the training set  $~~$Convert categorical features  $~~$Impute (or throw out?) missing data  $~~$Normalize the features  $~~$  Data Preparation$~~$Query, observe, simulate, etc. - collect data to be modeled  $~~$Select features to use in the model  $~~$Determine “ground truth” or labels for the training set  $~~$Convert categorical features  $~~$Impute (or throw out?) missing data  $~~$Normalize the features  $~~$Visualize the data  Worry About the Data  Feature EngineeringAdd new features (if necessary)Remove noisy/uninformative/highly correlated features (if necessary)  Model Selection (credit: Scikit Learn)   Model EvaluationAvoid over-/under- fitting  Model EvaluationFind optimal trade-off between bias and variance (credit: <NAME>)   Model EvaluationUse *cross validation* (credit: Real World Machine Learning, Brink, Richards, Fetherolf (2016))   Model Evaluation**Metrics**True positive (TP) = + classified as +False positive (FP) = - classified as + (type I error)True negative (TN) = - classified as -False negative (FN) = + classified as - (type II error)  Model EvaluationConfusion matrix| | | Pred | Class || --- | ---: | --- | --- || | |  **+**  |  **-**  || **True** | **+** |  TP | FN  ||**Class** | **-** |  FP | TN  |  Model Evaluation (credit: Richards et al. (2012))   Model Evaluation|  |  | || :--- | :---: | ---: || True Positive Rate (TPR) | = | TP/(TP + FN) || False Positive Rate (FPR) | = | FP/(FP + TN) |  Model EvaluationROC curve (credit: Zahiri et al. 2013)   Model Evaluation|  |  | || :--- | :---: | ---: || Precision | = |  TP / (TP + FP)|| Recall | = | TP / (TP + FN) || $F_1$ | = | 2 * (P * R)/(P + R) |  Model Evaluation (credit: Huijsmans & Sebe 05)   Model Optimization (credit: Real World Machine Learning, Brink, Richards, Fetherolf (2016))   Predictions (credit: MIT OCW 15.097)   WORRY ABOUT THE DATA  (credit: Real World Machine Learning, Brink, Richards, Fetherolf (2016))   ConclusionSupervised methods are useful for classification/regression Cross validation helps prevent overfittingProcess for building an ML pipeline is algorithm independentWorry about the data  Appendix<jupyter_code>xgrid = np.linspace(13, 24, 1000)
magdist = lambda x, A, mu, sigma : A/(2*np.pi*sigma**2)**0.5*np.exp(-(x - mu)**2/(2*sigma**2))
fig, ax = plt.subplots()
ax.plot(xgrid, magdist(xgrid, 10, 18, 1), 
        'Crimson', lw=5, label='SDSS\nspectroscopic')
ax.fill(xgrid, magdist(xgrid, 10, 18, 1), 'Crimson', alpha=0.4)
ax.plot(xgrid, magdist(xgrid, 85, 19.5, 1.4), 
        '0.2', lw=2, label='SDSS\nphotometric')
ax.fill(xgrid, magdist(xgrid, 85, 19.5, 1.4), 
        '0.2', alpha=0.2)

ax.set_xlabel('r (mag)', fontsize=14)
ax.set_ylabel('N $\longrightarrow$', fontsize=14)
ax.set_yticklabels([])
ax.set_ylim(0,25)
ax.set_xlim(13.5,23.9)
ax.legend()
fig.tight_layout()
fig.savefig('./images/training_set.png')
X, y = make_blobs(n_samples=167, n_features=2, centers=2,
                  cluster_std=0.85, center_box=(- 3.0, 3.0), 
                  shuffle=True, random_state=1951)

from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X, y)
# mesh for predictions
h = 0.025
x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
Z = knn.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
cmap_light = ListedColormap(["#fc8d62", "#8da0cb"])
cmap_bold = ListedColormap(["#d95f02", "#7570b3"])

fig, (ax, ax2) = plt.subplots(1,2, figsize=(9,5)) 

for a in (ax, ax2):
    a.contourf(xx, yy, Z, cmap=cmap_light)
    a.scatter(
        x=X[:, 0],
        y=X[:, 1],
        c=y,
        cmap=cmap_bold,
        alpha=1.0,
        edgecolor="0.3",
    )
    a.set_xlabel('X1', fontsize=14)
    a.set_ylabel('X2', fontsize=14)
ax2.set_xlim(-2.5, 2.8)
ax2.set_ylim(-0.5,1.8)
fig.suptitle('k = 3 predictions', fontsize=16)
fig.tight_layout()
fig.savefig('./images/knn3_example.png')
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X, y)

# mesh for predictions
h = 0.025
x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
Z = knn.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
cmap_light = ListedColormap(["#fc8d62", "#8da0cb"])
cmap_bold = ListedColormap(["#d95f02", "#7570b3"])

fig, (ax, ax2) = plt.subplots(1,2, figsize=(9,5)) 

for a in (ax, ax2):
    a.contourf(xx, yy, Z, cmap=cmap_light)
    a.scatter(
        x=X[:, 0],
        y=X[:, 1],
        c=y,
        cmap=cmap_bold,
        alpha=1.0,
        edgecolor="0.3",
    )
    a.set_xlabel('X1', fontsize=14)
    a.set_ylabel('X2', fontsize=14)
ax2.set_xlim(-2.5, 2.8)
ax2.set_ylim(-0.5,1.8)
fig.suptitle('k = 5 predictions', fontsize=16)
fig.tight_layout()
fig.savefig('./images/knn5_example.png')
knn = KNeighborsClassifier(n_neighbors=15)
knn.fit(X, y)

# mesh for predictions
h = 0.025
x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
Z = knn.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
cmap_light = ListedColormap(["#fc8d62", "#8da0cb"])
cmap_bold = ListedColormap(["#d95f02", "#7570b3"])

fig, (ax, ax2) = plt.subplots(1,2, figsize=(9,5)) 

for a in (ax, ax2):
    a.contourf(xx, yy, Z, cmap=cmap_light)
    a.scatter(
        x=X[:, 0],
        y=X[:, 1],
        c=y,
        cmap=cmap_bold,
        alpha=1.0,
        edgecolor="0.3",
    )
    a.set_xlabel('X1', fontsize=14)
    a.set_ylabel('X2', fontsize=14)
ax2.set_xlim(-2.5, 2.8)
ax2.set_ylim(-0.5,1.8)
fig.suptitle('k = 15 predictions', fontsize=16)
fig.tight_layout()
fig.savefig('./images/knn15_example.png')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1490345390263802, 'fraction_numerical': 0.031184842715982233, 'mean_word_length': 4.088560885608856, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 11, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '71092', 'license': 'MIT', 'n_tokens_mistral': 3958, 'n_tokens_neox': 3661, 'n_words': 1226, 'path': 'Sessions/Session14/Day1/IntroductionToSupervisedMachineLearning.ipynb', 'repo_name': 'LuciaIllari/LSSTC-DSFP-Sessions', 'size': '11074'}