<jupyter_start><jupyter_text>Notebook and comments, <NAME>’s “Machine Learning: The DL on ML”By <NAME>, October 24, 2018. Last edited February 22, 2019.This replicates, in a jupyter notebook, the code found in Chapter 7 (“Machine Learning: The DL1 on ML”) of Meredith Broussard’s _Artificial Unintelligence: How Computers Misunderstand the World_ (MIT Press, 2018, https://mitpress.mit.edu/books/artificial-unintelligence). I have also added an additional section on “Overfitting” at the end. As Broussard notes, her treatment is a simplified version of the Kaggle Python Tutorial on Machine Learning at Datacamp (https://www.datacamp.com/community/open-courses/kaggle-python-tutorial-on-machine-learning), but I think it is also helpful to have a companion following the book chapter. I also address one subtle but enormously consequential mistake Broussard makes: reporting training accuracy as test accuracy. This is a typo, but it is a significant one. (Note that the datacamp tutorial discusses overfitting on its 6th lesson of the second unit, [https://campus.datacamp.com/courses/kaggle-python-tutorial-on-machine-learning/predicting-with-decision-trees?ex=6](https://campus.datacamp.com/courses/kaggle-python-tutorial-on-machine-learning/predicting-with-decision-trees?ex=6), although this is cursory.) The mistake notwithstanding, I think Broussard’s chapter is the single best overview of machine learning for a non-technical audience in terms of (hopefully) demystifying machine learning, and providing an alternative narrative to those of commonly made public claims. The commentary she provides while going through the example emphasizes - the _tedium_ of doing machine learning, - the surprising simplicity (and even ”stupidity”) of much of it, and - some fundamental limitations in data and modeling (such as through her point about numbering of rafts; this can be phrased as a problem of a latent variable being causal, but so can many cases of features that predict well in data from one case but will fail to generalize). This notebook is to help anybody who might want to move from that treatment to a technical one, or perhaps see the visual artifact of what doing machine learning/data science would look like, in context (on an analyst's computer, rather than filtered through the book’s typography). To actually run this notebook, you will unfortunately need to go through a standard but still laborious process of installing multiple things on your computer. There is one largely unimportant mistake:- Feature importance is NOT the same as statistical significance! They frequently coincide, but it is possible to have an important feature that is not statistically significant, and conversely, it is possible to have a statistically significance variable low in feature importance. Feature importance has to do with the (normalized) size of an effect, whereas statistical significance has to do with the amount of variance of that effect across observations, given the sample size. This difference is further explored in the literature around “post-selection inference.”But there is one major mistake:- The test accuracy is .72, NOT .97 as Broussard claims! > page 114: “We can write the predictions to a CSV file called `my_solution_one.csv`, upload the file to DataCamp, and verify that our predictions were 97 percent accurate. Ta-da! We just did machine learning.”    The training accuracy is 0.9776 (also, this should round up to 0.98, not down to 0.97, but that’s really not important here), but submitting the ‘predicted’ values to Kaggle gives an accuracy of 0.72248. _The fact that test performance is always worse than training performance is critical_, as it illustrates the common problem of _overfitting_: I discuss this more at the bottom, but this is important because overfitting means that machine learning, in the hypothetical settings of data analysis (like in this notebook), can easily seem to perform far better than it actually will when deployed in the world. Additionally, there is one typographic error: all of the python code has commas inside quotes, perhaps the result of a careless “find+replace all” of  with . Be aware of this if you copy code from the book, but it’s fixed in the copy below. In the book chapter, there are some redundant commands (in the text, each instance has a different explanation); I kept these for faithfulness to the chapter. These commands follow `train.describe()` (inputs `[5]` to `[10]`). > 1 “DL” here stands for “down-low” and is queer black slang. Here it means something like “secrets”, or “summary”.<jupyter_code>import pandas as pd
import numpy as np
from sklearn import tree, preprocessing
train_url = "http://s3.amazonaws.com/assets.datacamp.com/course/Kaggle/train.csv"
train = pd.read_csv(train_url)
test_url = "http://s3.amazonaws.com/assets.datacamp.com/course/Kaggle/test.csv"
test = pd.read_csv(test_url)
print(train.head())
print(test.head())
train.describe()
train["Pclass"].value_counts()
train["Survived"].value_counts() # Note: redundant with two lines down
print(train["Survived"].value_counts(normalize = True)) # Note: redundant with two lines down
# Passengers that survived vs passengers that passed away
print(train["Survived"].value_counts())
# As proportions
print(train["Survived"].value_counts(normalize = True))
# Males that survived vs males that passed away
print(train["Survived"][train["Sex"] == 'male'].value_counts())
# Females that survived vs females that passed away
print(train["Survived"][train["Sex"] == 'female'].value_counts())
# Normalized male survival
print(train["Survived"][train["Sex"] == 'male'].value_counts(normalize=True))
# Normalized female survival
print(train["Survived"][train["Sex"] == 'female'].value_counts(normalize=True))
train["Age"] = train["Age"].fillna(train["Age"].median())
# Print the train data to see the available features
print(train)
# Create the target and features numpy arrays: target, features_one
target = train["Survived"].values
# Preprocess
encoded_sex = preprocessing.LabelEncoder()
# Convert into numbers
train.Sex = encoded_sex.fit_transform(train.Sex)
features_one = train[["Pclass", "Sex", "Age", "Fare"]].values
# Fit the first decision tree: my_tree_one
my_tree_one = tree.DecisionTreeClassifier()
my_tree_one = my_tree_one.fit(features_one, target)
# Look at the importance and score of the included features
print(my_tree_one.feature_importances_)
print(my_tree_one.score(features_one, target))
# Fill any missing fare values with the median fare
test["Fare"] = test["Fare"].fillna(test["Fare"].median())
# Fill any missing age values with the median age
test["Age"] = test["Age"].fillna(test["Age"].median())
# Preprocess
test_encoded_sex = preprocessing.LabelEncoder()
test.Sex = test_encoded_sex.fit_transform(test.Sex)
# Extract important features from the test set: Pclass, Sex, Age, and Fare
test_features = test[["Pclass", "Sex", "Age", "Fare"]].values
print('These are the features:\n')
print(test_features)
# Make a prediction using the test set and print
my_prediction = my_tree_one.predict(test_features)
print('This is the prediction:\n')
print(my_prediction)
# Create a data frame with two columns: PassengerId & Survived 
# Survived contains the model’s prediction
PassengerId =np.array(test["PassengerId"]).astype(int)
my_solution = pd.DataFrame(my_prediction, PassengerId, columns = ["Survived"])
print('This is the solution in toto:\n')
print(my_solution)
# Check that the data frame has 418 entries
print('This is the solution shape:\n')
print(my_solution.shape)
# Write the solution to a CSV file with the name my_solution.csv
my_solution.to_csv("my_solution_one.csv", index_label = ["PassengerId"])<jupyter_output><empty_output><jupyter_text>Now, you can submit this csv at https://www.kaggle.com/c/titanic/leaderboard. This requires registering with Kaggle. If you do submit, you should get an accuracy of 0.72248, like I did:  ![Kaggle.png](https://github.com/momin-malik/guides/raw/master/Kaggle.png)     ***  OverfittingHere I go further into the critical point that the training accuracy is different from the test accuracy, and specifically, the test accuracy is almost always going to be lower than the training accuracy (if that ever seems to not be the case, it’s more likely something is wrong with the code or data). What is overfitting? If we imagine that there is some underlying ‘signal’ in our data, but also ‘noise’, then overfitting is when our models pick up on the noise rather than the signal (yes, machine learning and statistics involve an implicit metaphysical commitment—with consequences for how we interact with the world—although that is a much larger discussion). The noise, of course, never stays constant, and so when we see new data, with different noise, our previous model will fail. In the metaphorical language of machine learning, the model ‘memorizes’ the data and then can’t generalize. The technique of cross-validation attempts to address this by splitting the data into two parts, ‘training’ and ‘testing’ (the ratio of splitting the data into training and test may be 1:1, or maybe 4:1, or 9:1). The idea is that the signal will be the same across the split, but not the noise; by fitting our model on the training set and  then comparing it against the test set, we can get a more honest assessment of whether or not we have captured some signal, or if we are only picking up noise. Note that even test accuracy is not reliable as an estimate of how well a model will do when deployed: there are ways that the performance on truly new data might again be worse than even the test performance (if there was sampling bias towards observations with a strong signal; if there are temporal, spatial, network, or other dependencies in the data that the data splitting does not or cannot control for; and if we set aside data to serve only as a test set but try so many models that we end up overfitting via a distribution over models). Overfitting has much to do with how complicated a model is; complicated models are far more likely to be picking up noise than reflecting complexity in the world. For this reason, simple models often work better than complicated ones. *** We can see how a complicated model can lead to failure by visualizing the decision tree that gave the .98 accuracy. In order to plot the tree, you will need to install graphviz, e.g., `conda install python-graphviz` in the command line. Warning: this can be finicky. I exported the tree as a file in order to convert to an image with `tree.export_graphviz(my_tree_one, out_file='tree.dot', feature_names=["Pclass", "Sex", "Age", "Fare"])`, and that image is what is shown here.<jupyter_code>from graphviz import Source
Source(tree.export_graphviz(my_tree_one, out_file='None', feature_names=["Pclass", "Sex", "Age", "Fare"]))<jupyter_output><empty_output><jupyter_text>![tree.png](https://github.com/momin-malik/guides/raw/master/tree.png) The tree has about 400 splits for only four variables! To zoom in, you can right click and open the above image in a new tab, or look at the pdf at [https://github.com/momin-malik/guides/blob/master/tree.pdf](https://github.com/momin-malik/guides/blob/master/tree.pdf).We can also see that the terminal nodes (also called ‘leaf nodes’) at the end of branching often contain only 1 or 2 samples: this is what is called “memorizing” the data, where there can be a separate decision for every data point.A better approach is to use some limits on how complex the tree can be. Specifically, I use the [default parameters](https://www.rdocumentation.org/packages/tree/versions/1.0-39/topics/tree.control) of the `R` implementation of decision trees in the `tree` package. These parameters are:- `mincut = 5`- `minsize = 10`- `mindev = 0.01`In the `scikitlearn` [implementation of decision trees](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html), the corresponding parameters are- `min_samples_split` (default = 2)- `min_samples_leaf` (default = 1)- `min_impurity_decrease` (default = 0.)Note that the datacamp tutorial at [https://campus.datacamp.com/courses/kaggle-python-tutorial-on-machine-learning/predicting-with-decision-trees?ex=6](https://campus.datacamp.com/courses/kaggle-python-tutorial-on-machine-learning/predicting-with-decision-trees?ex=6) addresses overfitting using the parameters `max_depth = 10, min_samples_split = 5, random_state = 1`, although these are just presented to the tutorial-taker with no explanation of where they come from. Granted, there is no explanation in the `R` documentation either of why the default parameters are what they are, although the fact that they are the defaults here reflects a philosophy that the defaults should encode what works, rather than the base form of the model. As for where these come from—these settings are known as “tuning parameters”, also known as “hyperparameters” (and, confusingly, we sometimes talk of “tuning the hyperparameters”) in that they ‘tune’ how the model is fit to data but (unlike _model_ parameters, like the feature importance from decision trees) are not something we interpret to try and understand something about the data. Default values of tuning parameters sometimes have theoretical reasons (like histogram bin width), but in this case they probably are the result of the trial and error of dozens of the heavy users who wrote the `R` code in question, a sort of folk wisdom (which may be reliable, just without theoretical justification). In practice, we could optimize these parameters as well, by splitting the data into not only training  and test, but training, validation, and test; we would fit multiple decision trees, with different combinations of tuning parameters, on the training set, see which leads to the highest accuracy on the validation set, choose that one as our model, and then see how well it works by testing on the test set. The additional split into a validationset is to prevent overfitting on the tuning parameters, which is also possible and easy to fall into. But often, the choice of tuning parameters is very much of a subjective process, or even a “dark art”, something without justification but that can determine whether or not a model is useless or miraculous (although if tuning parameters are not chosen systematically, it can lead to overfitting).<jupyter_code># Fit the second decision tree: my_tree_two
my_tree_two = tree.DecisionTreeClassifier(min_samples_split = 5, 
                                          min_samples_leaf = 10, 
                                          min_impurity_decrease = 0.01)
my_tree_two = my_tree_two.fit(features_one, target)
print(my_tree_two.score(features_one, target))<jupyter_output>0.819304152637<jupyter_text>The training accuracy is substantially lower! It seems like we did worse, right? Well, using the same `test_features`, we can make another set of values to submit to Kaggle.<jupyter_code>my_prediction_two = my_tree_two.predict(test_features)
PassengerId = np.array(test["PassengerId"]).astype(int)
my_solution_two = pd.DataFrame(my_prediction_two, PassengerId, columns = ["Survived"])
my_solution_two.to_csv("my_solution_two.csv", index_label = ["PassengerId"])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07630337961750065, 'fraction_numerical': 0.007401100340581609, 'mean_word_length': 5.262920426579163, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 20, 'https://': 13, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '598343', 'license': 'MIT', 'n_tokens_mistral': 4171, 'n_tokens_neox': 3935, 'n_words': 1942, 'path': 'Broussard_DL_on_ML.ipynb', 'repo_name': 'momin-malik/guides', 'size': '15288'}