<jupyter_start><jupyter_text>![DSL_logo](https://github.com/BrockDSL/Machine_Learning_with_Python/blob/master/dsl_logo.png?raw=1) Introduction to Machine Learning with PythonIn our [Data Science](https://brockdsl.github.io/Python_2.0_Workshop/) workshop we introduced some concepts by looking at some fictional data about people that got sick with a mysterious illness. In this session we are going to see if we can build a machine learning model to see if we can predict who has the illness based on the answers to some questions. As a further exercise we'll setup two examples that try to guess the quality of wine. I encourage you to try out these examples after class is done.  First, a brief recap on Python codeThe following code should look familiar to you<jupyter_code>import pandas as pd

#Load the file into a dataframe using the pandas read_csv function
data = pd.read_csv("https://brockdsl.github.io/Python_2.0_Workshop/canadian_toy_dataset.csv")

#Tell it what our columns are by passing along a list of that information
data.columns = ["city","gender","age","income","ill"]


data.head()<jupyter_output><empty_output><jupyter_text>Machine Learning BasicsDon't let the impressive name fool you. Machine learning is more or less the following steps1. Getting your data and cleaning it up1. Identify what parts of your data are **features**1. Identify what is your **target variable** that you'll guess based on your features1. Split your data in **training and testing sets**1. **Train** your model against the training set1. **Validate** your model against the testing set1. ????1. ProfitWe are going to use the Python library [scikit-learn](https://scikit-learn.org/stable/) and we are going to be doing a [classification](https://en.wikipedia.org/wiki/Statistical_classification) problem.![classification](https://raw.githubusercontent.com/BrockDSL/Machine_Learning_with_Python/master/classification.png)  Decision TreeThis is one of the most basic machine learning model you can use. It is considered a [supervised learning](https://en.wikipedia.org/wiki/Supervised_learning) method. You create the best [decision tree](https://en.wikipedia.org/wiki/Decision_tree_learning) that you can based on your training data. Here's an example tree that shows your chance of surviving the Titanic disaster. What we are creating is series of question that when answered will put observations into a _bucket_ or in other terms one of the classification options. We also devise a probability associated with an observation falling into that _bucket_.The features are described by the labels, however ``sibsp`` - is the number of spouses or siblings on board.![dtree](https://upload.wikimedia.org/wikipedia/commons/e/eb/Decision_Tree.jpg)So in this tree the most important question to ask first is what is the gender of the person you are considering, then next most important question is age above 9 and a half, followed lastly by, does this person have less than three spouses or siblings on board. Let's start by loading the Libraries we need<jupyter_code>#This should look familar
import pandas as pd
#import numpy as np


#We'll draw a graph later on
import matplotlib.pyplot as plt

#Our 'Machine Learning pieces'
from sklearn.tree import DecisionTreeClassifier 
from sklearn.model_selection import train_test_split
from sklearn.tree import export_text
from sklearn import metrics 
from sklearn import tree

print("Ready to proceed!")<jupyter_output><empty_output><jupyter_text>Getting the data readyNow, let's load our data. Our decision tree can only work with numerical values, so we'll have to modify the columns of data that are text based. As stated preparing the data is usually the most difficult part of the process.<jupyter_code>data = pd.read_csv("https://brockdsl.github.io/Python_2.0_Workshop/canadian_toy_dataset.csv")
data.columns = ["city","gender","age","income","ill"]
data.head()<jupyter_output><empty_output><jupyter_text>This dataset is fairly clean, we just need to represent it all as numbers instead of text labels. So that means we need to change the columns:- `ill` - instead of a No / Yes label we'll use 0 and 2 instead- `city` - this will break out the column into 8 different columns- `gender` - this will break out the column into 2 different columns<jupyter_code>#Instead of yes/no we'll use a 0 or 2
#We use the value '2' to make our analysis later on less ambiguous
data["ill"].replace({"No":0, "Yes":2},inplace=True)

#We change categorical values into numeric ones using `dummies`
data = pd.get_dummies(data, columns=['city','gender'])
data.head(5)<jupyter_output><empty_output><jupyter_text>The example above shows 5 entries that that come from Montreal.<jupyter_code>data.tail(5)<jupyter_output><empty_output><jupyter_text>This example shows the last 5 entries in the dataframe that come from Edmonton.  Now we are done the most difficult part of the process, understanding the data and getting it ready.----  Building and Running the Model We now have our data cleaned up, and represented in a way that Scikit will be able to analyze. To be honest the most difficult part of the process is done.We now need to split our columns in two types:- **features** represent the data we use to build our guess- **target variable** the thing our model hopes to guess<jupyter_code>#all of the following columns are features, we'll make a list of their names
features = ["age",\
            "income",\
            "city_Edmonton",\
            "city_Halifax",\
            "city_Montreal",
            "city_Ottawa",\
            "city_Regina",\
            "city_Toronto",
            "city_Vancouver",\
            "city_Waterloo",\
            "gender_Female",\
            "gender_Male"]

X = data[features]

#We want to target the ill column
y = data.ill<jupyter_output><empty_output><jupyter_text>Training and testingNow that we have built our model we need to get the data ready for it. We do this by breaking it into two different pieces. The diagram shows a conceptualization of how this is proportioned.![Train Test Split](https://raw.githubusercontent.com/BrockDSL/Machine_Learning_with_Python/master/train_test.png)- **Training set** - This is what is used to build the model. If we set this value too large the ML Model just _memorizes_ the data so we need to be careful when setting this value. This is called _overfitting_ the data.- **Testing set** - This is used to see if our guesses are correctBefore we were looking at the **columns** of the data, this investigation of training/testing looks at the **rows** of data.<jupyter_code>#Training and test together make up 100% of the data!
#We start with a baseline of 30% of our data as testing

test_percent = 30
train_percent = 100 - test_percent

X_train, X_test, y_train, y_test = train_test_split(X, \
                                                    y, \
                                                    test_size=test_percent/100.0,
                                                   random_state=10)<jupyter_output><empty_output><jupyter_text>Now the interesting part, we build our model, **train** it against the **training set** and see how it **predicts** against the **testing set**<jupyter_code># Create Decision Tree classifer object
treeClass = DecisionTreeClassifier()

# Train
treeClass = treeClass.fit(X_train,y_train)

#Predict
y_pred = treeClass.predict(X_test)<jupyter_output><empty_output><jupyter_text>Accuracy of the ModelTo see how good our machine learning model is we need to see how accurate our predictions are. `Scikit` has built in functions and [metrics](https://scikit-learn.org/stable/modules/model_evaluation.html) to do this for us.<jupyter_code>print("Accuracy: ")
print(metrics.accuracy_score(y_test,y_pred))<jupyter_output><empty_output><jupyter_text>Making PredictionsNot bad. We can use our model to predict a guess for **ill** if we pass along all of the other parameters. Our model only tells us if someone is ill or not. This is directly asking our classification model to give us a prediction based on a pretend record.Since this classifier tells us if someone is ill or someone is not ill, it has two outputs.<jupyter_code>data.ill.unique()
# I randomly picked a record in the dataset to test if the prediction is correct. 
# This is from line: 149120 of the datafile
person_x_yes = [
        32, #age
        82311, #income
        1, #city_Edmonton
        0, #city_Halifax
        0, #city_Montreal
        0, #city_Ottawa
        0, #city_Regina
        0, #city_Toronto
        0, #city_Vancouver
        0, #city_Waterloo
        0, #gender_Female
        1, #gender_Male
]

person_x_yes = pd.DataFrame([person_x_yes],columns=X_test.columns)

print("Someone who is ill")
print("Class predicted by model: ",treeClass.predict(person_x_yes))
print("Probablity associated with the guess: ",treeClass.predict_proba(person_x_yes))



# I randomly picked a record in the dataset to test if the prediction is correct. 
# This is from line: 149121 of the datafile
person_x_no = [
        40, #age
        89780, #income
        1, #city_Edmonton
        0, #city_Halifax
        0, #city_Montreal
        0, #city_Ottawa
        0, #city_Regina
        0, #city_Toronto
        0, #city_Vancouver
        0, #city_Waterloo
        1, #gender_Female
        0, #gender_Male
]

#Use the dataframe of our fictional person in our model and get our prediction
person_x_no = pd.DataFrame([person_x_no],columns=X_test.columns)

print("\nSomeone who was not ill")
print("Class predicted by model: ", treeClass.predict(person_x_no))
print("Probablity associated with the guess: ", treeClass.predict_proba(person_x_no))<jupyter_output><empty_output><jupyter_text>Our model is very confident in it's ability to made predictions!With this model constucted we can make ask it question so to speak. We can provide it with details about a pretend person and see what classification the model will place this person. Making a prediction with our modelTry to set some parameters in the `pretend_person` variable below to make the prediction determine that the person is **ill**. If you can find one please copy and paste it into the chat box for others to try. You can do this by:- changing the values **line 2 & line 3** for age and income- pick one line from **line 4 to line 11** and change a single row to a value of 1- pick one line from **line 12 to line 13** and change a single row to a value of 1 Question 1- Try to come up with some set of values that creates an `ill` person. Share your choices in the chat box- Try to come up with some set of values that doesn't create an `ill` person. Share you choices in the chat box<jupyter_code>pretend_person = pd.DataFrame([
        30, #age - FILL IN
        5000, #income - FILL IN
        1, #city_Edmonton - ONLY 1 city at a time
        0, #city_Halifax
        0, #city_Montreal
        0, #city_Ottawa
        0, #city_Regina
        0, #city_Toronto
        0, #city_Vancouver
        0, #city_Waterloo
        1, #gender_Female - ONLY 1 gender at a time
        0, #gender_Male
])


#turn our pretend person into a dataframe that is the correct dimensions
pretend_person = pretend_person.T 
pretend_person.columns = X_test.columns

print("\Pretend person details")
print(pretend_person.head())

print("Pretend person Class predicted")
print(treeClass.predict(pretend_person))

print("Pretend person probablity of guess")
print(treeClass.predict_proba(pretend_person))<jupyter_output><empty_output><jupyter_text>Visualizing our Decision TreeWe can 'visualize' the decision tree to trace through the decisions it makes. In this case we can tell that **income level** is the most important factor that we consider since we ask so many questions about that before looking at any of the other features. Question 2What is the first question the tree is asking you to make? Share your thoughts in the chat box.<jupyter_code>printed_tree = export_text(treeClass,feature_names=features)
print(printed_tree)<jupyter_output><empty_output><jupyter_text>Tuning parameters - Testing Set SizesTo make our models run better we can tweak _many, many, many_ different parameters. For example, we can vary the testing data size percentage. We'll try some different values and plot our our accuracy of our predictions.<jupyter_code>testing_percents = [1,5,10,20,30,99]
accuracy = []
training_percents = []

for test_ratio in sorted(testing_percents):
    X_train, X_test, y_train, y_test = train_test_split(X, \
                                                        y, \
                                                        test_size=test_percent/100.0,
                                                        random_state=10)
    treeClassTest = DecisionTreeClassifier()
    treeClassTest = treeClassTest.fit(X_train,y_train)
    y_pred = treeClassTest.predict(X_test)
    score = metrics.accuracy_score(y_test,y_pred)
    accuracy.append(score)
    training_percents.append(100 - test_ratio)

    
plt.plot(training_percents,accuracy)
plt.ylabel("Accuracy in %")
plt.xlabel("Training Size %")
plt.show()<jupyter_output><empty_output><jupyter_text>(Your graph might look different, this is a statistical operation and will probably vary across different machines)  Tuning Parameters - Maximum depth of the treeWe can restrict how deep our tree will be by setting `max_depth` in our `DecisionTreeClassifier` variable. Below is another example of trying different values in our ML model for this parameter and plotting out the accuracy of our model.<jupyter_code>test_percent = 70
max_options = [5,10,15,20,25,30]

accuracy = []
tree_max = []

for max_d in sorted(max_options):
    X_train, X_test, y_train, y_test = train_test_split(X, \
                                                        y, \
                                                        test_size=test_percent/100.0,
                                                        random_state=10,
                                                       )
    
    #We set maximum depth in the DecisionTreeClassifer when we first create the variable
    treeClassTest = DecisionTreeClassifier(max_depth=max_d)
    treeClassTest = treeClassTest.fit(X_train,y_train)
    y_pred = treeClassTest.predict(X_test)
    score = metrics.accuracy_score(y_test,y_pred)
    accuracy.append(score)
    tree_max.append(max_d)

    
plt.plot(max_options,accuracy)
plt.ylabel("Accuracy")
plt.xlabel("Maximum Depth of Tree")
plt.show()<jupyter_output><empty_output><jupyter_text>Example 2We are going to look at a [cancer survivor data set](http://archive.ics.uci.edu/ml/datasets/Haberman%27s+Survival) from the UCI machine learning archive.![cancer_description](https://raw.githubusercontent.com/BrockDSL/Machine_Learning_with_Python/master/cancer_characteristics.png)<jupyter_code>cancer_data = pd.read_csv("http://archive.ics.uci.edu/ml/machine-learning-databases/haberman/haberman.data",header=None)
cancer_data.columns = ["age","operation_year","positive_nodes","survival_status"]
cancer_data.describe()<jupyter_output><empty_output><jupyter_text>Setting up our model Question 2- What should our features list look like?- What should or target column beYou need to do two things:1. uncomment one line between **line 3 & line 7**2. uncomment one line between **line 13 & line 16**<jupyter_code>#What features list should we use, uncomment the correct answer

#cancer_features = ["age","operation_year","positive_nodes","survival_status"]
#cancer_features = ["age","operation_year","positive_nodes"]
#cancer_features = ["age","operation_year"]
#cancer_features = ["age"]
#cancer_features = ["operation_year","positive_nodes","survival_status"]


#What target should we use, uncomment the correct answer

#cancer_target = cancer_data.age
#cancer_target = cancer_data.operation_year
#cancer_target = cancer_data.positive_nodes
#cancer_target = cancer_data.survival_status

X = cancer_data[cancer_features]
y = cancer_target<jupyter_output><empty_output><jupyter_text>Question 2  ContinuedNow that you have your features and target set, run the next cell to build and test your model.Once you are done type "finished" in the chatbox.<jupyter_code>#We'll start with 40 just for fun
test_percent = 40

X_train, X_test, y_train, y_test = train_test_split(X, \
                                                    y, \
                                                    test_size=test_percent/100.0,
                                                   random_state=10)
# Create Decision Tree classifer object
treeClass = DecisionTreeClassifier()

# Train
treeClass = treeClass.fit(X_train,y_train)

#Predict
y_pred = treeClass.predict(X_test)

#Accuracy
print("Accuracy:")
print(metrics.accuracy_score(y_test,y_pred))<jupyter_output><empty_output><jupyter_text>Visualizing the treeNow that we have a tree built for this scenario let's display it to the screen.<jupyter_code>printed_tree = export_text(treeClass,feature_names=cancer_features)
print(printed_tree)<jupyter_output><empty_output><jupyter_text>Tuning the testing set size Question 3Experiment with adding some values in the `testing_percents_cancer` list on **line 3**. In the chat box put in the testing set size that produced the best results.When you are done experiementing type "Done!" in the chat box.<jupyter_code># put in some values between 0 - 99
# add at least 4 values in between the commas
testing_percents_cancer = [,,,]

#Let's make sure the numbers are all increasing so our graph looks good
testing_percents_cancer = sorted(testing_percents_cancer)

accuracy = []
training_percents = []

for test_ratio in testing_percents_cancer:
    X_train, X_test, y_train, y_test = train_test_split(X, \
                                                        y, \
                                                        test_size=test_percent/100.0,
                                                        random_state=10)
    treeClassTest = DecisionTreeClassifier()
    treeClassTest = treeClassTest.fit(X_train,y_train)
    y_pred = treeClassTest.predict(X_test)
    score = metrics.accuracy_score(y_test,y_pred)
    accuracy.append(score)
    training_percents.append(100 - test_ratio)

    
plt.plot(training_percents,accuracy)
plt.ylabel("Accuracy in %")
plt.xlabel("Training Size %")
plt.show()<jupyter_output><empty_output><jupyter_text>Turning Parameters, Maximum Tree Depth Question 4Experiment with adding some values in the `max_options_cancer` list on **line 5**. In the chat box put in the testing set size that produced the best results.When you are done experiementing type "Done!" in the chat box.<jupyter_code>test_percent = 30

#Put in some options between 1 and 40
#add at least 4 values between the commas
max_options_cancer = [,,,]

#Let's make sure the numbers are all increasing so our graph looks good
max_options_cancer = sorted(max_options_cancer)

accuracy = []
tree_max = []

for max_d in max_options_cancer:
    X_train, X_test, y_train, y_test = train_test_split(X, \
                                                        y, \
                                                        test_size=test_percent/100.0,
                                                        random_state=10,
                                                       )
    
    #We set maximum depth in the DecisionTreeClassifer when we first create the variable
    treeClassTest = DecisionTreeClassifier(max_depth=max_d)
    treeClassTest = treeClassTest.fit(X_train,y_train)
    y_pred = treeClassTest.predict(X_test)
    score = metrics.accuracy_score(y_test,y_pred)
    accuracy.append(score)
    tree_max.append(max_d)

    
plt.plot(max_options_cancer,accuracy)
plt.ylabel("Accuracy")
plt.xlabel("Maximum Depth of Tree")
plt.show()<jupyter_output><empty_output><jupyter_text>Maximizing Accuracy Question 5- What is the best combination of `max_depth` and `testing size` that produced the *highest* accuracy?- What is the worst combination of `max_depth` and `testing size` that produced the *lowest* acccuracy? You just need to add some values to **line 3** & **line 4**.Share your answers in the chat.You can experiment using the following cell:<jupyter_code>#Change the following values
test_percent = 
max_tree_depth = 

#HINT: You can use the previous graphs to help you pick your values

X_train, X_test, y_train, y_test = train_test_split(X, \
                                                    y, \
                                                    test_size=test_percent/100.0,
                                                   random_state=10)
# Create Decision Tree classifer object
treeClass = DecisionTreeClassifier(max_depth=max_tree_depth)

# Train
treeClass = treeClass.fit(X_train,y_train)

#Predict
y_pred = treeClass.predict(X_test)

#Accuracy?
print("\nAccuracy of our tree: ")
print(metrics.accuracy_score(y_test,y_pred))


#Display our final tree
print("\nBest tree found:\n")
printed_tree = export_text(treeClass,feature_names=cancer_features)
print(printed_tree)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0775086834467336, 'fraction_numerical': 0.010515297140410143, 'mean_word_length': 3.6021458287716226, 'pattern_counts': {'":': 2, '<': 85, '<?xml version=': 0, '>': 85, 'https://': 13, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 46, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '196493', 'license': 'MIT', 'n_tokens_mistral': 6168, 'n_tokens_neox': 5798, 'n_words': 2164, 'path': 'Machine_Learning_Basics.ipynb', 'repo_name': 'abakusinfopl/testy-i-variables', 'size': '21017'}