<jupyter_start><jupyter_text>OverviewIn this notebook, we'll explore a typical workflow for applying data science to a problem and creating a predictive model.Though the process of applying data science to problems can vary somewhat from one problem to another, the journey to AI follows the following steps - which we call the AI ladder - at its core.1. Collect2. Organize3. Analyze4. InfuseThis notebook has selected some of the most common activities from the AI ladder and takes you through them. Specifically, this is the workflow that we will be covering. Notebook's workflow- Preparation    1. Problem Definition    2. Goal Setting    3. Data Collection- Data preparation    4. Data Exploration    5. Data Cleaning    6. Data Wrangling- Data Analysis    7. Data Visualization and insight    8. Model Training        Tools we will useThroughout this notebook, you will learn to use a few different Python libraries. We introduce those tools here and import them to the project. To learn more about each one, you can click on their names:- [`matplotlib`](https://matplotlib.org/): A library for creating static, animated, and interactive visualizations.- [`seaborn`](https://seaborn.pydata.org/): Another visualization library based on matplotlib that provides simpler higher-level plotting tools.- [`pandas`](https://pandas.pydata.org/docs/user_guide/10min.html): A library for data manipulation and analysis.- [`numpy`](https://numpy.org/): A library for working with arrays, matrices, and linear algebra.- [`sklearn`](https://scikit-learn.org/stable/getting_started.html): Scikit-learn is a library that provides machine learning algorithms and other relevant tools.<jupyter_code># Let's import these libraries
# Note that we give some of them short names for ease of use

# Visualization
import matplotlib.pyplot as plt
import seaborn as sns

# Data Analysis
import pandas as pd
import numpy as np
import sklearn<jupyter_output><empty_output><jupyter_text>PreprationIn this section, we take the steps necessary to set up our problem to stay focused and achieve the results that we need.  1. Problem DefinitionThe first step to successfully applying machine learning to an AI problem is defining the problem and setting goals. Let's solidify the problem statement and gather the necessary domain information.**The problem**In this case, the problem that we are trying to solve is:> Given a subset of Titanic passengers' survival status, train a predictive model that predicts which of the remaining passengers also survived. The model should be optimized for accuracy.**Background Information**It is also helpful to know the event that happened, that is, gather the domain knowledge. This would help us put together specific hypotheses when exploring the data and creating new features to help our predictive models.If you are unfamiliar with the event, look at the [Wikipedia](https://en.wikipedia.org/wiki/Titanic) article on the event paying particular attention to the "[Survivors and victims](https://en.wikipedia.org/wiki/TitanicSurvivors_and_victims)" section. That should provide some insight into the kinds of patterns you might want to explore in the data.  2. Goal SettingThe next step is to define the goal we want to achieve. In this case, we will set the following goals to address our problem definition1. Exploring the Data2. Cleaning the Data3. Wrangling the Data4. Training a classifier    The next steps are all focused on the following goals  3. Data CollectionThe last step before we start working with the data is collecting the data. In this case, we can load the data in from a file that we have prepared; however, in real-world use-cases, one might need to use data mining processes and get the appropriate approvals before using the data.  For our purpose, you can load the data using the [`read_csv`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) method of the [`pandas`](https://pandas.pydata.org/docs/user_guide/10min.html) library.<jupyter_code>#Import the data
df = pd.read_csv(r'https://raw.githubusercontent.com/IBM/python-and-analytics/master/data/titanic.full.csv', na_values='?')
# TODO remove
df = pd.read_csv(r'../data/titanic.full.csv',na_values='?')<jupyter_output><empty_output><jupyter_text>Notice the `na_values='?'` within the parenthesis after body. That's an additional argument that we passed into the `read_csv()` function of pandas to specify `?` in our datasets denotes blanks. You might need to adjust this for your datasets. Our data is now loaded into our notebook in a `dataframe`. Think of a `dataframe` as a table in python with rows and columns. A `dataframe` allows you to manipulate the data. We will look at some of these data manipulations below. Finally, we need to understand our data. For that, here is a quick description of what each column entails:| Column    | Description                                                          || :-------- | :------------------------------------------------------------------- || Pclass    | Passenger Class (1 = 1st; 2 = 2nd; 3 = 3rd)                          || survival  | Survival (0 = No; 1 = Yes)                                           || name      | Name                                                                 || sex       | Sex                                                                  || age       | Age                                                                  || sibsp     | Number of Siblings/Spouses Aboard                                    || parch     | Number of Parents/Children Aboard                                    || ticket    | Ticket Number                                                        || fare      | Passenger Fare (British pound)                                       || cabin     | Cabin                                                                || embarked  | Port of Embarkation (C = Cherbourg; Q = Queenstown; S = Southampton) || boat      | Lifeboat                                                             || body      | Body Identification Number                                           || home.dest | Home/Destination                                                     | At this point, we are done with acquiring the data and defining our problem, and we are ready to move on to the Data Preparation step.  Data Prepration  4. Data ExplorationBefore we start cleaning our dataset, we need to understand our data. Let's do that by first visualizing the data and then perform simple statistcs on it.  Using a Pandas DataframeFirst, let's take a look at our `dataframe` to become familiar with operations that we can perform on it and what is inside it. Vieweing your data<jupyter_code># Let's print our DataFrame
df<jupyter_output><empty_output><jupyter_text>This form of printing the `dataframe` takes a lot of space which is unnecessary since we manipulate the table with code.  So from here on, we will use `df.head()` to only show the top of the table instead. You can also use `df.tail()` if you want to see the end of a table.<jupyter_code># Try df.head() and df.tail() in this cell.
# Notice that jupyter will show only one of them at a time. So if you type both, only the last one is shown.<jupyter_output><empty_output><jupyter_text>Next, let's look at the data types in each column (aka. feature). It is always of interest to know what numerical, categorical, and text features.Here are some common data types:- `float`: Numbers with decimals- `int`: Integers. Numbers without any decimals- `object`: Textual values- `category`: Categorical data- `bool`: logical `True` or `False` values- `datetime64`: A format for storing dates<jupyter_code># Exercise
# View the datatypes for each column
df.dtypes<jupyter_output><empty_output><jupyter_text>It looks like some of the features are incorrectly detected as strings (`Objects`). We will take a look at those next and fix them in the data cleaning section. For now, let's continue exploring our `dataframe.` Next, let's see what methods our dataframe supports. To list these, in the cell below, type `df.` and then press the `TAB` button on your keyboard. You should see a list of methods or attributes available.<jupyter_code># Exercise
# type df. and press TAB at the end of the following line and explore the options
# df.<TAB><jupyter_output><empty_output><jupyter_text>Note that you also see the feature (column) names as things that you can call on `df`. That is the easiest way to pull out a column for manipulation, or vieweing. For example:<jupyter_code>df.age<jupyter_output><empty_output><jupyter_text>`describe()` and `.info()` show quick summary of your dataframe. Let's give them a try:<jupyter_code># Exercise
# Try df.describe()

# Exercise
# Try df.info()<jupyter_output><empty_output><jupyter_text>`.T` quickly transposes the table.<jupyter_code># Exercise
# Try df.T<jupyter_output><empty_output><jupyter_text>`.sort_index()` and `.sort_values()` sort your table. if you pass `axis=1` to these functions they will operate on the columns. That is, they will sort the column names.<jupyter_code># Exercise
# Try df.sort_index(axis=1)

# Exercise
# Try df.sort_values(by='age')<jupyter_output><empty_output><jupyter_text>Selecting parts of the dataNext, let's look at how to select parts of our table. We will use data selection extensively in this notebook.<jupyter_code># Exercise:
# Column Selection
# try df.fare or df["fare"]


# Exercise:
# Row Selections by row number. The syntax is df[begining row : final row]
# try df[0:4]

# Exercise:
# Conditional row selection
# try the following. Reads "Select df where survived value is one"
# try df[df["survived"]==1]

# Selecting by row and column
# use the df.loc[rows , [column1, column2, ...] ]
# try the following. Reads "from df select rows 0 to 3 and columns name, sex, and age"
# try df.loc[0:3, ["name", "sex", "age"]]<jupyter_output><empty_output><jupyter_text>[Learn More](https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.htmlselection-by-label) about Data Selection.  setting valuesThere are many ways to set values in your table which you can find in the [documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.htmlsetting)You can, for instance, use their position, value, a condition about their value (eg. > 0), a condition on a different column(feature) (eg. if not survived, set age to 0!), etc... Let's explore some of them together. First, we make a copy of our data, so the original dataset remains intact. To do this, use the `.copy()` method. Note that simply assigning a different name to your  `dataframe` wouldn't be enough (ie. `df2 = df`). This would mean `df` and `df2` point to the same table in the memory! That is if you would modify `df2`, `df` would also reflect those changes. The right way of making a copy is using the `.copy()` function.<jupyter_code># Make a copy
df2 = df.copy()
# Next, let's see what would happen if we dropped every row that has any NaN (blank) values.

df2.dropna(how="any")<jupyter_output><empty_output><jupyter_text>Interesting... Nothing remained!  It looks like every row has some unknowns. We shall do some more specific cleaning in our data cleaning step. For now, since `dropna()` doesn't perform its operation [`inplace`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html) and since we didn't assign the output of `df2.dropna(how="any")` back to `df2`, the actual content of `df2` is unmodified, so we can move on. Let's verify `df2` is untouched.<jupyter_code># Checking df2 is unmodified
df2<jupyter_output><empty_output><jupyter_text>Operating on the data Now it's time for some exploration! You can see many of the possible operations in the [documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/basics.htmlbasics-binop). Here we will take a look at some of them to become familiar with the syntax. `.mean()`, `.median()`, `.max()`, `min()`, `skew()` are some of the basic statistical operations you can use. You can use it on the whole table, inidividual rows, or your selected data<jupyter_code>df.mean()
df["age"].median()
# Remember to select data conditionally you can do df[(condition1) & (condition2) & ...]

female_survivors = df[ (df["sex"]=='female') & (df["survived"] == 1) ]
male_survivors   = df[ (df["sex"]=='male')   & (df["survived"] == 1) ]

f_age_median = female_survivors["age"].median()
m_age_median = male_survivors["age"].median()

print("Age medians f=", f_age_median, "m=", m_age_median)
f_age_skew = female_survivors["age"].skew()
m_age_skew = male_survivors["age"].skew()

print("Age skew f=", f_age_skew, "m=", m_age_skew)<jupyter_output>Age skew f= 0.2953371209570967 m= 0.23856402501973314<jupyter_text>We can see a significant age difference between male and female survivors, and the skew in the data is different. So we will make sure to explore this further when we are visualizing the data. For now, you can explore the data even more with the statistical methods available before moving on. At this point, we are familiar enough with the dataframe that we can move on to the visualization section. Of course, there are many other operations that we haven't covered yet in the interest of time, such as joining tables, grouping items together, pivoting, time series, etc. You learn more about these operations by referring to the  [documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.htmlsetting).  Visual Exploration (matplotlib and seaborn)Now that we know how to use a dataframe and select different parts of it, we can create experimental plots using seaborn and matplotlib. You can find extensive information on how to make plots with these tools online. Here, we will cover some of the most commonly used features.  HistogramsLet's begin by exploring the survival rates based on age on a histogram. We will use the `plt.hist()` function. Remember that `plt` was the short name we assigned `matplotlib.pyplot` when we imported it. `plt.hist()` is used to create histograms. You need to select one column to create a histogram.<jupyter_code>plt.hist(df[df["survived"] == 1]["age"])<jupyter_output><empty_output><jupyter_text>Interesting... It looks like young children had high survival rates. Let's explore this more by increasing the number of bins.<jupyter_code>plt.hist(df[df["survived"] == 1]["age"], bins = 20)<jupyter_output><empty_output><jupyter_text>Conclusion: We can see that many people under the age of 4 and between age 70-80 survived the disaster. This indicates that there is some correlation between age and whether someone survived the disaster. Therefore, we make sure to include this feature in our model training later. Let's look at a more advanced syntax to simplify our visualization. here we will explore the survival by `pclass` by `age`.<jupyter_code># Create a seaborn grid of empty plots with as many rows as unique values in "pclass" and columns as "survived" 
grid = sns.FacetGrid(df, row='survived', col="pclass")

# apply plt.hist on the 'age' column of each selected data in the grid. 
grid.map(plt.hist, 'age', bins = 20)

# So the first line splits the data by 'survived' and 'pclass' columns, and then grid.map applies a plt.hist(..., bins = 20) on the age column.<jupyter_output><empty_output><jupyter_text>Let's do the same for survivors based on age and sex as well.<jupyter_code># Create a seabord grid of empty plots with as many rows as unique "pclass" and columns as "survived" values
grid = sns.FacetGrid(df, row='survived', col='sex')

# apply plt.hist on the 'age' column of each selected data in the grid. 
grid.map(plt.hist, 'age', bins = 20)<jupyter_output><empty_output><jupyter_text>ObservationThese plots provide more intuition on who survived and who didn't. It appears that:- pclass 1 had the least passengers, but most survived- pclass 3 had the most passengers, and most didn't survive- most females survived while most men didn't- most children and all elderly survived- ...Conclusion: From this, we get the intuition that there might be a good correlation between age, sex, pclass, and whether someone survived or not. So these are features that we might consider for our model training. Next, let's learn how to use `plt.pointplot` and `plt.barplot` to explore correlations in categorical and numerical features.First, we will explore if the port of boarding (`embark`) seems to correlate with the survival rate.<jupyter_code>grid = sns.FacetGrid(df, col='embarked')
grid.map(sns.pointplot, 'pclass', 'survived', 'sex')
grid.add_legend()<jupyter_output><empty_output><jupyter_text>Conclusion:  It appears that port of entry has an impact on whether males or females survived, so we should definitely consider adding this to our model. Next let's see if those who paid more, were more likely to survive. For this, we will use a `plt.barplot`<jupyter_code>grid = sns.FacetGrid(df, row="survived", col='embarked')
grid.map(sns.barplot, 'sex', 'fare')
grid.add_legend()<jupyter_output><empty_output><jupyter_text>Conclusion:  It also appears that though not as significant as other features, the paid fare played a role in whether someone survived or not. And this seems to be impacted by the port of entry as well as their gender. So let's consider `fare` for our model training as well. Exercise:You are not well-equipped to explore your assumptions about this data. Let's see if you can find correlations that could indicate your assumptions are correct or not.<jupyter_code># Room for exercise<jupyter_output><empty_output><jupyter_text>This brings us to the end of our data exploration; we are now ready to clean and wrangle our data and then proceed to model training.  5. Data CleaningLet's clean our data, remove what we don't need, and fill in the blanks.  Dropping unimportant features.While we can't know if a feature is useful or not initially, we can get a good feeling about it based on our data exploration. It is always a good practice to start with fewer features and slowly add more since fewer features mean faster model training and iteration time.In this case, based on our observations in the previous section, we are going to keep `pclass`, `age`, `sex`, `embarked`, and `fare`. Additionally, we would see correlations from the following if we explored them, so we will keep those as well: `name`, `sibsp`, `parch`And of course, our target column: "survived."<jupyter_code>df = df.loc[:, ["pclass", "age", "sex", "embarked", "fare", "name", "sibsp", "parch", "survived"]]
df.head()<jupyter_output><empty_output><jupyter_text>Since most Machine Learning models cannot deal with blank data, let's see which columns have empty values and address those. For this, we use the `.isnull()` function that puts a true/false on each cell indicating if it is blank, and then we use the `.sum()` function on the result to sum up how many trues (blanks) are in each column.<jupyter_code>df.isnull().sum()
# Exercise
# You can try df.isnull() by itself to see what the output looks like<jupyter_output><empty_output><jupyter_text>Let's start with `age` as it has the most blanks. We shall replace the blank ones with a guessed value. Based on your knowledge of the domain, you can choose different methods for guessing empty numbers; here, we can fill in the age median and move on. Of course, this is a very simplistic method and introduces unwanted error. A better approach would be to guess the age based on the median of age in each group separated based on `sex`, `pclass`, `embarked`. But that is out of the scope of this introductory notebook.`.fillna()` is used to fill in the NaN values.<jupyter_code># Let's get the Mode of age
age_mode = df['age'].mode()
age_mode
# Since age_mode is a Series, we get the first and only element, so we have a scalar number to use
age_mode = age_mode[0]

# Then we can use the fillna()
df["age"] = df["age"].fillna(age_mode)
# Let's verify we have no more NaN's in the age column
df["age"].isnull().sum()<jupyter_output><empty_output><jupyter_text>Exercise: Do the same for the embarked and fare columns.<jupyter_code># Exercise 
# Fill NaN embarked values with the mode of that column<jupyter_output><empty_output><jupyter_text>Our Data is now relatively clean, so we can move on to the next step.  6. Data WranglingMany Machine Learning algorithms operate on numerical values only, so we need to convert our textual and categorial fields to numbers<jupyter_code># Let's see the types first
df.dtypes<jupyter_output><empty_output><jupyter_text>One way to do this is to replace each value with a number that you want. You would use the following syntax for that.```pythondf['sex'] = df['sex'].map( {'female': 1, 'male': 0} )df['sex'] = df['sex'].astype(int)```However, since we don't care about which number represents which column, there is a simpler way using `.astype('category')` and `cat.codes`<jupyter_code># Convert the column type to categorical
df["sex"] = df["sex"].astype('category')
df.dtypes
# Replace the categorical column with numberical codes
df["sex"] = df["sex"].cat.codes
df.dtypes
# Exercise 
# print the values inside the "sex" column to see if they are what you expected<jupyter_output><empty_output><jupyter_text>Next, Let's do the same for the other categorical columns too.<jupyter_code>df["pclass"] = df["pclass"].astype('category').cat.codes
df["embarked"] = df["embarked"].astype('category').cat.codes
df["survived"] = df["survived"].astype('category').cat.codes

# Let's check the head to see if it looks ok
df.dtypes<jupyter_output><empty_output><jupyter_text>Next, let's take care of the `name` column.Perhaps the titles of the passengers would have some corrolation with their survival. Let's see.<jupyter_code># Extract Titles from name

# Here's what we are doing
# Left handside: create a new column called titles
# Right handside: 
#   .str: treat the names as string
#   .extract: extract the part of string that matches our regular rexpression ' ([A-Za-z]+)\.'
# See the next cell for what a regular expression is.
df['title'] = df["name"].str.extract(' ([A-Za-z]+)\.')
df['title']<jupyter_output><empty_output><jupyter_text>Regular Expressions: Also called regex, regular expressions are a syntax that describes textual patterns. You can read this tutorial on [python regex](https://docs.python.org/3/howto/regex.html) to learn more.Specifically, here our regex ` ([A-Za-z]+)\.` means to find any word pattern that starts with one space(` `), ends with one dot (`\.`), and has one or more `+` upper or lower case letters `[Az-az]`. [Click Here](https://regexr.com/5n4td) to see a visual description of what this regex does or create your own. Now that we have our titles let's group the rare ones since the specific title probably won't have much value, but whether it's a generic or rare one might. `.count_values()` counts unique values in a column and shows you how many times each repeats.<jupyter_code># Let's see the unique titles:

df["title"].value_counts()<jupyter_output><empty_output><jupyter_text>`.replace(x,y)` replaces all occurrences of `x` with `y` within the given column.<jupyter_code># Let's fix the typos
df["title"] = df["title"].replace('Mlle', "Miss")
df["title"] = df["title"].replace('Ms', "Miss")
df["title"] = df["title"].replace('Mme', "Miss")
df["title"] = df["title"].replace('', "None")
# Let's replace the rare ones with keyword 'rare'
# Here we are selecting the rare titles as anything not in a list
df["title"][~df["title"].isin(["Mr", "Miss", "Mrs", "Master"])] = "Rare"

df["title"].value_counts()<jupyter_output><empty_output><jupyter_text>Now let's see if our hypothesis seems reasonable that rare titles have a different chance of survival. To do this, we will make a bar plot using `sns.barplot` and see if there is a notable difference between the averages.<jupyter_code>sns.barplot(x='title', y='survived', data=df)<jupyter_output><empty_output><jupyter_text>There seems to be a correlation between the title and survival rate. Even within the same gender (for instance, male), the title seems to make a difference in the survival rate.Conclusion: So let's keep the new `title` column and drop the `name`. There seems to be value to be gained from the title column.`.drop()` is used for this, and we use `axis=1` to indicate it is a column that we are dropping, not a row.<jupyter_code>df = df.drop("name", axis=1)
df.head()<jupyter_output><empty_output><jupyter_text>Finally, let's replace the Title column with numerical values and we are ready to train own Machine Learning Model.<jupyter_code># we will do this in steps so it's easier to follow
titles = df["title"].astype('category').cat.codes
titles_as_category = titles.astype('category')
title_codes = titles_as_category.cat.codes

# Assign the title codes 
df["title"] = title_codes

# Print the top of the table to check
df.head()<jupyter_output><empty_output><jupyter_text>Data AnalysisNow that we have collected, cleaned, and wrangled our data, it is time to build our predictive model. Note that there are different families (classes) of predictive models (supervised, unsupervised, neural networks, ...), and each family has many models to select from. The choice of algorithm is one that you make in a real-life scenario, experiment and see the results of your technique chosen, and then decide how to proceed:- Explore another family/model.- Keep the one you have trained.- Create a hybrid (bagging, boosting, ensemble, etc) to capitalize on multiple models' strength.In a real-life scenario, you would want to do some research on each model class, learn their strengths, weaknesses, input/output format, and then pick the most promising ones based on your knowledge of the problem that you are solving.To keep this notebook within our timelimit, we will explore only one "classifier" model from the "supervised learning" class, called "Decision Tree".  What is a Decision TreeFrom scikit-learn's documentation:> Decision Trees (DTs) are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. A tree can be seen as a piecewise constant approximation.Simply put, this model is a tree-like set of decisions that finally lead to a prediction. As for its strengths, here are some of the ones that relate to our problem today from scikit-learn's documentation:>    Some advantages of decision trees are:        - Simple to understand and to interpret. Trees can be visualised.        - Requires little data preparation. Other techniques often require data normalisation, dummy variables need to be created and blank values to be removed. Note however that this module does not support missing values.        - Able to handle both numerical and categorical data. However scikit-learn implementation does not support categorical variables for now. Other techniques are usually specialised in analysing datasets that have only one type of variable.        - Uses a white box model. If a given situation is observable in a model, the explanation for the condition is easily explained by boolean logic. By contrast, in a black box model (e.g., in an artificial neural network), results may be more difficult to interpret.        - Possible to validate a model using statistical tests. That makes it possible to account for the reliability of the model.And as for relevant weaknesses:>    The disadvantages of decision trees include:        - Decision-tree learners can create over-complex trees that do not generalise the data well. This is called overfitting. Mechanisms such as pruning, setting the minimum number of samples required at a leaf node or setting the maximum depth of the tree are necessary to avoid this problem.        - Decision trees can be unstable because small variations in the data might result in a completely different tree being generated. This problem is mitigated by using decision trees within an ensemble.        - Predictions of decision trees are neither smooth nor continuous, but piecewise constant approximations. Therefore, they are not good at extrapolation.        - There are concepts that are hard to learn because decision trees do not express them easily, such as XOR, parity or multiplexer problems.        - Decision tree learners create biased trees if some classes dominate. It is therefore recommended to balance the dataset prior to fitting with the decision tree.With these pros and cons in mind, Decisions Trees seem like a strong candidate for our problem, so let's use them and see how they perform.[Learn more](https://scikit-learn.org/stable/modules/tree.html)  Packages used: scikit-learnWe will learn and use scikit learn, which implements user-friendly methods to perform the training and testing of an algorithm for us. There are other libraries out there that could do the same. However, they often require more in-depth knowledge of how the algorithms work internally to configure correctly.A few concepts that we should learn before moving further: training/validation and test splitWe often split our dataset into three sections (train/validate/test), often with ratios (50/25/25). This is, however, not always the case, and in simplified scenarios, such as our use-case, one might only use a (train/validate) split with a ratio of (70/30). We discuss the use of the "test" below.- The training set is shown to the model, so it learns the patterns to use for its predictions- The validation set is then used to assess the true performance of the model. This technique helps find out if the model just memorized the input data or if it can in-fact predict unseen data correctly (aka. [generalize](https://en.wikipedia.org/wiki/Generalization_error)).- The test set is used when the validation error is used for selecting the best model or when performing [hyper-parameter optimization](https://en.wikipedia.org/wiki/Hyperparameter_optimization)). In those cases, because we are using the error rate from the validation set to pick the best model, our validation error rate will be lower than the model's true error rate. Therefore, we need a set of data that we never used to optimize or pick our model to assess an unbiased error rate.[Learn More](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets)Since, in this case, we are just using one model (no model selection), and once trained, we aren't tuning it any further (no Hyperparameter_optimization), we can skip the test set and use a 70/30 train/validate split. Model fitIt is crucial to know how a model's prediction fits the actual predictions. If the model is not adequately trained, the training phase's predictions can't closely match the true predictions. This is called underfitting. In an underfit case, the model will also perform poorly on the evaluation set.On the other hand, if a model performs amazingly on the training set (think 99% accurate) but poorly on the validation set, it is most likely overfit. You can think of this as a model that has almost memorized the training set and therefore is incapable of performing well in predicting new inputs. If a model is neither underfitting nor overfit, it is called balanced. This is the place we want to be in, and this is why we keep unseen data (validation and test splits). We use those unseen data to detect if our model is overfitting the data.[Learn More](https://en.wikipedia.org/wiki/Overfitting) Uppercase `X` and lowercase `y`A tradition among data scientists is to use uppercase `X` to denote the input data to a model and the lowercase `y` the target predictions or values that the model is expected to learn and predict. That is, each set (train/validate/test) is split into `X` and `y`, where `y` is the predicted column, and `X` is all the other columns.With that, let's split our data, perform our training, and see how they perform on unseen data (validation).<jupyter_code># import the libraries we need
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

# Split target column our
X = df.drop("survived", axis = 1)
y = df["survived"]

# perform the train/validation split
# test is our validation here since we won't further split out train set
# We specify the random state here for reprocucable results. 
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=20)

# Let's check the shapes (table dimentions) to make sure it makes sense
X_train.shape, y_train.shape, X_test.shape
# Exercise:
# Take a look at the data splits [X_train, X_test, y_train, y_test] and see if they make sense to you. You can explore, length, columns, dtypes, ...<jupyter_output><empty_output><jupyter_text>Note: The following line won't work, once you see the error, read it try to debug the issue in the next cell, then move forward to see the solution.<jupyter_code># Decision Tree
decision_tree = DecisionTreeClassifier()

# Note: This won't work, once you see the error, read it try to debug the issue, then move forward to see the solution.
decision_tree.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Looks like something is wrong. Take a look at the ValueError message and use the cell below to explore and see if you can findout what is wrong. We left a small mistake intentionally to experience the debugging process<jupyter_code># Exrcise 
# Explore the copy of our original table below and find out what is wrong
# Make sure not to modify to original df, and only work on the df2
df2 = df.copy()

# You may explore df2 here<jupyter_output><empty_output><jupyter_text>Solution:As the error says:> Input contains NaN, infinity or a value too large for dtype('float32')So let's find the column with NaN and fill it with an average for that column.We will use the `.isnull()` to fill the table with true wherever there is a null, and then use the `.any()` to show if in each column there is any True (which would mean a `NaN`). You can run these steps one-by-one<jupyter_code>df.isnull().any()<jupyter_output><empty_output><jupyter_text>Now let's count the `NaN` in fare column. Note that `df["fare"].isnull().count()` wouldn't work as it would count all values: `true` and `false`! Instead, use `.sum()` that treats `true` as 1 and `false` as 0, hence counting the `true`s by summing.<jupyter_code>df["fare"].isnull().sum()<jupyter_output><empty_output><jupyter_text>ok there is only one missing value, so let's replace it with the mean and move on.abs<jupyter_code>fare_mode = df["fare"].mode()[0]
df["fare"] = df["fare"].fillna(fare_mode)

# and let's make sure there are no more nulls
df.isnull().any()<jupyter_output><empty_output><jupyter_text>Perfect, no more nulls, we can now train our model<jupyter_code># Redo our split with fixed data
X = df.drop("survived", axis = 1)
y = df["survived"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)

# Decision Tree
decision_tree = DecisionTreeClassifier()

# Train Model
decision_tree.fit(X_train, y_train)

# Make predictions for validation set
y_pred = decision_tree.predict(X_test)

# calculate models accuracy on training set
train_score = decision_tree.score(X_train, y_train)
test_score = decision_tree.score(X_test, y_test)

# Print results
# We know this is a new syntax to format and round numbers to two decimal places within a string. 
# You can learn more at https://www.w3schools.com/python/ref_string_format.asp
print("Trainins Score: {:.2f} validation score: {:.2f}".format(train_score, test_score))<jupyter_output>Trainins Score: 0.97 validation score: 0.74<jupyter_text>It looks like our decision tree is overfitting! It does very well with the training data 97% accuracy and performs poorly on unseen data (validation set). How to counter that is beyond this notebook's scope, but you can begin by forcing each branch of the tree to have at least 10 leaves. That is replacing the following line```pythondecision_tree = DecisionTreeClassifier()```with:```pythondecision_tree = DecisionTreeClassifier(min_samples_leaf=10)```This should reduce overfitting and improve our accuracy to about 80%!<jupyter_code># Tuning the Tree Parameters
decision_tree = DecisionTreeClassifier(min_samples_leaf=10)

# Train Model
decision_tree.fit(X_train, y_train)

# Make predictions for validation set
y_pred = decision_tree.predict(X_test)

# calculate models accuracy on training set
train_score = decision_tree.score(X_train, y_train)
test_score = decision_tree.score(X_test, y_test)

# Print results
print("Trainins Score: {:.2f} validation score: {:.2f}".format(train_score, test_score))<jupyter_output>Trainins Score: 0.85 validation score: 0.79
{'fraction_non_alphanumeric': 0.08300890256186873, 'fraction_numerical': 0.005199967330048188, 'mean_word_length': 4.404944084755739, 'pattern_counts': {'":': 0, '<': 167, '<?xml version=': 0, '>': 173, 'https://': 23, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 30, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '42', 'dir': 'jupyter-structured-clean-dedup', 'id': '279666', 'license': 'Apache-2.0', 'n_tokens_mistral': 10265, 'n_tokens_neox': 9750, 'n_words': 4838, 'path': 'notebooks/Titanic.ipynb', 'repo_name': 'IBM/python-and-analytics', 'size': '36731'}