<jupyter_start><jupyter_text>1. Getting started: Kaggle's Titanic CompetitionKaggle is already established as the best place which hosts machine learning competitions. If you do not know it already, then it's time to do it.__[Titanic Competition](https://www.kaggle.com/c/titanic)__ is perhaps the first competition which one should try. Of course, if you are already an experienced data scientist, than you can skip to an advanced competition.The purpose of the competition is to learn if a passenger has survived or not. We illustrate some steps and ideas one can apply to compete in this learning competition using the available tools one can find in rapaio library.<jupyter_code>%jars /home/ati/work/out/artifacts/rapaio_jar/rapaio.jar /home/ati/work/out/artifacts/rapaio_jar/fastutil-8.2.1.jar
import java.util.stream.*;
import java.util.function.*;

import rapaio.datasets.*;
import rapaio.data.*;
import rapaio.sys.*;
import rapaio.graphics.*;
import rapaio.graphics.plot.*;
import static rapaio.graphics.Plotter.*;
import rapaio.io.*;
import rapaio.core.*;
import rapaio.core.distributions.*;
import rapaio.core.tests.*;
import rapaio.core.tools.*;
import rapaio.core.stat.*;
import rapaio.ml.eval.*;
import rapaio.ml.common.*;
import rapaio.ml.classifier.*;
import rapaio.ml.classifier.tree.*;
import rapaio.ml.classifier.ensemble.*;

import rapaio.data.filter.*;
import rapaio.data.filter.var.*;
import rapaio.data.filter.frame.*;

import rapaio.experiment.ml.eval.*;

WS.getPrinter().withGraphicWidth(800);
WS.getPrinter().withGraphicHeight(600);
WS.getPrinter().withTextWidth(100);<jupyter_output><empty_output><jupyter_text>1.2 Get the dataThe purpose of the competition is to predict which passengers have survived or not. The available data has two parts. The first part consists in a data set which contains what happened with some passengers and some related information like sex, cabin, age, class, etc. This data set contains information regarding their survival. The purpose why this data set contains survival data is because it will be used to train a model which learns how to decide if a passenger survives or not. This is the `train.csv`. The other file is a data set which contains data about another set of passenger, this time without knowing if they survived or not. They contain, however an identification number. This data set is `test.csv` and this is used to make predictions. Those predictions should be similar with the provided `gendermodel.csv`.We also have to take a look of the data description provided on __[contest dedicated page](https://www.kaggle.com/c/titanic/data)__:```VARIABLE DESCRIPTIONS:survival Survival(0 = No; 1 = Yes)pclass Passenger Class(1 = 1st; 2 = 2nd; 3 = 3rd)name Namesex Sexage Agesibsp Number of Siblings/Spouses Aboardparch Number of Parents/Children Aboardticket Ticket Numberfare Passenger Farecabin Cabinembarked Port of Embarkation(C = Cherbourg; Q = Queenstown; S = Southampton)SPECIAL NOTES:Pclass is a proxy for socio-economic status (SES)1st ~ Upper; 2nd ~ Middle; 3rd ~ LowerAge is in Years; Fractional if Age less than One (1)If the Age is Estimated, it is in the form xx.5With respect to the family relation variables (i.e. sibsp and parch)some relations were ignored. The following are the definitions usedfor sibsp and parch.Sibling: Brother, Sister, Stepbrother, or Stepsister of Passenger Aboard TitanicSpouse: Husband or Wife of Passenger Aboard Titanic (Mistresses and Fiances Ignored)Parent: Mother or Father of Passenger Aboard TitanicChild: Son, Daughter, Stepson, or Stepdaughter of Passenger Aboard TitanicOther family relatives excluded from this study include cousins,nephews/nieces, aunts/uncles, and in-laws. Some children travelledonly with a nanny, therefore parch=0 for them. As well, sometravelled with very close friends or neighbors in a village, however,the definitions do not support such relations.```The first step in our adventure is to download those 3 data file in *csv* format. You can do it from __[data section](https://www.kaggle.com/c/titanic/data)__ of the competition. Let's suppose you downloaded somewhere in a local folder. We will name this folder `data` folder, and actually it can have any name you would like. 1.2 Read train data from csv fileBecause the data is small we can load the whole data in memory with no problems.Let's see how we can load the data into memory. In rapaio the sets of data are loaded into the form of *frames* (`rapaio.data.Frame`). A frame is basically a tabular data, with columns for each variable (feature) and rows for each instance (in our case for each passenger).A first try of loading the train data set and see what has happened is the following:<jupyter_code>String root = "/home/ati/work/rapaio-kaggle/src/titanic/";
Csv.instance().read(root + "train.csv").printSummary();<jupyter_output>Frame Summary
=============
* rowCount: 891
* complete: 889/891
* varCount: 12
* varNames: 

 0. PassengerId : double  | 4.   Sex : nominal |  8.   Ticket : nominal |
 1.    Survived : binary  | 5.   Age : nominal |  9.     Fare : double  |
 2.      Pclass : double  | 6. SibSp : double  | 10.    Cabin : nominal |
 3.        Name : nominal | 7. Parch : double  | 11. Embarked : nominal |

      PassengerId         Survived           Pclass 
   Min. :   1.000     Min. : 0.000     Min. : 1.000 
1st Qu. : 223.500  1st Qu. : 0.000  1st Qu. : 2.000 
 Median : 446.000   Median : 0.000   Median : 3.000 
   Mean : 446.000     Mean : 0.384     Mean : 2.309 
2nd Qu. : 668.500  2nd Qu. : 1.000  2nd Qu. : 3.000 
   Max. : 891.000     Max. : 1.000     Max. : 3.000 
                                                    
                                                       Name           Sex            Age            SibSp 
                            "Braund, Mr. <NAME>" :   1    male : 577      [...]<jupyter_text>How can we interpret the output of the frame's summary?* We loaded a frame which has $891$ rows and $12$ columns (variables)* From all the rows, $889$ are complete (non missing data)* The name of the variables are listed, together with their types* It follows a data summary for the frame: 6 number summary for numeric variables, most frequent levels for nominal variablesLet's inspect each variable and see how it fits our needs.**PassengedId**The type for this variable is index (integer values). This field looks like an identifier for the passenger, so from our point of view the sorting is not required. What we can do, but is not required, is to change the field type to nominal. Anyway, we do not need this field for learning since it should be unique for each instance, thus the predictive power is null. We will ignore it for now since we will not consider it for learning**Survived**This is our target variable. It is parsed as binary, but since we do classification, we will change it's type to nominal. We do that directly from the csv parsing, by indicating that we want Survived parsed as nominal variable:<jupyter_code>Csv.instance()
.withTypes(VType.NOMINAL, "Survived")
.read(root + "train.csv")
.printSummary();<jupyter_output>Frame Summary
=============
* rowCount: 891
* complete: 889/891
* varCount: 12
* varNames: 

 0. PassengerId : double  | 4.   Sex : nominal |  8.   Ticket : nominal |
 1.    Survived : nominal | 5.   Age : nominal |  9.     Fare : double  |
 2.      Pclass : double  | 6. SibSp : double  | 10.    Cabin : nominal |
 3.        Name : nominal | 7. Parch : double  | 11. Embarked : nominal |

      PassengerId  Survived           Pclass                                                         Name 
   Min. :   1.000   0 : 549     Min. : 1.000                              "Braund, Mr. <NAME>" :   1 
1st Qu. : 223.500   1 : 342  1st Qu. : 2.000  "Cumings, Mrs. <NAME> (<NAME>)" :   1 
 Median : 446.000             Median : 3.000                               "<NAME>" :   1 
   Mean : 446.000               Mean : 2.309         "Futrelle, Mrs. <NAME> (<NAME>)" :   1 
2nd Qu. : 668.500            2nd Qu. : 3.000                             "A[...]<jupyter_text>And notice how type of the `Survived` variable changed to nominal.**Pclass**This variable has index type. We can keep it like it is or we can change it to nominal. Both ways can be useful. For example parsed as index could give an interpretation to the order. We can say that somehow, because of ordering class 1 is lower than class 2, and class 2 is between classes 1 and 3. At the same time we can keep it as nominal if we do not want to use the ordering. Let's choose nominal for now, considering that 1,2 and 3 are just labels for type of tickets, with no other meaning attached. We proceed in the same way:<jupyter_code>Csv.instance()
.withTypes(VType.NOMINAL, "Survived", "Pclass")
.read(root + "train.csv")
.printSummary();<jupyter_output>Frame Summary
=============
* rowCount: 891
* complete: 889/891
* varCount: 12
* varNames: 

 0. PassengerId : double  | 4.   Sex : nominal |  8.   Ticket : nominal |
 1.    Survived : nominal | 5.   Age : nominal |  9.     Fare : double  |
 2.      Pclass : nominal | 6. SibSp : double  | 10.    Cabin : nominal |
 3.        Name : nominal | 7. Parch : double  | 11. Embarked : nominal |

      PassengerId  Survived   Pclass                                                         Name 
   Min. :   1.000   0 : 549  3 : 491                              "<NAME>. <NAME>" :   1 
1st Qu. : 223.500   1 : 342  1 : 216  "Cumings, Mrs. <NAME> (<NAME>)" :   1 
 Median : 446.000            2 : 184                               "<NAME>" :   1 
   Mean : 446.000                            "Futrelle, Mrs. <NAME> (<NAME>)" :   1 
2nd Qu. : 668.500                                                "Allen, Mr. <NAME>" :   1 
   Max. : 891.00[...]<jupyter_text>Notice that we append the variable name after `Survived`. This is possible since the `withTypes` method specify a type, and follows a dynamic array of strings, for the names of variables.**Name**This is the passenger names and the values are unique. As it is, the predictive power of this field is null. We keep it as it is. Note that it contains valuable information, but not in this direct form.**Sex**This field specifies the gender of the passenger. We have $577$ males and $314$ females.**Age**This field specifies the age of an passenger. We would expect that to parse this variable as numeric or at leas index, but is nominal. Why that happened? Notice that the values looks like numbers. But the first value (the most frequent one, $117$ instances) has nothing specified. Well, the variable is nominal has to do with how *Csv* parsing handles missing values. By default, the *csv* parsing considers as missing values only the string "?". But the most frequent value in this field is empty string "". This means that empty string is not considered a missing value. Because empty string can't produce numbers from parsing, the variable is *promoted* to nominal.We can customize the missing value handling by specifying the valid strings for that purpose. We use `.useNAValues(String...naValues)` to tell the parser all the valid strings which are missing values. In our case we want just the empty string to be a missing value. When the parser will found an empty string it will set the variable value as missing value. It will *not promote* variable to nominal, since a missing value is a legal value.<jupyter_code>Csv.instance()
.withNAValues("")
.withTypes(VType.NOMINAL, "Survived", "Pclass")
.read(root + "train.csv")
.printSummary();<jupyter_output>Frame Summary
=============
* rowCount: 891
* complete: 183/891
* varCount: 12
* varNames: 

 0. PassengerId : double  | 4.   Sex : nominal |  8.   Ticket : nominal |
 1.    Survived : nominal | 5.   Age : double  |  9.     Fare : double  |
 2.      Pclass : nominal | 6. SibSp : double  | 10.    Cabin : nominal |
 3.        Name : nominal | 7. Parch : double  | 11. Embarked : nominal |

      PassengerId  Survived   Pclass                                                         Name 
   Min. :   1.000   0 : 549  3 : 491                              "Braund, Mr. <NAME>" :   1 
1st Qu. : 223.500   1 : 342  1 : 216  "Cumings, Mrs. <NAME> (<NAME>)" :   1 
 Median : 446.000            2 : 184                               "<NAME>" :   1 
   Mean : 446.000                            "Futrelle, Mrs. <NAME> (<NAME>)" :   1 
2nd Qu. : 668.500                                                "Allen, Mr. <NAME>" :   1 
   Max. : 891.00[...]<jupyter_text>Notice what happened: *Age* field is now numeric and it contains $177$ missing values.**SibSp**It's meaning is "siblings/spouses". It's parsed as index, which is natural. In pathological cases with sick imagination we can consider a "quarter of a wife" for example.**Parch**It's meaning is "parents/children". It is naturally parsed as index.**Ticket**This is the code of the ticket. Probably a family can have the same ticket, thus must be the reason why the frequencies have values up to $$7$$. This field is nominal. It has low predictive power used directly. Perhaps contains valuable information, but used directly in row format would not help much.**Fare**This is the price for passenger fare and should be numeric, like it is.**Cabin**Code of the passenger's cabin, parsed as nominal. Same notes as for `Ticket` variable.**Embarked**Code for the embarking city, which could be: C = Cherbourg, Q = Queenstown, S = Southampton. It's parsed as nominal and has $2$ missing values.If we are content with our parsing, we load data into a data frame for later use:<jupyter_code>Frame train = Csv.instance().withNAValues("").withTypes(VType.NOMINAL, "Survived", "Pclass")
.read(root + "train.csv");
train.printSummary();<jupyter_output>Frame Summary
=============
* rowCount: 891
* complete: 183/891
* varCount: 12
* varNames: 

 0. PassengerId : double  | 4.   Sex : nominal |  8.   Ticket : nominal |
 1.    Survived : nominal | 5.   Age : double  |  9.     Fare : double  |
 2.      Pclass : nominal | 6. SibSp : double  | 10.    Cabin : nominal |
 3.        Name : nominal | 7. Parch : double  | 11. Embarked : nominal |

      PassengerId  Survived   Pclass                                                         Name 
   Min. :   1.000   0 : 549  3 : 491                              "Braund, Mr. <NAME>" :   1 
1st Qu. : 223.500   1 : 342  1 : 216  "Cumings, Mrs. <NAME> (<NAME>)" :   1 
 Median : 446.000            2 : 184                               "<NAME>" :   1 
   Mean : 446.000                            "Futrelle, Mrs. <NAME> (<NAME>)" :   1 
2nd Qu. : 668.500                                                "Allen, Mr. <NAME>" :   1 
   Max. : 891.00[...]<jupyter_text>1.3 Read test data from *csv* fileOnce we have a training frame we can load also the test data. We do that to take a look at the frame and because data is small and there is no memory or time problem cost associated with it. To avoid adding again the *csv* options and to get identical levels nominal variables, we use a different way to parse the data set. We specify variable types by frame templates:<jupyter_code>Frame test = Csv.instance().withNAValues("").withTemplate(train).read(root + "test.csv");<jupyter_output><empty_output><jupyter_text>Instead to specify again the preferred types for variables, we use train frame as a template for variable types. This has also the side effect that the encoding of categorical variables is identical.<jupyter_code>test.printSummary();<jupyter_output>Frame Summary
=============
* rowCount: 418
* complete: 87/418
* varCount: 11
* varNames: 

 0. PassengerId : double  | 4.    Age : double  |  8.     Fare : double  |
 1.      Pclass : nominal | 5.  SibSp : double  |  9.    Cabin : nominal |
 2.        Name : nominal | 6.  Parch : double  | 10. Embarked : nominal |
 3.         Sex : nominal | 7. Ticket : nominal |                         

       PassengerId   Pclass                                                  Name           Sex 
   Min. :  892.000  3 : 218                          "<NAME>" :   1    male : 266 
1st Qu. :  996.250  1 : 107                              "<NAME>" :   1  female : 152 
 Median : 1100.500  2 :  93              "<NAME> (<NAME>)" :   1               
   Mean : 1100.500                              "Myles, Mr. <NAME>" :   1               
2nd Qu. : 1204.750                                       "<NAME>" :   1               
   Max. : 1309.000       [...]<jupyter_text>We can note that we don't have *Survived* variable anymore. This is correct since this is what we have to predict. Note also that the types for the remaining variables are the same with training data set.  2. Build simple models 2.1 Build a majority modelTo make a first submission we will build a very simple model, which classifies with a single value all instances. This value is the majority label.Let's inspect at how target variable look like.<jupyter_code>DVector.fromCount(false, train.rvar("Survived")).printSummary();<jupyter_output>0   1
    -   -
  549 342<jupyter_text>As we already new from the summary, the number of passengers who didn't survived is lower than those who did. Let's see percentages:<jupyter_code>DVector.fromCount(false, train.rvar("Survived")).normalize().printSummary();<jupyter_output>0         1
          -         -
  0.6161616 0.3838384<jupyter_text>We note that there are about $61\%$ of passengers who did not survived. We will create a submit data set, which we will save for later submission. How we can do that?<jupyter_code>VarNominal prediction = VarNominal.from(test.rowCount(), row -> "0").withName("Survived");
Frame submit = SolidFrame.byVars(test.rvar("PassengerId"), prediction);

Csv.instance().withQuotes(false).write(submit, root + "majority_submit.csv");<jupyter_output><empty_output><jupyter_text>In the first line we created a new nominal variable. The size of the new variable is the number of rows from the test frame. For each row we produce the same label `"0"`. We name this variable `Survived`.In the second line we created a new frame taking the variable named `PassengerId` from the test data set and the new prediction variable.In the last line we wrote a new csv file with the csv parsing utility, taking care to not write quotes. We can submit this file and see which are the results.![Submission result with majority classifier](images/titanic-majority-submit.png)  2.2 Build a simple gender modelIt has been said that "women and children first" really happened during Titanic tragedy. If this was true or not, we do not know. But we can use data to see if we are hearing the same story. For now we will take the gender and see if it had an influence. We will build a contingency table for variables `Sex` and `Survived`.<jupyter_code>DTable.fromCounts(train.rvar("Sex"), train.rvar("Survived"), false).printSummary();<jupyter_output>0   1 total
   male 468 109   577
 female  81 233   314
  total 549 342   891<jupyter_text>On rows we have levels of `Sex` variable. On columns we have levels of `Sex` variable. Cells are computed as counts. What we see is that there are a lot of men who did not survived and a lot of women who does. We will normalize on rows to take a closer look.<jupyter_code>DTable.fromCounts(train.rvar("Sex"), train.rvar("Survived"), false).normalizeOnRows().printSummary();<jupyter_output>0         1 total
   male 0.8110919 0.1889081     1
 female 0.2579618 0.7420382     1
  total 1.0690536 0.9309464     2<jupyter_text>It seems that men survived with a rate of $0.19$ and women with $0.74$. The values are so obvious, we need no hypothesis testing to check that this variable is significant for classification. We will build a simple model where we predict as survived all the women and not survived all the men.<jupyter_code>Var prediction = VarNominal.from(test.rowCount(), row -> test.getLabel(row, "Sex").equals("male") 
? "0" : "1").withName("Survived");
Frame submit = SolidFrame.byVars(test.rvar("PassengerId"), prediction);
Csv.instance().withQuotes(false).write(submit, root + "gender_submit.csv");<jupyter_output><empty_output><jupyter_text>__![Submission result with gender classifier](images/titanic-gender-submit.png)__  3. Tree modelBuilding models in the manual way is often not the way to go. This process is tedious and time consuming. There are already built automated procedures, which incorporate miscellaneous approaches to learn a classifier. One of the often used models is the decision tree. Decision trees are greedy local approximations build in a recursive greedy fashion. Often the split decision at node level uses a single feature. At leave nodes a simple majority classifier creates the classification rule. 3.1 Gender model with decision treeInitially we will build a CART decision tree using as input feature the *Sex* variable. We do this to exemplify how a manual rule can be created in an automated fashion.<jupyter_code>Frame tr = train.mapVars("Survived,Sex");
CTree tree = CTree.newCART();
tree.fit(tr, "Survived");
tree.printSummary();<jupyter_output>CTree model
================

Description:
CTree {varSelector=VarSelector[ALL];
minCount=1;
maxDepth=-1;
tests=NOMINAL:NominalBinary,DOUBLE:NumericBinary,BINARY:BinaryBinary,INT:NumericBinary;
func=GiniGain;
split=ToAllWeighted;
}

Capabilities:
types inputs/targets: BINARY,INT,NOMINAL,DOUBLE/NOMINAL
counts inputs/targets: [1,1000000] / [1,1]
missing inputs/targets: true/false

Learned model:
input vars: 
 0. Sex : NOMINAL  |               

target vars:
> Survived : NOMINAL [?,0,1]


total number of nodes: 3
total number of leaves: 2
description:
split, n/err, classes (densities) [* if is leaf / purity if not]

|- 0. root    891/342 0 (0.616 0.384 ) [0.139648]
|   |- 1. Sex = 'female'    314/81 1 (0.258 0.742 ) *
|   |- 2. Sex != 'female'    577/109 0 (0.811 0.189 ) *<jupyter_text>Taking a closer look at the last three rows from the output, one can identify our manual rule. Basically the interpretation is: *"all the females survived, all the males did not"*. For exemplification purposes we build also the submit file.<jupyter_code>// fit the tree to test data frame
CPrediction pred = tree.predict(test);
// build teh submission
Frame submit = SolidFrame.byVars(test.rvar("PassengerId"),pred.firstClasses().withName("Survived"));
// write to a submit file
Csv.instance().withQuotes(false).write(submit, root + "tree1-model.csv");<jupyter_output><empty_output><jupyter_text>3.2 Enrich tree by using other featuresOur training data set has more than a single input feature. Thus We can state we didn't use all the information available. We will add now the class and embarking port and see how it behaves.<jupyter_code>Frame tr = train.mapVars("Survived,Sex,Pclass,Embarked");

CTree tree = CTree.newCART();
tree.fit(tr, "Survived");
tree.printSummary();

CPrediction pred = tree.predict(test);
Frame submit = SolidFrame.byVars(test.rvar("PassengerId"),pred.firstClasses().withName("Survived"));
Csv.instance().withQuotes(false).write(submit, root + "tree2-model.csv");<jupyter_output>CTree model
================

Description:
CTree {varSelector=VarSelector[ALL];
minCount=1;
maxDepth=-1;
tests=NOMINAL:NominalBinary,DOUBLE:NumericBinary,BINARY:BinaryBinary,INT:NumericBinary;
func=GiniGain;
split=ToAllWeighted;
}

Capabilities:
types inputs/targets: BINARY,INT,NOMINAL,DOUBLE/NOMINAL
counts inputs/targets: [1,1000000] / [1,1]
missing inputs/targets: true/false

Learned model:
input vars: 
 0. Sex : NOMINAL  | 1. Pclass : NOMINAL  | 2. Embarked : NOMINAL  |

target vars:
> Survived : NOMINAL [?,0,1]


total number of nodes: 17
total number of leaves: 9
description:
split, n/err, classes (densities) [* if is leaf / purity if not]

|- 0. root    891/342 0 (0.616 0.384 ) [0.139648]
|   |- 1. Sex = 'female'    314/81 1 (0.258 0.742 ) [0.0204665]
|   |   |- 3. Pclass = '2'    76/6 1 (0.079 0.921 ) [0.0003369]
|   |   |   |- 7. Embarked = 'Q'    2/0 1 (0 1 ) *
|   |   |   |- 8. Embarked != 'Q'    74/6 1 (0.081 0.919 ) [0.0013737]
|   |   |   |   |- 13. Embarked = 'C'    7/0 1[...]<jupyter_text>The tree is much richer and there are more chances to be better. This is what happened after submission.__![Results after submission of enriched tree](images/titanic-tree2-submit.png)__**Nice!**. We advanced $704$ positions and improved our score with $0.01435$. On public leader board we have a nice $0.77990$ accuracy score.  3.3 Overfitting with treesWhat about using other input features to improve our prediction accuracy? There are some of them which we can include directly, with no changes: *Age*,*Fare*,*SibSp* and *Parch*.We can change the script slightly, to include those new input features. But we can do better, we can use cross-validation to estimate what will happen.<jupyter_code>CTree tree = CTree.newCART();
CEvaluation.cv(train.mapVars("Survived,Sex,Pclass,Embarked"),"Survived", tree, 10);
CEvaluation.cv(train.mapVars("Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch"),"Survived", tree, 10);<jupyter_output>CrossValidation with 10 folds
CV  1:  acc=0.766667, mean=0.766667, se=NaN
CV  2:  acc=0.775281, mean=0.770974, se=0.006091
CV  3:  acc=0.775281, mean=0.772409, se=0.004973
CV  4:  acc=0.786517, mean=0.775936, se=0.008139
CV  5:  acc=0.786517, mean=0.778052, se=0.008490
CV  6:  acc=0.820225, mean=0.785081, se=0.018817
CV  7:  acc=0.764045, mean=0.782076, se=0.018928
CV  8:  acc=0.764045, mean=0.779822, se=0.018648
CV  9:  acc=0.831461, mean=0.785560, se=0.024506
CV 10:  acc=0.797753, mean=0.786779, se=0.023424
==============
Mean accuracy:0.786779
SE: 0.023424     (Standard error)

CrossValidation with 10 folds
CV  1:  acc=0.788889, mean=0.788889, se=NaN
CV  2:  acc=0.775281, mean=0.782085, se=0.009622
CV  3:  acc=0.775281, mean=0.779817, se=0.007857
CV  4:  acc=0.775281, mean=0.778683, se=0.006804
CV  5:  acc=0.865169, mean=0.795980, se=0.039124
CV  6:  acc=0.764045, mean=0.790658, se=0.037343
CV  7:  acc=0.775281, mean=0.788461, se=0.034581
CV  8:  acc=0.764045, mean=0.785409, se=0.0[...]<jupyter_text>Notice that the 10-crossfold estimator of the accuracy has dropped with a large quantity. What happens? We can have an idea if we take a look at the learned tree:<jupyter_code>tree.fit(train.mapVars("Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch"), "Survived");
tree.printSummary();<jupyter_output>CTree model
================

Description:
CTree {varSelector=VarSelector[ALL];
minCount=1;
maxDepth=-1;
tests=NOMINAL:NominalBinary,DOUBLE:NumericBinary,BINARY:BinaryBinary,INT:NumericBinary;
func=GiniGain;
split=ToAllWeighted;
}

Capabilities:
types inputs/targets: BINARY,INT,NOMINAL,DOUBLE/NOMINAL
counts inputs/targets: [1,1000000] / [1,1]
missing inputs/targets: true/false

Learned model:
input vars: 
 0.      Sex : NOMINAL  | 3.   Age : DOUBLE  | 6. Parch : DOUBLE  |
 1.   Pclass : NOMINAL  | 4.  Fare : DOUBLE  |                     
 2. Embarked : NOMINAL  | 5. SibSp : DOUBLE  |                     

target vars:
> Survived : NOMINAL [?,0,1]


total number of nodes: 513
total number of leaves: 257
description:
split, n/err, classes (densities) [* if is leaf / purity if not]

|- 0. root    891/342 0 (0.616 0.384 ) [0.5731486]
|   |- 1. Age <= 37.5    703/271 0 (0.608 0.392 ) [0.6304902]
|   |   |- 3. Age <= 27.5    514/193 0 (0.606 0.394 ) [0.11952]
|   |   |   |- 7. Sex = 'female[...]<jupyter_text>Notice how large is the tree. Basically the tree was full grown and overfit the training data set too much. We can ask ourselves why that happens? Why it happens now, and did not happened when we had fewer inputs? The answer is that it happened also before. But it's consequences were not so drastic.The first tree used for training just $3$ input nominal features. Notice that all three features are nominal. The maximum number of groups which one can form is given by the product of the number of levels for each feature. This total maximal number is $2*3*3=18$. It practically exhausted the discrimination potential of those features. It did overfit in that reduced space of features. When we apply the model to the whole data set, the effect of exhaustion is not seen anymore.The second tree does the same thing, but this time in a richer space, with added input dimensions. Compared with the full feature space, we see the effect.There are two approaches to avoid overfit for a decision tree. The first approach is to stop learning up to the moment when we exhaust the data. The name for this approach is *early stop*. We can do that by specifying some parameters of the tree model:* Set a minimum number of instances for leaf node* Set a maximal depth for the tree* Not implemented yet, but easy to do: complexity threshold, maximal number of nodes in a treeThe second approach is to prune the tree. Pruning procedure consists of growing the full tree and later on removing some nodes if they do not provide some type of gain. Currently we implemented only *reduced error pruning strategy*.We will test with 10-fold cross validation an early-stopping strategy to see how it works.<jupyter_code>CEvaluation.cv(train.mapVars("Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch"),
"Survived", tree.withMaxDepth(12).withMinCount(4), 10);<jupyter_output>CrossValidation with 10 folds
CV  1:  acc=0.788889, mean=0.788889, se=NaN
CV  2:  acc=0.764045, mean=0.776467, se=0.017567
CV  3:  acc=0.752809, mean=0.768581, se=0.018463
CV  4:  acc=0.775281, mean=0.770256, se=0.015442
CV  5:  acc=0.775281, mean=0.771261, se=0.013561
CV  6:  acc=0.752809, mean=0.768186, se=0.014278
CV  7:  acc=0.775281, mean=0.769199, se=0.013307
CV  8:  acc=0.820225, mean=0.775577, se=0.021846
CV  9:  acc=0.764045, mean=0.774296, se=0.020793
CV 10:  acc=0.831461, mean=0.780012, se=0.026666
==============
Mean accuracy:0.780012
SE: 0.026666     (Standard error)<jupyter_text>I tried some values, just to show that we can do something about it, but the progress did not appear. We should try a different approach, and that is an ensemble. Next session contains directions on how to build such an ensemble.  4. CForest modelRandom forests are well-known to work well when the irreducible error from the training data is high. This is probably the case of this Titanic data set. We have reasons to believe that this is the situation since it was a tragedy. A lot of random or not-so-expected things happened. That happened despite of the bravery and the sacrifice of the crew and others.Random forests are the invention of [Leo Breiman](https://en.wikipedia.org/wiki/Leo_Breiman). The first design was a joint effort together with [<NAME>](http://www.math.usu.edu/adele/). The base of random forests is bagging (or **b**ootstrapp **ag**gregation). On top of that, selecting just a random limited number of variables at each node is the core of the algorithm.We will work with random forests for now. This ensemble is mode robust and is capable of obtaining much better results than a single tree. At the same time we will introduce 10-fold cross validation to check our progress and estimate the error produced.In the beginning we will use 10-fold cross validation for estimating the accuracy on public leader board.We will test first with 10 fold cv the tree model.<jupyter_code>CEvaluation.cv(train.mapVars("Survived,Sex,Pclass,Embarked"), "Survived", CTree.newCART(), 10);<jupyter_output>CrossValidation with 10 folds
CV  1:  acc=0.800000, mean=0.800000, se=NaN
CV  2:  acc=0.842697, mean=0.821348, se=0.030191
CV  3:  acc=0.820225, mean=0.820974, se=0.021358
CV  4:  acc=0.719101, mean=0.795506, se=0.053839
CV  5:  acc=0.820225, mean=0.800449, se=0.047918
CV  6:  acc=0.741573, mean=0.790637, se=0.049139
CV  7:  acc=0.808989, mean=0.793258, se=0.045391
CV  8:  acc=0.831461, mean=0.798034, se=0.044141
CV  9:  acc=0.730337, mean=0.790512, se=0.047054
CV 10:  acc=0.752809, mean=0.786742, se=0.045937
==============
Mean accuracy:0.786742
SE: 0.045937     (Standard error)<jupyter_text>4.1 Our first random forestThe name of the random forest implementation is `CForest`. To build a new ensemble of trees, one have to instantiate it in the following way:<jupyter_code>Classifier rf = CForest.newRF();<jupyter_output><empty_output><jupyter_text>There are a lot of things which can be customized for a random forest. Among them one can change:* Number of trees for classification* Which kind of weak classifier to use (you can customize this customized accordingly, like any other classifier)* Number of threads in pool (if you want to use parallelism)* What to do after each running stepLet's build one and use ore new cross validation procedure to estimate it's error.<jupyter_code>RandomSource.setSeed(123);
Frame tr = train.mapVars("Survived,Sex,Pclass,Embarked");
CForest rf = CForest.newRF().withRuns(100);
CEvaluation.cv(tr, "Survived", rf, 10);<jupyter_output>CrossValidation with 10 folds
CV  1:  acc=0.733333, mean=0.733333, se=NaN
CV  2:  acc=0.808989, mean=0.771161, se=0.053496
CV  3:  acc=0.842697, mean=0.795006, se=0.056006
CV  4:  acc=0.764045, mean=0.787266, se=0.048278
CV  5:  acc=0.842697, mean=0.798352, se=0.048607
CV  6:  acc=0.831461, mean=0.803870, se=0.045528
CV  7:  acc=0.730337, mean=0.793365, se=0.049998
CV  8:  acc=0.820225, mean=0.796723, se=0.047253
CV  9:  acc=0.820225, mean=0.799334, se=0.044890
CV 10:  acc=0.674157, mean=0.786816, se=0.057949
==============
Mean accuracy:0.786816
SE: 0.057949     (Standard error)<jupyter_text>Well, an identical output. This is due to the fact that our variables are already exhausted by the tree. It looks like an underfit. If one consider bias variance trade off, one can see this as high bias. We need to enrich our feature space to improve our performance.Let's be direct and test what would happen if we would use all our directly usable features? This time we will fit also the training data set, to see the distribution of the training error.<jupyter_code>RandomSource.setSeed(123);
Frame tr = train.mapVars("Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch");
CForest rf = CForest.newRF().withRuns(100);
CEvaluation.cv(tr, "Survived", rf, 10);

rf.fit(tr, "Survived");
CPrediction fit = rf.predict(test);
new Confusion(tr.rvar("Survived"), rf.predict(tr).firstClasses()).printSummary();<jupyter_output>CrossValidation with 10 folds
CV  1:  acc=0.822222, mean=0.822222, se=NaN
CV  2:  acc=0.853933, mean=0.838077, se=0.022423
CV  3:  acc=0.876404, mean=0.850853, se=0.027222
CV  4:  acc=0.842697, mean=0.848814, se=0.022598
CV  5:  acc=0.775281, mean=0.834107, se=0.038268
CV  6:  acc=0.842697, mean=0.835539, se=0.034407
CV  7:  acc=0.764045, mean=0.825325, se=0.041433
CV  8:  acc=0.831461, mean=0.826092, se=0.038421
CV  9:  acc=0.842697, mean=0.827937, se=0.036363
CV 10:  acc=0.719101, mean=0.817054, se=0.048579
==============
Mean accuracy:0.817054
SE: 0.048579     (Standard error)
> Confusion

 Ac\Pr |    0    1 | total
 ----- |    -    - | -----
     0 | >527   22 |   549
     1 |   46 >296 |   342
 ----- |    -    - | -----
 total |  573  318 |   891


Complete cases 891 from 891
Acc: 0.9236813         (Accuracy )
F1:  0.9393939         (F1 score / F-measure)
MCC: 0.8378872         (Matthew correlation coefficient)
Pre: 0.9197208         (Precision)
Rec: 0.9599271         (Recall)
G:[...]<jupyter_text>This time we have a good example of overfit. Why is that? Look at the confusion matrix on the training set. We fit too well the training data. This data set is well known for its high irreducible error. And there is an explanation for that. During the tragic event a lot of exceptional things happened. For example I read somewhere that an old lady which had a dog was not allowed to embark with her pet due to regulations. As a consequence she decided to not leave it and she chose to die with him. It's close to impossible to learn those kind of things, even if the information would be available. We should reduce the error somehow. We can try to decrease the overfit by adding more learners. Let's see if that would be enough for our purpose.<jupyter_code>RandomSource.setSeed(123);
Frame tr = train.mapVars("Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch");
CForest rf = CForest.newRF().withRuns(500);
CEvaluation.cv(tr, "Survived", rf, 10);

rf.fit(tr, "Survived");
CPrediction fit = rf.predict(test);
new Confusion(tr.rvar("Survived"), rf.predict(tr).firstClasses()).printSummary();<jupyter_output>CrossValidation with 10 folds
CV  1:  acc=0.822222, mean=0.822222, se=NaN
CV  2:  acc=0.831461, mean=0.826841, se=0.006533
CV  3:  acc=0.876404, mean=0.843362, se=0.028986
CV  4:  acc=0.820225, mean=0.837578, se=0.026343
CV  5:  acc=0.786517, mean=0.827366, se=0.032279
CV  6:  acc=0.831461, mean=0.828048, se=0.028919
CV  7:  acc=0.752809, mean=0.817300, se=0.038803
CV  8:  acc=0.831461, mean=0.819070, se=0.036271
CV  9:  acc=0.842697, mean=0.821695, se=0.034831
CV 10:  acc=0.719101, mean=0.811436, se=0.046162
==============
Mean accuracy:0.811436
SE: 0.046162     (Standard error)
> Confusion

 Ac\Pr |    0    1 | total
 ----- |    -    - | -----
     0 | >526   23 |   549
     1 |   44 >298 |   342
 ----- |    -    - | -----
 total |  570  321 |   891


Complete cases 891 from 891
Acc: 0.9248036         (Accuracy )
F1:  0.9401251         (F1 score / F-measure)
MCC: 0.8402332         (Matthew correlation coefficient)
Pre: 0.922807         (Precision)
Rec: 0.9581056         (Recall)
G: [...]<jupyter_text>This is slightly better than before. But the difference does not look significantly better than previous. We will use a simple pre-pruning strategy is to limit the number instances in leaf nodes. We set the minimum count to $3$.<jupyter_code>RandomSource.setSeed(123);
Frame tr = train.mapVars("Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch");
CForest rf = CForest.newRF()
.withClassifier(CTree.newCART().withMinCount(3))
.withRuns(100);
CEvaluation.cv(tr, "Survived", rf, 10);

rf.fit(tr, "Survived");
CPrediction fit = rf.predict(test);
new Confusion(tr.rvar("Survived"), rf.predict(tr).firstClasses()).printSummary();<jupyter_output>CrossValidation with 10 folds
CV  1:  acc=0.800000, mean=0.800000, se=NaN
CV  2:  acc=0.820225, mean=0.810112, se=0.014301
CV  3:  acc=0.887640, mean=0.835955, se=0.045889
CV  4:  acc=0.820225, mean=0.832022, se=0.038285
CV  5:  acc=0.775281, mean=0.820674, se=0.041752
CV  6:  acc=0.853933, mean=0.826217, se=0.039736
CV  7:  acc=0.775281, mean=0.818941, se=0.041066
CV  8:  acc=0.808989, mean=0.817697, se=0.038182
CV  9:  acc=0.853933, mean=0.821723, se=0.037703
CV 10:  acc=0.719101, mean=0.811461, se=0.048132
==============
Mean accuracy:0.811461
SE: 0.048132     (Standard error)
> Confusion

 Ac\Pr |    0    1 | total
 ----- |    -    - | -----
     0 | >520   29 |   549
     1 |   67 >275 |   342
 ----- |    -    - | -----
 total |  587  304 |   891


Complete cases 891 from 891
Acc: 0.8922559         (Accuracy )
F1:  0.915493         (F1 score / F-measure)
MCC: 0.7706188         (Matthew correlation coefficient)
Pre: 0.8858603         (Precision)
Rec: 0.9471767         (Recall)
G: [...]<jupyter_text>Notice that we changed the classifier used by `CForest`. This is the same classifier used by default by random forest. We do this because we customized the classifier by changing the min count parameter. That had indeed some effect. However after submitting to competition we did not saw any improvement. We should look forward to engineer a little bit our features for further improvements.  5. Feature engineering 5.1 Title featureIt is clear that we can't use directly the `"Name"` variable. This is due to the fact that names are almost unique, and that leads to a tiny generalization power. To understand that we should see that even if we learned that a passenger with a given name survived or not. We can't decide if another passenger survived, using only the name of the new passenger.Lets inspect some of the values from `"Name"` variable.<jupyter_code>train.rvar("Name").printLines(20);<jupyter_output>[0]                                 "Braund, Mr. <NAME>"
  [1]     "Cumings, Mrs. <NAME> (<NAME>)"
  [2]                                  "Heikkinen, <NAME>"
  [3]            "Futrelle, Mrs. <NAME> (<NAME>)"
  [4]                                "Allen, Mr. <NAME>"
  [5]                                        "Moran, Mr. James"
  [6]                                 "McCarthy, Mr. <NAME>"
  [7]                          "Palsson, Master. <NAME>"
  [8]       "Johnson, Mrs. <NAME> (<NAME>)"
  [9]                     "Nasser, Mrs. Nicholas (<NAME>)"
 [10]                         "Sandstrom, Miss. <NAME>"
 [11]                                "Bonnell, <NAME>"
 [12]                          "Saundercock, Mr. <NAME>"
 [13]                             "Andersson, Mr. <NAME>"
 [14]                    "Vestrom, Miss.[...]<jupyter_text>We notice that the names contains title information of the individual. This is valuable, but how can we benefit from that? First of all see that the format of that string is clear: space + title + dot + space. We can try to model a regular expression or we can take a simpler, but manual path. Intuition tells us that there should not be too many keys.We build a set with known keys. After that we filter out names with known titles, and print first twenty of them. We see that we have already `"Mrs"` and `"Mr"`. Let's find others.<jupyter_code>// build incrementally a set with known keys
HashSet<String> keys = new HashSet<>();
keys.add("Mrs");
keys.add("Mr");

// filter out names with known keys
// print first twenty to inspect and see other keys
train.rvar("Name").stream().mapToString().filter(txt -> {
    for(String key : keys)
        if(txt.contains(" " + key + ". "))
            return false;
    return true;
}).limit(20).forEach(WS::println);<jupyter_output>"Heikkinen, Miss. Laina"
"Palsson, Master. <NAME>"
"Sandstrom, Miss. <NAME>"
"Bonnell, Miss. Elizabeth"
"Vestrom, Miss. <NAME>"
"Rice, Master. Eugene"
"McGowan, Miss. Anna ""Annie"""
"Palsson, Miss. <NAME>"
"O'Dwyer, Miss. Ellen ""Nellie"""
"Uruchurtu, Don. <NAME>"
"Glynn, Miss. <NAME>"
"<NAME>, Miss. <NAME>"
"Nicola-Yarred, Miss. Jamila"
"Laroche, Miss. <NAME>"
"Devaney, Miss. <NAME>"
"O'Driscoll, Miss. Bridget"
"Panula, Master. <NAME>"
"Rugg, Miss. Emily"
"West, Miss. <NAME>"
"Goodwin, Master. <NAME>"<jupyter_text>We reduced our search and found other titles like `"Miss"`, `"Master"`. We arrive at the following set of keys:<jupyter_code>HashSet<String> keys = new HashSet<>();
keys.addAll(Arrays.asList("Mrs", "Mme", "Lady", "Countess", "Mr", "Sir","Don", "Ms", "Miss", 
"Mlle", "Master", "Dr","Col", "Major", "Jonkheer", "Capt", "Rev"));

VarNominal title = train.rvar("Name").stream().mapToString().map(txt -> {
    for(String key : keys)
        if(txt.contains(" " + key + ". "))
            return key;
    return "?";
}).collect(VarNominal.collector());
DVector.fromCount(true, title).printSummary();<jupyter_output>?  Mr Mrs Miss Master Don Rev Dr Mme Ms Major Lady Sir Mlle Col Capt Countess Jonkheer
 -  -- --- ---- ------ --- --- -- --- -- ----- ---- --- ---- --- ---- -------- --------
 0 517 125  182     40   1   6  7   1  1     2    1   1    2   2    1        1        1<jupyter_text>We note that we exhausted training data. This is enough. It is possible that in test data to appear new titles. We will consider them missing values. That is why we return `"?"` when no matching is found. Another thing to notice is that some of the labels have few number of appearances. We will merge them in a greater category.Another useful feature built in `rapaio` is filters. There are two types of filters: variable filters and frame filters. The nice part of frame filters is that learning algorithms are able to use frame filters naturally, in order to make feature transformations on data. This kind of filters are called *input filters* from the learning algorithm perspective. It is important that you know that input filters transforms features before train phase and also on fit phase.We will build a learning filter to create a new feature.<jupyter_code>class TitleFilter implements FFilter {
    private HashMap<String, String[]> replaceMap = new HashMap<>();
    private Function<String, String> titleFun = txt -> {
        for (Map.Entry<String, String[]> e : replaceMap.entrySet()) {
            for (int i = 0; i < e.getValue().length; i++) {
                if (txt.contains(" " + e.getValue()[i] + ". "))
                    return e.getKey();
                }
            }
        return "?";
    };

    @Override
    public void fit(Frame df) {
        replaceMap.put("Mrs", new String[]{"Mrs", "Mme", "Lady", "Countess"});
        replaceMap.put("Mr", new String[]{"Mr", "Sir", "Don", "Ms"});
        replaceMap.put("Miss", new String[]{"Miss", "Mlle"});
        replaceMap.put("Master", new String[]{"Master"});
        replaceMap.put("Dr", new String[]{"Dr"});
        replaceMap.put("Military", new String[]{"Col", "Major", "Jonkheer", "Capt"});
        replaceMap.put("Rev", new String[]{"Rev"});
    }

    @Override
    public Frame apply(Frame df) {
        VarNominal title = VarNominal.empty(0, new ArrayList<>(replaceMap.keySet())).withName("Title");
        df.rvar("Name").stream().mapToString().forEach(name -> title.addLabel(titleFun.apply(name)));
        return df.bindVars(title);
    }
    
    @Override
    public TitleFilter newInstance() {
        return new TitleFilter();
    }
    
    public String[] varNames() {
        return new String[0];
    }
}<jupyter_output><empty_output><jupyter_text>Now let's try a new random forest on the reduced data set and also on title.<jupyter_code>RandomSource.setSeed(124);

Frame df_train = train.fapply(new TitleFilter());
Frame df_test = test.fapply(new TitleFilter());

CForest rf = CForest.newRF()
.withClassifier(CTree.newCART().withMinCount(3)).withRuns(100);

CEvaluation.cv(df_train.mapVars("Survived,Sex,Pclass,Embarked,Age,Fare,SibSp,Parch"), "Survived", rf, 10);

rf.fit(df_train.mapVars("Survived,Sex,Pclass,Embarked,Title"), "Survived");
rf.printSummary();

CPrediction fit = rf.predict(df_test);

new Confusion(train.rvar("Survived"), rf.predict(df_train).firstClasses()).printSummary();<jupyter_output>CrossValidation with 10 folds
CV  1:  acc=0.822222, mean=0.822222, se=NaN
CV  2:  acc=0.741573, mean=0.781898, se=0.057028
CV  3:  acc=0.853933, mean=0.805909, se=0.057929
CV  4:  acc=0.808989, mean=0.806679, se=0.047324
CV  5:  acc=0.764045, mean=0.798152, se=0.045202
CV  6:  acc=0.808989, mean=0.799958, se=0.040671
CV  7:  acc=0.831461, mean=0.804459, se=0.038990
CV  8:  acc=0.853933, mean=0.810643, se=0.040112
CV  9:  acc=0.876404, mean=0.817950, se=0.043455
CV 10:  acc=0.786517, mean=0.814806, se=0.042159
==============
Mean accuracy:0.814806
SE: 0.042159     (Standard error)
CForest model
================

Description:
CForest{runs:100;
baggingMode:DISTRIBUTION;
oob:false;
sampler:Bootstrap(p=1);
tree:CTree {varSelector=VarSelector[ALL];
minCount=3;
maxDepth=-1;
tests=NOMINAL:NominalBinary,DOUBLE:NumericBinary,BINARY:BinaryBinary,INT:NumericBinary;
func=GiniGain;
split=ToAllWeighted;
}}

Capabilities:
types inputs/targets: BINARY,INT,NOMINAL,DOUBLE/NOMINAL
counts inputs/targets: [...]<jupyter_text>Now that looks slightly better than our best classifier. We submit that to kaggle to see if there is any improvement.![Progress after incorporating title name into input features](images/titanic-rf1-model.png) 5.2 Other featuresThere are various authors which published their work on solving this kaggle competition. Most interesting part of their work is the feature engineering section. I developed here some ideas in order to show how one can do this with the library. 5.2.1 Family sizeUsing directly `"SibSp"` and `"Parch"` fields yields no value for a random forest classifier. Studying this two features it looks like those values can be both combined into a single one by summation. This would give us a family size estimator.In order to have an idea of the performance of this new estimator I used a ChiSquare independence test. The idea is to study if those features taken separately worth less than combined.<jupyter_code>// convert sibsp and parch to nominal types to be able to use a chi-square test
Var sibsp = VarNominal.from(train.rowCount(), row -> train.getLabel(row, "SibSp"));
Var parch = VarNominal.from(train.rowCount(), row -> train.getLabel(row, "Parch"));

// test individually each feature
ChiSqIndependence.from(train.rvar("Survived"), sibsp, true).printSummary();
ChiSqIndependence.from(train.rvar("Survived"), parch, true).printSummary();

// build a combined feature by summation, as nominal
VarNominal familySize = VarNominal.from(train.rowCount(),
row -> "" + (1 + train.getInt(row, "SibSp") + train.getInt(row, "Parch")));

// run the chi-square test on sumation
ChiSqIndependence.from(train.rvar("Survived"), familySize, true).printSummary();<jupyter_output>> ChiSqIndependence

Pearson's Chi-squared test with Yates' continuity correction

X-squared = 31.9279202, df = 6, p-value = 0.000017

Observed data:
       1.0 0.0 3.0 4.0 2.0 5.0 8.0 total
     0  97 398  12  15  15   5   7   549
     1 112 210   4   3  13   0   0   342
 total 209 608  16  18  28   5   7   891

Expected data:
               1.0         0.0       3.0        4.0        2.0       5.0       8.0 total
     0 128.7777778 374.6262626 9.8585859 11.0909091 17.2525253 3.0808081 4.3131313   549
     1  80.2222222 233.3737374 6.1414141  6.9090909 10.7474747 1.9191919 2.6868687   342
 total         209         608        16         18         28         5         7   891

> ChiSqIndependence

Pearson's Chi-squared test with Yates' continuity correction

X-squared = 23.3892671, df = 6, p-value = 0.000676

Observed data:
       0.0 1.0 2.0 5.0 3.0 4.0 6.0 total
     0 445  53  40   4   2   4   1   549
     1 233  65  40   1   3   0   0   342
 total 678 118  80   5   5   4   1   891[...]<jupyter_text>How we can interpret the result? The test says that each feature brings value separately. The *p-value* associated with both test could be considered significant. That means there are string evidence that those features are not independent of target class. As a conclusion, those features are useful. The last test is made for their summation. It looks like he test is more significant than the previous two. As a consequence we can use the summation instead of those two values taken independently. 5.3 Cabin and TicketIt seems that cabin and ticket denominations are not useful as they are. There are some various reasons why not to do so. First of all they have many missing values. But a stringer reason is that both have too many levels to contain solid generalization base for learning.If we take only the first letter from each of those two fields, more generalization can happen. This is probably due to the fact that perhaps there is some localization information encoded in those. Perhaps information about the deck, the comfort level, auxiliary functions is encoded in there. As a conclusion it worth a try so we proceed with this thing.To combine all those things we can do it in a single filter or into many filters applied on data. I chose to do a single custom filter to solve all those problems.<jupyter_code>class CustomFilter implements FFilter {

    private HashMap<String, String[]> replaceMap = new HashMap<>();
    private Function<String, String> titleFun = txt -> {
        for (Map.Entry<String, String[]> e : replaceMap.entrySet()) {
            for (int i = 0; i < e.getValue().length; i++) {
                if (txt.contains(" " + e.getValue()[i] + ". "))
                    return e.getKey();
            }
        }
        return "?";
    };

    public void fit(Frame df) {
        replaceMap.put("Mrs", new String[]{"Mrs", "Mme", "Lady", "Countess"});
        replaceMap.put("Mr", new String[]{"Mr", "Sir", "Don", "Ms"});
        replaceMap.put("Miss", new String[]{"Miss", "Mlle"});
        replaceMap.put("Master", new String[]{"Master"});
        replaceMap.put("Dr", new String[]{"Dr"});
        replaceMap.put("Military", new String[]{"Col", "Major", "Jonkheer", "Capt"});
        replaceMap.put("Rev", new String[]{"Rev"});
    }

    public Frame apply(Frame df) {
        VarNominal title = VarNominal.empty(0, new ArrayList<>(replaceMap.keySet())).withName("Title");
        df.rvar("Name").stream().mapToString().forEach(name -> title.addLabel(titleFun.apply(name)));

        Var famSize = VarDouble.from(df.rowCount(), row ->
        1.0 + df.getInt(row, "SibSp") + df.getInt(row, "Parch")).withName("FamilySize");

        Var ticket = VarNominal.from(df.rowCount(), row ->
        df.isMissing(row, "Ticket") ? "?" : df.getLabel(row, "Ticket")
        .substring(0, 1).toUpperCase()
        ).withName("Ticket");

        Var cabin = VarNominal.from(df.rowCount(), row ->
        df.isMissing(row, "Cabin") ? "?" : (df.getLabel(row, "Cabin")
        .substring(0, 1).toUpperCase())
        ).withName("Cabin");

        return df.removeVars("Ticket,Cabin").bindVars(famSize, ticket, cabin, title).solidCopy();
    }

    public CustomFilter newInstance() { return new CustomFilter(); }
    
    public String[] varNames() { return new String[0]; }
}<jupyter_output><empty_output><jupyter_text>5.4 Another try with random forestsSo we have some new features and we look to learn from them. We can use a previous classifier like random forests to test it before submit.But we know that we are in danger to overfit is rf. One idea is to transform numeric features into nominal ones by a process named discretization. For this purpose we use a filter from the library called `FQuantileDiscrete`. This filter computes a given number of quantile intervals and put labels according with those intervals on numerical values. Let's see how we proceed and how the data looks like:<jupyter_code>FFilter[] inputFilters = new FFilter[]{
new CustomFilter(),
FQuantileDiscrete.split(VRange.of("Age"), 10),
FQuantileDiscrete.split(VRange.of("Fare"), 10),
FQuantileDiscrete.split(VRange.of("SibSp"), 3),
FQuantileDiscrete.split(VRange.of("Parch"), 3),
FQuantileDiscrete.split(VRange.of("FamilySize"), 8),
FRemoveVars.remove(VRange.of("PassengerId,Name"))
};

// print a summary of the transformed data
train.fapply(inputFilters).printSummary();<jupyter_output>Frame Summary
=============
* rowCount: 891
* complete: 183/891
* varCount: 12
* varNames: 

 0. Survived : nominal | 4.    SibSp : nominal |  8. FamilySize : nominal |
 1.   Pclass : nominal | 5.    Parch : nominal |  9.     Ticket : nominal |
 2.      Sex : nominal | 6.     Fare : nominal | 10.      Cabin : nominal |
 3.      Age : nominal | 7. Embarked : nominal | 11.      Title : nominal |

Survived   Pclass           Sex            Age         SibSp         Parch                 Fare    Embarked 
 0 : 549  3 : 491    male : 577  31.8~36 :  91  -Inf~0 : 608  -Inf~0 : 678     7.854~8.05 : 106     S : 644 
 1 : 342  1 : 216  female : 314    14~19 :  87   0~Inf : 283   0~Inf : 213      -Inf~7.55 :  92     C : 168 
          2 : 184                  41~50 :  78                                  27~39.688 :  91     Q :  77 
                                 -Inf~14 :  77                                  21.679~27 :  89  NA's :   2 
                                   22~25 :  70           [...]<jupyter_text>We can see that `"Age"` values are now intervals and still $177$ missing values.The values chosen for quantile numbers is more or less arbitrary. There is no *good* numbers in general, only for some specific purposes.As promised, we will give a try to another random forest to see if it can better generalize.<jupyter_code>RandomSource.setSeed(123);

CForest rf3 = CForest.newRF()
.withInputFilters(inputFilters)
//.withVarSelector(VarSelector.fixed(4))
//.withBaggingMode(BaggingMode.VOTING)
//.withClassifier()
.withRuns(200);

rf3.fit(train, "Survived");
CEvaluation.cv(train, "Survived", rf3, 10)<jupyter_output>CrossValidation with 10 folds
CV  1:  acc=0.822222, mean=0.822222, se=NaN
CV  2:  acc=0.865169, mean=0.843695, se=0.030368
CV  3:  acc=0.808989, mean=0.832127, se=0.029370
CV  4:  acc=0.764045, mean=0.815106, se=0.041640
CV  5:  acc=0.820225, mean=0.816130, se=0.036133
CV  6:  acc=0.764045, mean=0.807449, se=0.038686
CV  7:  acc=0.853933, mean=0.814090, se=0.039445
CV  8:  acc=0.831461, mean=0.816261, se=0.037031
CV  9:  acc=0.876404, mean=0.822944, se=0.040023
CV 10:  acc=0.797753, mean=0.820424, se=0.038566
==============
Mean accuracy:0.820424
SE: 0.038566     (Standard error)
{'fraction_non_alphanumeric': 0.1126640803528724, 'fraction_numerical': 0.07356822731218225, 'mean_word_length': 3.265034753702025, 'pattern_counts': {'":': 0, '<': 197, '<?xml version=': 0, '>': 221, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 56, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '58713', 'license': 'Apache-2.0', 'n_tokens_mistral': 20591, 'n_tokens_neox': 17764, 'n_words': 6587, 'path': "notebooks/Getting started with Kaggle's Titanic Competition .ipynb", 'repo_name': 'iromeo/rapaio', 'size': '57003'}