<jupyter_start><jupyter_text>Variable Ranking and Feature SelectionIn this document, we will be ranking and analyzing features to see which ones are important in predicting* null* goal* corner* freekick* penaltyusing a **single decision tree with very limited maximum depth**. We will draw the confusion matrices, feature importances and the decision tree itself to see which feature it uses at each level. This is going to be very useful in determining if the decision trees are learning the events we want them to learn or simply overfitting the data.<jupyter_code>%matplotlib inline
import pickle
import matplotlib.pyplot as plt

import numpy as np
import scipy as sc
import scipy.linalg as la

import pandas as pd
pd.set_option('display.max_columns', None)
pd.set_option('compute.use_bottleneck', True)
pd.set_option('compute.use_numexpr', True)

from sklearn import tree
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.utils import shuffle

from collections import Counter

with open('../data/event_names.pkl', 'rb') as f:
    event_names = pickle.load(f)
   
print('Event ids and names')
event_names<jupyter_output>Event ids and names<jupyter_text>Training Set<jupyter_code>data = pd.read_csv('../data/train/all_train.csv')
data = data.dropna().reset_index(drop=True)
X = data.iloc[:, 1:].values
display(data.tail())

print('ID\tName')
print('--\t----')
for id in np.unique(data['eventId']):
    print('{}\t{}'.format(id, event_names[id]))<jupyter_output><empty_output><jupyter_text>Decision Tree Variable RankingIn this section we build decision trees to see which variables are the most important when splitting the dataset according to Gini index. We proceed by performing cross validation on the training dataset. In each fold, a decision tree is trained to maximum depth $d$ from the training part. Then, the test part is predicted using the trained decision tree. In each fold, the resulting confusion matrix, feature importances and decision tree itself are drawn.<jupyter_code>from utils import plot_hbar_nameval, plot_confusion_matrix
from sklearn.model_selection import KFold
from sklearn.metrics import confusion_matrix
from IPython.display import Image, display

import pydotplus

def decision_tree_cv(X, y, depth=3, k=10, feature_names=None, class_names=None):
    label_ids = sorted(np.unique(y), key=abs)
    
    clf = tree.DecisionTreeClassifier(
        class_weight='balanced',
        max_depth=depth,
    )
    
    skf = KFold(n_splits=k, shuffle=True)
    skf.get_n_splits(X, y)
    
    cm_list = []
    importance_list = []
    tree_list = []
    for train_index, test_index in skf.split(X, y):
        X_train, X_test = X[train_index], X[test_index]
        y_train, y_test = y[train_index], y[test_index]
 
        clf = clf.fit(X_train, y_train)
        y_pred = clf.predict(X_test)
        
        conf_mat = confusion_matrix(y_test, y_pred, labels=label_ids)
        fig, ax = plot_confusion_matrix(conf_mat, class_names, 'Prediction', 'Truth', normalize=True)
        cm_list.append(fig)
        
        fig, ax = plot_hbar_nameval(feature_names, clf.feature_importances_, 'Feature importances', max_bars=10)
        importance_list.append(fig)
        
        dot_data = tree.export_graphviz(
            clf,
            out_file=None,
            feature_names=feature_names,
            class_names=class_names,
            filled=True,
        )
        graph = pydotplus.graph_from_dot_data(dot_data)
        fig = Image(graph.create_png())
        tree_list.append(fig)
        
    return cm_list, importance_list, tree_list<jupyter_output><empty_output><jupyter_text>Decision Tree Results and DrawingYou can change the depth parameter to change the decision tree depth. Lowering the depth decreases classifier performance as expected. In this example, we use depth 3 to see top 3 most important features drawn as a pydotplus graph. Increasing the depth to 5 actually enables the classifier to distinguish the events somewhat properly.We can also see that depth 3 decision tree is not able to distinguish the last 3 events from each other. This makes sense since a depth 3 decision tree makes only 3 decisions per path from root to a leaf. If we assume each decision "decides a class", then depth-3 decision tree would find 3 classes. In our case, it does this by dividing the dataset into* null* corner* freekick + goal + penaltyclasses.<jupyter_code>y = data.values[:, 0]
features = list(data.columns[1:])

events = [0, 60, 62, 80, 93]

mask = np.in1d(y, events)

cm_list_all, importance_list_all, tree_list_all = decision_tree_cv(
    X[mask],
    y[mask],
    depth=3,
    k=10,
    feature_names=features,
    class_names=[event_names[eid] for eid in sorted(events)]
)
index = 0
display(cm_list_all[index])
display(importance_list_all[index])
display(tree_list_all[index])<jupyter_output><empty_output><jupyter_text>You can double click the tree above to zoom in.  Predicting Individual EventsIn most cases, decision tree perform poorly in predicting the specified events. This may be due to1. **Very high class imbalance:** Since we collapse all other events apart from the one we are interested into a single class, we obtain a highly imbalanced training set. The algorithms obtain very high accuracy due to simply choosing the majority class, while in fact performing very poorly on the interesting cases.2. **Noisy labelling:** Since we have used a fixed amount of time intervals while labelling samples instead of manually checking to which event class a given frame belongs to, classifier may have difficulties in learning the representation of a given class.3. **Maximum depth:** The trees are grown to depth 3 only.  PenaltyPenalty prediction is problematic due to very few samples.Penalty class has few samples, but it is a very distinctive class. We might try the following to increase classifier performance for penalty events:1. Increase number of penalty samples by getting more frames for penalty events. (Increasing event interval in training set construction).  * How healthy is this?2. Use features that can distinguish penalty events. Examples are  * A way of measuring degree of local linearity. Defending players form a linear shape    just before a penalty kick.  * A way of measuring degree of tight clustering.3. Use oversampling methods to increase penalty samples.  * SMOTE may be useful.  Predicting Null Event vs PenaltyIn this section, we run the decision tree to do binary classification between the null event and penalty. Even a depth 2 decision tree achieves great performance while yielding interpretable results:* The fact that homeClusterDensity and refX are the most important features is in line with our football common sense. This also increases our confidence that our traning set is good.<jupyter_code>y = data['eventId'].values
mask = (y == 93) | (y == 0)
events = [0, 93]

cm_list_pen, importance_list_pen, tree_list_pen = decision_tree_cv(
    X[mask],
    y[mask],
    depth=2,
    k=10,
    feature_names=features,
    class_names=[event_names[eid] for eid in sorted(events)]
)
index = 8
display(cm_list_pen[index])
display(importance_list_pen[index])
display(tree_list_pen[index])<jupyter_output><empty_output><jupyter_text>Predicting Corner vs NullIn this section, we perform binary classification between corner and null event. We note that the the learned features in this case are different. Importance of inner distance and maximum X features make sense since during a corner, most of the players have either very high or very low x coordinates and the inner distance is generally very low.<jupyter_code>y = data['eventId'].values
mask = (y == 0) | (y == 60)
events = [0, 60]

cm_list_corner, importance_list_corner, tree_list_corner = decision_tree_cv(
    X[mask],
    y[mask],
    depth=2,
    k=10,
    feature_names=features,
    class_names=[event_names[eid] for eid in sorted(events)]
)
index = 5
display(cm_list_corner[index])
display(importance_list_corner[index])
display(tree_list_corner[index])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07311407262501528, 'fraction_numerical': 0.007213595794106859, 'mean_word_length': 4.572207084468665, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '559416', 'license': 'Apache-2.0', 'n_tokens_mistral': 2374, 'n_tokens_neox': 2288, 'n_words': 917, 'path': 'notebooks/feature_selection.ipynb', 'repo_name': 'eozd/SIU-2018', 'size': '8179'}