<jupyter_start><jupyter_text>Decision trees 1. [EDA - exploratory data analysis](section1)2. [Build a model](section2)    2.1 [Define](section2.1)        2.2 [Fit](section2.2)        2.3 [Predict](section2.3)        2.4 [Evalaute](section2.4)3. [Select informative attributes](section3)    3.1 [Petal vs. Sepal](section3.1) 4. [Overfitting and Underfitting](section4)<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sklearn as sk
from sklearn import tree
from sklearn.tree import DecisionTreeClassifier
url = 'https://raw.githubusercontent.com/nlihin/data-analytics/main/datasets/iris.csv'
iris_df = pd.read_csv(url)
iris_df.head()<jupyter_output><empty_output><jupyter_text>1. EDA<jupyter_code>pd.pivot_table(iris_df,index=['class'],values=['sepal_length','sepal_width','petal_length','petal_width'])
plt.figure(1, figsize=(5,5))
plt.title("Distribution of Species")
iris_df['class'].value_counts().plot.pie(autopct="%1.1f%%")
plt.show()
plt.figure(figsize=(15,10))
plt.subplot(2,2,1)
sns.boxplot(x = 'class', y = 'sepal_length', data = iris_df, palette="gist_ncar_r")
plt.subplot(2,2,2)
sns.boxplot(x = 'class', y = 'sepal_width', data = iris_df, palette="gist_ncar_r")
plt.subplot(2,2,3)
sns.boxplot(x = 'class', y = 'petal_length', data = iris_df, palette="gist_ncar_r")
plt.subplot(2,2,4)
sns.boxplot(x = 'class', y = 'petal_width', data = iris_df, palette="gist_ncar_r")
plt.show()
sns.pairplot(iris_df, hue='class')
plt.show()
plt.figure(figsize=(7,4)) 
sns.heatmap(iris_df.corr(),annot=True,cmap='Greens') #draws  heatmap with input as the correlation matrix calculted by(iris.corr())
plt.show()<jupyter_output><empty_output><jupyter_text>ObservationThe Sepal Width and Length are not correlated The Petal Width and Length are highly correlatedWe will use all the features for training the algorithm and check the accuracy.Then we will use 1 Petal Feature and 1 Sepal Feature to check the accuracy of the algorithm as we are using only 2 features that are not correlated. Thus we can have a variance in the dataset which may help in better accuracy. We will check it later.   2. Build a classification model  2.1 Define the model: we will use a decision tree based on the gini index (gini index is the default)<jupyter_code>model1 = sk.tree.DecisionTreeClassifier(criterion="gini")<jupyter_output><empty_output><jupyter_text>2.2 Fit the model on the data: Define what are the features (x) and what is the target (y)<jupyter_code>features = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width']
X = iris_df[features]
y = iris_df['class']<jupyter_output><empty_output><jupyter_text>Use the model we chose on the features we chose (X and y)<jupyter_code>model1.fit(X, y)<jupyter_output><empty_output><jupyter_text>now model1 is a tree, and it has tree features:<jupyter_code>type(model1)
model1.feature_importances_<jupyter_output><empty_output><jupyter_text>Let's see what we got:<jupyter_code>text_representation = tree.export_text(model1)
print(text_representation)<jupyter_output>|--- feature_2 <= 2.45
|   |--- class: Iris-setosa
|--- feature_2 >  2.45
|   |--- feature_3 <= 1.75
|   |   |--- feature_2 <= 4.95
|   |   |   |--- feature_3 <= 1.65
|   |   |   |   |--- class: Iris-versicolor
|   |   |   |--- feature_3 >  1.65
|   |   |   |   |--- class: Iris-virginica
|   |   |--- feature_2 >  4.95
|   |   |   |--- feature_3 <= 1.55
|   |   |   |   |--- class: Iris-virginica
|   |   |   |--- feature_3 >  1.55
|   |   |   |   |--- feature_2 <= 5.45
|   |   |   |   |   |--- class: Iris-versicolor
|   |   |   |   |--- feature_2 >  5.45
|   |   |   |   |   |--- class: Iris-virginica
|   |--- feature_3 >  1.75
|   |   |--- feature_2 <= 4.85
|   |   |   |--- feature_0 <= 5.95
|   |   |   |   |--- class: Iris-versicolor
|   |   |   |--- feature_0 >  5.95
|   |   |   |   |--- class: Iris-virginica
|   |   |--- feature_2 >  4.85
|   |   |   |--- class: Iris-virginica<jupyter_text>Vizualize:<jupyter_code>fig = plt.figure(figsize=(15,10))

tree.plot_tree(model1, 
               feature_names = features, 
               class_names = model1.classes_,
               filled=True, 
               rounded = True)  
plt.show()<jupyter_output><empty_output><jupyter_text>That's nice. But how well will it predict on new data?  2.3 Predict  Split the data to 70% train set and 30% test<jupyter_code>X_train, X_test, y_train, y_test = sk.model_selection.train_test_split(X, y, test_size=0.3, random_state=1) 
X_train.head(2)
X_test.head(2)
y_test.head(2)<jupyter_output><empty_output><jupyter_text>Use the training data to train the model (a new model - model2)<jupyter_code>model2 = sk.tree.DecisionTreeClassifier(criterion="gini")
model2.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Finally - predict:<jupyter_code>prediction2 = model2.predict(X_test)
prediction2<jupyter_output><empty_output><jupyter_text>present in a nicer format:<jupyter_code>df = pd.DataFrame(columns = ['class', 'predictions'])
df['class'] = y_test
df = df.reset_index(drop=True)
df['predictions'] = pd.Series(prediction2)
df.head()<jupyter_output><empty_output><jupyter_text>2.4 Evaluate There are various ways to measure accuracy: The most obvious - (number of correct)/(total)<jupyter_code>sk.metrics.accuracy_score(prediction2, y_test)<jupyter_output><empty_output><jupyter_text>A truth table<jupyter_code>sk.metrics.confusion_matrix(y_test, prediction2)
sk.metrics.plot_confusion_matrix(model2, X_test, y_test, cmap=plt.cm.Blues)<jupyter_output><empty_output><jupyter_text>Precision, Recall and F1 score - most common. But out of scope for this course<jupyter_code>print(sk.metrics.classification_report(y_test, prediction2, target_names=model2.classes_))<jupyter_output>precision    recall  f1-score   support

    Iris-setosa       1.00      1.00      1.00        14
Iris-versicolor       0.94      0.94      0.94        18
 Iris-virginica       0.92      0.92      0.92        13

       accuracy                           0.96        45
      macro avg       0.96      0.96      0.96        45
   weighted avg       0.96      0.96      0.96        45<jupyter_text>3. Feature importance Which attributes are the most important?<jupyter_code>model1.feature_importances_<jupyter_output><empty_output><jupyter_text>organize it in a nicer format:<jupyter_code>feature_importances = pd.DataFrame(model2.feature_importances_,
                                   index = iris_df.columns[:4],
                                    columns=['importance']).sort_values('importance',
                                                                        ascending=False)
feature_importances.head()<jupyter_output><empty_output><jupyter_text>3.1 Petal vs. Sepal  We see that petal is most important. What happens if we only use sepal?<jupyter_code>X_s = iris_df[['sepal_length','sepal_width']]
y_s = iris_df['class']
X_train_s, X_test_s, y_train_s, y_test_s = sk.model_selection.train_test_split(X_s, y_s, test_size=0.3, random_state=1) 
model_s = sk.tree.DecisionTreeClassifier(criterion="gini")
model_s.fit(X_train_s, y_train_s)
prediction_s = model_s.predict(X_test_s)
print('The accuracy of the Decision Tree is',sk.metrics.accuracy_score(prediction_s,y_test_s)) 
df2 = pd.DataFrame(columns = ['class', 'predictions'])
df2['class'] = y_test_s
df2 = df2.reset_index(drop=True)
df2['predictions'] = pd.Series(prediction_s) 
sk.metrics.plot_confusion_matrix(model_s, X_test_s, y_test_s, cmap=plt.cm.Blues)
#mixed_df = iris_df[['sepal_length','petal_width','class']]
X_m = iris_df[['sepal_length','petal_width']]
y_m = iris_df['class']
X_train_m, X_test_m, y_train_m, y_test_m = sk.model_selection.train_test_split(X_m, y_m, test_size=0.3, random_state=1) 
model_mix = sk.tree.DecisionTreeClassifier(criterion="gini")
model_mix.fit(X_train_m,y_train_m)
prediction_mix = model_mix.predict(X_test_m)
print('The accuracy of the Decision Tree is',sk.metrics.accuracy_score(prediction_mix, y_test_m))
sk.metrics.plot_confusion_matrix(model_mix, X_test_m, y_test_m, cmap=plt.cm.Blues)<jupyter_output><empty_output><jupyter_text>4. Overfitting and underfitting<jupyter_code>def get_acc(max_leaf_nodes, train_X, test_X, train_y, test_y):
    model = sk.tree.DecisionTreeClassifier(criterion="gini", max_leaf_nodes=max_leaf_nodes, random_state=0)
    model.fit(train_X, train_y)
    preds_val = model.predict(test_X)
    acc = sk.metrics.accuracy_score(preds_val, test_y)
    return(acc)
features = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width']
X = iris_df[features]
y = iris_df['class']
X_train, X_test, y_train, y_test = sk.model_selection.train_test_split(X, y, test_size=0.3, random_state=1) 
for max_leaf_nodes in [2, 4, 6, 8, 10]:
    my_acc = get_acc(max_leaf_nodes, X_train, X_test, y_train, y_test)
    print(max_leaf_nodes, my_acc)<jupyter_output>2 0.6
4 0.9555555555555556
6 0.9555555555555556
8 0.9555555555555556
10 0.9555555555555556
{'fraction_non_alphanumeric': 0.13210084033613445, 'fraction_numerical': 0.03675070028011204, 'mean_word_length': 4.27853341218214, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 82, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '457146', 'license': 'MIT', 'n_tokens_mistral': 3551, 'n_tokens_neox': 3356, 'n_words': 635, 'path': 'unit7.ipynb', 'repo_name': 'NirNagary/DataS', 'size': '8925'}