<jupyter_start><jupyter_text>training and visualizing a decision tree<jupyter_code>from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
iris = load_iris()
iris['DESCR']
X = iris.data[:, 2:] # petal length and width
y = iris.target
tree_clf = DecisionTreeClassifier(max_depth=2)
tree_clf.fit(X, y)<jupyter_output><empty_output><jupyter_text>**visualize the trained decision tree**<jupyter_code>from sklearn.tree import export_graphviz
export_graphviz(
    tree_clf,
    out_file='iris_tree.dot', 
    feature_names=iris.feature_names[2:],
    rounded=True,
    filled=True
)<jupyter_output><empty_output><jupyter_text>making predictions In particular, they don't require feature scaling or centering at all. **gini impurity**$$1-\sum_{k=1}^{n} p_{i, k}^2$$ scikit-learn uses the CART algorithm whihc only produces only binary tree. fairly intuitive and easy to interpret  estimate class probabilities 1. traverses the tree to find the leaf node for this instance2. returns the ratio of training instances of class k in this node<jupyter_code>tree_clf.predict_proba([[5, 1.5]])
tree_clf.predict([[5, 1.5]])<jupyter_output><empty_output><jupyter_text>CART training algorithm CART (classification and regression tree) algorithm**How to choose feature k and threshold $t_{k}$** searches for the pair that produces the smallest value of cost function$$J(k, t_{k}) = \dfrac{m_{left}}{m} G_{left} + \dfrac{m_{right}}{m} G_{right}$$- $m_{left/right}$ is the number of instances in the left/right subset.- $G_{left/right}$ measures the impurity**greedy algrithm** **Entropy**$$H_{i} = -\sum_{k=1,p_{i, k}\neq 0}^n p_{i, k}log(p_{i, k})$$ entropy is zero when all messages are identical. most of time, the two ways dont' make a big difference. Gini impurity is slightly faster to compute.  regularization hyperparameters - parametric model - nonparametric model (likely to overfitting) **regularization**:restrict the model's freedom through hyperparameters post pruning:     Standard	statistical	tests,	such	as	the	χ2	test,	are	used	to	estimate	the	probability	that	the	improvement	is	purely	the	result	of chance	(which	is	called	the	null	hypothesis).	If	this	probability,	called	the	p-value,	is	higher	than	a	given	threshold	(typically	5%, controlled	by	a	hyperparameter),	then	the	node	is	considered	unnecessary	and	its	children	are	deleted.	The	pruning	continues	until all	unnecessary	nodes	have	been	pruned  regression<jupyter_code>from sklearn.tree import DecisionTreeRegressor
tree_reg = DecisionTreeRegressor(max_depth=2)
tree_reg.fit(X, y)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09765474817377931, 'fraction_numerical': 0.007304882737408689, 'mean_word_length': 6.268156424581005, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '541043', 'license': 'MIT', 'n_tokens_mistral': 884, 'n_tokens_neox': 841, 'n_words': 279, 'path': 'decision_tree/decision_tree.ipynb', 'repo_name': 'zi-NaN/hands-on-machine-learning', 'size': '2601'}