ModelRefs / Decision Trees — Tutorial
Decision Trees — Tutorial
Interpretable tree-based models that make decisions through sequential splits. Covers How a tree makes decisions, Splitting criteria: Gini and entropy.
Overview
Interpretable tree-based models that make decisions through sequential splits
Level: Intermediate. Estimated reading time: 35 minutes.
How a tree makes decisions
A decision tree partitions the feature space with a series of binary questions: "Is feature X greater than threshold t?" At each internal node the algorithm picks the feature and threshold that best separates the classes. Leaf nodes hold predictions.
Training is greedy: at each node, evaluate all features and all possible thresholds; choose the split that maximally reduces impurity. This is CART (Classification and Regression Trees), the algorithm behind sklearn's DecisionTreeClassifier.
Splitting criteria: Gini and entropy
Gini impurity measures the probability of mislabelling a randomly chosen element: G = 1 − Σ pᵢ². Pure node (one class): G=0. Maximum disorder: G=0.5 for binary.
Information gain (entropy-based): H = −Σ pᵢ·log₂(pᵢ). A pure node has H=0. Split maximises the reduction in entropy.
In practice, Gini and entropy produce very similar trees. Gini is slightly faster (no log). Use criterion='gini' as the default.
Controlling tree depth to prevent overfitting
An unconstrained tree will grow until every leaf is pure — perfectly fitting training data but badly overfitting. Control via:
• max_depth: hard limit on depth. Try 3–10. • min_samples_split: minimum samples needed to split a node. • min_samples_leaf: minimum samples in any leaf. • ccp_alpha: cost-complexity pruning (post-training pruning).
Always compare train vs test accuracy. A 100% training accuracy with 70% test accuracy means your tree is memorising.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Decision Trees — Tutorial.