Summary
These models, applicable to both regression and classification, are excellent approximations to the data, without necessarily requiring prior knowledge of how the data behave or making assumptions about the underlying model. They are useful both in the data exploration stage and for building baseline models (because in practice they do not perform well enough to be deployed in production; they therefore serve as a point of comparison for building models that outperform them).
Like linear and logistic regression, their results are interpretable, although they do not have estimated parameters that can be analyzed afterward. Decision trees also have the advantage of being able to ingest any type of data, since in essence they are "yes or no" rules based on the discriminating power of the available data. Because they depend exclusively on the data, they are very sensitive to its quality, and also to the randomness of the starting point set by the analyst. Care must therefore be taken not to draw hasty conclusions from these models alone.
Later it will be shown that these are building blocks from which more complex models can be built (widely used in industry, such as random forests), with the same characteristics but overcoming the disadvantages mentioned.
Extension From the baseline to the model that really competes
The "later" that this chapter announces was never developed in the first edition, and it is worth completing, because tree ensembles, in particular gradient boosting, are today the practical reference for tabular data (rows and columns, like most of the data in this book).
A ladder of baseline models
A new model must beat every step of a ladder of simpler alternatives, evaluated with the same cross-validation split and the same metric:
- A trivial predictor: the mean (regression) or the majority class (classification).
- A linear or logistic model (see Linear regression and Logistic regression).
- KNN and a decision tree, like those in this chapter.
- A tree ensemble: random forest or gradient boosting.
If a complex model barely improves on a linear one, the additional complexity (of maintenance, of explanation) is rarely justified.
Ensembles: bagging and boosting
A random forest combines many deep trees, each trained on a bootstrap sample of the data (see Nonparametric statistics) and with a random subset of variables at each split; averaging their predictions reduces the variability that makes a single tree unstable. Gradient boosting, by contrast, builds shallow trees sequentially, each focused on correcting the errors of the ensemble before it. Its widespread implementations (XGBoost, LightGBM, CatBoost and the histogram-based one in scikit-learn) accept missing values and, in several cases, categorical variables without prior encoding. The main hyperparameters are the learning rate, the depth of the trees and the number of trees, which are best set with early stopping on validation data.
The evidence in its favor is solid: a 2022 comparative study (Grinsztajn, Oyallon and Varoquaux) of 45 datasets found that tree-based models still outperformed neural networks on medium-sized tabular data (on the order of 10,000 observations), even without counting their greater speed.
The cost: interpretability
An ensemble of hundreds of trees can no longer be read as a set of rules. To understand what it learned, methods that explain the model from the outside are used, such as permutation importance or SHAP values. It is worth remembering that they explain the model, not the phenomenon: a variable that is important for prediction is not necessarily a cause.
With AI assistants, putting together a boosting pipeline takes minutes (see From copilot to colleague). The risk shifts to evaluation: data leakage from the test set into the training set, poorly designed validation, unfair comparisons. The baseline ladder and a fixed validation split are the main defense (see Verifying what the machine says).
Further reading
General concepts
- Model-Agnostic Methods. Interpretable Machine Learning, Christoph Molnar, "Methods Overview".
- Example-Based Explanations. Interpretable Machine Learning, Christoph Molnar, "Methods Overview".
KNN
- K-Nearest Neighbors. An Introduction to Statistical Learning (2nd ed.), James, Witten, Hastie and Tibshirani, pp. 39–42.
- Nearest Neighbors. scikit-learn User Guide.
- The k-Nearest Neighbors (kNN) Algorithm in Python. Real Python.
- Comparison of Linear Regression with K-Nearest Neighbors. An Introduction to Statistical Learning (2nd ed.), James, Witten, Hastie and Tibshirani, pp. 105–110.
CART
- Decision Tree. Interpretable Machine Learning, Christoph Molnar, ch. 9.
- Tree-Based Methods. An Introduction to Statistical Learning (2nd ed.), James, Witten, Hastie and Tibshirani, pp. 327–339.
- Decision Trees. scikit-learn User Guide.
Extension
- Tree-Based Methods. An Introduction to Statistical Learning (2nd ed.), James, Witten, Hastie and Tibshirani, ch. 8.2: bagging, random forests and boosting.
- Ensembles: Gradient boosting, random forests, bagging, voting, stacking. scikit-learn User Guide.
- XGBoost: A Scalable Tree Boosting System. Chen and Guestrin, 2016.
- Why do tree-based models still outperform deep learning on tabular data?. Grinsztajn, Oyallon and Varoquaux, 2022.
- Permutation Feature Importance and SHAP. Interpretable Machine Learning, Christoph Molnar.