17. Logistic regression

Summary

Logistic regression is, in essence, a binary classification model (at least in its basic form), widely used in the credit industry for many years, and it has the particular feature of being, like linear regression, interpretable for decision-making. It is an intrinsically linear model and the parameter estimates it produces are accessible, so individual conclusions can be drawn about the importance each variable carries in the final estimate of the dependent variable, and statistical tests can also be applied to those parameters.

Unlike linear regression, the output of the model is not a quantity in measurable units but rather a probability, which lies between 0 and 1. Different criteria can therefore be set to finally choose one value or the other, usually with the midpoint (0.5) as the cutoff. A common scenario is also one in which the aim is to classify correctly in favor of one category or the other (for example, in medicine, where it is preferable to tell a patient that they are ill rather than that they are not), and for that the model is tuned to produce results that lean in one direction or the other. The usual measures for this are precision, recall, the false and true positive rates, and several other measures available through the construction of the confusion matrix, the standard tool for evaluating not only logistic regression but classification models (or classifiers) in general.

With the rise of data science / machine learning, logistic regression has gradually given way to other models that are simpler to compute / interpret, such as decision trees or random forests, and that also need no assumptions about the distribution of the data (the model itself) to be used and to draw conclusions from them. The next chapter covers some of them.

Extension Beyond the confusion matrix

The confusion matrix remains the starting point for evaluating a classifier, but it summarizes performance at a single cutoff point, and a complete evaluation requires looking further.

Choosing the threshold

The midpoint (0.5) is a convention, not an optimum. The right threshold depends on the cost of each type of error and on the frequency of the class of interest: in fraud detection, letting a case through can cost much more than reviewing a legitimate one. The recommended approach is to choose the threshold that minimizes the expected cost on validation data (not training data); scikit-learn includes tools to tune it with cross-validation.

Threshold-independent metrics

  • The ROC curve and the area under it (AUC) summarize performance across all possible thresholds.
  • When the class of interest is rare, the precision-recall curve is more informative than the ROC.
  • Accuracy is misleading with imbalanced classes: if 1% of transactions are fraudulent, a model that never detects fraud is right 99% of the time.

Calibration

If the model returns a probability and that probability is used as such (to set a price, prioritize cases or compute an expected value), ranking well is not enough: it has to be calibrated. That is, among the cases to which it assigns around 0.2, approximately 20% should belong to the positive class. This is checked with a calibration curve and summarized with measures such as the Brier score or the log loss. Logistic regression tends to produce well-calibrated probabilities, because it is fitted by optimizing precisely that loss; many other models need to be checked and, if necessary, recalibrated (for example, with Platt scaling or isotonic regression).

A clarification on the last paragraph

What the summary says about trees and random forests deserves a nuance: these models are more flexible (they do not require assuming a functional form), but a random forest is not simpler to interpret than a logistic regression; the opposite is true. Logistic regression remains standard in fields such as credit risk precisely because of its interpretability and its calibration, and it is a mandatory baseline against which to compare more complex models (see KNN and CART).

These same tools serve to evaluate a language model used as a classifier, a task developed in Verifying what the machine says.

Further reading

Extension