16. Linear regression

Summary

Linear regression is a model in very wide use, applied to the estimation of a continuous variable from continuous and/or qualitative (binary) variables. It first involves estimating certain parameters associated with each independent variable, which make up a mathematical expression from which, once those values are estimated, an approximation of the dependent variable is obtained.

For this approximation, it will be relevant to evaluate how close it is to the values actually observed and whether the residuals resulting from that evaluation show any particular behavior, which is a sign of how good the model is, as it seeks to be as complete a representation as possible of the true model underlying the data.

It should be noted that, like every model, it is a mathematical simplification of reality, so it will not necessarily reach the expected results. The estimates may be far from the observed values, possibly because other variables with an impact are being left out, or because the nature of the phenomenon itself is not linear in its parameters (an incorrect specification of the model), among several other reasons. Returning to what was said at the beginning, it is a widely used model, but that does not mean it is used correctly. The data may present a set of problems such as multicollinearity, heteroscedasticity or autocorrelation, which must be addressed properly; doing so saves time in concluding that the linear regression model will not, in fact, be adequate.

Very complete implementations exist in both Python and R, in multiple libraries. In data science / machine learning, scikit-learn (sklearn) is commonly used, and it is even considered a standard for the way it structures model building. This is precisely the library used from here on for the rest of the chapters.

Further reading

General concepts

  • Relación entre variables (in Spanish). Kelmansky, Estadística para todos, pp. 168–196.
  • El estudio de la relación entre variables. Aguirre, Niño and Simonetti, Estadística aplicada en las ciencias sociales y humanas (in Spanish), pp. 105–130.

Linear regression

  • Regresión lineal y correlación. Lind, Marchal and Wathen, Estadística aplicada a los negocios y la economía (in Spanish), 15th ed., pp. 461–494.
  • Análisis de regresión múltiple. Lind, Marchal and Wathen, Estadística aplicada a los negocios y la economía (in Spanish), 15th ed., pp. 513–542.
  • Análisis de regresión: el modelo lineal simple. Canavos, Probabilidad y estadística. Aplicaciones y métodos (in Spanish), pp. 443–478.
  • Fundamentals of Data Visualization. Claus O. Wilke, part I: ch. 12.
  • Linear Regression Foundations. STAT 500: Applied Statistics, Penn State, lesson 9.
  • Linear Regression. Interpretable Machine Learning, Christoph Molnar, ch. 6.
  • Regresión lineal (in Spanish). Laboratorio de Datos, first term 2021, UBA: class 6.
  • Regresión lineal y correlación. Introducción a la estadística (in Spanish), OpenStax, ch. 12.

Implementation in Python