Summary
Starting from an estimator that has already been computed (that is, an approximation of some characteristic of a variable has been obtained), the aim is often to check the result against some assumption about the true value, or perhaps against a value taken as given. This putting of what is known to the test is called a hypothesis test. A hypothesis test, using the evidence (data) collected, seeks to reject (or not) what is held to be true so far. It is worth noting that this is precisely how science works in conceptual terms; hypothesis tests are therefore the tool par excellence of scientific research.
As with the computation of estimators, the result of a hypothesis test depends on the quality of the data obtained and on the representativeness of the sample. The data must be obtained at random, at least at some point in the overall procedure, and are also subject to whether or not the variable being tested is assumed to follow a given distribution. If it is not, then, just as with estimation, nonparametric methods are used.
In addition, statistical tests vary depending on whether there are one or more samples, one or more variables, independent or paired variables (for example, the same variable at two different points in time), known or unknown dispersion, among other things. It is a truly extensive battery that can hardly be memorized, so it is advisable instead to learn to recognize the scenarios in which a test can be applied, and then look into which one would be most appropriate. In that sense, the approach is similar to that of data visualization, but it calls for much more caution because, being a critical tool for drawing conclusions, it can lead to poorly grounded actions.
Extension Statistical significance and practical significance
Hypothesis tests are a central tool of scientific research, though not the only one, and their use in isolation has drawn growing criticism. In 2016, the American Statistical Association published a statement of principles on the use of the p-value, among them: that it does not measure the probability that the hypothesis is true; that it does not measure the size of an effect or the importance of a result; and that conclusions and decisions should not be based only on whether the p-value passes a threshold. Several practices now considered basic follow from this.
Report the effect and its interval
The main result of a comparison is the effect size: the estimated difference in the units of the problem (pesos, percentage points, minutes) or a standardized measure (such as Cohen's d), accompanied by its confidence interval (see Point and interval estimation). The p-value complements it; it does not replace it. This is especially important with today's volumes: with very large samples, differences that are negligible in practice turn out statistically significant; with small samples, relevant effects may not. That is why it pays to define, before looking at the data, the smallest difference that would have practical consequences.
Many tests, many false positives
At a 5% significance level, roughly one in every twenty comparisons with no real effect will come out significant by chance. When testing many metrics, segments or variants, finding "something significant" is almost certain. There are corrections for multiple comparisons (Bonferroni, Holm) and procedures that control the expected proportion of false discoveries (Benjamini-Hochberg). An equivalent problem arises from looking at the results repeatedly while data are being collected and stopping when it is convenient.
Plan before analyzing
- Set the hypothesis, the primary metric and the analysis before seeing the data (in research, through preregistration).
- Compute the power and the sample size needed to detect the smallest relevant difference.
- Report all the tests performed, not only the significant ones.
These precautions carry more weight with AI assistants, which can run dozens of tests in seconds and return only the ones that "worked". Knowing how many analyses were run, and why, remains the analyst's responsibility (see Verifying what the machine says). How all this applies to controlled experiments in digital products is taken up again in From insight to action.
Further reading
- Decisiones en el campo de la estadística (in Spanish). Kelmansky, Estadística para todos, pp. 231–241.
- Pruebas de hipótesis de una muestra. Lind, Marchal and Wathen, Estadística aplicada a los negocios y la economía (in Spanish), 15th ed., pp. 333–346.
- Pruebas de hipótesis estadísticas. Canavos, Probabilidad y estadística. Aplicaciones y métodos (in Spanish), pp. 303–346.
- Hypothesis Testing for One-Sample Proportion. STAT 500: Applied Statistics, Penn State, lesson 6, part A.
- Hypothesis Testing for One-Sample Mean. STAT 500: Applied Statistics, Penn State, lesson 6, part B.
- Pruebas de hipótesis con una muestra (in Spanish). Introducción a la estadística, OpenStax, ch. 9.
Extension
- The ASA's Statement on p-Values: Context, Process, and Purpose. Wasserstein and Lazar, The American Statistician, 70(2), 2016.
- Improving Your Statistical Inferences. Daniël Lakens. Open book on p-values, effect sizes, power and multiple comparisons.