What does a p-value actually tell you, and what doesn't it?

Muriel Bristol, a phycologist, claimed she could tell whether the tea or the milk had been poured first into a cup. Her future husband, William Roach, suggested that the statistician Ronald Fisher give her eight cups, four of each kind, in random order. The question Fisher asked is the root of the p-value: if she were just guessing, how likely is it that she would pick all of them right? The chance of getting all eight cups right by guessing is only 1 in 70, the combinations of 8 taken 4 at a time. According to a colleague of Fisher, in the actual experiment she succeeded in identifying all eight cups correctly.

The assumption to be tested, that she had no ability to distinguish the teas, is the null hypothesis, and Fisher held that a null hypothesis is never proved or established, but is possibly disproved. The p-value is the probability of obtaining test results at least as extreme as the result actually observed, under the assumption that the null hypothesis is correct. Fisher was willing to reject the null hypothesis only if the lady got all 8 right, which has 1 chance out of 70, about 1.4%. Getting at least 3 of the 4 milk-first cups right would happen by chance about 24.3% of the time, too often to reject the null hypothesis.

That is also the limit of what a p-value says. In 2016 the American Statistical Association stated that p-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone, and do not measure the size of an effect or the importance of a result. And the famous 0.05 cutoff was only a suggestion: in 1925 Fisher proposed a probability of one in twenty as a convenient cutoff level to reject the null hypothesis, and later recommended that it be set according to specific circumstances.