Explain like I'm five · SW Eng

What is Statistical Hypothesis Testing?

Statistical hypothesis testing, told as a story about a publishing house using numbers to tell whether a new cover really sells better.

old covernew coverI feel the new cover will sell better!the editor has a hunch — but no real certainty

The editor has a hunch — but no real certainty.Changing the cover on a hunch alone is risky.

the new cover sold more — but was it luck?a hunch can't tell a real gap from chancewe need numbers to separate chance from a real difference

The new cover may just have had one lucky day, not a real difference.We need numbers to separate chance from a real difference.

showing each group a different coverp-value: odds of a gap at least this big if no differencehypothesis testing = start by assuming 'no difference', then use data to check if it's realif the p-value falls below the threshold, drop the 'no difference' assumption

Hypothesis testing starts by assuming 'no difference', then checks with data.If the p-value falls below the threshold, you drop the 'no difference' assumption.

Set up the null hypothesisassume no difference, for now
Gather the datacompare sales between the two cover groups
Compute the p-valuemeasures the odds of a gap at least this big if there's no difference
Compare to the thresholdif the p-value is below it, the assumption is dropped
JUDGMENT TABLEnull hypothesis — no differencealternative — there is a differencesmall p-value → reject the nulllarge p-value → can't reject the nullTWO WAYS TO BE WRONGtype I — seeing a difference that isn't theretype II — missing a difference that is therehypothesis tests can be wrong too — you judge while accepting a risk of error

If the p-value is below the significance level, reject the null; otherwise you fail to reject it — which doesn't prove there's no difference.Hypothesis tests can be wrong too — you judge while accepting a risk of error.

p-value below the threshold: we call it a differencebut only one extra book per 1,000 readersnot worth what the new cover coststatistically significant ≠ a difference that actually matters

Statistically significant isn't the same as practically important.Even when the p-value points to a difference, you still have to ask if it's worth what the new cover cost.

In one breath

Statistical hypothesis testing starts by assuming there's no difference, then checks how badly the data clashes with that assumption to judge whether a real difference exists.

Hypothesis testing sets up a null hypothesis (no effect/difference) and an alternative (there is one), computes a p-value from sample data — the probability of seeing a result this extreme by chance if the null were true — and compares it to a significance level (often 0.05) to decide whether to reject the null. Wrongly rejecting a true null is a Type I error; wrongly failing to reject a false null is a Type II error.

When grown-ups say it

Null Hypothesis
The starting assumption of 'no difference'. You begin by assuming there's no difference, for now.
Alternative Hypothesis
The claim that there IS a difference. Accepted when the null gets rejected.
p-value
The chance of a result this extreme or more by luck, if there's really no difference. The smaller it is, the worse the data fits 'no difference'.
Significance Level
The judgment threshold (often 0.05). Reject the null when the p-value falls below it.
Type I Error
Wrongly saying there IS a difference. The risk of jumping to conclusions.
Type II Error
Wrongly saying there is NO difference. The risk of missing a real difference.
Performance Testing
A test that measures a system's limits with numbers. A different goal, same reliance on numbers. → the pretend readers who rush in early
Metamorphic Testing
Checking via relationships, even without a known answer. A different approach for when there is no known answer at all. → checking by relationship, with no answer key
Next story
Same Plot, Retouched Sentences →
Want another term as a picture book? Request →