The Logic of NHST

Lecture · Module 8 · Mon Oct 5

How today works

You’ve already met the machinery. The Module walked the five steps; Friday’s pre-study had you build a null distribution by hand, resample by resample. None of that gets repeated today.

Instead we’ll do something you cannot do alone: use the whole room as a sampling distribution. Twelve people running twenty studies each is 240 studies — and 240 studies is enough to watch a hypothesis test misbehave in exactly the way the theory says it will.

Today’s class has five parts:

  1. The room cries wolf (18 min) — we run studies where we know there’s nothing there, and count how many find something
  2. The room misses something real (20 min) — same idea flipped: an effect that genuinely exists, and who fails to see it
  3. There is only one test (15 min) — why the shelf of tests in M09 is really one idea
  4. α is a decision, and it’s yours (17 min) — what an error costs in your research
  5. Misconception gauntlet (10 min) — four sentences, class vote

Pair-and-share

Sections marked like this are two-minute conversations with the person next to you. Predict before you run — being wrong out loud is the fastest way to remember something.

Recap · where we are (5 min)

Three things from the Module and pre-study, then we move:

  • A null distribution is a “what if” world — what your statistic would look like if \(H_0\) were true. You built one by hand on Friday.
  • A p-value is the proportion of that world at least as extreme as what you actually saw. It is \(P(\text{data} \mid H_0)\) — never the reverse.
  • A decision rule compares that p-value to \(\alpha\), a threshold you choose in advance.

That’s the theory, and it’s clean. Today we find out what it feels like when it goes wrong.

Part 1 · The room cries wolf (18 min)

Here is a world with nothing in it. Every “study” below draws 30 observations from a population whose mean is exactly zero, and tests whether the mean is zero.

The null hypothesis is not approximately true here. It is true by construction. We built the world; there is no effect to find.

So: how many of us will find one?

Pair-and-share · predict first (2 min)

You’re about to run 20 studies on data with no effect, at \(\alpha = .05\).

  • How many of your 20 do you expect to come back significant?
  • Across the room — 12 people, 240 studies — how many should we see in total?
  • Would you be surprised if someone here got three?

Say a number out loud before you click Run.

Set your seed to something personal — your birthday as MMDD, the last four digits of your phone number, anything. Then run it.

Collect the room (6 min)

Go around: how many of your 20 were significant? I’ll put them on the board and total them.

Most of you will say 0 or 1. Someone will say 2. Occasionally somebody says 3 — and that person, in a real career, would be writing up a finding right now.

With 12 people × 20 studies = 240 studies, we expect about 12 false positives, and an observed rate near 5%.

Then the question that matters:

Of all the studies this room just ran, which ones would get published?

The ~228 that found nothing are unpublishable. The ~12 that “found” something are exactly the ones a journal would take. Every one of them is wrong.

What just happened

Nobody made a mistake. Nothing was coded wrong. The test worked exactly as designed — α = .05 means “I accept being wrong 5% of the time when the null is true.”

That’s the deal you sign every time you run a test. The 5% isn’t a flaw to be fixed; it’s a price you agreed to pay. What makes it dangerous is that the literature only ever shows you the 5%.

That’s the file-drawer problem, and you just generated it in eight minutes.

Part 2 · The room misses something real (20 min)

Flip it. Now there is an effect. The question is who sees it.

Four groups (2 min)

Group Sample size True effect
A n = 12 d = 0.2 (small)
B n = 80 d = 0.2 (small)
C n = 12 d = 0.8 (large)
D n = 80 d = 0.8 (large)

Count off 1–2–3–4 around the room, then set my_n and my_d below to match your group.

Pair-and-share · predict (2 min)

Before anyone runs: rank the four groups from “detects it most often” to “least often,” and guess a percentage for your own group.

Hold onto your guess — we’ll check it in five minutes.

Fill in the grid (8 min)

Total each group’s detections and divide by (3 people × 20 studies = 60). Write the four percentages on the board:

d = 0.2 (small) d = 0.8 (large)
n = 12 ~10% ~71%
n = 80 ~42% ~100%

Three things to pull out of the grid:

  1. Group A ran 60 studies of a real effect and missed it roughly 54 times. Nothing was wrong with the effect. Nothing was wrong with the test. There just weren’t enough people.
  2. Group D never missed. Same test, same α — but a large effect and a decent sample make failure nearly impossible.
  3. Compare A to B (same effect, more people) and A to C (same people, bigger effect). Both raise detection — but only one of them is under your control as a researcher.

The picture your four numbers live on

Your grid has four points in it. Here is the curve they came from — power against sample size, for each of the two effect sizes, with your four cells marked.

Read the picture (3 min)

Find your group’s letter. Two things to notice:

A sits near the floor. At n = 12 with a small effect, the curve has barely left the ground — and note it doesn’t start at zero. Even with no effect worth finding, you’d “detect” something 5% of the time. That’s α again, showing up as the curve’s floor.

The gold curve never reaches the dashed line. Even at n = 100, a small effect stays under 80% power. To study a small effect properly you need a sample far bigger than most of us casually assume — which is the single most common reason a well-designed study fails.

The two ways to be wrong, both now felt

  • In Part 1 the null was true and we rejected it anyway — a Type I error, rate α, the price you set.
  • In Part 2 the null was false and we failed to reject it — a Type II error, rate β. Its complement, power = 1 − β, is what Group A didn’t have.

Notice the asymmetry in how you control them. You choose α outright. You buy power — with sample size, with better measurement, or by studying a bigger effect.

Group A is what underpowered research looks like from the inside: you run a real study of a real effect, get p = .31, and conclude there’s nothing there.

Part 3 · There is only one test (15 min)

In M09 you’ll meet a shelf of named tests: two-sample t, paired t, one-way ANOVA, chi-square, correlation. It looks like a lot of machinery to memorize.

It isn’t. Every one of them is the same three moves:

  1. Pick a number that summarizes your data — a statistic
  2. Build the distribution that statistic would have if \(H_0\) were true
  3. Ask how far out your actual value falls

The named tests differ only in step 2 — in which shortcut they use to skip the simulation.

Here are the three moves in code, on the college-mobility data you know:

Now watch what changes when the question changes completely.

Look at what didn’t change (5 min)

Put the two chunks side by side. specify → hypothesize → generate → calculate. The verbs are identical.

One tested a mean against $32,000. One tested a proportion against .5. Swap in two groups and you’d have a two-sample t-test; swap in two categorical variables and you’d have a chi-square.

So why does your stats textbook have twelve chapters?

Because the computers came later. Fisher and Gosset couldn’t run 1,000 simulations — they had pencils. So for each situation they worked out a mathematical approximation to the null distribution: the t-distribution, the F-distribution, χ². Those tables are the shortcuts, and the “different tests” are really different shortcuts to one idea.

You live after the computers. You can build the null world directly and see that it was one idea all along.

One honest note about tools

That four-verb pipeline is a teaching tool. It exists to make the logic visible, and nothing does that better.

It is not what you’ll write in your dissertation. On Wednesday you’ll use t_test() — one line, tidy output — because that’s what your field writes and what reviewers expect. aov(), chisq.test(), and lm() are the same story.

Those functions are shortcuts to what you just watched. Knowing that is the difference between running a test and understanding one.

Part 4 · α is a decision, and it’s yours (17 min)

We’ve been writing α = .05 all week as if it came down from a mountain. It didn’t. Fisher offered it as a convenience and later warned against using it rigidly.

α is a choice about which error you’d rather make. Lower it and false positives get rarer — but you lose power, pushing more studies into Group A. No setting avoids both.

So the right α depends on what each mistake costs. And that depends on what you study.

Two scenarios, same α? (6 min)

Scenario 1. A screening test for a treatable childhood illness. A false positive means an unnecessary follow-up appointment. A false negative means a missed diagnosis.

Scenario 2. A confirmatory trial for a drug with serious side effects, going to a regulator. A false positive means an ineffective drug reaches patients. A false negative means a useful drug is shelved.

Should both use α = .05? Argue it out. Which way would you move each one, and what would it cost you?

Pair-and-share · your own research (6 min)

Answer for your own work — thesis, lab, whatever you’re closest to:

  1. What’s one hypothesis you might actually test?
  2. If you got a false positive — you publish it and it isn’t real — what does that cost? Who’s affected?
  3. If you got a false negative — the effect is real and you miss it — what does that cost?
  4. Which would you rather risk? Does .05 sit in the right place for you?

Most people have never been asked question 4. There’s no correct answer — but there’s a big difference between choosing .05 and inheriting it.

Write the rule before you look

Here’s the habit that makes all of this honest, and it takes thirty seconds.

Before you see your data, write down your α, your sample size, and your decision rule. Then follow it.

Why it matters: once you’ve seen the data, every choice is contaminated by knowing what would help. Dropping two outliers, switching to one-tailed “because the direction was predicted,” collecting twenty more participants because p = .07 — each is defensible on its own, and together they can push a true α of .05 past .30.

You’ll do this formally in Project 2, where replication-plan.md asks you to commit to a success bar before you run anything. That file isn’t busywork; it’s what makes your result mean what you say it means.

Part 5 · Misconception gauntlet (10 min)

Four sentences, each one something a real researcher has written. Thumbs up if it’s fine, thumbs down if it’s wrong — then we argue.

Vote, then defend (10 min)

1. “We obtained p = .03, so there is a 3% probability that the null hypothesis is true.”

Wrong. This inverts the conditional. p is \(P(\text{data} \mid H_0)\) — the chance of data this extreme if the null holds. \(P(H_0 \mid \text{data})\) is a different quantity and requires a prior. The single most common error in the published literature.

2. “The effect did not reach significance (p = .07), indicating no difference between groups.”

Wrong. Failing to reject is not accepting. Ask Group A — they failed to reject sixty times on an effect that was unambiguously real. Absence of evidence is not evidence of absence.

3. “The intervention had a highly significant effect (p < .001), demonstrating its substantial impact.”

Wrong — the second half doesn’t follow from the first. A tiny p means the data are very incompatible with the null; it says nothing about magnitude. With a big enough sample a trivial difference gets p < .001. Magnitude is the effect size’s job, which is why you’ll report Cohen’s d alongside every test on Wednesday.

4. “Our result (p = .04) suggests a 96% chance of replicating in a future study.”

Wrong. Replication probability depends on the true effect size, the new study’s sample size, and its variability — none of which the original p-value contains. A p = .04 study with low power often replicates well under half the time.

Why these keep happening

All four try to make p answer a question it can’t: how likely is my hypothesis to be true?

That’s a reasonable thing to want — it’s just not what a frequentist p-value provides. As you saw in M07, a Bayesian analysis answers it directly. These aren’t word games; they change what a reader concludes, and they’re a documented contributor to why so many published findings fail to replicate.

Wrapping up (5 min)

What we built today

  • You generated the file-drawer problem yourselves. 240 studies of nothing, about a dozen “discoveries” — and only those dozen would ever be published.
  • You watched an underpowered study from the inside. Group A ran a real effect sixty times and missed it nine times in ten.
  • You saw that there’s one test, not twelve. Four verbs, any statistic; the named tests are pre-computer shortcuts to the same idea.
  • You put a price on being wrong in your own research — which is what choosing α actually means.

What’s next.

Wednesday’s lab turns this into what researchers actually produce. You’ll run one test on the Hofman boulder data with t_test(), size the effect, build a figure a stranger could read, and write it up as an APA manuscript — title page, citation, reference list and all. It’s the first paper you’ll write in this course, and the format carries straight into your thesis.

Friday’s pre-study opens M09, which takes the four verbs from Part 3 and walks the whole shelf: two-sample t, paired t, ANOVA, chi-square. You already know the logic — what’s left is bookkeeping.