Two Groups, One Logic

Lecture · Module 9 · Mon Oct 12

How today works

Two lectures ago the county asked where people start. Last week it asked whether one group changed. Today it asks the question last week’s pilot could not answer: did the workbook group change more than people would have changed anyway?

The tools are last week’s. What is new is a second group, and with it three questions that are easy to blur together. We will keep them apart all session:

Three questions to keep apart

  1. What happened in our trial? One trial gives one estimate: the difference between the two groups’ average changes.
  2. How unusual is our result in a no-effect world? We build the null world, where the workbook does nothing, by reshuffling the group labels. The share of shuffled differences at least as far from zero as ours is a p-value. Welch’s t then gives the model-based route: the difference in standard errors (\(t_\text{obs}\)), its interval, and a second p-value.
  3. Will the design work? Before any data exist, we imagine a real-change world and ask how often a trial this size would detect the effect. That share is power.

We start with a quick recap of last week’s test and end by giving your own study a comparison group.

Recap · last week’s test in five questions

Last Monday the county rehearsed a before-and-after pilot: 40 adults, two weeks of the workbook, and each person’s change in benevolence (after minus before). Rebuild its test from memory before you open the answers.

Retrieval warm-up · five questions

  1. What was the estimand, the population quantity the pilot was after?
  2. What was the estimate?
  3. What measured the estimate’s uncertainty (i.e., wobble)?
  4. What did the null world assert, and what did the test find?
  5. What did the 95% interval say, and does it agree with the test?
  1. The population’s true average change, \(\mu_D\).
  2. The pilot’s average change, \(\bar D\) = +0.54.
  3. Its standard error, \(SE_{\bar D} = s_D / \sqrt{n}\) = 0.158.
  4. \(H_0: \mu_D = 0\). The test gave \(t_\text{obs} = (\bar D - 0)/SE_{\bar D}\) = 3.41 and p = .002.
  5. 95% CI [0.22, 0.86]. Zero is outside it, so the interval and the test reach the same decision.

So the pilot gives evidence that the average change is not zero. It does not yet tell us how much of that change the workbook caused, because there was no group that went without it.

The pilot looked great — until the waitlist showed up

So the rehearsed pilot came back the way the team hoped: an average change of +0.54 points, 95% CI [0.22, 0.86], zero nowhere in it, p < .05.

Pair-and-share · so what did the workbook do?

In REACH, participants were randomly assigned to get the workbook or be on a waitlist. The workbook arm changed by +0.55 and the waitlist by +0.06. What is the workbook’s effect? Compute a number, and say what you did to get it.

Subtract. The waitlist’s +0.06 estimates the average change over those same two weeks without the workbook — it bundles whatever would have happened anyway: time, settling, answering the same questions a second time, and other influences. The workbook arm changed by +0.55, so the simple difference-in-change estimate is about 0.55 − 0.06 = 0.49 points, not 0.55. Because treatment was randomized, it is the between-arm comparison — not the workbook arm’s change by itself — that gives the trial its causal leverage.1

The sentence of the day

Change is not an intervention effect. An intervention effect is the change beyond what would have happened anyway.

Here the waitlist barely moved, so the subtraction barely mattered — this time, over two weeks. But nothing guaranteed that, and you only know +0.06 because the trial measured a group that waited. People often sign up when a hurt feels at its worst, and a moment that raw is usually followed by a calmer one, so scores can improve with no help from the workbook at all. Other things change over the same weeks too, and a second sitting with the same questionnaire is not quite the first. In a longer study or a different population, that natural change could be far larger than 0.06 — and a pilot with no comparison group can never tell the workbook from it. No statistic computed on one group’s change can split them apart. What splits them is a second group, measured over the same weeks with the same instrument, where chance decides who waits. Today the county plans that trial.

Question 1 · What happened in our trial?

The county’s answer is a randomized controlled trial (RCT): chance decides who gets the workbook now and who waits, and both groups are measured before and after, over the same two weeks. The outcome is still each person’s change. What the trial estimates is new: the difference between the two groups’ average changes, workbook minus waitlist,

\[\bar D_\text{workbook} - \bar D_\text{waitlist},\]

which estimates the population difference δ: the change beyond what would have happened anyway.

The county’s plan for the RCT

Waitlist Workbook
Average change 0 (REACH: +0.06) 0.5 (REACH: +0.55)
SD of change 1.0 (REACH: 0.81) 1.0 (REACH: 1.04)
People 40 40

The planning numbers are tidy versions of REACH’s two arms, and 40 per arm is what the budget covers. The plan assumes δ = 0.50, the gap between the two average changes.

Draw your trial

Why real data this time? The county still has no data of its own. In the last two lectures we filled that gap by simulating data from its planning numbers. For the first two questions today, we rehearse analyzing a finished trial, so we borrow a real one: REACH ran the same basic design, and its 3,965 completers’ change scores come with the quirks real data bring, such as scores that live on a 1–5 scale and two arms whose changes don’t spread equally. A random 40 from each arm is what one 40-per-arm trial of the workbook could look like if we ran the study in our county. Question 3 goes back to planning, and so back to simulating.

Choose your own seed, so your trial is yours.

See what the trial estimates. Each thin line below is one person in your trial: their benevolence before the two weeks and after. The thick line is each arm’s average. The trial’s question is about the two thick lines: how much more did the workbook arm’s average change than the waitlist arm’s? That difference in average change is the quantity the trial estimates.

Your observed difference, workbook minus waitlist:

Record your trial

Write down your observed difference. Is it big? Question 2 asks how unusual a difference like yours would be if the workbook did nothing.

Question 2 · How unusual is our result in a no-effect world?

Your trial gave one difference. Is it big? Looking at it can’t tell you, because even if the workbook did nothing at all, two random groups of 40 would still differ a little, just from who happened to land in which group. To judge your difference, you need to know how big the differences produced by luck alone tend to be. Randomization lets you find out, using nothing but your own 80 people.

The key idea: if the workbook did nothing, the group labels are just tags. Each person’s change score belongs to that person. If the workbook had no effect on anyone, someone whose benevolence rose by 1.2 points would have risen by 1.2 points on the waitlist, too. All that random assignment decided was which 40 people got the tag “workbook.” A different random assignment would have put different people under that tag and produced a different difference, purely by luck. So you can rerun the random assignment yourself: keep every change score as it is, deal the 40 “workbook” tags out again at random, and recompute the difference. That is a shuffle.

Shuffling, with six people

Here are six people, three per group, with numbers invented to keep the arithmetic easy. Each shuffle keeps all six change scores and deals the three “workbook” tags out again:

Person Change Real assignment Shuffle 1 Shuffle 2
A +1.2 workbook waitlist workbook
B +0.9 workbook workbook waitlist
C +0.3 workbook waitlist waitlist
D +0.6 waitlist waitlist workbook
E 0.0 waitlist workbook waitlist
F −0.3 waitlist workbook workbook
Workbook average − waitlist average 0.8 − 0.1 = +0.7 0.2 − 0.7 = −0.5 0.5 − 0.4 = +0.1

The change scores never move; only the tags do. In a shuffle the tags mean nothing, so each shuffle gives one difference that luck alone produced. Here the two shuffles gave −0.5 and +0.1, both closer to zero than the real +0.7. Shuffle many more times, and you see how big luck-only differences tend to get.

Your trial works the same way, just bigger: the computer reshuffles your 80 people’s tags, first 20 times and then 1,000. Those shuffled differences are the null world, built from your own data. If your real difference sits out where few shuffles reach, it would be unusual in a no-effect world; the share of shuffles at least as far from zero as yours is your p-value. Statisticians call this a permutation test (to permute is to rearrange).

Pair-and-share · predict first

Suppose the workbook changed nobody’s outcome. Your 80 people would still have 80 different change scores, and the 40 labeled “Treatment” would still have some average change. How big a difference could the labels alone produce by luck: bigger than yours, or smaller? Jot down a guess before you run.

In the code, the tags are the treat column ("Treatment" = workbook, "Control" = waitlist). sample() deals them out again in a random order, and the rest of the function recomputes the difference, exactly as in the six-person table:

Collect the room

Call out that last number: how many of your 20 shuffles matched or beat your real difference. Twenty shuffles are enough to feel the null world but far too few for a precise p-value, so now run 1,000 shuffles of your own study; the instructor projects one. Everyone’s histogram will look a little different, because everyone’s 80 people are different, so judge your difference against your own shuffles.

The table counts the shuffles that landed at least as far from zero as your real difference and divides that count by 1,000: the share is your permutation p. If your count is 0, report p < .001 (fewer than 1 in 1,000) rather than p = 0, because 1,000 shuffles can’t measure anything rarer than about 1 in 1,000. Hold on to it: the model-based route below gives a second p-value to set beside it. A p above .05 means your difference wasn’t unusual enough in the null world to reject \(H_0\); it does not show that the workbook has no effect.

Two wobbles: the ruler for the t-test

The permutation test needed no formula, only the design. The other route, the model-based one the Module and Wednesday’s lab use, does what last week’s test did: it measures your difference in standard errors. So it first needs to know how much a difference between two averages wobbles.

Pair-and-share · predict first

Last Monday one group’s average change wobbled by \(\sigma/\sqrt{n}\), about 0.158 at the plan’s numbers. Now the estimate is a difference between two group averages. Does that difference wobble less than one average, the same amount, or more? If more, by how much?

More. A difference between two uncertain averages inherits the wobble of both. At the plan’s numbers, one group’s average change has an SE of about 0.158 (1.0 ÷ \(\sqrt{40}\)), and the difference between two such averages has an SE of about 0.22 (\(\sqrt{2}\) × 0.158).

You cannot subtract away uncertainty. When two independent groups are the same size and have about the same spread, the difference between their averages wobbles about 1.4 times (\(\sqrt{2}\)) as much as either group’s average does, however many people are in each group.

In symbols. For independent groups, the SE of a difference is \(SE_\text{diff} = \sqrt{SE_1^2 + SE_2^2}\). Write \(n\) for the number of people in each group. When the two groups are the same size and share the same spread σ, it simplifies in three steps:

\[SE_\text{diff} = \sigma\sqrt{\frac{1}{n} + \frac{1}{n}} = \sigma\sqrt{\frac{2}{n}} = \sqrt{2}\cdot\frac{\sigma}{\sqrt{n}}\]

The last piece, \(\sigma/\sqrt{n}\), is one group’s SE. The reason is that for independent groups, variances add: each group’s average brings its own variance, \(\sigma^2/n\); the two add to \(2\sigma^2/n\); and the square root that turns a variance back into an SE turns that 2 into \(\sqrt{2}\).

Why \(\sqrt{2}\) and not 2? Each group’s average misses its own true value by a random amount, and the difference inherits both misses. When the two averages miss in the same direction, the misses partly cancel in the subtraction. When they miss in opposite directions, the misses add up. If they always added up, the difference would wobble twice as much as one average; if they always canceled, it would not wobble at all. With independent groups, each happens about half the time, so the wobble lands in between, at \(\sqrt{2} \approx 1.41\) times one average’s.

The 40 isn’t what makes the \(\sqrt{2}\). The \(n\) stays inside one group’s SE, so the \(\sqrt{2}\) is left over at any group size. With the plan’s σ = 1.0:

People per group One group’s SE, \(\sigma/\sqrt{n}\) SE of the difference, \(\sigma\sqrt{2/n}\) Ratio
40 1.0 / 6.32 ≈ 0.158 1.0 × 0.224 ≈ 0.224 \(\sqrt{2} \approx 1.41\)
80 1.0 / 8.94 ≈ 0.112 1.0 × 0.158 ≈ 0.158 \(\sqrt{2} \approx 1.41\)

Compare the two 0.158s: at 80 per group, the difference’s SE matches one group’s SE at 40. Doubling each group is what it takes to undo the \(\sqrt{2}\), which is why the comparison group in Question 3 costs so many people. The tidy \(\sqrt{2}\) also needs two groups with similar spreads. When the sizes or spreads differ, \(\sqrt{SE_1^2 + SE_2^2}\) with each group’s own SE still gives the right answer (it is the SE Welch’s t uses); the factor just won’t be exactly \(\sqrt{2}\). The standard t-test instead estimates one shared σ by pooling both groups’ SDs (the pre-study’s \(s_\text{pooled}\)). With equal group sizes, its SE matches Welch’s exactly; the two tests then differ only in their degrees of freedom.

Same skeleton, a new SE

The SE of the difference, \(SE_\text{diff}\), is the denominator of the two-group test:

\[t_\text{obs} = \frac{(\bar D_\text{workbook} - \bar D_\text{waitlist}) - 0}{SE_\text{diff}}\]

Last week’s denominator was \(SE_{\bar D}\), the SE of one group’s average change. Today’s is \(SE_\text{diff}\), the SE of the difference between two groups’ average changes.

Fit Welch’s t

Now run the model-based route on your trial:

Read it in the Module’s order: estimate (your difference in average change, the same number as obs_diff), lower_ci/upper_ci (its interval: is zero in or out?), statistic (\(t_\text{obs}\): your difference divided by its \(SE_\text{diff}\)), t_df (its degrees of freedom; Welch’s are usually not a whole number), and p_value. By default t_test() uses Welch’s version, which lets each arm keep its own spread. That suits REACH: across all completers, the waitlist’s changes spread less than the workbook arm’s (SD 0.81 against 1.04).

Set the two p-values side by side: your permutation p from the 1,000 shuffles, and the p_value from Welch’s t. They usually tell the same story; the Going further box below explains why they can differ.

The shuffle needs no formula and no curve, only the design. Its null hypothesis is the strict one: the workbook changed nobody’s outcome, so each person would have changed by the same amount in either group, and the labels are arbitrary. Randomization doesn’t just justify the causal comparison; it also supplies this reference distribution. Welch’s t is the model-based route, and its null hypothesis is about averages: the two population average changes are equal. (That could be true even if the workbook helped some people and set others back by the same total amount.) Because the two nulls claim slightly different things, the two p-values usually lead to the same decision, but not always. If yours disagree, the difference between those two claims is the place to start.

Hands up

Raise your hand if your t_test() p_value was below .05. Remember roughly what share of the room that was. Question 3 explains it.

The analysis ends here. You now have everything needed to report one completed trial: the two groups’ average changes and SDs, the difference and its interval, \(t_\text{obs}\) with its degrees of freedom, and the p-value. Power is not another step in this test. It belongs to planning, so Question 3 rewinds the clock to before the trial was run.

Question 3 · Will the design work?

Now rewind the clock: the county hasn’t collected any data yet. To plan, it imagines a real-change world (the plan’s waitlist change of 0, workbook change of 0.5, and SD of 1.0) and asks: if the true effect is δ = 0.50, how often would a trial of 40 per arm detect it? That share of repeated trials is power.

You have already run this experiment. Think back to the show of hands for Welch’s t. Everyone drew a different 40 + 40 people from the same two pools of real change scores, where the workbook arm’s average change really is higher (by 0.49 points across all completers), and yet some trials rejected \(H_0\) and some didn’t. Same truth, different conclusions. To see how often that happens, we drew 2,000 trials ahead of time, each a fresh random 40 + 40 from REACH’s completers, tested with Welch’s t: 67% rejected, about two in three. That share gives an empirical picture of the power of a 40-per-arm trial, if the two arms’ change scores looked like REACH’s. The trials that missed made a Type II error: the effect was there, and they didn’t detect it. As in M08, power is the complement of the Type II error rate: Power = 1 − β, where β is the chance of a Type II error. Here about 67% of trials detected the effect, so β is about 33%.

Simulating the county’s trial

We’ll simulate data for our power exploration — just like we did last week. Here is last week’s simulate_study() with one line added: two rnorm() calls now, one per group, and the test compares them with each other instead of with zero change. First, the county’s plan as budgeted, 40 per arm:

Starting point

Your study plan now names the change you expect in each of two groups — a comparison group whose average change is mu_control and an intervention group whose average change is mu_treat, sharing an SD of change sigma — and you want the power of a trial that measures n_per_group people in each.

What the code does

Inside replicate(), each rehearsal draws two samples with rnorm(), runs a two-sample t-test between them, and returns TRUE if p fell below alpha. The mean of those TRUE/FALSE values is the share of rehearsals that rejected: power.

Set mu_treat equal to mu_control and the same function reports the Type I rate, exactly as last week’s did.

Key functions

Function What it does
rnorm(n_per_group, mean, sd) × 2 Two simulated groups — the second line is the only thing new this week.
t.test(treat, control)$p.value Welch’s two-sample t-test (R’s default; the Module’s Test 1 shows it beside the standard version), with the p-value pulled out.
replicate(n_sims, { … }) Runs the study n_sims times and stacks the results.
mean(rejected) The proportion of TRUEs: the power.

How to read output

One number: the power of the county’s trial at 40 per arm, near 0.60. (That reference value comes from the standard equal-variance test’s formula. The function runs Welch’s test, which, with equal groups and equal SDs, lands within simulation noise of it.)

The plan’s power at 40 per arm, about 60%, sits a little below the room’s 67%, mainly because the plan assumes both arms spread like the workbook arm (SD 1.0), while REACH’s waitlist changes spread less.

How power grows with the size of the trial

One number answers one plan. To see how power depends on the size of the trial and the size of the effect, run the same function across sample sizes from 10 to 500 per group, for a small, a medium, and a large effect. The effects are Cohen’s d: the difference between the two groups’ average changes divided by the SD of change. With an SD of 1, the effect in points is the d, and the county’s 0.50 ÷ 1.0 = 0.5 is a medium effect by Cohen’s benchmarks (0.2 small, 0.5 medium, 0.8 large). The graph below does exactly that: 30 power simulations of 1,000 trials each, one for every point. That takes a while to run, so it is drawn for you; open the code underneath to see how it was built.

Show the code that built this graph 
# The same function as in the chunk above
simulate_two_groups <- function(n_per_group, mu_control, mu_treat, sigma,
                                alpha = 0.05, n_sims = 1000) {
  rejected <- replicate(n_sims, {
    control <- rnorm(n_per_group, mean = mu_control, sd = sigma)
    treat   <- rnorm(n_per_group, mean = mu_treat,   sd = sigma)
    t.test(treat, control)$p.value < alpha
  })
  mean(rejected)
}

set.seed(2026)

# Power at each sample size, for each effect size (SD 1, so each effect is a Cohen's d)
power_curve <- expand_grid(n_per_group = c(10, 25, 40, 64, 100, 150, 200, 300, 400, 500),
                           d           = c(0.2, 0.5, 0.8)) |>
  mutate(power = map2_dbl(n_per_group, d, \(n, d)
           simulate_two_groups(n_per_group = n, mu_control = 0, mu_treat = d, sigma = 1.0)))

power_curve |>
  ggplot(aes(x = n_per_group, y = power, color = factor(d))) +
  geom_hline(yintercept = 0.80, linetype = "dashed", color = "#C05852") +   # the 80% convention
  geom_line(linewidth = 1.1) +
  geom_point(size = 2.5) +                                                  # each point = 1,000 simulated trials
  scale_color_manual(values = c("#AD872B", "#4E5EAA", "#348C9E"),
                     labels = c("0.2 (small)", "0.5 (medium)", "0.8 (large)")) +
  scale_y_continuous(labels = scales::percent, limits = c(0, 1)) +
  labs(title = "Power of a two-group trial, by sample size and effect size",
       subtitle = "Each point = 1,000 simulated trials  ·  rose dashed line = 80% power",
       x = "People per group", y = "Power", color = "Cohen's d")

A line graph of simulated power against people per group, from 10 to 500, for three effect sizes, with a dashed line at 80 percent. The large-effect curve (d = 0.8) climbs steeply and crosses 80 percent at about 25 per group. The medium-effect curve (d = 0.5) is near 60 percent at 40 per group and crosses 80 percent at about 64 per group. The small-effect curve (d = 0.2) rises slowly and reaches 80 percent only at about 400 per group. Each point is the share of 1,000 simulated trials that rejected the null hypothesis, so the lines are slightly uneven.

Read it the way you read last week’s power curve. At 40 per group, the county’s medium effect has about 60% power. The medium curve crosses 80% at about 64 per group; a large effect gets there with about 26, and a small effect needs nearly 400 per group (394 by the exact formula in Your study), close to 800 people in all. Two things to carry into your own planning: the n on this axis is per group, so double it for the total; and a modest drop in the effect you expect raises the n a lot, so choose a realistic d. If your expected effect comes from one small published study, plan for a somewhat smaller one.

What the comparison group costs

Same effect, same σ, same α. Only the design changed. (The n for 80% power is where each design’s power curve crosses 80%.)

Design Power at n = 40 n for 80% power People measured
One group’s change against zero (last week’s pilot) 0.87 34 34
Two groups compared with each other (today) 0.60 64 per group 128

Under these planning assumptions, answering the two-group question takes nearly four times as many participants as detecting a 0.5-point change from zero in one group: twice as many per group, because the difference’s SE is \(\sqrt{2}\) times one mean’s and only doubling each group shrinks it back, and then two groups. What those extra participants give you is the thing last week’s pilot could not — a comparison measured the same way, at the same time, on the same kind of people, with chance deciding who got the workbook. That randomized comparison is what gives the board a basis for a causal claim about the average difference.

The nearly fourfold figure follows from these assumptions (the same effect, the same σ, equal groups); it is not a fixed rule. Change any of them, as your own study will, and the cost changes too.

Your study · add a comparison group

Everything so far ran on the county’s trial. Now give your study a comparison group, the same step the county just took. Start Step 1 in class; whatever you don’t finish, including Step 2, complete on your own after class. The boxes below don’t save, so a refresh or a closed tab clears them: copy your rows and sentences into your own notes. If you didn’t finish last week’s self-paced Part 4 (Your pilot), do that first, since this builds on its section 2.

Step 1 · Your comparison group, rehearsed

Where to get your SD of change. It is often the hardest planning number to find, because papers usually report the SD of scores at each time point but not the SD of the change. In order of preference:

  1. A study like yours that reports it, with the same measure and a similar time between measurements, or your own pilot data.
  2. Build it from the two SDs and the correlation. If a paper gives the SD at each time point (\(SD_1\) and \(SD_2\)) and the correlation \(r\) between people’s before and after scores, then \(SD_\text{change} = \sqrt{SD_1^2 + SD_2^2 - 2\,r\,SD_1\,SD_2}\). If the correlation isn’t reported, the measure’s test-retest reliability (the correlation between two administrations of the same measure, over a similar interval, when nothing is meant to change) is a reasonable stand-in. If you expect people to respond to the intervention differently, assume a somewhat smaller r.
  3. A quick rule when the two SDs are about equal: \(SD_\text{change} = SD \times \sqrt{2(1 - r)}\). At r = .5 the change score varies as much as the scores themselves; at r = .8 it varies about 0.63 times as much, and at r = .3 about 1.18 times as much. The more consistent people are from one time to the next, the smaller the SD of change.

For the county, both routes land near 1.0: REACH’s workbook arm had an SD of change of 1.04, and last week’s simulated pilot (scores with SD 1.0 and r = .5) gives 1.0 × \(\sqrt{2(1 - 0.5)}\) = 1.0. When you’re unsure, lean a little high, because an SD that is too small makes the study look more powerful than it is. Then rerun the chunk with a second, larger SD to see how much your answer depends on it.

Two ways to compute your study’s power: Question 3’s simulation function, and R’s formula, power.t.test(). Enter your numbers and run it:

Ways 1 and 2 should agree to within a couple of percentage points. The simulation rehearses 1,000 trials, so it wobbles a little (near 80% power, its Monte Carlo SE is about .013, from \(\sqrt{0.8 \times 0.2 / 1000}\)); the formula computes power exactly for this simple design. The last line runs the formula backward to give the n per group for 80% power. To check it by simulation, set my_n_per_group to that n and run the chunk again.

  • In power.t.test(), delta is the difference between the two groups’ expected changes, and sd is the SD of change in each group. type = "two.sample" is this design, and n counts people per group. strict = TRUE counts both tails, matching a two-sided test.
  • Why learn both? Two groups with a continuous outcome is one of the most common designs in psychology, and the formula handles it in one line. But the formula assumes roughly Normal outcomes with equal SDs in the two groups. When your design differs (unequal spreads, dropouts, clustering), simulation still works: change the code that generates the data, and count the rejections.

Your study plan, section 3 · Your comparison group

Use the numbers you just ran. Fill in the new rows — and notice what the third one does to your δ:

Element Your value
The comparison group — who they are, and what they get (waitlist, usual care, an active control)
The change you expect in the COMPARISON group — change over time, rawness easing on its own, practice with the questionnaire
The effect your trial is powered for: δ = your change − theirs — the difference between the two expected changes; for the county it is smaller than last week’s workbook-arm change
Power at your feasible n per group
The n per group for 80% power (from power.t.test()) — and whether you can afford it

If you cannot afford it, last week’s four things that move power are what you have to work with. And notice the design is already helping: each arm is measured before and after, so the outcome is a change score. The M09 bonus activity shows when that within-person step helps and when it doesn’t.

Step 2 · Your design in six sentences

Your study plan now has three sections, and together they describe a design. Turn the plan into six sentences a colleague could act on.

Six sentences

Six sentences, in this order. Use the numbers in your study plan.

  1. The outcome, in its units, and the scale it lives on.
  2. The comparison and assignment — who is compared with whom (one group before and after, or two groups measured alongside each other), whether observations are independent or paired, and, for a causal trial, how participants are assigned to conditions.
  3. The world — the change you expect in each group, the SD of change, and where each number came from.
  4. The sample — n (per group), and the power that n gives you against the effect you expect.
  5. The rule — α, two-sided or not, and what you will do with each of the two possible answers.
  6. Which of the four things that move power you would act on if the reviewer says the sample is too small.

Talk your partner through your six sentences and check one thing: could they reconstruct your statistical design and analysis plan from those six sentences alone? If not, the missing sentence is the one to fix.

Wrapping up

Keep the three questions apart

Question Where we look What it gives us
What happened in our trial? Your one trial The difference between the two groups’ average changes
How unusual is our result in a no-effect world? Your 80 people, labels reshuffled as if the workbook did nothing; then the t curve The null world and the p-value; from Welch’s t, \(t_\text{obs}\) and the interval
Will the design work? Many imagined future trials, drawn from a real-change world Power, and the n the design needs

Inference asks what we can learn from the trial we got. Power asks how the design would behave across the trials we might get.

Three lectures, one increasingly ambitious question

The estimand The question
M07 \(\mu\) Where do people start?
M08 \(\mu_D\) Did they change?
M09 \(\mu_{D,\text{workbook}} - \mu_{D,\text{waitlist}}\) Did they change more than they would have anyway?

For the mean questions that carried M07–M09, the same skeleton kept returning:

\[\frac{\text{estimate} - \text{null value}}{SE}\]

What changed today was the design: the estimate became a difference between two averages, and its SE carries two wobbles.

What’s next.

The county’s story ends here — for now. In Part 3 of the course you will analyze the real REACH trial, the study whose numbers the team borrowed all along.

Wednesday’s lab is a computational reproduction: Zhang et al.’s illusion-of-predictability study, with the two-sample Welch’s t you just ran — rebuilt end to end on a published paper’s own data, an estimation plot beside it, and your numbers checked against the printed ones.

Footnotes

  1. For teaching we use the change scores of the 3,965 participants who completed follow-up, out of the 4,598 in the trial’s analytic sample. (The authors kept everyone, using an intention-to-treat analysis with multiple imputation.) A real trial’s analysis also has to handle missing follow-up and its prespecified primary estimand, because analyzing only completers can weaken the protection randomization provides.↩︎