Revenge, Adjusted

Lecture · Module 11 · Mon Nov 9

How today works

The M11 Module added baseline benevolence and then site to the REACH treatment model. Today we fit the same models on revenge, the outcome we met last Monday. Same data, same code as the Module, new answers. Then last week’s researcher returns with a new study, and a new mistake that adjusting for baseline can’t fix.

Today’s plan

Part The question The Module section it revisits
Part 1 Do the arms differ in what baseline revenge didn’t predict? Part 1: residualized gain
Part 2 What does adjusting for baseline revenge change? Part 2: the two-predictor model, partial effects, the precision gain, R²; Part 3: adjusted means, the CI and the PI
Part 3 What does adding site change? Part 4: ΔR², the nested F-test, why coefficients move
Part 4 What if we analyze only the people who used the workbook? Part 6: what licenses a causal claim, and variables measured after treatment

Each part follows the Module’s steps in the Module’s order, so you can keep the Module open beside this page.

Part 1 · Do the arms differ in what baseline revenge didn’t predict?

We start where the Module’s Part 1 starts: everyone with both revenge scores, in both arms and all six sites. The chunk below builds the analysis sample with the Module’s recipe, including the centered baseline we’ll need in Part 2.

That’s 3,964 people: 2,057 on the waitlist and 1,907 who got the workbook.

Step 1 · Fit baseline alone, then look at what it missed

Fit follow-up revenge on baseline revenge, with no treatment information in the model. Each person’s residual is what baseline alone failed to predict: their residualized gain.

Pair-and-share · predict first

The workbook is meant to lower revenge. If it works, which arm’s residuals should sit mostly below zero, ending lower than baseline alone predicted? Decide with your neighbor, then run the chunk.

The workbook arm. Its average residual is −0.181: given where they started, workbook participants ended a little lower in revenge than baseline alone predicted. The waitlist’s average residual is 0.168, a little higher. The gap between them, −0.181 − 0.168 = −0.349, is the residualized treatment contrast, and we found it with a model that never saw treat.

Step 2 · The MLR finds the same number in one step

The treatTreatment row says −0.349, almost exactly the residual gap from Step 1 (−0.349). As in the Module, the two-step shortcut is a close preview; the one-step regression is the version to report, because it residualizes both sides and uses the right degrees of freedom.

Part 2 · What does adjusting for baseline revenge change?

The model, centered

As in the Module’s Part 2, we refit with the centered baseline, so the intercept describes someone at the average baseline instead of an impossible baseline of 0.

Read the output · three coefficients

Answer each from your output before you open the answers. The phrase that matters in each answer is who is being compared with whom.

  1. The intercept, 1.893
  2. The baseline slope, 0.493
  3. The treatment slope, −0.349
  1. The predicted follow-up revenge for a waitlist participant at the average baseline (2.00). Centering moved the intercept to a real, in-data profile; the slopes are the same as in the uncentered fit.
  2. Two people in the same arm whose baseline revenge differs by 1 point are predicted to differ by 0.49 points two weeks later.
  3. A workbook participant and a waitlist participant who started at the same baseline revenge: the workbook participant is predicted to score 0.35 points lower, 95% CI [−0.393, −0.305]. That “same baseline” is what makes each slope a partial effect.

Each row’s t, p, and CI come from M10’s recipe, with df = n − 3 = 3,961 because the model estimates three coefficients.

The precision gain

Pair-and-share · predict first

Last week’s treat-only model said the workbook lowered revenge by 0.359 points. Now baseline is in the model. Which do you expect to change more: the treatment estimate, or its standard error? Make your guess, then open the table.

Quantity lm(t2_revenge ~ treat) lm(t2_revenge ~ t1_revenge.c + treat)
Treatment coefficient −0.359 −0.349
Its SE 0.0275 0.0225
\(\sigma\) 0.865 0.707
R² .041 .361
Adjusted R² .041 .361

The estimate barely moves: randomization left the two arms almost even on baseline revenge, so baseline has almost nothing to correct. The SE drops by about 18%: baseline accounts for much of the person-to-person spread in follow-up revenge (\(\sigma\) falls from 0.87 to 0.71), so the same difference is estimated against less leftover noise. That’s the Module’s lesson, repeated with revenge: adjusting for a strong pre-treatment predictor in a randomized trial doesn’t change what we’re estimating; it estimates it more precisely.

In each column, R² and adjusted R² are nearly identical: with n = 3,964, the penalty for one or two slopes is tiny.

The adjusted means, the CI, and the PI

Name the quantity first, then get it from the model. Here: predicted follow-up revenge in each arm, for people at the average baseline. predict() returns it with either interval, exactly as in M10.

Pair-and-share · what does each interval say?

  1. What are the two adjusted means, and where have you seen the waitlist’s before?
  2. A clinic asks how much the workbook lowers revenge on average. A client asks how vengeful she is likely to feel in two weeks. Which interval answers which?
  3. Look at the bottom of the pale bars. What’s odd about them?
  1. 1.89 on the waitlist and 1.54 with the workbook, for people at the average baseline. The waitlist’s is the intercept, and the difference is the treatment slope: 1.543 − 1.893 = −0.349.
  2. The clinic’s question is about averages, so the CIs answer it: each is about 0.06 of a point wide, so the two averages are pinned down precisely. (For the difference, read the treatment slope’s CI, not whether the two intervals overlap.) The client’s question is about one person, so the PIs answer it: about 2.77 points wide, roughly 45 times the CI, and the two arms’ PIs overlap almost entirely. Knowing which arm someone was in tells you much less about their own score than the CIs suggest.
  3. They run below 1, to 0.51 and 0.16, on a scale where 1 is the lowest possible score. The PI assumes Normal scatter around the prediction, and revenge piles up at the floor (Meet revenge, last week). The Module’s benevolence PI ran past the top of its scale; revenge runs past the bottom. M12 checks what the floor does to the model’s assumptions.

Part 3 · What does adding site change?

Site was REACH’s randomization stratum: people were randomized within each site. As in the Module’s Part 4, we add it to absorb the between-site differences in revenge you saw last week.

What moved, and what didn’t

Quantity Treat only + baseline + baseline + site
Treatment coefficient −0.359 −0.349 −0.349
Its SE 0.0275 0.0225 0.0219
R² .041 .361 .395

The treatment estimate barely moves at either step: within-site randomization balanced the arms on site, too. The biggest precision gain came from baseline; adding site changes the SE very little.

Does site add information? The nested-model test says yes: F(5, 3,956) = 44.88, p < .001. But ΔR² = .395 − .361 = .034: beyond baseline and treatment, site accounts for about 3.4 more percentage points of the variation in follow-up revenge. As the Module says, that test isn’t why site is in the model. Site is there because it’s how the trial was randomized.

Why do the site coefficients move?

The treatment coefficient barely changed when baseline entered. The site coefficients are a different story. Compare them with and without baseline in the model:

Pair-and-share · why does Hong Kong’s gap shrink?

Without baseline, Hong Kong’s follow-up revenge sits 1.07 points above Colombia’s. With baseline in the model, the gap is 0.56. Last week, Hong Kong had the highest baseline revenge of the six sites. Why does adding baseline shrink its gap, when it barely touched the treatment coefficient?

Part of Hong Kong’s higher follow-up revenge is carried over from where its participants started. Once baseline is in the model, the Hong Kong coefficient compares people with the same baseline revenge, and that carried-over part is credited to baseline instead. In this table, all five site gaps shrink toward zero once baseline enters. Treatment is different: randomization left the two arms almost even at baseline, so baseline has almost nothing to carry. A coefficient changes when a predictor enters because the comparison it makes changes.

The result to report

Among participants with follow-up data, those randomly assigned to the workbook scored an estimated 0.35 points lower on follow-up revenge than those on the waitlist, adjusting for baseline revenge and site, 95% CI [−0.39, −0.31], t(3,956) = −15.93, p < .001.

The estimand leads; the baseline and site coefficients stay in the table.

Part 4 · A common mistake: analyzing only the people who used the workbook

Last week’s researcher took the lesson to heart. His new study is randomized, just like REACH: half his participants get the workbook now, half wait two weeks. At follow-up, he asks the workbook arm one extra question: did you actually use it? About a third say no.

He reasons: “Those people never received the treatment, so they can’t tell me what the workbook does.” He drops them, and compares the workbook arm’s users with the whole waitlist. Analyses that keep only the people who followed the plan are often called per-protocol analyses.

Pair-and-share · predict first

  1. Compared with analyzing everyone as randomized, will his estimate of the workbook’s effect be bigger, smaller, or about the same?
  2. In what ways might the people who skipped the workbook differ from the people who used it?

Why we simulate

Our REACH file doesn’t record who actually used the workbook, so we can’t run his analysis on REACH. Instead we’ll build a simulated trial in which we write the truth, so we can check each analysis against it. (No REACH participants are in this part; every person below is simulated.)

Here is the truth we write:

  • Each person has an engagement level: how actively they’re working on the hurt. Nobody measures it.
  • More engaged people are more likely to use the workbook if they’re given it. About two-thirds would.
  • Engagement lowers revenge on its own: engaged people would end up a little less vengeful even without the workbook.
  • Using the workbook lowers revenge by exactly 0.50 points. This is the true effect of using it.

Two pieces of code may be new. plogis() turns any number into a probability between 0 and 1, so higher engagement means a higher chance of using the workbook. rbinom() with size = 1 flips one weighted coin per person with that probability.

The 0.65 and 0.62 in the t2 line are last week’s waitlist regression line, rounded, and baseline revenge gets REACH’s mean and SD, about 2 and 1. The other numbers (0.85, 1.2, 0.35, 0.70) are choices we made so that about two-thirds use the workbook and engagement matters a little. (To keep the simulation simple, its scores aren’t held between 1 and 5.)

Run his analysis

What would he conclude?

Analyzed as randomized, the workbook lowered revenge by 0.33 points. Dropping the 34% of the workbook arm who didn’t use it, he gets 0.62, nearly twice as large. What would he write in his abstract? And which number is right, given that we wrote the true effect of using the workbook as 0.50?

Neither number is 0.50. The as-randomized estimate is smaller and his is larger, and they’re off for very different reasons.

The as-randomized estimate answers a different question: what happens when you offer people the workbook. Only about two-thirds use it, so the average effect of being offered it is 0.50 × 66% ≈ 0.33. That’s a fair answer to a real question (should a clinic hand out this workbook?), and it’s the one randomization protects. It’s called the intention-to-treat (ITT) estimate, and the Module’s footnote on ITT describes it.

His estimate overshoots the 0.50. To see why, look at who is in each group he compares.

The people he can’t see

His users averaged 0.36 on engagement; the waitlist averaged −0.01. He didn’t compare similar people: he compared a more engaged two-thirds of the workbook arm with all of the waitlist.

The waitlist has non-users too. About a third of the waitlist would also have skipped the workbook if they’d been given it. He can’t see them, because they never had a workbook to skip, so all of them stay in his comparison group. The exclusion happens in one arm only, and on a waitlist there is no way to make the matching exclusion.

Where his extra points come from. Across many trials, users average about 0.32 SD more engaged than the waitlist as a whole (0.37 in this one), and engagement lowers revenge by 0.35 points per SD. So the users would have ended about 0.35 × 0.32 ≈ 0.11 points lower even without the workbook. His analysis credits that to the workbook: −0.50 − 0.11 ≈ −0.61.

A simulation can make the fair comparison directly. In a simulation we know would_use for everyone, so we can compare users with the waitlist’s would-be users, like with like:

That gives −0.51, close to the true −0.50. A real study can’t run this chunk: would_use is never observed on the waitlist.

Can adjusting for baseline fix it?

Pair-and-share · predict first

Today’s whole Module is about adjusting. Suppose he adds baseline revenge to his model, lm(t2 ~ t1 + treat), on the same users-only data. Will his estimate move toward −0.50? Decide with your neighbor, then run the chunk.

It stays at −0.619. Adjusting removes differences in the variables you put in the model, and only those. Here his users and the waitlist started at about the same baseline revenge; what differs is engagement, which nobody measured. Adjusting for baseline makes his estimate more precise, as Part 2 showed, but it stays off target. (In real data, engagement might be related to baseline revenge, and then adjusting for baseline would remove the part of the difference that baseline captures. It can’t remove the rest.)

This is the Module’s Part 6 warning in a new form. Whether someone used the workbook was decided after randomization, and it can be observed in one arm only. Selecting on it compares groups that chance no longer made comparable.

When is dropping non-users harmless?

Suppose instead that non-use had nothing to do with engagement: say, about 30% of the download links broke at random. Swap the would_use line for a plain 70% chance and rerun his analysis:

Now his analysis lands at −0.50, right on the true −0.50. Dropping non-users is harmless only when the reason for non-use is unrelated to the outcome. He has no way to check that from his data: the reason here was engagement, and engagement was never measured.

So what should he report?

Pair-and-share · his write-up

He still wants to know what the workbook does for the people who actually use it. What should he report, and is his question a bad one?

His question is a good one; his analysis can’t answer it. Start from the estimand, as the Module says:

  • “What does offering the workbook do?” The as-randomized (ITT) estimate answers it, and randomization protects it. This is the headline most trials report, and it’s what REACH reported: its authors analyzed everyone as randomized and used multiple imputation for missing follow-ups.
  • “What does using it do, for people who would use it?” This is a different estimand. Estimating it from a real trial takes methods beyond this course, built on extra assumptions. Dropping non-users from one arm doesn’t estimate it.

He can report the ITT estimate, and report how many people used the workbook alongside it.

One more connection: the M10 lecture noted that more of REACH’s workbook arm missed the follow-up (16.7% against 10.9%). Our complete-case analysis drops those people, which is a milder version of the same problem, and that’s why REACH’s authors used multiple imputation instead.

The sentence of the day

Randomization makes the arms comparable, and dropping people from one arm for something that happened after randomization breaks that. Adjusting for what you measured can’t repair a difference in what you didn’t measure.

The same applies whenever an exclusion depends on what happened after assignment: dropping students who skipped tutoring sessions, patients who stopped taking a pill, or app users who never logged in.

It happens in lab experiments too. Many psychology experiments drop participants who fail an attention check or don’t follow the instructions. Everyone takes the same check, so this can look fair. But if the check comes after the manipulation, the condition itself can change who fails it: a long, dull, or upsetting condition may lose more of its less attentive participants than a short, easy one. The people left in each condition then differ in ways chance didn’t create, just as his users and the waitlist did.

The fix is to decide who’s in before anyone is assigned. Measure attentiveness before randomizing, for example with an attention check placed before the manipulation, exclude on that, and then randomize the people who remain. Nothing that happens in either condition can change who was excluded, so randomization keeps the conditions comparable. The trade-off is the estimand: the result now describes attentive participants, so the write-up should say so.

A bigger trial makes every estimate more precise. Does it bring his estimate closer to −0.50? Run it, then try n <- 40000, ten times larger.

Even at 4,000 people his CI, about [−0.69, −0.55], already misses the true −0.50. At 40,000 it narrows to about [−0.64, −0.59], still centered near −0.62. More data shrink the uncertainty around whatever an analysis estimates; they don’t change what it estimates.

Wrapping up

What today covered, with revenge

The Module’s idea Today’s revenge result
Residualized gain previews the MLR coefficient Residual gap −0.349; the MLR’s treatment coefficient −0.349
Partial effects: each slope compares people who match on the other predictors Same baseline, different arm: −0.35 points
Adjusting in an RCT sharpens the estimate without changing it Estimate −0.359 → −0.349; SE down about 18%
Adjusted means, CI vs. PI 1.89 and 1.54; the PIs run below the scale’s floor
ΔR² and the nested F-test Site adds ΔR² = .034, F(5, 3,956) = 44.88
Coefficients move when the comparison changes Hong Kong’s gap 1.07 → 0.56 once baseline enters
Variables measured after treatment Dropping non-users: −0.62, against a true −0.50, and adjusting doesn’t move it

What’s next.

Wednesday’s lab continues the Bread + Peace analysis: you’ll put growth and fatalities in one model, see which elections’ misses shrink, and re-forecast 2020 and 2024.

Next Monday, revenge comes back one more time. You’ll check the assumptions behind today’s model, and the floor you saw in the PIs is one of the first things you’ll look at.