Skip to main content

Revenge, Adjusted

Lecture · Module 11 · Mon Nov 9

How today works

The M11 Module added baseline benevolence and then site to the REACH treatment model. Today we fit the same models on revenge, the outcome we met last Monday. Same data, same code as the Module, new answers.

Today’s plan

Part The question The Module section it revisits
Part 1 Do the arms differ in what baseline revenge didn’t predict? Part 1: residualized gain
Part 2 What does adjusting for baseline revenge change? Part 2: the two-predictor model, partial effects, the precision gain, R²; Part 3: adjusted means, the CI and the PI
Part 3 What does adding site change? Part 4: ΔR², the nested F-test, why coefficients move
Part 4 What if we adjust for something the workbook changed? Part 6: what licenses a causal claim, and variables measured after treatment

Each part follows the Module’s steps in the Module’s order, so you can keep the Module open beside this page.


Part 1 · Do the arms differ in what baseline revenge didn’t predict?

We start where the Module’s Part 1 starts: everyone with both revenge scores, in both arms and all six sites. The chunk below builds the analysis sample with the Module’s recipe, including the centered baseline we’ll need in Part 2.

That’s 3,964 people: 2,057 on the waitlist and 1,907 who got the workbook.

Step 1 · Draw the line that baseline alone predicts

Fit follow-up revenge on baseline revenge, with no treatment information in the model, and draw the line. This is the Module’s Step 1 figure with revenge in place of benevolence. The points are colored by arm, but the line doesn’t know that: it’s fit to everyone at once.

Read the line as an expectation. The fitted line is \(\widehat{\text{T2}} = 0.73 + 0.49 \cdot \text{T1}\). It answers one question: if all you knew about someone was their baseline revenge, what follow-up score would you expect? For someone who started at 3, it expects 2.22. Each person’s vertical distance from the line is their residual. A point above the line ended with more revenge than their start predicted. A point below the line ended with less revenge than expected.

Watch the direction. In the Module, the workbook was meant to raise benevolence, so success meant sitting above the line. The workbook is meant to lower revenge, so here success means sitting below it.

With 3,964 points piled on top of each other, the cloud is too dense to judge by eye. So let’s zoom in on people who all started at the same baseline score.

Step 2 · Zoom in on people who started in the same place

Take the 197 people who started at exactly 3 on baseline revenge: the vertical stack of points directly above 3 on the x-axis. The line gives every one of them the same expectation, 2.22, because baseline is all it knows about them.

Pair-and-share · same start, same expectation

Everyone in this group started at 3, and the line expected all of them to end at 2.22. Which arm ended with less revenge than expected? Is it fair to credit that gap to the workbook? (Hint: who decided which of these people got it?)

The workbook arm ended below the line. Waitlist participants who started at 3 averaged 2.67 at follow-up, above the line. Workbook participants who started at 3 averaged 1.96, below it. Credit the workbook? Yes, because both groups started in the same place, and a coin flip decided which of them got the workbook. This is what “lower than expected” means: lower than people who started where you did. One arm sitting above the line and the other below it is what a treatment effect looks like here. The line is fit to both arms together, so it runs between them.

Step 3 · Everyone, at every baseline score: what baseline didn’t predict, by arm

Step 2 compared the arms only among people who started at 3. The residual makes that same comparison for everyone: it measures each person against the line’s expectation for their own baseline score. Each person’s residual is what baseline alone failed to predict: their residualized gain.

Pair-and-share · predict first

The workbook is meant to lower revenge. If it works, which arm’s residuals should sit mostly below zero, ending lower than baseline alone predicted? Decide with your neighbor, then run the chunk.

The workbook arm. Its average residual is −0.181: given where they started, workbook participants ended a little lower in revenge than baseline alone predicted. The waitlist’s average residual is 0.168, a little higher. The gap between them, −0.181 − 0.168 = −0.349, is the residualized treatment contrast, and we found it with a model that never saw treat.

The dashed zero line is Step 1’s black line, laid flat. A residual below zero is a point below the line. 73% of the workbook arm sit below it, compared with 53% of the waitlist. That’s Step 2’s pattern for people who started at 3, repeated at every baseline score.

What’s the dense stripe just below zero in each arm? It’s the 824 people who scored 1, the lowest possible revenge, at both baseline and follow-up. The line predicts 1.23 for anyone who starts at 1, so every one of them gets the same residual, 1 − 1.23 = −0.23, and their points stack into a solid line. It’s the floor again, and M12 comes back to it.

Step 4 · The MLR finds the same number in one step

The treatTreatment row says −0.349, almost exactly the residual gap from Step 3 (−0.349). As in the Module, the two-step shortcut is a close preview; the one-step regression is the version to report, because it residualizes both sides and uses the right degrees of freedom.


Part 2 · What does adjusting for baseline revenge change?

The model, centered

As in the Module’s Part 2, we refit with the centered baseline, so the intercept describes someone at the average baseline instead of an impossible baseline of 0.

Read the output · three coefficients

Answer each from your output before you open the answers. The phrase that matters in each answer is who is being compared with whom.

  1. The intercept, 1.893
  2. The baseline slope, 0.493
  3. The treatment slope, −0.349
  1. The predicted follow-up revenge for a waitlist participant at the average baseline (2.00). Centering moved the intercept to a real, in-data profile; the slopes are the same as in the uncentered fit.
  2. Two people in the same arm whose baseline revenge differs by 1 point are predicted to differ by 0.49 points two weeks later.
  3. A workbook participant and a waitlist participant who started at the same baseline revenge: the workbook participant is predicted to score 0.35 points lower, 95% CI [−0.393, −0.305]. That “same baseline” is what makes each slope a partial effect: an adjusted comparison between people who match on the other predictor. “Partial” describes the comparison and doesn’t make it causal. The treatment slope has a causal reading because treatment was randomized; the baseline slope compares people who differ in baseline revenge, and nobody assigned those differences.

Each row’s t, p, and CI come from M10’s recipe, with df = n − 3 = 3,961 because the model estimates three coefficients.

The precision gain

Pair-and-share · predict first

Last week’s treat-only model said the workbook lowered revenge by 0.359 points. Now baseline is in the model. Which do you expect to change more: the treatment estimate, or its standard error? Make your guess, then open the table.

Quantity lm(t2_revenge ~ treat) lm(t2_revenge ~ t1_revenge.c + treat)
Treatment coefficient −0.359 −0.349
Its SE 0.0275 0.0225
Width of its 95% CI 0.108 0.088
\(\sigma\) 0.865 0.707
R² .041 .361
Adjusted R² .041 .361

The estimate barely moves: in this sample, the randomized arms were already nearly balanced on baseline revenge (randomization balances them on average, not exactly in every sample), so baseline has almost nothing to correct. The SE drops by about 18%: baseline accounts for much of the person-to-person spread in follow-up revenge (\(\sigma\) falls from 0.87 to 0.71), so the same difference is estimated against less leftover noise. That’s the Module’s lesson, repeated with revenge: adjusting for a strong pre-treatment predictor in a randomized trial doesn’t change what we’re estimating; it estimates it more precisely.

How big is that 18% drop? Two ways to see it:

  • The CI tightens. The treatment coefficient’s 95% CI shrinks from 0.108 points wide to 0.088.
  • It’s worth about 1,980 more participants. As a rough rule, an SE shrinks with the square root of the sample size, so to cut it this much without baseline you’d need \(\left(\frac{0.0275}{0.0225}\right)^2 = 1.50\) times as many people: about 5,940 instead of 3,964. Baseline revenge was already sitting in the data set. Adding it to the model bought the precision of roughly 1,980 extra participants for free.

In each column, R² and adjusted R² are nearly identical: with n = 3,964, the penalty for one or two slopes is tiny.

The adjusted means, the CI, and the PI

Picture two people reading these results, each with a different question.

  • The director of a counseling center is deciding whether to offer the workbook to everyone on the center’s waitlist. The director’s question is about averages: across many clients, how much lower does revenge end up with the workbook than without it?
  • Jordan was deeply hurt by a close friend and is wondering whether two weeks with the workbook is worth it. Jordan’s revenge score today is about average for REACH (around 2.0). Jordan’s question is about one person: if I do the workbook, where am I likely to land?

The same model answers both questions, but with different intervals.

Name the quantity first, then get it from the model. Here: predicted follow-up revenge in each arm, for people at the average baseline. predict() returns it with either interval, exactly as in M10.

Pair-and-share · what does each interval say?

  1. What are the two adjusted means, and where have you seen the waitlist’s before?
  2. Cover the right panel. From the left panel alone, how different would you expect one workbook participant and one waitlist participant to be at follow-up? Now uncover the right panel. What changed?
  3. Which interval answers the director’s question, and which answers Jordan’s? Based on the right panel, what would you honestly tell Jordan?
  4. Look at the bottom of the pale bars. What’s odd about them?
  1. 1.89 on the waitlist and 1.54 with the workbook, for people at the average baseline. The waitlist’s is the intercept, and the difference is the treatment slope: 1.543 − 1.893 = −0.349.
  2. The left panel makes the arms look worlds apart. The two bars sit far from each other, with empty space between them, so it’s tempting to read it as “workbook people end up much less vengeful.” Two things produce that impression. First, the y-axis: ggplot zoomed in to fit just the CIs, so a difference of 0.35 points fills the whole panel. Second, the question: a CI describes an average, and with about 2,000 people per arm, each average is pinned down to within a few hundredths of a point. The right panel adds the PIs for one person in each arm, and its y-axis widens to fit them. Now the same CIs shrink to thin slivers. 87% of each arm’s PI overlaps the other arm’s. The averages clearly differ, but individual people in the two arms overlap almost completely: pick one waitlist participant and one workbook participant, and the workbook participant could easily be the more vengeful of the two.
  3. The director’s question is about averages. The two CIs describe each arm’s mean follow-up revenge at the average baseline: 1.54 with the workbook and 1.89 without it, each pinned down to within about 0.06 of a point. But the director’s real question is the difference between those means, and for that you read the treatment coefficient and its CI: −0.35, 95% CI [−0.39, −0.31], not whether the two arm CIs overlap. The average difference is estimated quite precisely. Jordan’s question is about one person, so the PIs are the relevant intervals. Each is about 2.77 points wide, roughly 45 times the CI, and the workbook PI and the waitlist PI overlap by 87%. The honest answer for Jordan: “On average, people who started where you did ended up less vengeful when they were assigned to the workbook. But the model can’t tell whether you would score lower with the workbook than you would without it. The wide prediction intervals show that individual outcomes vary a great deal in either arm.” (Each person is observed in only one arm, so the two PIs describe the spread of scores for new people in each arm, not how much the workbook would change any one person’s score.)
  4. They run below 1, to 0.51 and 0.16, on a scale where 1 is the lowest possible score. The PI assumes Normal scatter around the prediction, and revenge piles up at the floor (Meet revenge, last week). The Module’s benevolence PI ran past the top of its scale; revenge runs past the bottom. M12 checks what the floor does to the model’s assumptions.

Part 3 · What does adding site change?

Site was REACH’s randomization stratum: people were randomized within each site. As in the Module’s Part 4, we add it to absorb the between-site differences in revenge you saw last week.

What moved, and what didn’t

Quantity Treat only + baseline + baseline + site
Treatment coefficient −0.359 −0.349 −0.349
Its SE 0.0275 0.0225 0.0219
R² .041 .361 .395

The treatment estimate barely moves at either step: within-site randomization balanced the arms on site, too. The biggest precision gain came from baseline; adding site changes the SE very little.

Does site add information? The nested-model test says yes: F(5, 3,956) = 44.88, p < .001. But ΔR² = .395 − .361 = .034: beyond baseline and treatment, site accounts for about 3.4 more percentage points of the variation in follow-up revenge. As the Module says, that test isn’t why site is in the model. Site is there because it’s how the trial was randomized.

Why do the site coefficients move?

The treatment coefficient barely changed when baseline entered. The site coefficients are a different story. Compare them with and without baseline in the model:

Pair-and-share · why does Hong Kong’s gap shrink?

Without baseline, Hong Kong’s follow-up revenge sits 1.07 points above Colombia’s. With baseline in the model, the gap is 0.56. Last week, Hong Kong had the highest baseline revenge of the six sites. Why does adding baseline shrink its gap, when it barely touched the treatment coefficient?

Part of Hong Kong’s higher follow-up revenge is carried over from where its participants started. Once baseline is in the model, the Hong Kong coefficient compares people with the same baseline revenge, and that carried-over part is credited to baseline instead. In this table, all five site gaps shrink toward zero once baseline enters. Treatment is different: in this sample the randomized arms were nearly balanced at baseline, so baseline has almost nothing to carry. A coefficient changes when a predictor enters because the comparison it makes changes.

The result to report

Among participants with follow-up data, those randomly assigned to the workbook scored an estimated 0.35 points lower on follow-up revenge than those on the waitlist, adjusting for baseline revenge and site, 95% CI [−0.39, −0.31], t(3,956) = −15.93, p < .001.

The estimand leads; the baseline and site coefficients stay in the table.


Part 4 · What if we adjust for something the workbook changed?

So far we’ve adjusted for two things: baseline revenge, measured before anyone was randomized, and site, fixed by the design. The Module’s Part 6 warns against adjusting for variables measured after treatment, with a hypothetical empathy measure as its example. REACH has a real one: benevolence at the two-week follow-up, measured at the same time as follow-up revenge. The workbook moved it too: adjusting for baseline revenge and site, the workbook arm’s follow-up benevolence was about 0.51 points higher.

Pair-and-share · predict first

Add follow-up benevolence to Part 3’s model. Will the workbook’s estimated effect on revenge get bigger, smaller, or stay about the same? Why?

The estimate shrinks, from −0.35 to −0.23, about 33% smaller. Nothing about the trial changed; only the comparison did. With follow-up benevolence in the model, the treatment coefficient compares workbook and waitlist participants who ended up with the same benevolence. Follow-up benevolence is something the workbook itself changed, so holding it fixed causes two problems. First, if the workbook lowers revenge partly by raising benevolence, holding benevolence fixed blocks that pathway and takes part of the workbook’s effect away. Second, it compares groups that are no longer randomized: waitlist participants who reached a given level of benevolence without the workbook may differ in other ways from workbook participants at that level. Either way, the new coefficient isn’t the workbook’s total effect. It’s still a well-defined adjusted comparison, but we can’t automatically read it as a causal “direct effect” either. Questions about how the workbook works (mediation) need their own designs and assumptions, which more advanced courses take up.

The sentence of the day

For a treatment’s total effect, adjust for well-chosen pre-treatment variables, not for variables the treatment itself could change.

Baseline revenge and site were fixed before anyone was assigned, so adjusting for them leaves the randomized comparison intact. Randomization is what gives Part 3’s result its causal basis; the adjustment makes it more precise. (“Well-chosen” matters: being measured before randomization makes a variable safe to adjust for, not automatically worth adjusting for. The Module’s model-building guide covers how to choose.) Follow-up benevolence came after assignment, so adjusting for it changes the question. The same logic covers dropping people for something that happened after assignment, such as not using the workbook. The box below shows how badly that can go, in a simulated study.

Optional, and not on the exam. This simulated study shows the same logic as Part 4 from another angle: what happens when a randomized study drops the people who didn’t use the treatment, a mistake that adjusting for baseline can’t fix.

Last week’s researcher took the lesson to heart. His new study is randomized, just like REACH: half his participants get the workbook now, half wait two weeks. At follow-up, he asks the workbook arm one extra question: did you actually use it? About a third say no.

He reasons: “Those people never received the treatment, so they can’t tell me what the workbook does.” He drops them, and compares the workbook arm’s users with the whole waitlist. Analyses that keep only the people who followed the plan are often called per-protocol analyses.

Pair-and-share · predict first

  1. Compared with analyzing everyone as randomized, will his estimate of the workbook’s effect be bigger, smaller, or about the same?
  2. In what ways might the people who skipped the workbook differ from the people who used it?

Why we simulate

Our REACH file doesn’t record who actually used the workbook, so we can’t run his analysis on REACH. Instead we’ll build a simulated trial in which we write the truth, so we can check each analysis against it. (No REACH participants are in this part; every person below is simulated.)

Here is the truth we write:

  • Each person has an engagement level: how actively they’re working on the hurt. Nobody measures it.
  • More engaged people are more likely to use the workbook if they’re given it. About two-thirds would.
  • Engagement lowers revenge on its own: engaged people would end up a little less vengeful even without the workbook.
  • Using the workbook lowers revenge by exactly 0.50 points. This is the true effect of using it.

Two pieces of code may be new. plogis() turns any number into a probability between 0 and 1, so higher engagement means a higher chance of using the workbook. rbinom() with size = 1 flips one weighted coin per person with that probability.

The 0.65 and 0.62 in the t2 line are last week’s waitlist regression line, rounded, and baseline revenge gets REACH’s mean and SD, about 2 and 1. The other numbers (0.85, 1.2, 0.35, 0.70) are choices we made so that about two-thirds use the workbook and engagement matters a little. (To keep the simulation simple, its scores aren’t held between 1 and 5.)

Run his analysis

What would he conclude?

Analyzed as randomized, the workbook lowered revenge by 0.33 points. Dropping the 34% of the workbook arm who didn’t use it, he gets 0.62, nearly twice as large. What would he write in his abstract? And which number is right, given that we wrote the true effect of using the workbook as 0.50?

Neither number is 0.50. The as-randomized estimate is smaller and his is larger, and they’re off for very different reasons.

The as-randomized estimate answers a different question: what happens when you offer people the workbook. Only about two-thirds use it, so the average effect of being offered it is 0.50 × 66% ≈ 0.33. That’s a fair answer to a real question (should a clinic hand out this workbook?), and it’s the one randomization protects. It’s called the intention-to-treat (ITT) estimate, and the Module’s footnote on ITT describes it.

His estimate overshoots the 0.50. To see why, look at who is in each group he compares.

The people he can’t see

His users averaged 0.36 on engagement; the waitlist averaged −0.01. He didn’t compare similar people: he compared a more engaged two-thirds of the workbook arm with all of the waitlist.

The waitlist has non-users too. About a third of the waitlist would also have skipped the workbook if they’d been given it. He can’t see them, because they never had a workbook to skip, so all of them stay in his comparison group. The exclusion happens in one arm only, and on a waitlist there is no way to make the matching exclusion.

Where his extra points come from. Across many trials, users average about 0.32 SD more engaged than the waitlist as a whole (0.37 in this one), and engagement lowers revenge by 0.35 points per SD. So the users would have ended about 0.35 × 0.32 ≈ 0.11 points lower even without the workbook. His analysis credits that to the workbook: −0.50 − 0.11 ≈ −0.61.

A simulation can make the fair comparison directly. In a simulation we know would_use for everyone, so we can compare users with the waitlist’s would-be users, like with like:

That gives −0.51, close to the true −0.50. A real study can’t run this chunk: would_use is never observed on the waitlist.

Can adjusting for baseline fix it?

Pair-and-share · predict first

Today’s whole Module is about adjusting. Suppose he adds baseline revenge to his model, lm(t2 ~ t1 + treat), on the same users-only data. Will his estimate move toward −0.50? Decide with your neighbor, then run the chunk.

It stays at −0.619. Adjusting removes differences in the variables you put in the model, and only those. Here his users and the waitlist started at about the same baseline revenge; what differs is engagement, which nobody measured. Adjusting for baseline makes his estimate more precise, as Part 2 showed, but it stays off target. (In real data, engagement might be related to baseline revenge, and then adjusting for baseline would remove the part of the difference that baseline captures. It can’t remove the rest.)

This is the Module’s Part 6 warning in a new form. Whether someone used the workbook was decided after randomization, and it can be observed in one arm only. Selecting on it compares groups that chance no longer made comparable.

When is dropping non-users harmless?

Suppose instead that non-use had nothing to do with engagement: say, about 30% of the download links broke at random. Swap the would_use line for a plain 70% chance and rerun his analysis:

Now his analysis lands at −0.50, right on the true −0.50. In this simulation, dropping non-users becomes harmless when use is random with respect to everything that drives the outcome. In real studies that’s a strong assumption, and he has no way to check it from his data: the reason here was engagement, and engagement was never measured.

So what should he report?

Pair-and-share · his write-up

He still wants to know what the workbook does for the people who actually use it. What should he report, and is his question a bad one?

His question is a good one; his analysis can’t answer it. Start from the estimand, as the Module says:

  • “What does offering the workbook do?” The as-randomized (ITT) estimate answers it, and randomization protects it. This is the headline most trials report, and it’s what REACH reported: its authors analyzed everyone as randomized and used multiple imputation for missing follow-ups.
  • “What does using it do, for people who would use it?” This is a different estimand. Estimating it from a real trial takes methods beyond this course, built on extra assumptions. Dropping non-users from one arm doesn’t estimate it.

He can report the ITT estimate, and report how many people used the workbook alongside it.

One more connection: the M10 lecture noted that more of REACH’s workbook arm missed the follow-up (16.7% against 10.9%). Our complete-case analysis drops those people, which is a milder version of the same problem, and that’s why REACH’s authors used multiple imputation instead.

The takeaway

Randomization makes the arms comparable, and dropping people from one arm for something that happened after randomization breaks that. Adjusting for what you measured can’t repair a difference in what you didn’t measure.

The same applies whenever an exclusion depends on what happened after assignment: dropping students who skipped tutoring sessions, patients who stopped taking a pill, or app users who never logged in.

It happens in lab experiments too. Many psychology experiments drop participants who fail an attention check or don’t follow the instructions. Everyone takes the same check, so this can look fair. But if the check comes after the manipulation, the condition itself can change who fails it: a long, dull, or upsetting condition may lose more of its less attentive participants than a short, easy one. The people left in each condition then differ in ways chance didn’t create, just as his users and the waitlist did.

Two clean options. If attentiveness is genuinely an eligibility criterion, measure it before randomizing (for example, with an attention check placed before the manipulation), exclude on that, and randomize only the people who remain. Nothing that happens in either condition can change who was excluded, so randomization keeps the conditions comparable. The trade-off is the estimand: the result now describes attentive participants, so the write-up should say so. Otherwise, keep everyone who was randomized in the primary analysis, rather than excluding people because of something that happened after assignment.

A bigger trial makes every estimate more precise. Does it bring his estimate closer to −0.50? Run it, then try n <- 40000, ten times larger.

Even at 4,000 people his CI, about [−0.69, −0.55], already misses the true −0.50. At 40,000 it narrows to about [−0.64, −0.59], still centered near −0.62. More data shrink the uncertainty around whatever an analysis estimates; they don’t change what it estimates.


Wrapping up

What today covered, with revenge

The Module’s idea Today’s revenge result
Residualized gain previews the MLR coefficient Residual gap −0.349; the MLR’s treatment coefficient −0.349
Partial effects: each slope compares people who match on the other predictors Same baseline, different arm: −0.35 points
Adjusting in an RCT sharpens the estimate without changing it Estimate −0.359 → −0.349; SE down about 18%
Adjusted means, CI vs. PI 1.89 and 1.54; the PIs run below the scale’s floor
ΔR² and the nested F-test Site adds ΔR² = .034, F(5, 3,956) = 44.88
Coefficients move when the comparison changes Hong Kong’s gap 1.07 → 0.56 once baseline enters
Variables measured after treatment Adding follow-up benevolence: −0.35 → −0.23, about 33% smaller

What’s next.

Next Monday, revenge comes back one more time. You’ll check the assumptions behind today’s model, and the floor you saw in the PIs is one of the first things you’ll look at.