The same clinician, twice: a paired look at Study 1
Self-paced bonus activity · what a within-subjects design buys you, and what it costs
Welcome to a self-paced bonus activity. In the M09 lab, Group A ran a two-sample Welch’s t on Study 1 and found that clinicians shown standard-error bars estimated a much higher probability of superiority than clinicians shown standard-deviation bars.
That analysis used one estimate per clinician. But Study 1 was a within-subjects design — every clinician eventually saw both figures and gave two estimates. Here you’ll run the paired analysis the lab set aside, and find out two things that are genuinely surprising:
- Pairing does buy precision here — but only through one of the two routes it normally works by. The other route is missing entirely, and you can see why.
- The size of the within-person effect depends on presentation order, which you can test directly. That is a sequence effect, and it is why the first-estimate comparison is the cleaner one.
Everything happens in the sandbox chunks on this page — nothing here goes into your lab notebook. The blood-pressure subset (paired_bp) is already loaded, and tidyverse, infer, and effectsize are attached.
The data, in wide form
The lab’s file kept each clinician’s first answer. This one keeps both, one row per clinician:
Two columns worth reading carefully. psup_se is the estimate a clinician gave for the SE figure and psup_sd the estimate for the SD figure — whenever in the session each appeared. The separate first_condition column records which came first. So for a clinician who saw SDs first, psup_sd is their first answer and psup_se their second.
Your task: compute the within-person difference and look at its distribution. The difference we want is psup_se - psup_sd, so that a positive value means this clinician gave a higher estimate for the SE figure.
Positive should mean “higher for SE,” so psup_se goes first.
The average clinician’s estimate dropped by more than 20 points when the same result was drawn with SD bars instead of SE bars. Same person, same trial, the same underlying result — only the picture changed.
How lopsided is it?
A mean can hide a lot. Count the directions:
Out of 75 clinicians, the overwhelming majority moved the same way and only a handful moved against the effect. That is a stronger statement than the mean alone can make — it shows the positive average is not produced by a small minority moving a long way. Note what it does not settle: the magnitude of the mean difference can still be influenced by how far individual clinicians moved, and counting directions deliberately ignores distance.
The paired test
The Module’s Test 5 makes the key move: a paired t-test is a one-sample t-test on the difference scores, tested against zero.
Your task: two blanks — the response is the diff column you just built, and the null value is 0.
Base R’s t.test() will do the subtraction for you if you hand it both columns and set paired = TRUE — same answer, less bookkeeping:
And the standardized effect for a paired design is Cohen’s \(d_z\) — the mean difference over the SD of the differences:
Surprise 1 — the usual covariance advantage is absent
Here is where this dataset stops behaving like a textbook example — though not in the way you might expect.
A paired design can improve precision in two separate ways.
- Every participant contributes under both conditions. All 75 clinicians give you a difference score. The between-groups comparison, by contrast, splits those same 75 people into two groups of roughly 37 and compares their averages.
- Positive within-person correlation cancels stable individual differences. Whatever makes a given clinician generally optimistic sits in both of their answers and subtracts away. This is the effect textbook examples usually showcase, because they usually feature strongly correlated repeated measures.
Those are independent. Route 1 depends only on the design; route 2 depends on the correlation between the two measurements. Check the correlation here:
The correlation is essentially zero, so route 2 contributes nothing: knowing a clinician’s SE answer tells you almost nothing about their SD answer.
Notice that sd_diff comes out larger than either raw SD. That is not evidence of anything going wrong — it is arithmetic. Since
\[ \text{Var}(X - Y) = \text{Var}(X) + \text{Var}(Y) - 2\,\text{Cov}(X, Y) \]
a correlation near zero means the covariance term vanishes and the variance of the difference is simply the sum of the two variances, which must exceed either one. Comparing sd_diff to a raw SD therefore tells you nothing about precision. Precision is about the standard error of an estimator, not the SD of a score — so compare the standard errors directly.
Almost every point sits above the diagonal — the effect is overwhelming — but the cloud has no upward tilt, which is the zero correlation you just computed.
Now compare the two standard errors
This is the comparison that actually settles whether pairing bought precision. Put the paired estimator’s SE next to the between-groups (Welch) SE for the same 75 clinicians:
Pairing did buy precision — through one route, not two
The paired standard error is about 71% of the between-groups standard error. That is a real precision gain: the paired 95% interval is roughly nine percentage points wide against thirteen for the between-groups comparison, which is why the paired t (9.42) is larger than the lab’s between-groups t (6.83).
So the headline is not that pairing failed. It is that pairing delivered route 1 and only route 1:
| Operating here? | Why | |
|---|---|---|
| Route 1 — everyone contributes under both conditions | ✅ fully | 75 difference scores instead of two groups of ~37 |
| Route 2 — positive correlation cancels individual differences | ❌ not at all | r ≈ -0.05, so nothing cancels in the subtraction |
The lesson worth keeping: “paired designs are more precise” is really two claims, and they can come apart. Textbook examples usually feature strongly correlated repeated measures, so both routes fire at once and the two get conflated. Here you can watch them separate.
One thing this is not: a rule for choosing your test. The design decides that — these are two measurements on the same clinicians, so the paired analysis is the one that matches the data structure. What the correlation explains is how much precision you ended up with, not which test was appropriate. Prior evidence about within-person correlation is useful when you’re planning a study and want to anticipate the payoff from a repeated-measures design.
Surprise 2 — the effect depends on presentation order
Now the reason the first-estimate comparison is the cleaner one.
Every clinician answered about one figure, then about the other. If the sequence they happened to get changed how they answered, then the within-person difference is not a pure display effect — it also carries whatever the sequence did.
Order was randomized, which gives us a way to check. If sequence made no difference, the within-person contrast should be the same size for clinicians who saw SEs first as for those who saw SDs first.
Your task: one blank — group by first_condition.
Those two numbers should not be so far apart. Test it:
The within-person effect is substantially larger for clinicians who happened to see the SE figure first, and the difference between those two within-person effects is itself statistically significant.
What this establishes, stated carefully: the size of the within-person display contrast depends on presentation order. In design terms that is a sequence effect — equivalently, a display-by-order interaction — and it means a single pooled paired estimate is averaging over two genuinely different quantities.
What it does not establish is the mechanism. Any of these would produce the same signature, and this test cannot tell them apart:
- Anchoring — the first answer constrains the second.
- Carryover — seeing one figure format changes how the next one is read.
- A period effect — second answers differ systematically from first answers for reasons unrelated to format (fatigue, practice, a shifted sense of the scale).
- Ceiling effects — the scale tops out at 100, so clinicians who started high have less room to move.
Separating those needs a model with period and carryover terms, which is beyond this course. It is enough — and more honest — to say the contrast varies by sequence and name the candidates.
What this explains about the paper
The first-estimate comparison sidesteps all of this: it compares randomized groups before either clinician has seen the alternative figure, so there is no sequence for order to act through. That is what makes it clean.
Zhang et al. do lead with the first-estimate comparison, and report the within-subjects analysis only in their SI Appendix, noting it shows “a similar pattern.” Our order result is consistent with that choice — but the paper does not state this as its reason, so treat the connection as a reasonable interpretation rather than something the authors said.
Both analyses are defensible, and they answer different questions. The paired one carries a caveat the between-groups one does not:
| Between-subjects (the lab) | Paired (this page) | |
|---|---|---|
| Question | Do two groups of clinicians differ? | Does the same clinician answer differently? |
| Uses | first estimates only | both estimates |
| Clean? | yes — no prior figure could have influenced the answer | the contrast varies by sequence (shown above) |
| The paper | headline result | SI Appendix |
The transferable habit: when a design gives you more than one way to analyze it, the right choice is rarely “whichever gives the bigger t.” It is whichever question you can answer most cleanly — and saying out loud what the other analysis showed, rather than quietly reporting only one.
Try it on the other scenario
Study 1 randomized clinicians to a blood-pressure or a COVID-19 trial. Everything above used the blood-pressure arm. Rerun it on the other one and see whether the same two surprises hold:
The effect reproduces in the other arm. Whether the sequence effect does is worth checking for yourself — and is a good reminder that a pattern found in one arm of a study is a hypothesis about the other, not a finding.
Back to the M09 lab → return to the lab