Reproducing the Illusion of Predictability

Lab · Module 9 · Wed Oct 14

Welcome to the Module 9 lab

This lab lands at the end of hypothesis-testing week. The M09 Module and Monday’s lecture walked one logic across five designs; today you put one of them — the two-sample Welch’s t — to work on real published data.

In research-methods terms, what you’re about to do is a computational reproduction: you take a published paper’s own data and re-run its analysis to see whether the reported numbers come back out. That is a different thing from a direct replication, which collects a new sample using the original procedure to see whether the finding recurs. Reproduction asks “can this reported result be recovered from these data and these analytic procedures?”; replication asks “does the finding recur when new data are collected under the same or a closely related design?” Both matter, and they fail for different reasons — a result can reproduce perfectly and still not replicate, or fail to reproduce over something as mundane as an undocumented exclusion.

Reproduction is the one you can do in a single class session, which is why it’s today’s lab. Project 2 is a reproduction too — the same move at full scale, and mostly on your own: you find the paper, procure the data, rebuild the analytic sample, and defend what you got. Today the paper, the data, and the analytic decisions are handed to you. In Project 2 all three are yours.

We’ll use data from An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability (Zhang et al., 2023, PNAS) — open access, and a copy is in the course readings folder as zhang_2023.pdf. The authors publish their data and analysis code at github.com/jhofman/illusion-of-predictability, which is where our three course datasets come from. It is co-authored by the same Dr. Jake Hofman behind the boulder-sliding study you’ve worked with all Module — and you already met its headline numbers in the M07 Module’s “illusion of predictability” section. Same core question — do experts confuse a chart’s inferential uncertainty with its outcome variability? — carried from a fictional game to clinicians, data scientists, and tenure-track faculty. Today you reproduce those numbers yourself.

You’ll run the same focal test on response data generated by three groups of experts — they saw the stimulus figures; these are the judgments they gave — line your numbers up against the paper’s, and decide whether the finding holds. This is exactly the workflow Project 2 asks of you in a few weeks — today you rehearse it in a single lab session, as a class.

Note

Two papers this week — keep them straight. The Module and pre-study work with Hofman et al. (2020), the boulder-sliding paper, which has two experiments we refer to as Experiment 1 (2 × 2: interval format × caption text) and Experiment 2 (4 × 2: four visualizations × two effect sizes).

Today’s paper is a different one — Zhang et al. (2023) — and it reports three studies on three expert populations. To avoid a collision of numbers, we call those Study 1, Study 2, and Study 3 throughout this lab. Different paper, different participants, same underlying question.

What you’ll leave with

A reproduced Welch’s t-test and Cohen’s d for one expert-audience study, run with infer and effectsize
A picture of your study’s results showing error bars and individual points — built to the standard the paper argues for
An estimation plot putting the effect and its uncertainty on their own axis — a strong contemporary way to present a two-group comparison, and a figure you can reuse for Project 2
A completed APA result line, your numbers next to the paper’s, and a reproduction verdict
A shared 3-row reproduction checklist for the whole paper, built with the class

This is a jigsaw lab: your group reproduces one study, then we reconvene so all three fit together. Each analysis step has a ✍️ Your Code tab, a 💡 Hint, and a 👀 Spoiler. As in the M06–M08 labs, the ✍️ box starts empty — a Targets list above it tells you what to build. Two of the figure steps are different: they hand you the plot’s scales and labels and ask you to add only the layers that carry the statistics. If you have been stuck for more than a few minutes, open 💡, then 👀, and keep moving — your group is waiting on your numbers for the share-back.

Before you start — set up your notebook

Like M07 and M08, there is no per-lab template — you copy lab_template.qmd and build the structure yourself. Third rep, so you know the two surfaces by now: this page’s sandboxes are the scratchpad where your group gets the code working; your own Quarto notebook is the individually-submitted deliverable. Step 0 below scaffolds it in a few minutes. The three illusion datasets are already in your project’s data/ folder, each with a codebook in documentation/ — nothing to download.

How the time works. Your group’s reproduction (Steps 1–3), the share-back (Step 4), and the debrief all happen in class; the write-up — assembling your notebook, the interpretation prose, the final render, and the Canvas submission — is yours to finish at home. Each step ends with an In your notebook note saying exactly what to add.

Carrying code into your notebook — the same rules as M06

The code boxes on this page run in the browser; your notebook runs on your copy of R. So anything you carry across has to sit inside an R chunk you insert yourself, with a label on a #| line. If code lands outside a chunk it renders as plain text and never runs.

The M06 lab’s Step 0 walks through the keystrokes for inserting and labelling a chunk, and what goes wrong when code lands outside one — worth a look if any of that is hazy.

Step 0 · Get set up

Fourth time through — this should take only a few minutes now. (The M06 lab’s Step 0 has the line-by-line explanations if you need them.)

Start in GitHub Desktop, before RStudio. Select PSY652_project, click Fetch origin, and click Pull origin if it appears. That is the pull half of the pull → edit → commit → push loop — the same loop you’ll close at the end of today’s lab.
Open RStudio by double-clicking PSY652_project.Rproj, then in the Files pane open programs/lab_template.qmd.
File → Save As… a copy inside programs/ named m09_lab.qmd — the same move as M06 through M08.
Set the YAML. The template’s header is already there. Change the title, and bump toc-depth: 3 to toc-depth: 5 so this report’s ## subsections reach the table of contents. Leave the rest alone.
```
title: "Reproducing Zhang et al. (2023): The Illusion of Predictability"
toc-depth: 5
```

Give the notebook this section structure — the skeleton you fill in as the lab proceeds:

# Introduction
# Data
# Results
## Picture of the data
## The effect, drawn
## Welch's t-test and effect size
## APA result and reproduction verdict
# Discussion

Add the setup chunk — packages only, right after the YAML. Give it the label: setup option so you can find it later:

```{r}
#| label: setup

library(tidyverse)
library(infer)
library(effectsize)
library(gt)
```

Then import the data in its own chunk, under the # Data heading. Keeping packages and data apart is the pattern in lab_template.qmd, and it earns its keep the first time something breaks: an error in the setup chunk means a package isn’t installed, an error in the import chunk means a file path is wrong. One glance tells you which, instead of hunting through a chunk that does both.

```{r}
#| label: import-data

# Medical providers -- headline test is the blood-pressure scenario only
medical_bp <- read_rds(here::here("data", "illusion_medical.Rds")) |>
  filter(scenario == "Blood pressure scenario") |>
  mutate(condition = fct_relevel(condition, "Saw SDs first", "Saw SEs first"))

# Data scientists
datasci_analysis <- read_rds(here::here("data", "illusion_datasci.Rds")) |>
  mutate(condition = fct_relevel(condition, "SE + Points", "SE Only"))

# Faculty
faculty_analysis <- read_rds(here::here("data", "illusion_faculty.Rds")) |>
  mutate(condition = fct_relevel(condition, "SE + Points", "SE Only"))
```

All three files load even though your report analyzes only your group’s — that keeps the share-back comparisons reproducible. You may drop the two you didn’t use once the share-back is done.

Step 1 · The paper in three minutes

Zhang et al. (2023) · An illusion of predictability in scientific results

Citation. Zhang, S., Heck, P. R., Meyer, M. N., Chabris, C. F., Goldstein, D. G., & Hofman, J. M. (2023). An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability. Proceedings of the National Academy of Sciences, 120(33), e2302491120.

The central claim. When a chart displays inferential uncertainty — for example standard-error bars or confidence intervals around group means — but hides outcome variability — for example standard deviations, prediction intervals, or the individual observations themselves — viewers may overestimate how predictable the result is for an individual. Zhang and colleagues found this pattern even among highly trained experts. Across the three studies, showing outcome variability substantially improved the accuracy of participants’ judgments. The authors therefore recommend plotting individual outcomes alongside statistical estimates when possible.

The design pattern—one core contrast, three expert populations. All three studies examine the same underlying question and share one focal outcome measure that we reproduce in this lab. They differ, however, in how the visualization contrast was created, and that distinction matters.

1. One focal question, shared across all three studies

Each participant saw results from a real scientific study and estimated the probability of superiority — if one person were selected from the treatment group and one from the comparison group, how likely would it be that the treated person had the better outcome? We call this quantity PSup.

Participants responded on a 50–100 scale. The lower bound of 50 is intentional: a PSup of 50% means that a randomly selected treated outcome is no more likely to be better than a randomly selected comparison outcome than the reverse. Values below 50 would indicate an advantage for the comparison group, which the response scale did not permit. The scale therefore runs from “no advantage either way” to “a sure thing.” Higher responses indicate that the participant believes the treatment outcome is more predictable for an individual.

PSup was not the only outcome collected in the studies, but it provides a common measure that can be examined in the same way across all three expert populations.

2. One visual feature changed

Within each experimental comparison, the underlying scientific result was held fixed while its visual presentation changed.

Study 1—Medical Providers. Clinicians saw either group means with standard-error bars, emphasizing inferential uncertainty, or group means with standard-deviation bars, emphasizing variation among individual outcomes.
Studies 2 and 3—Data Scientists and Faculty. Participants saw either group means with standard-error bars only or the same means and error bars with the individual observations added. The second display therefore showed inferential uncertainty and outcome variability together.

Only the visual representation changed; the underlying study results did not.

How the contrast was implemented also differed across the studies:

Studies 2 and 3 used straightforward between-subjects designs. Each data scientist and each faculty member was randomly assigned to one condition — SE Only or SE + Points — and saw only that version.
Study 1 used a within-subjects design with randomized order. Every medical provider eventually saw both figures. What was randomized was which figure appeared first. After responding to the first figure, participants were shown “another scenario” that was actually the same underlying result displayed with the other type of error bar.

That is why the Study 1 condition is named Saw SEs first rather than simply Saw SEs. The focal outcome for the paper’s between-groups comparison is each clinician’s first estimate, recorded before the clinician had seen the alternative display. Comparing those first responses produces a clean randomized comparison between two independent groups.

Study 1 also randomized whether a clinician saw a blood-pressure or a COVID-19 medication scenario. The paper reports results for both scenarios in its main text. Group A reproduces the blood-pressure comparison.

3. One focal comparison per study—for this lab

The complete paper contains additional outcomes and analyses. For this lab, we isolate one common comparison from each study:

Does mean PSup differ between the two visualization conditions?

For each focal comparison, we therefore have:

two independently randomized groups;
one continuous outcome;
and a question about the difference between two population means.

The M09 decision map points to a two-sample Welch’s t-test.

So, for this lab: three expert populations, one shared outcome, one focal comparison, and one test applied three times. If the authors’ account is correct, participants shown inferential uncertainty without visible outcome variability — SE bars alone — should report higher PSup estimates than participants shown SD bars or individual outcomes. In other words, they should judge individual outcomes as more predictable than participants who can see how widely those outcomes actually vary.

Study 1’s within-subjects structure also permits a paired analysis: for each clinician, subtract the PSup estimate given for the SD figure from the estimate given for the SE figure, then test whether the mean within-person difference equals zero.

That analysis asks a different question from today’s focal comparison:

Between-groups analysis: Do clinicians who saw SE bars first differ from clinicians who saw SD bars first?
Paired analysis: Does the same clinician respond differently to the two displays?

We reproduce the randomized first-response comparison reported in the paper’s main results, because it is the cleaner estimate of the visualization effect — the paired version runs into two complications this one avoids. (An optional activity at the end of this lab works through the paired analysis and both complications, if you want to see why.)

The three studies you will reproduce:

Study	Population	Visualization contrast	N	Published focal test
Study 1—Medical Providers	clinicians	SE bars versus SD bars, blood-pressure medication scenario	75	t(65.1) = 6.83, p < .001
Study 2—Data Scientists	professional data scientists at a software company	SE-only bars versus SE bars plus individual points	175	t(159.4) = 6.34, p < .001
Study 3—Faculty	tenure-track academics	SE-only bars versus SE bars plus individual points	368	t(363.9) = 4.52, p < .001

The N column is computed from the course files you are about to load. The final column is quoted from the published paper. Those are different kinds of numbers: one comes from the dataset in front of you; the other is the published result you are attempting to reproduce.

How close should your numbers come?

Our course datasets were prepared from the authors’ public repository, and we run the same general analysis they report: R’s default two-sided Welch t-test. Participants with missing PSup estimates were removed when the course files were created, so the N shown in each file is the N analyzed in the lab.

Two of the three analyses should reproduce the published result to the reported decimal:

Study 1—Medical Providers. Expect t(65.1) = 6.83. This should match exactly. Group A analyzes the blood-pressure scenario only, one of the two scenario-specific comparisons reported in the paper’s main text.
Study 3—Faculty. Expect t(363.9) = 4.52. This should also match exactly.
Study 2—Data Scientists. Expect approximately t(159.2) = 6.41, compared with the paper’s t(159.4) = 6.34. The results should be close, but not identical.

Why Study 2 differs. The published analysis applied a preregistered eligibility exclusion based on participants’ reported experience with statistics or scientific literature. Those eligibility variables are stored in a separate background file keyed to participant IDs. Our simplified course dataset does not include that file or implement the corresponding join, so the lab analyzes everyone in the course file who provided a usable PSup estimate.

In the published analysis that exclusion removed exactly one participant, and that single case is the entire gap between 6.41 and 6.34. (Why one, when the paper’s Methods say two? That question has an interesting answer — see the box below.)

That preprocessing difference does not change the direction, magnitude, or substantive conclusion of the comparison.

This is a realistic computational-reproduction outcome rather than a defect. Published statistics depend not only on the final test function, but also on the decisions that determine the analytic sample. A report that says,

“Our statistic was 6.41 rather than 6.34 because our course file retains one participant the published analysis excluded,”

is more scientifically informative than a number that matches for reasons you cannot explain. Group B should state that difference explicitly in its reproduction verdict. In Project 2, you will be responsible for reconstructing and documenting analytic-sample decisions like this yourself.

When the prose and the code disagree

Here is what a computational reproduction check is actually for, and Study 2 gives us a remarkably clean example. In short: the paper’s Methods say two participants were removed, but running the authors’ own published script removes one — and the reason is a silent R type coercion that no amount of re-reading the Methods could reveal. That single participant is the whole 6.41-vs-6.34 gap.

The full walk-through is below. It is optional — Group B needs only the one-participant fact above — but it is the clearest example in this course of why code is part of the scientific record.

Show me the code, and how one becomes two

The paper’s Methods report that the authors removed two participants. But the published analysis script builds the exclusion object like this — quoted from analysis/shared_analysis.R, de-indented (it sits inside a function) and still in the older %>% pipe rather than the native |> this course uses:

assignments_to_drop1 = background_ds %>%
  filter((has_any_stats_training == 0) | (rctComfort == 1)) %>%
  select(assignmentId)

assignments_to_drop = c(
  background_with_number_done %>%
    filter(number_done == 0) %>% select(assignmentId),
  assignments_to_drop1)

(background_with_number_done is the background survey file with one column added: a per-person tally of how many experience categories they reported, so number_done == 0 finds anyone who reported none.)

Between them the two filters identify three unique participant IDs. The first identifies one participant; assignments_to_drop1 identifies three, including that same participant.

The problem is the use of c(). Each input is a one-column data frame, and c() does not stack their rows into one vector of IDs. Instead it creates a list with two elements:

one element containing a single ID;
one element containing a vector of three IDs.

That structure produces two subtle consequences.

First, the script counts the number excluded with:

background_numbers["num_dropped"] = length(assignments_to_drop)

The length of this object is 2 — because it holds two list elements, not because it holds two participant IDs. That count is saved to results/das_results.Rdata, the file the manuscript reads its numbers from, which appears to explain the figure reported in the paper’s Methods.

Second, the analysis later filters participants with:

filter(!(assignmentId %in% assignments_to_drop))

The first list element holds one bare ID, so that participant is matched and removed. The second holds three IDs bundled as a single vector, which coerces to the literal text c("id1", "id2", "id3") and matches no individual assignmentId. The published code therefore removes one participant, not the three the filters identify.

The numbers confirm it:

Analytic sample	n	Welch test
No participants removed — our course file	175	t(159.2) = 6.41
Published code — one participant removed	174	t(159.4) = 6.34
All three flagged participants removed	172	t(155.3) = 6.26

The middle row is the statistic printed in the paper. The entire difference between our result and the published one is therefore a single participant — and a quiet data-structure error that becomes visible only when the code is executed and its intermediate objects are inspected.

The intended construction would be something like:

assignments_to_drop <- bind_rows(
  background_with_number_done |>
    filter(number_done == 0) |>
    select(assignmentId),
  assignments_to_drop1
) |>
  distinct(assignmentId) |>
  pull(assignmentId)

Two things to take from this:

The discrepancy does not threaten the study’s conclusion. Whether zero, one, or all three flagged participants are removed, the estimated difference stays large and the evidence against equal group means stays overwhelming. The preprocessing error moves the reported statistic slightly; it does not move the finding.
Code is part of the scientific record. A Methods section describes the analysis the researchers intended to run; executable code reveals the analysis the computer actually ran. A strong reproduction does not merely ask whether the final number matches — it traces any difference back through the data-processing decisions that produced it.

When you complete Project 2, write your code as though another researcher will run it — because reproducible science assumes that eventually someone will.

Your assignment

We’re splitting into three groups, one per study:

Group A → Medical Providers (Study 1)
Group B → Data Scientists (Study 2)
Group C → Faculty (Study 3)

Each group works through the same six-step pipeline (a through f) on their dataset, then we reconvene to share back. Don’t peek at the other groups’ tabs until then.

Checkpoint 1 · You can frame the reproduction

Before you touch code, you can say in one sentence each: (a) the paper’s central claim — even experts confuse a chart’s inferential uncertainty with its outcome variability; (b) what your assigned study manipulated; and (c) that every study’s headline test is a two-sample Welch’s t on a 50–100 probability-of-superiority estimate. You know which group — A, B, or C — is yours.

In your notebook — your Introduction borrows straight from this step: two or three sentences giving the paper’s question and what your group’s study manipulated.

Step 2 · Whole-class setup

We’ll load all three datasets up front so everyone sees the same starting state. Each .Rds file is the tidied analytic dataset for one study, prepared from the paper’s public-repo CSVs. The schema is shared across all three:

One decision to make before anything else: which condition counts as the baseline. A factor remembers its level order, and that order silently decides which group R treats as the reference — which in turn decides the sign of every difference you report.

In all three studies the same contrast is at stake: a group shown only error bars against a group that also saw outcome variability. We put the outcome-variability group first in every dataset, so a positive difference always means the error-bars-only group estimated higher — matching the direction of the sentence you’ll write.

Dataset	Baseline (saw outcome variability)	Order as saved
illusion_medical	`Saw SDs first`	`Saw SEs first`, `Saw SDs first`	needs flipping
illusion_datasci	`SE + Points`	`SE + Points`, `SE Only`	already first
illusion_faculty	`SE + Points`	`SE + Points`, `SE Only`	already first

Only the medical file actually needs reordering — but the code below calls fct_relevel() on all three anyway. That is deliberate. Depending on the order a file happens to have been saved in is fragile: re-export the data, change a recode upstream, and your signs flip with no error message. Stating the order you want is how you stop that from being possible.

The same chunk also filters the medical file to the blood-pressure scenario, because that is the comparison the paper’s headline number reports. After this chunk, each group’s dataset is ready to analyze — no further reshaping anywhere in the lab.

Two surfaces, two file paths. In the sandbox below, the data are mounted at shared_data/, so the path is a plain string and there is no here package. In your own notebook the same files live in your project’s data/ folder, so you write here::here("data", "illusion_medical.Rds") — exactly as in the Step 0 import chunk you already pasted. Same files, different location, so the path differs. If you paste one version into the other, R will tell you it can’t find the file.

illusion_medical / illusion_datasci / illusion_faculty · 163 / 175 / 368 observations · shared schema · Zhang et al. (2023)

Three tidied analytic datasets — one per expert population — prepared from the paper’s public-repo CSVs (the medical file spans both scenarios; its headline test uses the 75 blood-pressure rows). They share a single analytic schema:

participant_id character — anonymized worker ID
condition factor — the chart the participant was shown (levels differ per study; see your tab)
psup_estimate numeric — the participant’s estimate of probability-of-superiority, on a 50–100 scale
scenario factor — medical only: which medication scenario the participant was assigned (blood-pressure vs COVID-19)
study character — constant within each dataset (medical / datasci / faculty)

Before you test: what a t-test actually assumes

psup_estimate is bounded — it cannot go below 50 or above 100 — and once you plot it you’ll see responses stacked against those edges. So the individual scores are not Normally distributed, and they never could be.

That is fine, and it is worth being precise about why. The classical t-test is exact under a Normal-population model, but it is also a good large-sample approximation well beyond that model. What the approximation actually needs is that the sampling distribution of the mean difference be well behaved — and by the Central Limit Theorem you met in M07, that holds at reasonable sample sizes even when the raw values are lopsided or bounded.

The size that matters is the per-group size, not the total. Study 1 has 75 providers in the blood-pressure arm, but they split into two independent groups of roughly 34 and 41 — so each mean rests on about forty observations, not seventy-five. Studies 2 and 3 are more comfortable still (81/94 and 169/199). Those group sizes are moderate to large, the outcome is bounded so it cannot produce wild outliers, and the plots you’re about to draw show no isolated values capable of dragging a group mean on their own. On those three grounds Welch’s large-sample approximation is reasonable here.

What the test does still lean on is independence, and that comes from the design: each participant contributed one estimate and was randomly assigned to one condition.

Welch’s t adds one more piece of freedom — it does not assume the two groups share a variance, which is why it reports fractional degrees of freedom like 65.1 rather than a whole number. The Module works through why that matters.

You have already watched this happen. In M06 you built a binomial distribution and grew n from 8 to 80 to 800, and watched a lumpy, skewed, bounded count distribution turn into a smooth bell. Nothing about the individual coin-flips changed — what changed was that a sum of many of them has room to even out. psup_estimate is in the same position: individual responses are bounded and piled against the edges, but the thing we are testing is a mean, and means of 34-plus observations behave far better than the raw values do. That M06 figure is the picture to hold in your head here.

The habit worth building: “is my outcome Normal?” is usually the wrong question. “Is the mean well estimated at this sample size, and are the observations independent?” is the right one.

Checkpoint 2 · Data loaded and understood

All three datasets load, and your glimpse() shows the shared schema — condition and psup_estimate are the two columns every test needs. You can say what one row represents — for Study 1 that is a clinician’s first estimate; for Studies 2 and 3 it is each expert’s focal Part 1 estimate.

In your notebook — this load already lives in the import chunk you added under your Data heading in Step 0 (note the path differs there: here::here("data", ...), not the sandbox’s shared_data/). Add a sentence to that Data section naming your group’s dataset and what one row represents; the glimpse() is a check you don’t need to keep.

Step 3 · Reproduce your study

How to use this part

Click the tab for your assigned study. Work through the six sub-steps a → f in order. Each substantive step has a ✍️ Code tab (your starting point) and a 👀 Spoiler tab (only peek if stuck). The Cohen’s d and APA-result steps are short — fill them in directly.

a · Meet the data

Your analytic dataset is medical_bp, built for you in Step 2. Everything you report — the counts, the plot, the descriptives, the test, the effect size — comes from that one object, so the N behind every number is provably the same. Start by looking at it:

Two things were decided in Step 2 that shape everything downstream, and they are worth being able to explain:

The condition order. fct_relevel() put “Saw SDs first” — the group that saw outcome variability — first, making it the baseline. That fixes the sign of every comparison: a positive difference now means the Saw SEs first group estimated higher, which is the direction your APA sentence will read.
The rows. Nothing is filtered out from here on. What count() prints above is the N that reaches your t-test.

You’re working with the blood-pressure scenario only — that’s the headline test in the paper, and Step 2 filtered to it for you. The conditions are “Saw SEs first” and “Saw SDs first”; each participant saw one chart type first and the other second, but the paper compares their first estimate.

b · Picture of the data

First, the numbers your APA line will need. Two of the six columns below you already know how to build; the other four are the ones worth slowing down on.

Before you write anything, before you run it, go round your group and say out loud what each of the six columns will contain and what it is for. If you can’t name what a line produces, you can’t defend the number it puts in your write-up.

Targets. Build one row per condition, each carrying a mean and its interval.

Start from medical_bp, group_by() condition, and summarize().
Build six columns: the cell size, the mean of psup_estimate, its SD, its standard error, and the two 95% bounds.
The SE of a mean is the SD divided by the square root of n — this is the quantity M07 was about, now in service of a comparison.
For the bounds, the multiplier is the t critical value for that group’s degrees of freedom, from qt() — the Hint names the exact call if you need it.
Name the columns n, M, SD, SE, lo, hi — the figure you build next refers to them by those names.
Set .groups = "drop", assign to group_descriptives, and print it.

The standard error of a mean is the SD divided by the square root of the group size — you have both as columns already, so SD / sqrt(n).

For the bounds, the multiplier is not 1.96. That value is the large-sample shortcut; the model-based multiplier comes from the t-distribution with this group’s degrees of freedom, which qt() gives you: qt(0.975, n - 1). Note 0.975, not 0.95 — a two-sided 95% interval leaves 2.5% in each tail.

👀 Check your group’s answers

Column	What it is	What it’s for
`n`	how many providers in that condition	the N in your APA line, and the df below
`M`	the group mean	the M in your APA line
`SD`	how much individual providers differ from each other	the SD in your APA line
`SE`	how precisely that group’s mean is estimated — `SD / sqrt(n)`	the width of that group’s interval. Welch’s test later combines both groups’ uncertainty into the standard error of the difference — neither group’s SE is itself the test’s denominator
`lo`, `hi`	the 95% confidence interval on that group’s mean	the error bars in the chart below

The distinction that matters: SD is about people, SE is about the estimate. They answer different questions, and only one of them shrinks as you collect more data. If your group hesitated anywhere, it is almost always here.

Now plot your study’s results.

Be clear about what this figure is. It shows what your participants estimated — one point per person, split by the condition they were assigned to. It is not a recreation of the SE-bar and SD-bar figures that Zhang et al. showed their participants. Those were the stimulus, the thing being manipulated; this is the outcome, the thing being measured. Keeping those two straight is the difference between describing a study and describing its findings.

What your figure has to show, and why — note the self-reference. The paper’s finding is that a chart showing only means and error bars leads readers to overestimate how predictable individual outcomes are. So when you plot your own results, you follow the paper’s recommendation and show the individual observations as well. A results figure with error bars alone would commit, in your write-up, precisely the error the paper documents. Yours needs three layers:

Every individual observation, jittered so overlapping points stay visible.
Each group’s mean, marked clearly enough to read above the dots.
Each group’s 95% interval, drawn from the lo/hi you just computed.

The construction detail worth thinking about: layers 2 and 3 come from group_descriptives (two rows), while layer 1 comes from medical_bp (one row per person). A single ggplot() can draw from two different data frames — you pass data = inside the geom, and add inherit.aes = FALSE so that geom ignores the plot-level aes() and uses only its own.

Targets. Three layers to add to a figure that is otherwise given.

The scales, zoom, labels and legend are already written below — that chrome is not what this step is testing. What you add is the part that carries the statistics:

The raw observations. Add the geom that scatters individual points with a little horizontal jitter, so overlapping responses stay visible. Its width/height/alpha settings are already filled in.
The interval. Point geom_errorbar() at the summary table you built in part b, and map ymin and ymax to its two bound columns.
Leave inherit.aes = FALSE alone on both summary layers — it is what stops them inheriting the color mapping from the raw points and re-splitting by condition.

Read the result before moving on: the cloud is the people, the dot is the group mean, and the bar is how well you know that mean.

For the individual points you want the jittered scatter from M03 — geom_jitter(), which nudges points sideways so ties don’t hide each other. (Plain geom_point() would stack them into a single vertical line.)

The error-bar layer reads from the summary table you built above, and geom_errorbar() wants a bottom and a top: ymin and ymax, which are exactly the two bound columns in that table.

A note on what we’re plotting, versus what the paper plots. Zhang et al.’s figures show the mean with one standard error above and below it. We draw 95% confidence intervals instead, because they answer the question a reader actually has — what range of population means is this sample consistent with? — and because it keeps this chart consistent with the interval you build in section c. So your figure is deliberately not a pixel match for theirs. Say so if you reproduce it in your write-up.

c · Draw the effect

The plot you just built shows the two groups. It does not show the thing your paper is actually about — the difference between them, and how precisely you know it. A reader is left to eyeball the gap between two error bars, which is precisely the visual judgment the paper you are reproducing shows people get wrong.

The fix is an estimation plot (Ho et al., 2019) — a strong contemporary approach for presenting a two-group comparison, and a good standard to hold your own figures to. It puts three things in one frame:

Every observation, so nobody has to take a summary on trust.
Each group’s mean with its 95% interval, on the left axis, in the outcome’s own units.
The difference itself, on its own axis at the right, with an interval showing the uncertainty around it rather than a bare point.

That third element is the point of the whole design. The effect gets its own axis, its own uncertainty, and equal visual weight, instead of being something the reader has to compute by eye from two error bars.

Two construction details worth understanding rather than copying:

sec_axis() builds the second axis. The right-hand ruler is the left-hand ruler shifted so that zero sits at the baseline group’s mean — the dashed line. It is one coordinate system with two labels, which is why the difference lines up with the group means instead of floating on an unrelated scale.

“Wait — didn’t M03 say two y-axes are bad?”

It did, and the rule is a good one. M03’s trap list classes dual y-axes as bad, on the grounds that putting two variables on different y-axes lets you manufacture any correlation you like by adjusting the scales. (The critique is an old and well-argued one — see Few, 2008 — and it is why ggplot2 makes a genuine second axis so awkward to build.)

Read that reason carefully and you’ll see this figure isn’t the thing being warned about. The problem case has two different variables and two independently chosen scales — and it’s that second freedom that does the damage, because sliding one scale against the other can make an association appear, vanish, or reverse.

Here there is one variable (probability of superiority) on one scale. The right-hand axis is not a second measurement; it is the same ruler, relabeled so that zero falls at the baseline group’s mean — which is exactly what sec_axis(~ . - baseline) says: take this axis, subtract a constant. Nothing is free to be tuned, so nothing can be manufactured. Slide the data and both axes move together.

The transferable lesson is about how to hold a design rule. “Never use two y-axes” is a shortcut for “don’t give yourself a free parameter that can invent a relationship.” When you meet a figure that breaks the surface form of a rule, check whether it also breaks the reason. Sometimes it doesn’t — and knowing why is what separates following conventions from understanding them.

The shaded curve is a bootstrap — M07’s resampling machinery, aimed at a difference rather than a single mean. Its width is the uncertainty in the effect.

Two kinds of interval in one figure

This figure deliberately mixes two procedures, and you should be able to say which is which:

The group means on the left carry parametric t-intervals — the qt(0.975, n - 1) bounds you computed in section b.
The difference on the right carries a percentile bootstrap interval — resampled, nonparametric, straight out of M07. (Nonparametric means it assumes no distributional shape — but it does still assume your observations are suitable units to resample, which is the independence you get from the design.)

Both are 95% intervals, and here they will tell the same story. They are not the same procedure, though, and they are not guaranteed to agree exactly — the bootstrap interval on the difference need not match the Welch interval your t-test reports in section d. Naming which interval came from which method is part of describing a figure honestly.

This is the one genuinely analytic decision in this section — the baseline goes first, and that choice sets the sign of every number that follows. Use the exact labels count() printed back in section a.

Targets. Name the two groups, baseline first.

Set ref_group to the condition that acts as the baseline — the one you are comparing against.
Set comp_group to the other one.
Both are the exact level labels as they appear in condition, in quotes. Get them from the table you just printed rather than from memory.

Order matters here and it will matter again in the t-test: it decides the sign of the difference, and therefore whether your effect reads as an increase or a decrease.

The baseline is the group that saw outcome variability — the condition the paper treats as the better display. The other group is the one whose estimates the paper expects to be inflated. Both strings must match the printed factor levels exactly, including capitalization and spacing.

With those two names set, the rest of the figure is bookkeeping. One piece of it deserves a word first, because it is the only code in today’s lab that no module has shown you.

The infer verbs, in their estimation form

Step 2 of the code below bootstraps the difference between the two group means, using the verb pipeline from the M08 lecture:

your_data |>
  specify(psup_estimate ~ condition) |>          # outcome ~ grouping variable
  generate(reps = 2000, type = "bootstrap") |>   # resample 2,000 times
  calculate(stat = "diff in means", order = c(comp_group, ref_group))

One verb is missing, on purpose. In lecture the pipeline had a fourth verb — hypothesize(), sitting between specify() and generate() — which imposed a null so you could build a null distribution and read a p-value off it. Here we are estimating, not testing: the question is how big the difference is and how precisely it is pinned down. Drop hypothesize() and the remaining three verbs give you a bootstrap distribution of the difference itself, centered on what your data actually show. Hand that to get_confidence_interval() and you have the interval the plot draws.

That single omission is the entire difference between the M08 lecture’s question and this one — which makes it worth remembering, because it is the difference between testing and estimating in general.

Now run it:

Print the difference and its interval so you have the numbers in front of you — these are the raw effect and its uncertainty, and they lead your APA line:

Read it back. The dashed line is the baseline group’s mean, so the right-hand axis reads zero there. The blue dot is your estimated effect; the vertical bar is its 95% interval; the curve shows which values of the difference the data support most strongly. If that interval clearly excludes zero, you can already anticipate how the t-test in the next section will come out — you have read the result off the picture before computing a p-value. (Near zero the two can part company, since the bootstrap interval and the Welch test are different procedures. Here the effect is nowhere near the boundary.) That is the argument for estimation plots in one sentence.

👀 Check your thinking

The effect is the colored dot on the right-hand axis; the vertical bar through it is its 95% interval, and the shaded curve behind shows which values the resampling supported most often. The group means on the left are inputs to that difference, not the effect itself.

Covering the right axis leaves you eyeballing a gap between two error bars — which is exactly the judgment the paper shows people get wrong, and exactly why this figure exists.

Narrower interval: more participants, or less person-to-person variability in responses. Unchanged: anything that doesn’t touch either — relabelling the axis, changing colors, or (importantly) collecting more data on a different question.

The confusion is between individual spread and precision of a mean. Overlapping dots say individuals vary a lot. The interval on the difference says the average gap is pinned down well. Both are true at once — the same point the CLES and overlap numbers make later in this step.

d · The Welch’s t-test

Write the hypotheses first — before you run anything. This is the transfer from M08 and M09, and it is the step most people skip. In your notebook, state both, in words and in symbols:

\[ H_0: \mu_\text{Saw SEs first} - \mu_\text{Saw SDs first} = 0 \qquad H_a: \mu_\text{Saw SEs first} - \mu_\text{Saw SDs first} \neq 0 \]

Two things to be careful about in the wording:

The parameters are population means for the two conditions — not individual people’s estimates, and not a claim about any one participant.
The paper predicts a direction (the group that could not see outcome variability should estimate higher). But the published test is non-directional, so you test two-sided and let the estimate carry the direction. Predicting a direction and testing two-sided is the conservative, conventional choice; it is not a contradiction.

Watch the direction. The order must run the same direction as your estimation plot and your hypotheses above, or every sign flips.

Targets. Run the focal comparison.

Pipe medical_bp into t_test().
Give it a formula of the form outcome ~ grouping variable — the outcome is psup_estimate.
Set order using the two objects you just defined, comparison first, baseline second, so the reported difference is comparison minus baseline.
Ask for a two-sided test with alternative.
Assign to test_result and print it.

t_test() from infer runs Welch’s version by default — it does not assume the two groups share a variance. That is the right default, and the M09 Module explains why.

The formula puts the outcome on the left and the grouping variable on the right — the same two column names you saw in glimpse(). For order, you already stored the right two strings as comp_group and ref_group in section c; pass them in that order. The paper’s headline test is non-directional, so alternative is the quoted string "two-sided".

Read the output in the order the Module taught. estimate is the raw mean difference in percentage points — the number your APA line leads with. lower_ci/upper_ci bound it. statistic is t_obs, t_df its degrees of freedom, p_value the two-sided tail area.

Notice that t_df is not a whole number. That is Welch’s degrees of freedom: because the two groups are allowed to have different variances, the df is a weighted blend of each group’s contribution to the standard error rather than a simple count. Compare your numbers to the paper’s t(65.1) = 6.83.

👀 Check your thinking

The p-value is: assuming the two population means are equal and the model’s assumptions are reasonable, the probability of getting a t-statistic at least as far from zero as the one you observed — in either direction. Two details matter. It is built on the standardized statistic, not the raw gap, because the same gap is more or less surprising depending on how precisely it was measured. And “either direction” is what makes it two-sided. It is a statement about data-under-a-model, not about the model being true.

Is it big? The t-test is silent on that. It reports how cleanly the difference is separated from zero, not whether the difference matters. Judging size needs the raw difference in percentage points, and a standardized effect to compare across studies — which is exactly why step e comes next.

With n = 20: shrinking the sample does not change the population quantity you are estimating — but be careful, that is not the same as saying your observed estimate would stay put. A 20-person estimate is far less stable; it could land well away from the current value depending on which 20 people you happened to draw. What changes systematically is the standard error: it grows, so |statistic| generally shrinks and p_value generally rises. That asymmetry — the target holding still while the evidence about it weakens — is the difference between an effect and the evidence for it.

e · Effect size (Cohen’s d)

The t-test says the difference is hard to attribute to sampling variation. Cohen’s d says how big it is, in standard-deviation units.

Why `pooled_sd = FALSE` belongs with a Welch test

Cohen’s d needs a standardizer — the SD you divide the mean difference by — and by default cohens_d() uses a pooled SD, which is built on the assumption that the two populations share one variance. That is exactly the assumption Welch’s test declines to make. Pairing a Welch test with a pooled-SD d isn’t catastrophic, but it does quietly reintroduce the assumption you just avoided.

Setting pooled_sd = FALSE uses the average of the two group SDs instead, which makes no equal-variance claim. This is the package’s own recommendation: ?cohens_d says to set it “for effect sizes that are to accompany a Welch’s t-test,” citing Delacre and colleagues (2021).

The test and the effect size still answer different questions, and it’s worth being able to say how:

Welch’s t measures the difference against its sampling uncertainty — how well pinned down is it?
Cohen’s d measures the same difference against the spread of individual scores — how big is it relative to how much people differ?

One reporting habit follows: name your standardizer. “Cohen’s d using the unpooled (average) SD” is a complete description; a bare “d = 1.60” is not, because other conventions exist (Glass’s Δ uses only the reference group’s SD; Hedges’s g adds a small-sample correction).

Putting d into plain language — an extension for after class

This section is not part of the in-class path. If your group is still working through the Welch test and the APA line, skip it and come back — it’s a good first thing to add when you assemble your notebook at home, and it is the extension most worth your time.

Cohen’s d is in standard-deviation units, which is precise and almost impossible to say out loud to a non-statistician. The common-language effect size (CLES) offers a different description of the same group difference, as a probability:

If I pick one clinician at random from each condition, how often is the one who saw SEs first the higher estimator?

There is a pleasing coincidence here. That quantity — the chance a randomly drawn member of one group exceeds a randomly drawn member of the other — is the probability of superiority. It is exactly what Zhang et al. asked their participants to estimate about a medication. So in describing your own effect you are computing, for your study, the very statistic the study is about.

CLES has a definition you can compute directly: form every possible pair, one from each group, and count how often the saw SEs first member is higher. Ties count as half.

Seeing it rather than computing it

Both numbers are easier to feel than to read. Dr. Kristoffer Magnusson’s interactive Cohen’s d visualization — the one the M09 Module pointed you to — lets you drag a slider for d and watch two distributions slide apart while the overlap and CLES update live. Set the slider near your own d and you are looking at your study.

One honest difference: that app draws smooth Normal curves, because it is illustrating a model. Your participants answered on a coarse scale — run n_distinct(psup_estimate) and you’ll find only a handful of distinct values, piled on round numbers. So treat the app as intuition for what your numbers mean, not as a picture of your data.

👀 Check your thinking

No. CLES counts pairs, not people, and it only asks who was higher — never by how much. A CLES of 86% means that if you draw one person from each condition and compare them, the Saw SEs first one is higher about 86 times in 100. It says nothing about the size of the gap in any particular pair, and it certainly does not mean one group’s scores sit entirely above the other’s. Look back at your figure from section b: the two clouds of points overlap heavily. Both facts are true at once — a reliable tilt, not a separation.

The correction. Something close to: “On average the display shifted estimates substantially, and if you drew one person from each condition the Saw SEs first one would give the higher estimate about [your CLES]% of the time. But the two groups’ answers still overlap a great deal — knowing someone’s condition tells you which way to bet, not what number they gave.” That sentence is defensible; “basically determines the answer” is not.

f · APA result line

Fill this in with your numbers

Among N = ___ medical providers viewing a hypothetical blood-pressure medication trial, those who first saw the standard-error visualization estimated a higher probability of superiority (M = ___, SD = ___) than those who first saw the standard-deviation visualization (M = ___, SD = ___), a difference of ___ percentage points, 95% CI [___, ___], t(___) = ___, p ___, Cohen’s d = ___, 95% CI [___, ___].

Paper reports: t(65.1) = 6.83, p < .001

Where each number comes from — every one is already on your screen, and this is the order the Module’s reporting rule asks for: raw effect first, then its uncertainty, then the standardized effect.

Slot	Where you got it
N	`count(condition)` in section a, or the `n` column of `group_descriptives`
M, SD per group	the `M` and `SD` columns of `group_descriptives`, section b
difference in percentage points	`estimate` from `test_result`, section d (the printout in c matches it)
95% CI on the difference	`lower_ci` / `upper_ci` from `test_result`
t, df, p	`statistic`, `t_df`, `p_value` from `test_result`
Cohen’s d and its CI	the cohens_d() output, section e

Two conventions to honor: report p as < .001 rather than as a rounded zero, and keep the direction of your sentence matching the sign of your estimate.

a · Meet the data

Your analytic dataset is datasci_analysis, built for you in Step 2. Everything you report — the counts, the plot, the descriptives, the test, the effect size — comes from that one object, so the N behind every number is provably the same. Start by looking at it:

Two things were decided in Step 2 that shape everything downstream, and they are worth being able to explain:

The condition order. fct_relevel() put “SE + Points” — the group that saw outcome variability — first, making it the baseline. That fixes the sign of every comparison: a positive difference now means the SE Only group estimated higher, which is the direction your APA sentence will read.
The rows. Nothing is filtered out from here on. What count() prints above is the N that reaches your t-test.

The conditions are “SE Only” (just error bars) and “SE + Points” (error bars plus individual data points). The paper’s claim: the SE-only group will overestimate the probability of superiority because they can’t see the spread of individual outcomes.

A heads-up before you run the test. Your t will land near 6.41 — close to, but not exactly, the paper’s 6.34. That’s expected: the authors applied a preregistered exclusion keyed to a background file our course dataset doesn’t ship, and the entire gap comes down to one participant. A near-match with a documented reason is still a successful reproduction check — and this is exactly the kind of divergence your Project 2 write-up should notice and explain rather than hide. (The “When the prose and the code disagree” box back in Step 1 has the full story, and it is worth reading before you write your verdict.)

b · Picture of the data

First, the numbers your APA line will need. Two of the six columns below you already know how to build; the other four are the ones worth slowing down on.

Targets. Build one row per condition, each carrying a mean and its interval.

Start from datasci_analysis, group_by() condition, and summarize().
Build six columns: the cell size, the mean of psup_estimate, its SD, its standard error, and the two 95% bounds.
The SE of a mean is the SD divided by the square root of n — this is the quantity M07 was about, now in service of a comparison.
For the bounds, the multiplier is the t critical value for that group’s degrees of freedom, from qt() — the Hint names the exact call if you need it.
Name the columns n, M, SD, SE, lo, hi — the figure you build next refers to them by those names.
Set .groups = "drop", assign to group_descriptives, and print it.

The standard error of a mean is the SD divided by the square root of the group size — you have both as columns already, so SD / sqrt(n).

👀 Check your group’s answers

Column	What it is	What it’s for
`n`	how many data scientists in that condition	the N in your APA line, and the df below
`M`	the group mean	the M in your APA line
`SD`	how much individual data scientists differ from each other	the SD in your APA line
`SE`	how precisely that group’s mean is estimated — `SD / sqrt(n)`	the width of that group’s interval. Welch’s test later combines both groups’ uncertainty into the standard error of the difference — neither group’s SE is itself the test’s denominator
`lo`, `hi`	the 95% confidence interval on that group’s mean	the error bars in the chart below

Now plot your study’s results.

Every individual observation, jittered so overlapping points stay visible.
Each group’s mean, marked clearly enough to read above the dots.
Each group’s 95% interval, drawn from the lo/hi you just computed.

The construction detail worth thinking about: layers 2 and 3 come from group_descriptives (two rows), while layer 1 comes from datasci_analysis (one row per person). A single ggplot() can draw from two different data frames — you pass data = inside the geom, and add inherit.aes = FALSE so that geom ignores the plot-level aes() and uses only its own.

Targets. Three layers to add to a figure that is otherwise given.

The scales, zoom, labels and legend are already written below — that chrome is not what this step is testing. What you add is the part that carries the statistics:

The raw observations. Add the geom that scatters individual points with a little horizontal jitter, so overlapping responses stay visible. Its width/height/alpha settings are already filled in.
The interval. Point geom_errorbar() at the summary table you built in part b, and map ymin and ymax to its two bound columns.
Leave inherit.aes = FALSE alone on both summary layers — it is what stops them inheriting the color mapping from the raw points and re-splitting by condition.

Read the result before moving on: the cloud is the people, the dot is the group mean, and the bar is how well you know that mean.

The error-bar layer reads from the summary table you built above, and geom_errorbar() wants a bottom and a top: ymin and ymax, which are exactly the two bound columns in that table.

c · Draw the effect

Every observation, so nobody has to take a summary on trust.
Each group’s mean with its 95% interval, on the left axis, in the outcome’s own units.
The difference itself, on its own axis at the right, with an interval showing the uncertainty around it rather than a bare point.

Two construction details worth understanding rather than copying:

sec_axis() builds the second axis. The right-hand ruler is the left-hand ruler shifted so that zero sits at the baseline group’s mean — the dashed line. It is one coordinate system with two labels, which is why the difference lines up with the group means instead of floating on an unrelated scale.

“Wait — didn’t M03 say two y-axes are bad?”

The shaded curve is a bootstrap — M07’s resampling machinery, aimed at a difference rather than a single mean. Its width is the uncertainty in the effect.

Two kinds of interval in one figure

This figure deliberately mixes two procedures, and you should be able to say which is which:

The group means on the left carry parametric t-intervals — the qt(0.975, n - 1) bounds you computed in section b.
The difference on the right carries a percentile bootstrap interval — resampled, nonparametric, straight out of M07. (Nonparametric means it assumes no distributional shape — but it does still assume your observations are suitable units to resample, which is the independence you get from the design.)

Targets. Name the two groups, baseline first.

Set ref_group to the condition that acts as the baseline — the one you are comparing against.
Set comp_group to the other one.
Both are the exact level labels as they appear in condition, in quotes. Get them from the table you just printed rather than from memory.

Order matters here and it will matter again in the t-test: it decides the sign of the difference, and therefore whether your effect reads as an increase or a decrease.

With those two names set, the rest of the figure is bookkeeping. One piece of it deserves a word first, because it is the only code in today’s lab that no module has shown you.

The infer verbs, in their estimation form

Step 2 of the code below bootstraps the difference between the two group means, using the verb pipeline from the M08 lecture:

your_data |>
  specify(psup_estimate ~ condition) |>          # outcome ~ grouping variable
  generate(reps = 2000, type = "bootstrap") |>   # resample 2,000 times
  calculate(stat = "diff in means", order = c(comp_group, ref_group))

Now run it:

Print the difference and its interval so you have the numbers in front of you — these are the raw effect and its uncertainty, and they lead your APA line:

👀 Check your thinking

Covering the right axis leaves you eyeballing a gap between two error bars — which is exactly the judgment the paper shows people get wrong, and exactly why this figure exists.

d · The Welch’s t-test

Write the hypotheses first — before you run anything. This is the transfer from M08 and M09, and it is the step most people skip. In your notebook, state both, in words and in symbols:

\[ H_0: \mu_\text{SE Only} - \mu_\text{SE + Points} = 0 \qquad H_a: \mu_\text{SE Only} - \mu_\text{SE + Points} \neq 0 \]

Two things to be careful about in the wording:

The parameters are population means for the two conditions — not individual people’s estimates, and not a claim about any one participant.
The paper predicts a direction (the group that could not see outcome variability should estimate higher). But the published test is non-directional, so you test two-sided and let the estimate carry the direction. Predicting a direction and testing two-sided is the conservative, conventional choice; it is not a contradiction.

Watch the direction. The order must run the same direction as your estimation plot and your hypotheses above, or every sign flips.

Targets. Run the focal comparison.

Pipe datasci_analysis into t_test().
Give it a formula of the form outcome ~ grouping variable — the outcome is psup_estimate.
Set order using the two objects you just defined, comparison first, baseline second, so the reported difference is comparison minus baseline.
Ask for a two-sided test with alternative.
Assign to test_result and print it.

t_test() from infer runs Welch’s version by default — it does not assume the two groups share a variance. That is the right default, and the M09 Module explains why.

👀 Check your thinking

e · Effect size (Cohen’s d)

The t-test says the difference is hard to attribute to sampling variation. Cohen’s d says how big it is, in standard-deviation units.

Why `pooled_sd = FALSE` belongs with a Welch test

The test and the effect size still answer different questions, and it’s worth being able to say how:

Welch’s t measures the difference against its sampling uncertainty — how well pinned down is it?
Cohen’s d measures the same difference against the spread of individual scores — how big is it relative to how much people differ?

Putting d into plain language — an extension for after class

If I pick one data scientist at random from each condition, how often is the one who saw error bars only the higher estimator?

CLES has a definition you can compute directly: form every possible pair, one from each group, and count how often the saw error bars only member is higher. Ties count as half.

Seeing it rather than computing it

👀 Check your thinking

No. CLES counts pairs, not people, and it only asks who was higher — never by how much. A CLES of 75% means that if you draw one person from each condition and compare them, the SE Only one is higher about 75 times in 100. It says nothing about the size of the gap in any particular pair, and it certainly does not mean one group’s scores sit entirely above the other’s. Look back at your figure from section b: the two clouds of points overlap heavily. Both facts are true at once — a reliable tilt, not a separation.

The correction. Something close to: “On average the display shifted estimates substantially, and if you drew one person from each condition the SE Only one would give the higher estimate about [your CLES]% of the time. But the two groups’ answers still overlap a great deal — knowing someone’s condition tells you which way to bet, not what number they gave.” That sentence is defensible; “basically determines the answer” is not.

f · APA result line

Fill this in with your numbers

Among N = ___ data scientists, those shown standard-error error bars only estimated a higher probability of superiority (M = ___, SD = ___) than those also shown individual data points (M = ___, SD = ___), a difference of ___ percentage points, 95% CI [___, ___], t(___) = ___, p ___, Cohen’s d = ___, 95% CI [___, ___].

Paper reports: t(159.4) = 6.34, p < .001

Slot	Where you got it
N	`count(condition)` in section a, or the `n` column of `group_descriptives`
M, SD per group	the `M` and `SD` columns of `group_descriptives`, section b
difference in percentage points	`estimate` from `test_result`, section d (the printout in c matches it)
95% CI on the difference	`lower_ci` / `upper_ci` from `test_result`
t, df, p	`statistic`, `t_df`, `p_value` from `test_result`
Cohen’s d and its CI	the cohens_d() output, section e

Two conventions to honor: report p as < .001 rather than as a rounded zero, and keep the direction of your sentence matching the sign of your estimate.

a · Meet the data

Your analytic dataset is faculty_analysis, built for you in Step 2. Everything you report — the counts, the plot, the descriptives, the test, the effect size — comes from that one object, so the N behind every number is provably the same. Start by looking at it:

Two things were decided in Step 2 that shape everything downstream, and they are worth being able to explain:

The condition order. fct_relevel() put “SE + Points” — the group that saw outcome variability — first, making it the baseline. That fixes the sign of every comparison: a positive difference now means the SE Only group estimated higher, which is the direction your APA sentence will read.
The rows. Nothing is filtered out from here on. What count() prints above is the N that reaches your t-test.

Same conditions as Study 2 — “SE Only” vs “SE + Points” — but the sample is tenure-track academics drawn from psychology, sociology, physics, biology, business, and computer science.

A documentation wrinkle in this study too, worth one sentence in your verdict. Your test will reproduce the paper’s t(363.9) = 4.52 exactly, using all 368 responses. The article also reports that 63 participants were excluded. Those two statements cannot both describe the same analysis: Welch’s degrees of freedom can never exceed N − 2, so a 305-person sample could not produce df = 363.9. We reproduce the published statistic as reported; how the stated exclusion count relates to that particular test isn’t clear from the article. Note it and move on — it does not change the finding.

b · Picture of the data

First, the numbers your APA line will need. Two of the six columns below you already know how to build; the other four are the ones worth slowing down on.

Targets. Build one row per condition, each carrying a mean and its interval.

Start from faculty_analysis, group_by() condition, and summarize().
Build six columns: the cell size, the mean of psup_estimate, its SD, its standard error, and the two 95% bounds.
The SE of a mean is the SD divided by the square root of n — this is the quantity M07 was about, now in service of a comparison.
For the bounds, the multiplier is the t critical value for that group’s degrees of freedom, from qt() — the Hint names the exact call if you need it.
Name the columns n, M, SD, SE, lo, hi — the figure you build next refers to them by those names.
Set .groups = "drop", assign to group_descriptives, and print it.

The standard error of a mean is the SD divided by the square root of the group size — you have both as columns already, so SD / sqrt(n).

👀 Check your group’s answers

Column	What it is	What it’s for
`n`	how many faculty in that condition	the N in your APA line, and the df below
`M`	the group mean	the M in your APA line
`SD`	how much individual faculty differ from each other	the SD in your APA line
`SE`	how precisely that group’s mean is estimated — `SD / sqrt(n)`	the width of that group’s interval. Welch’s test later combines both groups’ uncertainty into the standard error of the difference — neither group’s SE is itself the test’s denominator
`lo`, `hi`	the 95% confidence interval on that group’s mean	the error bars in the chart below

Now plot your study’s results.

Every individual observation, jittered so overlapping points stay visible.
Each group’s mean, marked clearly enough to read above the dots.
Each group’s 95% interval, drawn from the lo/hi you just computed.

The construction detail worth thinking about: layers 2 and 3 come from group_descriptives (two rows), while layer 1 comes from faculty_analysis (one row per person). A single ggplot() can draw from two different data frames — you pass data = inside the geom, and add inherit.aes = FALSE so that geom ignores the plot-level aes() and uses only its own.

Targets. Three layers to add to a figure that is otherwise given.

The scales, zoom, labels and legend are already written below — that chrome is not what this step is testing. What you add is the part that carries the statistics:

The raw observations. Add the geom that scatters individual points with a little horizontal jitter, so overlapping responses stay visible. Its width/height/alpha settings are already filled in.
The interval. Point geom_errorbar() at the summary table you built in part b, and map ymin and ymax to its two bound columns.
Leave inherit.aes = FALSE alone on both summary layers — it is what stops them inheriting the color mapping from the raw points and re-splitting by condition.

Read the result before moving on: the cloud is the people, the dot is the group mean, and the bar is how well you know that mean.

The error-bar layer reads from the summary table you built above, and geom_errorbar() wants a bottom and a top: ymin and ymax, which are exactly the two bound columns in that table.

c · Draw the effect

Every observation, so nobody has to take a summary on trust.
Each group’s mean with its 95% interval, on the left axis, in the outcome’s own units.
The difference itself, on its own axis at the right, with an interval showing the uncertainty around it rather than a bare point.

Two construction details worth understanding rather than copying:

sec_axis() builds the second axis. The right-hand ruler is the left-hand ruler shifted so that zero sits at the baseline group’s mean — the dashed line. It is one coordinate system with two labels, which is why the difference lines up with the group means instead of floating on an unrelated scale.

“Wait — didn’t M03 say two y-axes are bad?”

The shaded curve is a bootstrap — M07’s resampling machinery, aimed at a difference rather than a single mean. Its width is the uncertainty in the effect.

Two kinds of interval in one figure

This figure deliberately mixes two procedures, and you should be able to say which is which:

The group means on the left carry parametric t-intervals — the qt(0.975, n - 1) bounds you computed in section b.
The difference on the right carries a percentile bootstrap interval — resampled, nonparametric, straight out of M07. (Nonparametric means it assumes no distributional shape — but it does still assume your observations are suitable units to resample, which is the independence you get from the design.)

Targets. Name the two groups, baseline first.

Set ref_group to the condition that acts as the baseline — the one you are comparing against.
Set comp_group to the other one.
Both are the exact level labels as they appear in condition, in quotes. Get them from the table you just printed rather than from memory.

Order matters here and it will matter again in the t-test: it decides the sign of the difference, and therefore whether your effect reads as an increase or a decrease.

With those two names set, the rest of the figure is bookkeeping. One piece of it deserves a word first, because it is the only code in today’s lab that no module has shown you.

The infer verbs, in their estimation form

Step 2 of the code below bootstraps the difference between the two group means, using the verb pipeline from the M08 lecture:

your_data |>
  specify(psup_estimate ~ condition) |>          # outcome ~ grouping variable
  generate(reps = 2000, type = "bootstrap") |>   # resample 2,000 times
  calculate(stat = "diff in means", order = c(comp_group, ref_group))

Now run it:

Print the difference and its interval so you have the numbers in front of you — these are the raw effect and its uncertainty, and they lead your APA line:

👀 Check your thinking

Covering the right axis leaves you eyeballing a gap between two error bars — which is exactly the judgment the paper shows people get wrong, and exactly why this figure exists.

d · The Welch’s t-test

Write the hypotheses first — before you run anything. This is the transfer from M08 and M09, and it is the step most people skip. In your notebook, state both, in words and in symbols:

\[ H_0: \mu_\text{SE Only} - \mu_\text{SE + Points} = 0 \qquad H_a: \mu_\text{SE Only} - \mu_\text{SE + Points} \neq 0 \]

Two things to be careful about in the wording:

The parameters are population means for the two conditions — not individual people’s estimates, and not a claim about any one participant.
The paper predicts a direction (the group that could not see outcome variability should estimate higher). But the published test is non-directional, so you test two-sided and let the estimate carry the direction. Predicting a direction and testing two-sided is the conservative, conventional choice; it is not a contradiction.

Watch the direction. The order must run the same direction as your estimation plot and your hypotheses above, or every sign flips.

Targets. Run the focal comparison.

Pipe faculty_analysis into t_test().
Give it a formula of the form outcome ~ grouping variable — the outcome is psup_estimate.
Set order using the two objects you just defined, comparison first, baseline second, so the reported difference is comparison minus baseline.
Ask for a two-sided test with alternative.
Assign to test_result and print it.

t_test() from infer runs Welch’s version by default — it does not assume the two groups share a variance. That is the right default, and the M09 Module explains why.

👀 Check your thinking

e · Effect size (Cohen’s d)

The t-test says the difference is hard to attribute to sampling variation. Cohen’s d says how big it is, in standard-deviation units.

Why `pooled_sd = FALSE` belongs with a Welch test

The test and the effect size still answer different questions, and it’s worth being able to say how:

Welch’s t measures the difference against its sampling uncertainty — how well pinned down is it?
Cohen’s d measures the same difference against the spread of individual scores — how big is it relative to how much people differ?

Putting d into plain language — an extension for after class

If I pick one faculty member at random from each condition, how often is the one who saw error bars only the higher estimator?

CLES has a definition you can compute directly: form every possible pair, one from each group, and count how often the saw error bars only member is higher. Ties count as half.

Seeing it rather than computing it

👀 Check your thinking

No. CLES counts pairs, not people, and it only asks who was higher — never by how much. A CLES of 65% means that if you draw one person from each condition and compare them, the SE Only one is higher about 65 times in 100. It says nothing about the size of the gap in any particular pair, and it certainly does not mean one group’s scores sit entirely above the other’s. Look back at your figure from section b: the two clouds of points overlap heavily. Both facts are true at once — a reliable tilt, not a separation.

f · APA result line

Fill this in with your numbers

Among N = ___ tenure-track academic faculty, those shown standard-error error bars only estimated a higher probability of superiority (M = ___, SD = ___) than those also shown individual data points (M = ___, SD = ___), a difference of ___ percentage points, 95% CI [___, ___], t(___) = ___, p ___, Cohen’s d = ___, 95% CI [___, ___].

Paper reports: t(363.9) = 4.52, p < .001

Slot	Where you got it
N	`count(condition)` in section a, or the `n` column of `group_descriptives`
M, SD per group	the `M` and `SD` columns of `group_descriptives`, section b
difference in percentage points	`estimate` from `test_result`, section d (the printout in c matches it)
95% CI on the difference	`lower_ci` / `upper_ci` from `test_result`
t, df, p	`statistic`, `t_df`, `p_value` from `test_result`
Cohen’s d and its CI	the cohens_d() output, section e

Two conventions to honor: report p as < .001 rather than as a rounded zero, and keep the direction of your sentence matching the sign of your estimate.

Checkpoint 3 · Your study reproduced

For your assigned study you have all six pieces: a data picture showing error bars and points, an estimation plot putting the difference and its uncertainty on their own axis, a Welch’s t-test, a Cohen’s d with its 95% CI, a completed APA result line, and a reproduction verdict in one of three words: exactly reproduced, closely reproduced (with the discrepancy explained), or not reproduced. Your numbers are ready to share.

One check worth making before you present: does your estimation plot agree with your APA line? The difference the plot shows should have the same sign and roughly the same interval as the t-test’s. If the plot says +15 and your sentence says the groups went the other way, one of the two has its conditions reversed.

In your notebook — this whole step is the deliverable core: from your group’s tab, add the data picture (label the chunk fig-psup with a fig-cap), the estimation plot (label it fig-effect, and give it a caption naming the effect it shows), the Welch’s t-test, the Cohen’s d with its CI, your completed APA result line, and the reproduction verdict. Interpretation prose after each. Your Data section should still say in a sentence or two where the file came from, what one row is, and which rows you analyzed — that is the documentation habit Project 2 will ask you to formalize.

Step 4 · Share back and synthesize

Each group’s spokesperson takes ~3 minutes to walk us through:

Question — what was the manipulation, in plain English?
Sample — who, how many, any meaningful demographic notes?
Result — your APA line, on screen.
Reproduction verdict — your t and d next to the paper’s. Did the numbers come back out?
One sentence — what does your study say about the paper’s central claim that experts confuse inferential uncertainty with outcome variability?

As each group reports, we’ll fill in this synthesis table on the screen. By the end, we have a 3-row reproduction-checklist artifact for the whole paper.

Put the three estimation plots side by side

Once all three groups have reported, pull up the three estimation plots together and read the right-hand axes across them. The difference shrinks as you move from Study 1 to Study 3 — and the intervals tell you how confident to be that the ordering is real rather than an accident of which experts happened to be sampled.

Two questions worth arguing about before you leave:

Do all three effects point the same way? If every interval sits entirely above zero, all three studies agree on the direction: hiding outcome variability raises average PSup judgments.
How similar are the magnitudes? Compare the point estimates and interval widths descriptively — and stop there. Whether two intervals happen to overlap is not a test of whether the effects are equal, and non-overlap is not proof that they differ. A formal comparison is a different procedure than eyeballing two pictures.

And there’s a reason this lab cannot settle it anyway. Study 1 differs from Studies 2 and 3 in two ways at once — a different expert population and a different visualization contrast (SE vs SD bars, against SE-only vs SE-plus-points). When two things change together, no amount of staring at the gradient can tell you which one produced it; population, manipulation, scenario, and ordinary sampling variation are all tangled. Describing the pattern is honest. Explaining it would require a design built for that question.

This is still the payoff of plotting effects rather than p-values: three p-values would all read “< .001” and tell you nothing about how big the illusion is in each audience.

Checkpoint 4 · The whole paper, assembled

The class synthesis table has all three rows filled — each study’s t, d, and verdict beside the paper’s. You can describe the effect-size gradient across the three studies, and explain why this set of studies cannot by itself identify what produced it.

In your notebook — the synthesis table is a class artifact, not required in your submission — but your Discussion draws on it. Here is everything that section needs.

What your `# Discussion` needs — four things

Your Results section had six labelled parts to work through. Discussion has four, and it is the shortest section in the report — aim for one paragraph, not a page.

Did you reproduce it? One sentence, naming your own numbers: “Our reanalysis recovered the published contrast, t(…) = …, d = …” — or saying plainly where it diverged.
Where your study sits. One sentence placing your group against the other two — the effect-size gradient across the three audiences.
One limitation. The strongest candidate is right in front of you: your group is one audience, and the three studies differ in sample and in what was manipulated, so a gradient across them is not a clean test of who is more susceptible.
What a reproduction does and does not establish. You re-ran their analysis on their data. Agreement shows the published numbers follow from the data as analyzed — computational reproducibility. It is not independent replication, which would need new participants.

If you missed the synthesis — the three studies’ reproduced numbers

Fill the table in live if you are in class; this is the backstop if you were away or are working ahead. Each row is what that group’s own lab path produces from the shipped data.

Study	N	Paper’s t	Our t	Our d [95% CI]
1 · Medical Providers	75	6.83	t(65.1) = 6.83	1.60 [1.06, 2.12]
2 · Data Scientists	175	6.34	t(159.2) = 6.41	0.96 [0.64, 1.27]
3 · Faculty	368	4.52	t(363.9) = 4.52	0.47 [0.26, 0.67]

The gradient is in the last column: 1.60 → 0.96 → 0.47. All three reproduce the paper’s direction and significance, and the standardized effect shrinks markedly from Study 1 to Study 3.

Before you write that up as “the illusion is weaker in faculty,” re-read limitation 3 above. The three studies differ in audience and in what was manipulated — Study 1 contrasts SE against SD error bars, Studies 2 and 3 contrast SE alone against SE plus individual points. A gradient across them cannot separate those two explanations.

Lab debrief · 5 minutes

That’s the reproduction done, and the whole paper assembled between you. With a few minutes left, save your work and look up — this is the last lab of Part 2, so it is worth spending them on what the last four weeks were actually for.

Lab debrief · what did we learn by doing?

The sticking point. Where did today cost you the most time — reconstructing the analytic sample, choosing between the tests, getting cohens_d() to point the same direction as the t-test, or assembling the report? What got you past it, and what would have helped sooner?
Choosing the test was the hard part, not running it. Every group ran a two-sample t-test today, and the call itself was one line. The work was upstream: deciding which design you had, which variable was the outcome, and which rows belonged in the analysis. When you meet a new dataset with no lab page attached, what is the first question you ask to get to the right test?
The same logic, three different audiences. Your group reproduced one study; the other two reproduced the others. The t-values ranged from 6.83 down to 4.52 and the standardized effects from 1.60 down to 0.47 — yet all three would be reported as p < .001. What does that tell you about how much a p-value on its own says about the size of a finding?
What a reproduction is worth — and what it isn’t. You recovered the paper’s numbers from the paper’s data. Say precisely what that establishes and what it leaves open. If a reproduction had failed — your t nowhere near theirs — name three things that could explain it, only one of which is “the paper is wrong.”
Part 2 ends here, and Project 2 starts from it. Everything you did today is the shape of Project 2: find a published result, reconstruct the sample, re-run the test, and report honestly whether it came back. What was hardest today that you would want to have settled before committing to a paper of your own?

Final render and submit

This is the take-home half of the lab. You scaffolded m09_lab.qmd in Step 0 and each step’s In your notebook note told you what to drop in. Now assemble it into a report a stranger could follow cold — and keep building the habit you’ll lean on for Project 2:

Add your name to the author: field.
Do a final render. Click Render, or press Cmd/Ctrl + Shift + K.
Open the rendered HTML (it renders next to the .qmd in programs/).
Read it end to end — as if a stranger opened it cold. Is every result labeled? Is your APA line complete?
Submit it to Canvas under “Lab 9 — Reproducing the Illusion of Predictability.”
Commit and push. In GitHub Desktop, commit your lab notebook with a one-line summary, then Push origin. Project 2 starts next week and your team will work through this same loop — today is the last low-stakes rehearsal.

One thing to bring to the Project 2 launch. As you assemble the report, note which part of today’s workflow you’d feel ready to scale up to a full reproduction — and which part still feels under-scaffolded. We open next week’s session with that question.

Double check

Before you submit:

Your Introduction has two or three sentences giving the paper’s question and what your group’s study manipulated (Step 1)
Your group’s Welch’s t-test runs and its numbers match the shared checklist
Step 3’s deliverable core is all present: the data picture (fig-psup), the estimation plot, and the test — each labelled and captioned
Your Discussion has all four pieces (Step 4): whether you reproduced it with your numbers, where your study sits against the other two, one limitation, and one sentence on what a reproduction does and does not establish
Your APA result line is complete — M/SD per group, the raw mean difference with its 95% CI, then t, df, p, and Cohen’s d with its 95% CI
Your “picture of the data” shows error bars and individual points
You can say in one sentence whether your study reproduced the paper
You submitted the HTML to Canvas
You committed and pushed — GitHub Desktop’s Changes tab is now empty

What you just did, in research terms

You ran a computational reproduction — the same test, on the same data, as a published paper, with your numbers reported beside theirs. (Not a direct replication: that would mean collecting a new sample of clinicians, data scientists, or faculty and running the study again.) That is the credibility check the field increasingly asks for, and it’s the spine of Project 2. Notice what made it trustworthy: you pictured the data before testing it, you chose Welch’s t so the inference did not require equal population variances, you reported an effect size with a confidence interval rather than a lone p-value, and you documented the analytic dataset in your write-up so someone else could retrace your path from raw file to result. Across three expert audiences the illusion held — the same confusion Hofman first found in a boulder-sliding game turns up in clinicians reading a medication-trial scenario. Same recipe, real stakes.

More practice (optional)

After class, any time. Nothing here goes into your lab notebook, and none of it is graded.

The same clinician, twice: a paired look at Study 1 → — the paired analysis Step 1 set aside. Run the paired t-test on both of each clinician’s estimates, then meet the two complications the first-response comparison avoids: the two responses show essentially no positive within-person correlation, so pairing’s usual precision advantage is absent, and the size of the within-person contrast differs by presentation order. A useful corrective to the idea that pairing always helps.

Every optional activity in the course is also listed on one page: Optional Activities.

Welcome to the Module 9 lab

What you’ll leave with

Before you start — set up your notebook

Carrying code into your notebook — the same rules as M06

Step 0 · Get set up

Step 1 · The paper in three minutes

Zhang et al. (2023) · An illusion of predictability in scientific results

1. One focal question, shared across all three studies

2. One visual feature changed

3. One focal comparison per study—for this lab

How close should your numbers come?

Your assignment

Checkpoint 1 · You can frame the reproduction

Step 2 · Whole-class setup

illusion_medical / illusion_datasci / illusion_faculty · 163 / 175 / 368 observations · shared schema · Zhang et al. (2023)

Before you test: what a t-test actually assumes

Checkpoint 2 · Data loaded and understood

Step 3 · Reproduce your study

How to use this part

a · Meet the data

b · Picture of the data

c · Draw the effect

“Wait — didn’t M03 say two y-axes are bad?”

Two kinds of interval in one figure

The infer verbs, in their estimation form

Talk it through: read the figure before you test it

d · The Welch’s t-test

Talk it through: what the p-value does and doesn’t say

e · Effect size (Cohen’s d)

Why pooled_sd = FALSE belongs with a Welch test

Check yourself: what CLES does and doesn’t say

f · APA result line

Fill this in with your numbers

a · Meet the data

b · Picture of the data

c · Draw the effect

“Wait — didn’t M03 say two y-axes are bad?”

Two kinds of interval in one figure

The infer verbs, in their estimation form

Talk it through: read the figure before you test it

d · The Welch’s t-test

Talk it through: what the p-value does and doesn’t say

e · Effect size (Cohen’s d)

Why pooled_sd = FALSE belongs with a Welch test

Check yourself: what CLES does and doesn’t say

f · APA result line

Fill this in with your numbers

a · Meet the data

b · Picture of the data

c · Draw the effect

“Wait — didn’t M03 say two y-axes are bad?”

Two kinds of interval in one figure

The infer verbs, in their estimation form

Talk it through: read the figure before you test it

d · The Welch’s t-test

Talk it through: what the p-value does and doesn’t say

e · Effect size (Cohen’s d)

Why pooled_sd = FALSE belongs with a Welch test

Check yourself: what CLES does and doesn’t say

f · APA result line

Fill this in with your numbers

Checkpoint 3 · Your study reproduced

Step 4 · Share back and synthesize

Put the three estimation plots side by side

While the other groups present

Checkpoint 4 · The whole paper, assembled

What your # Discussion needs — four things

Lab debrief · 5 minutes

Lab debrief · what did we learn by doing?

Final render and submit

Double check

What you just did, in research terms

More practice (optional)

Why `pooled_sd = FALSE` belongs with a Welch test

Why `pooled_sd = FALSE` belongs with a Welch test

Why `pooled_sd = FALSE` belongs with a Welch test

What your `# Discussion` needs — four things