Conducting Common Hypothesis Tests

A decorative image of two distributions (group A and group B). — Image inspired by the R-centric artwork of Allison Horst

Learning Objectives

By the end of this Module, you should be able to:

Choose among five common hypothesis tests by identifying the outcome type, the number of groups or measurements, and whether observations are independent or paired.
State, in words and then in symbols, the population parameter and the hypotheses each test evaluates.
Run each test in R and read its output: the estimate, the test statistic, the degrees of freedom, the p-value, and the confidence interval.
Assess the design and model assumptions that matter for each test, and choose a defensible alternative when the standard analysis is unsuitable.
Report each result in APA style — the raw effect first, then its uncertainty, then a standardized effect size — with a claim whose scope matches the design.

Overview

Where this fits — the Part 2 inference arc

Modules 6–9 are one continuous story: how we reason from a sample back to the world. Each module is a single step in that arc.

M06 · Probability — what outcomes should we expect under a given data-generating process?
M07 · Confidence intervals — how much does a sample statistic wobble from one sample to the next?
M08 · The logic of NHST — how surprising is our statistic if a specific null value were true?
M09 · Conducting tests — which standard test matches this design, and how do I run, read, and report it? · ← you are here

In Module 8 you learned the logic of null-hypothesis significance testing, and you learned it on a single test: imagine a world where $H_0$ is true, ask how compatible your data are with that world, and decide against a pre-specified $\alpha$. That logic is now fixed. You will not learn a new version of it in this Module — or, honestly, ever. What changes from here on is something else entirely:

The logic never changes. What changes is the design — and the design chooses the test.

That sentence is the whole Module. A one-sample t-test, a two-sample t-test, an ANOVA, a chi-square test, a paired t-test — these are not five separate theories to memorize. They are one piece of logic pointed at five different research designs. Each design asks about a different population quantity, so each needs its own test statistic, reference distribution, and effect-size measure. But the reasoning underneath — how compatible is what I observed with what the null model predicts? — never moves.

This is also the Module built to be used. When Project 2 asks you to reproduce a published test in a few weeks, this is where you’ll look up how. When your own data arrive next semester — or mid-dissertation, three years from now — and you need to compare two groups, or four, or a table of counts, this is the chapter to come back to. Every test below answers the same four practical questions: when do I use it, how do I run it, how do I report it, and what does the result actually mean?

One study, five questions. To keep the statistics in front and the bookkeeping behind, the whole Module works inside a single research world — Hofman, Goldstein, and Hullman’s boulder-sliding visualization study, introduced just below. The data stay familiar; only the structure of the question changes:

Is one group’s mean calibrated to a benchmark? → one-sample t-test
Do two randomized groups differ? → two-sample (Welch’s) t-test
Do four randomized groups all have the same mean? → one-way ANOVA
Does a categorical outcome depend on group? → chi-square test of independence
What changes when the same people are measured twice? → paired-samples t-test

Each question is a small transformation of the one before it: a second group appears, then two more, then the outcome turns categorical, then the groups stop being independent. If you can see each test as the previous test plus one change of design, you will never again stare at a dataset wondering which test “applies.”

What this Module deliberately does not cover: regression (M10–M11), correlation as a standalone test (M10 shows it’s a special case of regression — easier to learn once, there), and factorial designs with interactions (a PSY 653 topic; more on that below). What you get here is the toolkit behind most published group comparisons in psychology — and the judgment to match the right tool to the right question.

Packages used in this module

library(tidyverse)
library(here)
library(broom)
library(infer)
library(effectsize)
library(scales)
library(patchwork)
library(gtsummary)
library(labelled)

Meet the data

All five tests in this Module share a single research context — a study by Hofman, Goldstein, and Hullman (2020) titled How visualizing inferential uncertainty can mislead readers about treatment effects in scientific results (CHI 2020). The full paper is in your reading list, but you don’t need it in front of you to follow the analyses below. Here’s what you need to know.

The Hofman et al. boulder-sliding study

Hofman and colleagues wanted to know whether the kind of error bar an author chooses — a 95% confidence interval (CI), a 95% prediction interval (PI), or something else entirely — changes how readers interpret a finding. They built a clever experimental scenario to test this. Participants on Amazon Mechanical Turk were told they were competing in a fictional boulder-sliding game against an equally skilled opponent named Blorg. The goal was to slide a boulder farther than Blorg’s; the contestant whose boulder slid farthest won an all-or-nothing prize of 250 Ice Dollars. Participants were given the chance to rent a “special boulder” (a treatment that was expected, but not guaranteed, to slide farther than the standard boulder). They were shown a visualization comparing the two boulders’ performance, then asked two questions:

Willingness to pay (WTP) — how much, in Ice Dollars, would you pay to rent the special boulder?
Probability of superiority — what’s the probability that the special boulder will out-slide the standard boulder on a given attempt?

The fictional cover story was deliberate: it strips away real-world expertise about the underlying domain so any systematic difference in responses across visualization conditions can be attributed to the visualization itself.

Behind the scenes, the simulated boulder distributions were Normal random variables. The standard boulder always had mean 100 m and standard deviation 15.3 m. The special boulder had the same SD but a higher mean — by 4 m in the small effect-size condition (104 m; Cohen’s d ≈ 0.25, true probability of superiority ≈ 57%) or by 16 m in the large effect-size condition (116 m; Cohen’s d ≈ 1.0, true probability of superiority ≈ 76%).¹ The “right” answers were therefore knowable: the risk-neutral willingness to pay was approximately 17.5 Ice Dollars for the small-effect condition and approximately 65 for the large-effect condition.

The study ran two experiments. Experiment 1 (used in Test 1) is a 2 × 2 design crossing interval format (CI vs PI) with caption text, run at the small effect size only. Experiment 2 (used in Tests 2–4) crossed four visualization formats with two effect sizes for an 8-cell between-subjects factorial, and the course version of the data (data/blorg_exp2.Rds) carries all of it.

The four displays, side by side. This is the figure to sit with for a minute, because it contains the paper’s entire argument. Every panel below plots exactly the same underlying data — a standard boulder averaging 100 m against a special boulder averaging 104 m, both with an SD of 15.3 m. Nothing about the boulders changes from panel to panel. Only the choice of what to draw changes.

Four panels, each plotting the standard boulder at 100 meters and the special boulder at 104 meters. Panel a, 95% confidence intervals: a y-axis labeled 'Average sliding distance' running from 99 to 105, with short error bars, so the two boulders appear far apart and clearly separated. Panel b, 95% prediction intervals: a y-axis labeled 'Sliding distance' running from about 70 to 135, with very long error bars that overlap almost completely. Panel c, rescaled 95% confidence intervals: the same short confidence-interval bars from panel a redrawn on panel b's wide 70-to-130 axis, so they shrink to small dots. Panel d, animated samples or HOPs: a stack of small chart frames receding along an arrow labeled 'Time', each frame showing one simulated pair of outcomes. — The four visualization formats tested by Hofman et al., all showing the same underlying data.

Look at what that does. In (a) the two boulders look worlds apart; in (b) they look almost indistinguishable. Panel (a) draws the mean ± 1.96 standard errors — uncertainty about where the average sits, which shrinks as the sample grows. Panel (b) draws the mean ± 1.96 standard deviations — the spread of individual slides, which does not shrink no matter how much data you collect. Panel (c) keeps (a)’s confidence interval but borrows (b)’s axis, separating “how wide the interval looks” from “what the interval means.” Panel (d) abandons error bars entirely and animates individual draws, one at a time.

Same data, four pictures, four very different impressions. That is the phenomenon every test in this Module is measuring.

Figure credit and reuse terms

Figure 1 from Hofman, J. M., Goldstein, D. G., & Hullman, J. (2020). How visualizing inferential uncertainty can mislead readers about treatment effects in scientific results. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), Paper 327, pp. 1–12. https://doi.org/10.1145/3313831.3376454

© 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Reproduced here for classroom use in PSY 652.

The authors also publish the data and code needed to reproduce their results — the analysis code and the cleaned datasets these labs are built from — at github.com/jhofman/visual-effects-chi2020, and an author-hosted copy of the paper is available at jakehofman.com.

A full factorial — and how we handle it

Experiment 2 is a complete factorial: every visualization was shown at every effect size. Tests 2–4 handle this by filtering to the Large effect-size condition and comparing visualizations there. That is a deliberate choice, and it’s worth seeing why: within one effect-size condition, every participant faces the same underlying decision problem, so the group comparisons are clean. (Pooling the two effect sizes would mix two different decision problems into one outcome distribution — a mean of that mixture is hard to interpret.)

We stop short of the question the full factorial could answer — does the effect of the visualization depend on which effect size the reader saw? That is an interaction, and testing it means fitting a factorial ANOVA. We don’t cover it here: factorial designs and interactions are a PSY 653 topic, where they’re developed properly under the name moderation.

One practical consequence for your writing: when your analysis uses one slice of a larger design, say so — “within the large-effect condition, displays differed.” The claim about a slice and the claim about the whole design are different findings.

blorg_exp2 · 4 visualization formats × 2 effect sizes · between-subjects · Hofman, Goldstein, & Hullman (2020), Experiment 2

A course-prepared version of Hofman, Goldstein, & Hullman (2020), Experiment 2 (data/blorg_exp2.Rds) — all 1,830 post-exclusion participants, all four visualization arms.

graph_type factor — Visualization shown to the participant.
- "CI" — Hofman et al.’s “95% confidence interval” display: error bars at mean ± 1.96 SE.
- "CI rescaled" — the same CI display with its y-axis stretched to span the 95% PI range. The paper included it to test whether the visual width of the interval, rather than its statistical meaning, was driving readers’ judgments.
- "PI" — the study’s PI-style outcome interval: error bars at mean ± 1.96 SD, visualizing roughly 95% of individual outcomes. (A textbook prediction interval built from sample estimates would carry a small additional correction; Hofman et al. displayed the ± 1.96 SD form.)
- "HOPS" — hypothetical outcome plots (HOPs): an animation showing draws from each boulder’s distribution one at a time, rather than summarising them with a static bar. Note the spelling: the s in HOPs is just the plural of “plot”, so the literature writes HOPs. The data file stores the level as "HOPS", so that is what you type when you filter — but in prose we follow the paper.
effect_size factor — Underlying gap between the standard and special boulders.
- "Small" — Cohen’s d ≈ 0.25 (true win probability ≈ 57%)
- "Large" — Cohen’s d ≈ 1.0 (true win probability ≈ 76%)
wtp numeric — Willingness to pay to rent the special boulder, in Ice Dollars (continuous slider, 0 to 250)
superiority numeric — Participant’s estimated probability that the special boulder will out-slide the standard boulder, on a 0–1 scale (typed as 0–100, then rescaled)

blorg_exp2 <- read_rds(here("shared_data", "blorg_exp2.Rds"))

blorg_exp2 |> glimpse()

Rows: 1,830
Columns: 4
$ graph_type  <fct> HOPS, CI, HOPS, PI, CI, CI, HOPS, HOPS, HOPS, CI, HOPS, CI…
$ effect_size <fct> Large, Small, Large, Large, Large, Large, Large, Small, La…
$ wtp         <dbl> 50, 0, 25, 50, 50, 50, 50, 25, 150, 11, 50, 0, 50, 20, 30,…
$ superiority <dbl> 0.75, 0.90, 0.73, 0.75, 1.00, 0.75, 0.75, 0.62, 0.90, 0.50…

The five questions, mapped to the data. Here is the whole Module on one card — the same study, interrogated five ways:

	The question	The design	The test
1	Is one group’s average WTP calibrated to the rational price?	One group (Exp 1, PI matching-text arm), external benchmark $17.50	One-sample t
2	Do CI viewers and PI viewers pay different amounts?	Two randomized groups (Exp 2, Large)	Two-sample t (Welch)
3	Do the four display formats all produce the same mean WTP?	Four randomized groups (Exp 2, Large)	One-way ANOVA
4	Does overpaying (WTP above the $65 rational price) depend on the display?	Group × categorical outcome (Exp 2, Large)	Chi-square
5	Does WTP change from before to after seeing a display?	Two measurements per person (simulated extension)	Paired-samples t

Two honesty notes on this arc, before we start. Test 1 is a course-designed analysis, not a reproduction of the paper’s central test: the paper’s preregistered hypotheses compared visualization formats, and the one-sample calibration question formalizes a descriptive pattern the authors note along the way. And Test 5 is simulated — the study had no pre–post measurement, so we build one, visibly, to teach the paired design. Everything else runs on the real data.

Part 1 — One logic, five designs

The question comes before the test

Here is the most practical thing this Module will tell you: statistical software does not choose the test — you do. R will happily run a t-test on data that needed a chi-square, and it will not warn you. The choice is yours, and it is made before any statistics happen, by answering four questions about your research design:

What kind of outcome do you have? A continuous measurement, or category membership?
How many groups or measurements are being compared? One group against a benchmark? Two groups? Several?
Are the observations independent or paired? Different people in each group, or the same people measured more than once?
What population parameter does your question name? A mean? A difference of means? A set of proportions?

Answer those four, and the test falls out. Here is the decision map for the five designs this Module covers — worth bookmarking, because it is the lookup you’ll use for Project 2 and beyond:

The decision map

Outcome	Design	The question	The test
Continuous	One sample	Is one population mean equal to a benchmark?	One-sample t
Continuous	Two independent groups	Are two population means equal?	Two-sample t (Welch)
Continuous	Three or more independent groups	Are all the population means equal?	One-way ANOVA
Categorical	One categorical variable	Do the category proportions match values specified in advance?	Chi-square goodness-of-fit
Categorical	Two categorical variables	Are the variables independent?	Chi-square test of independence
Continuous	Two measurements on the same units	Is the mean within-person difference zero?	Paired-samples t
Categorical (binary)	Two measurements on the same units	Did the proportion change?	McNemar’s test (named in Test 5, not developed)

Two edges of the map, for completeness: a categorical outcome with a continuous predictor calls for logistic regression (beyond this course), and a continuous outcome with a continuous predictor is regression — that’s M10.

And one honest caveat about the map itself. It covers the uncomplicated designs this Module develops: one outcome, independent units or a single pairing, no covariates. Real studies routinely add complications that take you off it — clustered or nested data, repeated measurements at more than two time points, covariates you need to adjust for, unequal follow-up, complex survey sampling. When the slogan says the design chooses the test, read it as useful shorthand: more precisely, the question, the outcome type, the dependence structure, and the modeling assumptions together narrow the set of defensible analyses. The map is where that narrowing starts, not where it ends.

What stays fixed, what changes

If the last Module taught the logic and this one applies it five ways, it helps to be explicit about exactly which parts travel and which parts swap out:

Component	What stays the same	What changes
The null logic	Ask how compatible the data are with $H_0$	The parameter $H_0$ constrains
The evidence	A test statistic and its p-value	t, F, or $\chi^2$
The uncertainty	Sampling variability under a model	The formula and reference distribution
The magnitude	Report an effect size with its CI	d, $\eta^2$, V, or $d_z$
The interpretation	Name the direction and size in plain language	The substantive claim

One clarification about the test statistic, because it’s easy to over-generalize. Every test statistic in this Module does the same job: it compares the departure you observed with the amount of departure the null model would ordinarily produce. For the t-tests, that comparison is literally a gap divided by its standard error. ANOVA’s F and the chi-square statistic do the same job differently — they aggregate many departures (between-group spread against within-group spread; observed counts against expected counts) into a single number. Same purpose, different arithmetic. Don’t force them into one formula; do carry the one idea.

And a reminder from M08 worth keeping precise: the p-value describes how incompatible the observed result is with the null model — not the probability the null is true, and not a measure of importance. Because incompatibility depends on precision, the same size of effect can produce different p-values at different sample sizes. That is exactly why every test below reports an effect size alongside the test: magnitude is a separate question, and it gets a separate number.

One decision rule, fixed now for the whole Module. Every test below uses $\alpha = .05$ and a two-sided alternative, carrying forward the convention M08 set. Stating it once here rather than re-arguing it five times is deliberate — and so is stating it before seeing any results, which is the only honest moment to choose it. Two things worth remembering about that number:

It is a convention, not a law of nature. There is nothing about .05 that makes a departure real at .049 and imaginary at .051. It is an agreed-upon tolerance for how often we are willing to reject a true null, and fields differ: some areas of physics and genomics work at thresholds thousands of times stricter.
Two-sided is our default because both directions would be scientifically meaningful here. Theory predicts a direction in every test below, but a result in the unexpected direction is a finding we would want to see rather than one our test should be blind to. The prediction still does work; it just does interpretive work, telling us what a rejection means, rather than buying us a smaller critical value. To be clear, a one-sided alternative is not inherently less honest: it is perfectly defensible when the direction is fixed in advance and a result the other way would not support the claim anyway. What is indefensible is choosing the direction after seeing which way the data fell.

A related point that is easy to garble. All three t-tests below use two-sided alternatives. ANOVA and chi-square instead use upper-tail reference distributions — but that is a different thing from a directional one-sided alternative. Their statistics are non-negative by construction, so any departure from $H_0$, in whatever direction, pushes the statistic up; the upper tail is simply where all the evidence lives. Nothing directional has been assumed. Separately, Test 3’s follow-up comparisons adjust $\alpha$ to account for testing six pairs at once.

Two kinds of assumptions — and only one is visible in a plot

M08 introduced the three conditions a one-sample t-test rests on — independence, a well-approximated sampling distribution for the mean, and a sample that represents the population you mean to describe. Every test in this Module carries its own list, and the single most useful thing you can learn about such lists is that their entries come in two kinds. The order of operations matters too — this is not a ritual you perform after picking a test, it’s part of how you pick:

Understand the design → choose a model appropriate to it → examine whether the model is reasonable.

Design assumptions — independence of observations, random assignment, whether observations are paired or independent, each person appearing exactly once. These come from how the data were collected. They are the most consequential assumptions and no plot can check them — you verify them by knowing the design. A violated design assumption isn’t patched with a histogram; it calls for a different model.
Model considerations — is the mean a sensible summary of this outcome? Could a few extreme values dominate it? Is the sampling distribution of the statistic well approximated at this sample size (the CLT’s job)? Do the group variances differ enough that the standard-error model matters? For chi-square: are the expected cell counts adequate? These you assess with plots, summaries, and judgment.
Sensitivity — the quiet third check: would a robust alternative change the conclusion? Would the answer survive a different defensible threshold? If a conclusion flips under a small, reasonable perturbation, that fragility is a finding.

Each test below walks the first two in that order, asking only the questions that matter for that test — no generic checklists — and raises sensitivity where it genuinely bites.

How we’ll run things in R

The five tests use a deliberately mixed set of R tools, and the rule behind the mix is worth stating so it never feels arbitrary:

Analysis	Primary tool
One-sample, two-sample, and paired t	t_test() from infer
One-way ANOVA	lm() + anova() + broom::tidy()
Chi-square	chisq.test()
Standardized effect sizes	effectsize functions

The rule: we use the tidy wrapper where the analysis reduces cleanly to a t-test — it returns a one-row tibble you can pull reported values from, and it’s the same function you used in M08. We use the standard model-fitting functions where the model’s internals are themselves the lesson: ANOVA’s variance decomposition arrives via lm() + anova() (and previews M10), and chisq.test() hands you the expected counts and residuals you need to interpret which cells drive an association. The logic is unified even when the function changes — that’s the point.

How to read the five tests

Every test below uses the same six headings, in the same order:

Recognize the design — the raw ingredients first; the test’s name second.
State the parameter and hypotheses — in words before symbols.
See the data and assess the model — descriptives, one good plot, and the assumptions that actually matter.
Run and read the test — the R call, and what each output value means.
Quantify and visualize the effect — the raw effect first, then the standardized one.
Write the result and define its scope — the APA sentence, the plain-language sentence, and one sentence on what the analysis does not establish.

Read Test 1 carefully — it teaches the template at full length. For Tests 2 through 5, each section opens by naming what’s new relative to the previous test, and leans on the template for everything else. That’s the cumulative structure of the Module: not five chapters, one chapter with five variations.

One deliberate exception: Test 4 opens with a short warm-up before the six headings begin, meeting the chi-square machinery on a single categorical variable — the goodness-of-fit test — where it is easiest to see, before extending it to two variables at once.

The reference distribution — and why each test needs its own

This is the part of the Module that pays off two earlier ones, so it is worth slowing down for.

In M06 you learned what a probability distribution is: a complete account of which values a quantity takes and how often — a PDF you can integrate, a CDF you can query with pnorm(), a z-score that converts any value into “how many SDs from the center.” In M08 you put that to work: you built a null distribution — the set of results the world would produce if $H_0$ were true — and asked where your observed statistic fell in it.

M09’s whole job is to run that same move five times. And here is the sentence that organizes it:

Every test standardizes your data into a single number, and every test has a known distribution telling you what that number does when $H_0$ is true. That distribution is the reference distribution, and the design picks which one you get.

You have met one already. M08’s t-distribution, with its heavier-than-normal tails, chosen because we estimate $\sigma$ from the sample rather than knowing it. Its shape depends on degrees of freedom — small df, fat tails; large df, indistinguishable from the normal.

Degrees of freedom: quantities minus constraints

Every reference distribution in this Module is selected by its degrees of freedom, so it is worth having one idea that explains df everywhere instead of five separate rules. Here it is:

Degrees of freedom = the number of quantities that went into the calculation − the number of constraints they must satisfy.

A constraint is anything the data are forced to satisfy because you already estimated something from them. Estimating a mean and then measuring deviations from it costs you one piece of information: once you know the mean and all but one of the deviations, the last one is determined — it has no freedom left to vary.

That single sentence generates every df you will meet below:

Test	Quantities	Constraints	df
One-sample t	n observations	1 (the sample mean)	$n - 1$
Two-sample t (equal-variance form)	N observations	2 (two group means)	$N - 2$
One-way ANOVA, within	N observations	k (one mean per group)	$N - k$
One-way ANOVA, between	k group means	1 (the grand mean)	$k - 1$
Chi-square, $r \times c$ table	$rc$ cell counts	row and column totals	$(r-1)(c-1)$
Paired t	n difference scores	1 (the mean difference)	$n - 1$

The chi-square row is the one worth staring at, because you can see the constraint working. In a 2 × 2 table with the margins fixed, fill in any one cell and the other three are forced — so one cell is free, and $(2-1)(2-1) = 1$. In our 4 × 2 table, three cells are free.

One honest exception, flagged now so it doesn’t surprise you in Test 2: Welch’s t-test breaks this heuristic. Its df is a weighted compromise between two groups’ information, and it comes out fractional rather than a whole number — you’ll see the actual value when we get there. That is not a counting argument at all; it is an approximation chosen to make the reference distribution fit well. The principle covers the other cases, which is most of what you need.

This Module adds two more reference distributions, and they are genuinely new — you have not seen them before:

Test	Statistic	Reference distribution	How many df values pick the curve
One-sample, two-sample, paired t	t	t-distribution	1
One-way ANOVA	F	F-distribution	2 — a numerator df and a denominator df
Chi-square	$\chi^2$	$\chi^2$-distribution	1

Read that last column carefully, because it is easy to misread. It counts how many degrees-of-freedom numbers you must supply to select one specific curve — not what those numbers are. The t and χ² families are each indexed by a single value, so naming that one number picks the curve — Test 1 will use a t-distribution at $df = n - 1$, which for its sample is in the hundreds. The F family is indexed by a pair, and you need both: Test 3’s ANOVA lands on F(3, 906), and changing either number gives you a different curve entirely.

Two of these are non-negative by construction — F is a ratio of variances and $\chi^2$ is a sum of squares, so neither can go below zero. That single fact changes the picture: the t-tests have rejection regions in both tails, while ANOVA and chi-square have one, in the upper tail only. Their curves are right-skewed rather than symmetric.

Show the code that built this figure

panel <- function(grid, crit, title, two_tailed) {
  # Tag each tail so geom_area does not fill across the gap between them.
  reject <- if (two_tailed) {
    grid |> filter(x <= -crit | x >= crit) |> mutate(tail = if_else(x < 0, "L", "R"))
  } else {
    grid |> filter(x >= crit) |> mutate(tail = "R")
  }
  ggplot(grid, aes(x, y)) +
    geom_area(fill = "#9CBFAA", alpha = 0.55) +
    geom_area(data = reject, aes(group = tail), fill = "#945297", alpha = 0.55) +
    geom_line(linewidth = 0.7, color = "#1C2433") +
    labs(title = title, x = NULL, y = NULL) +
    theme(plot.title = element_text(size = 11.5, face = "bold", hjust = 0.5),
          axis.text.y = element_blank(), axis.ticks.y = element_blank(),
          panel.grid.minor = element_blank())
}

# The F and chi-square panels use the df these data actually produce in
# Tests 3 and 4 (computed just above), so the labels can never drift from
# the analyses. The t panel is deliberately generic (df = 30): its shape is
# illustrative, not tied to any one test.
t_grid <- tibble(x = seq(-4, 4, length.out = 500))  |> mutate(y = dt(x, 30))
f_grid <- tibble(x = seq(0, 8, length.out = 500))   |> mutate(y = df(x, f_df1_p1, f_df2_p1))
c_grid <- tibble(x = seq(0, 20, length.out = 500))  |> mutate(y = dchisq(x, chi_df_p1))

(panel(t_grid, qt(.975, 30), "t, df = 30\n(two-tailed)", TRUE) |
 panel(f_grid, qf(.95, f_df1_p1, f_df2_p1),
       sprintf("F, df = (%d, %d)\n(upper tail only)", f_df1_p1, f_df2_p1), FALSE) |
 panel(c_grid, qchisq(.95, chi_df_p1),
       sprintf("chi-square, df = %d\n(upper tail only)", chi_df_p1), FALSE)) +
  plot_annotation(
    title = "The three reference distributions in this Module",
    subtitle = "Green = values compatible with H0.   Heather = the rejection region at alpha = .05.",
    theme = theme(plot.background = element_rect(fill = "#FCFDFD", color = NA))
  )

Three panels showing the reference distributions used in this Module: a symmetric t-distribution at df 30, a right-skewed F-distribution at df 3 and 906, and a right-skewed chi-square distribution at df 3. The t panel shades both tails as rejection regions; the F and chi-square panels shade only the upper tail.

How to read any of these panels. The curve is what the test statistic does across many hypothetical samples when $H_0$ is true. The heather region is the set of values you would call “too extreme to be comfortable” — its area is exactly $\alpha$ = .05. The boundary is the critical value. And the p-value is the area beyond wherever your observed statistic actually landed. Those are two ways of reading one picture: is my statistic in the heather? and how much area is past it? They always agree.

The colours are the same ones M08 used, and they carry the same meanings throughout: heather marks the rejection region, the territory we are testing against; rose marks our own observed statistic. Every figure below is that pair.

Two symbols you need to keep apart

Those panels involve two numbers on the horizontal axis, and confusing them is one of the most common sources of trouble in a first inference course. Both are carried over from M08, and the notation is worth pinning down before we start using it in earnest:

	What it is	Where it comes from	How we write it
Observed statistic	the single number your data produced	your sample — you cannot know it until you collect and compute	$t_\text{obs}$, and by extension $F_\text{obs}$, $\chi^2_\text{obs}$
Critical value	the cutoff for “extreme enough to reject”	$\alpha$, the reference distribution, and its df — fixed before you look at your data	$t^$, $F^$, $\chi^{2*}$

The shorthand is easy once you see the logic behind it: the asterisk marks a threshold, and the “obs” subscript marks your result. So $t^*$ is a property of your decision rule; $t_\text{obs}$ is a property of your data. The whole test is then a single comparison between them — reject when the observed value passes the starred one.

Two practical notes. First, in running prose we will often just say “our F of 6.57” rather than writing $F_\text{obs}$ every time; the subscript is there when the contrast with $F^*$ needs to be explicit. Second, this is exactly the notation M08 used — and $t^*$ is also the multiplier you met in M07, where $\bar{x} \pm t^* \cdot SE$ built a confidence interval. That is not a coincidence, and it is the reason a CI and a test agree: they are built from the same $t^*$.

Every test below draws this picture with its own distribution, its own df, and your observed statistic marked on it. That figure is the moment where M06’s distributions and M08’s null-world reasoning become the thing you actually use.

Critical values across df.

The p-value framing and the critical-value framing are two reads of the same decision. Instead of computing the tail probability of your observed statistic, you can fix $\alpha$ first and ask: how extreme would the statistic need to be to reject? That cutoff is the critical value ($t^*$, $F^*$, $\chi^{2*}$), and the region beyond it is the rejection region. The two framings always agree — reject when $p \leq \alpha$ is the same decision as reject when the statistic passes its critical value.

For a two-sided t-test at $\alpha = .05$, the critical value puts $\alpha/2 = .025$ in each tail. A small lookup across df:

Show the code that built this table

# Critical t-values for a two-sided test at alpha = .05, across several df.
tibble(df = c(5, 10, 25, 50, 100, 500, Inf)) |>
  mutate(
    `t* (α = .05, two-sided)` = qt(p = 0.975, df = df) |> round(3)
  )

Notice the pattern: as df grows, $t^*$ shrinks toward 1.96 (the analogous z-value). Smaller samples demand more extreme statistics for the same $\alpha$ — the formal version of M08’s lesson that smaller $n$ means lower power.

Show the code that built this figure

# A generic illustrative df: visibly t-shaped (heavier tails than z),
# round number, NOT tied to any specific worked example. The same shape
# applies to any df, just compressed toward z as df grows.
df_demo  <- 30
t_star   <- qt(p = 0.975, df = df_demo)

curve_df <- tibble(t = seq(-4, 4, length.out = 400)) |>
  mutate(density = dt(t, df = df_demo),
         region = case_when(
           t <= -t_star ~ "reject_left",
           t >=  t_star ~ "reject_right",
           TRUE         ~ "fail_to_reject"
         ))

curve_df |>
  ggplot(aes(x = t, y = density, fill = region)) +
  geom_area(alpha = 0.7, position = "identity") +
  geom_line(linewidth = 0.8, color = "#1C2433") +
  geom_vline(xintercept = c(-t_star, t_star),
             color = "#945297", linewidth = 0.9, linetype = "dashed") +
  annotate("label", x = c(-t_star, t_star), y = 0.18,
           label = c(sprintf("t* = -%.2f", t_star), sprintf("t* = +%.2f", t_star)),
           color = "#945297", fontface = "bold", fill = "white", size = 3.6) +
  scale_fill_manual(values = c(
    "reject_left" = "#945297",
    "fail_to_reject" = "#9CBFAA",
    "reject_right" = "#945297"
  ), guide = "none") +
  labs(
    # Markdown titles via ggtext: <br> breaks the line and *...* italicises,
    # and unlike plotmath's atop() both lines stay flush left.
    title = "Rejection region for a two-sided *t*-test at &alpha; = .05",
    subtitle = paste0(
      "Illustrative — at *df* = 30, the heavier tails of the ",
      "*t*-distribution are clearly visible.<br>",
      "The same shape applies to any *df*, compressed toward *z* as *df* grows."
    ),
    x = "t",
    y = "density"
  ) +
  theme(
    plot.title = ggtext::element_markdown(hjust = 0),
    plot.subtitle = ggtext::element_markdown(hjust = 0, lineheight = 1.3)
  )

A t-distribution density curve at df 30 with its two tails, beyond the dashed critical values near minus and plus 2.04, shaded heather to mark the rejection region for a two-sided test at alpha .05; the central region is shaded green as the fail-to-reject region.

The shape changes with the test — ANOVA’s F and the chi-square statistic are non-negative, so their rejection regions live in the upper tail only — but the decision logic is identical: reject $H_0$ when the observed statistic lands in the shaded region.

The 95% CI and the two-sided test are the same decision.

For the three t-tests in this Module, an elegant identity holds: the 95% CI and the two-sided test at $\alpha = .05$ always give the same decision — not empirically, but algebraically. Both lean on the same critical value $t^*$:

\[\underbrace{\bar{x} \pm t^* \cdot SE}_{\text{the 95\% CI}}, \qquad \underbrace{\text{reject } H_0 \text{ iff } |t_\text{obs}| > t^*}_{\text{the two-sided NHST}}\]

The CI excludes the null value exactly when $|\bar{x} - \mu_0|/SE > t^*$ — which is literally the rejection condition. The two statements are therefore equivalent, which is what the shorthand iff in that expression is claiming.² So: the one-sample CI excludes $\mu_0$ iff the one-sample test rejects; the CI on a difference of means excludes 0 iff the two-sample test rejects; the CI on the mean within-person difference excludes 0 iff the paired test rejects.

For ANOVA and chi-square the match is looser — a family resemblance rather than an identity. Those tests ask about several groups at once, so there is no single interval to line the test up against. For ANOVA, the clean correspondence reappears one level down: a Tukey-adjusted pairwise CI excludes 0 exactly when that Tukey-adjusted comparison reaches significance. For chi-square, the CI on Cramér’s V is an effect-size interval — it usually excludes 0 when the test rejects because both respond to the same evidence, not because they are the same statement.

This identity is one reason reporting guidelines push for CI alongside p-value: the CI contains the test’s decision and tells you where the parameter plausibly is. (It is also exact only for the parametric route — M08’s bootstrap CI and bootstrap p-value can disagree slightly, because the percentile CI comes from an unshifted resampling distribution while the p-value counts tails of a shifted one.)

Test 1: One mean against a benchmark — the one-sample t-test

Test 1 teaches the template

This first test runs at full length, because it carries the six-step template every later test reuses. Two things to know before we start. The machinery is M08’s — the one-sample t is the very test that Module built from scratch, so nothing here re-derives it; we practice using it well. And the analysis is course-designed: Hofman et al.’s preregistered hypotheses compared visualization formats to each other (that’s Test 2), while this test formalizes a descriptive pattern the authors note along the way — that participants paid more than the rational price in every condition. Formalizing someone’s descriptive claim as a test is a genuinely useful research move; it just isn’t the paper’s central contrast, and we won’t pretend it is.

Recognize the design

Start where every analysis should start — with the ingredients, not the test name:

The outcome is willingness to pay, in Ice Dollars: one continuous measurement per participant.
There is one group. We’ll look at the participants in Experiment 1 who saw a PI display with a plain matching caption — a single arm of the study, no comparison group in sight.
The comparison is external: a fixed benchmark from theory, the risk-neutral rational price for renting the special boulder.
Observations are independent — each MTurk worker was randomly assigned and appears once.
The parameter in question is that group’s population mean, $\mu$.

One continuous outcome, one group, one external benchmark, independent observations, a question about a mean. The decision map gives the answer: a one-sample t-test.

This is the pattern to practice on every dataset you ever meet: name the outcome type, count the groups, check independence, identify the parameter — then the test names itself.

State the parameter and hypotheses

Before symbols, we need the benchmark — because a one-sample test is only as meaningful as the number it tests against.

Where $17.50 comes from. The game is winner-take-all: the contestant whose boulder slides farthest takes home the entire $250 Ice-Dollar prize. Experiment 1 ran only the small effect size — the large-effect condition is an Experiment 2 addition — so throughout these data the standard boulder beats Blorg’s 50% of the time and the special boulder beats Blorg’s 57% of the time. Renting the special boulder therefore buys a 7-percentage-point increase in win probability — and a 7-point increase in expected prize money. For a risk-neutral player — someone who values an expected gain exactly at its dollar value — the most the upgrade is worth is the change in expected winnings:

\[\mu_0 \;=\; \$250 \times (P_\text{special} - P_\text{standard}) \;=\; \$250 \times (0.57 - 0.50) \;=\; \$17.50\]

Hofman et al. report this benchmark explicitly: a risk-neutral player should pay at most $17.50, and refuse to pay more.

Now the hypotheses — and first, be precise about what $\mu$ refers to. It is the population mean willingness to pay, in Ice Dollars, among people shown a PI display with matching caption text — not all participants, not all displays. Naming the population is half of stating a hypothesis well.

\[H_0: \mu = \$17.50 \qquad\qquad H_a: \mu \neq \$17.50\]

The same two statements in words, because the symbols alone don’t say what is at stake:

$H_0$ — the population mean WTP in this condition equals the risk-neutral expected-value benchmark of $17.50. That is the average a room full of risk-neutral expected-value maximizers would produce. Notice this is not a straw man³ — it is the prediction of a specific and respectable theory of decision-making, the one a standard economics course would hand you.
$H_a$ — the population mean WTP differs from that benchmark. The group’s average payment sits somewhere other than the expected-value price.

Be careful about what these statements are and aren’t. They are claims about one number — a population mean — not about how any individual decides. Even if the mean landed exactly on $17.50, that would not establish that participants were individually risk-neutral: some could overpay and others underpay, averaging out. And a mean above $17.50 establishes an average departure from the benchmark, not the psychology producing it. A one-sample test is a test about a mean; keeping the hypothesis that modest is what keeps the conclusion honest.

Which way do we expect it to go? Up. People tend to overweight an upside they can picture, to anchor on round numbers, and to treat a “better boulder” as worth more than the arithmetic says. So the substantive prediction is $\mu > \$17.50$. We nevertheless test two-sided at $\alpha = .05$, the same convention M08 used — a one-sided test buys us nothing here, and if players had under-paid we would certainly want to see it. What the prediction buys us is interpretive: if we reject and the sample mean sits above $17.50, that is over-payment specifically, not merely “a difference.”

And what would rejecting mean? That the risk-neutral benchmark fails to describe this group’s average behavior — and, given the direction, that they pay systematically more than the upgrade is worth.

See the data and assess the model

Experiment 1’s data live in their own file. The design was a 2 × 2: interval format (CI vs PI) crossed with caption text. In the “matching-text” arm, the caption reported only the statistics shown in the figure the participant saw; in the “extra-text” arm, it reported both the CI and the PI statistics regardless of which figure the participant saw.

blorg_exp1 · 1,743 observations · 5 variables · 2 × 2 between-subjects factorial · Hofman, Goldstein, & Hullman (2020), Experiment 1

A cleaned, course-prepared subset of Hofman, Goldstein, & Hullman (2020), Experiment 1 (data/blorg_exp1.Rds).

worker_id character — Anonymized Amazon Mechanical Turk worker identifier
condition.f factor — Four-level experimental condition crossing visualization format and caption text
interval_CI numeric — Visualization format shown to the participant
text_extra numeric — Whether the caption supplied information beyond the visualization
wtp_final numeric — Willingness to pay to rent the special boulder, in Ice Dollars

Full codebook for blorg_exp1 — values, levels, missingness, and how the file was prepared.

blorg_exp1 <- read_rds(here("shared_data", "blorg_exp1.Rds"))

blorg_exp1 |> glimpse()

Rows: 1,743
Columns: 6
$ worker_id   <hash> 75076b89f92c7cd0ccb18570daee59d3, 938460cb4b4b7a3de2fd845…
$ condition.f <fct> 2: CI with extra info, 4: PI with extra info, 1: CI with v…
$ interval_CI <dbl> 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1…
$ text_extra  <dbl> 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0…
$ wtp_final   <dbl> 0, 50, 50, 86, 50, 200, 100, 125, 25, 0, 25, 50, 50, 100, …
$ superiority <dbl> 0.60, 0.55, 0.90, 0.95, 0.95, 0.90, 0.90, 0.50, 0.55, 0.55…

We filter to the single arm our question concerns — participants who saw a prediction-interval display with matching text:

pi_matching <- blorg_exp1 |>
  filter(interval_CI == 0, text_extra == 0) |>
  select(wtp = wtp_final)

pi_matching |>
  summarize(
    n = n(),
    M = mean(wtp) |> round(2),
    SD = sd(wtp) |> round(2),
    SE = (sd(wtp) / sqrt(n())) |> round(2)
  )

So 486 participants reported a mean willingness to pay of M = $49.64 (SD = $49.38, SE = $2.24) — already sitting well above $17.50. The test will tell us whether that gap is large relative to the sampling uncertainty in a mean at this sample size.

Design assumptions first, checked against how the data were collected:

Independence. Each MTurk worker was randomly assigned to one condition and appears once. ✓
One observation per person, one group. ✓ — the design is exactly what a one-sample test wants.

Model considerations next, checked against the data:

Histogram of willingness to pay in the PI matching-text arm of Hofman Experiment 1: right-skewed and spiky at round-number anchors, tapering off toward a maximum response of $249.

Is the mean a sensible summary? The raw distribution is right-skewed and clumped at round numbers — but no single value dominates. The largest response is $249, and a 10%-trimmed mean ($41.79) sits somewhat below the untrimmed mean ($49.64) because of broad skew, not a few stray points. Worth remembering when interpreting the mean; not a reason to distrust the test. ✓
Is the sampling distribution of $\bar{x}$ well approximated? The assumption is about the mean, not the raw scores — and at n = 486, the Central Limit Theorem (CLT) does that work even though the raw distribution is skewed. ✓

Sensitivity, briefly: with 486 observations and no isolated dominating values, the conclusion is unlikely to be sensitive to a reasonable robust alternative (see the box at the end of this test). We are not claiming to have run one — if a result were borderline, you would run it and report both.

Run and read the test

M08 derived the machinery in full — the standard error, $t_\text{obs}$, the reference distribution. Nothing changes, so we quote the recipe:

\[t_\text{obs} = \frac{\bar{x} - \mu_0}{SE}, \qquad SE = \frac{s}{\sqrt{n}}, \qquad df = n - 1\]

Read it as a ratio of two distances. The numerator is the gap you actually observed — how far this sample’s mean sits from the benchmark, in Ice Dollars. The denominator is how far a sample mean would typically wander from the truth just by chance at this sample size. So t answers: how many “typical wanderings” is my observed gap?

Now put our numbers in. The sample mean was $49.64, the benchmark is $17.50, and the standard error is $2.24:

\[t_\text{obs} \;=\; \frac{\bar{x} - \mu_0}{SE} \;=\; \frac{49.64 - 17.50}{2.24} \;\approx\; 14.35\]

In R, that arithmetic and the two numbers we need to judge it are four short lines. Everything comes straight from the summary you computed above — nothing here is magic, and the figure below is built from exactly these values:

test1_pieces <- pi_matching |>
  summarize(
    n_obs  = n(),
    M      = mean(wtp),
    SE     = sd(wtp) / sqrt(n()),
    # the observed statistic: the gap, measured in standard errors
    t_obs  = (M - 17.50) / SE,
    # quantities minus constraints
    df     = n_obs - 1,
    # the two-tailed critical value: the t that leaves .025 in the upper tail
    t_star = qt(0.975, df = df),
    # the p-value: area beyond our statistic, counted in BOTH tails
    p      = 2 * pt(-abs(t_obs), df = df)
  )

test1_pieces

Two functions there are worth naming, because you will reach for them constantly. qt() is the t-distribution’s quantile function — hand it a probability, it returns the cutoff. pt() is the cumulative function running the other way — hand it a cutoff, it returns the area below. They are the t-family versions of the qnorm() and pnorm() you met in M06, and between them they generate every number in this section: qt() gives the critical value, pt() gives the p-value.

So this sample’s mean sits about 14 standard errors above the risk-neutral price. Hold onto that number — everything that follows is about deciding whether it is large.

Now the reference distribution — this test’s own. Our df is $n - 1 = 485$ — and now the counting principle earns its keep. We put 486 observations into the calculation and imposed one constraint on them: the sample mean, which we estimated from those same observations in order to measure deviations from it. Quantities minus constraints, 486 − 1 = 485. So the curve that describes what t does when $H_0$ is true is a t-distribution with 485 degrees of freedom. At that many df it is visually indistinguishable from the standard normal, and the two-tailed critical value is $t^* = 1.965$ — essentially the 1.96 you know from M06’s Empirical Rule.

Here is where our statistic falls on it:

Show the code that built this figure

ref_dist_plot(
  dens = function(x) dt(x, df = df_1),
  crit = c(-t_star_1, t_star_1),
  obs  = t_obs_1,
  xlim = c(-16, 16),
  stat_label = "t",
  subtitle = sprintf(
    "t-distribution, df = %d\nHeather = rejection region (alpha = .05, two-tailed), boundaries at t* = %s%.2f",
    df_1, "\u00b1", t_star_1)
)

The t-distribution with 485 degrees of freedom, spanning roughly minus 16 to plus 16, with narrow heather rejection regions in both tails beyond plus and minus 1.97. The observed t of 14.35 is marked far out in the right tail, well beyond the critical value.

Notice the axis. To fit our observed t on the same picture, the plot has to run out to ±16 — which squashes the entire distribution into a narrow spike near zero. That squashing is the result. Under $H_0$, essentially all the probability sits between −4 and +4; our statistic landed at 14.35. The p-value is the area beyond that mark, and there is almost none of it.

infer’s t_test() runs the one-sample form when you give it a response but no formula. Pass mu = 17.50 to anchor the null:

t_result_1 <- pi_matching |>
  t_test(
    response = wtp,
    mu = 17.50,
    alternative = "two-sided"
  )

t_result_1

Study the output before you read on. Across the analyses in this Module you will be looking for the same handful of quantities. Find each one in the tibble above and say to yourself what it means here:

the estimate — what parameter is being estimated, and what’s its value?
the test statistic — in which direction, and how far from what $H_0$ predicts?
the degrees of freedom — what determines them for this test?
the p-value — compatible or incompatible with $H_0$?
the confidence interval — for which quantity, and does it contain $\mu_0$?

Check your answers

estimate = 49.64 — the sample mean WTP, in Ice Dollars. The parameter being estimated is $\mu$, this arm’s population mean.
statistic = 14.35 — the observed gap is about 14 standard errors above the benchmark, matching the hand calculation we did just above.
t_df = 485 — $n - 1$: a whole number, because there is a single group and a single mean to estimate.
p_value < .001 — a sample mean this far from $17.50 would essentially never arise if the population mean really were $17.50. The data are highly incompatible with $H_0$. We reject.
95% CI = [$45.24, $54.05] — the plausible range for $\mu$ itself. It sits entirely above $17.50, which is the interval form of the same decision — and it tells you by how much, which the p-value never does.

Three readings, one decision — and they agree by construction

Notice that the output just gave you three separate-looking reasons to reject $H_0$:

Reading	Here	Verdict
The statistic clears the critical value	$\lvert t_\text{obs} \rvert = 14.35$ against $t^* = 1.96$	beyond it
The p-value falls below $\alpha$	p < .001 against $\alpha = .05$	below it
The confidence interval excludes the null value	[$45.24, $54.05] against $\mu_0$ = $17.50	excludes it

These are not three independent confirmations. It is tempting to read agreement across three rows as three pieces of evidence stacking up, and it isn’t. They are one comparison written three ways, and all three are built from the same two ingredients: the observed statistic and the critical value $t^*$.

Follow the chain. The p-value is the tail area beyond $t_\text{obs}$; $\alpha$ is the tail area beyond $t^*$. So p < $\alpha$ is $|t_\text{obs}| > t^*$ — the same sentence, one written in areas and the other in distances. And the interval is $\bar{x} \pm t^* \cdot SE$, so it clears $\mu_0$ at exactly the moment the gap $|\bar{x} - \mu_0|$ grows past $t^* \cdot SE$ — which is that same comparison a third time, rearranged. Part 1’s box on the CI and the test works the algebra through.

The practical upshot: if these three ever disagree in your own output, you have not found something interesting — you have made an error. The usual culprits are comparing against the wrong $\alpha$, reading a one-sided interval beside a two-sided test, or pairing a CI with a p-value that came from a different method (M08’s bootstrap CI and bootstrap p-value can part company for exactly this reason).

One scope note for later: the p-value-and-critical-value pair agree for every test in this Module. The CI leg is the special privilege of the three t-tests, because each has a single parameter to build an interval around. ANOVA and chi-square ask about several groups at once, so there is no one interval to line them up against — Tests 3 and 4 will say more.

Quantify and visualize the effect

Report the effect in raw units first — they need no translation. The mean is $49.64 against a benchmark of $17.50: participants paid about $32 more than the rational price, roughly 2.8 times the benchmark, with a 95% CI on the mean of [$45.24, $54.05].

Then the standardized companion. Cohen’s d for a one-sample design is the same gap scaled by the standard deviation rather than the standard error — $d = (\bar{x} - \mu_0)/s$ — so unlike t, it doesn’t mechanically grow as $n$ grows. (effectsize::cohens_d() returns the point estimate with a 95% CI; report both — the point estimate alone hides how precisely d was measured.)

cohens_d_1 <- pi_matching |>
  pull(wtp) |>
  cohens_d(mu = 17.50)
cohens_d_1

Cohen’s d = 0.65, 95% CI [0.55, 0.75] — a medium effect by the conventional labels, estimated precisely enough that its CI sits comfortably away from zero. (What d means visually — overlap between distributions, and what “medium” buys you — is developed properly in Test 2, where there are two real groups to draw.)

Write the result and define its scope

APA-style reporting · One-Sample t-Test

PI matching-text participants reported substantially higher willingness to pay (M = $49.64, SD = $49.38, n = 486) than the risk-neutral break-even price of $17.50, t(485) = 14.35, p < .001, mean WTP 95% CI [$45.24, $54.05], Cohen’s d = 0.65, 95% CI [0.55, 0.75].

Note the structure, because every test below reuses it:

descriptives → test statistic → degrees of freedom → p-value → CI on the parameter being tested → effect size with its own CI

The opening parenthetical names the parameter in plain language, so a reader who skips the equation still knows what the benchmark is. The second CI (on d) reports the precision of the effect-size estimate — a different question from the precision of the mean.

In plain language: even participants shown the plainest display with the plainest caption paid, on average, nearly three times the rational price for the special boulder.

What this analysis does not establish. Departing from a risk-neutral benchmark is not, by itself, proof of irrationality — participants may be risk-seeking, may value the thrill of winning nonlinearly, may misread the fictional currency, or may be using sensible heuristics for an unfamiliar task. A one-sample mean comparison detects the departure; it cannot distinguish among the psychological mechanisms behind it. And because this test looks at a single arm, it says nothing about whether the visualization matters — that comparison needs two randomized groups, which is exactly where Test 2 picks up.

When the standard analysis is unsuitable: Wilcoxon signed-rank (one-sample form).

First, what a “rank-based” test is, since several of the alternatives in this Module are described that way. To rank data is to throw away the numbers and keep only the order: line the values up smallest to largest and replace each one with its position. Willingness-to-pay values of $5, $20, $25, and $250 become ranks 1, 2, 3, and 4 — the $250 response is still the largest, but it is now only one step above $25 rather than ten times its size.

That single move is the whole idea, and it buys two things. A wild outlier can no longer drag the analysis around, because the biggest value contributes rank n whether it is $250 or $250,000. And the ranks of any sample are always just the numbers 1 through n, whatever shape the raw data had — so the test’s reference distribution can be worked out without assuming Normality at all. That is what makes these tests nonparametric: they don’t lean on a model for the population’s shape.

The cost is the flip side of the benefit. Throwing away the spacing between values means you can no longer say anything about a mean, because means depend on exactly the magnitudes you discarded. So rank-based tests answer a genuinely different question — which is the caution below, and the reason we don’t treat them as drop-in substitutes.

If the raw distribution is heavily skewed and the sample is too small for the CLT to cover the mean (typically n < 20–30), the Wilcoxon signed-rank test is the usual alternative. One caution before reaching for it: it does not test the same parameter. Strictly, it tests whether the distribution of $X - \mu_0$ is symmetric around zero — which coincides with a test on the median only under symmetry. Don’t switch tests just because a histogram looks non-Normal; first decide whether your research question is about the mean (stay parametric if the CLT plausibly covers you) or is equally well served by a location shift.

pi_matching |> pull(wtp) |> wilcox.test(mu = 17.50)

With n = 486, the one-sample t is the right choice here; the Wilcoxon is on the shelf for the small-sample case.

Test 2: Two randomized groups — the two-sample t-test

What’s new in Test 2

One change of design, three consequences. The design now has two independent groups, so: (1) the parameter is no longer one mean but a difference between two population means; (2) the standard error must combine two groups’ worth of uncertainty (Welch’s formula); and (3) the degrees of freedom stop being a whole number. The logic, the template, and everything else carry over from Test 1 unchanged.

Recognize the design

Test 1 looked at one arm of the study and asked whether its mean was calibrated to a benchmark. It could not, even in principle, say anything about whether the visualization matters — for that, you need groups to compare. Experiment 2 provides them, and this is the paper’s own preregistered question: do readers respond differently to a CI display than to a PI display?⁴

The ingredients:

The outcome is willingness to pay — still one continuous measurement per participant.
There are two groups considered here, and participants were randomly assigned to them: some saw the CI display, others the PI display.
Observations are independent — different participants in each group, each appearing once.
The parameter in question is the difference between the two population means.

Continuous outcome, two independent groups, a question about a difference of means: a two-sample t-test — in its modern default form, Welch’s t-test.

As set out in Meet the data, we work within the Large effect-size condition, so both groups face the identical decision problem (a special boulder genuinely worth $65). Random assignment then leaves only two explanations for a group difference — the display, or sampling variability — and separating those two is exactly the test’s job:

ci_pi <- blorg_exp2 |>
  filter(effect_size == "Large", graph_type %in% c("CI", "PI")) |>
  select(graph_type, wtp) |>
  drop_na() |>
  mutate(graph_type = fct_relevel(droplevels(graph_type), "CI", "PI"))

State the parameter and hypotheses

In words first: the parameter is $\mu_{CI} - \mu_{PI}$, the population difference in mean willingness to pay between readers assigned to the CI display and readers assigned to the PI display, in the large-effect condition. Writing the null as a difference equal to zero keeps continuity with Test 1 — it is still one parameter tested against one benchmark; the benchmark is now 0, and the parameter is a difference:

\[H_0: \mu_{CI} - \mu_{PI} = 0 \qquad\qquad H_a: \mu_{CI} - \mu_{PI} \neq 0\]

$H_0$ — the display makes no difference to average willingness to pay. Readers of either chart land in the same place, on average.
$H_a$ — the display shifts average willingness to pay, in one direction or the other.

Which way do we expect it to go? The paper’s argument predicts $\mu_{CI} > \mu_{PI}$: a CI shows uncertainty about the mean, which is narrow at large n, so a CI display makes the special boulder’s advantage look tight and dependable — while a PI shows the spread of individual outcomes, which stays wide. A reader anchored on the tight interval should value the upgrade more. As always, we test two-sided at $\alpha = .05$ and let the prediction do interpretive work: rejecting with $\bar{x}_{CI}$ on top is evidence for the paper’s account specifically.

See the data and assess the model

ci_pi |>
  group_by(graph_type) |>
  summarize(
    n = n(),
    M = mean(wtp) |> round(2),
    SD = sd(wtp) |> round(2),
    SE = (sd(wtp) / sqrt(n())) |> round(2)
  )

CI viewers report a mean WTP of $86.69 (SD = $62.7); PI viewers report $64.67 (SD = $53.57). The observed gap is $22.01. The question for the rest of this section is whether a gap that size would commonly arise from sampling variation if the two population means were equal — and then, how big it is in standardized terms.

Show the code that built this figure

ci_pi |> ggplot(aes(x = graph_type, y = wtp, fill = graph_type)) +
  geom_violin(alpha = 0.35, color = NA) +
  geom_boxplot(width = 0.15, fill = "white", outlier.alpha = 0.3) +
  stat_summary(fun = mean, geom = "point", shape = 21, size = 3,
               fill = "#AD872B", color = "#AD872B") +
  scale_fill_manual(values = c("CI" = "#4E5EAA", "PI" = "#3A8055"),
                    guide = "none") +
  labs(title = "Willingness to pay by visualization condition",
       subtitle = "Gold dot = group mean",
       x = "Visualization type",
       y = "Willingness to pay (Ice Dollars)")

Violin-and-boxplots of willingness to pay for the CI and PI conditions, each with a gold dot at the group mean; the CI condition's distribution and mean sit higher than the PI condition's.

The CI condition sits visibly higher — and yet look how much the two distributions overlap: most CI viewers are indistinguishable from most PI viewers, even though the means clearly differ. Hold onto that observation; it is exactly what Cohen’s d will quantify below.

Design assumptions. Independence holds by design — each participant was randomly assigned to exactly one condition and appears once. And random assignment is doing quiet, crucial work here that no statistic can do: it is what makes the two groups comparable, in expectation, in every respect except the display.

Model considerations. Both distributions are right-skewed and clumped at round numbers — but with 231 and 236 participants, the CLT covers the sampling distribution of each mean, and therefore of their difference. On variances: the two SDs ($62.7 and $53.57) are similar but not identical, and we don’t need to adjudicate — Welch’s t, the modern default, doesn’t assume equal variances. It costs essentially nothing when variances are equal and protects you when they aren’t, which is why we use it by default rather than gatekeeping the analysis on a preliminary variance test (those tests carry their own error rates, and “test, then pick your test” quietly distorts $\alpha$).

Run and read the test

The new machinery, in two short pieces.

First, the idea that will carry you through Test 3 as well: signal against noise.

Every test statistic in this Module is a fraction, and it is worth naming what lives upstairs and downstairs:

\[\text{test statistic} \;=\; \frac{\textbf{signal — how far apart the groups are}}{\textbf{noise — how much individuals vary anyway}}\]

In Test 1 the “signal” was one mean’s distance from a benchmark, and the “noise” was how much a single mean wanders by chance. Here the signal is the gap between two group means — variability between groups — and the noise is built from how much participants scatter within their own group.

That between/within split is the engine of this test and, in Test 3, of ANOVA. Hold onto it in this form: a difference between group means is only impressive relative to how much people differ from each other inside a group. A $22 gap between two conditions means something quite different if individuals within a condition typically vary by $5 than if they typically vary by $60. The denominator is what supplies that context — which is why a test statistic, not a raw difference, is what gets compared to a distribution.

Quantifying the noise: the pooled standard deviation. Before we can divide by the noise, we have to measure it — and with two groups there is a decision to make that Test 1 never faced. We now have two estimates of within-group spread, $s_{CI}$ and $s_{PI}$. How do we get one number out of two?

The classical answer is to pool them, and the cleanest way to see what pooling does is to stop thinking of it as averaging two SDs and start thinking of it as computing one variance around two centers. Recall how an ordinary variance works: take each observation’s deviation from the mean, square it, add them up, divide. The pooled variance does exactly that — except each participant’s deviation is measured from their own group’s mean rather than from a single overall mean:

\[s_p^2 \;=\; \frac{\sum_{\text{all participants}} (x_i - \bar{x}_{\text{their group}})^2}{N - 2}\]

And now look at the denominator, because it is the counting principle again. We have $N = 467$ observations and we estimated two means from them in order to measure those deviations — so two constraints, and $N - 2 = 465$ degrees of freedom. Test 1 divided by $n - 1$ because it estimated one mean; here we estimate two, so we divide by $N - 2$. Nothing else changed.

Written in terms of the two group SDs you already have, that same quantity is a weighted average of the two variances, each weighted by its group’s df:

\[s_p^2 \;=\; \frac{(n_{CI} - 1)\,s_{CI}^2 + (n_{PI} - 1)\,s_{PI}^2}{N - 2}\]

# Pull each group's n and SD out of the data, so nothing here is a mystery
# object: these are the same numbers as the summary table above.
grp2 <- ci_pi |>
  group_by(graph_type) |>
  summarize(n = n(), SD = sd(wtp), .groups = "drop")

grp2

# The weighted average of the two variances, each weighted by its own df.
s_pooled_sq <- sum((grp2$n - 1) * grp2$SD^2) / (sum(grp2$n) - 2)

sqrt(s_pooled_sq)

[1] 58.26184

So the typical distance between a participant and their own group’s mean is about $58.26. That is the noise floor, in Ice Dollars — and it is worth fixing in memory, because in Test 3 this exact quantity comes back under a different name, computed the same way across four groups instead of two.

The standard error of a difference. Test 1 used one SE, $s/\sqrt{n}$. A difference of two independent means has more uncertainty than either mean alone — the variances add — so:

\[SE_\text{diff} \;=\; \sqrt{\frac{s_{CI}^2}{n_{CI}} + \frac{s_{PI}^2}{n_{PI}}} \;=\; \sqrt{4.13^2 + 3.49^2} \;\approx\; 5.4\]

# ci_M / ci_SE / ci_n -- and their PI counterparts -- are the group means,
# standard errors, and sample sizes from the summary table above.

# Each group keeps its own variance; the two contributions add under the root.
SE_diff <- sqrt(ci_SE^2 + pi_SE^2)

# Welch-Satterthwaite: NOT a count of anything, which is why it comes out
# fractional. This is the one df in the Module you do not derive by counting.
welch_df <- (ci_SE^2 + pi_SE^2)^2 /
            (ci_SE^4 / (ci_n - 1) + pi_SE^4 / (pi_n - 1))

t_obs_2  <- (ci_M - pi_M) / SE_diff
t_star_2 <- qt(0.975, df = welch_df)                 # two-tailed cutoff
p_obs_2  <- 2 * pt(-abs(t_obs_2), df = welch_df)     # both tails, so doubled

c(SE_diff = SE_diff, df = welch_df,
  t_obs = t_obs_2, t_star = t_star_2, p = p_obs_2) |> round(4)

 SE_diff       df    t_obs   t_star        p 
  5.4014 450.8369   4.0757   1.9652   0.0001

Same two functions as Test 1 — qt() for the cutoff, pt() for the tail area — just handed a different, fractional df.

This is Welch’s form — notice that the pooled SD we just built does not appear in it. Each group keeps its own variance, divided by its own n. That is the whole difference between the two versions of this test:

Student’s t-test assumes the two populations share one standard deviation, so it uses the pooled estimate for both groups: $SE_\text{diff} = s_p\sqrt{1/n_{CI} + 1/n_{PI}}$, with $df = N - 2 = 465$ — the counting principle, exactly as derived above.
Welch’s t-test makes no such assumption, so it lets each group’s own variance stand.

The two agree closely when the group variances are similar and diverge when they aren’t, which is why Welch is the modern default: it costs almost nothing in the equal-variance case and protects you in the unequal one.

So why did we build the pooled SD at all, if Welch never touches it? Two reasons, and it is worth being blunt about this rather than leaving you wondering. First, it is the quantity Student’s version of this test divides by — so seeing it is how you understand what Welch is choosing not to assume. Second, and more importantly for what comes next, it is the direct ancestor of $MS_\text{within}$, the noise term ANOVA is built on. Test 3 will compute exactly this quantity across four groups instead of two.

You can see both versions with one argument. t_test() passes var.equal straight through to base R, so setting it to TRUE swaps Welch’s separate variances for the pooled estimate we just built:

bind_rows(
  ci_pi |>
    t_test(formula = wtp ~ graph_type, order = c("CI", "PI")) |>
    mutate(version = "Welch (the default)"),
  ci_pi |>
    t_test(formula = wtp ~ graph_type, order = c("CI", "PI"), var.equal = TRUE) |>
    mutate(version = "Student (pooled SD)")
) |>
  select(version, statistic, t_df, p_value)

The df is the tell: Welch’s fractional 450.8 against Student’s whole-numbered $N - 2 = 465$. Here the two barely differ, because these groups have similar SDs — which is exactly the situation in which Welch costs you nothing. Leave var.equal alone and you get Welch, which is the right default; we show the switch so you know what the other version is doing, not because you should reach for it.

Welch’s degrees of freedom. Dropping the equal-variance assumption costs something, and this is where it shows up. With two groups contributing unequally to the SE, df is no longer a count of anything — it becomes a weighted compromise, the Welch–Satterthwaite approximation, chosen so that the t-distribution fits the statistic’s actual behavior as closely as possible. It lands between the smaller group’s $n - 1$ and Student’s $N - 2$, and it comes out fractional: for our data, 450.8 rather than 465. This is the exception flagged back in Part 1 — a fractional df looks odd the first time, but it is routine, and software handles it invisibly.

With those two pieces, the statistic is Test 1’s recipe with the new SE:

\[t_\text{obs} \;=\; \frac{\bar{x}_{CI} - \bar{x}_{PI}}{SE_\text{diff}} \;=\; \frac{86.69 - 64.67}{5.4} \;\approx\; 4.08\]

And this test’s reference distribution. Welch’s df of 450.8 picks the curve; the two-tailed critical value is $t^* = 1.965$:

Show the code that built this figure

ref_dist_plot(
  dens = function(x) dt(x, df = welch_df),
  crit = c(-t_star_2, t_star_2),
  obs  = t_obs_2,
  xlim = c(-5, 5),
  stat_label = "t",
  subtitle = sprintf(
    "t-distribution, df = %.1f\nHeather = rejection region (alpha = .05, two-tailed), boundaries at t* = %s%.2f",
    welch_df, "\u00b1", t_star_2)
)

The t-distribution with about 451 degrees of freedom spanning minus 5 to plus 5, with heather rejection regions beyond plus and minus 1.97. The observed t of 4.08 is marked in the right tail, clearly inside the rejection region but much closer to the boundary than Test 1's statistic was.

Compare this picture with Test 1’s. Same distribution family, same $\alpha$, same two-tailed logic — but our statistic now sits at 4.08 rather than out past 14, so the whole curve fits comfortably on the axis and you can see the observed value clearing the boundary rather than vanishing off it. Both reject; they are not equally emphatic, and the picture shows the difference in a way the word “significant” cannot.

t_test() takes a formula — the syntax you’ll meet again in lm() — with order setting the direction of subtraction (CI − PI):

t_result <- ci_pi |>
  t_test(
    formula = wtp ~ graph_type,
    order = c("CI", "PI"),
    alternative = "two-sided"
  )

t_result

Study the output before you read on. Same five numbers as Test 1 — but two of them now mean something subtly different. Find the estimate and the CI, and ask: estimate of what? interval for what?

Check your answers

estimate = 22.01 — no longer a group mean but the difference in means (CI − PI), in Ice Dollars. The parameter is $\mu_{CI} - \mu_{PI}$.
statistic = 4.08 — the observed difference is about 4 standard errors away from the null value of 0. Matches the hand calculation.
t_df = 450.8 — Welch’s fractional df.
p_value < .001 — a difference this large would be very unusual if the two population means were equal. We reject $H_0$.
95% CI = [$11.4, $32.63] — the plausible range for the difference, not for either mean. It excludes 0 (the test’s decision, in interval form) and, just as usefully, it tells you the difference could plausibly be as small as $11 or as large as $33.

A note on the sign of t, because it trips up almost everyone. That order argument was not cosmetic. Had we written order = c("PI", "CI"), R would have subtracted the other way and reported t = -4.08 with a CI of [−$32.63, −$11.4] — every sign flipped. The p-value would be identical, and so would the conclusion.

This matters in two practical ways. First, when you reproduce a published analysis and your t comes out negative where theirs was positive, you have almost certainly not made an error — you subtracted in the other direction. Compare the group means, not the sign. Second, when you write results up, choose the direction that matches your sentence: if you write “CI viewers paid more than PI viewers,” report the positive t that goes with CI − PI. A negative statistic beside the word more reads as a contradiction even when it isn’t. (This advice is specific to t-tests — F and $\chi^2$ are non-negative by construction, so the question never arises for them.)

Quantify and visualize the effect

Raw units first. Readers randomized to the CI display were willing to pay about $22 more on average than readers randomized to the PI display — 95% CI [$11.4, $32.63] — against a rational price of $65. That sentence, in Ice Dollars, is the primary result. Everything standardized comes second.

Now the standardized companion. For two groups, Cohen’s d is the mean difference scaled by the pooled within-group SD:

cohens_d(wtp ~ graph_type, data = ci_pi, pooled_sd = FALSE)

Cohen’s d = 0.38, 95% CI [0.19, 0.56].

Notice the pooled_sd = FALSE, because this is a decision, not a default. Cohen’s d divides the mean difference by a standardizer, and you have to choose one. By default cohens_d() uses a pooled SD — the same quantity we built by hand above — which is constructed on the assumption that the two populations share one variance. But that is precisely the assumption Welch’s test just declined to make. Asking for pooled_sd = FALSE instead standardizes by $\sqrt{(s_1^2 + s_2^2)/2}$, which makes no equal-variance claim and so keeps the effect size consistent with the test beside it. This is the package’s own advice: ?cohens_d says to set it “for effect sizes that are to accompany a Welch’s t-test,” citing Delacre et al. (2021).

The two numbers still answer different questions, and it is worth being able to say how. Welch’s t measures the difference against its sampling uncertainty — how well pinned down is it? Cohen’s d measures the same difference against the spread of individual observations — how large is it relative to how much people differ? That is also why d is less sample-size-dependent than t: it doesn’t mechanically grow with $n$. It isn’t a context-free constant either, though — it depends on the population’s variability, the measure’s reliability, and the standardizer you chose. Which is the last reason to name your standardizer whenever you report a d.

\|d\|	Conventional label
< 0.2	Negligible
0.2 – 0.5	Small
0.5 – 0.8	Medium
≥ 0.8	Large

Treat these as rough, context-free conventions — orientation, not verdicts.

What a d of this size actually looks like

A number like “d = 0.38” is hard to feel. A picture is not.

Start with the general case. Imagine a randomized trial: a treatment group and a control group, everyone measured on the same outcome. Each group produces a distribution of scores, and Cohen’s d is the distance between the two distribution centers, expressed in standard-deviation units. That “in standard deviations” step is the same standardizing trick you’ve used all term — M07 standardized a sample mean into a z, M08 standardized a gap into a t — and it’s what makes d portable across outcomes measured in points, dollars, or millimeters of mercury.

The panels below draw two Normal populations at three benchmark values of d. Two things to watch:

The shaded region is the overlap — the ground the two distributions share. Big overlap means the groups are mostly indistinguishable at the individual level even when their averages differ.
CLES, the common-language effect size, is the probability that a randomly drawn member of the higher group scores above a randomly drawn member of the lower group. If the grouping made no difference, it would be 50%.

Show the code that built this figure

library(tidyverse)
library(patchwork)

teal <- "#348C9E"
rose <- "#C05852"
panel_bg <- "#FCFDFD"

cohens_d_panel <- function(d, label) {

  x_grid <- seq(-4, 4 + d, length.out = 600)
  density_data <- tibble(
    x = rep(x_grid, 2),
    density = c(dnorm(x_grid, mean = 0, sd = 1),
                dnorm(x_grid, mean = d, sd = 1)),
    group = rep(c("Group A", "Group B"), each = length(x_grid))
  )

  overlap_grid <- seq(-4, 4 + d, length.out = 600)
  overlap_density <- pmin(
    dnorm(overlap_grid, mean = 0, sd = 1),
    dnorm(overlap_grid, mean = d, sd = 1)
  )
  overlap_data <- tibble(x = overlap_grid, density = overlap_density)

  pct_overlap <- round(100 * (2 * pnorm(-abs(d) / 2)), 0)
  cles <- round(100 * pnorm(d / sqrt(2)), 0)

  density_data |>
    ggplot(aes(x = x, y = density, color = group, fill = group)) +
    geom_area(data = overlap_data,
              aes(x = x, y = density),
              fill = "grey60",
              alpha = 0.45,
              inherit.aes = FALSE) +
    geom_line(linewidth = 1.1) +
    scale_color_manual(values = c("Group A" = teal, "Group B" = rose)) +
    scale_fill_manual(values = c("Group A" = teal, "Group B" = rose)) +
    coord_cartesian(xlim = c(-4, 4 + d), ylim = c(0, 0.44), clip = "off") +
    labs(
      title = paste0("d = ", d, " (", label, ")"),
      subtitle = paste0("Overlap: ", pct_overlap, "%   |   CLES: ", cles, "%"),
      x = NULL, y = NULL
    ) +
    theme(legend.position = "none",
          plot.title = element_text(size = 13, face = "bold", hjust = 0.5,
                                    margin = margin(b = 2)),
          plot.subtitle = element_text(size = 10.5, color = "grey25", hjust = 0.5,
                                       margin = margin(b = 6)),
          axis.text.y = element_blank(),
          axis.ticks.y = element_blank(),
          panel.grid.minor = element_blank())
}

(cohens_d_panel(0.2, "small") |
  cohens_d_panel(0.5, "medium") |
  cohens_d_panel(0.8, "large")) +
  plot_annotation(theme = theme(plot.background = element_rect(fill = panel_bg, color = NA)))

Three side-by-side panels of two overlapping Normal curves separated by Cohen's d of 0.2, 0.5, and 0.8. The grey overlap region shrinks from about 92 percent at d = 0.2 to 69 percent at d = 0.8, while the common-language effect size rises from 56 to 71 percent.

Even at d = 0.8 — Cohen’s “large” — the distributions still overlap substantially, and plenty of lower-group members outscore plenty of higher-group members. A real average effect is entirely compatible with enormous individual variation. Our d of 0.38 sits between the first two panels: the CI display reliably shifts the average, while most individual CI viewers remain indistinguishable from most individual PI viewers. Both halves of that sentence belong in your understanding of the result.

The common-language effect size (CLES).

Under an equal-variance Normal model, d converts to the probability that a random draw from the higher-mean population exceeds a random draw from the lower-mean one: $\Phi(d/\sqrt{2})$. For d = 0.2 → 56%; d = 0.5 → 64%; d = 0.8 → 71%.

Used carefully, this can be an intuitive supplement for non-statistical audiences — “about a 64% chance a random treated person scored higher than a random control person.” The caution is in the premise: the conversion assumes Normal distributions with equal variances. With skewed outcomes like WTP, quote it as an approximation under that model, not as a measured fact about your sample.

Dr. Kristoffer Magnusson’s interactive visualization at rpsychologist.com/cohend lets you drag a slider for d and watch the overlap, CLES, and the two distributions update in real time. Highly recommended for building intuition.

Write the result and define its scope

APA-style reporting · Two-Sample t-Test

Participants in the CI condition (M = $86.7, SD = $62.7) reported significantly higher willingness to pay than participants in the PI condition (M = $64.7, SD = $53.6), t(450.8) = 4.08, p < .001, 95% CI for the mean difference [$11.4, $32.63], Cohen’s d = 0.38, 95% CI [0.19, 0.56].

In plain language: within the large-effect condition, readers randomized to a CI display were willing to pay about $22 more for the special boulder than readers randomized to a PI display — a small effect by the conventional labels, in the direction the paper’s account predicts.

Where does the causal claim come from?

Not from the t-test. The test quantifies how incompatible the observed difference is with a null model of no difference between the population means; it neither knows nor cares how the groups were formed. The causal reading — the display caused the difference — comes from random assignment, which makes the conditions comparable in expectation — the realized groups can still differ on baseline characteristics by luck, but nothing systematic distinguishes them except the chart they saw. Randomization supplies the causal warrant; the t-test supplies the uncertainty quantification for the randomized contrast. Keep the two contributions separate, because in observational data you’ll have the second without the first — same test, much weaker claim.

What this analysis does not establish. It compares two of the four displays, within one effect-size condition: it cannot say where HOPs or the rescaled CI land (Test 3’s job), or whether the display effect differs by effect size (an interaction — PSY 653). And a note on narrative: Test 1 found overpayment relative to a benchmark in Experiment 1; this test found a display effect in Experiment 2. The two findings rhyme — displays that emphasize inferential uncertainty push payment up — but they come from different experiments and different kinds of comparison (calibration vs randomized contrast), so treat “the CI display amplifies baseline overpayment” as the paper’s synthesis across studies, not as something either test shows alone.

When the standard analysis is unsuitable: Mann-Whitney U (Wilcoxon rank-sum).

For heavily skewed outcomes with small per-group samples (typically n < 20–30 per group), the Mann-Whitney U is the usual alternative — with the same caution as before: it tests a different population quantity. Strictly it tests stochastic dominance — whether draws from one group tend to exceed draws from the other — which coincides with a comparison of medians only when the two distributions share a shape. It is not, in general, a test of means. Decide what your question needs before switching.

wilcox.test(wtp ~ graph_type, data = ci_pi)

With hundreds of observations per group, Welch’s t is the right choice here.

Test 3: Four randomized groups — one-way ANOVA

What’s new in Test 3

The design gains groups: four randomized conditions instead of two. That changes the kind of question the main test can answer — it becomes an omnibus⁵ question (“could all four population means be equal?”) that deliberately does not say which groups differ. Two pieces of machinery arrive with it: the F-statistic, which aggregates many departures at once, and the multiple-comparisons problem, which governs how you follow up.

Recognize the design

Test 2 compared CI against PI and left two threads hanging. Where does HOPs — the animated display that shows individual draws one at a time — land? And what about CI rescaled, the display the authors built specifically to separate how wide an interval looks from what it statistically means?

The ingredients: the outcome is still continuous WTP; there are now four independent, randomized groups (still within the Large effect-size condition); the parameters in question are four population means.

Why not just run all the pairwise t-tests? Count them: four groups make $4 \times 3 / 2 = 6$ pairs. At $\alpha = .05$ per test, the chance of at least one false positive across the family is roughly $1 - (.95)^6 \approx 26\%$⁶ — a one-in-four chance of “discovering” something produced by nothing but looking six times. A one-way ANOVA answers the question differently: a single omnibus test of equality across all four means, whose one decision carries a single 5% false-positive rate. It does not by itself dispose of the multiplicity problem — any follow-up comparisons still need an appropriate adjustment or a set of contrasts specified in advance.

all_groups <- blorg_exp2 |>
  filter(effect_size == "Large") |>
  select(graph_type, wtp) |>
  drop_na()

State the parameter and hypotheses

In words: the parameters are the four population mean WTPs, one per display condition, and the null claims the display makes no difference to any of them.

\[H_0: \mu_{CI} = \mu_{CI\,rescaled} = \mu_{PI} = \mu_{HOPS}\] \[H_a: \text{at least one population mean differs}\]

Note what $H_a$ does not say: which mean differs, or by how much, or in what direction. The omnibus alternative is deliberately unspecific — that’s the price of testing four groups with one decision, and it’s why a significant ANOVA is the start of the answer, not the end.

The same two hypotheses, written as models

There is a second way to write those hypotheses, and it is worth the two minutes it takes, because it changes what a hypothesis test is in your mind — from a procedure you run to a comparison between two accounts of how the data came to be. This is the framing that carries you into regression in M10, and further still if you ever move to Bayesian modeling, where writing the model down is not optional.

Let $Y_{ik}$ be the willingness to pay of participant i in display condition k. The two hypotheses become two models:

\[H_0: \; Y_{ik} = \mu + \epsilon_{ik} \qquad\qquad H_a: \; Y_{ik} = \mu_k + \epsilon_{ik}\]

Read them out loud, because they say something concrete:

Under $H_0$, everyone’s score is generated from one population mean $\mu$ — the same for every condition — plus an individual departure $\epsilon$.
Under $H_a$, each condition has its own mean $\mu_k$, and everyone’s score is generated from their condition’s mean plus an individual departure.

Both models assume the departures behave the same way: $\epsilon_{ik} \sim \text{Normal}(0, \sigma^2)$ — centered on zero, with one spread $\sigma$ shared across all four groups. That shared $\sigma$ is where the equal-variance assumption actually lives; it is not an extra rule bolted on afterward, it is written into the model.

Now the payoff, and it is the sentence that makes F make sense:

$MS_\text{within}$ estimates $\sigma^2$. $MS_\text{between}$ estimates $\sigma^2$ plus something that grows with the true differences among the group means.

So the F-ratio is, roughly,

\[F \;\approx\; \frac{\sigma^2 + (\text{true differences among the } \mu_k)}{\sigma^2}\]

If $H_0$ is true, the added term is zero, both mean squares estimate the same $\sigma^2$, and the ratio hovers around 1. If $H_a$ is true, the numerator carries something extra and the ratio is pushed above 1. That is the entire logic of ANOVA in one fraction — and notice that it explains not just why we compare against 1 but why the comparison is one-tailed: the extra term can only ever add.

This model view also reaches backward. Test 1 compared $Y_i = \mu_0 + \epsilon_i$ against $Y_i = \mu + \epsilon_i$; Test 2 compared a model with one mean against a model with two. Every test in this Module is a contest between a simpler account of the data and a more elaborate one, asking whether the elaboration earns its keep.

See the data and assess the model

all_groups |>
  group_by(graph_type) |>
  summarize(
    n = n(),
    mean = round(mean(wtp), 1),
    sd = round(sd(wtp), 1)
  )

Show the code that built this figure

all_groups |> ggplot(aes(x = graph_type, y = wtp, fill = graph_type)) +
  geom_violin(alpha = 0.35, color = NA) +
  geom_boxplot(width = 0.12, fill = "white", outlier.alpha = 0.3) +
  stat_summary(fun = mean, geom = "point", shape = 21, size = 3,
               fill = "#AD872B", color = "#AD872B") +
  scale_fill_manual(values = c("CI" = "#4E5EAA", "CI rescaled" = "#348C9E",
                               "PI" = "#3A8055", "HOPS" = "#945297"),
                    guide = "none") +
  labs(title = "Willingness to pay across the four visualization conditions",
       subtitle = "Gold dot = group mean",
       x = "Visualization type",
       y = "Willingness to pay (Ice Dollars)")

Violin-and-boxplots of willingness to pay across the four visualization conditions — CI, CI rescaled, PI, and HOPS — each with a gold dot at the group mean; the CI condition's mean sits highest and PI's lowest, with CI rescaled and HOPS in between.

The four means step downward from CI to PI, with the rescaled CI and HOPs in between — but each pair of distributions overlaps heavily, so eyes alone can’t settle whether four population means differ.

Design assumptions. Random assignment to one of four conditions; each participant once; independence across groups. ✓

Model considerations. The standard one-way ANOVA is an equal-variance model: it pools all four groups’ spread into one within-group variance. Our four SDs are similar (from $48 to $63), the group sizes are large and nearly balanced, and under those conditions the standard ANOVA behaves well — so we use it here, follow it with Tukey, and note the alternative: if group variances differ substantially, the direct unequal-variance extension is Welch’s ANOVA (the multi-group analog of the Welch t you just used), paired with Games–Howell follow-ups. Details in the box below; the principle is the same one as in Test 2 — prefer methods that don’t need a preliminary variance test to justify them.

A use for the coefficient of variation

M01 introduced the coefficient of variation — the SD as a percentage of the mean — as a way to compare spreads that sit on different scales. This is where it earns its keep in inference. Comparing the groups’ CVs is a quick read on whether the equal-variance assumption is plausible: when groups have noticeably different means but similar CVs, their SDs must differ systematically, which is the signature of variance that scales with the mean. That pattern is common in positive, right-skewed measures — income, counts, reaction times — and it is exactly the situation Welch’s ANOVA (or a transformation) is built for.

⚠️ Do not read the CV as a measure of how confident you can be in a group difference. A small within-group CV does not mean the groups are more clearly separated. What determines the evidence is the gap between the means relative to the within-group SD — that ratio is Cohen’s d, and it is what $F = MS_\text{between} / MS_\text{within}$ measures. The CV divides each group’s SD by its own mean, and that mean has nothing to do with whether the gap is detectable. Two groups with means of 1,000 and 1,010 and SDs of 50 have CVs under 5% and no detectable difference; two groups with means of 2 and 4 and SDs of 0.5 have CVs above 10% and an overwhelming one. CV diagnoses the assumption; d and F measure the evidence.

You will nonetheless meet such a test constantly in published papers, so you should know its name: Levene’s test (car::leveneTest()) evaluates the null hypothesis that all groups share one variance. Its mechanism is charming — it replaces each score with its absolute distance from its own group’s center, then runs an ANOVA on those distances, since “are the groups equally spread?” becomes “do these distance-scores have equal means?” One detail worth knowing, because the output announces it and students routinely miss it: car::leveneTest() centers on the median by default, which is the more robust Brown–Forsythe variant; the original mean-centered form has to be requested explicitly with center = mean. We name the test rather than use it, for the reason just given — making one test’s legitimacy conditional on another test’s outcome inflates the error rate of the pair. Read the group SDs, know your design, and prefer Welch when in doubt.

Welch’s ANOVA and Games–Howell.

The standard ANOVA’s F can drift from its advertised $\alpha$ when group variances differ, and the distortion is worst when smaller groups carry larger variances. Rather than running a variance test as a gatekeeper (preliminary assumption tests have their own power and multiplicity problems), know the direct alternative:

Welch’s ANOVA — same omnibus logic, degrees of freedom adjusted for unequal variances: oneway.test()(wtp ~ graph_type, data = all_groups) (its default is the Welch form).
Games–Howell — the pairwise follow-up that assumes nothing about equal variances: rstatix::games_howell_test().

The pairing rule: Tukey after standard ANOVA; Games–Howell after Welch’s ANOVA. If you’re unsure which world you’re in, running both is cheap — and if they disagree, the Welch/Games–Howell answer is the more defensible.

Run and read the test

How F does its job. Test 2’s fraction — signal over noise — is about to become literal. The F-statistic is built by taking the total variability in willingness to pay and splitting it into two piles.

Start with a single question: why do these 910 people not all report the same WTP? Two reasons, and only two:

Because they saw different displays. Some were shown a CI, others a PI — and if display matters, that pushes their answers apart. This is between-group variability.
Because people differ. Two participants who saw the identical chart still answer differently — different risk appetites, different reading of a fictional currency, different moods. This is within-group variability, and it is the irreducible noise floor.

The remarkable fact — and it is exact algebra, not an approximation — is that those two pieces add up to the whole:

\[\underbrace{SS_\text{total}}_{\text{everyone vs the grand mean}} \;=\; \underbrace{SS_\text{between}}_{\text{group means vs the grand mean}} \;+\; \underbrace{SS_\text{within}}_{\text{each person vs their own group mean}}\]

where each SS (“sum of squares”) is exactly what M01 taught you variance is built from: take a deviation from a mean, square it, add them all up. Let’s compute all three from our data and check that the identity really holds:

grand_mean <- mean(all_groups$wtp)

group_means <- all_groups |>
  group_by(graph_type) |>
  summarize(n = n(), group_mean = mean(wtp), .groups = "drop")

ss_total <- sum((all_groups$wtp - grand_mean)^2)

ss_between <- sum(group_means$n * (group_means$group_mean - grand_mean)^2)

ss_within <- all_groups |>
  left_join(group_means, by = "graph_type") |>
  summarize(ss = sum((wtp - group_mean)^2)) |>
  pull(ss)

tibble(
  piece = c("SS_between", "SS_within", "sum of the two", "SS_total"),
  value = round(c(ss_between, ss_within, ss_between + ss_within, ss_total))
)

The two pieces reconstruct the total exactly. Every dollar of variation in this dataset is now labeled: some of it attributable to which chart you saw, the rest to being a different person.

Here is the same partition as a picture. The left panel measures every participant against the grand mean; the right measures the same people against their own group’s mean. Watch the gold diamonds — they are the four group means, and they are the only thing that actually moves:

Show the code that built this figure

dev_data <- all_groups |>
  left_join(group_means, by = "graph_type") |>
  mutate(
    from_grand = wtp - grand_mean,
    from_group = wtp - group_mean
  )

# Where each GROUP mean sits under each reference point. Against the grand
# mean the four differ; against their own means they are all zero by
# construction -- that contrast is what the figure exists to show.
mean_markers <- group_means |>
  mutate(from_grand = group_mean - grand_mean, from_group = 0)

# Both panels MUST share a y-range: the whole point is comparing spread
# between them, and different scales would make that comparison meaningless.
dev_range <- range(c(dev_data$from_grand, dev_data$from_group))

palette_4 <- c("CI" = "#4E5EAA", "CI rescaled" = "#348C9E",
               "PI" = "#3A8055", "HOPS" = "#945297")

partition_panel <- function(dat, yvar, ttl, sub, ylab) {
  ggplot(dat, aes(x = graph_type, y = .data[[yvar]], color = graph_type)) +
    geom_jitter(width = 0.22, alpha = 0.18, size = 0.9) +
    geom_hline(yintercept = 0, color = "#1C2433", linewidth = 0.9) +
    geom_point(data = mean_markers, aes(y = .data[[yvar]]),
               shape = 23, size = 4.2, stroke = 1.1,
               fill = "#AD872B", color = "#1C2433") +
    scale_color_manual(values = palette_4, guide = "none") +
    coord_cartesian(ylim = dev_range) +
    labs(title = ttl, subtitle = sub, x = NULL, y = ylab) +
    # Titles sized down and subtitles broken onto two lines so the two
    # panels cannot collide with each other at this figure width.
    theme(plot.title = element_text(size = 12),
          plot.subtitle = element_text(size = 10, lineheight = 1.2))
}

p_total <- partition_panel(
  dev_data, "from_grand",
  "Distance from the GRAND mean",
  paste0("SS_total = ", format(round(ss_total), big.mark = ","),
         "\ngroup means SPREAD OUT"),
  "deviation (Ice Dollars)")

p_within <- partition_panel(
  dev_data, "from_group",
  "Distance from their OWN group mean",
  paste0("SS_within = ", format(round(ss_within), big.mark = ","),
         "\ngroup means ALL AT ZERO"),
  NULL)

(p_total | p_within) +
  plot_annotation(
    title = "The variance partition, drawn",
    subtitle = paste0("Same ", nrow(all_groups),
                      " people, same shared y-axis — only the reference point changes. Gold diamonds = group means."),
    theme = theme(plot.background = element_rect(fill = "#FCFDFD", color = NA))
  )

Two panels showing the same 910 participants as jittered points. In the left panel, measured against the grand mean, the four gold diamonds marking the group means sit at different heights, spread from about minus 11 to plus 11 Ice Dollars. In the right panel, measured against each participant's own group mean, all four diamonds sit exactly on the zero line. The point clouds themselves look almost identical between panels, because the group means differ by far less than individuals differ from one another.

What to look at, and what it means. The two point clouds look nearly identical, and that is not a failure of the figure — it is the result. The gold diamonds are what changed. On the left they sit at four different heights, spanning about $-9 to $13; on the right they are pinned to zero, because subtracting each group’s own mean is precisely what removes them.

That collapse of the diamonds is $SS_\text{between}$ — everything the display condition explains. Everything still scattered around them after the collapse is $SS_\text{within}$ — the part being a different person explains.

“Shrinkage” is the word for how much the cloud tightens when you switch reference points, and it deserves a number rather than an adjective. Re-measuring everyone against their own group’s mean removed 60,946 of the 2,861,857 total — a reduction of 2.1%. Which is why your eye cannot see it: the diamonds are separated by tens of dollars while individuals scatter across hundreds. Small effect, large noise — visible in the arithmetic, invisible in the cloud. Hold that contrast, because it is the same one the F-test is about to formalize.

From sums of squares to a fair comparison. We cannot compare $SS_\text{between}$ and $SS_\text{within}$ directly, because they are built from different numbers of free quantities — and Part 1’s counting principle tells us exactly how many each has. For the between piece, the quantities are the four group means and the constraint is the grand mean they must average to: $k - 1 = 3$. For the within piece, the quantities are all 910 observations and the constraints are the four group means we estimated in order to measure deviations from them: $N - k = 910 - 4 = 906$. Dividing each sum of squares by its own df converts it into an average squared deviation per free piece — a mean square, which is just a variance:

# Degrees of freedom, counted rather than typed.
n_obs_3 <- nrow(all_groups)
k_grp_3 <- nrow(group_means)
df_b_3  <- k_grp_3 - 1          # k - 1
df_w_3  <- n_obs_3 - k_grp_3    # N - k

# Each sum of squares divided by its own df is a mean square -- a variance.
ms_between <- ss_between / df_b_3
ms_within  <- ss_within / df_w_3

# F is simply their ratio.
f_from_ss <- ms_between / ms_within

c(df_between = df_b_3, df_within = df_w_3,
  MS_between = ms_between, MS_within = ms_within, F = f_from_ss) |> round(2)

df_between  df_within MS_between  MS_within          F 
      3.00     906.00   20315.42    3091.51       6.57

\[MS_\text{between} = \frac{SS_\text{between}}{df_\text{between}} = \frac{60,946}{3} = 20,315 \qquad MS_\text{within} = \frac{SS_\text{within}}{df_\text{within}} = \frac{2,800,910}{906} = 3,092\]

$MS_\text{within}$ deserves a moment, because you have already met it. In Test 2 we built the pooled variance for two groups: deviations from each participant’s own group mean, squared, summed, divided by $N - 2$. $MS_\text{within}$ is that same quantity with the 2 replaced by 4 — deviations from each participant’s own group mean, squared, summed, divided by $N - k$. Two groups or four, it is one idea: pool everyone’s distance from their own group’s center into a single estimate of $\sigma$, the noise the model says all groups share.

The numbers make the continuity visible. With two groups (CI and PI) the pooled SD came to $58.26; here, across all four, $\sqrt{MS_\text{within}} =$ $55.6. Similar, as they should be — both are estimating the same $\sigma$. That figure is the typical distance between a participant and their own group’s mean: the noise floor, in Ice Dollars.

And now F is just the ratio:

\[F = \frac{MS_\text{between}}{MS_\text{within}} = \frac{20,315}{3,092} = 6.57\]

Why 1 is the number to compare against. If $H_0$ were true — if the four population means were identical — then the group means would still not be exactly equal in any sample; they would scatter a bit, purely by sampling luck. And here is the key: under $H_0$, that accidental scatter is generated by the same within-group noise. Both mean squares would be estimating the same underlying variance, so their ratio would hover around 1. Values much above 1 mean the group means are further apart than within-group noise alone can manufacture. Our 6.57 says the between-group mean square is about 6.6 times the within-group mean square — more separation among the group means than the null model would ordinarily produce.

That also explains the shape you saw in Part 1. F is a ratio of two non-negative quantities, so it cannot go below zero, and under $H_0$ it is typically near 1, with a right-skewed distribution — a completely different picture from the symmetric t, for a completely understandable reason.

The two numbers that turn our F into a decision come from the F distribution the same way Test 1’s came from the t distribution — a quantile function for the cutoff, a cumulative function for the tail area:

# Both use the SAME pair of df. Upper tail only, because F cannot be negative --
# so unlike the t-tests, there is no doubling for a second tail.
f_star_3 <- qf(0.95, df1 = df_b_3, df2 = df_w_3)
p_from_f <- pf(f_from_ss, df1 = df_b_3, df2 = df_w_3, lower.tail = FALSE)

c(F_obs = f_from_ss, F_star = f_star_3, p = p_from_f) |> round(4)

 F_obs F_star      p 
6.5714 2.6147 0.0002

So we reject when our F of 6.57 clears the critical value of 2.61 — which it does. Note lower.tail = FALSE in pf(): it asks for the area above our statistic, which for a one-tailed test is the whole p-value.

Show the code that built this figure

ref_dist_plot(
  dens = function(x) df(x, df1 = df_b_3, df2 = df_w_3),
  crit = qf(0.95, df_b_3, df_w_3),
  obs  = f_from_ss,
  xlim = c(0, 8),
  stat_label = "F",
  subtitle = sprintf(
    "F-distribution, df = (%d, %d) — upper tail only, because F cannot be negative\nCritical value F* = %.2f",
    df_b_3, df_w_3, qf(0.95, df_b_3, df_w_3))
)

The F-distribution with 3 and 906 degrees of freedom on a zero-to-eight axis, right-skewed with its bulk near 1. The upper-tail rejection region beyond the critical value of 2.61 is shaded heather, and the observed F of 6.57 is marked well inside it.

Read it exactly as before: the curve is what F does when all four population means are equal, the heather region holds 5% of the area, and our observed F sits well inside it. One number, aggregating all four groups’ departures at once — which is precisely why it cannot tell you which group is responsible.

We fit the model with lm() and ask anova() for the F-table — deliberately, because ANOVA is a linear model with a categorical predictor, and this is the same workflow M10 builds on:

anova_model <- lm(wtp ~ graph_type, data = all_groups)

anova(anova_model) |> tidy()

Study the output before you read on. The ANOVA table looks different from a t-test tibble, and it does not carry every quantity a t-test does — an omnibus test has no single estimate or interval. Find: the two df values (where does each come from?), the F statistic (what would 1 mean?), and the p-value.

Check your answers

Two rows: graph_type is the between-group signal; Residuals is the within-group noise floor.
df = (3, 906): the numerator df is groups − 1 = 3; the denominator is $N -$ groups $= 906$.
statistic = 6.57 — the between-group mean square is about 7 times the within-group mean square. (Under $H_0$ both estimate the same population variance, so an F near 1 is what the null model predicts.)
p-value < .001 — four equal population means would essentially never produce an F this large. We reject $H_0$: at least one display’s mean WTP differs.
No confidence interval in this table — an omnibus test has no single parameter to build one around. Intervals reappear at the pairwise step, next.

With two groups, ANOVA is the t-test

The Module has been claiming that these are not different theories, only one logic pointed at different designs. Here is the place that claim becomes checkable rather than rhetorical.

Run an ANOVA on just two groups and you do not get something similar to a two-sample t-test. You get the same test — with the statistics related by $F = t^2$. Take Test 2’s CI-versus-PI data and analyze it both ways:

# Student's t-test (the equal-variance form, so both analyses assume one shared sigma)
student_t <- t.test(wtp ~ graph_type, data = ci_pi, var.equal = TRUE) |> tidy()

# The same comparison as a one-way ANOVA
two_group_f <- anova(lm(wtp ~ graph_type, data = ci_pi)) |> tidy()

tibble(
  quantity = c("t (Student)", "t squared", "F (ANOVA)"),
  value = c(
    student_t$statistic,
    student_t$statistic^2,
    two_group_f$statistic[1]
  )
)

Not approximately equal — equal, to every decimal R will print: 16.6671. The p-values match too, and so do the degrees of freedom, once you know where to look: Student’s t had $df = N - 2 = 465$, and the ANOVA reports $df = (1, 465)$. The numerator 1 is $k - 1$ with two groups.

Why does this happen? Look back at the two models. With $k = 2$, “each group has its own mean” is “the two means differ,” so the two tests are asking one question in two notations. Squaring is what erases the direction: a t of $+4.08$ and a t of $-4.08$ describe equally strong evidence, and F — which cannot be negative and has no direction to report — collapses them onto the same value. That is also why F’s rejection region sits in the upper tail only. The two-tailed t-test and the one-tailed F-test are, here, literally the same decision.

Keep this in your pocket for Project 2: if a paper reports F(1, df) for a two-group comparison and you ran a t-test, take the square root before deciding you disagree with them.

Quantify and visualize the effect

Raw units first. The group means run from $65 (PI) up to $87 (CI) — a spread of about $22 across displays, against a rational price of $65.

Follow-up comparisons — matched to the question. The omnibus result tells us the four means are not all equal, but not where the differences lie. The follow-up should fit what you actually want to know — and note that it is not gated on the omnibus test: planned contrasts can be tested regardless of how F came out, and Tukey-adjusted comparisons already control their own familywise rate without needing a significant F as a gatekeeper. If your hypotheses named specific comparisons in advance, planned contrasts test just those, at full power. If — as here — the question is exploratory (“which displays differ from which?”), the standard tool is Tukey’s HSD, which tests all pairs while holding the familywise error rate at $\alpha$ (it draws its critical values from the studentized range distribution, built for the largest-gap-among-$k$-means question):

TukeyHSD(aov(wtp ~ graph_type, data = all_groups)) |>
  tidy() |>
  select(contrast, estimate, conf.low, conf.high, adj.p.value) |>
  mutate(across(where(is.numeric), ~ round(.x, 3)))

Each row is one pair: the estimated difference, a Tukey-adjusted 95% CI, and a Tukey-adjusted p-value. A pair whose CI excludes 0 differs significantly after accounting for the fact that six comparisons were made. Tukey isn’t the only adjustment, and the alternatives sort into a clear order:

Bonferroni — multiply every p-value by the number of comparisons. Controls the familywise rate, and it is the easiest to explain, which is why it is still everywhere.
Holm — the same idea applied sequentially: rank the p-values, and scale each by how many comparisons remain rather than by the full count. It controls the familywise rate at exactly the same level as Bonferroni while rejecting at least as often — never less. Holm uniformly dominates Bonferroni, so there is no situation in which Bonferroni is the better choice; prefer Holm whenever you would have reached for Bonferroni.
Tukey’s HSD — purpose-built for all pairwise comparisons of means, which is exactly our situation, and more powerful than the general-purpose corrections for that specific job. Hence its use here.
Benjamini–Hochberg — changes the goal rather than the arithmetic: it controls the false-discovery rate (the expected share of your rejections that are wrong) instead of the probability of any error. Suited to screening many comparisons where a few false leads are tolerable.

See the Multiple comparisons deep-dive for the arithmetic and for when each fits.

The standardized effect for ANOVA is $\eta^2$ (eta-squared) — and you have already computed everything it needs. Look again at the partition: $SS_\text{total}$ was all the variation in willingness to pay, and $SS_\text{between}$ was the slice attributable to which display someone saw. Their ratio is the proportion of variance explained:

\[\eta^2 = \frac{SS_\text{between}}{SS_\text{total}} = \frac{60,946}{2,861,857} = 0.021\]

ss_between / ss_total

[1] 0.02129605

So display format accounts for about 2.1% of the variation in WTP, and the other 97.9% is people differing from one another within their condition. That is the same story the partition figure told: moving from the left panel to the right one — from measuring everybody against the grand mean to measuring them against their own group’s mean — barely tightened the spread.

Notice how tightly $\eta^2$ and F are related — both are built from the same two piles, just combined differently. F asks “is the between-group signal bigger than noise?”; $\eta^2$ asks “what share of everything does the signal account for?” A large sample can make a small share highly detectable, which is exactly the case here.

effectsize confirms the hand calculation and adds a confidence interval:

eta_squared(anova_model, alternative = "two.sided")

(The alternative = "two.sided" argument requests a conventional two-sided 95% CI; effectsize’s default for variance-explained measures is a one-sided interval whose upper bound is always 1.)

η²	Conventional label
< 0.01	Negligible
0.01 – 0.06	Small
0.06 – 0.14	Medium
≥ 0.14	Large

Note what $\eta^2$ can and cannot do: it summarizes how much of the outcome the factor explains, but — like the omnibus F it accompanies — it says nothing about which means differ or in what direction. The Tukey table carries that information; the two are complements, not substitutes.

Write the result and define its scope

APA-style reporting · One-Way ANOVA

A one-way ANOVA indicated a significant effect of visualization condition on willingness to pay, F(3, 906) = 6.57, p < .001, η² = 0.021, 95% CI [0.005, 0.041]. Tukey-adjusted comparisons identified significant differences for the following pairs: PI-CI (mean difference = -22.015 Ice Dollars, 95% CI [-35.26, -8.769], adjusted p < .001); HOPS-CI (mean difference = -16.299 Ice Dollars, 95% CI [-29.573, -3.026], adjusted p = .009).

Two notes on this sentence. Report η² alongside F: a significant F with a small η² means the effect, though detected, accounts for little of the outcome’s variance. And notice the adjusted p-values are formatted the APA way — the smallest one is reported as < .001, never as 0. That is not cosmetic. Rounding a p of .000123 to three decimals would print 0, which claims an impossibility: no observed result has zero probability under the null. When a full table of comparisons would crowd the paragraph, give the reader the pattern in prose and put the complete table alongside, as we did above.

In plain language — where the four displays land. The Tukey table draws a clean substantive map. The CI display stands apart: it sits significantly above both PI ($87 vs $65) and HOPs ($87 vs $70). The two displays that show outcome variability — PI as a wide interval, HOPs as individual draws — sit together at the bottom (adjusted p = .681 for their difference). And CI rescaled lands in between ($75): not reliably different from the CI display it copies (p = .120) nor from PI (p = .210). That intermediate position echoes the paper’s own reading — stretching the CI’s axis to look as wide as a PI “helps somewhat,” but doesn’t fully reproduce the effect of actually showing outcome variability.

One honest caveat, courtesy of the effect size: η² ≈ 0.021 means display format explains only about 2% of the variance in WTP — most of what drives any individual’s willingness to pay is everything else they bring to the decision. “Significant” answers whether the group differences are larger than sampling variation under $H_0$ would typically produce, not how much of the outcome the factor controls; keeping those apart is exactly what the effect-size step is for.

What this analysis does not establish. All conclusions live within the Large effect-size condition — whether the display ranking changes at the small effect size is an interaction question this one-way analysis cannot touch. The HOPs–PI non-difference is a failure to detect, not a demonstration of equivalence. And the omnibus F itself established only “not all equal” — every specific claim above rests on the adjusted pairwise comparisons, not on F.

When the standard analysis is unsuitable: Kruskal-Wallis.

For heavily skewed outcomes with small per-group samples, the Kruskal-Wallis test is the usual rank-based alternative — with the now-familiar caution: it tests whether values from at least one group tend to run larger than the others’ (stochastic dominance), which is a statement about distributions, not means. Its pairwise follow-up is Dunn’s test (dunn.test).

kruskal.test(wtp ~ graph_type, data = all_groups)

ANOVA and regression

We fit the ANOVA with lm() on purpose. ANOVA is regression with a categorical predictor: the F-statistic tests whether adding group membership improves on an intercept-only model (the overall mean). In Module 10 you’ll see this connection run deep — the same lm(), tidy(), anova() workflow carries through the rest of the course.

Test 4: A categorical outcome — the chi-square test of independence

What’s new in Test 4

The outcome type changes: instead of comparing means of a continuous variable, we ask whether category membership depends on group. The machinery swaps accordingly — expected counts replace expected means, and the $\chi^2$ statistic aggregates the cell-by-cell departures between the counts you observed and the counts a null model would predict.

Because that observed-versus-expected idea is the whole engine, this test gets a warm-up section before the usual six headings: we meet the machinery first on a single categorical variable, where it is easiest to see, and then extend it to two.

Warm-up: one categorical variable — the goodness-of-fit test

Before comparing two categorical variables, meet the simpler case: one categorical variable, tested against a set of proportions you specify in advance. This is the $\chi^2$ goodness-of-fit test, and it is the categorical twin of Test 1 — one variable, one externally supplied benchmark:

	Continuous outcome	Categorical outcome
One variable vs a benchmark	one-sample t (Test 1)	$\chi^2$ goodness-of-fit
Outcome vs a grouping variable	two-sample t / ANOVA (Tests 2–3)	$\chi^2$ independence (below)

A question worth actually asking. The study intended to allocate participants equally across the four displays. Are the observed cell sizes compatible with that intended 1:1:1:1 allocation? With 910 participants in the Large-effect condition, equal allocation would put roughly a quarter in each. Let’s check — reporting the realized cell sizes is a routine part of describing a randomized study:

display_counts <- all_groups |> count(graph_type)

display_counts

The null hypothesis supplies the expected counts. Here $H_0$ says all four displays were equally likely: $P_\text{CI} = P_\text{CI\,rescaled} = P_\text{PI} = P_\text{HOPS} = 1/4$. Multiply each probability by the total sample size and you have what the null model predicts:

\[E_i \;=\; N \times P_i \;=\; 910 \times 0.25 \;=\; 227.5\]

Notice how different this is from the independence test coming next. In a goodness-of-fit test, you bring the expected proportions with you — from a theory, a known population distribution, or, as here, a study design. They are not estimated from the data.

See it before you test it. Every other test in this Module opens with a plot of the data, and a goodness-of-fit test is no different — the figure just has one job the others don’t: it has to show the expected counts alongside the observed ones, because the null model is something you brought rather than something the data suggest.

display_counts |>
  ggplot(aes(x = graph_type, y = n)) +
  geom_col(fill = "#4E5EAA", alpha = 0.85) +
  # Gold, not rose: in this section gold is the null model's prediction (see the
  # observed-vs-expected figure below), while heather means a rejection region.
  # The white-filled label keeps the text legible where the line meets the bars --
  # with near-equal allocation the line lands right at the bar tops.
  geom_hline(yintercept = gof_n / nrow(display_counts),
             linetype = "dashed", color = "#AD872B", linewidth = 1.1) +
  annotate("label", x = 0.55, y = gof_n / nrow(display_counts),
           vjust = -0.35, hjust = 0, color = "#AD872B", fontface = "italic",
           fill = "white", size = 4.1,
           label = paste0("expected under equal allocation = ",
                          round(gof_n / nrow(display_counts), 1))) +
  scale_y_continuous(expand = expansion(mult = c(0, 0.14))) +
  labs(
    title = "Realized allocation sits close to the intended 1:1:1:1 split",
    x = NULL, y = "Participants assigned"
  )

Bar chart of the number of participants assigned to each of the four display types, with a dashed horizontal line marking the count expected under equal allocation. All four bars sit close to the line.

Read the figure the way the test reads it: the dashed line is $E_i$, each bar is $O_i$, and the statistic below is just a scaled summary of the gaps between them. When a goodness-of-fit result surprises you, this plot is where you find out which category drove it — something the single χ² number cannot tell you.

Design and model assumptions. The design side is the same as everywhere else in this Module: each participant appears in exactly one category, and the categories are mutually exclusive and exhaustive. ✓ Randomized allocation, one row per person.

The model side has one check that belongs to the chi-square family specifically: the expected counts must be large enough for the χ² approximation to hold. The usual working rule is that every expected count should be at least 5. That is a statement about $E_i$, not $O_i$ — an observed count of 2 is not itself a problem if the null predicts 30, though it would certainly be an interesting result.

gof_test$expected

         CI CI rescaled          PI        HOPS 
      227.5       227.5       227.5       227.5

All 4 expected counts are 227.5, comfortably above 5, so the reference distribution is trustworthy here. When expected counts fall below that threshold — small samples, or a category the null model predicts is rare — the fix is to collapse sparse categories into a substantively sensible larger one, or to use Fisher’s exact test, which does not rely on the approximation.

The statistic is the one you’ll use for the rest of this section:

\[\chi^2 \;=\; \sum_i \frac{(O_i - E_i)^2}{E_i}\]

Two symbols, and they are the whole vocabulary of every chi-square test in this Module. $O_i$ is the observed count in category $i$ — how many participants actually landed there, straight from your data. $E_i$ is the expected count in that category — how many the null model says should have landed there. The subscript $i$ just runs over the categories, and $\sum_i$ says to add the contributions up across all of them.

Each category contributes its own squared departure, scaled by what was expected there, and the contributions are added up:

display_counts |>
  mutate(
    expected = gof_n / n(),
    contribution = (n - expected)^2 / expected
  )

all_groups |> pull(graph_type) |> table() |> chisq.test() |> tidy()

The df is the counting principle again, and it is easy to see here: we have four category counts, and one constraint — they must add to 910, the total we already know. Fix any three and the fourth is determined. So $df = k - 1 = 3$.

χ²(3, N = 910) = 2.06, p = .560. The observed counts sit close to 227.5 apiece, and a departure this small is entirely ordinary under equal allocation.

And this is a case where failing to reject is the reassuring outcome. The observed allocation shows no detectable departure from the intended equal proportions. Be precise about how little that claims, though — and this is the part students routinely overstate:

It is no evidence of a departure, not proof there was none. The M08 rule still holds: failing to reject is never accepting.
It says nothing about whether the assignment procedure was implemented correctly, whether allocation was concealed, or whether the groups are balanced on participant characteristics. Those are different questions needing different evidence.
Unequal cell sizes would not by themselves indicate failure — simple random assignment routinely produces uneven counts — and perfectly equal counts could be produced by a procedure that wasn’t random at all.

So this is a description of the realized design, not a licence for the causal readings in Tests 2 and 3. Those rest on the assignment procedure itself, not on a balance test failing to reject.

Two cautions to carry forward. A goodness-of-fit test is only as meaningful as the proportions you feed it — and if you estimate those proportions from the same data rather than specifying them in advance, you spend additional degrees of freedom, one per estimated quantity. Here we specified 1/4 from the design, so nothing was estimated.

Now we take that same observed-versus-expected machinery and point it at two variables at once.

Recognize the design

Tests 2 and 3 asked whether the display shifts the mean of a continuous outcome. Some research questions are natively categorical — diagnosed or not, retained or dropped out, above a threshold or below it — and here we can build a sharp one from the study’s own economics: did this participant overpay? In the Large-effect condition, the risk-neutral price is exactly $65 — so paying more than $65 has a precise meaning: paying more than the upgrade is worth.

One flag before we build it: dichotomizing a continuous variable discards information, and we do it here to teach the categorical test, not because it improves on Test 3’s analysis of the full WTP scale. The one good reason to cut a continuous variable is a substantively meaningful cutpoint — and a benchmark derived from the study’s own payoff structure is exactly that. (Cutting at a convenient round number, or hunting for the threshold that makes p smallest, is how dichotomization earns its bad reputation.)

The ingredients: two categorical variables — display condition (four levels) and overpay status (two levels) — measured on independent, randomized participants. The decision map says: chi-square test of independence.

chi_data <- blorg_exp2 |>
  filter(effect_size == "Large") |>
  mutate(
    overpay = case_when(
      wtp > 65 ~ "overpaid",
      wtp <= 65 ~ "did not overpay"
    ) |>
      factor(levels = c("did not overpay", "overpaid"))
  ) |>
  select(graph_type, overpay) |>
  drop_na()

chi_data |>
  count(graph_type, overpay) |>
  pivot_wider(names_from = overpay, values_from = n)

That tidy form is what our plotting and modeling code wants. But the shape that makes the test’s logic visible is a proper cross-tabulation with its margins — and you already have the tool for it. tbl_cross() from gtsummary is the same function you used in M05 to describe two categorical variables and again in M06 to read a contingency table as probabilities. Same function, third lens:

chi_data |>
  set_variable_labels(
    graph_type = "Visualization condition",
    overpay    = "Paid more than the $65 rational price?"
  ) |>
  tbl_cross(
    row     = graph_type,
    col     = overpay,
    percent = "row"
  )

	Paid more than the $65 rational price?		Total
	did not overpay	overpaid	Total
Visualization condition
CI	107 (46%)	124 (54%)	231 (100%)
CI rescaled	114 (55%)	95 (45%)	209 (100%)
PI	153 (65%)	83 (35%)	236 (100%)
HOPS	131 (56%)	103 (44%)	234 (100%)
Total	505 (55%)	405 (45%)	910 (100%)

Read it the way you read M06’s tables. Each cell gives a count and, because we set percent = "row", the percentage within that display condition — so 124 (54%) means 124 of the 231 CI viewers overpaid, or 54% of them. That row percentage is a conditional probability: $P(\text{overpaid} \mid \text{saw a CI})$, exactly the quantity M06 taught you to read off a cross-tab.

Now look down the “overpaid” column and the null hypothesis writes itself. The Total row says 405 (45%) — across everyone, 45% overpaid. If the display were irrelevant, every row should sit near that same 45%. Instead CI runs at 54% and PI at 35%. Independence is the claim that all four row percentages are equal; the chi-square test asks whether the spread we see is more than sampling noise would produce.

The margins — the Total row and Total column — earn their keep twice more before this section ends: they generate the expected counts, and they fix the degrees of freedom.

This is our 4 × 2 contingency table: four display conditions by two overpay statuses.

State the parameter and hypotheses

In words: the parameters are the four population proportions of participants who overpay, one per display, and the null states that those four proportions are equal across display conditions.

$H_0$: graph_type and overpay are independent — the population proportion overpaying is the same under every display.
$H_a$: they are not independent — the probability of overpaying depends on which display a participant saw.

Independence has a concrete meaning worth internalizing: if $H_0$ were true, knowing which chart someone saw would tell you nothing about whether they overpaid.

See the data and assess the model

The chi-square question — do the four displays split the same way across the two overpay statuses? — is a picture before it’s a statistic. A stacked-proportion bar chart shows it directly: each bar is normalized to 100%, so identical splits would mean independence.

Show the code that built this figure

chi_plot_data <- chi_data |>
  count(graph_type, overpay) |>
  group_by(graph_type) |>
  mutate(
    prop = n / sum(n),
    cell_text = sprintf("n = %d\n(%.0f%%)", n, 100 * prop)
  ) |>
  ungroup()

chi_plot_data |>
  ggplot(aes(x = graph_type, y = prop, fill = overpay)) +
  geom_col(width = 0.62, color = "white", linewidth = 1) +
  geom_text(
    aes(label = cell_text),
    position = position_stack(vjust = 0.5),
    color = "white", fontface = "bold", size = 4.6, lineheight = 0.95
  ) +
  scale_y_continuous(labels = scales::percent_format(accuracy = 1),
                     expand = expansion(mult = c(0, 0.02))) +
  scale_fill_manual(
    values = c("did not overpay" = "#4E5EAA",
               "overpaid" = "#AD872B"),
    breaks = c("overpaid", "did not overpay")
  ) +
  labs(
    title = "Overpaying the $65 rational price, by visualization condition",
    subtitle = paste0("Each bar = 100% of participants in that condition.\n",
                      "Independence would mean identical splits."),
    x = "Visualization condition",
    y = "Proportion of participants",
    fill = "Overpay status"
  )

Stacked bar chart showing, for each of the four visualization conditions, the percentage of participants who paid more versus at most the rational price of $65; the overpaying share is largest for CI, smallest for PI, with CI rescaled and HOPS in between.

The splits are not identical: 54% of CI participants paid more than the boulder is worth, against 35% of PI participants — with CI rescaled (45%) and HOPs (44%) in between. The question is whether gaps that size would commonly arise from sampling variation if the display made no difference.

Design assumptions. Same as Tests 2–3: randomized, independent, one row per participant. ✓

Model considerations. Chi-square’s reference distribution is a large-sample approximation, and the adequacy check is on the expected cell counts. The common guideline is that none should fall below 1 and no more than about 20% should fall below 5 — rough guidance rather than a threshold at which a binomial abruptly becomes Normal. We compute them directly (this is also your first look at the machinery inside the test):

chi_data |> with(table(graph_type, overpay)) |> chisq.test() |> pluck("expected")

             overpay
graph_type    did not overpay  overpaid
  CI                 128.1923 102.80769
  CI rescaled        115.9835  93.01648
  PI                 130.9670 105.03297
  HOPS               129.8571 104.14286

The smallest expected count is about 93 — far above 5. ✓ (When expected counts are small, the standard move is Fisher’s exact test — see the box at the end.)

Run and read the test

How $\chi^2$ does its job. For each of the eight cells, independence predicts an expected count from the margins, $E_{ij} = (\text{row}_i \times \text{col}_j)/N$ — that’s the table you just printed. The statistic then aggregates each cell’s departure from its prediction:

\[\chi^2 = \sum_{i,j} \frac{(O_{ij} - E_{ij})^2}{E_{ij}}\]

Two features of that formula are worth pausing on, because they explain the distribution’s shape.

The numerator is squared, so a cell that comes in below expectation contributes exactly as much as one equally far above. Departures in either direction count as evidence, which is why — unlike a t-test — there is no “direction” to a chi-square and no two-tailed/one-tailed choice to make. And the denominator scales each departure by what was expected there: being 20 counts off matters enormously in a cell expecting 30 and hardly at all in a cell expecting 3,000. Every cell is put on the same footing before being added up.

Because it is a sum of squared, scaled departures, $\chi^2$ is non-negative by construction and grows as the table drifts from independence. Zero would mean every cell landed exactly on its expectation.

Let’s see the departures themselves — observed against expected, cell by cell:

Show the code that built this figure

expected_tbl <- chi_data |>
  with(table(graph_type, overpay)) |>
  chisq.test() |>
  pluck("expected") |>
  as_tibble(rownames = "graph_type") |>
  pivot_longer(-graph_type, names_to = "overpay", values_to = "expected")

chi_data |>
  count(graph_type, overpay, name = "observed") |>
  mutate(graph_type = as.character(graph_type), overpay = as.character(overpay)) |>
  left_join(expected_tbl, by = c("graph_type", "overpay")) |>
  pivot_longer(c(observed, expected), names_to = "source", values_to = "count") |>
  mutate(graph_type = factor(graph_type, levels = levels(chi_data$graph_type))) |>
  ggplot(aes(x = graph_type, y = count, fill = source)) +
  geom_col(position = position_dodge(width = 0.75), width = 0.68) +
  facet_wrap(~ overpay) +
  scale_fill_manual(values = c("observed" = "#4E5EAA", "expected" = "#AD872B")) +
  labs(
    title = "Observed counts against what independence predicts",
    subtitle = "Chi-square adds up the squared, scaled gaps between these bar pairs.",
    x = "Visualization condition", y = "Number of participants", fill = NULL
  )

Grouped bar chart comparing observed and expected counts for all eight cells of the four-by-two table. The CI row shows more observed overpayers than expected and fewer non-overpayers; PI shows the reverse; CI rescaled and HOPS sit close to their expected values.

The CI and PI pairs visibly disagree; CI rescaled and HOPs nearly match their expectations. Chi-square converts exactly those eight gaps into one number. Like F, it aggregates many departures at once — and so, like F, it cannot tell you which cells are responsible. (That question has its own tool, just below.)

Why the reference distribution is a chi-square distribution

Every other test in this Module borrowed its reference distribution from somewhere you had already been. Here we can at least see where the chi-square shape comes from, using ideas M06 already gave you — and it is worth doing, because the sketch explains the name, the shape, and the one condition this test actually leans on.

Follow a single cell. How many of the 231 participants who saw the CI display will overpay? Each of them independently either does or doesn’t, and under $H_0$ each does so with the same probability. That is precisely a binomial count — M06’s first named distribution.

Now the three moves:

A cell count is binomial. $O_{ij} \sim \text{Binomial}(n, p)$, with mean $E_{ij}$.
At large expected counts, a binomial is approximately Normal. This is the Central Limit Theorem doing the work M06 showed it doing: a binomial count is a sum of many independent 0/1 outcomes, and sums of many independent things go Normal. This is the move that requires the expected counts to be reasonably large — and it is exactly where the guideline on expected counts comes from. That rule is not arbitrary etiquette; it is the condition under which this approximation holds.
Scale each departure and square it. Divide each cell’s departure by $\sqrt{E_{ij}}$, square the result, and add across cells. Dividing by $\sqrt{E_{ij}}$ is what puts a cell expecting 30 and a cell expecting 3,000 on the same footing, and squaring makes over- and under-shooting count alike.

\[\chi^2 = \sum_{i,j} \frac{(O_{ij} - E_{ij})^2}{E_{ij}} = \sum_{i,j} \left( \frac{O_{ij} - E_{ij}}{\sqrt{E_{ij}}} \right)^{2}\]

Summing squared Normal-ish quantities is what produces a chi-square shape — a chi-square distribution is, by definition, what you get when you square standard Normal variables and add them up. That is why the statistic cannot be negative, why its distribution is right-skewed, and why it carries the name. Squaring throws away the sign, which is the formal reason a chi-square test has no direction: a cell below its expectation and a cell equally far above contribute identically.

Why that is a sketch and not a proof.

Three things are glossed over above, and it is worth knowing they are there rather than believing the argument is airtight.

First, the scaled departure $(O_{ij} - E_{ij})/\sqrt{E_{ij}}$ is not exactly a standard Normal. A binomial count has standard deviation $\sqrt{np(1-p)}$, not $\sqrt{np} = \sqrt{E}$, so the scaled quantity has variance $1 - p$ rather than 1. Second, the eight cell counts are not independent of one another — they are one draw from a joint (multinomial) distribution, and the row and column totals constrain them. Third, the expected counts are not handed to us by theory; they are estimated from the margins of the very table being tested.

Each of those pulls in a direction the simple sketch ignores, and remarkably they work out: the Pearson statistic really does converge to a chi-square distribution — but with $(r-1)(c-1)$ degrees of freedom rather than one per cell, which is precisely the accounting the next paragraph does. Getting there properly requires the joint multinomial and a rewrite of the sum in terms of genuinely independent pieces, and that is beyond this course. What survives, and what you should carry: larger values mean a bigger aggregate departure from the null model, and the chi-square curve is the right yardstick once expected counts are large enough.

Which brings us to df, and it is the counting principle one last time. A chi-square’s degrees of freedom count how many of those squared z-scores are free. Writing $r$ for the number of rows and $c$ for the number of columns, our table has $r \times c = 4 \times 2 = 8$ cells, but they are not eight free quantities: the expected counts were computed from the row and column totals, so those totals are fixed, and once you know them, filling in some cells forces the rest. Work along one row of two cells — fix one and the row total determines the other, so each row donates $c - 1 = 1$ free cell. Work down the columns the same way and you have $r - 1 = 3$ free rows. Multiply:

\[df = (r - 1)(c - 1) = (4 - 1)(2 - 1) = 3\]

Try it on a 2 × 2 table and the arithmetic gives $df = 1$: fill in a single cell and the margins determine the other three. That is a genuinely useful thing to be able to see, and it is the same idea that gave Test 1 its $n - 1$.

Now compute it, exactly as we did in the warm-up — each cell’s squared departure, scaled by what independence expected there, then summed:

chi_cells <- chi_data |>
  count(graph_type, overpay, name = "observed") |>
  mutate(graph_type = as.character(graph_type), overpay = as.character(overpay)) |>
  left_join(expected_tbl, by = c("graph_type", "overpay")) |>
  mutate(contribution = (observed - expected)^2 / expected)

chi_cells

sum(chi_cells$contribution)

[1] 16.29931

$\chi^2 = 16.3$, accumulated across all eight cells. Look at which cells did the work: the CI and PI rows contribute most of the total, while CI rescaled and HOPs contribute almost nothing — the same pattern the bar chart showed, now in numbers.

This test’s reference distribution is a $\chi^2$ curve with 3 degrees of freedom, and here is where 16.3 falls on it:

Show the code that built this figure

chi_tbl <- chi_data |> with(table(graph_type, overpay))

# (r - 1)(c - 1), read off the table itself rather than typed in
chi_df_ref <- (nrow(chi_tbl) - 1) * (ncol(chi_tbl) - 1)

ref_dist_plot(
  dens = function(x) dchisq(x, df = chi_df_ref),
  crit = qchisq(0.95, chi_df_ref),
  obs  = chi_hand,   # the value we just summed by hand, above
  xlim = c(0, 20),
  stat_label = "chi-square",
  subtitle = sprintf(
    "chi-square distribution, df = %d — upper tail only, because the statistic cannot be negative\nCritical value = %.2f",
    chi_df_ref, qchisq(0.95, chi_df_ref))
)

The chi-square distribution with 3 degrees of freedom on a zero-to-twenty axis, right-skewed with most of its mass below 5. The upper-tail rejection region beyond the critical value of 7.81 is shaded heather, and the observed chi-square of 16.3 is marked far out in that tail.

chi_result <- chi_data |>
  with(table(graph_type, overpay)) |>
  chisq.test() |>
  tidy()

chi_result

Study the output before you read on. Find the statistic, the df, and the p-value — and ask yourself where a contingency table’s df comes from.

Check your answers

statistic = 16.30 — the accumulated, scaled departure of all eight observed counts from their independence expectations.
parameter (df) = 3 — for a table, $df = (r-1)(c-1) = (4-1)(2-1) = 3$: once the margins are fixed, only three cells are free to vary.
p-value < .001 — if display and overpaying were truly independent, a $\chi^2$ this large would be very rare. We reject $H_0$: overpaying depends on the display.

One thing R does silently that you need to know about: Yates’s continuity correction.

The derivation above approximated a discrete thing (counts, which arrive in whole numbers) with a continuous one (the Normal, and hence the $\chi^2$ curve). That approximation is roughest when the table is smallest, and by convention R patches it — but only for 2 × 2 tables. When $df = 1$, chisq.test() automatically applies Yates’s continuity correction, subtracting 0.5 from each absolute departure before squaring:

\[\chi^2_\text{Yates} = \sum_{i,j} \frac{\left(|O_{ij} - E_{ij}| - 0.5\right)^2}{E_{ij}}\]

This shrinks the statistic and enlarges the p-value. It happens without you asking, and the only announcement is a line in the output reading Pearson's Chi-squared test with Yates' continuity correction.

Why this matters for Project 2: if you reproduce a 2 × 2 analysis and your $\chi^2$ comes out slightly smaller than the published one, you have very likely not made a mistake — you applied the correction and the original authors didn’t, or the reverse. Suppress it with correct = FALSE to compare like with like, and say in your write-up which you used. Our table is 4 × 2, so $df = 3$ and no correction is applied here; the statistic you see is the uncorrected Pearson $\chi^2$.

(The correction is a pragmatic patch rather than a result derived from theory — Yates proposed it because the corrected test simply behaved better, and statisticians still disagree about whether it over-corrects.)

Quantify and visualize the effect

Raw units first. The effect is a set of proportions, and they are the most interpretable numbers in this whole test: 54% of CI viewers overpaid, versus 35% of PI viewers — a 19-percentage-point gap in the probability of paying more than the upgrade is worth, with CI rescaled and HOPs in between.

The standardized companion is Cramér’s V — chi-square’s analog of Cohen’s d, scaling the association to a 0–1 range comparable across tables of different sizes:

chi_data |> with(table(graph_type, overpay)) |>
  cramers_v(adjust = TRUE, alternative = "two.sided")

(As with η², alternative = "two.sided" requests a conventional two-sided 95% CI rather than effectsize’s one-sided default.)

Two things about that call are worth flagging, because they will otherwise cost you a confused half-hour. First, adjust = TRUE returns the bias-adjusted form of Cramér’s V, which is what we report throughout. The textbook formula, $V = \sqrt{\chi^2 / [N(k-1)]}$ with $k$ the smaller of the two table dimensions, gives 0.134 here — close to, but not the same as, the 0.121 printed above. Neither is wrong; the adjusted version corrects a small upward bias that matters most in small samples. Say which one you report. Second, the column in the output is named Cramers_v_adjusted precisely so you cannot confuse them.

V	Conventional label
< 0.1	Negligible
0.1 – 0.3	Small
0.3 – 0.5	Medium
≥ 0.5	Large

These thresholds are rough conventions; the right reading depends on the table’s dimensions and the substantive stakes.

Which cells drive the association? Cramér’s V, like η², is silent on that. The tool is standardized residuals — one per cell, roughly standard-Normal under independence. As a descriptive heuristic, cells beyond about $\pm 2$ are the ones carrying the association; treat that as a way of reading the pattern rather than as eight formal tests, since the residuals are correlated and a cellwise claim would need its own multiplicity care:

chi_data |> with(table(graph_type, overpay)) |> chisq.test() |> pluck("stdres") |> round(2)

             overpay
graph_type    did not overpay overpaid
  CI                    -3.25     3.25
  CI rescaled           -0.31     0.31
  PI                     3.35    -3.35
  HOPS                   0.17    -0.17

The pattern is Test 3’s map, redrawn in counts: the CI row is over-represented among overpayers (standardized residual = 3.2) and PI under-represented (-3.4), while CI rescaled and HOPs sit essentially at independence. Two tests, two outcome types, one substantive story.

Write the result and define its scope

APA-style reporting · Chi-Square Test of Independence

A chi-square test of independence indicated a significant association between visualization condition and overpaying the risk-neutral price of $65, χ²(3, N = 910) = 16.30, p < .001, V = 0.121, 95% CI [0, 0.183]. Participants who saw a 95% confidence interval were the most likely to overpay (54%) and participants who saw a 95% prediction interval the least (35%), with the rescaled-CI (45%) and hypothetical-outcome-plot (44%) conditions in between.

Note the APA format for chi-square: χ²(df, N = sample size) = value. Reporting N matters here more than usual: for a fixed pattern of proportions, the Pearson χ² statistic scales directly with N — double the sample, double the statistic — so N is what lets a reader interpret the statistic alongside the effect size.

In plain language: the display didn’t just nudge average payment — it changed how many people crossed the line from a defensible price into overpayment, from 35% under PI to 54% under CI.

What this analysis does not establish. The conclusions live within the Large effect-size condition and depend on the $65 cutpoint — defensible here because the design fixes it, but a different (equally defensible) threshold would be a different analysis, and agreement across thresholds is worth checking before leaning on any single one. And remember what this test gave up: Test 3 used every dollar of the WTP scale; this test compressed it to one bit per person. When both analyses are available and agree — as they do here — the continuous one is the primary evidence and the categorical one is the more communicable restatement.

When expected counts are small: Fisher’s exact test.

Pearson’s chi-square relies on a large-sample approximation that degrades when expected cell counts fall below about 5. Fisher’s exact test is not a “nonparametric fallback” in the usual sense — it is an exact test for contingency tables, computing the exact probability of tables at least as extreme as yours under independence. It’s the standard move for small expected counts, and it agrees with chi-square asymptotically⁷ at large N.

chi_data |> with(table(graph_type, overpay)) |> fisher.test()

(Directional alternatives apply only to 2×2 tables; for a 4×2 like ours, Fisher’s exact is inherently two-sided.)

Test 5: A new design — when the same people are measured twice

A design clinic — on simulated data

Everything so far compared different people in different groups. This last section changes the design one more time: the same people, measured twice. It is deliberately shorter than the other tests, because its machinery reduces to something you already own — and it runs on simulated data. Hofman et al. measured each participant once; we imagine an extension in which each participant rated WTP both before and after seeing the visualization, and simulate it with known properties. The numbers below describe our simulation, not the study’s participants — what’s real here is the design lesson.

Recognize the design

The ingredients: one continuous outcome (WTP), two measurements per participant (before and after), and — the decisive feature — the two measurements are not independent. Your pre score and your post score share everything that’s stable about you: your baseline generosity, your attitude to fictional currency, your mood. The parameter in question is the population mean within-person change, $\mu_d$.

Why not treat pre and post as two groups and run Test 2? Because person-to-person differences are usually the largest source of variability in psychological data — and in a two-sample analysis all of it lands in the noise. The paired design’s move is to subtract each person’s pre from their own post and analyze the change scores. Whatever is stable within a person is present in both measurements and cancels in the subtraction, so the noise floor drops.

One condition on that, worth stating plainly. The cancelling only happens to the extent that the two measurements are positively correlated — which is what $\text{Var}(d) = \text{Var}(post) + \text{Var}(pre) - 2\,\text{Cov}(pre, post)$ says: no covariance, no reduction. Repeated measures on the same person usually are correlated, and our simulation below assumes a within-person correlation of .5, so the benefit is real here. But it is a property of the measurements, not a guarantee of the design. A paired design also gains precision a second, independent way — every participant contributes under both conditions, instead of the sample being split between two groups — and that gain holds regardless of the correlation. The two routes are worth keeping separate in your head, because they can come apart: the M09 bonus activity works through real repeated-measures data where the correlation is essentially zero, so the second route delivers and the first does not.

And one cell of the design grid we’re leaving empty. This Module now covers three of the four combinations of outcome type × independent or paired:

	Independent observations	Same units measured twice
Continuous outcome	two-sample t (Test 2)	paired t (here)
Categorical outcome	$\chi^2$ independence (Test 4)	McNemar’s test

The missing cell has a name worth knowing: McNemar’s test (mcnemar.test()), for a categorical outcome measured twice on the same people — did they intend to vote for the candidate before the ad, and after? You cannot use a chi-square test of independence there, and the reason is the one this whole section is about: 100 people measured twice give you 200 observations that are not independent, and chi-square’s derivation assumed they were. McNemar’s fix is elegant — it tabulates each person once, in a table of before against after, and then examines only the off-diagonal cells, the people who changed. Those who answered the same way both times carry no information about whether opinions shifted, so the test ignores them. That is the same instinct as the paired t-test’s: throw away everything stable within a person and analyze only the change.

The trick: a paired t is a one-sample t on the differences

Compute $d_i = post_i - pre_i$ for each participant, and test $H_0: \mu_d = 0$ with a one-sample t-test — Test 1’s machinery, unchanged, applied to a column of change scores against a benchmark of zero. The whole game is in taking the differences first.

State the parameter and hypotheses

In words: $\mu_d$ is the population mean of the within-person change in WTP from before to after viewing the visualization.

\[H_0: \mu_d = 0 \quad \text{(seeing the display produces no average change)} \qquad H_a: \mu_d \neq 0\]

See the data and assess the model

We simulate 40 participants with a $5 average increase, realistic spread, and a plausible illustrative within-person correlation of .5. These three numbers are ours to choose rather than estimates from data, so they are stated here and set in the code below. The construction is fully visible:

Show the simulation code

# Simulate a pre-post WTP study on n participants. The simulation produces a
# moderate within-subject effect: post is, on average, $5 higher than pre,
# with a realistic between-subject SD and a target within-subject correlation
# of about 0.5 (a plausible illustrative value for repeated measures).
#
# Construction: every participant has a "shared" component (drives within-
# subject correlation) plus an independent residual on each occasion. This is
# the textbook way to hit a *named* target rho with equal pre/post variances —
# unlike the simpler `post = pre + e` form, which yields a much higher rho.

set.seed(909)

n_pp       <- 40                                # participants
true_pre   <- 50                                # population pre-mean WTP
true_diff  <- 5                                 # population mean change (post - pre)
sd_between <- 12                                # marginal SD on each occasion
rho        <- 0.5                               # target within-subject correlation
shared_sd  <- sd_between * sqrt(rho)            # shared (subject-level) SD
indep_sd   <- sd_between * sqrt(1 - rho)        # independent residual SD per occasion

shared_component <- rnorm(n_pp, mean = 0, sd = shared_sd)

blorg_prepost <- tibble(
  participant = 1:n_pp,
  pre = round(true_pre + shared_component +
                 rnorm(n_pp, mean = 0, sd = indep_sd)),
  post = round(true_pre + true_diff + shared_component +
                 rnorm(n_pp, mean = 0, sd = indep_sd))
)

blorg_prepost |> head()

Design assumptions — and for paired data, the crucial one is bookkeeping: the participant column is what pairs each pre with its own post. Lose the ID and the pairing is gone — the analysis silently degrades into an inappropriate two-sample test. The pairs must be independent of each other (one participant’s change doesn’t influence another’s ✓), while the two observations within a pair are intentionally dependent — that dependence is the design’s whole point.

Model considerations — the t-test now runs on the differences, so the differences are what we inspect:

Show the code that built this figure

diffs_df <- blorg_prepost |> mutate(d = post - pre)

p_hist <- diffs_df |>
  ggplot(aes(x = d)) +
  geom_histogram(binwidth = 2, fill = "#945297", color = "white", alpha = 0.7) +
  geom_vline(xintercept = 0, color = "#1C2433", linewidth = 0.6, linetype = "dashed") +
  labs(title = "Distribution of within-subject differences",
       x = "Δ WTP (post − pre)", y = "count")

p_qq <- diffs_df |>
  ggplot(aes(sample = d)) +
  stat_qq(color = "#945297", alpha = 0.7) +
  stat_qq_line(color = "#1C2433") +
  labs(title = "Q-Q plot of differences", x = "theoretical", y = "sample")

p_hist + p_qq

Two panels checking the within-subject differences: a histogram of pre-to-post differences centered above zero, and a Q-Q plot of those differences lying close to the diagonal reference line, indicating approximate Normality.

How to read the right-hand panel, since this is the Module’s only Q-Q plot and it repays thirty seconds. A Q-Q plot (“quantile–quantile”) sorts your observations from smallest to largest and plots each one against the value it would have had at that position if the data were perfectly Normal. Perfect Normality would put every point exactly on the diagonal line. So you are not looking for a pattern — you are looking for departures from a straight line, and their shape is diagnostic:

Points hugging the line — Normal enough.
Both ends curving away from the line in the same direction — skew.
The middle flat and both ends bending sharply away — heavy tails, more extreme values than a Normal would produce.
A few isolated points off the line at one end — outliers, rather than a wrong distributional shape.

The reason a Q-Q plot beats a histogram for this job is resolution in the tails, which is where non-Normality actually threatens a t-test and where a histogram’s bin widths hide everything.

Our differences are roughly bell-shaped and the points lie close to the line — comfortable territory for a t-test even at n = 40.

You will also see this assessed with a formal test, usually the Shapiro–Wilk (shapiro.test()), whose null hypothesis is that the data are Normal. We look at the plot instead, for two reasons that are by now familiar. It has the gatekeeping problem — making one test conditional on another distorts the error rate of the pair. And it has a perverse sensitivity to sample size: at n = 40 it will miss all but gross non-Normality, while at n = 5,000 it will flag departures far too small to trouble a t-test at all. The plot answers the question you actually have — is this bad enough to matter? — which a p-value cannot.

The most informative display for paired data connects each person’s two scores:

Show the code that built this figure

paired_long <- blorg_prepost |>
  pivot_longer(c(pre, post), names_to = "time", values_to = "wtp") |>
  mutate(time = factor(time, levels = c("pre", "post")))

paired_summary <- paired_long |>
  group_by(time) |>
  summarize(mean_wtp = mean(wtp), se = sd(wtp) / sqrt(n()))

paired_long |>
  ggplot(aes(x = time, y = wtp)) +
  # Per-participant lines, semi-transparent
  geom_line(aes(group = participant), color = "#945297", alpha = 0.3) +
  geom_point(aes(group = participant), color = "#945297", alpha = 0.5, size = 1.5) +
  # Group means with SE error bars, in dark ink, bigger
  geom_errorbar(data = paired_summary,
                aes(y = mean_wtp, ymin = mean_wtp - se, ymax = mean_wtp + se),
                color = "#1C2433", width = 0.12, linewidth = 1) +
  geom_point(data = paired_summary, aes(y = mean_wtp),
             color = "#1C2433", size = 4) +
  geom_line(data = paired_summary, aes(y = mean_wtp, group = 1),
            color = "#1C2433", linewidth = 1) +
  labs(
    title = "Within-subject change: pre to post WTP",
    subtitle = "Each light line = one participant.   Dark line = group mean ± SE.",
    x = NULL,
    y = "Willingness to pay (Ice Dollars)"
  )

A paired-connection plot: thin lines link each participant's pre and post scores, with the group pre and post means and their error bars overlaid; most lines slope upward, showing a general pre-to-post increase.

Most lines slope upward; a handful slope down — individual variation doing what it always does. One caution when reading this plot: the error bars show the SE of the raw pre and post means, but the paired test’s inference rests on the SE of the differences — typically smaller, because differencing removes each person’s baseline.

Run and read the test

Compute the differences, then run Test 1’s machinery against zero:

diffs_only <- blorg_prepost |> mutate(diff = post - pre)

paired_result <- diffs_only |>
  t_test(response = diff, mu = 0)

paired_result

Equivalently, base R’s t.test() takes paired = TRUE and does the subtraction internally — identical answer:

blorg_prepost |>
  with(t.test(post, pre, paired = TRUE)) |>
  tidy()

Study the output before you read on. It is Test 1’s tibble, but two of the five numbers have quietly changed meaning. Ask: the estimate is an average — of what? And: the df counts something — participants, or measurements?

Check your answers

estimate = 5.42 — $\bar{d}$, the mean within-person change in Ice Dollars. Not a group mean; an average of 40 individual differences.
statistic = 2.52 — that mean change divided by the SE of the differences.
t_df = 39 — $n - 1$ where n counts participants, not measurements. Eighty numbers went in; forty differences came out, and the df follows the differences.
p_value = .016 — a mean change this size is highly incompatible with “no average change.”
95% CI = [$1.07, $9.78] — brackets $\mu_d$, the population mean change, and excludes zero.

And the reference distribution — smallest df in the Module. With 40 participants, $df = 39$, and this is the one place where the t-distribution’s heavier tails actually matter. Same two functions as every other t-test in this Module, handed the smallest df we have used:

df_5      <- paired_result |> pull(t_df) |> unname()
t_obs_5   <- paired_result |> pull(statistic) |> unname()
t_star_5  <- qt(0.975, df = df_5)                   # two-tailed cutoff
p_check_5 <- 2 * pt(-abs(t_obs_5), df = df_5)       # both tails, so doubled

c(df = df_5, t_obs = t_obs_5, t_star = t_star_5, p = p_check_5) |> round(4)

     df   t_obs  t_star       p 
39.0000  2.5214  2.0227  0.0159

That critical value of 2.023 sits noticeably above the 1.96 that Tests 1 and 2 used at their much larger df — and the p-value reproduces what t_test() reported, from nothing but the statistic and the df.

Show the code that built this figure

ref_dist_plot(
  dens = function(x) dt(x, df = pp_df),
  crit = c(-t_star_5, t_star_5),
  obs  = paired_result$statistic,
  xlim = c(-5, 5),
  stat_label = "t",
  subtitle = sprintf(
    "t-distribution, df = %d\nNote t* = %s%.2f — further out than the %s1.97 of Tests 1 and 2, because df is small",
    pp_df, "\u00b1", t_star_5, "\u00b1")
)

The t-distribution with 39 degrees of freedom spanning minus 5 to plus 5, with heather rejection regions beyond plus and minus 2.02. The observed t is marked in the right tail beyond the critical value.

That is M08’s df lesson made concrete: fewer degrees of freedom, fatter tails, a higher bar to clear. Same $\alpha$, same logic — a different curve, because the design gave us less information.

Quantify and visualize the effect

Raw units first: WTP rose by about $5.42 per person on average, 95% CI [$1.07, $9.78]. The standardized companion for paired designs is Cohen’s $d_z$ — the mean difference scaled by the SD of the differences (not of the raw scores; the $z$ subscript flags that the relevant variability is within-person change):

\[d_z = \frac{\bar{d}}{s_d}\]

# effectsize emits an informational note here pointing at
# repeated_measures_d(); it is package chatter, not part of the result.
dz_obj <- blorg_prepost |>
  with(cohens_d(post, pre, paired = TRUE))
dz_obj

The usual 0.2 / 0.5 / 0.8 conventions get applied to $d_z$ too — with the usual caveat that they are field-blind rules of thumb. One structural note: because $s_d$ shrinks as the pre–post correlation grows, the same raw change yields a larger $d_z$ in a well-paired design — standardized effect sizes are not comparable across designs that standardize by different quantities.

Write the result and define its scope

APA-style reporting · Paired-Samples t-Test (Repeated Measures)

A paired-samples t-test indicated that participants’ willingness-to-pay scores were significantly higher after viewing the visualization than before, $\bar{d}$ = 5.42 Ice Dollars, 95% CI for the mean difference [1.07, 9.78], t(39) = 2.52, p = .016, $d_z$ = 0.4, 95% CI [0.07, 0.72].

In plain language: in our simulated extension, mean willingness to pay was about $5.42 higher after the visualization than before. (Note the deliberately descriptive wording — a before-and-after change on its own does not establish that the display caused it, for the reasons in the next paragraph.)

What this analysis does not establish. Everything — the design lesson aside — is bounded by the word simulated: these numbers characterize a data-generating process we chose, not Hofman et al.’s participants. (That’s also this section’s quiet second lesson: simulating a dataset with known properties is how you rehearse an analysis plan before real data arrive — a practice you’ll meet again as a research skill in its own right.) And even in a real pre–post study, a before–after change conflates the display’s effect with everything else that happens between the two measurements — time, familiarity, fatigue. Randomized between-person comparisons (Tests 2–3) and within-person designs answer related but different questions, each with its own vulnerabilities.

When the standard analysis is unsuitable: Wilcoxon signed-rank (paired form).

For heavily skewed differences with small n (typically < 20), the Wilcoxon signed-rank test ranks the absolute differences and signs the ranks. Same estimand caution as its one-sample form: it tests symmetry of the difference distribution around zero, not the mean difference per se.

blorg_prepost |> with(wilcox.test(post, pre, paired = TRUE))

More than two time points? The paired-t logic generalizes to models that handle the dependence among repeated measurements — classically repeated-measures ANOVA (which adds a sphericity assumption), more flexibly linear mixed-effects models. The statistic becomes an F; the tidyverse-friendly entry points are rstatix::anova_test() and afex::aov_car(). Both lie past this course’s edge, but you now hold the concept they build on.

The five tests, side by side

The Module opened with a claim: one logic, five designs. Here is that claim, made checkable — every row answers the same six questions:

The synthesis table

Test	The unit & design	The parameter	$H_0$ predicts	Statistic (APA form)	Raw effect to report	Standardized ES
One-sample t	Person; one group vs benchmark	$\mu$	$\bar{x}$ near $\mu_0$	t(df)	mean, vs the benchmark	Cohen’s d
Two-sample t (Welch)	Person; two independent groups	$\mu_1 - \mu_2$	$\bar{x}_1 - \bar{x}_2$ near 0	t(df)	difference in means, with CI	Cohen’s d
One-way ANOVA	Person; 3+ independent groups	the $k$ population means $(\mu_1, \dots, \mu_k)$	group means spread ≈ within-group noise	F(df₁, df₂)	the group means	$\eta^2$ (+ pairwise CIs)
Chi-square	Person; two categorical variables	conditional proportions across groups	observed counts ≈ expected counts	$\chi^2$(df, N = total)	the group proportions	Cramér’s V
Paired t	Person measured twice	$\mu_d$	$\bar{d}$ near 0	t(df)	mean within-person change	Cohen’s $d_z$

Shared across all five: independent sampling units (for paired designs, independent pairs), $\alpha$ set in advance, an effect size with CI reported alongside the p-value, the raw effect reported before the standardized one, and the direction named in plain English.

Follow-ups: the three t-tests deliver their specific comparison directly. A significant ANOVA calls for follow-up matched to the question — Tukey-adjusted pairwise comparisons when all pairs are of interest, planned contrasts when hypotheses were specified in advance (Multiple comparisons deep-dive). A significant chi-square is unpacked with standardized residuals to find the driving cells.

Two more tests were named along the way, both for designs the five rows above don’t reach. The chi-square goodness-of-fit test (Test 4’s warm-up) takes one categorical variable and compares its proportions against values you specify in advance — the categorical twin of the one-sample t. McNemar’s test (named in Test 5) handles a categorical outcome measured twice on the same people, where the independence that chi-square requires fails by construction.

Alternatives when the model is unsuitable (Wilcoxon forms, Mann-Whitney, Kruskal-Wallis, Fisher’s exact): remember that most of them test a different population quantity than their standard counterpart. Choose by asking which parameter your research question needs — not by glancing at a histogram.

Keep this table close. It is the one-page version of the Module — built for the moment, a semester or three from now, when your own data are in front of you and you need to pick the right test, run it, and report it without re-reading the whole chapter.

Looking ahead

Each of the five tests connects naturally to the regression-modeling framework. The three t-tests and the ANOVA are — literally — special cases of the linear model; the chi-square test connects to its categorical-outcome cousin, the generalized linear model (logistic or log-linear). The linear model is one model with the unifying form

\[Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \epsilon\]

where the $X$’s can be continuous, binary, or categorical, and (with link functions) the response $Y$ doesn’t even need to be continuous. Once you see regression as the parent framework, the five tests above stop looking like five different things and start looking like five ways of writing — or closely approximating — the same model:

M09 test	What the GLM equivalent looks like in R	Notes
One-sample t	`lm(y - mu0 ~ 1)`	Intercept-only regression on the centered outcome; the intercept tests whether $\mu - \mu_0 = 0$.
Two-sample t	`lm(y ~ group)`	Regression with one binary predictor; the slope on `group` is the difference in means.
Paired-samples t	`lm(diff ~ 1)` or `lmer(y ~ time + (1\|id))`	Intercept-only regression on the difference scores; the intercept is $\bar{d}$.
One-way ANOVA	`lm(y ~ factor)`	Regression with one categorical predictor; the F-test on the factor is what `aov()` reports.
Chi-square (2×2, binary outcome)	`glm(y ~ x, family = binomial)`	For a 2×2 table, logistic regression with one binary predictor asks a closely related independence question; larger tables call for log-linear or multinomial models.

Why does this matter? Because once you can write any of these tests as a regression model, extending it is just a matter of adding to the formula — more predictors, their interactions — without re-learning a new “test” for every design. (The same regression logic reaches further still, to mixed-effects models and to different outcome families for non-normal data, but those lie beyond this course.) The five tests in this Module are not the destination; they’re the foundation for the regression-based second half of the course.

Module 10 introduces simple linear regression. Module 11 layers in multiple predictors. The same lm() / tidy() / anova() workflow you already saw in Test 3 carries you all the way from there to your dissertation analyses.

Summary

One logic, five designs. Every test in this Module asked how compatible the observed data are with a null model, decided against a pre-set $\alpha$, and then did the more important work: reporting the effect in raw units, bounding it with a confidence interval, standardizing it for comparability, and stating a claim whose scope matches the design. What changed from test to test was never the reasoning — it was the design: one group against a benchmark, then two randomized groups, then four, then a categorical outcome, then two measurements per person.

When your own data arrive, start where this Module starts: outcome type, number of groups, independent or paired, and the parameter your question names. The decision map turns those answers into a test; the synthesis table turns the test into a complete, honest report.

Footnotes

The large-effect condition is the one place the published numbers don’t quite reconcile, and it’s worth knowing why. The concrete stimulus parameters are the boulder distances: 116 m against 100 m with $\sigma = 15.3$ m implies Cohen’s d = 1.046 and a probability of superiority of .770. The article rounds that effect to “approximately 1.0” and reports the probability as 76% — which is the value a d of exactly 1.0 gives, $\Phi(1/\sqrt{2}) = .760$. The gap is about one percentage point. The small-effect condition has no such gap: 104 m gives d = 0.261 and P = .573, and both the article’s “0.25” and the pre-registration’s “approximately 0.26” round to the same stated 57%. We use 76% throughout because it is the benchmark the authors themselves analyze against — their normative willingness-to-pay of $65 is computed as $250 \times .76 - 250 \times .50$. The habit worth taking from this: published stimulus parameters don’t always reconcile to the last decimal, and recomputing them is how you find out.↩︎
iff is standard mathematical shorthand for “if and only if” — it is not a typo for “if,” and you will meet it throughout the statistics literature. It is a stronger claim than a plain “if,” because it asserts the implication in both directions at once. Saying “the CI excludes $\mu_0$ if the test rejects” would leave room for a CI that excludes $\mu_0$ even though the test failed to reject. Saying iff rules that out: rejecting guarantees the CI excludes $\mu_0$, and a CI that excludes $\mu_0$ guarantees the test rejects. That two-way guarantee is exactly why the CI and the test can be called the same decision rather than merely similar ones. Whenever you meet iff, the useful habit is to check that you believe both directions — that is the entire content of the word.↩︎
A straw man is a position set up mainly so that it can be knocked over — in hypothesis testing, a null that nobody seriously entertains, so that rejecting it establishes very little. The criticism has real force against some uses of the nil null ($\mu = 0$): if no one ever believed the effect was exactly zero, “we rejected zero” is a thin finding. But it is not a general indictment of nulls at zero. A null of zero often encodes a genuine scientific claim — this treatment does nothing, these two groups do not differ — and rejecting it is then informative. What separates a useful null from a straw man is not the number but whether someone could defensibly hold it. M08’s worked example tested against $32,000 rather than zero for exactly this reason, and $\mu_0 = \$17.50$ here qualifies on the same grounds: it is what a rational-choice model actually predicts.↩︎
A reminder about what we are and aren’t using here. Experiment 2 is an 8-cell design: four visualization formats (CI, CI rescaled, PI, HOPs) crossed with two effect sizes (Small, Large). Test 2 deliberately narrows that twice over. We keep only two of the four displays — CI and PI, the pair the paper’s central hypothesis contrasts — and only the Large effect-size condition, so that both groups face the identical decision problem (a special boulder genuinely worth $65). That leaves 467 of Experiment 2’s participants in this analysis. Neither cut is a limitation of the t-test; a two-sample t simply answers a two-group question, so we hand it two groups. Test 3 puts all four displays back in and needs a different test to do it, and the Small effect-size condition stays out of every analysis in this Module — see the Meet the data box on the full factorial.↩︎
Omnibus is Latin for “for all” — the same root as the vehicle that carries everybody. An omnibus test asks one question about all the groups at once: could every one of these population means be equal? Its answer is a single yes-or-no covering the whole set, which is both its strength and its limitation. The strength is bookkeeping: one test, one decision, one 5% false-positive rate, no matter how many groups you have. The limitation is precision: rejecting tells you “not all equal” and nothing more — not which group stands out, not by how much, not in which direction. That is why a significant omnibus result is the beginning of the analysis, and why the follow-up comparisons later in this section exist. You will meet the same word attached to other tests (an omnibus F in regression, for instance, asking whether any predictor matters); it always means the same thing — one test spanning a whole set of possibilities.↩︎
Where that 26% comes from — the probability rules of M06. Three of them, chained. (1) If a single test has a 5% chance of a false positive, then by the complement rule it has a $1 - .05 = .95$ chance of not producing one. (2) The six comparisons are (near enough) independent, so the multiplication rule for independent events says the chance that all six stay clean is $.95 \times .95 \times \dots = (.95)^6 = 0.735$. (3) “At least one false positive” is the complement of “none at all,” so we flip back: $1 - (.95)^6 = 0.265$, about 26%. The same at-least-one-out-of-many move you used in M06 for coin flips and for the fake-news detective; only the labels have changed. (Read this as the arithmetic if the six tests were independent. Pairwise comparisons that share a group are correlated, so the exact familywise rate for this particular family differs from .26 — the qualitative point, that unadjusted repeated testing pushes the chance of at least one false positive well above .05, is unaffected.)↩︎
Asymptotically means in the limit, as the sample size grows without bound — so “these two tests agree asymptotically” is shorthand for “the bigger your sample, the closer their answers get, and in the limit they coincide.” It is one of the most common words in statistics writing and one of the least often explained. You have already relied on the idea twice in this Module without the label: the Central Limit Theorem is an asymptotic result (the sampling distribution of a mean approaches Normal as n grows), and the derivation above made the χ² curve an asymptotic approximation to a discrete reality (which is exactly why it needs expected counts of at least 5, and why the continuity correction exists). The practical translation is always the same, and it is a caution rather than a reassurance: an asymptotic guarantee tells you what happens with plenty of data, and says nothing about whether your sample is big enough. That question you answer by checking the specific condition — here, the expected cell counts.↩︎