Four Tests, One Logic

Lecture · Module 9 · Mon Oct 12

How today works

Today we apply the NHST framework to four different types of research question — all using real data. For each test, the workflow is the same:

  1. State the hypotheses
  2. Check assumptions
  3. Run the test in R
  4. Interpret the output (p-value, CI, effect size)
  5. Write an APA-style result

Setup

Test 1: Two-Sample t-Test

Research question

Do participants who see a confidence interval (CI) perceive the probability of winning differently than participants who see a prediction interval (PI)?

Hypotheses

  • \(H_0\): \(\mu_{CI} = \mu_{PI}\) (no difference in mean perceived probability)
  • \(H_A\): \(\mu_{CI} \neq \mu_{PI}\) (the means differ)

The data

Run the test

Effect size

APA write-up

Writing the result

An APA-style result for a two-sample t-test includes:

Participants in the CI condition (M = X.XX, SD = X.XX) perceived the probability of winning [higher/lower/similarly] compared to participants in the PI condition (M = X.XX, SD = X.XX), t(df) = X.XX, p = .XXX, 95% CI [X.XX, X.XX], d = X.XX.

Fill in the values from the output above.

Key takeaway

The t-test told us whether the difference is statistically significant. Cohen’s d told us whether the difference is practically meaningful. Both matter.

Test 2: One-Way ANOVA

Research question

Does mean perceived probability differ across all four visualization conditions (CI, PI, CI Rescaled, HOPs)?

Hypotheses

  • \(H_0\): \(\mu_{CI} = \mu_{PI} = \mu_{CIR} = \mu_{HOPS}\) (all means equal)
  • \(H_A\): At least one mean differs

What this test does and does not answer

Note what we are not conditioning on. This ANOVA pools both effect-size conditions — Small and Large — and asks only whether the four display types differ on average across both. That is a legitimate question, but it is a narrower one than it may look:

  • It can show that mean perceived probability differs somewhere among the four displays.
  • It cannot show which displays differ — that needs the follow-up comparisons.
  • It cannot show whether a display’s effect depends on the effect size shown. That is an interaction, and testing it requires putting both factors in the model together.

Say which question you answered when you write it up. A pooled main effect and an interaction are different claims.

Interactions get their proper treatment later — M12 meets them as the additivity assumption, and PSY 653 develops them under their other name, moderation.

The data

Run the test

Effect size

Writing the result

Perceived probability of winning differed significantly across visualization conditions, F(df1, df2) = X.XX, p = .XXX, \(\eta^2\) = X.XX.

If significant, follow up with pairwise comparisons to identify which groups differ.

Test 3: Pearson Correlation

Research question

Is there a relationship between participants’ perceived probability of winning (superiority) and their willingness to pay for the advantage (wtp)?

Hypotheses

  • \(H_0\): \(\rho = 0\) (no linear relationship)
  • \(H_A\): \(\rho \neq 0\) (there is a linear relationship)

The data

Run the test

Writing the result

Perceived probability of winning was [positively/negatively] correlated with willingness to pay, r(df) = .XX, p = .XXX, 95% CI [.XX, .XX].

Note that r is both the test statistic and the effect size — no separate effect size needed.

Test 4: Chi-Square Test of Independence

Research question

Is there an association between exclamation mark presence and article type (real vs. fake) in the fake news dataset?

Hypotheses

  • \(H_0\): Exclamation mark presence and article type are independent
  • \(H_A\): There is an association between exclamation marks and article type

The data

Run the test

Effect size

Writing the result

There was a significant association between exclamation mark presence and article type, \(\chi^2\)(df) = X.XX, p = .XXX, Cramér’s V = .XX. Articles with exclamation marks were disproportionately fake.

The Unifying Message

Whole-class moment

Look at all four tests. Despite different data types and different formulas, every test followed the same logic:

  1. Define what “nothing interesting” looks like (\(H_0\))
  2. Compute how far your data are from that null model
  3. Ask: “How rare is this under the null?”
  4. Report the answer (p-value) alongside the size of the effect (CI + effect size)

All four tests are special cases of the general linear model — a unifying framework you’ll learn in Part 2 of this course.

Key message

Statistical significance means “rare under the null.” It does not mean “important,” “large,” or “certain.” Always report effect sizes alongside p-values.

Why it matters: The replication crisis taught us that \(p < .05\) alone tells you almost nothing. Effect size + CI + p-value together tell the full story.

Wrapping Up

What we covered today

Test Data types R function Effect size
Two-sample t-test Continuous ~ 2 groups t.test(y ~ x) Cohen’s d
One-way ANOVA Continuous ~ 3+ groups aov(y ~ x) \(\eta^2\)
Pearson correlation Continuous ~ Continuous cor.test(x, y) r
Chi-square Categorical ~ Categorical chisq.test(x, y) Cramér’s V

What’s next

  • Wednesday Lab: Full NHST workflow — you’ll run complete analyses independently on new research questions, write APA paragraphs, and see what p-hacking looks like.