Precision and Power

Lecture · Module 7 · Mon Sep 28

How today works

Today’s class has four parts:

  1. A recap of what the Module and Friday’s pre-study gave us.
  2. A thinking activity — you’ll pick a mean you actually want to estimate in your own research, and we’ll talk about what a useful CI for it would look like.
  3. CI width under different sample sizes — we’ll simulate and watch the interval shrink as \(n\) grows.
  4. Power — the main new content of the day. Given a CI-based study design, how large a sample do you need to reliably detect a real effect? We’ll simulate the answer, not memorize a formula.

Recap (what you’ve learned so far)

In the Module and pre-study, you built the full toolkit for quantifying uncertainty about a single sample mean:

  • The one-sample problem. You observe one sample, but you want to say something about the population. Every sample mean you could have observed is slightly different — that spread is the sampling distribution, and its SD is the standard error (SE). Bigger \(n\) shrinks SE as \(1/\sqrt{n}\). Using the pre-study’s college-mobility sample, where \(s = \$11{,}031.86\), watch what \(\sqrt{n}\) does in the denominator:

\[\begin{aligned} \text{SE}_{n=25} &= \frac{\$11{,}031.86}{\sqrt{25}} &&= \frac{\$11{,}031.86}{5} &&= \$2{,}206.372 \\[2pt] \text{SE}_{n=100} &= \frac{\$11{,}031.86}{\sqrt{100}} &&= \frac{\$11{,}031.86}{10} &&= \$1{,}103.186 \\[2pt] \text{SE}_{n=400} &= \frac{\$11{,}031.86}{\sqrt{400}} &&= \frac{\$11{,}031.86}{20} &&= \$551.593 \end{aligned}\]

  • Two construction routes for a 95% CI, both frequentist:
    • Bootstrap percentile: rep_slice_sample() + quantile() — middle 95% of 1,000 bootstrap resamples.
    • Parametric t: t_test() (or the by-hand \(\bar{x} \pm t^{\star} \cdot s/\sqrt{n}\)). Think of it like this: \(t^\star\) is the how-many-SEs multiplier — it tells us how many standard errors to go out from the estimate on each side.
  • The long-run interpretation of 95%. The 95% describes the procedure, not this interval. The RPsychologist simulator in the pre-study made this visible.

Same data, two paths to the same CI

This figure shows the two-methods comparison in the M07 Module — a single sample of \(n = 250\) colleges from Chetty et al.’s mobility data. Left panel: the bootstrap sampling distribution of the sample mean, with the middle 95% (the bootstrap CI) shaded in green. Right panel: the bootstrap CI and the parametric t-CI plotted side by side, with the margin of error annotated.

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.2.0
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.3     ✔ tibble    3.3.1
✔ lubridate 1.9.4     ✔ tidyr     1.3.2
✔ purrr     1.2.1     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

Attaching package: 'scales'


The following object is masked from 'package:purrr':

    discard


The following object is masked from 'package:readr':

    col_factor

Two panels side by side. Left: a histogram of bootstrap sample means, roughly bell-shaped and centred near $37,000, with the middle 95% shaded green between dashed lines at about $35,700 and $38,800. Right: two horizontal 95% confidence intervals for mean earnings at age 34, one labelled Bootstrap in green and one labelled Parametric in blue, each with a dot at its point estimate just above $37,000. A gold dashed vertical line marks the true population mean, and both intervals contain it. An annotated arrow beneath marks the half-width, or margin of error, of about $1,622.

Both methods land on essentially the same interval. What you’d report (parametric t-CI, the standard for published work): “Mean earnings at age 34 = $37,187 (95% CI: $35,565, $38,809), n = 250.” The half-width — about ±$1,622 — is the margin of error for that estimate.

Imagine an intervention you’d design

Before we touch R, take a minute to reflect. Much of psychological research comes down to designing an intervention or testing a manipulation — something you change, deliver, or recommend — and then asking, with statistics, whether the intervention shifted the dependent variable (i.e., outcome) you care about. Today’s question is how big a study you’d need to credibly detect that shift.

Whole-class moment

Pick a continuous outcome from your own research area, and imagine you’ve designed an intervention or manipulation to optimize it. Record:

Element Your value
The outcome variable (the continuous variable you’d hope to shift)
Your intervention (one-line description)
Plausible mean without your intervention (the status quo — what the outcome would typically be)
Plausible SD of the outcome (\(\sigma\))
Plausible mean for participants in your intervention group (\(\mu\))
A feasible sample size (\(n\)) you could actually recruit

Keep these numbers handy. We’ll work through the rest of the lecture using a calibrated birth-weight scenario, but at the very end you’ll plug your own numbers into a couple of small self-check exercises to apply the framework to your own design.

Today’s premise

Today we take the same CI machinery and ask two new, more design-focused questions:

If I only have one shot at collecting data in my optimization effort, how large does my sample need to be so my CI for the intervention group is both (a) narrow enough to be useful, and (b) likely to exclude the status-quo value — so I can credibly claim my intervention did something?

The first half of that question is about precision. The second half is statistical power.

CI width at different sample sizes

For the rest of the lecture we’ll use a scenario from M06 PS2: newborn birth weights are approximately normally distributed with \(\mu = 3254\) grams and \(\sigma = 588\) grams.

Imagine you’ve designed a prenatal-nutrition intervention for mothers who are at risk of having a low birth weight baby, and you’re running it as a pilot trial. You recruit \(n\) high-risk pregnant participants, deliver the program, and at delivery you measure their newborns’ birth weights. The report you’ll write — for a journal, a funder, or a hospital board — will present the intervention group’s mean birth weight with a 95% CI: “Among participants in our nutrition program, mean newborn birth weight was \(X\) g (95% CI: \(Y\), \(Z\)).”

For now we’re focused on how tight that CI needs to be to credibly report your intervention’s outcome. (We’ll get to “did the intervention beat the status quo?” — that’s power, in the next section.)

Click Run Code on the chunk below. It simulates four samples of different sizes — \(n = 30\), \(n = 100\), \(n = 500\), and \(n = 2000\) — from the same birth-weight population, computes a 95% CI for each, and reports the half-width (how far the bounds are from the point estimate — a direct measure of precision: smaller is more precise).

Starting point

You don’t have a real sample yet — you want to simulate what would happen if you drew samples of three different sizes from a known birth-weight population (so you can compare CI widths cleanly).

What the code does

For each of \(n = 30, 100, 500, 2000\):

  1. Draws n random observations from a Normal population with \(\mu = 3254\) and \(\sigma = 588\).
  2. Computes the 95% CI for the mean of that simulated sample.
  3. Tags the row with its sample size.

Then the four single-row results are stacked together, the half-width is computed, and the relevant columns are selected for display.

Key functions

Function What it does
set.seed(652) Fixes the random-number generator so everyone in class sees the same simulated sample — without it, every run would draw fresh random numbers.
rnorm(n, mean = 3254, sd = 588) Draws n random values from a Normal distribution with that mean and SD. This is your simulated sample.
tibble(bw = …) Wraps the simulated values into a tibble column called bw so t_test() can use it.
t_test(response = bw, conf_level = 0.95) Computes the 95% parametric t-CI for the mean of bw, returning a tibble with estimate, lower_ci, upper_ci.
bind_rows() Stacks the four single-row tibbles into one four-row tibble.

How to read output

A four-row tibble with one row per sample size. Compare the half_width column across rows: it should shrink as \(n\) grows (specifically, as \(1/\sqrt{n}\)).

CI width and sample size

The expected half-width of a CI (i.e., the margin of error) is:

\[t^{\star} \cdot \frac{\sigma}{\sqrt{n}}\]

As \(n\) grows, the denominator grows, so the half-width gets narrower. But it gets narrower slowly. The key rule of thumb is that precision improves with \(\sqrt{n}\), not with \(n\) itself: doubling \(n\) does not cut the half-width in half. To get about half the half-width, you need roughly four times as many participants.

How much precision do you need?

The math above tells you how the half-width shrinks with \(n\). It doesn’t tell you whether \(\pm 240\) g, \(\pm 117\) g, \(\pm 51\) g, or \(\pm 26\) g is good enough. That depends entirely on the decision your CI is supposed to support — precision is a property of the CI relative to the question you’re asking, not a property of the CI alone.

A useful frame: the CI is “narrow enough” when a reader who reads only the lower bound and a reader who reads only the upper bound would not draw different substantive conclusions. If both would shrug and say “about what I expected,” it’s precise enough. If one concludes “big problem” and the other “totally fine,” it isn’t.

Working backwards from precision to \(n\)

You can solve the half-width formula for \(n\) to get the sample size needed to deliver a target precision:

\[n \approx \left( \frac{1.96 \cdot \sigma}{E} \right)^2\]

where \(E\) is your target half-width — the precision the decision requires. This uses the asymptotic1 \(z^{\star} = 1.96\); at small \(n\) the right answer involves \(t^{\star}\), but for sample-size planning \(z^{\star}\) is plenty accurate.

To use this formula, you need three numbers:

  1. \(\sigma\): how variable the outcome is.
  2. An expected effect: the size of the effect you think would be meaningful or plausible.
  3. \(E\): how much uncertainty you are willing to tolerate — the CI half-width (i.e., margin of error).

Notice that the expected effect does not appear directly in the formula. It helps us decide what would count as a useful margin of error. In the examples below, we use a simple planning rule: set \(E\) to half of the expected effect. That way, if the observed estimate is close to what we expected, the 95% CI should still point clearly in the same substantive direction.

For example, if you expect an intervention to reduce a symptom score by 4 points, setting \(E = 2\) means you are aiming for a 95% CI about as wide as:

\[-4 \pm 2 = [-6, -2]\]

That interval would still tell the same basic story at either end: symptoms decreased. If the margin of error were \(\pm 6\), the interval would be \([-10, +2]\), and readers could reasonably disagree about whether the intervention helped at all.

Three applied examples

Three pilot studies, three precision targets

Each example below is framed as a single-arm pilot study you’d propose in a federal grant application. Each specifies an expected effect size in the outcome’s units and a desired precision — and from those, the required \(n\) falls out of the formula above.

1. Prenatal-nutrition pilot — NIH R01 application (our running example).

You’re writing the pilot-data section of an NIH R01 for a future multi-year RCT of a prenatal-nutrition intervention in mothers at risk of having a low-birth-weight baby. The single-arm pilot needs to estimate the intervention group’s mean birth weight with enough precision that reviewers can judge whether the program is promising enough for a larger trial. From prior studies, you anticipate an effect of \(+250\) g relative to the usual-care/status-quo mean (i.e., the status-quo mean for at-risk mothers is 2500 g), with population \(\sigma = 588\) g. You target a CI margin of error no more than \(\pm 125\) g — half of the expected effect, so that, if the observed effect is close to the anticipated \(+250\) g, the 95% CI lower bound would remain above the status-quo value.

\[n \approx \left( \frac{1.96 \cdot 588}{125} \right)^2 \approx 86\]

2. Digital CBT pilot for adolescent depression — NIMH R34/R01 application.

You’re writing the pilot-data section of a federal grant application for a future randomized trial of a brief digital CBT intervention for adolescents with elevated depressive symptoms. The single-arm pilot needs to estimate the average reduction in depressive symptoms with enough precision that reviewers can judge whether the intervention is promising enough for a larger trial. The outcome is change in PHQ-9 score from baseline to 8-week follow-up. From prior studies, you anticipate an effect of \(-4\) points, with population \(\sigma = 6\) points for the change score. You target a CI margin of error no more than \(\pm 2\) points — half of the expected effect, so that, if the observed effect is close to the anticipated \(-4\) points, the 95% CI would still suggest a meaningful reduction.

\[n \approx \left( \frac{1.96 \cdot 6}{2} \right)^2 \approx 35\]

3. Supervisor-support pilot for employee burnout — NSF/NIOSH application.

You’re writing the pilot-data section of a federal grant application for a future randomized trial of a supervisor-support intervention designed to reduce burnout in high-stress workplaces. The single-arm pilot needs to estimate the post-intervention mean burnout score with enough precision that reviewers can judge whether the intervention is likely to move employees below a high-risk burnout threshold. The outcome is burnout score at 12-week follow-up on a standardized scale, where scores of 30 or higher are considered high risk. From prior workplace-intervention studies, you anticipate a post-intervention mean of 25 points — an effect of \(-5\) points relative to the high-risk cutoff — with population \(\sigma = 10\) points. You target a CI margin of error no more than \(\pm 2.5\) points — half of the expected distance from the cutoff, so that, if the observed mean is close to 25, the 95% CI upper bound would remain below the high-risk threshold.

\[n \approx \left( \frac{1.96 \cdot 10}{2.5} \right)^2 \approx 62\]

A 3-question framework for your own design

Before you commit to an \(n\), walk through these three questions for your study. They turn “I should pick some \(n\)” into “I should pick the \(n\) that delivers the precision the decision requires.”

Three questions to set your precision target

  1. Who’s the audience for your intervention’s reported outcome, and what action will they take based on the number? Concretely — name the audience and the action. In all three pilots above, the audience is federal grant reviewers (NIH, NIMH, NSF/NIOSH) and the action is fund the full randomized trial.

  2. What’s the smallest change in your reported intervention-group mean that would change that action? Give a number with units, defended from your literature or your decision context. In the three pilots above: +250 g of birth weight (Pilot 1), −4 points on PHQ-9 (Pilot 2), and −5 points on the burnout scale (Pilot 3).

  3. For today’s activity, set \(E\) (your precision target) at half of (2). This is a useful rule of thumb: it makes the CI narrow enough that, if your estimate lands near the effect you care about, the interval still points clearly in the same substantive direction. In real studies, researchers may choose a different \(E\) based on clinical, practical, or budget constraints.

If the resulting \(n\) is realistic for a study you could run, you have a defensible design. If not, the path forward is one of three honest options: a more precise measurement (smaller \(\sigma\)), a different question (a larger smallest-change-that-matters), or a more expensive study.

Compute the \(n\) you need for your scenario

Plug your \(\sigma\) and your precision target \(E\) into the chunk below. It applies \(n \approx (1.96 \cdot \sigma / E)^2\) and rounds up to a whole sample size.

Try it for one of the example scenarios above to verify the numbers we computed by hand. Then plug in your scenario.

Now build the CI you’d actually report

The chunk above used the asymptotic \(z^{\star} = 1.96\) to plan a sample size — that’s exactly what you want for a back-of-the-envelope \(n\) calculation. Once you’ve actually committed to an \(n\) and (hypothetically) collected data, the CI you’d report uses the exact \(t^{\star}\) at \(df = n - 1\) instead. Plug your \(\mu\) (the plausible intervention-group mean), \(\sigma\), and the \(n\) you just computed into the chunk below. It returns the same machinery the t_test() call from the pre-study would deliver — \(t^{\star}\), SE, half-width, and the CI bounds.

Compare the halfwidth you got to the precision target \(E\) you set above (half the smallest change in your reported mean that would change your audience’s action). Does the half-width come in at or below your target?

  • If yes, your \(n\) is defensible for the decision the CI will support.
  • If no, modify my_n until halfwidth drops to \(E\) — that’s the \(n\) you’d need to hit your precision target. Hold that number in your head. When we get to power in the next section, we’ll see the full set of levers you have for fixing an underpowered design — and you can decide then which one makes the most sense for your study.

Power via simulation

So far we’ve been treating the CI as a precision tool: “how tightly can I pin down the mean?” But the CI also does a decision job: given my sample, can I confidently rule out some particular value — say, the status-quo value my intervention is supposed to beat?

That’s power.

Statistical power is the chance that your study would successfully show an effect, assuming the effect is really there.

Imagine repeating the same study over and over: same sample size, same population, same true effect, same analysis. Some samples would produce a clear result, and some would not. Power is the proportion of those repeated studies that would succeed.

For the CI approach we’re using here, a study “succeeds” when the 95% CI lands entirely on the effect side of the status-quo value. In other words, the interval is narrow enough, and far enough from the status-quo value, that it rules out the value that would mean “no meaningful effect.”

What counts as the status-quo value depends on how you define the outcome:

  • If the outcome is a raw post-intervention mean, it’s typically the status-quo population mean the intervention group needs to beat.
  • If the outcome is a clinically meaningful score, it might be a cutoff value participants need to fall below.
  • If the outcome is a change score, it’s usually zero change.

So in this section, power means:

How often would our 95% CI exclude the status-quo value, if the intervention really does have the effect we care about?

Let’s make that concrete. Suppose the prenatal nutrition intervention genuinely raises mean birth weight by 250 grams. The status quo for at-risk mothers is 2500 g; the intervention-group population mean would be 2750 g. The population SD (how much individual babies vary) is unchanged at 588 g.

You’ll never recruit the whole intervention population. You recruit \(n\) participants, estimate the group’s mean with a 95% CI, and ask: does my CI exclude 2500 — the status-quo mean? If yes, you have evidence the intervention did something. If no, you can’t distinguish the intervention from no-effect.

How likely are you to succeed? That’s what we’ll simulate. We’ll work the rest of this section using the birth-weight scenario above (status quo = 2500 g, true mean for our optimized intervention = 2750 g, \(\sigma\) = 588 g); at the end of the lecture you’ll get a chance to plug in your own numbers.

A picture of the idea

Before we automate this, let’s eyeball what “my CI excludes the status-quo value” actually looks like at different sample sizes. Below, we draw 50 hypothetical studies at each of five sample sizes (using the birth-weight scenario’s true mean of 2750 g and \(\sigma\) of 588 g), compute the 95% CI for each, and color the interval rose if it excludes the status-quo value (2500 g) — a successful study — or gray if it still covers it.

Starting point

You want to see what 50 different studies’ CIs would look like at each of five sample sizes — so you need a way to (1) draw a simulated sample of size \(n\), (2) compute its 95% CI, and (3) repeat that 50 times for each of several \(n\)’s.

What the code does

The chunk has three parts:

  1. Defines a small helper function sim_one_ci() that draws one simulated sample of size \(n\) from the birth-weight population and returns a one-row tibble with the sample mean and the 95% CI bounds.
  2. Builds ci_demo by running sim_one_ci() 250 times — 50 reps × 5 sample sizes — and tagging each row with whether its CI excludes the status-quo value (2500 g).
  3. Plots the 250 intervals as horizontal error bars, faceted by sample size, colored by whether they exclude the status quo.

Key functions

Function What it does
function(n) { … } Defines a custom function that takes one argument (n) and returns whatever the last expression evaluates to (here, a tibble).
qt(p = 0.975, df = n - 1) Returns the \(t^\star\) critical value for a 95% CI at \(df = n - 1\) — the how-many-SEs multiplier from earlier.
expand_grid(n = c(…), rep = 1:50) Creates every combination of the values supplied — here, 5 × 50 = 250 rows.
map(n, sim_one_ci) Applies sim_one_ci() to each value of n and returns a list-column of one-row tibbles.
unnest(ci) Flattens the list-column of tibbles into proper data-frame columns.

How to read output

A 5-panel plot, one row per sample size. Each panel shows 50 horizontal intervals (one per simulated study) — rose if the interval excludes 2500 g, gray if it covers it. Vertical lines mark the status-quo value (2500, purple) and the true mean (2750, green).

Read the panels top-to-bottom — and count the rose intervals in each. For the birth-weight scenario (a +250 g intervention effect on an outcome with \(\sigma = 588\) g — a moderate effect relative to noise), here’s what you should see:

  • At \(n = 10\), only a handful of intervals are rose. The CIs are so wide that nearly all of them cover both the status-quo value (2500 g) and the true mean (2750 g) — at this sample size you cannot credibly distinguish a +250 g intervention from “no effect at all.”
  • At \(n = 25\), roughly half the intervals are rose. CIs are tighter, but whether your study would clear the status-quo value is nearly a coin flip — well below the standard 80%-power target.
  • At \(n = 50\), most intervals are rose. A clear majority of studies succeed, but a meaningful minority still fail even though the +250 g effect is real. Right around the 80%-power threshold.
  • At \(n = 100\), nearly every interval is rose, though some lower bounds sit close to the status-quo value. This is the \(n\) a careful researcher would aim for: high power, with cushion.
  • At \(n = 500\), every interval is rose, with comfortable margin. Powerful, but expensive — well past the point of diminishing returns for an effect of this size.

The two paths converge. Recall the precision calculation we did a few subsections back: for this same birth-weight scenario, hitting a target half-width of \(E = 125\) g required \(n \approx 86\). That falls right between the \(n = 50\) and \(n = 100\) panels — exactly where the picture starts to look like a defensible design.

That’s the punchline that motivates this whole section: the \(n\) you need depends critically on the size of your effect relative to outcome noise. A 250 g shift on a 588 g-SD outcome is moderate in those terms. Try editing the simulation chunk above — change mean = 2750 or sd = 588 — to see how the picture shifts when you make the effect bigger or the outcome less variable.

Power is just the share of rose intervals in a panel — formalized below by replacing “50 hypothetical studies” with “1,000 hypothetical studies” inside a function.

A simulation function

Here’s a single R function that does the whole power simulation.

Four arguments control the simulation:

  • n — your sample size
  • true_mean — the population mean under your intervention.
  • pop_sd — the population SD of your outcome variable
  • status_quo — the value the CI needs to exclude (the status-quo population mean, a clinical cutoff, or 0 for a change score)

Defaults: 95% confidence, 1,000 simulations per call.

Click Run Code to set up the function, then you can call it with custom arguments to see how power changes.

Starting point

You want a function that takes a study design (sample size, true mean, \(\sigma\), status-quo value) and returns its power — the probability that a 95% CI from such a study would exclude the status-quo value. Running this chunk doesn’t return a number yet; it just defines the function so you can call it.

What the code does

Inside function(…) {…}, the body does three things:

  1. Repeats a block of code n_sims times (default 1,000). Each repetition simulates one hypothetical study: draws an n-observation sample from the true population, computes the 95% CI for the mean, and returns TRUE if that CI excludes status_quo or FALSE if it covers it.
  2. Stores the 1,000 TRUE/FALSE outcomes in excludes_status_quo.
  3. Returns the proportion of TRUEs — that’s the power.

Key arguments and functions

Function or argument What it does
function(n, true_mean, pop_sd, status_quo, conf_level = 0.95, n_sims = 1000) Defines six arguments. The last two have defaults, so you don’t have to supply them every call.
replicate(n_sims, { … }) Runs the code in { ... } repeatedly (n_sims times) and stacks the results into a vector. Here, the result of each rep is a single TRUE/FALSE.
qt(1 - (1 - conf_level) / 2, df = n - 1) Computes \(t^\star\) for the chosen confidence level. For conf_level = 0.95 this evaluates to qt(0.975, df = n - 1).
mean(excludes_status_quo) Because TRUE = 1 and FALSE = 0, the mean of a TRUE/FALSE vector is the proportion of TRUEs — the power.

How to read output

Running this chunk doesn’t print anything; it just makes simulate_power_mean() available for the rest of the session. The next chunk will call it.

Sweep \(n\) and plot the power curve

Each call to simulate_power_mean() returns one number — the power at a given sample size, i.e., the proportion of simulated CIs that exclude the status-quo value. A common design target is 0.80 (80% power); the natural design question is what \(n\) first hits that target? So we’ll call the function across a grid of sample sizes and plot the curve.

Starting point

You have simulate_power_mean() from the previous chunk. You want to call it once per sample size in a grid and store the results in a single tibble — so you can plot \(n\) vs. power.

What the code does

  1. Stores the sample sizes you want to try in n_grid.
  2. Builds a tibble with one row per sample size.
  3. For each row, calls simulate_power_mean() with that row’s n (the true mean, \(\sigma\), and status-quo value are held fixed at the birth-weight scenario).
  4. Stores each row’s resulting power as a new column called power.

Key functions

Function What it does
tibble(n = n_grid) Builds a one-column tibble with one row per element of n_grid.
rowwise() Tells mutate() to evaluate the next expression one row at a time. Without it, mutate(power = simulate_power_mean(n = n, ...)) would try to pass the whole n column as a vector — and the function only handles one n at a time.
ungroup() Removes the row-wise grouping after the mutate() so subsequent operations behave normally.

How to read output

A two-column tibble: n (the sample size) and power (the proportion of 1,000 simulated CIs that excluded the status-quo value at that n). Power should rise as n grows.

Reading the power curve

  • Power starts low at small \(n\) — which makes sense: at \(n = 10\) or so, your CI is so wide it almost always covers both the true mean and the status-quo value. You can’t distinguish them.
  • Power climbs rapidly through the middle range.
  • Power plateaus for very large \(n\) — past some point, you’d be spending money for certainty you already had.

The sweet spot — where you first cross 0.80 — is the minimum design-defensible \(n\) for the effect size you’re targeting. Everything larger is insurance; everything smaller is underpowered.

Whole-class moment

Bring back the \(n\) you parked in your head from the precision section. Is either number — your precision-target \(n\) or your power-target \(n\) — realistic for a study you could actually run?

If not, you have four levers, in rough order of how realistically researchers can pull them:

  1. Sample size (\(n\)). Tunable, but expensive — and quadratic (\(4\times\) the \(n\) buys \(2\times\) the precision).
  2. Measurement precision (\(\sigma\)). A more reliable instrument, controlled covariates, a within-subject design — anything that shrinks the noise relative to the signal.
  3. Effect size you’re targeting. Design a more potent intervention, or pick a more responsive sample. You can’t will a bigger effect, but you can sometimes design for one.
  4. Confidence level. Dropping from 95% to 90% buys a small precision boost — cheap, but at the cost of a less stringent claim.

Which lever has the most room to give in your design? Discuss with your two-person partner.

Apply it to your own scenario

You started today by recording five things on a card: an outcome, an intervention, a plausible \(\mu\) for participants in your intervention group, a plausible \(\sigma\), and a feasible \(n\). Now spend a few minutes plugging your numbers into the two self-check exercises below.

Activity 1 — How big does your sample need to be for the precision you want?

First, decide on a target half-width \(E\) — the smallest change in your reported intervention-group mean that would change your audience’s action, divided by two. Then plug your \(\sigma\) and \(E\) into the chunk below to get the \(n\) you’d need.

Is the resulting \(n\) realistic for a study you could actually run?

Activity 2 — How much power would you have at your feasible \(n\)?

Now plug your scenario’s expected mean, \(\sigma\), status-quo value, and a feasible \(n\) into the simulation function. The output is the share of hypothetical studies of that size that would correctly exclude your status-quo value.

Heads up: simulate_power_mean() was defined in the earlier “A simulation function” chunk. If you haven’t already, scroll up and click Run Code on that chunk first — otherwise the chunk below will throw a could not find function "simulate_power_mean" error.

Is your power at or above 0.80? If not, scroll back up to the four levers — which one has the most room to give in your design? Note your answer and bring it to Wednesday’s lab, where we’ll have time to push your design further.

Wrapping up

What we built today

Four things to carry into M08

  • CI half-width shrinks as \(1/\sqrt{n}\). Quadruple \(n\), halve the CI. That’s the precision story.
  • Power is the decision-side cousin of precision. It’s the probability that your 95% CI will exclude a specific status-quo value, given a real effect. It emerges from the CI machinery you already know — no new math, just a simulation.
  • Power depends on three dials: \(n\), true effect size, and \(\sigma\). Confidence level is a fourth (reversed) dial but rarely the one you tune.
  • Your \(n\) decision should come from the power curve for your effect size, not from a convention. “We recruited 60 per group because that’s what they used in the prior study” is not a defense of a design if the prior study was also underpowered.

What’s next

  • Wednesday’s lab extends confidence intervals to a new statistic — a proportion — using the NSDUH adolescent depression data. You’ll replicate the trend graph from Daly (2022) with 95% CI error bars added. The machinery is a minor extension of the t-interval formula.
  • Next week (M08 — The Logic of NHST) gives a name to what we’ve been calling the status-quo value. Statisticians call it the null, and the question “does my 95% CI exclude the status quo?” becomes the \(p < .05\) decision in Null Hypothesis Significance Testing. M08 makes that equivalence explicit, gives you the \(p\)-value vocabulary, and connects it back to power.

Before you leave

In one sentence: for the mean you wrote down at the start of class, how large a sample would you need to achieve 0.80 power for credibly beating the status quo, assuming a meaningful effect size you care about?

If you don’t know your effect size — that’s homework to think about before M08.

Footnotes

  1. In statistics, an asymptotic result is one that becomes approximately true as \(n\) approaches infinity. For example, an estimator may be asymptotically normal, meaning its sampling distribution becomes closer to a normal distribution as the sample size gets larger.↩︎