From Sample to Population

Lecture · Module 5 · Mon Sep 14

How today works

This page is your guide for today’s class. Some moments are yours to explore on screen; others ask you to look up, make a prediction, or talk something through with a neighbor.

Whole-class moment

When you see this box, stop working on your screen and come back to the room. We’ll work through the idea together.

Today we make one of the biggest shifts in the course: from describing the people in our dataset to learning about people who are not in it.

Up to now, most of your numbers have described the rows in front of you. If 42% of the people in your file had some characteristic, then 42% of those rows had it. That statement is exact.

Starting today, we ask that same 42% to do a second job: help us learn about a larger population we did not fully observe. The statistic is still a fact about the sample; when we use it to learn about a population, it becomes an estimate.

The arithmetic barely changes. The meaning does. The new question is: what does this number let me say beyond the rows I happened to observe?

We will build a population together

Unlike the datasets you have used so far, today’s data do not come from a file. In a few minutes, we are going to build a fictional population together. That is deliberate: if we create the population, we get to know things a real researcher almost never knows, and we can study how sampling processes succeed or fail against a known truth.

The study we’ll follow today

Imagine that a doctoral student in the Clinical Psychology program here at CSU is designing a smartphone app to reduce anxiety in young adults in Fort Collins, Colorado. Before she builds the app, she wants to understand the population she hopes to serve.

Her first question is simple:

What proportion of 18–24-year-olds in Fort Collins are experiencing moderate-to-severe anxiety, defined as a GAD-7 score of 10 or higher?

Her study, in four lines

Who it’s for	Young adults aged 18–24 living in Fort Collins, Colorado
How many that is	About 36,701 people, using 171,500 residents × 21.4% ages 18–24
What she wants to know	The proportion with moderate-to-severe anxiety — GAD-7 score of 10 or higher
Who she can realistically reach	The PSY 100 participant pool at CSU — roughly 2,200 students across two semesters, all of them enrolled in college

That last row is not evidence of bad planning. It is a very normal research problem: the group you care about and the group you can realistically recruit are often not the same group.

It also has a name. Recruiting whoever is reachable — a subject pool, a clinic’s caseload, a class, whoever answers the flyer — is convenience sampling, and it is not an unfortunate exception in psychology. It is the norm. A large share of the studies you will read this semester, and most of the ones you will run, draw their samples this way.

So today is not a lecture about a doctoral student who planned badly. It is a lecture about the design most psychological research actually has, and about learning to say precisely what such a design can and cannot support.

She has now done something important. She has named both her target population and the quantity she wants to learn about it. That population quantity is her estimand.

Naming the estimand does not tell her its value. It gives her a target against which every later decision — whom to recruit, how many people to recruit, and what to claim — can be judged.

From population to sample

Before we judge her study, we need four words that the rest of the course will use constantly.

Four words of statistical inference

Population — every unit you want your conclusion to cover. Here: the 36,701 Fort Collins residents aged 18–24.
Parameter — a number describing that population. We often write p for a population proportion and μ for a population mean. It is usually fixed but unknown.
Sample — the units you actually observe.
Statistic — the corresponding number computed from the sample. We write \(\hat{p}\) (“p-hat”) for a sample proportion and \(\bar{x}\) (“x-bar”) for a sample mean.

Here, the estimand is the population proportion \(p\) who would screen positive for moderate-to-severe anxiety.

\[ \underbrace{p}_{\text{parameter · unknown}} \;\longrightarrow\; \text{sampling} \;\longrightarrow\; \underbrace{\hat{p}}_{\text{statistic · observed}} \]

Nature runs that arrow left to right — from population to sample to statistic. That is one piece of the data-generating process.

Statistical inference asks us to reason in the other direction: from the one \(\hat p\) we observed toward the unknown \(p\) we want to learn about.

Today, we get to cheat

A real researcher cannot inspect every 18–24-year-old in Fort Collins and look up \(p\). Today we are going to do something she cannot: we will build the population ourselves.

This is our first use of simulation. A simulation creates a simplified world by specifying some rules, letting a random process generate data from those rules, and then seeing what happens.

Why bother? Because if we create the world, we know the truth inside it. We can then watch different study designs succeed or fail against an answer that is normally hidden.

The rules of our toy world

Two pieces are anchored to the real setting:

Fort Collins has roughly 171,500 residents, and about 21.4% are ages 18–24.
The GAD-7 is a real anxiety screener; a score of 10 or higher is commonly used as the threshold for moderate-to-severe anxiety.

Everything else below is a teaching assumption, not a claim about the actual young adults of Fort Collins:

60% of our simulated young adults are enrolled in college.
30% of enrolled young adults have a chance of screening positive for moderate-to-severe anxiety.
40% of non-enrolled young adults have a chance of screening positive.
The PSY 100 participant pool contains 2,200 students, all from the enrolled group.

We chose these values because they create two sizable groups that differ enough to make today’s design problems visible.

Here are the assumptions in R. These numbers are the important part of the code:

Now we turn those rules into 36,701 fictional people. You do not need to learn this code today — read what it is doing.

We created one row per fictional young adult. About 60% are enrolled, and each person gets a random yes/no anxiety outcome using the probability assigned to their group.

Because the process is random, the realized rates will not be exactly 30% and 40%. Once this population has been generated, however, whatever proportion it contains is now the fixed truth inside our toy world.

Finally, we create the group our researcher can actually reach:

The most important line is not really code at all. It is a design fact:

Everyone in psy100 is enrolled. A non-enrolled young adult has no route into this sampling frame.

Now look behind the curtain:

In our toy world, anxiety is about 30.1% among enrolled young adults and 39.8% among non-enrolled young adults.

And because we own the whole fictional population, we can do the one thing a real researcher cannot:

The parameter is \(p\) = 34.0%. That is the number our researcher wants to learn about — and, in a real study, the number she would never get to see directly.

You are her committee

Your concerns will usually fall into three different buckets. They sound similar at first, but they need different solutions.

Three questions to ask about any estimate

Lens 1 · Sampling variability. How much could the number change simply because a different set of people happened to be sampled?
Lens 2 · Selection bias. Could the process that gets people into the study systematically push the estimate away from the quantity we want?
Lens 3 · Generalizability. Does the sampling frame support the population claim we want to make, for the particular quantity we are estimating?

The rest of class is one question asked three times: would more people solve it?

Lens 1 · Different samples contain different people

For this first lens, set every recruitment problem aside. Pretend that the PSY 100 pool itself is the population we want to describe and that we can take a clean random sample from it.

The true anxiety rate in that pool is 29.1%.

Whole-class activity · 25 people, many answers

Run this once and write your answer down:

We’ll collect several answers around the room.

Same population. Same sample size. Same procedure. Why aren’t our answers identical?

Different random samples contain different people. That alone is enough to make the statistic change.

No lab made a mistake. No one used a bad measure. The variation came from sampling itself.

Now change just one thing:

Collect a few answers again.

What changed? The estimates are less scattered when each sample contains 250 people than when it contains 25.

Lens 1 takeaway

Even with a clean random sampling process, different samples produce different statistics by chance. That is sampling variability.

A larger sample generally reduces that variability. It does not make randomness disappear; it makes the statistic more stable from sample to sample.

If we could collect every possible sample and stack all of their \(\hat p\) values together, that collection would have a name: the sampling distribution.

That is all you need from the sampling distribution today. M07 is where we slow down, study that object carefully, and learn how its variability becomes something we can quantify from one sample.

So for Lens 1, the committee’s answer is:

Would 250 help? Yes. If the problem is ordinary sampling variability, more information helps.

Lens 2 · Who chooses to enter?

Now bring the real recruitment process back.

Our doctoral student posts her study to the PSY 100 pool under the title:

A study of anxiety in college students

A student who has been struggling with anxiety may read that title differently from one who has not. The study feels more relevant to them, so they may be more likely to sign up.

Nothing dishonest has happened. People are simply not entering the study with equal probabilities.

Before you run anything, make a prediction:

If anxious students are more likely to volunteer, will the anxiety rate among volunteers be above, below, or about the same as the pool’s true rate of 29.1%?

Commit first. Then run it.

The volunteer group should land well above the pool’s actual anxiety rate. The problem is not the number of volunteers — it is which students they are. The probability of entering the sample is related to the outcome we are trying to estimate.

Whole-class moment · make participation lower but fairer

Change both probabilities to the same value:

p_signup_anxious    <- 0.45
p_signup_notanxious <- 0.45

Run the chunk again.

Notice what you did not do: you did not increase participation. Anxious students are now less likely to sign up than before.

What you removed was differential participation related to anxiety. The volunteer rate moves back toward the pool rate.

That gives us an important distinction:

A low response rate does not automatically imply serious bias, and a high response rate does not guarantee its absence.

The deeper question is: could the process that puts someone in or out of my sample be related to the quantity I am trying to estimate?

A larger volunteer sample can make the estimate more stable, but it does not make the recruitment process less selective.

Lens 2 takeaway

Lens 1 asks whether the estimate bounces around by chance.
Lens 2 asks whether the recruitment process systematically points the estimate away from its target.

More observations from the same selective process can give you greater precision without removing the selection problem.

Lens 3 · A good sample from the wrong frame

Now suppose she solves Lens 2 completely. No differential volunteering. She takes a beautiful random sample from the PSY 100 pool.

There is still one problem: the PSY 100 pool is not her target population.

The sampling frame

Her target population contains 36,701 young adults:

22,021 are enrolled in college.
14,680 are not.

Her sampling frame — the people who could possibly be selected through PSY 100 — contains only enrolled students.

So about 40% of her target population has no route into the study at all. Recruiting more people from PSY 100 does not change that.

For her original prevalence question, we can see exactly what that costs in our toy world:

The whole target population is about 34.0% anxious; the pool is about 29.1%. In this particular toy world, the difference is about 4.9 percentage points.

That is a real mismatch — but now comes the more important lesson.

The same frame can matter differently for a different estimand

Suppose she now finishes the app and runs a randomized trial. To make the generalizability problem visible, add one more teaching assumption:

Among young adults in college, the app lowers GAD-7 scores by 4 points on average.
Among young adults not in college, it lowers scores by 1 point on average.

We are not claiming those are real Fort Collins effects. We are deliberately creating effect modification in the toy world.

Note

A word that now does two jobs. From here on there are two ways a person can be “enrolled” — enrolled in college, and enrolled in her trial. They are unrelated, and the split that matters for the app’s effect is the college one. To keep them apart, this section says in college and not in college for the population split, and reserves trial language for who takes part in the study.

The average effect she wants for all Fort Collins young adults is:

\[ (.60)(-4) + (.40)(-1) = -2.8 \]

But a trial drawn entirely from the PSY 100 pool is aimed at an effect of about −4 points.

Whole-class moment · same frame, different consequence

Compare the two questions:

Question	Target-population quantity	What the PSY 100 frame is aimed at
How common is anxiety?	34.0%	29.1%
How well does the app work?	-2.8 GAD-7 points	-4.0 GAD-7 points

Why does the same sampling frame create a modest difference for one estimand and a much larger difference for the other?

Turn the dial · who is in the trial

Lenses 1 and 2 each gave you something to turn. Here is Lens 3’s. One dial: what fraction of her 200 trial participants comes from the PSY 100 pool.

Whole-class moment · slide from 100 to 0

Run it at 100, then 75, 50, 25, 0. The dashed line walks steadily from −4 toward −1, crossing the green line somewhere in the middle.

Every one of those trials is internally valid. Random assignment centers each on the average treatment effect for the kind of participants in that trial — the estimates are precise and unbiased for the people who were in the room. What changes is who those people are, and therefore which question the trial answered.

At 100% pool, she has a clean estimate of the effect among college students. It is simply not the number she promised.

The answer is not that prevalence is always forgiving or that treatment effects are always fragile. It is more general:

The consequence of a sampling-frame mismatch depends on the estimand and on how the people missing from the frame differ on the quantity that matters for that estimand.

For a population prevalence or mean, a mismatch matters when the groups differ on the outcome. For a population treatment effect, a mismatch matters when the groups differ in how they respond to treatment — what we later call effect modification.

“Is my sample representative?” is not quite the right question

A sample does not have to mirror the population on every characteristic.

Ask instead:

Representative for which estimand, with respect to which characteristics?

That is why “Is this a good sample?” has no useful answer until you finish the sentence: good enough for what population claim?

The two validities · finally, the words

That contrast has standard names, and this is the moment they become useful rather than jargon.

Internal and external validity

Internal validity asks: within this study, is the comparison sound? Did the design support the causal claim for the people who were actually in it? Random assignment is the main tool, and our researcher’s trial has it — she randomized, so the comparison between her arms is clean.

External validity asks: does the finding travel? Do the results apply beyond the particular people, setting, and moment studied? Probability sampling and a well-matched sampling frame are the tools, and this is where her trial struggles.

Her trial is internally valid and externally limited, and those are separate verdicts. Random assignment does not buy external validity, and a representative sample does not buy internal validity. Each is earned by a different design decision.

That asymmetry is worth sitting with, because it is the usual situation. Well-run randomized experiments on convenience samples are the bread and butter of experimental psychology: strong internal validity, external validity that has to be argued for rather than assumed.

So when is the subject pool fine?

Most of you will use a participant pool — this one or one like it. That is not a confession; it is how the field works. The useful question is not whether a convenience sample is acceptable in general, but what you are trying to estimate with it.

Three things you might use the pool for

What you are using it for	Does the frame bite?
Estimating a prevalence or mean for a population	Yes, if pool members differ from the population on the outcome. Our researcher’s 4.9-point gap is this case.
Estimating a treatment effect for a population	Yes, if pool members differ in how they respond to the treatment — effect modification. This is the case the dial just showed you, and it was the larger error.
Testing whether an effect exists at all	Often defensible. If the claim is “this manipulation can move this outcome in people like these,” a convenience sample can support it — because the claim is about existence, not about a population’s average.

That third row is the honest defense of convenience sampling, and it is worth stating plainly because you will use it: “I am not estimating a population quantity — I am testing whether a process operates.”

It is a real argument, and it is also the one most easily abused. It holds when your claim genuinely stays at the level of existence. It stops holding the moment you write “these findings suggest that college students in general…” or attach a population-sounding number to the effect. The discipline is not avoiding convenience samples; it is keeping the claim the same size as the design.

So for Lens 3, the committee’s answer is again:

Would 250 help? Not by itself. More PSY 100 students are still PSY 100 students.

Apply the three questions to your own research

Whole-class activity · pressure-test a study you might actually run

Think about a thesis, pilot, or other study you could realistically conduct. Fill in four lines:

Question	Your answer
1. What population do you ultimately want to say something about?
2. What population quantity do you want to estimate?
3. Who could realistically enter your sample — and who could not?
4. Which concern is biggest here: sampling variability, selective participation, or mismatch with the target population? Why?

Take two minutes on your own, then compare with a neighbor. We’ll hear two or three examples from the room.

The goal is not to design a perfect study. Most real studies involve constraints. The goal is to know which problem your design has, because different problems need different responses.

So what should she actually do?

We have spent the hour pressure-testing her study. She is still a doctoral student with one semester, no budget, and a subject pool. So what is the most useful advice we can give her?

Four things she can actually do

A citywide probability sample would be wonderful and is not going to happen. Everything below is something she could do this semester.

Narrow the claim to match the frame. If she can only recruit from the pool, she can make the target population correspondingly narrower. The study becomes less ambitious and the claim becomes defensible. This is free, and it is the one people skip.
Add even a modest bridge beyond the frame. A smaller sample of non-college 18–24-year-olds — through community organizations, clinics, workplaces, libraries — will be imprecise on its own, but it gives her direct evidence about whether the omitted group looks different. A rough answer to “does this generalize?” beats an assumption.
Make recruitment less outcome-linked. “A study of daily experiences in college students” rather than “a study of anxiety” weakens the tie between the outcome and the decision to participate. Not a cure for Lens 2, but a real improvement at no cost.
Say plainly what remains uncertain. Name the frame, name who had no route in, and say which direction the bias likely runs only where theory or evidence supports a direction. In real research — unlike today — she cannot look up the answer.

Three of those four are free. That is the encouraging part: most of the distance between a study that overclaims and one that doesn’t is covered by clear thinking about the target, the frame, and the wording — before a single row of data exists.

Your committee verdict

If the problem is…	Would a larger sample help?
Sampling variability	Yes. More information generally makes the statistic less variable from sample to sample.
Selective participation	Not by itself. More people entering through the same selective process do not remove the selection mechanism.
Sampling-frame / generalizability mismatch	Not by itself. More people from the same frame do not add the people the frame cannot reach.

The useful habit is not “get a bigger sample.” It is:

First diagnose the problem. Then choose the remedy.

The doorway to Part 2

Is the statistic we observe plausibly aimed at the population quantity we care about?

Once we are willing to treat the sampling process as defensible, a different problem remains even in a beautifully designed study:

How much could my statistic have changed simply because I happened to observe these people rather than another sample?

That is where Part 2 begins.

Three sentences to carry forward

A statistic describes the sample you observed; inference uses it to learn about a population parameter you usually cannot see.
Larger samples reduce ordinary sampling variability, but they do not automatically repair selection or coverage problems.
Whether a sample is adequate depends on the estimand — the population quantity you are trying to learn.

Next, M06 gives us the language of probability for describing what could happen under repeated random processes. Then M07 slows down and studies the sampling distribution — the object that tells us how a statistic behaves across repeated samples and ultimately lets us quantify sampling uncertainty from the one sample we actually have.

That is the handoff: today, understand the inferential problem; next, build the machinery for solving it.

Wednesday’s lab turns the camera onto a real survey that spends substantial resources addressing the population-sampling problem: the Pew American Trends Panel. You will procure the data yourself and build a Table 1 with gtsummary. The contrast with today’s PSY 100 frame is deliberate.