From Probability to Inference

Lecture · Module 6 · Mon Sep 21

Where we are: from mechanics to inference

Today’s throughline

Friday’s pre-study handed you the mechanics of probability: the complement, addition, and multiplication rules, conditional probability from a cross-tab, and two named distributions — the binomial and the normal.

Today those mechanics get a job. Both activities build the same object:

a distribution of what could have happened

and then use it to answer the two questions the rest of this course is built on.

Activity	The question	The object we build
1 · Sampling	How much does an estimate move from sample to sample?	The sampling distribution
2 · Detective	Is this result more than chance would produce?	A reference distribution

Same machinery, two jobs. By the end of Activity 2 you will have computed a p-value with your own data — before we give it that name.

Two notes on how today runs. Activity 1 is on your screen: you will run code, and the numbers you get are yours. Activity 2 is in the room: you will be the data.

Activity 1: Sampling from the fake news population

Here is the core research problem, in one sentence: you see one sample, but you want to learn about the population. The pre-study’s dbinom() and pbinom() are about to earn their keep. We’ll build that move step by step — one draw, then 10 draws, then the full binomial distribution those counts come from.

Setting up the population

fake_news · 150 articles · treated as the full population for today

We’ll pretend the 150-article fake_news collection is our full population. That lets us peek behind the curtain in a way actual research never can — we actually know the true population parameter, because we can just count: 60 of the 150 articles are Fake, so \(p = 60/150 = 0.40\). Every sample you draw in the next three steps will come from this 150-article pile, and we’ll check your results against that known truth of \(p = 0.40\).

(In actual research, of course, you never get to see the full population. Today’s pretend-population setup is a teaching trick — it lets you see exactly what sampling does before we ask you to reason in the dark.)

Step 1 — Draw one article

Imagine reaching into the 150-article pile, pulling one article, recording Fake vs Real, then putting it back before the next draw. Replacing after each draw keeps the probability fixed at \(p = 0.40\), which matches the binomial model we use below. The chunk does one such draw — slice_sample() grabs one random row from the dataset.

Everyone in the room will run this chunk once. I’ll ask for a show of hands: who pulled a Fake? If the population really is 40% Fake, roughly four out of every ten hands should go up.

One draw is almost uninformative on its own — you either got a Fake or you didn’t. But across the full classroom, the pattern of Fakes and Reals starts to reveal the population’s 40/60 split.

Step 2 — Draw a sample of 10

Actual research doesn’t rely on one observation. Now each of you runs a tiny study: pull a sample of 10 articles at random from the 150-article pile, and count how many are Fake.

Share your numbers with the class. You’ll see n_fake values like 4, 5, maybe 2, maybe 8 — all from the same 150-article population with the same underlying \(p = 0.40\). The p_fake column is just n_fake / 10.

Class discussion (4 min)

List the p_fake values the class shouted out. How spread out are they? If two of you reported your own proportion as the estimate of the population parameter, how far apart would your claims be? That spread is sampling variability — the signature of doing research with finite samples. It never goes away entirely; the job of inference is to reason carefully in spite of it.

Quick sidebar — three tools, three jobs

Steps 1 and 2 used slice_sample() to draw rows from the actual 150-article dataset. In Step 3 we’ll use rbinom() to simulate 1,000 studies. In Step 4 we’ll use dbinom() to compute the exact probabilities. You met dbinom() in Friday’s pre-study; today it earns its keep, so the distinction is worth pinning down:

Tool	What it does	When to use it
slice_sample()	Randomly selects rows from an existing data frame	You already have data in hand and want a random subset of it
rbinom()	Simulates random draws from the binomial model	Imagining what the distribution of outcomes would look like if you ran the experiment many times
dbinom()	Computes the exact probability of each outcome	When you want the math, not a simulation

Rule of thumb: slice_sample() and rbinom() both hand you samples; dbinom() hands you the exact probabilities those samples are approximating.

Step 3 — Simulate 1,000 studies with rbinom()

We just ran the 10-article study roughly 10-15 times (depending on class size). What if we could run it 1,000 times and look at the full pattern? rbinom() does exactly that — it simulates random draws from the binomial model. Each number it returns is one complete 10-article study.

Read the shape. The bars pile up around 4 — that’s \(n \cdot p = 10 \cdot 0.4\) — because 4 is the most likely count in this setup. Counts of 3 and 5 are almost as common; 0 and 10 appear, but rarely.

This thing has a name

What you are looking at is a sampling distribution: the distribution of a statistic across hypothetical repetitions of the same sampling process.

Follow the chain, because the whole rest of the course runs along it:

Population parameter	\(p = 0.40\) — one fixed number, and in real research unknown
One study	gives you one statistic, $ = $ n_fake / 10
Repeated studies	give you a distribution of \(\hat{p}\) — the thing plotted above

Each of your classmates in Step 2 produced one \(\hat{p}\). The 1,000 bars above are what happens when you keep going. The parameter never moved; the statistic did.

Two different kinds of variation are on this screen, and it is worth separating them.

Run the chunk a second time. Two things could have changed:

• The spread among the 1,000 values — some studies got 2 Fakes, some got 7. That is sampling variability: real, irreducible, and the thing inference exists to reason about. It would still be there if you simulated a million studies.
• The fact that the histogram itself shifts slightly on a re-run. That is Monte Carlo variability — an artifact of having simulated 1,000 studies rather than infinitely many. Simulate more and it shrinks toward nothing.

Only the first one is a fact about research. The second is a fact about our simulation, and Step 4 removes it entirely by computing the exact answer instead of simulating it.

Step 4 — The exact target: dbinom()

rbinom() drew a histogram by simulating. dbinom() draws the same shape by computing — it returns the exact probability of each possible count under the binomial model (independent draws with constant \(p = 0.40\)). If you re-ran Step 3 with 10,000 studies, then 100,000, the histogram would settle down onto exactly this shape.

Same shape as Step 3 — but now the bar heights are exact probabilities, not jiggly counts. rbinom() draws samples from this distribution; dbinom() gives the exact probability attached to each possible value of it. This is the landscape every one of your individual samples came from.

Key take-homes — from one sample to inference

1. Every actual research sample is ONE DRAW from a distribution like this. Your count of fakes was one dot on this landscape. If you ran your study again, you’d get a different dot from the same landscape.
2. Across infinitely many such studies, outcomes cluster around the expected value (\(n \cdot p\)) — but no single study is guaranteed to land there. This is the irreducible variability that inference is built to reason about.
3. Actual research flips the logic. You don’t know \(p\); you have one sample. The central move of inference is to reason backward from your one observation to the underlying distribution — and therefore to the underlying \(p\). We’ll start formalizing that move soon as we study confidence intervals and keep developing it for the rest of the semester.

If the room is moving quickly — what happens when the study gets bigger?

Everything so far used \(n = 10\). Run this to put \(n = 10\) and \(n = 100\) side by side, both drawn from the same \(p = 0.40\) population:

Three questions for the room, in order:

1. Did \(p\) change? No — same population both times.
2. Did the center move? No — both pile up on 0.40.
3. So what changed? The spread.

That spread is the whole story. A bigger study does not move the truth and does not move the center — it narrows the range of answers you are likely to get. The spread of a sampling distribution has its own name: the standard error, and pinning it down is what M07 does first.

---

Activity 2: The fake news detective

Same machinery, different reason — read this before you start

Activity 1 built a sampling distribution by asking: if \(p = 0.40\) — which we happen to know, because we can count — what would repeated studies produce?

Activity 2 is about to build the same kind of object for a different reason. We will assume a value we do not know to be true, ask what repeated studies would produce if it were, and then check whether the result we actually got looks at home there.

A null distribution is not a new kind of object. It is a sampling distribution built under an assumed null hypothesis.

And watch the number change. In Activity 1, \(p = 0.40\). In Activity 2 it will be \(p = 0.50\) — and the population did not change. What changed is the event we are counting: Activity 1 counted “this article is Fake” (40% of the pile). Activity 2 counts “this guess is correct”, which for someone flipping a fair coin happens half the time.

That is worth more than it looks. A probability only means something once you have said what event it is about. The symbol \(p\) does not name a fixed quantity in the world; it names whatever event you defined.

You’re about to play a guessing game. I’ll show you 10 article titles — some Fake, some Real — and your job is to classify each one.

Here’s the harder question, the one that comes right after the game: if you end up scoring 7 correct out of 10, is that impressive? Could a person who’s literally flipping a coin score 7 out of 10 just by luck? To answer that, you need to know what “just luck” actually produces.

That’s the core move of statistical inference in one sentence: figure out what chance alone would produce, then compare your actual result to that reference. Today we’ll do that in reverse order: commit to guesses first, build the coin-flip benchmark, then score your performance against it. Holding onto an unknown score for a few minutes is part of the point.

• If your detective score lands somewhere pennies could easily produce → your “detection ability” is indistinguishable from guessing.
• If your detective score lands in a region pennies almost never reach → you’re picking up on genuine signal in the titles.

I’ll give everyone a penny — we’ll need it for Step 2.

Step 1 — The detective experiment

Read each title carefully — use whatever cues you notice (writing style, punctuation, topic, gut feel) — and decide whether it’s Fake or Real. Don’t worry about scoring yet — we’ll reveal the answers after we’ve built the tool for judging your performance.

The detective experiment (10 min)

Instructions:

1. I will display 10 actual article titles, one at a time.
2. For each one, tick either the Real box or the Fake box in the tally below to record your guess.
3. No discussion during the guessing phase.
4. After all 10 titles are displayed, we will not reveal the answers yet — we first need to build the benchmark for judging performance. Hold onto your tally.

Detective tally

Real Fake Title 1 Correct Real Fake Title 2 Correct Real Fake Title 3 Correct Real Fake Title 4 Correct Real Fake Title 5 Correct Real Fake Title 6 Correct Real Fake Title 7 Correct Real Fake Title 8 Correct Real Fake Title 9 Correct Real Fake Title 10 Correct

Step 2 — Null simulation: what pure coin-flipping produces

Before we can tell whether your detective score is good, we need a benchmark. What does the score distribution look like for someone whose cues don’t work — a pure coin-flipper? Let’s find out by simulating it in the room.

Penny-flip simulation (5 min)

Use the checklist below. For each of the ten title slots, I will pause. During each pause:

1. Flip your penny.
2. Tick the H box (Heads) or the T box (Tails) for that title.
3. The rule we’re using: Heads → F (Fake), Tails → R (Real).

No actual article content is shown — this is pure coin-flipping. You’re simulating what a detective whose cues don’t work would produce.

Once everyone has flipped for all ten slots, I will reveal the secret truth for each title (Fake or Real). For each row:

• If your flip matched the truth (H on a Fake slot, or T on a Real slot), tick the Correct box.
• If it didn’t, leave Correct unchecked.

Count your checked Correct boxes — that’s your null-simulation score.

The ten title slots

H T Title 1 Correct H T Title 2 Correct H T Title 3 Correct H T Title 4 Correct H T Title 5 Correct H T Title 6 Correct H T Title 7 Correct H T Title 8 Correct H T Title 9 Correct H T Title 10 Correct

Where did your coin-flip score land? (3 min)

Count your checked Correct boxes from Step 2 — that’s your null-simulation score.

By show of hands: how many of you got 0–3 correct? 4–6? 7–10?

This is what pure-chance performance looks like in one room.

Step 3 — The reference distribution: what pure guessing produces

Say exactly what the null model is

Our null here is independent fair guessing: every answer is correct with probability .50, independently of the others.

Notice that this is narrower than “the titles carry no information.” Activity 1 just told you 60% of the pile is Real — so a classifier that ignores every title but always answers “Real” would score about 60%, well above a coin, while using no title information at all. “No signal” and “fair coin” are not the same claim.

The null model is something you specify, not something nature hands you. We choose the fair coin here because it is the benchmark our pennies actually simulated, and because it is the one you can reason about without knowing the Fake/Real split of the ten titles in advance.

The coin-flip scores your class just produced in Step 2 are noisy samples from one specific distribution. If we could plot infinitely many coin-flippers’ scores, they would pile up into exactly this shape — the theoretical distribution under pure random guessing: Binomial(10, 0.5). The probability equals 0.5 here because each guess from a pure coin-flipper is a 50/50 Bernoulli trial. Let’s visualize it:

This is what pure-chance performance looks like – what we’d expect under a pure random guessing scenario. Keep this shape in mind — it’s the reference distribution we’ll use to evaluate your detective performance next.

Step 4 — Reveal the answers and place your detective score on the reference

Time to score your detective tally. I’ll now reveal the Fake/Real answers for all 10 article titles from Step 1. For each title on your Detective tally:

• If your guess matched the truth, tick the Correct box.
• If it didn’t, leave Correct unchecked.

Count your checked Correct boxes — that’s your detective score.

You now have two numbers:

• From Step 2: your null-simulation score — number of checked Correct boxes from pure coin-flipping
• From Step 1: your detective score — number of checked Correct boxes from the actual article titles

Compare your detective score to the Binomial(10, 0.5) distribution from Step 3.

(8 min) Where does your detective score land on the reference distribution?

By show of hands: how many of you got 0–3 correct on the detective experiment? 4–6? 7–8? 9 or 10?

Now look back at the Binomial(10, 0.5) reference plot.

• If your detective score is near the middle (4–6), it looks like what a coin-flipper would produce. Not very surprising — you might as well have been flipping pennies.
• If your detective score is out in the upper tail, chance alone produces it rarely. Here are the three that matter, computed under fair guessing:

Score	\(P(X \geq \text{score})\)
7 or better	0.172
8 or better	0.055
9 or better	0.011

Sit with the awkward one

Eight out of ten feels impressive. Under fair guessing its probability is 0.055 — which does not clear the conventional .05 threshold you’ll meet in a moment.

So: is 0.055 meaningfully different from .049? The arithmetic says one rejects and the other doesn’t. Your intuition probably says they are the same result.

Hold onto that discomfort. It is not a flaw in your reasoning — it is a genuine and much-argued problem with treating a threshold as a verdict, and M08 takes it up directly.

Step 5 — Your score on the null distribution

Let’s make this concrete. The plot below is the same Binomial(10, 0.5) reference from Step 3, but now it highlights the tail that matters for you — every score at least as high as the one you got.

Change my_score to the number you got in Step 4 (your detective score), then run the chunk.

Look at the rose-shaded bars. Their total height — the number in the subtitle — is the probability, computed under the null of pure coin-flipping, of scoring at least as high as you actually did. That one number is the whole question:

If I were guessing independently at 50/50, how often would chance alone produce a score at least as high as mine?

• If that number is large (say, > 0.10), a coin-flipper would routinely produce a score at least this high. We have insufficient evidence to reject the null of random guessing. (This is not the same as concluding the null is true — it means the data don’t rule it out.)
• If that number is small (say, < 0.05), coin-flipping would rarely produce a score this extreme. Under the standard decision rule we reject the null in favor of the alternative: your score is evidence against the fair-guessing model, and in favor of above-chance performance on these ten titles.

Be precise about what you just learned (4 min)

Suppose a classmate scores 9 out of 10, giving \(P(X \geq 9) =\) 0.011. Take thirty seconds with your neighbour and separate two lists.

What the test supports: 9/10 is unusual under independent fair guessing on these particular ten titles.

What it does not establish: that this person has a durable ability to spot fake news; that they would do as well on ten different articles; or that any specific cue — exclamation marks, say — is what did the work.

The arithmetic is about one score on one set of items. How far the claim travels is a question about design and sampling, not about the p-value — and no test result will settle it for you.

Key message — you just computed a p-value

A p-value is the probability — computed under the assumption that the null model is true (here: independent fair guessing) — of observing a result at least as extreme as what you actually observed, in the direction of the alternative (here: scoring above chance). In this plot, the rose-shaded area is that probability.

The decision rule, in one sentence: if the p-value falls below a threshold (conventionally 0.05) that was chosen before examining the data, we reject the null in favor of the alternative.

Note on directionality — this activity shades only the upper tail because our research question is directional (“did you detect genuine signal?”). When a question is symmetric (“is the result different in either direction?”), a two-sided p-value counts every result at least as incompatible with the null in either direction — which for a symmetric distribution amounts to both tails, though that is a consequence rather than the definition. M08 will formalize both cases.

Why it matters: Every statistical test you’ll meet this semester has this shape. Define what “nothing interesting” looks like (the null distribution) → locate your observation → measure how far into the tail it landed → decide whether that tail probability is small enough to reject the null. M08 gives this framework a name and formal notation — Null Hypothesis Significance Testing — but you just did it firsthand.

What was the signal?

Students who scored well probably noticed patterns in the titles. What cues did you use?

Common signals:

• Exclamation marks in the title
• ALL CAPS words
• Emotionally charged language

Let’s check whether exclamation marks really are a signal of fakeness:

Among articles with an exclamation mark: the vast majority are Fake. Among articles without one: most are Real.

What about the other two cues?

Exclamation marks are binary (a title has one or it doesn’t), so a cross-tab was the natural tool. caps_pct and negative are continuous — each title gets a numeric score. To see whether they track fakeness too, we compare their distributions between Fake and Real articles:

Fake titles use ALL CAPS more than Real titles, and they carry a more negative emotional tone. Different measurement, same story.

What this means

Whatever processes produced these articles left observable differences in punctuation, capitalization, and tone. That is the data-generating process showing through: articles of the two types were not produced the same way, and the traces are visible in the data.

Those differences create potentially useful cues — features a reader could exploit to classify better than a coin. What we have not shown is that any particular cue is what you used, or what did the work for anyone who scored well. We know the differences exist; we did not measure which ones drove your answers.

Think about it

Today’s task was “given a title, can I tell Fake from Real?” — a prediction problem. For prediction, a cue does not have to be causal to be useful: exclamation marks earn their keep by reliably co-occurring with fakeness, whether or not they cause anything. Confounding, which would wreck a causal claim, does not disqualify a predictor in the same way.

But prediction has its own hard requirement, and it is not confounding — it is generalization. A cue is only useful if it keeps predicting on new cases. Every one of these three cues was measured on the same 150 articles we have been staring at all lecture; whether they hold up on articles from a different year, platform, or election is a question these data cannot answer.

---

Wrapping up

What we built today

Two activities, one theme:

1. Sampling simulation: Every sample from the same population gives a different estimate. The estimates cluster around the truth, but individual samples can be far off. This is why we need methods to quantify uncertainty and tools for statistical inference.

2. Fake News Detective: To judge whether a result is surprising, you need a reference distribution — a model of what chance alone would produce. This is the logic that powers every statistical test.

The big idea

Your sample is one of infinitely many possible samples. Each would give a slightly different answer. Understanding that variation — its shape, its size, its structure — is what the rest of this course is about.

What’s next

• Wednesday Lab 1: Cross-tabs and probability rules in practice — you’ll use R to compute marginal, joint, and conditional probabilities from an actual clinical and public-health dataset (NSDUH).
• Friday’s pre-study (M07): Confidence intervals — turning the sampling variability you felt today into a stated range of plausible values for a parameter you cannot see.
• Next Lecture (M07): We build confidence intervals directly, first by bootstrap and then with the t-distribution — the other major use of a sampling distribution.