Project 2 · NHST Reproduction

Part 2 capstone · M01–M09 · Proposal due Fri Nov 6, 5 PM · Report due Fri Dec 4, 5 PM · Presentations December 7–11

Working in a team of 2–3, you will find a published behavioral-science study with accessible data, identify one published inferential result that uses a method taught in M09, and determine whether you can reproduce that result from the original data.

You are not reproducing an entire paper. You are reconstructing one clearly defined target result and the analytic steps required to obtain it.

Your two final products are:

  1. a polished Quarto HTML reproduction report; and
  2. a shared GitHub repository that allows another analyst to understand and re-run the reproduction.

You will also give a 30-minute group presentation explaining the published claim, how you reconstructed the analytic sample and test, whether the numerical result reproduced, and one important lesson from the reproduction process.

How it is graded: 100 points across six dimensions, set out in the Project Rubric. Read it now, not the week it is due — and keep in mind that whether the result reproduced is not one of the grading criteria.

Start here: download project-2-starter.zip. It contains your repository skeleton — folders, .gitignore, and every documentation template described on this page — so you begin with the structure already in place rather than building it. Project setup and reproducibility walks you through it.

What you’ll build

Here is the whole thing on one screen. Everything below this box is how to accomplish it.

Target One published inferential result, using an M09-eligible test, from a paper with open data
The report 1,500–2,000 words, rendered from programs/p2_reproduction.qmd
Reproduction plan Written before you run the target test, and appended to rather than edited
Sample The paper’s analytic N reconstructed, or the gap explained
Descriptives Reconciled against the paper before the test is run
The test The focal test, its effect size, and the assumptions that matter
Figure One results figure showing the focal effect in the raw or minimally summarized data
Comparison An explicit original-versus-reproduced table
Discussion What happened, and what would explain any discrepancy
Repository README.md, data-provenance.md, codebook.csv, shared Git history, clean-clone test passed
Presentation 30-minute slot — about 20 minutes presenting, 10 for questions

Whether the result reproduced is not one of the criteria. Characterizing honestly what you found is.

Why this project

A published statistical result is the end of a long chain:

question → measurement → data → analytic sample → descriptive results → statistical model/test → reported claim

Project 2 asks you to walk that chain backward and rebuild it.

This is a reproduction, not a replication:

  • Reproduction: same study data, same target analysis — can you recover the published result?
  • Replication: a new sample or new data collection — does the finding recur in new data?

You will not collect new data in this project.

The project integrates M01–M09:

Course work What you will demonstrate Where you learned it
M01 · Measurement Trace the paper’s constructs and variables into the shared dataset. M01 module
M02 · Tools Build a reproducible R/Quarto/GitHub project. M02 module · M02 lab
M03 · Visualization Visualize the raw effect or group pattern clearly. M03 module · M03 lab
M04 · Wrangling Reconstruct the analytic sample, exclusions, recodes, and derived variables. M04 module · M04 lab
M05 · Description Verify analytic N, group sizes, means/SDs or counts/proportions before testing. M05 module · M05 lab
M06–M08 · Inference Explain the sampling logic, uncertainty, hypotheses, and p-value correctly. M06 · M07 · M08
M09 · Common tests Choose, run, interpret, report, and scope the target NHST procedure. M09 module · M09 lab

You have already done a reproduction

The M09 lab was a computational reproduction of a published result — the same focal test, on the authors’ own data, checked against what they reported. That lab is the single most useful thing to re-read before starting, because this project is that exercise scaled up: you choose the paper, you reconstruct the analytic sample, and nobody has tidied the data for you first. If the project feels ambitious at first, that is part of the point — but it is also very close to something you have already practiced.

One thing worth hearing early: reproductions are often untidy, and that is a fact about published research rather than about you. Sample sizes disagree with the text. Exclusions are described in a sentence that could mean two things. A number lands close but not exactly. Every one of those is a normal reproduction experience, and none of them means your team has done something wrong — characterizing the discrepancy honestly is worth more here than matching the paper.

The project calendar · what to do, and when

This is the whole project on one screen. Everything below this table explains how to do a row; come back here whenever you lose the thread.

Week What you are doing Due
1 Find a feasible study data-first, confirm its data open in R, set up the repository, and fill in documentation/project-plan.md — the proposal template in your starter Proposal — Fri Nov 06, 5 PM
2 Study approved; write the reproduction plan before running the test; document provenance; start the codebook; begin reconstructing the analytic sample Reproduction plan — Fri Nov 13, 5 PM
3 Analytic N and descriptives reconciled; target test run; effect size and results figure complete; original-vs-reproduced table drafted —
4 Polish the report and discussion, finish the README, pass the clean-clone test, rehearse Report — Fri Dec 04, 5 PM
5 Presentations, December 7–11 — 30-minute slot per team —

The order matters more here than in Project 1. The reproduction plan is written in Week 2 and not revised afterwards, so a team that runs the test first has lost the thing that makes the plan worth anything.

If the descriptives still do not match at the start of Week 4, ask for help immediately rather than forcing the target test to agree. A documented discrepancy is a good outcome; a number massaged into place is not.

The professional workflow both projects use

Project 2 uses the same repository habits as Project 1, but now you are expected to carry them out more independently. The project goes much more smoothly if you keep these seven habits in view from the start.

1 · Start with a clear target

Identify the exact published result you intend to reproduce before doing the analysis.

2 · Preserve the raw data

The original shared file goes in data/raw/ and stays exactly as the authors published it.

Every change to the data happens in code — no hand edits, and nothing edited in SPSS or Excel. If a value needs recoding, a missing-value code needs converting, a variable needs collapsing into groups, or a column needs renaming, that change is written as a line of R.

This one is worth explaining rather than simply asserting, because these files open straight into SPSS’s data editor or a spreadsheet, where clicking a cell and typing is genuinely the faster-feeling option in the moment. The trouble is that a hand edit exists only on the laptop where it happened. Your teammates re-download the file and get something different. The change leaves no trace of what it was or why you made it, so a reader — including you, three weeks later — cannot reconstruct it. And if you ever need to redo the analysis on a corrected file, every hand edit has to be remembered and repeated from scratch.

Worse, if you save, you have written over the pristine original that everything else depends on — and that is easier to do by accident in SPSS than in a spreadsheet, because the file it opens is your raw .sav. Reading a file to look at it is fine; the line is saving anything back.

A line of R costs about the same effort and gives all of that back: everyone gets the identical dataset, the reasoning sits right there in the code, and re-running it is one click. So when you hit something that needs fixing, the answer is always a line of code — and if you are not sure how to write it, that is exactly the kind of question to bring to lab or office hours.

This matters even more in a reproduction than it did in Project 1. Your claim is that you rebuilt someone else’s analytic sample from their published data — and that claim only means something if every step between their file and your numbers is written down and re-runnable. A hand edit breaks the chain at exactly the point a careful reader would want to inspect.

3 · Document provenance

A reader should be able to find the article, repository, exact data file, version, and access information.

4 · Document the analytic variables

Your documentation/codebook.csv traces the variables used in your reproduction back to their raw source.

See the Analytic Codebook Guide.

5 · Make the analysis reproducible

Use project-relative paths and complete code. Do not rely on an object created manually in the Environment or a result copied from the Console.

6 · Use Git as a record of the work

Commit meaningful increments throughout the project, and keep data or credentials out of the repository when redistribution is not allowed.

7 · Test the project from a clean start

A teammate who was not the primary data-prep person must be able to clone the repository, follow the README, place the source data in the documented location, restart R, and render the reproduction report from beginning to end.

The central rule · one target result

A paper may report dozens of analyses. Project 2 does not ask you to reproduce them all.

Choose one central result from one study within the paper.

Examples:

Participants in the treatment condition scored lower than controls, t(98) = 2.31, p = .023.

Mean scores differed among four experimental conditions, F(3, 176) = 5.62, p = .001.

Treatment condition was associated with response category, χ²(2, N = 244) = 8.41, p = .015.

Your reproduction may require descriptive calculations, sample reconstruction, or follow-up output to understand that target result. That supporting work is part of the project.

The target itself remains one clearly specified inferential result.

Eligible M09 methods

The target result must use one of the uncomplicated designs developed in M09.

Design Eligible test
One continuous outcome; one sample against a fixed benchmark One-sample t-test
One continuous outcome; two independent groups Two-sample t-test
One continuous outcome; three or more independent groups One-way ANOVA
One categorical variable against specified expected proportions Chi-square goodness-of-fit
Two categorical variables measured on independent units Chi-square test of independence
One continuous outcome measured twice on the same units Paired-samples t-test

Each of these is developed in the M09 module, and the M09 pre-study is the decision map for matching a design to a test — use it when you are deciding whether a candidate paper is eligible.

The word “ANOVA” is not enough

A paper is not eligible merely because the Results section contains a familiar test name.

The target analysis should not require:

  • factorial or two-way ANOVA,
  • repeated-measures ANOVA with more than two occasions,
  • ANCOVA,
  • MANOVA,
  • regression or logistic regression,
  • moderation or mediation,
  • mixed/multilevel models,
  • clustered or nested data,
  • complex survey hypothesis testing,
  • survival analysis,
  • SEM,
  • or another method beyond M09.

A paper may contain those analyses elsewhere. The one result you reproduce must fit the simple M09 decision map.

The data requirement

Project 2 requires the data used for the published analysis.

Do not choose a paper whose relevant data are inaccessible.

Accepted sources include:

  • an OSF project,
  • PsychArchives,
  • openICPSR,
  • Harvard Dataverse,
  • a journal’s open-data supplement,
  • a GitHub repository maintained by the research team,
  • or another instructor-approved open repository.

A public-use dataset may be approved if the target result can genuinely be reconstructed with an M09-level design and does not require complex survey inference or extensive undocumented data engineering.

Simulating observations from published summary statistics does not count as a reproduction for this project.

Week 1 proposal · the approval packet

Due Friday, November 06 at 5 PM, at the end of the project’s first week.

Please do not begin the reproduction until the study is approved. The proposal is a scope check: it tells us the paper is eligible, the data actually open, and the target result is reconstructible in the time available. Approving it early is how we keep a team from spending three weeks on a paper whose data turn out to be unusable.

Where it goes. The proposal lives in your repository as documentation/project-plan.md, and it is submitted by pushing it. Your repository needs to exist and be shared with us by this deadline — but the raw data file should not be committed to it. You do need the data locally by then: the proposal asks for glimpse() output showing the file reads into R and carries the variables you expect. What waits until Week 2 is the formal work of documenting provenance and reconstructing the analytic sample.

So by 5 PM on November 06 we should be able to open your repository and find documentation/project-plan.md filled in, with both of us added as collaborators.

Your starter already contains the template. Open documentation/project-plan.md and fill in each section — the proposal is that file, kept in the repository and updated as the project develops. Its sections are:

Section What to provide
Team Names, and who takes first responsibility for which section
How we will work together Meeting rhythm, and what the team will do if someone goes quiet
Published study Full citation + DOI/link + PDF, and which Study/Experiment if the paper has more than one
Target result The exact sentence, table row, or figure result you will reproduce, with its page or section
Reported result Statistic, df, p-value, and effect size or CI, exactly as printed. Write “not reported” where the paper omits one
Which M09 test this is The eligible procedure the result uses, and why that is the match
Reported analytic N Total, plus group n’s if the paper reports them
Open data Repository link and the exact filename you will download
Proof that the data open The output of glimpse() on the file — enough to show it reads into R and carries the variables you expect
Likely variables Raw outcome, grouping, and any ID or pairing variables
Exclusions we will need to reconstruct The rules the paper describes, as best you can tell
Feasibility One sentence explaining why the result fits M01–M09
Backup candidate A second paper and target result, in case this one falls through

The template has one more section, Changes to this plan, which you leave empty for now. It is append-only: date each change and say why it happened.

Keep it in the repository in this shape, so the file doubles as your proposal and as the record of what you set out to do:

documentation/project-plan.md
# Project plan

## Team
[Names, and who takes first responsibility for which section]

**How we will work together**
- *Meeting rhythm:* [when and where you will meet — a standing time beats "we'll message each other"]
- *If someone goes quiet:* [how long you wait before checking in, and who tells the instructor — agree this now, while it is hypothetical]

## Published study
[Full citation, DOI or link. Note which Study/Experiment if the paper
contains more than one.]

## Target result
[Quote the exact sentence, table row, or figure result you will
reproduce, with its page or section.]

## Reported result
[Statistic, df, p-value, and effect size or CI, exactly as printed.
Write "not reported" where the paper omits one.]

## Which M09 test this is
[The eligible procedure the result uses, and why that is the match]

## Reported analytic N
[Total, plus group n's if the paper reports them]

## Open data
[Repository link and the exact filename you will download]

## Proof that the data open
[Paste the output of glimpse() on the file — enough to show it reads
into R and carries the variables you expect.]

## Likely variables
[Raw outcome, grouping, and any ID or pairing variables]

## Exclusions we will need to reconstruct
[The rules the paper describes, as best you can tell]

## Feasibility
[One sentence: why this fits in four weeks with M01–M09 tools]

## Backup candidate
[A second paper and target result, in case this one falls through]

## Changes to this plan
[Append-only. Date each change and say why it happened.]

This is not the same file as documentation/reproduction-plan.md. The project plan is what you propose in Week 1 — the paper, the data, and why the target is feasible. The reproduction plan, written in Week 2 once the study is approved, is the detailed analytic specification you commit to before running the target test. Keeping them separate is what lets a reader see that your analysis plan was fixed in advance rather than adjusted to fit the result.

Have a second candidate identified before you submit. Papers fall through for ordinary reasons — the data turn out to be summary tables rather than raw records, or the target analysis is more complicated than the Results section made it sound. A team with a backup is never the team scrambling.

Writing the code yourself

Many of the repositories you might choose include the authors’ own analysis script. Some are in R. You will be tempted to open it and run it.

Do not build your reproduction by copying it. Not because it is cheating in the plagiarism sense — the code is public and you are welcome to read it — but because doing so skips the entire thing this project is meant to teach. A reproduction assembled by pasting someone else’s script teaches you very little about wrangling, and leaves you unable to answer the first question anyone asks: why does your number differ from theirs?

The same applies to AI. An AI assistant that writes your filter() chain and an author’s script that already contains it are the same temptation wearing different clothes: both put code in your report that you did not reason through. So one rule covers both.

This is a step up from Project 1, deliberately. In Project 1 the standard was that you understand and can defend every line, whatever its source. Here you are additionally asked to attempt each analytic step yourself before seeking help of any kind — from the authors’ script, from a classmate, from a model. The reason is specific to this project: nobody has told you what the analysis should be, and working that out from a paper is the skill being assessed. Reading a solution before you have tried the problem removes the part that teaches you.

The rule · you must be able to explain and defend every line

If you cannot say what a line does, why it is there, and what would change if you removed it, it does not belong in your report. That is the standard, and it is the one we will actually test — in the presentation, you walk us through your code.

This is not a rule about detection. We are not going to try to tell whether a line came from you, from the authors, or from a model. We are going to ask you about it.

What you do yourself

These are the decisions the project exists to teach, and they are yours:

  • Deciding what the analysis needs — which test, which sample, which exclusions, which variables. This is the M09 reasoning, applied to a paper nobody walked you through.
  • Writing the first version of each step. Attempt it before you look anywhere else. A wrong first attempt that you then fix is worth more than correct code you did not write.
  • Reading your own error messages. Read the message, look at the object, form a guess. Then get help if you need it.
  • Every word of interpretation. What the result means, whether it reproduces, and what the discrepancy implies are the parts we are actually grading.

Where AI genuinely helps

Used well, an AI assistant is closer to a patient teaching assistant than to a code vending machine. These uses make you better at R:

  • “Explain this error message to me.” You have read it, you are still stuck, and you want it in plain language.
  • “What does the .groups argument in summarise() actually do?” Looking up an argument is what practising analysts do all day.
  • “Here is the code I wrote. Is there a clearer way to express it?” You wrote it, it works, and you want to learn the idiom.
  • “Why might my n be 412 when the paper says 405?” — as a source of hypotheses to check, not an answer to copy.

Where it substitutes for the learning

  • “Write the code that reproduces Table 2.” Now you have code you cannot defend.
  • Pasting an error into a chat before you have read it yourself.
  • Asking for the interpretation, the discussion, or the limitations.
  • Using in-editor completion that writes whole pipelines before you have decided what the pipeline should do.

How this is enforced · the presentation

Part 4 of your talk is where you walk us through a piece of your own code and explain what it does and why. We will ask follow-ups, and they will be about your choices: why that exclusion, why that join, what happened when it did not work the first time.

A team that wrote its own analysis will usually find this the easiest part of the talk — you are describing something you built. A team that assembled one from other people’s code will usually find it the hardest. That asymmetry is deliberate, and it is why we do not need a detection policy.

If you do use AI, say so briefly in your README.md — what you used it for, in one or two sentences. Using it well is not something to hide, and telling us costs you nothing.

Reproduction plan · written before running the target test

Once approved, create:

documentation/reproduction-plan.md

This document specifies what you believe the authors did. It is written before you run the target test, and not edited afterwards except to append — the record that your success bar was set in advance rather than chosen to match whatever you got.

This is not a prediction of whether the paper is “right.” It is a record of the analysis you intend to reconstruct before comparing your test result with the published number. Writing it down first helps keep the project honest and makes later discrepancies much easier to diagnose.

Your starter already contains the template:

documentation/reproduction-plan.md
# Reproduction plan

Written **before** you run the target test, and appended to rather than edited.
The point of the file is that your success bar was set before you saw your own
result. If something here turns out to be wrong, do not rewrite it — add a
dated note at the bottom saying what changed and why.

## Team
[Names, and who takes first responsibility for which section]

## Target result
[Quote the sentence from the paper containing the result you will
reproduce. Include the page or section.]

## The design, as we read it

| Decision | Planned reproduction |
|---|---|
| Article / Study | [citation, and which Study/Experiment] |
| Target published claim | [the claim in one sentence] |
| Unit of observation | [person, trial, school, ...] |
| Outcome type | [continuous / categorical count] |
| Outcome variable | [name in the authors' file] |
| Groups / measurements | [how many, and what they are] |
| Independent or paired? | [and how you can tell from the design] |
| Population parameter | [the quantity the test is about] |
| Null hypothesis | [in your own words] |
| Alternative hypothesis | [one- or two-sided, and why] |
| Statistical test | [which M09 test, and why it matches] |
| Analytic sample / exclusions | [stated N, plus each exclusion the paper describes] |
| Missing-data rule | [what you will do, and whether the paper says what they did] |

## Numbers we will compare against

Copy each one exactly as printed. Write "not reported" where the paper omits a
quantity, and "not available to us" where the paper reports it but you cannot
get at it — a figure you can only read approximately, a supplement behind a
paywall. Those are different problems and your discussion should treat them
differently. Your comparison table in the report is built from this list.

| Published quantity | As the paper reports it |
|---|---|
| Analytic N | |
| Group sizes | |
| Descriptives (M, SD, or counts) | |
| Test statistic | |
| df | |
| p-value | |
| Effect size | |
| Confidence interval | |

## Assumptions we will verify
[Which assumptions this test carries, and how you will check each one from
the design and the data — not from the paper's say-so.]

## Known ambiguities before we start
[Anything the paper leaves unclear that you expect to matter. An exclusion
it does not spell out, a variable it names differently from the file, a
missing-data rule it never states.]

## Appended notes
[Date-stamped, append-only. Record what you learned as you went —
especially anything that turned out to differ from the plan.]

Do not delete from this file. A plan that changed, with the change recorded, is evidence of careful work. A plan that matches your results perfectly because you rewrote it is not evidence of anything.

You do not set your own bar for “reproduced”

The plan records what you intend to reconstruct — the sample, the exclusions, the variables, the test, the numbers you will compare against. It does not define what counts as success. Every team is held to the same three-way standard — near-exact, partial, or not reproduced — set out in Original versus reproduced result below, so that verdicts mean the same thing across the class.

In particular, do not make your bar “the p-value lands on the same side of .05.” That is the most tempting criterion and one of the worst. Two analyses that differ materially can land on the same side of .05, and two nearly identical analyses can land on opposite sides of it when p is near the threshold — so a significance match is weak evidence of agreement and a significance mismatch is weak evidence of disagreement. Compare the quantities: the analytic N, the descriptives, the statistic, and the effect size.

Your repository

Your team works in a new shared GitHub repository — not a copy of your Project 1 repo. Keep it private through grading. If your team later wants to make it public, check the data’s licence, the article’s citation requirements, and the instructor’s guidance first — reproduction data often carries terms your own analysis does not. Starting fresh keeps the Git record readable for both projects. Download project-2-starter.zip and you begin with the structure, the .gitignore, and the documentation templates already in place.

Project setup and reproducibility is the walkthrough: the structure, publishing it, what each documentation file is for, and how to run the clean-clone test. It is the same guide you used for Project 1, so most of it will be familiar.

Project 2 adds a few files. documentation/reproduction-plan.md is the pre-analysis specification described above — the record that your success bar was set before you ran the target test, rather than after you saw the result. The starter also ships the citation machinery from your M08 paper: documentation/references.bib holds the entries and programs/apa.csl formats them, so the report’s References section is generated from your @key citations rather than typed by hand. Nothing else about the structure changes.

Two points from the guide matter more here than they did in Project 1:

Raw data stays raw — and here it is someone else’s data. The authors’ file lives in data/raw/ exactly as they published it. Every step from their file to your numbers is written as code, because that chain is your reproduction claim. A hand edit breaks it at precisely the point a careful reader would want to inspect.

The clean-clone test is the reproduction of your reproduction. If a teammate cannot clone the repo, follow the README, obtain the data, and re-run your analysis to the same numbers, then you have not demonstrated what the project asks you to demonstrate. Run it with time left to fix what it finds.

documentation/data-provenance.md · where the authors’ data came from

The setup guide explains what provenance is for. Project 2’s version asks for different things than Project 1’s, because you are documenting someone else’s published data rather than a survey you downloaded:

documentation/data-provenance.md
# Data provenance

## Article
[Full citation]
DOI: [article DOI]

## Data repository
[OSF / Dataverse / figshare / journal supplement / other]
URL: [direct link to the data page, not the article page]
DOI: [data DOI, if the repository issues one separately]

## Version or release
[Repositories often host several versions. Name the one you used.
If there is only one, say so explicitly.]

## Date accessed
[YYYY-MM-DD]

## Downloaded filename(s)
[Exactly as they arrived, before any renaming]

## Supporting documentation used
[Codebook, README, analysis script, supplement — whatever you relied on
to interpret the variables. Note if the authors provided none.]

## Access conditions
[Open? Registration required? Data-use agreement?]

## Expected local path
data/raw/[filename]

## Redistribution
[May this file be shared? Many repositories permit it under a licence;
some do not. Name the licence if there is one.]

## Notes
[Anything a future reader needs: a file that differs from what the paper
describes, a variable the codebook does not mention, an ambiguity you
had to resolve by judgment.]

Fill this in the day you obtain the file. The access date and the exact filename are the two things people forget, and they are the two that make a dataset findable again — which is the whole point of a reproduction.

data/README.md · the one tracked file in data/

Your starter ships this, and it is worth a look because it is the only thing inside data/ that a teammate sees after cloning — everything else is ignored. It differs slightly from Project 1’s, because it points at this project’s report and at someone else’s data:

data/README.md
# Data

The data files themselves are **not** tracked by Git. This file is.

## data/raw/
The authors' file exactly as downloaded, never edited by hand.

Expected file: `[filename]`

To obtain it, follow `documentation/data-provenance.md`.

## data/derived/
Our reconstruction of the authors' analytic sample, built from the raw
file by `programs/p2_reproduction.qmd`. Do not edit by hand.

Expected file: `analytic.rds`

## If you just cloned this repository
Both folders will be empty. Obtain the raw file, place it in
`data/raw/`, then render `programs/p2_reproduction.qmd` — it rebuilds
`data/derived/` for you.

The reproduction report

Read an example first

An annotated example report walks through a short, deliberately competent reproduction with notes on what would raise each section. A reproduction report is a genre almost nobody has written before, and the example shows what “characterize the discrepancy rather than smooth it over” actually looks like in prose.

Aim for approximately 1,500–2,000 words, not counting code and tables.

The report deliberately follows the logic of M09. If you are unsure what belongs in a section, go back to the way M09 asked you to move from design, to parameter, to hypotheses, to test, to interpretation. The clearest reports usually feel like a guided reconstruction: here is the claim, here is the sample, here is the test, here is what matched, and here is what did not.

1 · The published question and claim

Open by orienting a reader who has not read the paper. Cover, briefly:

  • The research question. What did the authors want to know, in one sentence of plain English?
  • The study design. How was it run — an experiment or an observational study, between- or within-subjects, how many groups or measurements, and how participants came to be in the conditions they were in.
  • The variables. Name the outcome and the grouping or predictor variable as the paper describes them, with their units or levels. If the paper’s name for a variable differs from the column name in the data file, say both — that mapping is the thing your codebook records, and readers will need it to follow your code.
  • Who was studied. The target sample and the analytic sample the paper reports.
  • The exact result you are reproducing. The specific sentence, table row, or figure value.

Cite the original paper with a @key citation — the same automatic-references workflow as your M08 paper. The starter ships references.bib and the APA style file already wired up: insert citations from the visual editor (Insert → Citation → From DOI) and the References section at the end of the report builds itself from what you cite. Nothing in that section is typed by hand.

A reader should be able to finish this section and, without having opened the paper, state what was measured on whom and what the authors concluded. Everything after this assumes they can.

2 · Recognize the design

Before naming the test, identify:

  • unit of observation,
  • outcome type,
  • number of groups or measurements,
  • whether observations are independent or paired,
  • and the population parameter the question targets.

Then explain why the published test matches that design.

3 · Parameter and hypotheses

State:

  • the population parameter,
  • (H_0),
  • (H_a),
  • and whether the original analysis was one- or two-sided if the paper specifies it.

Use the M09 convention of explaining the hypotheses in words as well as symbols.

4 · Data and analytic-sample reconstruction

Describe:

  • where you obtained the data,
  • the raw file,
  • the variables needed,
  • exclusions,
  • missing-data handling,
  • recodes or derived variables,
  • and final analytic N.

The code performing these steps belongs in the .qmd.

5 · Reconstruct the descriptive results

Do this before the test. If the group sizes or means do not match the paper, the disagreement is in the sample rather than the analysis — and that is a much easier problem to diagnose now than after a mismatched p-value. The tools are the M05 module’s.

This step is worth taking seriously. Teams often want to rush to the target NHST, but the descriptives are what tell you whether you are even analyzing the same sample the authors did.

Create a comparison such as:

Quantity Published Reproduced
Analytic N
Group n
Mean / proportion
SD / counts

Then ask:

Do our descriptives match closely enough that we appear to have reconstructed the same analytic sample?

If not, stop and investigate before interpreting the hypothesis test.

The results figure · required

Build one ggplot2 figure showing the effect you are reproducing. This is a graded deliverable, not a decoration, and it is the thing you will put on screen during your talk.

Build one ggplot2 figure that makes the focal effect or association visible in the raw or minimally summarized data. What that looks like depends on your design:

Your target test A figure that shows the effect
Two-sample t or one-way ANOVA Group means or medians with a display of uncertainty, and — where the n allows it — the individual observations underneath
Paired t Paired trajectories linking each unit’s two measurements, or the distribution of the difference scores
One-sample t The distribution of the outcome with the benchmark value marked, plus the mean and its interval
Chi-square (either kind) Counts or proportions by category, with the denominator unmistakable — say what each percentage is a percentage of

The M09 lab is the worked model for the first row: points jittered under group means with geom_errorbar() for the interval. If your design sits in another row, the principle is the same — show the data the conclusion rests on, not a picture of the machinery that tested it.

Hold it to the M03 standard, because a reproduction figure is read by someone deciding whether to believe you:

  • title states the takeaway, not the variables — “Treatment scores exceeded control by 4.2 points”, not “Score by condition”.
  • subtitle carries scope: n, the year or wave, any transformation.
  • Axis labels are human-readable with units — “Aggression score (0–10)”, never agg_score.
  • caption carries provenance: "Source: " plus the authors, year, and the file you analyzed.
  • theme_minimal(), and color used deliberately or not at all.

The test your figure has to pass

Show it to someone who has not read your report. Can they say what was compared, roughly how big the difference is, and how much uncertainty surrounds the estimate you are reporting — without you narrating?

If they cannot, the figure is not finished. That is the same standard M03 set with the Rosling chart, applied to your own result.

6 · Assess the design and model

The assumptions behind each test, and what to do when one is questionable, are in the M09 module. For the confidence-interval half of the reporting, see M07; for what a p-value does and does not license, M08.

Following M09, distinguish:

Design assumptions

  • independence or pairing,
  • unit of analysis,
  • treatment/group formation,
  • other design facts that cannot be diagnosed from a histogram.

Model considerations

  • whether the mean is a sensible summary,
  • influential/extreme observations,
  • adequacy of the sampling approximation,
  • variance considerations where relevant,
  • expected counts for chi-square.

Do not perform a generic assumption ritual. Discuss the considerations that actually matter for your chosen test.

7 · Reproduce the test

Run the M09 analysis and report:

  • raw effect,
  • 95% CI where applicable,
  • test statistic,
  • degrees of freedom,
  • p-value,
  • course-standard effect size,
  • and effect-size CI when your M09 workflow provides one.

Examples of course-standard effect sizes:

Test Effect size
One-sample t Cohen’s d
Two-sample t Cohen’s d
One-way ANOVA (^2)
Chi-square Cramér’s V
Paired-samples t Cohen’s (d_z)

One choice the table hides: which SD standardizes Cohen’s d. If you ran Welch’s two-sample t — and you usually will, since it is R’s default — pair it with the unpooled standardizer, which is what M09 uses:

my_data |>
  mutate(group = fct_rev(group)) |>       # see the warning below
  cohens_d(outcome ~ group, data = _, pooled_sd = FALSE)

cohens_d() pools by default, so this is a deliberate argument rather than something you get for free.

The sign trap: t and d do not agree by default

t_test() takes the direction of the comparison from its order argument. cohens_d() ignores that entirely and subtracts in factor-level order — so the same comparison can come back as t = +9.21 with d = −2.38. The magnitude is right and only the sign is wrong, which is exactly why it survives into an APA sentence unnoticed.

Reverse the factor before calling cohens_d(), as the code above does, then check that your t and your d carry the same sign before you write either one down. The M09 lab uses the same fct_rev() step for this reason.

The pooled and unpooled versions often agree to two decimals, which means a mismatch with the paper is rarely explained by this choice — but say which one you used, because a reader cannot tell from the number alone.

If the original paper did not report the course-standard effect size, label it clearly as an additional result calculated for this course, not something you are claiming the authors reported.

8 · Original versus reproduced result

Use one comparison table:

Quantity Original paper Our reproduction Match?
Analytic N
Raw effect / key descriptives
Test statistic
df
p-value
Effect size, if reported

Then classify the outcome:

Near-exact reproduction

The analytic N, descriptives, test statistic, and inferential result agree within ordinary reporting or rounding differences.

Partial reproduction

The qualitative conclusion is similar, but one or more numerical quantities differ beyond ordinary rounding and your team identifies a plausible analytic reason.

Not reproduced

The target result differs materially and the discrepancy remains unresolved after careful reconstruction.

Your grade does not depend on the category.

It depends on whether your reconstruction and diagnosis are transparent and defensible.

9 · Discussion and scope

Discuss:

  • what reproduced or failed to reproduce,
  • the most likely reason for any discrepancy,
  • what you learned from reconstructing the analysis,
  • and what the design permits you to conclude.

Remember an important M09 principle:

The statistical test does not create the study’s causal or generalizability warrant. Those claims come from the design.

Common sources of disagreement

If your result does not match, do not assume the project has failed. Start by investigating possibilities such as:

  • wrong study or dataset file,
  • different analytic N,
  • an exclusion you missed,
  • missing-data handling,
  • different coding or category reference,
  • different threshold for a derived categorical variable,
  • Student versus Welch two-sample t — and there is a quick way to tell which the authors ran, even when the paper does not say: check whether the reported df equals \(n_1 + n_2 - 2\). If it does, they used Student’s; if it is a non-integer or simply smaller, they used Welch’s. A paper reporting df = 861.22 for two groups of 419 and 486 has told you it used Welch’s without writing the word,
  • one- versus two-sided test,
  • ANOVA subset or follow-up differing from the paper,
  • weighting,
  • rounding,
  • software defaults,
  • undocumented author decisions,
  • an error in shared code or the published report,
  • or an implementation error in your own analysis.

Because a reproduction uses the same underlying data, a mismatch is not explained by “sampling variability from a new sample.”

The presentation · 30 minutes

Every team member must speak and be prepared to answer questions about the full project.

Format · remote, and the choice of medium is yours

Presentations are delivered remotely over Microsoft Teams, across the two class meetings of presentation week (December 7–11) — two teams on Monday, two on Wednesday. You will know your slot in advance. One person shares their screen; you decide how to hand off between speakers.

How you present the material is up to you. Slides are one option, not a requirement. A rendered Quarto report scrolled through deliberately, a live notebook, or a small set of figures shown full-screen can all work — often better than slides, since your figures already exist and re-making them as slide images only invites version drift.

Whatever you choose, two things matter more than the medium:

  • Everything is legible at screen-share resolution. Test it by sharing to a teammate first — axis labels and table text that read fine on your own monitor frequently do not survive Teams compression.
  • You can move to a specific figure on request. Questions will not arrive in your running order, so know how to jump to Figure 2 without scrolling past everything else.

Have one person hold a backup copy of whatever you are presenting, in case the presenter’s connection drops.

Part 1 · The published claim · about 5 minutes

Your audience has not read this paper. Orient them first:

  • The research question, in one sentence.
  • The study design — experiment or observational, between- or within-subjects, how many groups or measurements.
  • The variables — the outcome and the grouping variable, with units or levels.
  • Who was studied, and the analytic sample the paper reports.
  • The exact result you targeted, and why it was an eligible M09 analysis.

Five minutes is enough for this only if you have decided in advance what to leave out. Practice it.

Part 2 · Reconstructing the analysis · about 4 minutes

  • What data did you obtain?
  • What variables mapped to the paper’s constructs?
  • What exclusions/missing-data rules mattered?
  • Did the analytic N and descriptives match?

Part 3 · Did it reproduce? · about 7 minutes

Put your results figure on screen for this part and talk from it rather than from a table of numbers.

Show:

  • the raw effect figure,
  • original versus reproduced numbers,
  • test statistic and uncertainty,
  • effect size,
  • and your reproduction classification.

Part 4 · Behind the reproduction · about 4 minutes

Put your code on screen and walk us through a piece of it. Choose something that took work — a wrangling step, the analytic-sample reconstruction, the test itself — and talk us through what it does and why you wrote it that way. Expect follow-up questions about your choices: why that exclusion, why that join, what happened the first time it did not work.

This is the part of the talk where writing the code yourself pays off. Anyone can read code aloud; explaining why it is shaped the way it is only works if you shaped it.

Then teach one transferable lesson:

  • how you mapped the design to the right test,
  • how you found an undocumented exclusion,
  • how the codebook crosswalk helped,
  • why two apparently equivalent analysis choices produced different results,
  • or another reproducibility lesson.

Part 5 · Questions and discussion · about 10 minutes

Ten minutes, and they are not a formality — this is where you get to show what your team understood, where the reconstruction was difficult, and how you reasoned through it.

Everyone should expect to answer something. Questions come from the class first, then from us, and they will be aimed at whoever did the part being asked about. Reasonable things to be ready for: why did you exclude those cases? · what would change if you had kept them? · why that test and not the obvious alternative? · what did you get wrong the first time?

That last one is worth preparing deliberately. “Here is what we got wrong and how we found it” is the most useful thing a team can give the rest of the class, and it is the answer that most convinces us you did the work.

What to submit

Your repository is the submission. There is no separate file to upload — we grade the GitHub repo directly, at the state it is in when the deadline passes.

Add both instructors as collaborators by the end of the project’s first week. That is well before the final deadline on purpose: we read your project plan during that week, and a private repository we have not been invited to is one we cannot open. On github.com, open your repository → Settings → Collaborators → Add people, and add KimberlyHenry (instructor) and alliekom (lab instructor). Type the usernames exactly rather than searching by name — GitHub’s search will happily offer you someone else entirely.

Week 1, not the last day

Your repository is private, which is correct — but it also means that until the invitations are accepted, nobody but your team can see a thing. That is why this belongs in the first week rather than the last: your project plan is due then and we read it then, so an uninvited repository is an unread plan.

Add both of us as soon as the repository exists — KimberlyHenry (instructor) and alliekom (lab instructor) — and confirm in Settings → Collaborators that each appears.

Adding us early costs you nothing. We read the plan in the first week, and we look again when the deadline passes; nobody is watching your commits in between.

Two consequences of grading the repo rather than a file, both in your favour once you know them:

  • The commit history is visible, and that is the point. It is the evidence that the work was shared. A single last-minute commit cannot demonstrate the ongoing, shared workflow this part of the project is designed to assess.
  • What is not committed does not exist. Anything sitting only on someone’s laptop — a figure, a fixed typo, the final render — is not part of what we see. Push before the deadline, then check the repository on github.com in a browser and confirm what you expect is actually there.

Data files excluded by your .gitignore are meant to be absent; that is the design, not an omission. data/README.md and documentation/data-provenance.md are what tell us how to obtain them.

Final checklist

Before submitting:

Then read the Project Rubric one more time with your finished repository open beside it. Most points lost on this project are lost to something the rubric names plainly and no one checked at the end.