Project 2 · NHST Reproduction

Part 2 capstone · M01–M09

Working in a team of 2–3, you will find a published behavioral-science study with accessible data, in the research area of at least one team member, identify one published inferential result that uses a method taught in M09, and determine whether you can reproduce that result from the original data.

You are not reproducing an entire paper. You are reconstructing one clearly defined target result and the analytic steps required to obtain it.

Your two final products are:

  1. a polished Quarto HTML reproduction report; and
  2. a shared GitHub repository that allows another analyst to understand and re-run the reproduction.

You will also give a 30-minute group presentation explaining the published claim, how you reconstructed the analytic sample and test, whether the numerical result reproduced, and one important lesson from the reproduction process.

Start here · what to do this week

This week, do only four things:

  1. Form your team of 2–3 and download project-2-starter.zip, your repository skeleton with every template already in it. Project setup and reproducibility walks you through it.
  2. Find one candidate paper in one team member’s research area, with open data and an eligible M09 result, using the data-first search below.
  3. Open the data in R and confirm that the variables you need are there.
  4. Fill in documentation/project-plan.md and push it by Friday, November 06 at 5 PM.

Don’t run the focal test yet. Once your study is approved, Week 2 covers the reproduction plan and the analytic reconstruction.

Milestone What it is Due
Proposal The paper, its data, and the exact target result Fri Nov 06, 5 PM
Reproduction plan Your analytic specification, written before you see your result Fri Nov 13, 5 PM
Final report The reproduction report and a complete repository Fri Dec 04, 5 PM
Presentation A 30-minute slot per team December 7–11

How it is graded: 100 points across six dimensions, set out in the Project Rubric. Read it now, not the week it is due — and keep in mind that whether the result reproduced is not one of the grading criteria.

What you’ll build

Here is the whole thing on one screen. Everything below this box is how to accomplish it.

Target One published inferential result, using an M09-eligible test, from a paper with open data in a team member’s research area
The report 1,500–2,000 words, rendered from programs/p2_reproduction.qmd
Reproduction plan Written before you run the target test, and appended to rather than edited
Sample The paper’s analytic N (the number of cases left in the analysis after exclusions) reconstructed, or the gap explained
Descriptives Reconciled against the paper before the test is run
The test The focal test, its effect size, and the assumptions that matter
Figure One results figure showing the focal effect in the raw or minimally summarized data
Comparison An explicit original-versus-reproduced table
Discussion What happened, and what would explain any discrepancy
Repository README.md, data-provenance.md, codebook.csv, shared Git history, clean-clone test passed
Presentation 30-minute slot — about 20 minutes presenting, 10 for questions

Whether the result reproduced is not one of the criteria. Characterizing honestly what you found is.

Why this project

A published statistical result is the end of a long chain:

question → measurement → data → analytic sample → descriptive results → statistical model/test → reported claim

Project 2 asks you to walk that chain backward and rebuild it.

This is a reproduction, not a replication:

  • Reproduction: same study data, same target analysis — can you recover the published result?
  • Replication: a new sample or new data collection — does the finding recur in new data?

You will not collect new data in this project.

The project integrates M01–M09:

Course work What you will demonstrate Where you learned it
M01 · Measurement Trace the paper’s constructs and variables into the shared dataset. M01 module
M02 · Tools Build a reproducible R/Quarto/GitHub project. M02 module · M02 lab
M03 · Visualization Visualize the raw effect or group pattern clearly. M03 module · M03 lab
M04 · Wrangling Reconstruct the analytic sample, exclusions, recodes, and derived variables. M04 module · M04 lab
M05 · Description Verify analytic N, group sizes, means/SDs or counts/proportions before testing. M05 module · M05 lab
M06–M08 · Inference Explain the sampling logic, uncertainty, hypotheses, and p-value correctly. M06 · M07 · M08
M09 · Common tests Choose, run, interpret, report, and scope the target NHST procedure. M09 module · M09 lab

You have already done a reproduction

The M09 lab was a computational reproduction of a published result — the same focal test, on the authors’ own data, checked against what they reported. That lab is one of the most useful things to re-read before starting, because this project is that exercise scaled up: you choose the paper, you reconstruct the analytic sample, and nobody has tidied the data for you first. If the project feels ambitious at first, that is part of the point — but it is also very close to something you have already practiced.

One thing worth hearing early: reproductions are often untidy, and that is a fact about published research rather than about you. Sample sizes disagree with the text. Exclusions are described in a sentence that could mean two things. A number lands close but not exactly. Every one of those is a normal reproduction experience, and none of them means your team has done something wrong — characterizing the discrepancy honestly is worth more here than matching the paper.

The project calendar · what to do, and when

This is the whole project on one screen. Everything below this table explains how to do a row; come back here whenever you lose the thread.

Week What you are doing Due
1 Find a feasible study data-first, confirm its data open in R, set up the repository, and fill in documentation/project-plan.md — the proposal template in your starter Proposal — Fri Nov 06, 5 PM
2 Study approved; write the reproduction plan before running the test; document provenance; start the codebook; begin reconstructing the analytic sample Reproduction plan — Fri Nov 13, 5 PM
3 Analytic N and descriptives reconciled; target test run; effect size and results figure complete; original-vs-reproduced table drafted —
4 Polish the report and discussion, finish the README, pass the clean-clone test, rehearse Report — Fri Dec 04, 5 PM
5 Presentations, December 7–11 — 30-minute slot per team —

The order matters more here than in Project 1. The reproduction plan is written in Week 2 and not revised afterwards. A team that runs the test first loses what makes the plan useful: a record of the analysis written before anyone saw the answer.

If the descriptives still do not match at the start of Week 4, ask for help immediately rather than forcing the target test to agree. A documented discrepancy is a good outcome; a number massaged into place is not.

The central rule · one target result

A paper may report dozens of analyses. Project 2 does not ask you to reproduce them all.

Choose one central result from one study within the paper.

The result has to come from the research area of at least one team member, and it has to be substantively interesting: a finding people in that area care about, not one picked only because it was easy to find. The report and the presentation are built around why the finding matters, and that is much easier to write, and much better to listen to, when someone on the team knows the area. Every area has studies with open data and simple comparisons, so start your search there.

Examples:

Participants in the treatment condition scored lower than controls, t(98) = 2.31, p = .023.

Mean scores differed among four experimental conditions, F(3, 176) = 5.62, p = .001.

Treatment condition was associated with response category, χ²(2, N = 244) = 8.41, p = .015.

Your reproduction may require descriptive calculations, sample reconstruction, or follow-up output to understand that target result. That supporting work is part of the project.

The target itself remains one clearly specified inferential result.

Eligible M09 methods

The target result must use one of the uncomplicated designs developed in M09.

Design Eligible test Estimand: in words, then in symbols
One continuous outcome; one sample against a fixed benchmark One-sample t-test The population mean, compared with a fixed benchmark: \(\mu\) (against \(\mu_0\))
One continuous outcome; two independent groups Two-sample t-test The difference between two population means: \(\mu_1 - \mu_2\)
One continuous outcome; three or more independent groups One-way ANOVA The k population means: \(\mu_1, \mu_2, \dots, \mu_k\)
One categorical variable against specified expected proportions Chi-square goodness-of-fit The population proportion in each category: \(p_1, \dots, p_k\) (against values set in advance)
Two categorical variables measured on independent units Chi-square test of independence The proportion with the outcome in each group: for two groups, \(p_A\) and \(p_B\)
One continuous outcome measured twice on the same units Paired-samples t-test (taught only in the M09 bonus activity, not the core Module) The population mean of the within-pair differences: \(\mu_d\)

The last column is what your proposal’s Estimand field asks for. Write it for your target, with your own groups named: \(\mu_\text{workshop} - \mu_\text{waitlist}\) says more than \(\mu_1 - \mu_2\).

Each of these is developed in the M09 module except the paired-samples t, which was taught only as a bonus activity. It is still eligible, but if you choose it, plan to work through the M09 bonus activity first, because it is the only place the course develops that test. The M09 pre-study is the decision map for matching a design to a test — use it when you are deciding whether a candidate paper is eligible.

The word “ANOVA” is not enough

A paper is not eligible merely because the Results section contains a familiar test name.

The target analysis should not require:

  • factorial or two-way ANOVA,
  • repeated-measures ANOVA with more than two occasions,
  • ANCOVA,
  • MANOVA,
  • regression or logistic regression,
  • moderation or mediation,
  • mixed/multilevel models,
  • clustered or nested data,
  • complex survey hypothesis testing,
  • survival analysis,
  • SEM,
  • or another method beyond M09.

A paper may contain those analyses elsewhere. The one result you reproduce must fit the simple M09 decision map.

The data requirement

Project 2 requires the data used for the published analysis.

Do not choose a paper whose relevant data are inaccessible.

Accepted sources include:

  • an OSF project,
  • PsychArchives,
  • openICPSR,
  • Harvard Dataverse,
  • a journal’s open-data supplement,
  • a GitHub repository maintained by the research team,
  • or another instructor-approved open repository.

A public-use dataset may be approved if the target result can genuinely be reconstructed with an M09-level design and does not require complex survey inference or a large amount of undocumented data preparation.

Simulating observations from published summary statistics does not count as a reproduction for this project.

Week 1 proposal · the approval packet

Due Friday, November 06 at 5 PM, at the end of the project’s first week.

Please do not begin the reproduction until the study is approved. The proposal is a scope check: it tells us the paper is eligible, the data actually open, and the target result is reconstructible in the time available. Approving it early is how we keep a team from spending three weeks on a paper whose data turn out to be unusable.

Where it goes. The proposal lives in your repository as documentation/project-plan.md, and it is submitted by pushing it. Your repository needs to exist and be shared with us by this deadline — but the raw data file should not be committed to it. You do need the data locally by then: the proposal asks for glimpse() output showing the file reads into R and carries the variables you expect. What waits until Week 2 is the formal work of documenting provenance and reconstructing the analytic sample.

So by 5 PM on November 06 we should be able to open your repository and find documentation/project-plan.md filled in, with both of us added as collaborators.

Your starter already contains the template. Open documentation/project-plan.md and fill in each section; the proposal is that file, kept in your repository. Its sections are:

Section What to provide
Team Names, and who takes first responsibility for which section
How we will work together Meeting rhythm, and what the team will do if someone goes quiet
Published study Full citation + DOI/link + PDF, and which Study/Experiment if the paper has more than one
Target result The exact sentence, table row, or figure result you will reproduce, with its page or section
Why this result matters Whose research area it connects to, what the finding contributed, and what it would mean if it did or did not reproduce
Reported result Statistic, df, p-value, and effect size or CI, exactly as printed. Write “not reported” where the paper omits one
Estimand The population quantity the target result is about, in words first and then in symbols (the Eligible M09 methods table has the pattern for each test)
Which M09 test this is The eligible procedure the result uses, and why that is the match
Reported analytic N Total, plus group n’s if the paper reports them, and whether your file’s row count matches the paper’s recruited N or its final analytic N (see Check whether the exclusions have already been applied)
Open data Repository link and the exact filename you will download
Proof that the data open The output of glimpse() on the file — enough to show it reads into R and carries the variables you expect
Likely variables Raw outcome, grouping, and any ID or pairing variables, with a check that the grouping variable holds exactly the groups you compare
Exclusions we will need to reconstruct The rules the paper describes, as best you can tell
Feasibility One sentence explaining why the result fits M01–M09
Backup candidate A second paper and target result, in case this one falls through, meeting the same requirements

The template has one more section, Changes to this plan, which you leave empty for now. After approval, don’t rewrite the other sections. If something changes, record it under Changes to this plan with the date and the reason. That way the file stays a record of what you set out to do, and of what changed and why.

documentation/project-plan.md
# Project plan

## Team
[Names, and who takes first responsibility for which section]

**How we will work together**
- *Meeting rhythm:* [when and where you will meet — a standing time beats "we'll message each other"]
- *If someone goes quiet:* [how long you wait before checking in, and who tells the instructor — agree this now, while it is hypothetical]

## Published study
[Full citation, DOI or link. Note which Study/Experiment if the paper
contains more than one.]

## Target result
[Quote the exact sentence, table row, or figure result you will
reproduce, with its page or section.]

## Why this result matters
[Whose research area it connects to (name the team member), what the
finding contributed to that area, and what it would mean for the area
if the result did or did not reproduce. One to three sentences.]

## Reported result
[Statistic, df, p-value, and effect size or CI, exactly as printed.
Write "not reported" where the paper omits one.]

## Estimand
[The population quantity the target result is about, in words first
and then in symbols: for example, "the difference in mean fatigue between
all students like these who take the workshop and all who wait,
mu_workshop - mu_waitlist."]

## Which M09 test this is
[The eligible procedure the result uses, and why that is the match]

## Reported analytic N
[Total, plus group n's if the paper reports them. If the paper doesn't
print an N for your target result itself, write the N it does print and
the stage it describes (recruited, screened, after exclusions).]

**Row-count check:** [How many rows does the downloaded file have? Does
that match the paper's recruited or screened N (so the exclusions are
yours to apply) or its final analytic N (so they were already applied)?]

## Open data
[Repository link and the exact filename you will download]

## Proof that the data open
[Paste the output of glimpse() on the file — enough to show it reads
into R and carries the variables you expect.]

## Likely variables
[Raw outcome, grouping, and any ID or pairing variables. Check that the
grouping variable holds exactly the groups your target compares: a column
that combines two factors (say, chart type and caption in one four-level
variable) has to be split first.]

## Exclusions we will need to reconstruct
[The rules the paper describes, as best you can tell]

## Feasibility
[One sentence: why this fits in four weeks with M01–M09 tools]

## Backup candidate
[A second paper and target result, in case this one falls through. It
has to meet the same requirements, including the research-area one.]

## Changes to this plan
[Append-only. Date each change and say why it happened.]

This is not the same file as documentation/reproduction-plan.md. The project plan is what you propose in Week 1 — the paper, the data, and why the target is feasible. The reproduction plan, written in Week 2 once the study is approved, is the detailed analytic specification you commit to before running the target test. Keeping them separate is what lets a reader see that your analysis plan was fixed in advance rather than adjusted to fit the result.

Have a second candidate identified before you submit. Papers fall through for ordinary reasons — the data turn out to be summary tables rather than raw records, or the target analysis is more complicated than the Results section made it sound. A team with a backup can switch quickly instead of starting the search over.

Writing the code yourself

Many of the repositories you might choose include the authors’ own analysis script, and an AI assistant can write a filter() chain in seconds. One rule covers both.

Attempt each analytic step yourself first. After that, you may use the authors’ code, classmates, documentation, or AI to understand an error, learn syntax, or improve code you have written. Don’t hand off the analytic decisions or the first attempt: working out what the analysis should be, from a paper nobody walked you through, is the skill this project assesses. That is a deliberate step up from Project 1, where the standard was that you understand and can defend every line, whatever its source.

The rule · you must be able to explain and defend every line

If you cannot say what a line does, why it is there, and what would change if you removed it, it does not belong in your report. That is the standard, and it is the one we will actually test — in the presentation, you walk us through your code.

This is not a rule about detection. We are not going to try to tell whether a line came from you, from the authors, or from a model. We are going to ask you about it.

How this is checked: the presentation. Part 4 of your talk is where you walk us through a piece of your own code. We will ask follow-ups about your choices: why that exclusion, why that join, what happened when it did not work the first time. A team that wrote its own analysis usually finds this the easiest part of the talk, because it is describing something it built.

If you use AI, say so briefly in your README.md: what you used it for, in a sentence or two. Using it well is nothing to hide.

Why not build the reproduction from the authors’ script? Not because it is cheating in the plagiarism sense — the code is public and you are welcome to read it — but because doing so skips what this project is meant to teach. A reproduction assembled by pasting someone else’s script teaches you very little about wrangling, and leaves you unable to answer the first question anyone asks: why does your number differ from theirs?

What you do yourself. These are the decisions the project exists to teach, and they are yours:

  • Deciding what the analysis needs — which test, which sample, which exclusions, which variables. This is the M09 reasoning, applied to a paper nobody walked you through.
  • Writing the first version of each step. Attempt it before you look anywhere else. A wrong first attempt that you then fix is worth more than correct code you did not write.
  • Reading your own error messages. Read the message, look at the object, form a guess. Then get help if you need it.
  • Every word of interpretation. What the result means, whether it reproduces, and what the discrepancy implies are the parts we are actually grading.

Where AI genuinely helps. Used well, an AI assistant is closer to a patient teaching assistant than to a tool that writes your code for you:

  • “Explain this error message to me.” You have read it, you are still stuck, and you want it in plain language.
  • “What does the .groups argument in summarise() actually do?” Looking up an argument is something practicing analysts do constantly.
  • “Here is the code I wrote. Is there a clearer way to express it?” You wrote it, it works, and you want to learn a clearer, more standard way to write it.
  • “Why might my n be 412 when the paper says 405?” — as a source of hypotheses to check, not an answer to copy.

Where it substitutes for the learning:

  • “Write the code that reproduces Table 2.” Now you have code you cannot defend.
  • Pasting an error into a chat before you have read it yourself.
  • Asking for the interpretation, the discussion, or the limitations.
  • Using in-editor completion that writes whole pipelines before you have decided what the pipeline should do.

Reproduction plan · written before running the target test

Once approved, create:

documentation/reproduction-plan.md

This document specifies what you believe the authors did. It is written before you run the target test, and not edited afterwards except to append — the record that your analysis plan was fixed in advance rather than adjusted to match whatever you got. Commit it before you run the test. What shows the plan came first is the order of the commits in your Git history, not how much time passes between them.

This is not a prediction of whether the paper is “right.” It is a record of the analysis you intend to reconstruct before comparing your test result with the published number. Writing it down first helps keep the project honest and makes later discrepancies much easier to diagnose.

Your starter already contains the template; open the file and answer each prompt.

documentation/reproduction-plan.md
# Reproduction plan

Written **before** you run the target test, and appended to rather than edited.
The point of the file is that your analysis plan was fixed before you saw your
own result. If something here turns out to be wrong, do not rewrite it — add a
dated note at the bottom saying what changed and why.

## Team
[Names, and who takes first responsibility for which section]

## Target result
[Quote the sentence from the paper containing the result you will
reproduce. Include the page or section.]

## The design, as we read it

| Decision | Planned reproduction |
|---|---|
| Article / Study | [citation, and which Study/Experiment] |
| Target published claim | [the claim in one sentence] |
| Unit of observation | [person, trial, school, ...] |
| Outcome type | [continuous / categorical count] |
| Outcome variable | [name in the authors' file] |
| Groups / measurements | [how many, and what they are] |
| Independent or paired? | [and how you can tell from the design] |
| Estimand | [the population quantity the result is about, in words, then in symbols] |
| Null hypothesis | [in your own words] |
| Alternative hypothesis | [one- or two-sided, and why] |
| Statistical test | [which M09 test, and why it matches] |
| Analytic sample / exclusions | [stated N, plus each exclusion the paper describes] |
| Missing-data rule | [what you will do, and whether the paper says what they did] |

## Numbers we will compare against

Copy each one exactly as printed. Write "not reported" where the paper omits a
quantity, and "not available to us" where the paper reports it but you cannot
get at it — a figure you can only read approximately, a supplement behind a
paywall. Those are different problems and your discussion should treat them
differently. Your comparison table in the report is built from this list.

| Published quantity | As the paper reports it |
|---|---|
| Analytic N | |
| Group sizes | |
| Descriptives (M, SD, or counts) | |
| Test statistic | |
| df | |
| p-value | |
| Effect size | |
| Confidence interval | |

## Assumptions we will verify
[Which assumptions this test carries, and how you will check each one from
the design and the data — not from the paper's say-so.]

## Known ambiguities before we start
[Anything the paper leaves unclear that you expect to matter. An exclusion
it does not spell out, a variable it names differently from the file, a
missing-data rule it never states.]

## Appended notes
[Date-stamped, append-only. Record what you learned as you went —
especially anything that turned out to differ from the plan.]

If the paper prints only t, df, and p for your target, with no group sizes or descriptives, record those and write “not reported” for the rest. The df can sometimes tell you the sample size, but only for some tests; Common sources of disagreement explains when.

Do not delete from this file. A plan that changed, with the change recorded, is evidence of careful work. A plan that matches your results perfectly because you rewrote it is not evidence of anything.

You do not set your own bar for “reproduced”

The plan records what you intend to reconstruct — the sample, the exclusions, the variables, the test, the numbers you will compare against. It does not define what counts as success. Every team is held to the same three-way standard — near-exact, partial, or not reproduced — set out in Original versus reproduced result below, so that verdicts mean the same thing across the class.

In particular, do not make your bar “the p-value lands on the same side of .05.” It is a tempting criterion, and a weak one. Two analyses that differ materially can land on the same side of .05, and two nearly identical analyses can land on opposite sides of it when p is near the threshold — so a significance match is weak evidence of agreement and a significance mismatch is weak evidence of disagreement. Compare the quantities: the analytic N, the descriptives, the statistic, and the effect size.

Your repository

Your team works in a new shared GitHub repository — not a copy of your Project 1 repo. Keep it private through grading. If your team later wants to make it public, check the data’s license, the article’s citation requirements, and the instructor’s guidance first — reproduction data often carries terms your own analysis does not. Starting fresh keeps the Git record readable for both projects. Download project-2-starter.zip and you begin with the structure, the .gitignore, and the documentation templates already in place.

Project setup and reproducibility is the walkthrough: the structure, publishing it, what each documentation file is for, and how to run the clean-clone test. It is the same guide you used for Project 1, so most of it will be familiar.

Project 2 adds a few files. documentation/reproduction-plan.md is the pre-analysis specification described above — the record that your analysis plan was fixed before you ran the target test, rather than after you saw the result. The starter also ships the citation setup from your M08 paper: documentation/references.bib holds the entries and programs/apa.csl formats them, so the report’s References section is generated from your @key citations rather than typed by hand. Nothing else about the structure changes.

The files in your starter, and when each one is done:

File Why it exists When you complete it
documentation/project-plan.md Your proposal: the scope check that gets the study approved Week 1
documentation/reproduction-plan.md Records your analytic decisions before you see your result Week 2
documentation/data-provenance.md Records exactly where the shared data came from The day you obtain the file
documentation/codebook.csv Maps the paper’s constructs to your analytic variables Weeks 2–3
README.md Tells another analyst how to reproduce your work Finish as you go
programs/p2_reproduction.qmd The reproduction report Weeks 2–4

Each file opens with its prompts: open it and answer each one. Where this page discusses a file, its template is shown in a folded box.

Project 2 uses the same repository habits as Project 1, but now you are expected to carry them out more independently. The project goes much more smoothly if you keep these seven habits in view from the start.

1 · Start with a clear target. Identify the exact published result you intend to reproduce before doing the analysis.

2 · Preserve the raw data. The original shared file goes in data/raw/ and stays exactly as the authors published it.

Every change to the data happens in code — no hand edits, and nothing edited in SPSS or Excel. If a value needs recoding, a missing-value code needs converting, a variable needs collapsing into groups, or a column needs renaming, that change is written as a line of R.

This one is worth explaining rather than simply asserting, because these files open straight into SPSS’s data editor or a spreadsheet, where clicking a cell and typing is genuinely the faster-feeling option in the moment. The trouble is that a hand edit exists only on the laptop where it happened. Your teammates re-download the file and get something different. The change leaves no trace of what it was or why you made it, so a reader — including you, three weeks later — cannot reconstruct it. And if you ever need to redo the analysis on a corrected file, every hand edit has to be remembered and repeated from scratch.

Worse, if you save, you have written over the pristine original that everything else depends on — and that is easier to do by accident in SPSS than in a spreadsheet, because the file it opens is your raw .sav. Reading a file to look at it is fine; the line is saving anything back.

A line of R costs about the same effort and gives all of that back: everyone gets the identical dataset, the reasoning sits right there in the code, and re-running it is one click. So when you hit something that needs fixing, the answer is always a line of code — and if you are not sure how to write it, that is exactly the kind of question to bring to lab or office hours.

This matters even more in a reproduction than it did in Project 1. Your claim is that you rebuilt someone else’s analytic sample from their published data — and that claim only means something if every step between their file and your numbers is written down and re-runnable. A hand edit breaks the chain at exactly the point a careful reader would want to inspect.

3 · Document provenance. A reader should be able to find the article, repository, exact data file, version, and access information.

4 · Document the analytic variables. Your documentation/codebook.csv traces the variables used in your reproduction back to their raw source.

See the Analytic Codebook Guide.

5 · Make the analysis reproducible. Use project-relative paths and complete code. Do not rely on an object created manually in the Environment or a result copied from the Console.

6 · Use Git as a record of the work. Commit meaningful increments throughout the project, and keep data or credentials out of the repository when redistribution is not allowed.

7 · Test the project from a clean start. A teammate who was not the primary data-prep person must be able to clone the repository, follow the README, place the source data in the documented location, restart R, and render the reproduction report from beginning to end.

Two points from the guide matter more here than they did in Project 1:

Raw data stays raw — and here it is someone else’s data. The authors’ file lives in data/raw/ exactly as they published it. Every step from their file to your numbers is written as code, because that chain is your reproduction claim. A hand edit breaks it at precisely the point a careful reader would want to inspect.

The clean-clone test is the reproduction of your reproduction. If a teammate cannot clone the repo, follow the README, obtain the data, and re-run your analysis to the same numbers, then you have not demonstrated what the project asks you to demonstrate. Run it with time left to fix what it finds.

documentation/data-provenance.md · where the authors’ data came from

The setup guide explains what provenance is for. Project 2’s version asks for different things than Project 1’s, because you are documenting someone else’s published data rather than a survey you downloaded:

documentation/data-provenance.md
# Data provenance

## Article
[Full citation]
DOI: [article DOI]

## Data repository
[OSF / Dataverse / figshare / journal supplement / other]
URL: [direct link to the data page, not the article page]
DOI: [data DOI, if the repository issues one separately]

## Version or release
[Repositories often host several versions. Name the one you used.
If there is only one, say so explicitly.]

## Date accessed
[YYYY-MM-DD]

## Downloaded filename(s)
[Exactly as they arrived, before any renaming]

## Supporting documentation used
[Codebook, README, analysis script, supplement — whatever you relied on
to interpret the variables. Note if the authors provided none.]

## Access conditions
[Open? Registration required? Data-use agreement?]

## Expected local path
data/raw/[filename]

## Redistribution
[May this file be shared? Many repositories permit it under a license;
some do not. Name the license if there is one.]

## Notes
[Anything a future reader needs: a file that differs from what the paper
describes, a variable the codebook does not mention, an ambiguity you
had to resolve by judgment.]

Fill this in the day you obtain the file. The access date and the exact filename are the two things people forget, and they are the two that make a dataset findable again — which a reproduction depends on.

data/README.md · the one tracked file in data/

Your starter ships this, and it is worth a look because it is the only thing inside data/ that a teammate sees after cloning — everything else is ignored. It differs slightly from Project 1’s, because it points at this project’s report and at someone else’s data:

data/README.md
# Data

The data files themselves are **not** tracked by Git. This file is.

## data/raw/
The authors' file exactly as downloaded, never edited by hand.

Expected file: `[filename]`

To obtain it, follow `documentation/data-provenance.md`.

## data/derived/
Our reconstruction of the authors' analytic sample, built from the raw
file by `programs/p2_reproduction.qmd`. Do not edit by hand.

Expected file: `analytic.rds`

## If you just cloned this repository
Both folders will be empty. Obtain the raw file, place it in
`data/raw/`, then render `programs/p2_reproduction.qmd` — it rebuilds
`data/derived/` for you.

The reproduction report

Read an example first

An annotated example report walks through a short, deliberately competent reproduction with notes on what would raise each section. A reproduction report is a genre almost nobody has written before, and the example shows what “characterize the discrepancy rather than smooth it over” actually looks like in prose.

Aim for approximately 1,500–2,000 words, not counting code and tables.

The report deliberately follows the logic of M09. If you are unsure what belongs in a section, go back to the way M09 asked you to move from design, to estimand, to hypotheses, to test, to interpretation. The clearest reports usually feel like a guided reconstruction: here is the claim, here is the sample, here is the test, here is what matched, and here is what did not.

The report has five sections. The starter’s report file uses the familiar APA headings, so here is where each section goes:

Report section In the starter’s programs/p2_reproduction.qmd
1 · The published study and target Introduction; Methods › The original study
2 · Data and analytic-sample reconstruction Methods › Our reproduction; The codebook; Results › Descriptive summary
3 · Reproduction analysis Results › Visualization, The reproduction test, APA-formatted result
4 · Original versus reproduced result Results › Original versus reproduced
5 · Discussion and scope Discussion

1 · The published study and target

Open by orienting a reader who has not read the paper. Cover, briefly:

  • The research question. What did the authors want to know, in one sentence of plain English?
  • Why it matters. Place the finding in its research area: what it contributed, and why a reader in that area would care whether it holds up.
  • The study design. How was it run — an experiment or an observational study, between- or within-subjects, how many groups or measurements, and how participants came to be in the conditions they were in.
  • The variables. Name the outcome and the grouping or predictor variable as the paper describes them, with their units or levels. If the paper’s name for a variable differs from the column name in the data file, say both — that mapping is the thing your codebook records, and readers will need it to follow your code.
  • Who was studied. The target sample and the analytic sample the paper reports.
  • The exact result you are reproducing. The specific sentence, table row, or figure value.

Cite the original paper with a @key citation — the same automatic-references workflow as your M08 paper. The starter ships references.bib and the APA style file already wired up: insert citations from the visual editor (Insert → Citation → From DOI) and the References section at the end of the report builds itself from what you cite. Nothing in that section is typed by hand.

A reader should be able to finish this section and, without having opened the paper, state what was measured on whom and what the authors concluded. Everything after this assumes they can.

The design, the estimand, and the hypotheses

Before naming the test, identify:

  • unit of observation,
  • outcome type,
  • number of groups or measurements,
  • whether observations are independent or paired,
  • and the estimand: the population quantity the question targets (M09 names it in words before symbols).

Then explain why the published test matches that design.

State:

  • the estimand, in words and then in symbols,
  • \(H_0\),
  • \(H_a\),
  • and whether the original analysis was one- or two-sided if the paper specifies it.

Use the M09 convention of explaining the hypotheses in words as well as symbols.

2 · Data and analytic-sample reconstruction

Describe:

  • where you obtained the data,
  • the raw file,
  • the variables needed,
  • exclusions,
  • missing-data handling,
  • recodes or derived variables,
  • and final analytic N.

The code performing these steps belongs in the .qmd.

Reconcile the descriptives before the test

Do this before the test. If the group sizes or means do not match the paper, the disagreement is in the sample rather than the analysis — and that is a much easier problem to diagnose now than after a mismatched p-value. The tools are the M05 module’s.

This step is worth taking seriously. Teams often want to rush to the target NHST, but the descriptives are what tell you whether you are even analyzing the same sample the authors did.

Create a comparison such as:

Quantity Published Reproduced
Analytic N
Group n
Mean / proportion
SD / counts

Then ask:

Do our descriptives match closely enough that we appear to have reconstructed the same analytic sample?

If not, stop and investigate before interpreting the hypothesis test.

3 · Reproduction analysis

The results figure · required

Build one ggplot2 figure showing the effect you are reproducing. This is a graded deliverable, not a decoration, and it is the thing you will put on screen during your talk.

Build one ggplot2 figure that makes the focal effect or association visible in the raw or minimally summarized data. What that looks like depends on your design:

Your target test A figure that shows the effect
Two-sample t or one-way ANOVA Group means or medians with a display of uncertainty, and — where the n allows it — the individual observations underneath
Paired t Paired trajectories linking each unit’s two measurements, or the distribution of the difference scores
One-sample t The distribution of the outcome with the benchmark value marked, plus the mean and its interval
Chi-square (either kind) Counts or proportions by category, with the denominator unmistakable — say what each percentage is a percentage of

The M09 lab is the worked model for the first row: points jittered under group means with geom_errorbar() for the interval. If your design sits in another row, the principle is the same — show the data the conclusion rests on, not a picture of the test itself.

Hold it to the M03 standard, because a reproduction figure is read by someone deciding whether to believe you:

  • title states the takeaway, not the variables — “Treatment scores exceeded control by 4.2 points”, not “Score by condition”.
  • subtitle carries scope: n, the year or wave, any transformation.
  • Axis labels are human-readable with units — “Aggression score (0–10)”, never agg_score.
  • caption carries provenance: "Source: " plus the authors, year, and the file you analyzed.
  • theme_minimal(), and color used deliberately or not at all.

The test your figure has to pass

Show it to someone who has not read your report. Can they say what was compared, roughly how big the difference is, and how much uncertainty surrounds the estimate you are reporting — without you narrating?

If they cannot, the figure is not finished. That is the same standard M03 set with the Rosling chart, applied to your own result.

Assess the design and model

The assumptions behind each test, and what to do when one is questionable, are in the M09 module. For the confidence-interval half of the reporting, see M07; for what a p-value does and does not license, M08.

Following M09, distinguish:

Design assumptions

  • independence or pairing,
  • unit of analysis,
  • treatment/group formation,
  • other design facts that cannot be diagnosed from a histogram.

Model considerations

  • whether the mean is a sensible summary,
  • influential/extreme observations,
  • adequacy of the sampling approximation,
  • variance considerations where relevant,
  • expected counts for chi-square.

Rather than running through every assumption check by habit, discuss the considerations that actually matter for your chosen test.

Run the test

Run the M09 analysis and report:

  • raw effect,
  • 95% CI where applicable,
  • test statistic,
  • degrees of freedom,
  • p-value,
  • course-standard effect size,
  • and effect-size CI when your M09 workflow provides one.

Examples of course-standard effect sizes:

Test Effect size
One-sample t Cohen’s d
Two-sample t Cohen’s d
One-way ANOVA \(\eta^2\)
Chi-square Cramér’s V
Paired-samples t Cohen’s \(d_z\)

One choice the table hides: which version of the two-sample t, and which SD standardizes Cohen’s d. M09 runs the two-sample t both ways, as the M09 pre-study’s Activity 2.3 does: the standard (equal-variance) version, with var.equal = TRUE, and Welch’s version, R’s default, which lets each group keep its own spread. In a reproduction, run both, then report the one the authors ran. The df usually tells you which one that is (see Common sources of disagreement). Each version has its own standardizer for Cohen’s d: the pooled SD beside the standard test, which is what cohens_d() uses by default, and the unpooled SD beside Welch’s.

# The standard two-sample t: one pooled SD for both groups
my_data |>
  t_test(formula = outcome ~ group, order = c("Treatment", "Control"), var.equal = TRUE)

# Welch's version, R's default: each group keeps its own spread
my_data |>
  t_test(formula = outcome ~ group, order = c("Treatment", "Control"))

# Cohen's d to match each one. First put the group you named first in `order`
# first in the factor too, so d subtracts the same way t does (see the warning below).
my_data |>
  mutate(group = fct_relevel(group, "Treatment")) |>
  cohens_d(outcome ~ group, data = _)                      # pooled: beside the standard test

my_data |>
  mutate(group = fct_relevel(group, "Treatment")) |>
  cohens_d(outcome ~ group, data = _, pooled_sd = FALSE)   # unpooled: beside Welch's

The two versions often agree closely, and so do the two ds, so a mismatch with the paper is rarely explained by this choice alone. It sometimes is, though, which is why running both is worth the extra line. Whatever you report, name the version and the standardizer, because a reader cannot tell them apart from the numbers.

The sign trap: t and d do not agree by default

t_test() takes the direction of the comparison from its order argument. cohens_d() ignores that entirely and subtracts in factor-level order — so the same comparison can come back as t = +9.21 with d = −2.38. The magnitude is right and only the sign is wrong, which is exactly why it survives into an APA sentence unnoticed.

Make the group you named first in order the factor’s first level before calling cohens_d(). fct_relevel(), as in the code above, does that whatever order the levels started in. The M09 lab uses fct_rev() instead, because it had set its levels baseline-first on purpose. Reversing levels that are already in the order you want flips a correct d to the wrong sign, which is why stating the order directly is safer. Then check that your t and your d carry the same sign before you write either one down.

If the original paper did not report the course-standard effect size, label it clearly as an additional result calculated for this course, not something you are claiming the authors reported.

Write the numbers with inline R

Every number your prose reports — the analytic N, the means, the test statistic, df, p, the effect size and its CI — comes from your code through inline R, not typed. It is the same requirement as Project 1, now with the APA formatting from your M08 paper. One chunk pulls each number from your results and formats it once (for example, p without a leading zero, and written as p < .001 when it is that small), and your sentence calls those names. If you later fix an exclusion and re-render, every number in the sentence updates. The Write it with inline R box in section f of the M09 lab is the full worked template, and Writing with Code collects the syntax. The one exception is the paper’s own numbers: copy those exactly as printed into your comparison table, because they are what you are checking against.

4 · Original versus reproduced result

Use one comparison table:

Quantity Original paper Our reproduction Match?
Analytic N
Raw effect / key descriptives
Test statistic
df
p-value
Effect size, if reported

Then classify the outcome:

Near-exact reproduction

The analytic N, the key descriptives, the focal effect and statistic, and the inferential result agree within ordinary reporting or rounding differences.

Partial reproduction

The reconstruction recovers an important part of the published result, but one or more key numerical quantities differ beyond ordinary rounding. The discrepancy is described and, where possible, traced to an analytic or documentation difference.

Not reproduced

The focal numerical result differs materially, or cannot be recovered, after careful reconstruction, and the discrepancy remains unresolved.

Your grade does not depend on the category.

It depends on whether your reconstruction and diagnosis are transparent and defensible.

5 · Discussion and scope

Discuss:

  • what reproduced or failed to reproduce,
  • the most likely reason for any discrepancy,
  • what you learned from reconstructing the analysis,
  • and what the design permits you to conclude.

Remember an important M09 principle:

The statistical test does not create the study’s causal or generalizability warrant. Those claims come from the design.

Common sources of disagreement

If your result does not match, do not assume the project has failed. Start by investigating possibilities such as:

  • wrong study or dataset file,
  • different analytic N,
  • an exclusion you missed,
  • missing-data handling,
  • different coding or category reference,
  • different threshold for a derived categorical variable,
  • the standard (Student’s) versus Welch two-sample t. The reported df is a useful clue when the paper doesn’t say which it ran. The standard test always has \(df = n_1 + n_2 - 2\). If the reported df clearly differs from that, and especially if it isn’t a whole number, the analysis was almost certainly Welch’s: a paper reporting df = 861.22 for two groups of 419 and 486 has told you it used Welch’s without writing the word. If the df equals \(n_1 + n_2 - 2\), the standard test is plausible but not proven, because Welch’s df can match or round to that value when the two groups’ spreads are similar. Check the Methods, the authors’ code, or their software’s defaults. One trap: only the standard test’s df converts to a sample size (\(N = df + 2\)). A Welch df is not a head count, so don’t read an N off it; compare it with the Welch df your own data produce instead,
  • one- versus two-sided test,
  • ANOVA subset or follow-up differing from the paper,
  • weighting,
  • rounding,
  • software defaults,
  • undocumented author decisions,
  • an error in shared code or the published report,
  • or an implementation error in your own analysis.

Because a reproduction uses the same underlying data, a mismatch is not explained by “sampling variability from a new sample.”

The presentation · 30 minutes

Every team member must speak and be prepared to answer questions about the full project.

Format · remote, and the choice of medium is yours

Presentations are delivered remotely over Microsoft Teams, across the two class meetings of presentation week (December 7–11) — two teams on Monday, two on Wednesday. You will know your slot in advance. One person shares their screen; you decide how to hand off between speakers.

How you present the material is up to you. Slides are one option, not a requirement. A rendered Quarto report scrolled through deliberately, a live notebook, or a small set of figures shown full-screen can all work — often better than slides, since your figures already exist and re-making them as slide images only invites version drift.

Whatever you choose, two things matter more than the medium:

  • Everything is legible at screen-share resolution. Test it by sharing to a teammate first — axis labels and table text that read fine on your own monitor frequently do not survive Teams compression.
  • You can move to a specific figure on request. Questions will not arrive in your running order, so know how to jump to Figure 2 without scrolling past everything else.

Have one person hold a backup copy of whatever you are presenting, in case the presenter’s connection drops.

Part 1 · The published claim · about 5 minutes

Your audience has not read this paper. Orient them first:

  • The research question, in one sentence.
  • Why it matters in its research area, and why your team chose it.
  • The study design — experiment or observational, between- or within-subjects, how many groups or measurements.
  • The variables — the outcome and the grouping variable, with units or levels.
  • Who was studied, and the analytic sample the paper reports.
  • The exact result you targeted, its estimand, and why it was an eligible M09 analysis.

Five minutes is enough for this only if you have decided in advance what to leave out. Practice it.

Part 2 · Reconstructing the analysis · about 4 minutes

  • What data did you obtain?
  • What variables mapped to the paper’s constructs?
  • What exclusions/missing-data rules mattered?
  • Did the analytic N and descriptives match?

Part 3 · Did it reproduce? · about 7 minutes

Put your results figure on screen for this part and talk from it rather than from a table of numbers.

Show:

  • the raw effect figure,
  • original versus reproduced numbers,
  • test statistic and uncertainty,
  • effect size,
  • and your reproduction classification.

Part 4 · Behind the reproduction · about 4 minutes

Put your code on screen and walk us through a piece of it. Choose something that took work — a wrangling step, the analytic-sample reconstruction, the test itself — and talk us through what it does and why you wrote it that way. Expect follow-up questions about your choices: why that exclusion, why that join, what happened the first time it did not work.

This is the part of the talk where writing the code yourself matters most. Anyone can read code aloud; explaining why it is shaped the way it is only works if you shaped it.

Then teach one transferable lesson:

  • how you mapped the design to the right test,
  • how you found an undocumented exclusion,
  • how the codebook crosswalk helped,
  • why two apparently equivalent analysis choices produced different results,
  • or another reproducibility lesson.

Part 5 · Questions and discussion · about 10 minutes

Ten minutes, and they are not a formality — this is where you get to show what your team understood, where the reconstruction was difficult, and how you reasoned through it.

Everyone should expect to answer something. Questions come from the class first, then from us, and they will be aimed at whoever did the part being asked about. Reasonable things to be ready for: why did you exclude those cases? · what would change if you had kept them? · why that test and not the obvious alternative? · what did you get wrong the first time?

That last one is worth preparing deliberately. “Here is what we got wrong and how we found it” is one of the most useful things a team can give the rest of the class, and it is strong evidence that you did the work.

What to submit

Your repository is the submission. There is no separate file to upload — we grade the GitHub repo directly, at the state it is in when the deadline passes.

Add both instructors as collaborators by the end of the project’s first week. That is well before the final deadline on purpose: we read your project plan during that week, and a private repository we have not been invited to is one we cannot open. On github.com, open your repository → Settings → Collaborators → Add people, and add KimberlyHenry (instructor) and alliekom (lab instructor). Type the usernames exactly rather than searching by name — GitHub’s search will happily offer you someone else entirely.

Week 1, not the last day

Your repository is private, which is correct — but it also means that until the invitations are accepted, nobody but your team can see a thing. That is why this belongs in the first week rather than the last: your project plan is due then and we read it then, so an uninvited repository is an unread plan.

Add both of us as soon as the repository exists — KimberlyHenry (instructor) and alliekom (lab instructor) — and confirm in Settings → Collaborators that each appears.

Adding us early costs you nothing. We read the plan in the first week, and we look again when the deadline passes; nobody is watching your commits in between.

Two consequences of grading the repo rather than a file, both in your favour once you know them:

  • The commit history is visible, and that is the point. It is the evidence that the work was shared. A single last-minute commit cannot demonstrate the ongoing, shared workflow this part of the project is designed to assess.
  • What is not committed does not exist. Anything sitting only on someone’s laptop — a figure, a fixed typo, the final render — is not part of what we see. Push before the deadline, then check the repository on github.com in a browser and confirm what you expect is actually there.

Data files excluded by your .gitignore are meant to be absent; that is the design, not an omission. data/README.md and documentation/data-provenance.md are what tell us how to obtain them.

Final checklist

Before submitting:

Analysis

Report

Repository

Presentation

Then read the Project Rubric one more time with your finished repository open beside it. Points are often lost to something the rubric names plainly that no one checked at the end.