Skip to contents

Folder replication checklist

Source: vignettes/folder-replication-checklist.Rmd
folder-replication-checklist.Rmd

See vignette("why-replicateEverything", package = "replicateEverything") for motivation and a tour of the consumer API.

Two models

Model Registry Materials
Folder-backed studies/<folder>.yml stub only Study repo: code/, data/, outputs/
Package-backed studies/<folder>.yml stub only Study R package on GitHub

This checklist covers folder-backed studies (simple Git repo, not an R package).

Study repo layout 

rep-<doi-slug>/
  replication.yml
  README.md
  code/                 # one script per step
  data/                 # optional raw inputs
  outputs/              # step outputs + display files + manifest.json
  tests/
    testthat.R
    testthat/
      test-<id>.R
    substantive/          # published-value benchmarks (recommended)
      <step_id>.R

There is no study-local registry/ folder. A maintainer with a local registry checkout writes studies/<folder>.yml directly from this repo’s root replication.yml via sync_study_to_registry() (step 4) — nothing is generated or committed here.

Required replication.yml fields

Paper metadata

  • paper.doi — full DOI URL
  • paper.title
  • paper.article_url — optional publisher landing page when https://doi.org/... does not resolve (common for older Cambridge Core / APSR entries)
  • repo or paper.study_repo — GitHub slug (org/repo); inferred from folder name when it starts with rep-

Maintainer and collections (required for registry sync)

  • maintainer.name and maintainer.email — person responsible for keeping the study repo and server dependencies current; shown as [maintainer] on the Studies tab
  • collections — one or more tags for bibliography filtering (APSR, PED, World Bank, IPI, …); copied to registry index.csv
  • languages — every engine used in steps (r, stata, python); precompiled into index.csv for the Studies tab
maintainer:
  name: Jane Maintainer
  email: maintainer@example.org

collections:
  - APSR

languages:
  - r
  - stata

Steps (required)

Declare a single steps: DAG — yaml is the authority for execution order. Legacy prep: / replications: blocks are a hard error; there is no compatibility shim.

steps:
  - id: prep_data
    type: transform
    parents: []
    inputs: [data/myfile.dta]
    outputs: [outputs/prep_data/analysis.rds]
    engine: r
    code: code/steps/prep_data.R

  - id: fig_1
    type: figure
    parents: [prep_data]
    data: outputs/prep_data/analysis.rds
    code: code/fig_1.R
    outputs: [outputs/fig_1.png]

  - id: fig_1_format
    type: format
    parent: fig_1
    code: code/format_fig_1.R
    outputs: [outputs/fig_1_display.png]
    # no label: — format children are sidebar-hidden wiring

  - id: tab_1
    type: table
    parents: [prep_data]
    code: code/tab_1.R
    outputs: [outputs/tab_1.html]

Format steps (type: format) run when run_replication(..., format = TRUE). Omit label: on format children (unused in Display).

Blocked / incomplete steps

When a step cannot be produced here (missing Mathematica, proprietary data, etc.):

- id: tab_restricted
  type: table
  incomplete: true
  data_unavailable: proprietary   # or requires_engine: mathematica
  blocked_reason: "Proprietary microdata not in the deposit."
  # code: / outputs: optional until unblocked

incomplete: true excludes the step from baking and from audit_everything() (neither success nor failure). Prefer structured requires_engine: / data_unavailable: so Shiny can show the right icon and partial-replication popup. Put a step in steps: only if it is a replication claim; blocked prep off the claim path stays in README / popup — not as orphan Unavailable nodes. Recover DAGs at wrapper granularity from author README tables — especially OpenICPSR AER packages. See Contributing principles.

Data patterns (short)

  • Pattern B (default): surgical Dataverse file-id access → outputs/.
  • Pattern A: materialize → data/ only when fetch is not a claimed step.
  • OpenICPSR: usually commit needed inputs only (no public file API); keep the full unzip under monorepo original_studies/.

Workflow from the study repo

1. Build display artifacts

Run from the study repository root (or pass its path):

library(replicateEverything)

options(
  replicateEverything.registry_root = "../registry",
  replicateEverything.use_sibling_packages = TRUE
)

build_study_outputs(
  location = ".",
  install_deps = TRUE
)

This runs every table and figure, saves display files under outputs/, and writes outputs/manifest.json.

Confirm outputs exist before handoff:

validate_outputs(location = ".")

2. Add tests

Each tests/testthat/test-<id>.R should:

  • call replicateEverything::run_replication(doi, id) (analysis object)
  • call run_replication(..., format = TRUE) when a format child step exists
  • compare formatted output to the committed file under outputs/ (HTML normalize or PNG md5)

Run:

testthat::test_dir("tests/testthat")

3. Validate (contributor)

Before running the full checklist, a quick manual smoke check from the study repo root confirms the study resolves and runs without any registry setup — pass "local" (the working-directory study) instead of a DOI:

library(replicateEverything)

list_replications("local")        # what does this study expose?
describe_study_dag("local")       # sanity-check the parsed step DAG
run_replication("local", "tab_1") # one light step end-to-end

Then run the full contributor checklist:

check_and_bake_study(
  location = ".",
  build_artifacts = FALSE   # already built above
)

This runs [check_replication()] against your study root replication.yml. It validates only — it does not write a registry stub or any handoff file into the study repo. Commit your study once checks pass; a registry maintainer writes the stub directly from your yaml (step 4).

4. Sync to the central registry (maintainer)

Maintainers work from a local registry checkout:

options(replicateEverything.registry_root = "../registry")

sync_study_to_registry(
  "../rep-10.1177-00491241211036161",
  registry_root = "../registry"
)

This builds a lightweight stub straight from the study repo’s root replication.yml (maintainer, collections, languages, and repo pointers — never the full steps: list), writes it to registry/studies/<folder>.yml, and runs [build_registry_index()]. register_study(".", registry_root = "../registry") runs check_and_bake_study() then sync_study_to_registry() in one call.

After syncing several studies, refresh the whole registry and rerun the audit:

refresh_registry("../registry", audit = TRUE, patience = 20)
testthat::test_dir("tests/testthat")

Run tests before check_and_bake_study() so the checklist reflects a merge-ready study.

Substantive (published-value) checks

Smoke tests confirm that run_replication() returns an object. Substantive checks go further: they compare replicated estimates to numbers in the published table (coefficients, standard errors, sample sizes, etc.).

Submitters: add tests/substantive/<step_id>.R for each table or figure where you can quote benchmarks from the paper. Reference implementation: Fearon & Laitin tab_1.

# tests/substantive/tab_1.R — see Fearon & Laitin reference implementation
substantive_check_tab_1 <- function(object) {
  check_glm_table_benchmark(object, tab_1_prior_war_benchmark())
}

check_glm_table_benchmark() is injected when the package loads substantive check scripts (via [audit_everything()] or [check_replication()]). Study-specific benchmark values belong in the study repo, not the package.

Call the same function from tests/testthat/test-<step_id>.R so testthat::test_dir() exercises it locally.

Maintainers: [check_replication()] reports substantive coverage (substantive_<step_id> rows). Use full_replication = TRUE to run defined checks. [audit_everything()] runs them registry-wide by default (substantive = TRUE); failures appear as [substantive] in the audit summary.

Maintainer shortcut 

options(replicateEverything.registry_root = "../registry")
sync_study_to_registry("../rep-10.1177-00491241211036161", audit = TRUE)

Reference implementations

Package-backed studies

For studies maintained as R packages, see vignette("package-replication-checklist", package = "replicateEverything") and check_replication().