Introduction to Preregistration

Sara Lil Middleton
Sara.Middleton@psy.lmu.de

LMU Open Science Center

Felix Schönbrodt
felix.schoenbrodt@psy.lmu.de

Ludwig-Maximilians-Universität München

2026-05-22

Open Science in the research process


get all material here: https://osf.io/zjrhu/

Threats to research validity …

p-hacking & friends

Threat Remedy
Switching of primary outcome to a variable that “works better” Publicly write down the primary outcome before data collection
HARKing: “Hypothesizing after results are known” Publicly write down hypotheses before data collection
Flexibility in data analysis (e.g., trying out different outlier exclusions to see which “work best”) Specify exclusion rules and the full analysis plan before seeing the data

Threats to research validity …

Biases

Threat label Explanation Remedy
Hindsight bias Tendency to frame previous decisions or events as more predictable after the outcome is known (“I knew it all along”) Write down your hypotheses and analysis plans before you see the data
Confirmation bias Tendency to seek out, interpret, favor and recall information in a way that supports one’s prior expectations Write down your decision rules and analysis plan before you see the data

Preregistration is the specification of …

  • the research design (including sample size planning),
  • operationalizations,
  • hypotheses,
  • analysis plan, and
  • inferential criteria

… prior to observing the outcomes of a study.

adapted from Nosek & Lindsay (2018), PRP-QUANT template, and RESQUE.

Distinguishing confirmatory from exploratory analyses

Confirmatory analysis

  • Number of hypothesis tests: Known.
  • Error rate control: Possible.
  • p-value: Meaningful.

Exploratory analysis

  • Number of hypothesis tests: Unknown.
  • Error rate control: Impossible.
  • p-value: “essentially uninterpretable”.

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p -Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108

Convince others that you did not engage in QRPs

How can you convince a skeptical reader?

  • Publish your preregistration openly
  • Use a third-party platform that freezes the file (so that you cannot edit it later) with a timestamp
  • Nullius in verba: The less trust is necessary, the more convincing.

Discuss: What challenge does a secondary data analysis (from an existing dataset) pose?

Why do we preregister?

Summary

  1. Don’t do QRPs: Safeguard ourselves from (unconscious) biases and QRPs
  2. Convince others that you did no QRPs: Build your reputation (pre-diction instead of post-diction; transparency)
  3. Restrict your degrees of freedom: Clearly distinguish confirmatory from exploratory results
  4. Intended side-effect: We think more clearly about our design and data collection before we start collecting data → quality improvements.

Registered Report

Elements of a preregistration

Elements of a preregistration

1. Hypotheses

✅ Essential:

  • Describe hypotheses as relationships between variables
  • Describe shape of interaction effects
  • Describe manipulation checks (or why they are not included)

☑️ Recommended:

  • Figures / tables to describe interaction effects
  • Rationales / theoretical frameworks to justify the hypotheses

van t‘Veer & Giner-Sorolla (2016)

Elements of a preregistration

2. Operationalizations

✅ Essential:

  • List, based on your hypotheses,
    • Independent variables (describe variable, all levels, between- or within-person?)
    • Dependent variables
    • Third variables (covariates, moderators, control variables etc.)

van t‘Veer & Giner-Sorolla (2016)

Elements of a preregistration

3. Planned sample

✅ Essential:

  • Pre-selection rules (e.g., age limits)
  • Where, from whom, and how will the data be collected?
  • Justify planned sample size (power analysis or Bayesia design analysis)
  • Describe data collection termination rule

van t‘Veer & Giner-Sorolla (2016), Schönbrodt & Wagenmakers (2017)

Elements of a preregistration

4. Exclusion criteria

✅ Essential:

  • Describe all anticipated exclusion criteria, e.g.
    • Missing, erroneous, overly consistent responses
    • Failing check-tests or suspicion probes
    • Demographic exclusions
    • Data-based outlier criteria
    • Method-based outlier criteria (e.g., too long response times)

☑️ Recommended:

  • Set fail-save levels of exclusion at which whole study needs to be stopped, altered, and restarted

van t‘Veer & Giner-Sorolla (2016)

Elements of a preregistration

5. Analysis plan

✅ Essential:

  • Describe statistical analyses that test hypotheses. For each, include
    • Relevant variables and how they are calculated
    • Statistical technique
    • Each variable‘s role in the technique (e.g., IV, DV, covariate, …)
    • If covariates are used: Rationale for using them
    • Describe inferential criteria (i.e., when will you accept or reject a hypothesis?). This is typically the \(\alpha\) level and whether you test one- or two-sided.

van t‘Veer & Giner-Sorolla (2016)

Elements of a preregistration

5. Analysis plan

☑️ Recommended:

  • Specify contingencies and assumptions, such as
    • Method of correcting for multiple testing
    • Method of missing data handling
    • Anticipated data transformations
    • Assumptions and assumption checks for analyses and alternative plans for data analysis if assumptions are not met

van t‘Veer & Giner-Sorolla (2016)

Elements of a preregistration

6. Additional stuff

☑️ Recommended:

  • For exploratory analyses: “We don‘t have any hypotheses”
  • Transparency statement: How will research output be shared?
  • Conditional safeguards: What will happen if…?

Lin & Green (2016)

Preregistration templates

Recommended:

  • The Preregistration for Quantitative Research in Psychology Template (PRP-QUANT) was developed in a joint effort by a task force composed of members of the APA, BPS, DGPs, the COS, and ZPID.
  • https://prereg-psych.org/index.php/rrp/templates

Preregistration templates (alternatives)

A curated list of high quality preregistrations

Preregistration platforms

TODO

End

Contact

CC-BY-SA 4.0