Introduction to Preregistration
Open Science in the research process
Threats to research validity …
p-hacking & friends
| Threat | Remedy |
|---|---|
| Switching of primary outcome to a variable that “works better” | Publicly write down the primary outcome before data collection |
| HARKing: “Hypothesizing after results are known” | Publicly write down hypotheses before data collection |
| Flexibility in data analysis (e.g., trying out different outlier exclusions to see which “work best”) | Specify exclusion rules and the full analysis plan before seeing the data |
Threats to research validity …
Biases
| Threat label | Explanation | Remedy |
|---|---|---|
| Hindsight bias | Tendency to frame previous decisions or events as more predictable after the outcome is known (“I knew it all along”) | Write down your hypotheses and analysis plans before you see the data |
| Confirmation bias | Tendency to seek out, interpret, favor and recall information in a way that supports one’s prior expectations | Write down your decision rules and analysis plan before you see the data |
Preregistration is the specification of …
- the research design (including sample size planning),
- operationalizations,
- hypotheses,
- analysis plan, and
- inferential criteria
… prior to observing the outcomes of a study.
Distinguishing confirmatory from exploratory analyses
Confirmatory analysis
- Number of hypothesis tests: Known.
- Error rate control: Possible.
- p-value: Meaningful.
Exploratory analysis
- Number of hypothesis tests: Unknown.
- Error rate control: Impossible.
- p-value: “essentially uninterpretable”.
Convince others that you did not engage in QRPs
How can you convince a skeptical reader?
- Publish your preregistration openly
- Use a third-party platform that freezes the file (so that you cannot edit it later) with a timestamp
- Nullius in verba: The less trust is necessary, the more convincing.
Discuss: What challenge does a secondary data analysis (from an existing dataset) pose?
Why do we preregister?
Summary
- Don’t do QRPs: Safeguard ourselves from (unconscious) biases and QRPs
- Convince others that you did no QRPs: Build your reputation (pre-diction instead of post-diction; transparency)
- Restrict your degrees of freedom: Clearly distinguish confirmatory from exploratory results
- Intended side-effect: We think more clearly about our design and data collection before we start collecting data → quality improvements.
Registered Report
Elements of a preregistration
Elements of a preregistration
1. Hypotheses
✅ Essential:
- Describe hypotheses as relationships between variables
- Describe shape of interaction effects
- Describe manipulation checks (or why they are not included)
☑️ Recommended:
- Figures / tables to describe interaction effects
- Rationales / theoretical frameworks to justify the hypotheses
Elements of a preregistration
2. Operationalizations
✅ Essential:
- List, based on your hypotheses,
- Independent variables (describe variable, all levels, between- or within-person?)
- Dependent variables
- Third variables (covariates, moderators, control variables etc.)
Elements of a preregistration
3. Planned sample
✅ Essential:
- Pre-selection rules (e.g., age limits)
- Where, from whom, and how will the data be collected?
- Justify planned sample size (power analysis or Bayesia design analysis)
- Describe data collection termination rule
van t‘Veer & Giner-Sorolla (2016), Schönbrodt & Wagenmakers (2017)
Elements of a preregistration
4. Exclusion criteria
✅ Essential:
- Describe all anticipated exclusion criteria, e.g.
- Missing, erroneous, overly consistent responses
- Failing check-tests or suspicion probes
- Demographic exclusions
- Data-based outlier criteria
- Method-based outlier criteria (e.g., too long response times)
☑️ Recommended:
- Set fail-save levels of exclusion at which whole study needs to be stopped, altered, and restarted
Elements of a preregistration
5. Analysis plan
✅ Essential:
- Describe statistical analyses that test hypotheses. For each, include
- Relevant variables and how they are calculated
- Statistical technique
- Each variable‘s role in the technique (e.g., IV, DV, covariate, …)
- If covariates are used: Rationale for using them
- Describe inferential criteria (i.e., when will you accept or reject a hypothesis?). This is typically the \(\alpha\) level and whether you test one- or two-sided.
Elements of a preregistration
5. Analysis plan
☑️ Recommended:
- Specify contingencies and assumptions, such as
- Method of correcting for multiple testing
- Method of missing data handling
- Anticipated data transformations
- Assumptions and assumption checks for analyses and alternative plans for data analysis if assumptions are not met
Elements of a preregistration
6. Additional stuff
☑️ Recommended:
- For exploratory analyses: “We don‘t have any hypotheses”
- Transparency statement: How will research output be shared?
- Conditional safeguards: What will happen if…?
Preregistration templates
Recommended:
- The Preregistration for Quantitative Research in Psychology Template (PRP-QUANT) was developed in a joint effort by a task force composed of members of the APA, BPS, DGPs, the COS, and ZPID.
- https://prereg-psych.org/index.php/rrp/templates
Preregistration templates (alternatives)
A curated list of high quality preregistrations
Preregistration platforms
TODO
End
Contact
- @nicebread@scicomm.xyz
- ed.uml.ysp@tdorbneohcs.xilef
- https://www.nicebread.de
- https://github.com/nicebread
