%%{ init: { 'flowchart': { 'curve': 'natural' } } }%%
flowchart LR
T(Theory)
P(Phenomena)
D(Data)
T -- "Explanation" --> P
P -- "Abduction" --> T
P -- "Prediction" --> D
D -- "Generalization" --> P
FOMO-Psy 1: Data, phenomena,
and the replication crisis
Data, phenomena,
and the replication crisis
The relation of data and phenomena
The relation of data and phenomena
%%{ init: { 'flowchart': { 'curve': 'natural' } } }%%
flowchart LR
P(Phenomena)
D(Data)
P -- "Prediction" --> D
D -- "Generalization" --> P
Phenomena: Stable and general features of the world in need of explanation. Can be understood as robust generalizations of patterns in empirical data. They are the explanatory targets for scientific theories. In psychology often called “effects” or “findings”.
Data: Relatively direct observations. Refer to particular empirical patterns in concrete data sets rather than empirical generalizations (which would be phenomenona).
The relation of data and phenomena
Generalization
%%{ init: { 'flowchart': { 'curve': 'natural' } } }%%
flowchart LR
P(Phenomena)
D(Data)
P -- "Prediction" --> D
D -- "Generalization" --> P
linkStyle 1 stroke-width:2px,stroke:red,color:red;
Data provide evidence for the existence of empirical phenomena: You generalize from one or more data sets with strong evidence to a general phenomenon.
Generalize to what? UTOS framework:
- Units (i.e., sample characteristics): To which other types of tested units does it generalize?
- Treatments: Does it generalize to other operationalizations of the independent variable/treatment?
- Outcomes: Does it generalize to other operationalizations of the dependent variable/outcome?
- Settings: Does it generalize to other settings, e.g. lab vs. field, other countries?
To claim a (robust) phenomenon, you ideally need:
- Independent replications with the same operationalizations (U, maybe S is varied): Often called “direct”, “exact”, or “close” replications
- Replications with different operationalizations (UTOS is varied): Often called “conceptual” replications
The relation of data and phenomena
%%{ init: { 'flowchart': { 'curve': 'natural' } } }%%
flowchart LR
P(Phenomena)
D(Data)
P -- "Prediction" --> D
D -- "Generalization" --> P
linkStyle 1 stroke-width:2px,stroke:red,color:red;
Probably most of psychological research is about establishing phenomena (disguised as “theories”).
Techniques used to detect data patterns:
- Factor analysis
- Principal components analysis
- Regression
- ANOVA
- …
The relation of data and phenomena
Prediction
%%{ init: { 'flowchart': { 'curve': 'natural' } } }%%
flowchart LR
P(Phenomena)
D(Data)
P -- "Prediction" --> D
D -- "Generalization" --> P
linkStyle 0 stroke-width:2px,stroke:red,color:red;
Phenomena (once their existence has been established) predict similar data patterns in new data sets of the same operationalization (as in “direct replication”) and ideally also for new operationalizations (as in “conceptual replication”, changing more UTOS dimensions).
Interlude: The risky shift phenomenon 1
The risky shift phenomenon: A group’s decisions are riskier than the average of the individual decisions of members before the group met (i.e., the group discussion made individuals riskier).
- Area of very active research in social psychology in the 1960s.
- “It could be easily replicated. Most of the replication studies […] employed the CDQ [Choice Dilemma Questionnaire] as their stimulus set, and they generally had no trouble obtaining the basic risky shift result.”
- Today we know that there is no risky shift. “It is now clear that the items contained in the original CDQ are in no sense a representative sample of the universe of all possible items. Instruments similar to the CDQ could readily be constructed whose scores would display risky shifts, cautious ones, or none at all” (Cartwright, 1971, p. 368).
- In the early risky shift literature, theoretical progress was unnecessarily impeded by multiple generations of replication studies (k ⋍ 200), nearly all relying on the same CQD questionnaire.
Interlude: The risky shift phenomenon 2
Questions for discussion (10 min.):
- Analyse the situation in the UTOS dimensions (units, treatments, outcomes, settings): To which dimensions did the successful replications (not) refer to?
- Is the “risky shift” finding - as demonstrated in the early publications - a phenomenon?
. . .
My take: It is a phenomenon (though a weak one), as it generalizes to new units (i.e., new participants) of the same operationalization. But the generalizability was much weaker than initially expected. It is a phenomenon of this specific stimulus set (outcome operationalization) and suggests certain types of research questions (e.g., “What is so specific to this stimulus set?”).
Replication crisis
Focus on phenomena
%%{ init: { 'flowchart': { 'curve': 'natural' } } }%%
flowchart LR
P(Phenomena)
D(Data)
P -- "Prediction" --> D
D -- "Generalization" --> P
The concerns of the replication crisis typically referred to the relation between data and phenomena:
- Does an empirical pattern even exist? (Or is it a false positive?)
- If we found a reproducible pattern with a specific operationalization: Is it generalizable (to other measures, other cultural contexts, other samples)?
- Do we even have a phenomenon in a particular research line? And how strongly should we belief in the existence of a phenomenon, given the empirical evidence?
Replication crisis
Focus on phenomena
%%{ init: { 'flowchart': { 'curve': 'natural' } } }%%
flowchart LR
T(Theory)
P(?? Phenomena ??)
D(Data)
T -- "Explanation" --> P
P -- "Abduction" --> T
P ---> D
D -. "?? Generalization ??" .-> P
linkStyle 2 stroke-width:0px,stroke:grey,color:grey;
linkStyle 3 stroke-width:2px,stroke:red,color:red;
Doubts about phenomena propagate to theories: If there is no phenomenon to explain, any explanatory theory gets obsolete.
From replication crisis to theory crisis
Reimagining psychology as a science
“Addressing the theory crisis in psychology”
“We argue that a further cause of poor replicability is the often weak logical link between theories and their empirical tests. […] A strong link between theories and hypotheses is best achieved by formalizing theories as computational models.”
“Many of these proposals [i.e., method reforms] are helpful ideas for raising the standards of good research practice, primarily ensuring more trustworthy inferences on the empirical level of scientific inference — the level connecting observations to empirical generalizations. Our concern is that shoring up the strength of inferences on the empirical level does not by itself address deficits on the theory level — the level connecting empirical generalizations to theories.”Oberauer & Lewandowsky (2019)
Obstacles to building useful theories in psychology
- Relative lack of robust phenomena that impose constraints on possible theories
- Problems of validity of psychological constructs
- Obstacles to discovering causal relationships between psychological variables
- Probably the most complex research object in the known universe …
- … and ridiculously small samples.
Slide adapted from Karolin Salmen, CC-BY
End
Contact
- @nicebread@scicomm.xyz
- ed.uml.ysp@tdorbneohcs.xilef
- https://www.nicebread.de
- https://github.com/nicebread
