2024-08-12
See Borsboom et al. (2021). Theory Construction Methodology: A Practical Framework for Building Theories in Psychology. Perspectives on Psychological Science, 16(4), 756–766. https://doi.org/10.1177/1745691620969647
Conjecture: We have a refined methodology¹ to test theories (e.g., experimental designs, statistical methods, preregistration, …). But we had (so far) no good methodology for constructing theories.
¹ “A scientific methodology is an ordered series of steps that assist a researcher in reaching a desired end state from a specified starting point.” (Borsboom et al., 2021)
Phenomena: Stable and general features of the world in need of explanation. Can be understood as robust generalizations of patterns in empirical data. They are the explanatory targets for scientific theories (the explanandum). In psychology often called “effects”.
Data: Relatively direct observations. Refer to particular empirical patterns in concrete data sets rather than empirical generalizations (which would be phenomenona).
Theories: Something that explains phenomena of interest (the explanans). But what is a theory?
What is an (explanatory) theory?
A theory is a set of statements about the relationship(s) between two or more constructs with a nomological (i.e. law-like) character.
What is a formal model?
A formal model is one possible implementation of a theory (typically using additional auxiliary assumptions). It is able to generate “fake” data and data patterns that would be observed in reality if the theory and the model are sufficiently accurate.
What is an explanation?
In the productive explanation framework, a theory T putatively explains a phenomenon P if and only if a formal model of the theory T produces a statistical pattern representing the empirical phenomenon P.
Borsboom et al. (2021). https://doi.org/10.1177/1745691620969647; van Dongen et al. (2023). https://doi.org/10.31234/osf.io/qd69g
Note
This methodology is “structured creativity” - you are allowed to tinker around as much as you like. The five steps are mere tools that help to structure your creative process.
„The phenomena most useful in theory building are not necessarily the most spectacular ones. Instead, it is vitally important to select phenomena that are well established, or even self-evident, because a solid foundation is essential to successful theory construction.“
Borsboom et al., 2021, p. 760
„Of the steps in TCM, the step of generating prototheories is the least methodologically developed. One methodological approach that is available is analogical abduction: If one finds a similar set of phenomena in another field that is better understood, then one can “borrow” explanatory principles from that field to inform one’s own.“
Borsboom et al., 2021, p. 761
A formal model captures the principles of the explanatory theory in a set of equations or rules (as implemented in a computer program or simulation).
Borsboom et al., 2021, p. 761
“To investigate this question, one must parse the phenomena in the same formal language as the theory. This means that the phenomena themselves have to be formalized.”
Before you observe any real data, ask (and test): Does the model even work in principle?
Borsboom et al., 2021, p. 761
(We’ll do that later)
Borsboom et al., 2021, p. 761
TODO
See McGuire, W. J. (1997). Creative hypothesis generating in psychology: Some useful heuristics. Annual Review of Psychology, 48, 1–30. https://doi.org/10.1146/annurev.psych.48.1.1
What is makeism?
Makeism: The view that computationalism implies that (a) it is possible to (re)make cognition computationally; (b) if we (re)make cognition then we can explain and/or understand it; and possibly (c) explaining and/or understanding cognition requires (re)making cognition itself.
Note that it is especially easy for makeists to fall into map-territory confusion - mistaking their modeling artefacts for cognition itself - due to the view that the made thing could be cognition.
➙ “Design and development as a research methodology” (Bisig & Pfeifer, 2008)
Abridged quote from van Rooij et al. (2023, August 1). Reclaiming AI as a theoretical tool for cognitive science. https://doi.org/10.31234/osf.io/4cbuv
Makeism | Demiurg |
---|---|
(a) it is possible to (re)make natural phenomena computationally | Yes - the systemic structure can be recreated as a (simplfied) model. |
(b) (Re)makeing a natural phenomenon is sufficient for being able to explain it | It is an explanation (cf. productive explanation). But its quality is only as good as its assumptions*. |
Computer simulations support and extend a scientist’s thinking capacity, and enable computerised ‘thought experiments’ (R. Cooper, 2005) to reason through ‘what ifs’ and answer questions like ‘how possibly’. These simulations […] are necessarily abstract and idealised
van Rooij et al. (2023, August 1). Reclaiming AI as a theoretical tool for cognitive science. https://doi.org/10.31234/osf.io/4cbuv, p. 13
“[…] first, one and the same problem can be computed by different algorithms, and second, one and the same algorithm can be physically realised in different ways.
This implies that we are dealing with massive underdetermination of theory by data: i.e., if we observe behaviours consistent with a computational level theory, we cannot infer which algorithms or neural processes underlie the behaviour.
Van Rooij et al. (2023). Reclaiming AI as a theoretical tool for cognitive science [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/4cbuv
TCM | Demiurg | |
---|---|---|
Starting point | set of relevant phenomena in need for explanation | an (evolutionary) problem that needs to be solved (v1) + prior knowledge about precursing organisms (v2) |
Primary heuristic for searching explanations | Look for analogous models/phenomena in other scientific disciplines | Look at existing capabilities of simpler organisms (biology); search for the simplest implementation (given existing biological structures) |
End state | A theory that offers a putative explanation of the phenomena |
Formal modeling in psychology - Empirisches Praktikum, Ludwig-Maximilians-Universität München