%%{
init: {
'theme': 'base',
'flowchart': { 'curve': 'natural' }
}
}%%
flowchart LR
T(Theory)
P(Phenomena)
D(Data)
T -- "Explanation" --> P
P -- "Abduction" --> T
P -- "Prediction" --> D
D -- "Generalization" --> P
Theory Construction Methodology
Theory Construction Methodology
Where do theories come from?
- Falsificationism starts with a (tentative) theory, which is repeatedly tested, refuted, and refined.
- But where does the initial theory come from? “Flair”? Flash of inspiration in the shower?
- Popper did not care much about the first stage of creating theories - just about the later testing of theories.
Conjecture: We have a refined methodology¹ to test theories (e.g., experimental designs, statistical methods, preregistration, …). But we had (so far) no good methodology for constructing theories.
¹ “A scientific methodology is an ordered series of steps that assist a researcher in reaching a desired end state from a specified starting point.” (Borsboom et al., 2021)
TCM: A methodology for constructing theories
Phenomena: Stable and general features of the world in need of explanation. Can be understood as robust generalizations of patterns in empirical data. They are the explanatory targets for scientific theories (the explanandum). In psychology often called “effects”.
Data: Relatively direct observations. Refer to particular empirical patterns in concrete data sets rather than empirical generalizations (which would be phenomenona).
Theories: Something that explains phenomena of interest (the explanans). But what is a theory?
Definitions
You know that already …
What is an (explanatory) theory?
A theory is a set of statements about the relationship(s) between two or more constructs with a nomological (i.e. law-like) character.
What is a formal model?
A formal model is one possible implementation of a theory (typically using additional auxiliary assumptions). It is able to generate “fake” data and data patterns that would be observed in reality if the theory and the model are sufficiently accurate.
What is an explanation?
In the productive explanation framework, a theory T putatively explains a phenomenon P if and only if a formal model of the theory T produces a statistical pattern representing the empirical phenomenon P.
TCM: Concrete Steps
- Identifying relevant phenomena
- Formulating a prototheory OR
- Formalizing an existing narrative theory
- Developing a formal model
- Checking the adequacy of the formal model
- Evaluating the overall worth of the constructed theory
Note
This methodology is “structured creativity” - you are allowed to tinker around as much as you like. The five steps are mere tools that help to structure your creative process.
TCM Step 1️⃣: Identifying relevant phenomena
„The phenomena most useful in theory building are not necessarily the most spectacular ones. Instead, it is vitally important to select phenomena that are well established, or even self-evident, because a solid foundation is essential to successful theory construction.“
- Assess evidential strength (meta-analyses?)
- Assess risk of bias (red flags for p-hacking?)
- Distinguish…
- core explanatory targets
- secondary explanatory targets
TCM Step 2️⃣a: Formulating a prototheory
„Of the steps in TCM, the step of generating prototheories is the least methodologically developed. One methodological approach that is available is analogical abduction: If one finds a similar set of phenomena in another field that is better understood, then one can “borrow” explanatory principles from that field to inform one’s own.“
- A small set of general principles, often causal relationships
TCM Step 2️⃣b: Formalizing an existing narrative theory
- Use the Visual Argument Structure Tool (VAST) to visualize an existing narrative theory
- Do not try to improve it (yet) - show the gaps and inconsistencies
TCM Step 3️⃣: Developing a formal model
A formal model captures the principles of the explanatory theory in a set of equations or rules (as implemented in a computer program or simulation).
- E.g., a Bischof control system model as an implementation of the VAST display. Record which gaps you had to fill, and which were directly derived from the theory (maybe with color coding?)
TCM Step 4️⃣: Checking the adequacy of the formal model
“To investigate this question, one must parse the phenomena in the same formal language as the theory. This means that the phenomena themselves have to be formalized.”
Before you observe any real data, ask (and test): Does the model even work in principle?
- Simulate data from the model: Does it produce the core target phenomena?
- Sensitivity analysis:
- What happens if parameters of the model are changed / pushed to the extreme? When does it break?
- How robust is the explanation across (realistic) changes of parameter settings?
- Does it produce the secondary phenomena, even if they have not been “baked into the model”? (Assuming that multiple phenomena are predicted) → maybe go back to step 3️⃣ or even 2️⃣ to refine theory/model
- Explanatory relevance: Are the explanatory principles actually necessary to explain phenomena? (e.g., it could happen that you remove an element in the model, and the phenomenon still is produced.)
TCM Step 5️⃣5️: Evaluating the overall worth of the constructed theory
(We’ll do that later)
TCM example
Creative hypothesis generating in psychology: Some useful heuristics
TODO
Resources
Speicher
“Makeism”
What is makeism?
Makeism: The view that computationalism implies that (a) it is possible to (re)make cognition computationally; (b) if we (re)make cognition then we can explain and/or understand it; and possibly (c) explaining and/or understanding cognition requires (re)making cognition itself.
Note that it is especially easy for makeists to fall into map-territory confusion - mistaking their modeling artefacts for cognition itself - due to the view that the made thing could be cognition.
➙ “Design and development as a research methodology” (Bisig & Pfeifer, 2008)
Is the demiurgic approach “makeism”?
| Makeism | Demiurg |
|---|---|
| (a) it is possible to (re)make natural phenomena computationally | Yes - the systemic structure can be recreated as a (simplfied) model. |
| (b) (Re)makeing a natural phenomenon is sufficient for being able to explain it | It is an explanation (cf. productive explanation). But its quality is only as good as its assumptions*. |
- “I assume an omnipotent agent, which knows everything about the world, its current state, and its history. With unlimited computing power, it determines the optimal action.”
About simulations as tools
Computer simulations support and extend a scientist’s thinking capacity, and enable computerised ‘thought experiments’ (R. Cooper, 2005) to reason through ‘what ifs’ and answer questions like ‘how possibly’. These simulations […] are necessarily abstract and idealised
Underdeterminism of theories
“[…] first, one and the same problem can be computed by different algorithms, and second, one and the same algorithm can be physically realised in different ways.
This implies that we are dealing with massive underdetermination of theory by data: i.e., if we observe behaviours consistent with a computational level theory, we cannot infer which algorithms or neural processes underlie the behaviour.
Comparison chart of methodologies
| TCM | Demiurg | |
|---|---|---|
| Starting point | set of relevant phenomena in need for explanation | an (evolutionary) problem that needs to be solved (v1) + prior knowledge about precursing organisms (v2) |
| Primary heuristic for searching explanations | Look for analogous models/phenomena in other scientific disciplines | Look at existing capabilities of simpler organisms (biology); search for the simplest implementation (given existing biological structures) |
| End state | A theory that offers a putative explanation of the phenomena |