New indicators for theory-guided research developed

A working group consisting of Nele Freyer, Jens Lange, Daniel Leising, Philipp Musfeld, and Felix Schönbrodt developed a set of indicators that allows to evaluate theory-guided research.

The indicators cover both narrative and formal theories, and they cover both theory development and theory testing.

In the general philosophy of RESQUE, these indicators mostly focus on “hygiene factors”: Properties of theory-guided research that are necessary but not sufficient for high-quality research.

Here are the indicators:

---

Theory usage and development

In the following, the term ‘theory’ is used for a set of (interconnected) ideas that explain empirical phenomena. Theories may be expressed in a narrative and/or a more formalized manner. The term ‘model’ is reserved for a (part of a) theory that is expressed mathematically or algorithmically.

---

[P_Theorizing] Theory-guided research and theory development

[ Condition: $P_Suitable === ‘Yes’ ]

[DescriptiveExploratory] This research was mainly descriptive and/or exploratory, without clear reference to a theory[TheoryGuided] In this research, at least one theory was (further) developed and/or evaluated[NotApplicable] Not applicable / cannot answer the question

---

[P_Theorizing_NAExplanation] ⤷ Not Applicable: Explanation

[ Condition: $P_Suitable === ‘Yes’ && $P_Theorizing === ‘NotApplicable’ ]

---

[P_Theorizing_Quality] The paper …

[ Condition: $P_Suitable === ‘Yes’ && $P_Theorizing === ‘TheoryGuided’ ]

[P_Theorizing_Quality_NarrativeAccount] contains an explicit narrative account of the phenomena that are to be explained, and of one or more hypothetical mechanisms accounting for them (i.e., a theory).[P_Theorizing_Quality_NarrativeDefinitions] contains explicit narrative definitions of each element (concepts, relationships, etc.) in the theory. All of this is made easily findable (e.g., under a standard heading ‘Definitions’).[P_Theorizing_Quality_FormalizedAccount] contains a formalized account of the theory (i.e., a model) using a commonly accepted standard notation (e.g., mathematical or logical operations, ODD protocol for ABMs). This includes all connections among the elements of the theory, and a formalization of the phenomena that are to be explained.[P_Theorizing_Quality_MathProps] contains a formalized account of the theory (i.e., a model) and explicates the basic mathematical properties of all elements of the model (e.g., scalar vs. matrix; dimensionality; units carrying the information).[P_Theorizing_Quality_PossibleValues] contains a formalized account of the theory (i.e., a model) and explicates the possible values for all elements of the model (i.e., upper and lower bounds, scale level).[P_Theorizing_Quality_ObjectiveDerivation] contains a formalized account of the theory (i.e., a model) and explicitly derives predictions from that. The derivation is objective, in the sense that any person would come to the exact same predictions.[P_Theorizing_Quality_Falsifiability] contains a formalized account of the theory (i.e., a model) and all stated predictions are in principle falsifiable (before considering operationalizations).

---

[P_Theorizing_Type] In what language or system is the theory formulated?

[ Condition: $P_Suitable === ‘Yes’ && $P_Theorizing === ‘TheoryGuided’ ]

[Narrative] Narrative[Mathematical] Mathematical[FormalLogic] Formal-Logic[ABM] Agent-based model[Other] Other (please specify)

---

[P_Theory_Type_Other] Theory type/ Language: Other

[ Condition: $P_Suitable === ‘Yes’ && $P_Theorizing === ‘TheoryGuided’ && $P_Theorizing_Type === ‘Other’ ]

---

[P_Theorizing_Contribution] How much did you personally contribute to this version of the theory?

[ Condition: $P_Suitable === ‘Yes’ && $P_Theorizing === ‘TheoryGuided’ ]

[No] I did not contribute to this version of the theory itself (i.e., I did no theoretical work beyond testing a theory)[Yes] I did contribute significantly to this version of the theory

---

[P_Theorizing_ContributionType] I significantly contributed to …

[ Condition: $P_Suitable === ‘Yes’ && $P_Theorizing === ‘TheoryGuided’ && $P_Theorizing_Contribution === ‘Yes’ ]

[P_Theorizing_ContributionType_NarrativeDevelopment] the overall development of an entire narrative theory[P_Theorizing_ContributionType_NarrativeRefinement] the expansion or refinement of an existing narrative theory[P_Theorizing_ContributionType_FormalizationFirst] the first formalization of an existing narrative theory[P_Theorizing_ContributionType_FormalizationStandAlone] the overall development of an entire formal model of which there was no narrative version[P_Theorizing_ContributionType_FormalizationRefinement] the expansion or refinement of an existing formal model

---

[P_Theorizing_Test] Did you test (a part of) this theory?

[ Condition: $P_Suitable === ‘Yes’ && $P_Theorizing === ‘TheoryGuided’ ]

[No] No[Yes] Yes

---

[P_Theorizing_Test_Quality] The paper …

[ Condition: $P_Suitable === ‘Yes’ && $P_Theorizing === ‘TheoryGuided’ && $P_Theorizing_Test === ‘Yes’ ]

[P_Theorizing_Test_Quality_Assumptions] describes all necessary assumptions that were necessary to make the theory testable (including ad-hoc assumptions).[P_Theorizing_Test_Quality_Operationalization] describes which, how, and why elements of the theory are measured, estimated, or fixed to specific values.[P_Theorizing_Test_Quality_Derivation] makes clear how the tested hypotheses logically and stringently follow from the theory.[P_Theorizing_Test_Quality_Evaluation] evaluates the relative and/or absolute model fit with appropriate procedures (e.g., out-of-sample cross-validation, AIC, BIC, LOO, Bayes Factors).[P_Theorizing_Test_Quality_Discussion] contains an explicit discussion as to what the results mean for the theory that was tested.