Session 9 - How to simulate virtual experiments

We practice how to generate data for simulated experiments

Overview

Topic Duration Notes
How to simulate data from model: Mathematical Models Step 5 120 Slides, Step 5
Start working on the new formalizations 45

Steps of formalization (overview)

(see also VAST application).

  1. Set up technical infrastructure.
    1. Create a fresh Github repository and a folder on Google Drive (or a comparable collaborative document) for the tables.
    2. Link the latter in the README of the repository.
    3. Invite the lecturer as a collaborator: Settings → Collaborators → “Add people”
  2. Choose your starting point:
    1. Existing theory –> Narrative Theory Reconstruction (NTR)
    2. Robust phenomenon –> Theory Construction Methodology (TCM)
  3. Define the scope
    1. Define scope of original literature that is used as basis for the formalization.
    2. Limit the number of phenomena that your model is supposed to explain.
    3. Limit the number of constructs and their relationships.
  4. Collect robust phenomena
    1. Evaluate the generalizability separately for each UTOS dimension
    2. Evaluate the strength of evidence (Quality of studies? Number of studies? Large samples?)
  5. Create your Construct Source Table
  6. Create your VAST display (see slides on VAST and VAST application)
    1. Distill a working definition for each construct
    2. Add relationships between constructs
  7. Formalize model: Create your Variable Table (see slides on Mathematical Modeling)
    1. List all concepts of your VAST display that are not higher-order concepts (because higher-order concepts typically are not reducible to a single metric)
    2. Define the scale level, the possible range of values, and semantic anchors
  8. Formalize model: Create your Relationship Table (see slides on Mathematical Modeling)
    1. Develop a mathematical function for each endogenous variable in the VAST display.
    2. What functional form does the theory imply? What are free parameters that can “bend” the function? Which parameters can be fixed?
  9. Program the functions in R
    1. Build atomic functions based on the Relationship Table
    2. Document the functions with roxygen2
    3. Combine atomic functions into one “super function” that takes all exogenous variables of the model as input and computes the focal output variable (typically, an observable behavior). Make sure that it can take vectors as input.
  10. Evaluate the model behavior
    1. Try different parameter settings. Under which conditions does the model produce the phenomenon?
    2. Plot the model behavior: Choose a focal predictor for the x-axis, focal outcome on y-axis, additional parameters as colors, shapes, or multiple plots.
  11. Simulate the model: Would a real study (with noise) be able to detect the phenomenon?
    1. Simulate data
    2. Do the statistical analysis on your simulated data: Is the focal effect significant?
    3. What is the necessary sample size to reliably detect the phenomenon (i.e., with a high statistical power)?