Power Analysis: Introduction

Slides: https://osf.io/t5rjf/

Felix Schönbrodt
felix.schoenbrodt@psy.lmu.de

Ludwig-Maximilians-Universität München

2024-04-19

Part I: General concepts of power analysis

What is statistical power?

A 2x2 classification matrix

Reality:
Effect present		 Reality:
No effect present
Test indicates:
Effect present		 True Positive False Positive
Test indicates:
No effect present		 False Negative True Negative

https://effectsizefaq.files.wordpress.com/2010/05/type-i-and-type-ii-errors.jpg

https://effectsizefaq.files.wordpress.com/2010/05/type-i-and-type-ii-errors.jpg

A priori power analysis

Note

A priori power analysis: We assume that the effect exists in reality.
→ We focus on the left column (“Reality: Effect present”)

Reality:
Effect present		 Reality:
No effect present
Test indicates:
p < .05		 Power = 1-β
True Positive		 α = 5%
False Positive
Test indicates:
p > .05		 β = 20%
False Negative
True Negative

Note

Power: The probability to detect an effect if an effect is present.

How to do a power analysis

flowchart LR
    L["<b>Significance Level</b><br>0.05? 0.005 (Benjamin et al., 2018)?<br>justify your alpha (Lakens et al., 2018)?"]
    D["<b>Desired Power</b><br>usually 80%, 90% recommended for<br>critical studies (Bondavera, 2013)"]
    E["<b>Effect Size</b><br>(see Part II of the workshop)"]
    S["<b>Sample Size</b>"]
    
    L --> S
    D --> S
    E --> S

Benjamin et al. (2018). Redefine statistical significance. doi:10.1038/s41562-017-0189-z; Lakens et al. (2018). Justify your alpha. doi:10.1038/s41562-018-0311-x

Power is a frequentist property - beware of fallacies!

Power is a pre-data measure (i.e., before data are collected) that averages over infinite hypothetical experiments

  • Only one of these hypothetical experiments will actually be observed
  • Power is a property of the test procedure/ the design – not of a single study’s outcome!

Power is conditional on a hypothetical effect size – not conditional on the actual data obtained

  • “Once the actual data are available, a power calculation is no longer conditioned on what is known, no longer corresponds to a valid inference, and may now be misleading.” ➙ for inference better use likelihood ratios or Bayes factors. Then pre-data power considerations are irrelevant.

Wagenmakers et al. (2014). A power fallacy. Behavior Research Methods, 47, 913–917. doi:10.3758/s13428-014-0517-4

Post hoc power considerations

  • Using the observed effect size to calculate “post hoc power” is meaningless (it‘s just a transformation of the p-value)
  • It is however meaningful to estimate the power you have achieved with your collected sample size and the a priori assumed effect size (“sensitivity power analysis”)

https://daniellakens.blogspot.com/2014/12/observed-power-and-what-to-do-if-your.html Gelman, A. (2019). Don’t calculate post-hoc power using observed estimate of effect size. Ann. Surg, 269, e9-e10

Why power is important

Exercise:

Given that p < .05:
What is the probability that a real effect exists in the population ➙ \(p(H_1 | D)\)?

Nuzzo, R. (2014). Statistical errors. Nature. Colquhoun, D. (2014). An investigation of the false discovery rate and the misinterpretation of p-values. Royal Society Open Science, 1(3), 140216–140216. http://doi.org/10.1073/pnas.1313476110

Nuzzo, R. (2014). Statistical errors. Nature. Colquhoun, D. (2014). An investigation of the false discovery rate and the misinterpretation of p-values. Royal Society Open Science, 1(3), 140216–140216. http://doi.org/10.1073/pnas.1313476110

Nuzzo, R. (2014). Statistical errors. Nature. Colquhoun, D. (2014). An investigation of the false discovery rate and the misinterpretation of p-values. Royal Society Open Science, 1(3), 140216–140216. http://doi.org/10.1073/pnas.1313476110

Assumed that our tested hypothesis are true in 30% of all cases (which is a not too risky research scenario):

  • A typical neuroscience study must “fail” (p > \(\alpha\)) in 90% of all cases
  • In the most likely outcome of p > .05, we have no idea whether a) the effect does not exist, or b) we simply missed the effect. Virtually no knowledge has been gained.

Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci, 14(5), 365–376. doi:10.1038/nrn3475

When a study is underpowered it most likely provides only weak inference. Even before a single participant is assessed, it is highly unlikely that an underpowered study provides an informative result.

Consequently, research unlikely to produce diagnostic outcomes is inefficient and can even be considered unethical. Why sacrifice people’s time, animals’ lives, and societies’ resources on an experiment that is highly unlikely to be informative?

Schönbrodt, F. D. & Wagenmakers, E.-J. (2018). Bayes Factor Design Analysis: Planning for compelling evidence. Psychonomic Bulletin & Review, 25, 128-142. doi:10.3758/s13423-017-1230-y

A power analysis helps you to find a balance between…

Pictures from nature.com/articles/srep43627, pngimg.com/download/23544, whyopenresearch.org/funding

Researcher‘s intuitions about power

Researcher’s intuitions about power

Bakker, M., Hartgerink, C. H. J., Wicherts, J. M., & van der Maas, H. L. J. (2016). Researchers’ Intuitions About Power in Psychological Research. Psychological Science, 27(8), 1069–1077. http://doi.org/10.1177/0956797616647519

Calibrate your power feeling

Calibrate your power feeling

Clever designs go a long way

The power of within-SS designs

  • Why? Each person is his/her own control group

  • For example, for the paired t-test:

    • By computing the within-person difference scores, all between-person variance (which contributes to error variance), gets removed
    • Less error variance → less noise → (relatively) more signal → larger effect size

Increase power with reliable measures

  • Cohen’s d = 0.4
  • N = 30
  • pre-post-test

Heo, M., Kim, N., & Faith, M. S. (2015). Statistical power as a function of Cronbach alpha of instrument questionnaire items. BMC Medical Research Methodology, 15. doi:10.1186/s12874-015-0070-6

Specific predictions?
Use one-tailed tests!

  • One-tailed tests have a higher power than two-tailed tests
  • Particularly recommended in combination with a preregistration
  • Most power analysis approaches (G*Power, R packages) allow you to chose between one- and two-tailed tests

Any questions so far?

Part II:
Effect sizes / smallest effects of interests

Common effect size metrics

Common effect sizes

Effect size transformations

Effect size transformations

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Effect sizes based on correlations. In Introduction to Meta-Analysis, p. 45-49. Brysbaert, M. (2019) How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables. Journal of Cognition, 2(1): 16, pp. 1–38. DOI: https://doi.org/10.5334/joc.72 Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4. https://doi.org/10.3389/fpsyg.2013.00863

Converting among effect sizes

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Converting among effect sizes. In Introduction to Meta-Analysis, p. 45-49. Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4. https://doi.org/10.3389/fpsyg.2013.00863

Converting among effect sizes

https://www.ibm.com/support/pages/effect-size-relationship-between-partial-eta-squared-cohens-f-and-cohens-d Brysbaert, M. (2019) How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables. Journal of Cognition, 2(1): 16, pp. 1–38. DOI: https://doi.org/10.5334/joc.72

Converting among effect sizes

Brysbaert, M. (2019) How Many Participants Do We Have to Include in Properly Powered Experiments? A Tutorial of Power Analysis with Reference Tables. Journal of Cognition, 2(1): 16, pp. 1–38. DOI: https://doi.org/10.5334/joc.72

Converting among effect sizes

Getting a feeling about effect sizes

How do these effect sizes look like?

https://rpsychologist.com/d3/cohend/

How do these effect sizes look like?

https://rpsychologist.com/d3/cohend/

How do these effect sizes look like?

https://rpsychologist.com/d3/cohend/

Guess the correlation

http://guessthecorrelation.com/

Guess the correlation

http://guessthecorrelation.com/

Understanding effect sizes

More understandable metrics: „Common Language Effect Size“, CLES:

  • …the probability that a randomly sampled person from one group will have a higher observed measurement than a randomly sampled person from the other group (for between designs)
  • …or (for within-designs) the probability that an individual has a higher value on one measurement than the other.

McGraw & Wong (1992). A common language effect size statistic. https://doi.org/10.1037/0033-2909.111.2.361

Understanding effect sizes

Example: d = 0.4, n = 55 in each group

  • Repeated-measures factor: 61% of the participants change into the expected direction
  • Between-groups factor: 61% chance of finding the expected ordering if you test a random participant from each sample

Typical effect sizes

Cohen‘s conventions

Is this reasonable?

Typical reported effect sizes I

Richard, Bond, & Stokes-Zoota (2003), correlation metric

  • Meta-meta-analysis; > 25.000 studies, > 8.000.000 participants
  • mean effect r = .21 (across literature SD = .15); median = .18

Richard, F. D., Bond, C. F. J., & Stokes-Zoota, J. J. (2003). One Hundred Years of Social Psychology Quantitatively Described. Review of General Psychology, 7(4), 331–363. doi:10.1037/1089-2680.7.4.331

Typical reported effect sizes I

Richard, Bond, & Stokes-Zoota (2003), correlation metric

Typical reported effect sizes II

Bosco et al. (2015), correlation metric

  • 147,328 correlations from Journal of Applied Psychology and Personnel Psychology
  • median effect: r = .16, mean effect r = .22 (SD = .20)

Bosco et al. (2015). Correlational effect size benchmarks. http://doi.org/10.1037/a0038047

Typical reported effect sizes III

Hill et al. (2008), Hedge’s g

  • How does the effect of an intervention compare to a typical year of growth in school?

Hill et al. (2008). Empirical Benchmarks for Interpreting Effect Sizes in Research. doi:10.1111/j.1750-8606.2008.00061.x

Typical reported effect sizes IV

Funder & Ozer (2019), correlation metric

Funder, D. C., & Ozer, D. J. (2019). Evaluating Effect Size in Psychological Research: Sense and Nonsense. Advances in Methods and Practices in Psychological Science, 2, 156–168. doi:10.1177/2515245919847202

Typical reported effect sizes V

Aguinis, Beaty, Boik, & Pierce (2005), \(f^2\)

Effect size of interaction from dichotomous moderator and continuous predictor

Aguinis et al. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review. Journal of Applied Psychology, 90(1), 94.

Other benchmarks I

Average placebo effect?

d = 0.24 [0.17; 0.31]!

Hróbjartsson, A., & Gøtzsche, P. C. (2004). Is the placebo powerless? Update of a systematic review with 52 new randomized trials comparing placebo with no treatment. Journal of internal medicine, 256(2), 91-100.

Other benchmarks II (ES: d)

Luhmann, M., Hofmann, W., Eid, M., & Lucas, R. E. (2012). Subjective well-being and adaptation to life events: a meta-analysis. Journal of Personality and Social Psychology, 102(3), 592–615. http://doi.org/10.1037/a0025948

The trustworthiness of effect sizes in the literature

Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science, 7(6), 543–554.

Fanelli, D. (2011). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891–904. doi:10.1007/s11192-011-0494-7

Can we base our power analyses on published effect sizes?

No.

  • See RP:P: 83% of all effect sizes are smaller than the original:
    Mean original: r = .40 ➙ Mean replication: r = .20
  • See also Franco et al. (2015):
    Reported ES 2x larger than unreported ES

Can we base our power analyses on published effect sizes?

• See Schäfer & Schwarz (2019), ES: r:

Schäfer, T., & Schwarz, M. A. (2019). The meaningfulness of effect sizes in psychological research: Differences between sub-disciplines and the impact of potential biases.Frontiers in Psychology,10, 813.

Can we base our power analyses on published effect sizes?

  • Suggestion 1: Divide reported effect by 2, compute power analysis.
  • Suggestion 2: Safeguard power (Perugini, 2014): Incorporate uncertainty in original study’s ES estimate. Aim lower end of 60%-CI.

Safeguard power

Perugini et al., 2014

  • Incorporate uncertainty in original study’s ES estimate
  • Aim for lower end of 60%-CI
  • Example:
    • Original study finds d = 0.5 (n = 30 in each group)
    • 60% CI = [0.28; 0.72]
    • Naive 80% power analysis: n = 64
    • Safeguard 80% power analysis: n = 202
  • Rewards precise estimates in original study
library(MBESS)
ci.smd(smd=0.5, n.1=30, n.2=30, conf.level=0.60)

End

Contact

CC-BY-SA 4.0