Session 09 - Data Analysis & Power Analysis

We review the data preprocessing and agree on a “best practice” version of it. We create coding instructions for free-text answers. We do a proper power analysis based on the original finding.

Overview

Topic	Duration	Notes
Do the preprocessing together	60
Create a codebook for rating strengths and weaknesses	45
Power analysis Intro	45	Slides
How (not) to report a power analysis	15	Slides
Do a power analysis in practice	15

General workflow of coding

Groups (or individuals) do a coding task on their own (as homework or in class)
In class, we discuss the approaches and potential problems and agree on a “best practice” version of the coding steps.
- The instructor live-codes a best-practice version and pushes it to a public repository.
- Participants can keep their own version (if it is functionally equivalent), or pull the joint version into their own script.
- Importantly, at the end all script version should lead to the exactly same result!

Coding instructions for counting the strengths and weaknesses

See Google doc with Coding Instructions.

Try it out in pairs on the old raw data:

Independently count strengths and weaknesses, then compare your numbers.
Where did you have deviations? What texts are in a grey area?
Make a suggestion how to improve the codebook so that these ambiguous cases are clearly defined.

When finished, convert the Coding Instructions to a PDF, store in the /doc folder, and upload to OSF materials.

Homework 1 (individual): Do the coding of the (so far existing) strengths and weaknesses

Download the xlsx file with the strengths and weaknesses data, replace the XX in the file name with your own initials. There is an individual file per rater.
In the columns labelled num_strengths_XX and num_weaks_XX, replace XX with your initials (e.g., num_strengths_FS, num_weaks_FS).
Count the strengths and weaknesses in the texts assigned to you, according to the coding manual. Each text is scored by two coders.
Enter “0” if no are present.
Do not communicate with anybody about your ratings! The ratings must be strictly independent in order to compute a valid interrater agreement.
Send the table back to the instructor via email.

Homework 2 (individual): Prepare your manuscript/course report

Note: Most of you work in groups on one project folder. When the course is complete, you will make a personal copy of that folder and continue to work in that (and also submit that at the end).

Add your manuscript.qmd file in the top level folder. I suggest directly to use the apaquarto extension and start with their template.

As homework, prepare the general structure of a scientific paper. Add all section headings; “Methods”, “Results”, “Discussion” should be Level 1 (with #).

Next, write the Methods section (see details in next Session Overview).

Whenever you are tempted to type a specific number, such as …

The final sample consisted of 95 participants.

… don’t do it - you would have to update it with every new data download. Instead use dynamic computations, either with inline computations:

The final sample consisted of `r nrow(dat_final)` participants.

Or use longer R chunks for multiline computations:

```r
# This is a longer R chunk
# The apaquarto extension automatically sets `#| echo: false` 
# so that the code is not shown in the output

dat <- read.csv("data.csv")
# do some data preprocessing ...
```

Look at some of the APA’s sample papers, and read the respective sections in a couple of papers from our literature list.

Some notes:

The questionnaire scales were computed as the mean of all items.
You do not have to report reliability statistics (you did not learn this yet).