PSY 3220 Module 5 Experimental Research Paper: Designing a Randomized Experiment and Preregistering It Example

Three Conditions, One Plan: Designing and Preregistering a Phone Experiment

Student Name

Bachelor of Arts in Psychology, Thomas Edison State University

PSY-3220: Research in Experimental Psychology

Mentor Name

Month Day, Year

Three Conditions, One Plan: Designing and Preregistering a Phone Experiment

Introduction

My survey can describe how adult learners study, but only an experiment can test whether phone location causes differences in comprehension. This paper explains the logic of experiments, describes my design and explains why I will preregister it.

The Logic of an Experiment

In an experiment, the researcher manipulates an independent variable, measures a dependent variable and holds other factors constant. Random assignment, giving each participant an equal chance of being in any condition, makes groups equivalent on average in characteristics the researcher did not measure, such as motivation or reading skill. When groups differ afterward, the manipulation is the most plausible cause. Threats to this inference, called confounds, occur when another variable changes along with the manipulation.

A Related Experiment

Stothart et al. (2015) gave participants a sustained attention task that required them to respond to most numbers on a screen but withhold responses to one target number. During the second half, some participants received calls or text messages on their own phones, which researchers had arranged without their knowledge, while others received nothing. Participants who received notifications made more errors, even though they did not answer or look at their phones, and the disruption was similar in size to that seen in studies where people actively use their phones. The study shows how a phone can capture attention without any interaction, and its random assignment allows a causal conclusion.

My Design

I will use a between-subjects design with three conditions: the phone lying face down beside the laptop, the phone zipped into a bag on the floor and the phone left in a different room. Each participant will be randomly assigned to one condition using a computer-generated sequence. All phones will be silenced to separate the effect of presence from notifications, the effect Stothart and colleagues studied. Each participant will study a 1,000-word passage for 15 minutes and then answer 20 comprehension questions. Phone dependence and state anxiety will be measured before the task and anxiety again after it.

Controlling Confounds

The same quiet room, passage, instructions and research assistant will be used for every session. The assistant will read instructions from a script and will not know the hypothesis, to prevent unintended cues. Session times will be balanced across conditions so that, for example, more desk-condition sessions do not occur late in the evening when people are tired.

Why Preregister?

Simmons et al. (2011) showed how ordinary flexibility in research decisions can produce false positive results. Choices such as how many participants to collect, whether to stop early when results look good, which outcome to report, whether to include a covariate and which participants to exclude each seem harmless, but together they can push the odds of declaring a nonexistent effect "significant" well past the 5 percent researchers assume. In one demonstration, they produced a statistically significant result showing that listening to a particular song made people younger. They recommended that researchers decide sample size in advance, report all variables and conditions and report results with and without exclusions and covariates.

Nosek et al. (2018) argued for preregistration: writing down hypotheses, design and analysis plan in a time-stamped public registry before collecting data. Preregistration distinguishes confirmatory tests, which test predictions made in advance, from exploratory analyses, which generate new ideas but should be labeled as such. It prevents researchers from presenting results discovered after the fact as if they had been predicted. Deviations from the plan are allowed but must be reported openly.

My Preregistration

Before recruiting anyone, I will file a plan stating the hypothesis, that comprehension will be lowest in the desk condition and highest in the other-room condition; the sample size, 50 per condition, based on a power analysis I will describe in the module on results; the stopping rule, stopping at 150 complete participants; exclusion criteria, such as failing to complete the passage or a fire alarm interrupting the session; and the main analysis, a one-way analysis of variance followed by planned comparisons. Any analyses of phone dependence as a moderator will be labeled exploratory.

Manipulation Check

A final question will ask participants where their phone was during reading, to confirm the manipulation worked as intended.

Threats From the Setting

Demand characteristics, cues that tell participants what the researcher expects, could bias results if participants guess the hypothesis. Using a neutral description of the study, a blind assistant and identical instructions in every condition reduces these cues. A final question asking participants what they think the study was about will show whether anyone guessed.

Within or Between?

A within-subjects design, in which each participant reads in all three conditions, would need fewer people, but practice and fatigue would build across sessions, and participants would quickly guess the purpose. Different passages and counterbalanced order could address some of these problems, but the between-subjects design keeps each person naive to the comparison.

Analysis Plan in Brief

The preregistered analysis will compare mean comprehension across the three conditions, followed by two planned comparisons: desk versus other room and bag versus other room. Anxiety will be examined to see whether it differs by condition, which would point to an alternative explanation.

Limits

A laboratory session may not reflect studying at home. Silencing phones removes notifications, which are part of real-world distraction. The study is planned, not yet conducted.

Conclusion

Stothart and colleagues showed how a randomized experiment can isolate a phone's effect on attention, Simmons and colleagues showed how flexible decisions produce false positives and Nosek and colleagues showed how preregistration protects against them. Random assignment, controlled procedures, a blind research assistant, a manipulation check and a preregistered plan give my experiment a fair chance of finding the truth, whatever it turns out to be.

References

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600-2606. https://doi.org/10.1073/pnas.1708274114

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632

Stothart, C., Mitchum, A., & Yehnert, C. (2015). The attentional cost of receiving a cell phone notification. Journal of Experimental Psychology: Human Perception and Performance, 41(4), 893-897. https://doi.org/10.1037/xhp0000100