PSY 3220 Module 7 Understanding Research Results Paper: Effect Sizes, Power and What a p-Value Means Example

Significant Is Not the Same as Large: Reading the Results of a Phone Experiment

Student Name

Bachelor of Arts in Psychology, Thomas Edison State University

PSY-3220: Research in Experimental Psychology

Mentor Name

Month Day, Year

Significant Is Not the Same as Large: Reading the Results of a Phone Experiment

Introduction

After designing my experiment, I needed to decide how many participants to recruit and how I will interpret the numbers. Many students, including me at first, treat a p-value below .05 as proof that an effect is real and important. This paper explains why that is mistaken and how effect sizes, confidence intervals and power give a fuller picture.

What a p-Value Is and Is Not

Six principles appear in a statement Wasserstein and Lazar (2016) prepared for the American Statistical Association. In plain terms, a small p-value says the observed data would be surprising if a model of no effect were true, and nothing more. Students often read it as the likelihood that their hypothesis is right, or as the likelihood that luck produced the result, and the statement rejects both readings. Treating .05 as a bright line invites poor decisions. Honest inference depends on reporting every analysis, not just the ones that worked. The size and importance of an effect must be judged separately, and a lone p-value is weak evidence on its own.

Effect Sizes

Lakens (2013) explains that effect sizes describe how large a difference or relationship is, in a form that can be compared across studies. For differences between two groups, Cohen's d expresses the difference in standard deviation units; for designs with more groups, measures such as eta squared or omega squared describe the proportion of variance explained by the manipulation. Lakens recommends reporting effect sizes with confidence intervals, which show the range of plausible values, and explains that effect sizes allow later researchers to plan sample sizes and combine studies in meta-analyses.

Power

Cohen (1992) wrote a brief primer urging researchers to plan for power, meaning the chance that a study will detect a true effect of a chosen size. Power depends on the significance level, the sample size and the size of the effect. Cohen offered rough conventions for differences between means, calling a d near one fifth of a standard deviation small, half a standard deviation medium and four fifths large. He provided tables showing how many participants are needed for power of .80 at a .05 significance level; for example, detecting a medium difference between two groups requires about 64 participants per group. Many studies of his time, he noted, were too small to detect the effects they sought.

My Power Analysis

Ward and colleagues' effects were modest. Planning for an effect between small and medium, about d = 0.45 for the comparison of desk and other-room conditions, with power of .80 and a two-tailed .05 significance level, a power calculation indicates roughly 80 participants per group for that comparison. Because recruiting 240 adults is beyond my resources, I will plan for 50 per condition, enough to detect a medium effect in that comparison with power near .70, and state this limitation openly. This is the sample size in my preregistration.

Interpreting Hypothetical Results

Suppose the desk condition averages 13.1 of 20 correct, the bag condition 13.8 and the other-room condition 14.6, with standard deviations near 3.0. A one-way analysis of variance gives F(2, 147) = 3.12, p = .047, eta squared = .04. A planned comparison of desk and other-room conditions gives d = 0.50, and its 95 percent interval runs from 0.10 to 0.90.

What These Numbers Mean

The p-value of .047 means the data would be fairly unusual if phone location had no effect, but it does not tell us how likely the hypothesis is. The effect size of d = 0.50 suggests a medium difference, about one and a half questions out of twenty. The wide interval, stretching from barely noticeable to quite large, shows considerable uncertainty. A results section that said only "phones significantly reduced comprehension" would overstate what the data show.

Writing the Results Paragraph

In APA style, I would report means and standard deviations for each condition, the test statistic with degrees of freedom, the exact p-value and the effect size with its confidence interval, followed by a plain-language summary that notes the uncertainty.

Statistical Versus Practical Significance

A difference of one and a half questions out of twenty may or may not matter. For a student near the boundary between two grades, it could be meaningful; across a semester of reading, small differences might add up. Interpreting effect sizes in terms readers can picture, such as questions answered correctly, helps judge practical importance in a way a p-value cannot.

What a Null Result Would Mean

If my study finds no significant difference, the confidence interval will matter most. An interval that includes both no effect and a medium effect would show that the study could not rule out a meaningful difference, while a narrow interval centered near zero would suggest that any effect is small.

Reporting Every Test

If I examine phone dependence as a possible moderator, that test will be reported as exploratory with its own effect size, so that readers do not mistake a pattern found by searching for one predicted in advance.

Rounding and Precision

I will report exact p-values to two or three decimals and effect sizes to two, avoiding false precision and the misleading habit of reporting "p < .05" alone.

Limits

Benchmarks for small, medium and large effects are rough conventions, and what counts as meaningful depends on context. The hypothetical results illustrate interpretation only.

Conclusion

Wasserstein and Lazar explained what p-values can and cannot tell us, Lakens showed how to calculate and report effect sizes with confidence intervals and Cohen showed why power must be planned before data collection. Reporting effect sizes and intervals alongside p-values, and planning the sample honestly, will let my study say not only whether phones matter but how much.

References

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. https://doi.org/10.1037/0033-2909.112.1.155

Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, Article 863. https://doi.org/10.3389/fpsyg.2013.00863

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. https://doi.org/10.1080/00031305.2016.1154108