PSY 3220 Module 2 Psychological Measurement Paper: Reliability, Construct Validity and Measuring Distraction Example

Measuring What We Mean: Reliability and Validity for a Study of Phones and Focus

Student Name

Bachelor of Arts in Psychology, Thomas Edison State University

PSY-3220: Research in Experimental Psychology

Mentor Name

Month Day, Year

Measuring What We Mean: Reliability and Validity for a Study of Phones and Focus

Introduction

My research question asks whether a phone's presence reduces reading comprehension among adult online learners. Before I can answer, I need measures that capture comprehension, phone dependence and anxiety accurately. A weak measure might mask an effect that exists or suggest one that does not. This paper explains the concepts of reliability and validity and applies them to the measures I plan to use.

Reliability

Reliability is about consistency: measuring the same thing twice under similar conditions should yield similar scores. For traits expected to stay steady, test-retest reliability compares scores from two occasions. Internal consistency asks whether items on a scale measure the same thing, often summarized with Cronbach's alpha. Interrater reliability asks whether different observers score the same behavior the same way. Without reliability there can be no validity, because random scores cannot track anything real, yet reliability alone does not guarantee validity, since a scale may be consistently off target.

Construct Validity

Cronbach and Meehl (1955) introduced the idea of construct validity for measures of qualities, such as anxiety or intelligence, that cannot be observed directly. Because there is no single criterion to compare such a measure against, validity must be established through a network of evidence. A construct is defined by its expected relationships with other constructs and behaviors, a set of relationships they called a nomological network. If scores on a new anxiety measure relate to other variables as theory predicts, for example rising before exams and relating to physiological arousal, that pattern supports the claim that the measure captures anxiety. Construct validation, in their view, is an ongoing process of testing the theory and the measure together.

Convergent and Discriminant Evidence

Campbell and Fiske (1959) proposed the multitrait-multimethod matrix, which measures several traits by several methods. Convergent validity is shown when different methods of measuring the same trait correlate highly, for example when self-reported and observer-rated anxiety agree. Discriminant validity is shown when measures of different traits do not correlate too highly, especially when they use the same method. Their approach also exposes method variance, shared variance that comes from using the same method, such as self-report questionnaires, rather than from the traits themselves.

How Well Researchers Do This

Flake et al. (2017) reviewed scales used in a sample of articles in a leading social and personality psychology journal. Many scales were reported with little more than a coefficient alpha, and some were created for the study or modified without any reported evidence of validity. The authors argued that weak measurement practices threaten the credibility of findings and recommended that researchers report the source, development and validity evidence for every scale, especially new or adapted ones.

My Measures

Reading comprehension: I will use passages and multiple-choice questions from a published, standardized reading test designed for adults, which reports alternate-form reliability and correlations with other reading measures. Using a standardized test avoids creating a new measure with unknown properties, the problem Flake and colleagues describe. Two parallel forms allow a practice session and a test session with different passages.

Phone dependence: I will use the same published self-report scale Ward and colleagues used, which allows comparison with their results. I will report its internal consistency in my sample.

Anxiety: I will measure state anxiety with a brief published scale before and after the reading task, to see whether separation from the phone raises anxiety, an alternative explanation identified in the first module.

Avoiding Method Variance

All three self-report measures share a method, so Campbell and Fiske's concern applies. The main outcome, comprehension, is a performance test rather than a self-report, which reduces the risk that shared method alone produces a relationship between phone dependence and the outcome.

Pilot Testing

Before the main study, I will pilot the procedure with eight volunteers to check that passages take about 15 minutes, instructions are clear and scores do not cluster at the top or bottom of the scale, problems known as ceiling and floor effects.

Reliability in My Sample

Published reliability figures describe the samples in which a measure was developed, not necessarily mine. Adult learners may answer a phone dependence scale differently from the undergraduates in the original study. I will calculate internal consistency for each scale in my own data and report it, and I will check whether the two forms of the reading test produce similar average scores in the pilot.

Validity of the Manipulation

Measurement also applies to the independent variable. A manipulation check, asking participants where their phone was during reading, will confirm that each condition was delivered as planned, so that any differences in comprehension can be attributed to phone location rather than to participants ignoring instructions.

Floor and Ceiling Effects

If the passages are too easy, most participants will answer nearly every question correctly, compressing scores at the top and hiding any difference between conditions. If they are too hard, scores will bunch at the bottom. The pilot will show whether average scores fall near the middle of the range, and passages can be swapped before the main study if they do not.

Limits

Standardized reading tests measure comprehension of short passages, which may differ from studying course material. Self-report measures depend on honesty and insight. No measure is perfectly valid.

Conclusion

Cronbach and Meehl showed that validity for abstract concepts rests on a network of expected relationships, Campbell and Fiske showed how convergent and discriminant evidence and attention to method variance strengthen that case and Flake and colleagues showed how often researchers neglect validity evidence. Choosing established measures, reporting their reliability and validity, pairing self-report with performance and piloting the procedure give my study measures that mean what I intend.

References

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81-105. https://doi.org/10.1037/h0046016

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302. https://doi.org/10.1037/h0040957

Flake, J. K., Pek, J., & Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370-378. https://doi.org/10.1177/1948550617693063