NUR 7100 Module 1 Test Validity Paper: A Test Blueprint and Validity Evidence for a Medical-Surgical Final Example

What the Final Should Measure: A Blueprint and Validity Evidence for a Middlesex County Medical-Surgical Examination

Student Name

M.S. in Nursing, Nurse Educator, Thomas Edison State University

NUR-7100: Testing, Assessment, and Evaluation

Mentor Name

Month Day, Year

What the Final Should Measure: A Blueprint and Validity Evidence for a Middlesex County Medical-Surgical Examination

The Complaint

Nothing in this Middlesex County example is real. A baccalaureate nursing program in Middlesex County gives a 100-item final examination at the end of its second medical-surgical course. After last semester's exam, students submitted a written complaint: many questions asked for normal laboratory values, drug classifications and definitions, while the course had emphasized prioritizing care for patients whose conditions change. Faculty reviewed the exam and found that it had grown by accretion, each instructor adding questions over the years, with no plan behind it. The course's coordinator, who is earning her master's in nursing education, was asked to rebuild the examination and show that its scores mean what the faculty intend.

What Validity Means

Messick (1995) held that validity belongs to the inferences drawn from scores rather than to the test, and that judging it means weighing how much evidence and theory back each interpretation and use. He described a unified concept of validity built around construct validity and identified six aspects of it: content, the relevance and representativeness of what the test covers; substantive, whether test takers engage the intended thinking; structural, whether scoring reflects the structure of the construct; generalizability, whether score meaning holds across tasks and settings; external, the relationships of scores with other measures; and consequential, the intended and unintended effects of score use. The view moved validation from a single coefficient to an argument.

Sources of Evidence

Downing (2003) translated this view for health professions educators. Drawing on the joint testing standards, he described five sources of validity evidence: content, such as a blueprint and expert review of items; response process, such as how students think when answering and the accuracy of scoring; internal structure, such as item statistics and reliability; relationship to other variables, such as correlation with other measures of the same ability; and consequences, such as the effects of pass and fail decisions. He argued that higher-stakes decisions require more evidence and that all assessments need some.

The joint testing standards (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014) set out the expectations behind these sources, including the expectation that test developers document the purpose of a test, the content specification and evidence that scores support the intended uses.

The Blueprint

The coordinator began with the course's six outcomes and the time spent on each. With two colleagues, she built a blueprint that set the number of items for each outcome and the cognitive level, using the revised taxonomy's categories, so that most items require applying and analyzing rather than remembering.

Under the old exam, 58 percent of items tested remembering. Under the blueprint, 17 percent do.

Gathering Evidence

Content evidence. Two faculty members who had not written the items reviewed each one against the blueprint and the course outcomes, rating relevance and level. Items they rated as irrelevant or at the wrong level were revised or replaced.

Response process evidence. Five students from the previous cohort, who had passed the course, talked aloud while answering twenty items. Their explanations revealed four items that could be answered by test-taking clues rather than nursing knowledge, such as an option much longer than the others, and two items where the intended answer was arguable. All six were revised.

Internal structure evidence. After the exam, the coordinator calculated item difficulty and discrimination and an overall reliability coefficient, which are reported in the course's item analysis file.

Relationship to other variables. Scores on the new exam were compared with students' scores on a standardized medical-surgical examination taken the same week; a moderate positive correlation supported the interpretation that both measure related knowledge.

Consequences. Faculty reviewed whether the pass score produced reasonable decisions by looking at students near the cut score and their clinical evaluations.

Setting the Pass Score

The old exam used a fixed passing score of 77 percent, the program's minimum course grade, with no review of whether that score represented the competence faculty expected. Using the blueprint, three faculty members judged, item by item, the probability that a minimally competent student, one who would be safe in a supervised clinical setting, would answer correctly. The average of their judgments suggested a cut score of 74 percent for the new, harder exam. The faculty decided to keep the program's 77 percent minimum for the course grade but to use the judgment exercise to check that the new exam did not unfairly fail students who were performing safely in clinical. This standard-setting step adds consequence evidence to the validity argument, because it ties the pass decision to a defined level of performance rather than to habit.

What Changed

The new exam has fewer items about normal values and more short patient scenarios in which students must decide what to do first. Faculty now meet before each exam to review new items against the blueprint, and the item bank tags each item by outcome and level.

Students' Response

The coordinator shared the blueprint with students at the start of the next course, showing how much of the exam each outcome would represent. Students said it helped them study, and complaints about trivia stopped. Some students found the new exam harder because it asked them to apply knowledge rather than recall it.

Limits

Five students talking aloud is a small sample for response process evidence. Correlation with a standardized exam supports but does not prove that the two measure the same thing. A single semester's data are not enough to judge reliability across cohorts.

Conclusion

An exam that grew without a plan tested what was easy to ask rather than what the course taught. Messick's view of validity as an argument, Downing's sources of evidence and the testing standards gave the faculty a way to rebuild the exam from outcomes and to show that its scores support their intended use.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

Downing, S. M. (2003). Validity: On the meaningful interpretation of assessment data. Medical Education, 37(9), 830-837. https://doi.org/10.1046/j.1365-2923.2003.01594.x

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749. https://doi.org/10.1037/0003-066X.50.9.741