NUR 7100 Module 2 Item Writing Paper: Reviewing a Test Bank Against Published Guidelines Example

Clues, Negatives and All of the Above: Auditing a Fundamentals Test Bank in Somerset County

Student Name

M.S. in Nursing, Nurse Educator, Thomas Edison State University

NUR-7100: Testing, Assessment, and Evaluation

Mentor Name

Month Day, Year

Clues, Negatives and All of the Above: Auditing a Fundamentals Test Bank in Somerset County

Why Audit

The program, faculty and items described here are invented. A fundamentals of nursing course in a Somerset County program uses a test bank of about 400 items, written over a decade by many faculty and drawn into four unit exams and a final. Students often challenge items after exams, and faculty spend hours debating answers. The fundamentals coordinator, partway through a nurse educator MSN, proposed an audit of the 120 items used on the most recent unit exams before the next semester.

Published Guidelines

Haladyna et al. (2002) reviewed measurement textbooks and research studies to identify guidelines for writing multiple-choice items for classroom tests. Their 31 guidelines fall under five headings, from content and format to style, the stem and the answer choices. Among them: focus each item on one important idea; avoid trivial content; write the stem so that the problem is clear without reading the options; word stems positively and avoid negatives such as "except" or "not"; keep options alike in length and grammar; steer clear of an "all of the above" option and treat "none of the above" cautiously; make every distractor plausible; and avoid clues such as words repeated from the stem in the correct option. They noted that research suggests three well-written options are often enough and that writing many weak distractors adds little.

What Flaws Do

Downing (2005) examined medical school examinations and compared items that violated standard writing principles with items that did not. Flawed items were common and were more difficult for students than standard items testing similar content. When he recalculated pass and fail decisions using only standard items, a meaningful share of students who had failed the examinations would have passed. Flawed items added difficulty that came from the item, not from the content, and so produced decisions that did not reflect what students knew.

Tarrant and Ware (2008) reviewed the high-stakes exams of one nursing school and found writing flaws in a large share of the items. Flawed items worked against high-achieving students, who were more likely than other students to be penalized by them, while some borderline students passed who might not have otherwise. The authors argued that faculty need training in item writing and that examinations should be reviewed before use.

The Audit

The coordinator and two colleagues reviewed each of the 120 items independently against a checklist of the guidelines most relevant to nursing items, then met to agree. They found at least one flaw in 71 items. The most common were negatively worded stems, 22 items; options of uneven length, with the correct answer longest, 18; options reading "all" or "none" of the above, 14; stems that could not be answered without reading the options, 13; implausible distractors, 12; and clues repeated from stem to answer, 9. Some items had more than one flaw. Thirty-one items tested recall of facts that the course outcomes did not require.

Revising Items

Two examples show the work. One item asked which action was not appropriate when giving an enema, with four options. Students who misread the negative chose a correct action. The revision asked which action best prevents a specific complication in a described patient, with three plausible options. A second item asked for the normal adult respiratory rate. Because the course outcomes required recognizing changes in a patient's condition, the revision described a patient whose respiratory rate had risen over the shift and asked what the nurse should do first.

The team revised 52 items and retired 19 that tested trivial content, replacing them with new items written to the course blueprint. Each revised item was reviewed by a second faculty member who had not written it.

Distractors That Teach

Writing plausible distractors was the hardest part of revision. The team found that the best distractors came from students' actual mistakes. Faculty collected common wrong answers from short-answer questions in the skills lab and from clinical conferences, such as believing that a patient with a fever should always be given fluids quickly or that raising the bed is always the right opening move when breathing worsens. Distractors built from these misconceptions attracted weaker students and gave faculty information about what students misunderstood. Distractors invented only to fill a slot, such as an obviously harmful action, were dropped, which is why many revised items have three options instead of four.

A Review Process for the Future

The faculty agreed on three practices. Every new item is reviewed by a colleague against the checklist before it enters the bank. Each exam is reviewed as a whole for balance and clues across items, such as one item giving away the answer to another. After each exam, item statistics are reviewed, and any item with unexpected results returns to the checklist. Faculty attend a two-hour workshop on item writing each year, using the program's own items as examples.

Preparing for Clinical Judgment Items

The licensing examination now uses case-based items that ask students to recognize and analyze cues, prioritize hypotheses and take action. The faculty agreed that clean single-best-answer items remain useful, but that each unit exam should include at least one short case with several linked items, written with the same care, so that students practice the reasoning they will meet on the licensing exam.

Students' Experience

In the semester after the audit, item challenges fell from an average of 14 per exam to 4, and faculty spent less time debating answers. Students said questions were clearer, though several noted that the new scenario items took longer to read.

Limits

The audit relied on faculty judgment, and reviewers sometimes disagreed about whether a distractor was plausible. Fewer challenges may reflect students' trust in the review rather than better items alone. Revised items need item statistics from several semesters before they can be judged.

Conclusion

A test bank written by many hands carried flaws in more than half of its audited items. Published guidelines gave the faculty a checklist, and studies of flawed items showed that the flaws changed which students passed and penalized the strongest students. Revising items, retiring trivia and building a review process make the program's exams fairer measures of what students know.

References

Downing, S. M. (2005). The effects of violating standard item writing principles on tests and students: The consequences of using flawed test items on achievement examinations in medical education. Advances in Health Sciences Education, 10(2), 133-143. https://doi.org/10.1007/s10459-004-4019-5

Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309-333. https://doi.org/10.1207/S15324818AME1503_5

Tarrant, M., & Ware, J. (2008). Impact of item-writing flaws in multiple-choice questions on student achievement in high-stakes nursing assessments. Medical Education, 42(2), 198-206. https://doi.org/10.1111/j.1365-2923.2007.02957.x