NUR 7100 Module 3 Item Analysis Paper: Difficulty, Discrimination and Reliability After a Pharmacology Exam Example

What the Numbers Say About Each Question: Item Analysis of a Hunterdon County Pharmacology Exam

Student Name

M.S. in Nursing, Nurse Educator, Thomas Edison State University

NUR-7100: Testing, Assessment, and Evaluation

Mentor Name

Month Day, Year

What the Numbers Say About Each Question: Item Analysis of a Hunterdon County Pharmacology Exam

The Exam

The program, the exam and every statistic here are invented. Sixty-two students in a Hunterdon County program took a 75-item multiple-choice pharmacology exam covering cardiovascular, endocrine and anti-infective drugs. The class average was 76 percent, and 14 students scored below the 77 percent passing score. The testing software produced an item analysis report. The pharmacology coordinator, who is completing an educator-track MSN, used the report to decide whether any items should be rescored before grades were released and which items needed work for future exams.

Difficulty and Discrimination

Ebel and Frisbie (1991) described two basic item statistics. The difficulty index, despite its name, rises as items get easier, because it is simply the share of students who got the item right. The discrimination index compares the performance of high-scoring and low-scoring students on the item; an item discriminates well when students who did well on the whole test answer it correctly more often than students who did poorly. They offered guidance for interpreting discrimination: values of about 0.40 and above indicate very good items, values from about 0.30 to 0.39 reasonably good items, values from about 0.20 to 0.29 marginal items that may need improvement and lower values poor items that should be revised or discarded. They cautioned that very easy or very hard items cannot discriminate much and that statistics should prompt review of an item rather than replace judgment.

Reliability

Kuder and Richardson (1937) derived formulas for estimating the reliability of a test from a single administration. The formula known as KR-20 applies to items scored right or wrong and estimates internal consistency, the degree to which items measure the same thing. Coefficient alpha generalizes the idea to items scored in other ways.

Tavakol and Dennick (2011) explained how to interpret alpha. Alpha rises with test length and with how closely the items hang together; a short test or a test covering very different content will have a lower alpha. Many sources treat a range of roughly 0.70 to 0.95 as acceptable, but very high values may mean that items are redundant. Alpha describes the scores from a particular group of students, not the test itself, and it does not show that a test is measuring what it is supposed to measure.

What the Report Showed

Reviewing Items

The coordinator reviewed each flagged item with a pharmacology colleague. The two items with negative discrimination, where weaker students outscored stronger ones, were examined first. In one, the key was wrong: the item had been revised the previous year, and the answer key had not been updated. In the other, the stem asked what to do first for a hypoglycemic patient receiving an insulin drip, and two options were defensible depending on whether the patient could swallow, which the stem did not say. Both items were rescored, the first by correcting the key and the second by accepting both defensible answers.

Of the four very hard items, one asked about a drug interaction the course had not covered; it was removed from scoring. The other three covered taught content and discriminated well, so they were kept. Among the nine low-discrimination items, a distractor analysis helped. In one item, almost no student chose two of the four options, so the item worked as a two-choice question; it was kept for this exam and marked for revision. In another, high scorers split between the key and one distractor, which on review was nearly as correct; the item was marked for rewriting.

Patterns Across Topics

The coordinator also grouped items by topic. Students did well on cardiovascular drugs and poorly on endocrine drugs, where average difficulty was 0.58 compared with 0.81 for cardiovascular items. Discrimination on endocrine items was acceptable, which suggested that the items were working and that the content had not been learned as well. Reviewing the course schedule, she found that endocrine drugs had been taught in a single week just before spring break. The next term spreads the topic over two weeks with a practice case, an example of item analysis informing teaching as well as the test.

Decisions About Scores

After rescoring, the class average rose slightly, and two students moved from below to above the passing score. The coordinator documented each decision and the reason in a memo to the course file, so that the decisions could be explained to students and reviewed by colleagues.

Explaining Results to Students

The coordinator posted a short note explaining that three items had been changed after review and why, without revealing the items. Students who asked for a review of their exam could see their responses in a supervised session.

Building the Item Bank

Each item's statistics are recorded in the bank with the semester. Items that discriminate well across several semesters are kept; items that remain marginal after revision are retired. Very easy items that test essential safety knowledge, such as a dangerous dose, are kept even though they discriminate little, because faculty want every student to answer them correctly.

Limits

Item statistics from 62 students are unstable, and one semester's results should not decide an item's fate. Discrimination indices assume the total score is a good measure of the ability being tested. Reliability does not establish validity.

Conclusion

The item analysis report pointed to a miskeyed item, an ambiguous item, an untaught item and several weak distractors. Ebel and Frisbie's guidance on difficulty and discrimination, Kuder and Richardson's reliability formula and Tavakol and Dennick's cautions about alpha helped the coordinator use the statistics as clues for review. Rescoring with documented reasons and building item history into the bank make the next exam stronger.

References

Ebel, R. L., & Frisbie, D. A. (1991). Essentials of educational measurement (5th ed.). Prentice Hall.

Kuder, G. F., & Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2(3), 151-160. https://doi.org/10.1007/BF02288391

Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach's alpha. International Journal of Medical Education, 2, 53-55. https://doi.org/10.5116/ijme.4dfb.8dfd