NUR 6310 Module 6 Evaluation Paper: Drug Interaction Alerts and the DeLone and McLean Success Model Example

Ninety Percent Ignored: Evaluating Drug Interaction Alerts One Year After Go-Live in Lawrenceville

Student Name

M.S. in Nursing, Nursing Informatics, Thomas Edison State University

NUR-6310: Nursing Informatics: Systems Life Cycle

Mentor Name

Month Day, Year

Ninety Percent Ignored: Evaluating Drug Interaction Alerts One Year After Go-Live in Lawrenceville

One Year Later

The hospital and its data are invented. A year after a 320-bed hospital in Lawrenceville moved to a new electronic health record, the medication safety committee asked for an evaluation of drug interaction alerts. Nurses and prescribers complained of constant interruptions, and pharmacy had noticed that serious interactions were sometimes ordered with an alert accepted in seconds. The informatics nurse pulled twelve months of alert data: 186,000 drug interaction alerts fired, 90 percent were overridden, and the median time spent on an alert before overriding was three seconds. Nurses received many of these alerts at medication administration, after a prescriber and a pharmacist had already overridden them.

Defining Success

DeLone and McLean (2003) updated their model of information systems success after a decade of research. The model has six dimensions: information quality, system quality, service quality, intention to use and use, user satisfaction and net benefits. Quality affects use and satisfaction, which together produce benefits, and benefits feed back into use and satisfaction. The model warns against judging a system by use alone, because clinicians may use a system heavily because they must, not because it helps.

Applied to the alerts, the model showed a split. Use was universal: every alert was seen. Information quality was poor: most alerts concerned interactions of little clinical importance. User satisfaction was low. Net benefits were uncertain, because the alerts that mattered were buried among those that did not.

Why Alerts Are Overridden

Pooling the published research on how clinicians respond to drug safety warnings, one review reported that between 49 and 96 percent of such warnings were overridden (van der Sijs et al., 2006). Common reasons included alerts of low clinical relevance, repeated alerts for the same combination, alerts that appeared too late in the workflow and the sheer number of alerts, which produced alert fatigue. The authors warned that high override rates risk serious alerts being ignored along with the trivial ones, and recommended improving the specificity of alerts, tiering them by severity and directing each one to whoever can act on it, when they can act.

Principles for Decision Support

Bates et al. (2003) offered ten commandments for effective clinical decision support, drawn from experience at a large academic center. Speed is everything. Anticipate needs and deliver in real time. Fit the way users already work. Little things can make a big difference. Recognize that physicians will strongly resist stopping. Changing direction is easier than stopping. Simple interventions work best. Ask for additional information only when you really need it. Monitor impact, get feedback and respond. Manage and maintain the knowledge-based systems.

The Lawrenceville alerts broke several: they interrupted every order regardless of severity, stopped rather than redirected and had no process for monitoring or maintenance. The last two commandments describe the evaluation and maintenance phase itself.

Recommendations

The informatics nurse and the pharmacy informaticist reviewed every interaction pair that fired more than 100 times in the year with a panel of two pharmacists, a hospitalist and a nurse. They recommended three tiers. Contraindicated combinations, about 3 percent of current alerts, become hard stops that require a pharmacist's call to override. Serious interactions, about 15 percent, remain interruptive for prescribers, with a suggested alternative or monitoring order built in, so the prescriber can change direction rather than only stop. All other interactions become non-interruptive information visible in the order details. Nurses no longer see interaction alerts at administration unless the pair is contraindicated and the order was entered without a pharmacist override.

Nurses and Alerts

The evaluation paid particular attention to nurses, who are often left out of decision support design. At medication administration, nurses saw interaction alerts for combinations that a prescriber had ordered and a pharmacist had verified, with no information about why either had accepted the risk. Nurses could not change the order and could only acknowledge the alert or hold the dose and call. Interviews with twelve nurses showed that most acknowledged these alerts automatically, and several said they no longer read them. Showing nurses only contraindicated combinations that bypassed pharmacist review, along with the reason any override was given, puts the alert in front of the person who can act on it at the moment it matters.

Testing the Changes

Before the tiers went live, the pharmacy informaticist ran the previous month's orders through the new configuration in a test environment. Interruptive alerts fell by 81 percent, and every contraindicated pair in the sample still produced a hard stop. The panel reviewed a random sample of the interactions moved to the information tier and moved six pairs back to the interruptive tier.

Governance and Maintenance

An alert governance group of pharmacy, nursing, medicine and informatics will meet quarterly, review override data and accept requests from staff to change alerts through a simple online form. Each new alert will have a named owner, a stated purpose and a review date. Drug database updates will be reviewed before release for new interaction pairs.

Measures

The hospital will track the number of interruptive alerts per 100 orders, override rates by tier, the share of contraindicated combinations dispensed, adverse drug events involving interactions and user satisfaction with alerts every six months.

Limits

Reducing alerts may let a rare but serious interaction pass if it is placed in the wrong tier. Override rates do not show whether an override was clinically justified. The success model was developed in general information systems and is not specific to health care.

Conclusion

A year after go-live, drug interaction alerts were used by everyone and helpful to few. The success model separated use from benefit, the override review explained alert fatigue and the commandments pointed to tiered, workflow-friendly alerts with ongoing maintenance. Evaluation is not the end of the life cycle but the start of the next round of improvement.

References

Bates, D. W., Kuperman, G. J., Wang, S., Gandhi, T., Kittler, A., Volk, L., Spurr, C., Khorasani, R., Tanasijevic, M., & Middleton, B. (2003). Ten commandments for effective clinical decision support: Making the practice of evidence-based medicine a reality. Journal of the American Medical Informatics Association, 10(6), 523-530. https://doi.org/10.1197/jamia.M1370

DeLone, W. H., & McLean, E. R. (2003). The DeLone and McLean model of information systems success: A ten-year update. Journal of Management Information Systems, 19(4), 9-30. https://doi.org/10.1080/07421222.2003.11045748

van der Sijs, H., Aarts, J., Vulto, A., & Berg, M. (2006). Overriding of drug safety alerts in computerized physician order entry. Journal of the American Medical Informatics Association, 13(2), 138-147. https://doi.org/10.1197/jamia.M1809