Overriding of Drug Safety Alerts in Computerized Physician Order Entry

Abstract

Many computerized physician order entry (CPOE) systems have integrated drug safety alerts. The authors reviewed the literature on physician response to drug safety alerts and interpreted the results using Reason's framework of accident causation. In total, 17 papers met the inclusion criteria. Drug safety alerts are overridden by clinicians in 49% to 96% of cases. Alert overriding may often be justified and adverse drug events due to overridden alerts are not always preventable. A distinction between appropriate and useful alerts should be made. The alerting system may contain error-producing conditions like low specificity, low sensitivity, unclear information content, unnecessary workflow disruptions, and unsafe and inefficient handling. These may result in active failures of the physician, like ignoring alerts, misinterpretation, and incorrect handling. Efforts to improve patient safety by increasing correct handling of drug safety alerts should focus on the error-producing conditions in software and organization. Studies on cognitive processes playing a role in overriding drug safety alerts are lacking.

Computerized physician order entry (CPOE) systems frequently include integrated decision support components, which can reduce errors and improve patient safety.1–6 Studies documenting positive effects of decision support on patient outcomes, including fewer duplicate orders, fewer overdoses, fewer allergic reactions, and reduced drug interactions, have prompted calls for additional safety-related, patient-specific advice.1,3,5 Yet, the burden of reminders and alerts must not be too high,1,2,6–8 or “alert fatigue” may cause clinicians to override both important and unimportant alerts,2,9 in a manner that compromises the desired safety effect of integrating decision support into CPOE.

This review attempts to provide insight into physicians' handling of safety alerts by asking the following questions: How often and in what situations are safety alerts overridden? Why do physicians override them? What effects ensue? What understanding of alert overrides can lead to improved alerting systems? The authors employ Reason's model of accident causation10 to understand overriding and its effects and to suggest new directions to improve alerting.

Methods

The MEDLINE and EMBASE databases from January 1980 to December 2004 were searched for English-language publications with the following MeSH headings and text words: computerized physician (medication) order entry, CPOE, electronic prescribing, computerized prescribing, medical record systems computerized and alert*, remind*, prompt*, order check, critic*, critiq*, decision support systems clinical, reminder systems, drug therapy computer assisted and overrid*, medical error, adverse drug events, and attitude. The authors also checked literature references of three recent systematic reviews and one synthesis of review paper.1,3,5,11

The authors selected publications discussing overriding of unsolicited drug safety alerts that appear during the prescription process because automatic provision of alerts has been proven to be a critical feature for changing clinician behavior.12 The term computerized physician order entry is used because interpretation and handling of drug safety alerts requires medical expertise. Full articles were included, but also proceedings when pertinent. The references of these publications were checked also. The refined selection was used for the first part of this review. To learn how alerting could be improved, the authors examined all publications from the search for characteristics of unsolicited safety alerts as well as measures to minimize error-producing conditions.

Results

Seventeen publications on overriding safety alerts in CPOE were identified2,13–28 (Table 1). Quantitative information on overriding was present in nine publications.13–21 This small yield does not pose a problem because the review focuses mainly on the conceptual analysis of the determinants of overriding. For the second part of this article, we selected from 193 papers of the first search those that described characteristics of safety alerts.

How Often and in What Kinds of Situations Are Safety Alerts Overridden?

Papers discussing percentages of overridden alerts of different types are summarized in Table 2.13–21 Except for serious alerts for overdose, which are overridden in one fourth of all alerts, safety alerts are overridden in 49% to 96% of cases. Taylor and Tamblyn21 show lower override rates for interactions (35%) and contraindications (43%), but this seems to be caused by an extra toxicity category also including interactions and contraindications. Bates et al.11 propose a maximum override rate of 40% but do not offer an explanation for this figure.

Low-level alerts appear to be overridden more often than high-level alerts (serious alerts), but this could not be completely confirmed in a study with three levels of alerts.13,14,19 Moreover, alert levels cannot be compared between studies because standardization of alert levels is absent. None of these quantitative studies discusses the relationship between different levels of alerts and override rates.

One study showed an increase in override rates from about 50% to 75% during a five-year period, indicating a declining compliance to safety alerts.14 A relationship between relative amount of alerts and percentages overridden cannot be observed, but this may be due to the small number of studies.13,16–19,21 High override rates were observed in drug renewals, in drug interactions with topical drugs, and in poorly defined drug allergies.14,16,19,20

Factors That Play a Role in Overriding

Three studies elucidated factors playing a role in overriding alerts by physicians in outpatient care.21–23,28 The most important reason for overriding was alert fatigue caused by poor signal-to-noise ratio because the alert was not serious, was irrelevant, or was shown repeatedly.2,21,22,28 Alert fatigue is as yet not thoroughly studied but is described as the mental state that is the result of too many alerts consuming time and mental energy, which can cause important alerts to be ignored along with clinically unimportant ones.9 Other reasons include the importance of the treatment not allowing a drug change, physicians' faith in their own knowledge or other information sources obtained, incorrect information, patients' resistance to drug change or lack of time.21,22 It was also mentioned that alerts were too long and difficult to interpret and that clinical consequences were not clear.23,28 Twenty-two percent of general practitioners admitted drug interaction overriding without checking.22 In a study on corollary orders, reasons not to accept reminders included inappropriate orders, disagreement with the guidelines, and lack of time.24 Lack of understanding about importance of the warning, technological problems, and unnecessary workflow interruptions also thwart correct and effective handling of safety alerts.2,26,27

Two studies reviewed appropriateness of the alerts and revealed that 36.5% and 39% of the alerts were false positive.17,19 Reviewers agreed with clinicians' decisions in 95.6% of cases where physicians overrode a valid alert.19 Oppenheim et al.17 found that 48% of the true positive alerts were overridden.

Effect of Overridden Alerts

The direct effect of overridden alerts on safety is mentioned in three publications. Adverse events were observed in 2.3%, 2.5%, and 6% of the overridden alerts, respectively, in studies with override rates of 57%, 90%, and 80%.15,19,20 Adverse events were preventable in 0.8% and none of the overrides, respectively.19,20

A high override rate can also indirectly impair patient safety. Too many alerts with low credibility may cause physicians to override important alerts along with unimportant ones. A high override rate might also result in the hospital decision to turn off a whole group of alerts, including relevant alerts, or in decreased user acceptance and distrust in both the alerting system and CPOE.16,22,27,29 Monitoring of overriding is said to be necessary to keep the override rate within acceptable limits and to ensure user trust and responsiveness to alerts.14

Understanding the Effect of Overriding Drug Safety Alerts on Patient Safety

Integrated decision support should prevent patients from receiving the wrong drug or the wrong dose when prescription errors are made. However, not all errors are caught because alerts are turned off, are not read, are misinterpreted, or are wrongly overridden. In Figure 1, the process of order entry, interpretation, and handling of drug safety alerts and the emergence of medication errors are presented schematically.

Figure 1 may help explain why overriding does not always result in a medication error. There can be good reasons for overriding (justified overriding), for example, when the benefits of the drug (combination) outweigh the disadvantages and potential adverse effects can be monitored.19 Conversely, a cancellation or change of a drug order due to a drug safety alert can itself result in a medication error. Overriding a safety alert is often seen as a problem in itself, a system violation, but it should be emphasized that only unjustified overriding (ignoring alerts, misinterpretation, wrong selection) poses a problem.

Justified overriding may be patient related or can occur when an alert is based on erroneous patient information. Patient-related reasons include, for example, clinically insignificant alerts, a limited treatment course, patient tolerance of the medication or dose in the past, discussion of potential adverse events with the patient or monitoring thereof, absence of a good alternative, and the benefits of the drug outweighing the disadvantages.19 Examples of erroneous patient information, justifying overriding, include inaccurate allergy information or medication lists that are out of date.19,30 Appropriate alerts can be defined as true positive alerts, alerts that are correct and current for the patient at hand. It does not imply that appropriate alerts are always perceived as useful.

The authors suggest that problems of safety alert overriding can be explained with the help of Reason's model of accident causation. This model is applicable to complex sociotechnical systems that require coordination of a large number of human and technologic elements and focuses on person, team, task, workplace, and organization.10,31 Alerting systems in CPOE are an example of such a complex sociotechnical system.

Reason distinguishes between active failures, error-producing conditions, and latent conditions.10,31–33 Active failures are errors (slips, lapses, and mistakes) and violations of an individual having an immediate adverse effect. Error-producing conditions are factors that affect performance of individuals, thus provoking active failures. These factors can originate in the environment, team of care providers, individual, or task at hand. Latent conditions are defensive gaps, weaknesses, and absences that are unwittingly created as the result of earlier decisions made by system designers, builders, regulators, and managers. Latent conditions can originate from organizational processes or management decisions. Reason's model shows that accidents result from a concatenation of several contributing factors at different levels: active failures, error-producing conditions, and latent conditions; individual and organizational factors. Simultaneous alignment of gaps or absences within diverse and redundant defenses results in accidents.10,31,32,34

Figure 2 shows how active failures leading to medication errors are the result of error-producing conditions in alerting system and physician and latent conditions in the organization.35 It shows how suboptimal decision support can reduce physicians' motivation, thus provoking active failures in alert handling.

Studies that linked overriding to adverse events showed that overriding did not result in adverse drug events in more than 97% of cases.15,19,20,Figure 1 shows that overrides might result in adequate therapy as well as medication errors. A medication error is any error in the process of prescribing, dispensing, or administering a drug, whether or not there are adverse consequences.36 Unjustified overriding of a drug safety alert (medication error) does not necessarily have adverse consequences. Overriding may, for example, result in suboptimal treatment or may be annulled by a dispensing or administration error. Conversely, adverse drug events can also result from justified overriding,20 or intrinsic drug toxicity.37 The relationship between overridden alerts and adverse drug events, presented in Figure 3, shows that appropriate alerts can be overridden and that overriding does not necessarily result in adverse drug events. It shows also that justified overriding (adequate therapy) cannot always prevent adverse events.

Two studies describe how expert panels review patient charts and score adverse events.15,19 For justification of overriding, it is easier to score override decisions, which gives more information about reasons.20,38 Requiring entry of reasons for overriding is more feasible for daily practice, triggers physicians to rethink the potential unsafe situation, gives pharmacists and other caregivers insight in the considerations, and can help adjust the knowledge database.

Alerting systems may contain several error-producing conditions increasing the risk of a medication error.

Improving Alerting

A safe alerting system has high specificity and sensitivity, presents clear information, does not unnecessarily disrupt workflow, and facilitates safe and efficient handling. Several qualitative studies of safety alerts in CPOE present requirements and suggestions for alerting systems.2,7,9,12,19,20,22,26,28,39–41 Alert factors applicable to unsolicited drug safety alerts are presented in Table 3, grouped by the five items for a safe alerting system and classified as necessary for appropriate alerts or useful for safe and efficient interpretation and handling.

Improvement of alerting systems should focus on prevention of active failures and individual error-producing conditions (distrust, alert fatigue, lack of motivation, and lack of time), by reducing or removing error-producing conditions of system, task and team, and latent conditions. Error-producing conditions have a more direct effect on active failures and can be influenced more easily than latent conditions.

If physicians ignore alerts, they often do so because alerts are inappropriate.2,22 The authors suggest that the first step to reduce the frequency of ignoring alerts should be reducing the number of inappropriate (false positive) alerts and to direct certain alerts to other care providers, thus preventing unnecessary disruptions in physicians' workflow. After increasing appropriateness, sensitivity, usefulness, and usability should be improved. Improving sensitivity may result in a unacceptable time burden for the physician if interpretation and handling are inefficient. If sensitivity is low and handling efficient, sensitivity should first be increased. When error-producing conditions are reduced or removed, latent conditions should be addressed.

Increasing Specificity

To increase the percentage of appropriate alerts (specificity), irrelevant and nonurgent alerts should not be shown and alerts should be patient tailored. Alerts that should not be shown are interactions between systemic and topical drugs, alerts of drug allergy in case of medication intolerance, and alerts without urgent or possible action.14,16

To tailor alerts to the patient at hand, age, gender, body weight, allergies, mitigating circumstances, and drug serum levels should be taken into account. The Dutch national drug database, updated monthly, recently added dose ranges dependent on age and body weight and included a gender indication on gender-specific drugs to improve specificity. If alerts are only important if specific serum levels are high, alerts should be suppressed if the level is low.9 Mitigating allergy alerting should be possible in case of medication intolerance.14,26 If a potential interaction did not result in problems in a particular patient, physicians should be able to prevent the interaction alert in dose adjustment in that patient. Entering (coded) overriding decisions should prevent future alert generation.20,38

A pharmacist or nurse can deal with low specificity alerts that can be checked once daily, for example, increasing laboratory values like creatinine, possible switch from intravenous to oral medication, or polypharmacy.41 They can present these alerts with additional information and a proposal to the physician.4 Directing alerts to other people in the workflow can also be used temporarily to test and improve new alerting features.41 Alerts about administration can be directed to nurses.26

Reducing Other Error-Producing Conditions

To reduce other error-producing conditions, sensitivity and information content should be improved, alerts should not unnecessarily disrupt workflow, and handling should be safe and efficient.

Sensitivity problems are common and a source of potential error.7,42,43 Alerting features can be lacking, alerts can be turned off or bypassed, the knowledge database can be incomplete or not up to date, and patient data, like body weight and drug allergies, can be incomplete, not coded (free text), or incorrect. Examples of sensitivity problems are the absence of a check (on drug-laboratory interactions, on correct administration routes or on interactions that become important when stopping a drug), bypassing medication control by using free text, and the absence of new registered drugs and ad hoc preparations and their drug safety data in the drug database.

To improve sensitivity, some relatively simple, albeit sometimes difficult to realize, measures can be taken. First, physicians should be encouraged to refrain from free text for drug name or dosage regimen.38 Second, allergies and body weight should be entered consequently if the software can handle these data elements.38 Furthermore, a regularly updated standardized drug database with allergies, interactions, and other safety data is strongly recommended.11,27,44 To further improve sensitivity and patient safety, technically more complex measures are necessary, like inclusion of laboratory data in alert generation.45

Performance is dependent on information content of the knowledge database,46 alerting features, and local customization.47 A performance study with a standard set of prescriptions for 16 clinically important drug-drug interactions in commercial and proprietary community pharmacy software revealed a sensitivity (ability to correctly identify clinically important alerts) ranging from 0.44 to 0.88 and a specificity ranging from 0.71 to 1.00.48 A gold standard for calculation of sensitivity and specificity does not exist; the ratios depend on the test performed.

The alert information should be unambiguous, justified, and presented concisely to enable easy understanding and initial triage at a glance. Users contend that more information should be accessible, although such sporadically read information has not been proven to result in a higher response rate to reminders.28,39,49

In CPOE systems with different alert levels, automatically overriding low-level alerts can result in motivation problems.14,19,22,23 High specificity implies that only urgent serious alerts are shown. Alerts with lower levels of seriousness can be absent or shown nonintrusively.14,20,27

Presentation of alternative actions on the same screen is subject to discussion. Advocates contend that such alerts are more effective in modifying physician behavior than alerts that only suggest stopping intended actions. Advice on dose adjustment in renal insufficiency resulted in improved care, they argue; standard doses and frequencies are the most important way to help physicians in their work. It must be emphasized that these advocates are all from the same center.11,12,50–52 Skeptics argue that the error risk will increase because physicians will rely on their own cognition and that decision support cannot incorporate all medical knowledge or all unanticipated events and situations in health care.7,36,53,54 Standardization of doses, frequencies, and route suggestions is thought to be an important source of erroneous orders and adverse events,36,54 although alerts suggesting controversial alternative orders were not acted on.55 To help physicians in alert handling without giving complete alternative actions, concise information can be presented on normal doses, last measured drug serum levels or creatinine levels, and also other interacting drugs.45,54

Turning off alerts without careful error management may impair patient safety.26,38 CPOE system design should meet physicians' preferences as well as safety requirements. One option is to turn off particular alerts for particular specialist groups,26,27 for example, showing interaction alerts on cyclosporine and nephrotoxic drugs for all specialties except nephrology because specialists should have enough knowledge to prevent unsafe situations. However, if specificity is high, specialists do only receive unsolicited alerts if their prescriptions cause potentially unsafe situations. A British study showed that 57% of prescribing errors were due to incorrectly executing an adequate plan because clinicians were busy or had been interrupted during routine tasks.34,35 It is often not a knowledge deficit that results in errors, but oversight, distraction, and forgetfulness.49 Another option mentioned by users was to turn off alerts (centrally or automatically) if users perform well (if serum levels are being ordered and if good reasons for overriding are entered).26 Automatically preventing alerts26 without informing physicians adds to nontransparency and does not seem desirable.38 If well-performing physicians are allowed to change the alert presentation from intrusive to nonintrusive, they keep informed and perhaps are educated without unnecessary workflow interruptions.14,20

CPOE systems may differ in their required actions to override alerts. In the system described by Nightingale et al.,13 a penicillin allergy alert can never be overridden, but allergy information is often based on patient information that is incomplete or partly correct.14,16,38 Some systems require entering a password or reason for overriding alerts that have the highest potential for causing adverse events,2,13,20,38 whereas other systems allow simply clicking away the alert.

Noninquisitive alerts guarantee efficient handling without disrupting workflow and are preferred.11,16,29,39,55 However, if the only safety check can be done after entering extra information or a choice between two alternatives, sensitivity can be increased.

To improve efficient handling of alerts, computer interface design should be logical; the number of keystrokes, mouse clicks, and screens should be kept as small as possible and speed should be sufficient.11,22,26,28 Detailed discussion of these technical, human computer interaction features falls outside of the scope of this paper.

Improving Latent Conditions

Safety of the alerting system may be further improved by influencing latent conditions (pharmacy check, training, and dependency), which have an indirect effect on the risk of a medication error. Alert overriding may be provoked by inappropriately high trust in the error-preventing actions of the pharmacy. Trust depends on the nature of the relationship between professionals and can only be influenced over time.

If a complete pharmacy check of all orders is absent, pharmacists should manually check dosage regimens and drugs that cannot be checked by the system. Information that dose checking cannot be done by the CPOE system should be directed to the pharmacy.

Adequate formal training and improving information content may prevent misinterpretation of alerts. Frequently shown informative alerts may also result in a learning effect,28 but this has not been studied thoroughly. In a study of automated alerts for 2,000 drug combinations, physicians spending more time in the clinic (using the CPOE system) recognized more interacting drug-condition pairs and more contraindicated pairs, suggesting a learning effect.22

Dependency can result from frequent alerts and poses a problem if users do not know the type of alerts the system is not checking for or if order check by supervisor or pharmacy is absent and not communicated to the users.2 If physicians rely too much on computerized decision support, they may not reconsider medication profiles or identify medication problems by themselves. However, in one study, false-positive alerts were not acted on.55 Another study of alerts for incorrect dosing showed no difference in alert rate between experienced and inexperienced house staff, suggesting that house staff was not dependent on the dosing assistance provided.17

Discussion and Conclusion

Many CPOE systems contain decision support by integrated drug safety alerts to improve patient safety. Very little research has been done on overriding drug safety alerts. Eight studies showed 49% to 96% alert overrides, except for high-level overdose alerts, which are overridden in 27%. Standardization of alert levels is largely absent, making comparison of override rates difficult. The Dutch drug database contains a coding system for drug-drug interactions.56

It should be emphasized that only unjustified overriding is problematic from a safety perspective. The authors advise entry of overriding decisions to gain deeper insight into (justified) overriding.20,22 Alerting systems may contain error-producing conditions and customizing is necessary, regardless of the use of a commercially available or a manually constructed database.44 Specificity or sensitivity should be increased as the result of consensus meetings between physicians and pharmacists.47

This customization process may be time-consuming and difficult because increasing sensitivity increases the total number of alerts and probably the percentage of inappropriate alerts, which decreases specificity. Required entry of reasons for overriding to prevent unintended overriding may result in an unacceptable time burden for physicians but gives useful information for system improvement.20,22,38 Disallowance of order entry (hard stops) is unacceptable in the opinion of the authors because decision support cannot replace the physicians' responsibility for the treatment of the patient.13 It is questionable whether entering a simple password will prevent unintended overriding.13

Many physicians complain about the poor signal-to-noise ratio and admit alert overriding because the alerts are not serious or are irrelevant.2,22 In studies on overriding, chart review did not reveal any adverse drug event in more than 97% of cases.15,19,20 Furthermore, in daily practice, adverse drug events often occur when the patient has moved to another point in the care chain, no longer within control of the physician(s) responsible for the event. Physicians believe that too many irrelevant alerts are presented and ask that alerts “they already know” be turned off. However, if specificity is high and alerts are only presented in potentially unsafe situations, specialists who already know them are not bothered by them. Furthermore, forgetfulness and oversight instead of a knowledge deficit are often the cause of generation of alerts and these problems can emerge in specialists as well as in residents.34,49 A testable hypothesis is whether specialists receive fewer alerts on their specialty than residents.

Presenting correct alternative actions is very difficult because they should include the right alternative drug, dose, and frequency for the patient's particular situation. The authors therefore propose to present concise information that can help physicians make a correct decision but to prevent selection of an alternative action with one click because indications may deviate from the indications on which the advice is based.

Decision support may result in physicians fully relying on the system and feeling safe if alerts are absent.2,7 Sensitivity problems can be divided between the absence of alerts within a particular alert feature and lacking alert features. Today, decision support on genetic profiles influencing drug-drug interaction effects is often lacking and physicians will not expect alerts of this type. If some type of alerting is present, physicians will have trust in complete decision support of that type, and increasing sensitivity as well as manually checking defensive gaps in the alerting system should achieve this. These gaps may change over time because of local customization and should result in a change in the pharmacy check to ensure patient safety. Which factors influence this pharmacy check are not clear.

The literature summarized in this paper focuses on the magnitude of overriding drug safety alerts, reasons and causes for overriding in general, effects of overriding, and suggestions for useful alerts. It is still not clear whether interactions on administration time, the level of seriousness, and the alternative action should be shown to the prescribing physician. The following hypotheses could be tested. Directing alerts on administration time to nurses or pharmacy technicians reduces the number of administration errors. Presentation of different levels of seriousness increases the override rate compared to one level of seriousness. Presentation of an alternative increases the number of unjustified cancellations or changes of order.

Before testing these hypotheses, it would be useful to gain insight in the cognitive processes playing a role when physicians are confronted with different types of alerts. None of the studies addressed this aspect of alert overriding. Rasmussen57 describes three levels of human performance (skill-, rule-, and knowledge-based behavior) and three corresponding ways in which information is perceived, depending on intentions and expectations of the receiver. It is not clear which level of human performance is used in interpretation and handling of drug safety alerts and which factors determine this performance level. Understanding reasons for and causes of overriding in particular cases is necessary for development of effective alerting systems that are acceptable to users.

Decision support in CPOE can be a good tool to improve patient safety but can also hamper patient safety if badly designed. The authors have argued how alert overriding can be understood with the help of Reason's model of accident causation and how decision support in CPOE might be designed to improve patient safety.

References