Learn more: PMC Disclaimer | PMC Copyright Notice
Do Clinicians Use Online Evidence to Support Patient Care? A Study of 55,000 Clinicians
Abstract
Objectives: To determine clinicians' (doctors', nurses', and allied health professionals') “actual” and “reported” use of a point-of-care online information retrieval system; and to make an assessment of the extent to which use is related to direct patient care by testing two hypotheses: hypothesis 1: clinicians use online evidence primarily to support clinical decisions relating to direct patient care; and hypothesis 2: clinicians use online evidence predominantly for research and continuing education.
Design: Web-log analysis of the Clinical Information Access Program (CIAP), an online, 24-hour, point-of-care information retrieval system available to 55,000 clinicians in public hospitals in New South Wales, Australia. A statewide mail survey of 5,511 clinicians.
Measurements: Rates of online evidence searching per 100 clinicians for the state and for the 81 individual hospitals studied; reported use of CIAP by clinicians through a self-administered questionnaire; and correlations between evidence searches and patient admissions.
Results: Monthly rates of 48.5 “search sessions” per 100 clinicians and 231.6 text hits to single-source databases per 100 clinicians (n = 619,545); 63% of clinicians reported that they were aware of CIAP and 75% of those had used it. Eighty-eight percent of users reported CIAP had the potential to improve patient care and 41% reported direct experience of this. Clinicians' use of CIAP on each day of the week was highly positively correlated with patient admissions (r = 0.99, p < 0.001). This was also true for all ten randomly selected hospitals.
Conclusion: Clinicians' online evidence use increases with patient admissions, supporting the hypothesis that clinicians' use of evidence is related to direct patient care. Patterns of evidence use and clinicians' self-reports also support this hypothesis.
Current literature1,2,3 indicates that clinicians do not routinely use the available evidence to support clinical decisions. Several studies have shown that simply disseminating evidence, for example, in the form of practice guidelines, does not lead to increased use of that information to inform clinical decisions.4 Clinicians apparently pursue answers to only a minority of their questions5,6 and, when they do so, they rely most heavily on colleagues for answers.5 Lack of easy access to up-to-date evidence is cited as a barrier to evidence-based practice by clinicians.7,8
Online clinical information resources have the potential to support clinicians who adopt an evidence-based approach by providing them with the information they need when they need it.9 We have some evidence that searches of bibliographic databases such as Medline are efficacious.5,10 Given sufficient time, clinicians are able to retrieve research evidence relevant to their clinical questions.11 Training in online searching techniques enhances the quality of evidence retrieved,12 whereas education in critical appraisal increases clinicians' abilities to apply the information obtained.13
However, measuring the actual uptake of online information retrieval systems is problematic and few studies have been attempted. Studies of intranet provision of online resources report monthly utilization rates of 30 to 720 searches per 100 person-months.14 However, most studies only report use rates that exclude clinicians who have access to the system but do not use it. Consequently, these studies do not provide a measure of actual uptake by the clinical population.
It is also difficult to measure the impact that online access to evidence has on clinical practice. Assessments of the impact of Medline information on decision making and patient care have relied primarily on self-reports of clinicians. Haynes and McKibbon15 provided training and access to online Medline to a group of 158 U.S. physicians. For 92% of searches related to a patient's care, clinicians reported that the information retrieved resulted in “at least some improvement” in care. Using the critical incident technique, Lindberg et al.16 interviewed U.S.-clinician Medline users about their searches. Of the 1,158 searches described, 41% were classified as affecting decisions regarding patient care. A survey of U.K. general practitioners who used the ATTRACT system, which provides rapid access to evidence-based summaries to clinical queries, found 60% (24 of 40 doctors) reported that the information gained had changed their practice.17
Based on the assumption that providing clinicians with easy access to “evidence” will support decision making and result in improvements in patient care, in 1997 the State Health Department in New South Wales (NSW), Australia, implemented the Clinical Information Access Program (CIAP; <http://www.ciap.health.nsw.gov.au/>). CIAP is a Web site providing point-of-care, 24-hour, online access to a wide range of bibliographic and other resource databases for the approximately 55,000 clinicians (doctors, nurses, and allied health staff) employed by the state and primarily working in public hospitals.
Qualitative data from case studies indicated that clinicians perceived a range of factors influenced their use of the system, including support from facility managers and direct supervisors, access to computer terminals, training and skills (appraisal of evidence, database searching, and computer skills), and the place of evidence-based practice in their professional work.18
We sought to test hypotheses generated as a result of these case studies using Web log and survey data. We aimed to determine the rates of “actual” and “reported” use of online evidence by the population of clinicians working in the public health system in NSW and to assess the extent to which system use was related to direct patient care.
We posed and tested two competing hypotheses. These hypotheses were formulated following a qualitative study examining clinicians' use of online evidence18: hypothesis 1—clinicians use online evidence primarily to support clinical decisions relating to direct patient care; and hypothesis 2—clinicians use online evidence predominantly for research and continuing education. These hypotheses were tested by the following methods.
Examination of Patterns of Online Evidence Use by Clinicians by Time, Day, and Location of SearchesHypothesis 1 would be supported by a pattern of use that coincided with patient care and peaked between the core working hours of 9 am and 5 pm, with most use originating from within hospitals (▶). Hypothesis 2 would be supported by a relatively wide distribution of use across the times of the day with sustained rates of use into the evening when clinicians have more free time for research (▶). A high proportion of access would be expected to occur from outside hospitals, e.g., at home.
Measuring the Association between Hospital Admissions and Use of Online Evidence
A significant positive correlation between patient admissions to hospitals and online evidence searches would provide support for hypothesis 1 by demonstrating that CIAP use is likely to be used primarily to inform patient care as opposed to meeting research or continuing education information needs. Absence of a correlation would support hypothesis 2.
Methods
Two sets of Web-log data, automatically generated for each CIAP search transaction and containing details such as the date and time of the search, resources examined, user identification, and Internet Protocol (IP) address, were analyzed. The first data set recorded searches of bibliographic databases (e.g., Medline) measured as sessions. A session is defined as a period of resource use demarcated by a log-in and log-out. A session is automatically logged out when a user has not accessed any documents for 30 minutes. The second data set documented individual hits to single source databases such as Cochrane, guidelines, and pharmaceutical references.
Seven months of bibliographic sessions (August 2000–February 2001) and five months of text document accesses (hits) to single source databases (October 2000–February 2001) were analyzed. A modified version of transaction log analysis was used.19,20 Each hospital is provided with a CIAP password, and therefore, a major limitation of the log data was the inability to identify the professional identity of individual CIAP users. IP addresses were used to estimate the proportion of CIAP use originating from hospital and home computers. An IP address is a unique string of numbers that identifies individual computers connected to the Internet. We used 100% of the available IP addresses that could be uniquely identified with a specific location.
Monthly utilization rates per 100 clinicians for each hospital in the state were calculated. The correlation (Pearson's product–moment) between patient admissions and bibliographic search sessions for each day of the week was examined for NSW as a whole, as well as for a random sample of ten hospitals. Furthermore, to examine different levels of use, for some analyses, individual hospitals were stratified according to relative rate of use: high monthly users (≥57 sessions per 100 clinicians), medium (< 57 and >34 sessions), and low (≤34 sessions per 100 clinicians). Cutoff points were determined by examination of the distribution of rates to identify natural breaks. Sensitivity analyses around the cutoff points were undertaken. Admission data for the ten hospitals were obtained from the NSW Department of Health, but the researchers were blind to these data at the time the hospitals were selected.
A survey of clinical staff's views and knowledge of CIAP was also undertaken. After the calculation of use rates for 81 hospitals in the state, a random selection of hospitals from the high-/medium- and low-use categories was made (▶). Hospitals continued to be selected until the numbers of clinical staff in each professional group (medicine, nursing, allied health) employed by those hospitals provided sufficient statistical power to allow differences between groups to be measured. In total, 65 hospitals (25 high/medium and 40 low CIAP users) covering 21,712 clinical staff were included. Each hospital was requested to meet a completed survey quota equal to 25% of staff in each of the three professional groups in the individual hospitals. A nominated survey coordinator at each hospital distributed surveys between November 2001 and February 2002 until the quotas were approached or met. Thus, within each randomly selected hospital, a convenience sample of clinicians, representing 25% of the total clinician workforce, was obtained. The number of completed surveys represented approximately 10% of the entire clinical staff population in the state's public hospital system.
Sampling strategy for statewide clinician survey. Ambulance officers and those clinicians with access to the clinical information access program working outside public hospitals, for example, in community health clinics, were excluded from the survey. NSW = New South Wales; CIAP = Clinical Information Access Program.
The survey items relevant to the present study were knowledge of CIAP, frequency of use, potential for CIAP to improve patient care, and direct experience of CIAP resulting in improved patient care. Seventeen potential reasons for use of the online evidence system were included. Most of these were developed by Lindberg et al.16 in their study of Medline use by more than 500 U.S. physicians. In previous case study research18 of CIAP use, we identified additional reasons for use such as to review policies and guidelines, and these were added to the survey. Comparisons between doctors' and nurses' responses to survey items were undertaken using chi-squared analyses with p set at <0.05.
Results
Rates, Location, and Time of Evidence Searching
The Web-log analysis showed that in total, 181,499 bibliographic search sessions were undertaken over the seven-month period, representing a monthly state rate of 48.5 sessions per 100 clinicians, along with 619,545 hits (231.6 per 100 clinicians) to the single-source databases. There was considerable variation in rates of sessions (range, 2.8–117.6 per 100 clinicians; ▶) and document accesses (0.8–1,144.4 per 100 clinicians) for the 81 individual hospitals examined.
Monthly rates of bibliographic sessions by hospital (N = 181,499). Please note: only the name of every second hospital appears on the graph.
Eighty-eight percent of all sessions occurred on weekdays and 74% between the hours of 9 am and 5 pm (▶). Patterns of online evidence use on the weekdays differed from those on the weekends. To examine clinicians' use of CIAP at home and in the hospital, IP addresses were used to identify the origin of evidence searches. Two-thirds of sessions and hits were conducted from hospitals, 22% from private residences and other organizations, and 11% from universities. However, the times at which hospital and home searches were conducted differed significantly as ▶ shows. During the weekdays, 79% of evidence searches in hospitals occurred between 9 am and 5 pm, whereas only 41% of home searches occurred at these times. During the weekends, the pattern of CIAP use in hospitals more closely resembled times of CIAP use at home (▶).
In rank order, the four most frequently viewed full-text medical journals were The Lancet, The British Medical Journal, New England Journal of Medicine, and Journal of the American Medical Association (JAMA).
Relationship of Evidence Searching to Patient Admissions
Clinicians' use of bibliographic databases on each day of the week was highly positively correlated with patient admissions (r = 0.99, p < 0.001) (▶). This association remained when the weekend was excluded from the analysis (r = 0.99, p < 0.001). This relationship was examined for individual hospitals with different levels of CIAP use (i.e., high, medium, and low) to further test this association. A random selection of ten hospitals across the three categories was selected: four high-, four medium-, and two low-use hospitals. Hospitals with monthly rates of less than 20 sessions per 100 clinicians were not included as a result of insufficient session data. All hospitals showed a positive significant relationship between evidence searching by CIAP and patient admissions (▶).
Distribution of patient admissions and bibliographic sessions by day of the week, August 2000 to February 2001.
Table 1.
Correlations between Hospital Admissions and Online Evidence Use
| Hospital | CIAP Use Rate | Pearson Correlation | Significance (One-tailed) |
|---|---|---|---|
| A | High | 0.92 | 0.01 |
| B | High | 0.95 | 0.001 |
| C | High | 0.85 | 0.01 |
| D | High | 0.91 | 0.01 |
| E | Medium | 0.74 | 0.03 |
| F | Medium | 0.87 | 0.01 |
| G | Medium | 0.91 | 0.01 |
| H | Medium | 0.96 | 0.001 |
| I | Low | 0.74 | 0.03 |
| J | Low | 0.87 | 0.01 |
CIAP = Clinical Information Access Program.
Clinician Reports of Evidence Use
Of the 5,511 clinicians surveyed, 63% (n = 3,471) had heard of CIAP and 47% (n = 2,603) had used it. However, 75% (n = 2,603) of those clinicians who were aware of CIAP had used it. Doctors were more likely to have heard of CIAP (71% compared with 58% of nurses, χ2 = 60.5, df = 1, p < 0.001) and of these, a greater proportion of doctors had used it (82% (n = 660) versus 71% (n = 1,308), χ2 = 37.4, df = 2, p < 0.001).
Sixty percent of users indicated they sought evidence more than twice a month, with 30% indicating they used CIAP once a week or more frequently. Consistent with the log results, the four most frequently used medical journals reported by clinicians were The British Medical Journal, New England Journal of Medicine, JAMA, and The Lancet.
Clinicians were asked to indicate the main reasons for CIAP use and were able to select as many of the 17 reasons as they wished. Of CIAP users, 77% reported use was for the generic purpose of filling a knowledge gap. Fifty-seven percent reported use for personal education and 46% for research. Eight further categories related directly to use of CIAP to support patient care (e.g., confirm a diagnosis, develop a treatment plan, make a diagnosis, select a diagnostic test). Seventy-five percent of users selected one or more of these clinical reasons for use. Four categories related to use of CIAP for the education of others (i.e., to educate others 41%, information for patients 24%, maintain an effective relationship with a patient 3%, and modify patient behaviors 3%). Thirteen percent reported accessing CIAP to settle a dispute and 3% for “other” reasons. Comparisons between nurses and doctors showed that doctors were significantly more likely to report using CIAP for clinical reasons (79.4% vs. 70.0%, respectively, χ2 = 19.6, df = 1, p < 0.001). Nurses were significantly more likely than doctors to report use for personal education (60.2% vs. 45.9%, respectively, χ2 = 36.1, df = 1, p < 0.001). Eighty-eight percent of clinician users (93% of doctors and 84% of nurses) indicated that they thought CIAP had the potential to improve patient care and 41% (54% doctors and 34% nurses) reported that they had direct experience of CIAP resulting in improved patient care.
Discussion
The call for studies to investigate how physicians use online evidence systems and the impact of these on patient care has been consistent since the 1990s,14,15,21 yet to date, the small number of studies conducted have relied almost entirely on results from physician surveys.15,17,22 The present study adopted a unique approach to this problem by examining in detail patterns of online evidence use and their relationship to patient admissions across a large clinician population.
The results showed that “actual” patterns of use, as measured by computer logs, supported the hypothesis that clinicians seek evidence to support patient care decisions. We found that patterns of online evidence use supported hypothesis 1 with the greatest use of CIAP occurring during the core working days and times, with most searches originating from hospitals. Over 40% of clinicians reported that they had direct experience of CIAP use resulting in improved patient care.
The pattern of hospital CIAP use during the week did not support hypothesis 2, which posited that use was primarily related to research and continuing education, and thus a reasonably even distribution of evidence use across the day, with greater use in the early evenings, would be apparent. In contrast, the pattern of online evidence use at clinicians' homes over the same weekday period closely matched hypothesis 2. Both home and hospital weekend use of the online evidence resources also more closely fitted hypothesis 2's pattern of use. Consequently, a mixed pattern of evidence use emerges, with clinically driven searching occurring during contact with patients, and educational- or research-motivated access occurring at other times.
The most significant finding was the strong positive relationship between patient admissions across the state and bibliographic sessions undertaken using CIAP. This correlation was found for the state as a whole and for each of the ten individual hospitals examined. This result cannot be explained by factors such as variation in staffing levels because these, particularly medical staffing, remain relatively constant across the weekdays in public hospitals.
These results are the first step in demonstrating that access to online clinical information is prompted by patient care questions and thus does have the potential to influence clinical decisions. Further work is required to demonstrate the clinical impact of information retrieval on decision making and patient care, and we are undertaking further studies to investigate these issues. However, the continued increase in use of CIAP four years since initial implementation23 suggests that clinicians believe it has clinical use. This view is supported by our survey results, and those of others, in which considerable proportions of clinicians reported access to online evidence as resulting in changes to patient care.15,16 Examination of the reasons for CIAP use showed that CIAP was used to support a range of clinical tasks as well as research and personal education activities. Although 75% reported clinical reasons for use, nearly 60% reported use for personal education and 46% for research. There were professional differences in use for some of the major categories, with doctors more likely to nominate use related to patient care compared with nursing staff. In many instances, these differences reflect differences in their clinical roles. The survey results gave an indication of the wide range of reasons why clinicians used CIAP but did not provide a measure of the frequency of use for specific activities. These findings are consistent with the case study data of clinicians' use of CIAP in which doctors were found to talk predominantly about the use of CIAP to answer clinical questions arising from diagnosis and treatment of individual patients, and nurses focused more on its use to support personal education.18 Nurses in the case studies raised issues regarding “… whether it was the nurse's role to initiate information seeking related to patient care” because doctors were the ultimate decision makers. Such views assist in explaining the smaller proportion of nurses reporting use of CIAP for patient care activities.18
Web-log data, which reflect actual use of clinical information systems, are a much underused resource in researching the information needs of clinicians. We supplemented these data with survey data of clinicians “reported” use and found consistency between sources. The random selection of the hospitals surveyed and the large, completed survey sample size negated many of the limitations of previous studies in this area that have relied on small, single-institution clinician survey samples. The Web-log data identified organizational users but not individuals. However, the survey results demonstrated that use was not confined to a small group of active clinician users, with 47% of all clinicians (75% of those who were aware of the system) reporting use of CIAP, and the majority undertaking searches at least once every two weeks.
Evidence-based concepts have been most influential in the medical profession. In general, nurses have been shown to be low users of research literature24 and to access online bibliographic databases less than doctors but more than allied health professionals.25 Thus, it is likely that if we had included only doctors in our study of online evidence use, overall utilization rates would have been higher. This was confirmed by the survey findings, in which nurses reported lower rates of CIAP use.
Use of CIAP was not consistent across the state, with considerable variation in CIAP rates between hospitals. Technical factors such as access to, and speed of, computers explain only a proportion of the variation in CIAP uptake.18 However, as other research on the adoption of evidence into practice has demonstrated, multiple organizational, professional, and education factors play important roles in determining uptake.26,27 The influence of these variables on clinicians' adoption of online evidence systems warrants further study to maximize the effective use of these systems.
Conclusion
We are reporting the first large-scale population study of evidence use in clinical practice, and our results paint a very different picture of evidence use from that arising out of previous small-scale studies. This study has demonstrated a strong positive association between patient admissions and use of online evidence by a large population of clinicians in everyday practice. We have proposed that this provides support for the hypothesis that clinicians primarily use online evidence to inform direct patient care questions. Perhaps most surprisingly, this relationship held for hospitals where rates of online evidence use were high and for hospitals where rates of use were comparatively low. Our conclusion is reinforced by the self-reports of more than 5,500 clinicians who indicated that they believed online evidence had the potential to improve patient care and many reported having had direct experience of this.
Notes
This research was funded by the New South Wales Health Department.