title: "Inter-rater reliability of the tracking and reporting outcomes of procedural sedation tool"

bibliography: [Extras/references.bib]

csl: "Extras/anaesthesia.csl"

author:

- Aaron Conway PhD:

email: aaron.conway@utoronto.ca

phone: +1 416 946 7112

address: 585 University Ave | Toronto, ON M5G 2N2

institute: [PMCC, LBFON, QUT]

correspondence: true

- Katina Corones-Watkins PhD:

institute: [Griffith]

correspondence: false

institute:

- PMCC: Peter Munk Cardiac Centre, University Health Network, Toronto, Canada

- LBFON: Lawrence S. Bloomberg Faculty of Nursing, University of Toronto, Toronto, Canada

- QUT: School of Nursing, Queensland University of Technology (QUT), Brisbane, Australia

- Griffith: School of Nursing, Giffith University, Brisbane, Australia

output:

officedown::rdocx_document:

reference_docx: Extras/manuscript_template.docx

pandoc_args:

- --lua-filter=Extras/scholarly-metadata.lua

- --lua-filter=Extras/author-info-blocks.lua

```{r setup, echo=FALSE}

\pagebreak

Clinical guidelines recommend that procedural sedation outcomes should be audited. The Tracking and Reporting Outcomes of Procedural Sedation (TROOPS) tool, developed by the International Consortium for the Advancement of Procedural Sedation, is intended to be suitable for use in all locations by sedation providers. We evaluated the feasibility and reliability of using a modified version of the TROOPS tool for sedation that is administered by nurses in the cardiac catheterization laboratory. The version of TROOPS we used contained items in the minor, intermediate and sentinel categories. Two nurses independently completed the TROOPS tool for 40 patients who underwent a procedure with sedation. A sedation-related adverse event was selected by at least one of the raters for 21 of the patients (52%; 95% CI = 0.38 to 0.67). Most were minor events related to the airway and breathing category of the TROOPS tool (n = 17; 42%; 95% CI = 0.29 to 0.58). The remaining were intermediate-severity events related to sedation quality. No events for the other categories of the TROOPS tool were selected. The intra-class correlation coefficient between paired nurse ratings of the TROOPS tool was `r total` (95% CI = `r total_lower` to `r total_upper`). Most nurses (85%, 95% CI = 73% to 92%) reported that it took less than 1 minute to complete the tool. In summary, paired ratings of sedation-related adverse events by nurses achieved moderate consistency using a modified version of the TROOPS tool containing items from the minor category of the airway and breathing domain. Use of this tool in clinical practice is feasible and reliable.

**Keywords**

Procedural sedation and analgesia; conscious sedation; nursing; patient safety; anesthesia

\pagebreak

The administration of sedative and analgesic medications during medical procedures is associated with several unintended effects with the potential to compromise patient safety. Serious adverse effects of sedation, such as respiratory compromise, are associated with worse clinical outcomes and higher costs.[@urman2019impact] However, if systems are in place to manage these complications, serious sedation-related serious adverse events are preventable. One central aspect of ensuring patient safety during sedation that is recommended in clinical guidelines is auditing of sedation-related outcomes.[@Dobson2018; @apfelbaum2018; @werthman2020administration; @hinkelbein2018; @conway2013] For example, the Practice Guidelines for Moderate Procedural Sedation and Analgesia from the American Society of Anesthesiologists Task Force in Moderate Procedural Sedation recommend the creation and implementation of a quality improvement process to measure adverse events and unsatisfactory sedation.[@apfelbaum2018] Likewise, the Canadian Anesthesiologists Society recommends that a specific mechanism must be established for auditing procedural sedation.[@Dobson2018]

The Tracking and Reporting Outcomes of Procedural Sedation (TROOPS) tool was developed by the International Consortium for the Advancement of Procedural Sedation.[@roback2018tracking] The tool is intended to be suitable for use in all locations by sedation providers regardless of the particular discipline. There are two versions of the tool. One is intended to be used as a strictly quality improvement tool and the other in research. The quality improvement version includes events categorized as intermediate and sentinel. Intermediate items in the TROOPS tool are defined as having the *potential* to endanger patients if not properly managed or reflecting suboptimal sedation quality. These items require timely reporting. Sentinel adverse events are defined as being life-threatening and warrant immediate reporting and the highest level of peer scrutiny.

The research tool includes additional *minor* adverse events, which are defined as those that pose little risk given appropriate provider skills and monitoring.[@roback2018tracking] Using the intermediate and sentinel event items included in the TROOPS tool for quality improvement seems appropriate due to the rigorous method in which it was developed. Although intended for a different purpose, specific items in the research version of the tool designated as *minor* may also be useful to track for the purposes of quality improvement because these events are more common. However, the feasibility and reliability of using the TROOPS tool for measuring the outcomes of sedation that is administered by nurses has yet to be evaluated. Such evaluation is required in order for the auditing component to be considered valid enough to inform practice improvement initiatives. For this reason, the aims of this study were to determine the:

1. Inter-rater reliability of nurses’ TROOPS ratings; and

2. Time taken for nurses to complete the TROOPS tool (calculated to determine the impact that integrating auditing of sedation outcomes into routine clinical practice on nurse workload).

A prospective study in an urban private hospital in Australia was undertaken. The study was approved by the human research ethics committee (UCH HREC 2016.23.201).

Patients undergoing procedures with nurse-administered sedation in a cardiac catheterization laboratory were eligible. 'Nurse-administered sedation' is when nurses administer a combination of sedative and analgesic medications prescribed by the proceduralist performing the procedure.[@conway2013a] A convenience sample was used.

A data collection form that contained three parts was developed to be used in this study. The first part contained items to collect demographic (e.g., age, sex) and clinical information (i.e., type of procedure, procedure start and end time, baseline oxygen supplementation and total doses of sedation used) about the included patients.

The second part consisted of the quality improvement version of the TROOPS tool, with the addition of minor adverse events from the 'airway and breathing' as well as 'gastrointestinal' domains from the research version of the tool.[@roback2018tracking] Minor events from the 'airway and breathing' domain used were 'increased or added supplemental oxygen', 'airway repositioning' and 'tactile stimulation'. Based on consultation with members of the clinical team, we elected not to include the minor items related to hypersalivation (i.e., suctioning for hypersalivation and anticholinergic for hypersalivation) because the medications used for sedation in the department (i.e., fentanyl and midazolam) are not associated with this particular adverse effect. Both minor events from the gastrointestinal domain of the research version were used. These were administration of anti-emetic and suction for emesis. The minor items from the allergy and neurological domains were not included because these adverse effects were not considered to occur frequently enough in this particular sedation context to warrant inclusion as a measure for quality improvement. Other modifications were made to the TROOPS tool through consultation with members of the clinical team in order to make it suitable for use by nurses in that particular department. Modifications included the removal of outcomes that were not feasibly assessed by nurses in the immediate post-procedure period, including permanent neurological deficit from the sentinel category of the neurological domain and pulmonary aspiration syndrome from the sentinel adverse events category of the airway and breathing domain. The TROOPS tool has not undergone validity or reliability assessment in any other clinical setting.

The third part of the data collection form included one item for nurses to rate how long it took to complete the TROOPS tool ratings. Possible selections were 'less than 1 minute', 'between 1 and 3 minutes', 'between 3 and 5 minutes', and 'more than 5 minutes'.

Two nurses who performed the ‘scout’ and ‘monitor’ role during procedures performed with nurse-administered sedation completed the modified version of the TROOPS tool. The nurse performing the 'scout' role was responsible for administering direct patient care and procedural support. The nurse performing the 'monitor' role was responsible for physiological monitoring and related documentation. Ratings were performed independently as soon as possible after the completion of the procedure. Prior to commencement of the project, all nurses involved in administering sedation in the clinical department were invited to attend in-services where information about the project and the assessment tool was provided. We did not collect information on the number of nurses attending the education sessions.

Inter-rater reliability was assessed by calculating the random effects one-way intra-class correlation between nurses’ paired ratings. Lower and upper confidence (95%) bounds associated with the intra-class correlation under the one-factor ANOVA model was used because each patient could have been rated by a different group of raters (i.e. nurses).[@gwet2014handbook] Feasibility was examined by calculating proportions for nurses' selections about the time taken to complete the tool. Analyses were undertaken using `R`.[@rcore2021] Data, statistical code and a computational environment required to reproduce the results is available.[@conway2021code]

Data were collected from January to November, 2017. Two nurses independently completed the modified version of the TROOPS tool for 40 patients who underwent a procedure with sedation. A summary of the patient characteristics is presented in Table \@ref(tab:tab1). Most patients were undergoing a cardiac implantable electronic device implant (e.g. pacemaker or implantable cardioverter defibrillator) or electrophysiology procedure (e.g., atrial fibrillation or ventricular tachycardia ablation). A combination of midazolam and fentanyl was used for all patients.

A sedation-related adverse event was selected by at least one of the raters for 21 of the patients (52%; 95% CI = 0.38 to 0.67). Most were minor events related to the airway and breathing category of the TROOPS tool (n = 17; 42%; 95% CI = 0.29 to 0.58). The remaining were intermediate-severity events related to sedation quality and patient experience. No events for the other categories of the TROOPS tool were selected. The intra-class correlation coefficient between paired nurse ratings of the modified TROOPS tool was `r total` (95% CI = `r total_lower` to `r total_upper`). Responses from fifty-one (85%, 95% CI = 73% to 92%) nurses indicated that it took less than 1 minute to complete the modified TROOPS tool. The remaining nurses responded that it took between 1 and 3 minutes to complete. There were 20 missing responses to this item for the 40 patients.

Reliability is a vital property of a measurement tool if results are intended to inform quality improvement. There was moderate consistency between paired ratings of the modified version of the TROOPS tool used in this study. However, it should be noted that confidence intervals for the intra-class correlation were wide, spanning from poor (`r total_lower`) to very high levels of consistency (`r total_upper`). Additional research with a larger sample size would increase confidence in estimates of inter-rater reliability.

Further research would be valuable to identify strategies that increase the reliability of using the TROOPS tool for rating sedation-related adverse events in practice. Pending such research, there are several recommendations for practice that can reasonably be drawn from these results. First, our study involved a fairly passive implementation process, which was limited to the provision of information. Instructions about how to use the tool were provided to nurses during scheduled in-service sessions. A more 'active' implementation process, including, for example, practical demonstrations of 'correct' ratings, may be needed to improve the reliability of TROOPS tool ratings. Video recordings of sedation encounters have been used to validate sedation scales in prior research.[@cravero2017validation] Using videos of sedation encounters could be a practical and efficient approach to facilitate demonstrations of the correct usage of the TROOPS tool for rating sedation-related adverse events. Alongside education and instruction on the correct usage of the TROOPS tool, periodic evaluation of inter-rater reliability should be undertaken by procedural sedation programs electing to use the TROOPS tool with items from the minor category of the airway and breathing domain.

In our study, both nurses who completed TROOPS ratings for each patient were involved in the procedure, but they performed different roles. In the department in which data collection was performed, nurses performed the roles of 'scout' nurse and 'monitor' nurse. This is commonplace in the cardiac catheterization laboratory setting.[@white2018consensus] Based on the authors' experiences of working in cardiac catheterization laboratory settings, often the responsibility for documentation, including that related to audits for quality improvement activities, is sometimes shared between the nurses performing these roles in a very inconsistent manner. Our findings show that TROOPS ratings for the same procedure were not in perfect agreement between nurses who performed these different roles. One course of action that would logically be expected to improve the reliability of measuring adverse sedation events using the TROOPS tool is to ensure that the rater is consistently only ever the nurse who performed the scout role *or* the nurse who performed the monitor role.

Few studies have evaluated the inter-rater reliability of measurement tools intended for quality improvement activities.[@redley2018mixed] The finding that ratings between nurses using the TROOPS tool were not in perfect agreement may be a somewhat surprising finding, considering the relatively simple measurement scale. This finding underscores the need for rigorous assessment of reliability for measurement tools intended for use not only in research but also in quality improvement.

It was identified in this study that minor sedation-related adverse events related to airway and breathing were common in this population with the sedation regimen used. Gaining a baseline assessment of the frequency of such events may be useful for departments so that sedation effectiveness can be periodically evaluated when new procedures are introduced that may have different sedation requirements. The event rate is higher than most other published studies of sedation in similar cohorts.[@conway2013risk; @conway2019sequence] There are no reports in the literature of the prevalence of sedation-related adverse events during procedures performed with sedation in the cardiac catheterization laboratory that have specifically used the TROOPS tool. However, serious adverse events that would meet 'sentinel' category definitions are rare.[@markham2015conscious; @sawhney2017moderate; @conway2011nurse]

We have confirmed in this study that the time required to complete the TROOPS tool is very minimal. Using an expanded version of the TROOPS tool including items from the minor category, either routinely for all procedures, or periodically as a 'spot-check' for quality assurance is unlikely to interfere with clinical workflow requirements for nursing staff in the cardiac catheterization laboratory.

We did not collect information on which nurses completed ratings so that anonymity was maintained. As such, the total number of nurses who completed ratings for patients in this study is not known. For the same reason, we can not report on the characteristics of the nurses who provided ratings, such as their experience or training.[@kottner2011guidelines] It should also be noted that, as expected, most events identified were from the minor category of the airway and breathing domain. Results should not be generalized outside of this context. The less common and more severe adverse events, such as the unintended requirement for tracheal intubation during sedation, is not likely to be inconsistently rated by nurses. Likewise, results should be considered most applicable to the context in which sedation was performed in this study. An approach that is common across many cardiac catheterization laboratories was used, with bolus doses of midazolam and fentanyl being administered by nurses.[@Conway_2014] Although there was a substantial proportion of events recorded by nurses in the sample (52%), it is possible that it over-estimates the true event rate because the analysis was performed using a convenience sample. The sample represents only a small proportion of the total number of procedures performed with sedation at the site over the data collection period. Estimating the prevalence of adverse sedation-related events was not the primary purpose of this study so we did not collect information about the total number of procedures performed with sedation over the data collection period to determine the exact proportion of completed TROOPS ratings. Finally, estimates of inter-rater reliability reported in this study should only be considered applicable to the modified version of the tool that we used, not the original research or QI versions of the TROOPS tool.

Paired ratings of sedation-related adverse events by nurses in the cardiac catheterization laboratory achieved moderate consistency using a modified version of the TROOPS tool containing items from the minor category of the airway and breathing domain. Use of this tool in clinical practice seems feasible. The time required to complete the ratings was very minimal, with most ratings taking less than one minute to complete.

\pagebreak

```{r tab.cap='Participant characteristics', tab.id = 'tab1'}

\pagebreak