Performance of the inFLUenza Patient-Reported Outcome Plus (FLU-PRO Plus) Instrument in Patients With Coronavirus Disease 2019

Abstract

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic has prompted research into how disease caused by the novel coronavirus (COVID-19) affects the human host. Current data show most patients experience symptoms that impair daily activities. A variety of web-based and mobile applications using self-reported symptom surveys have been developed [1–9]. Researchers have attempted to describe the symptoms and disease severity [10–13]; however, valid measurement of symptoms relevant to patients, including their intensity and duration, requires the use of a comprehensive, standardized, and interpretable measurement of patient symptoms over time. Patients are the reference standard for their own health; therefore, patient-reported outcome (PRO) instruments provide a well-defined and reliable method for assessment of patient symptoms. A standardized method of evaluating patient-reported outcomes can facilitate meta-analyses, vaccine effectiveness studies, and other COVID-19 research activities.

Viral respiratory tract diseases are common, and the current pandemic caused by SARS-CoV-2 is another example of their public health impact. The United States (US) military has been significantly impacted in the past by outbreaks of influenza and adenovirus infections [14]. Accurately tracking the symptoms and impacts of communicable infections is key to maximizing public health and maintaining force readiness.

The FLU-PRO is a PRO instrument developed to measure the intensity and frequency of symptoms of viral respiratory tract diseases [15]. Questions on loss of taste and smell have been added to the original FLU-PRO, resulting in the FLU-PRO Plus [16]. Designed to be administered daily, the instrument can also measure the duration of illness. The content validity of the instrument was developed through prospective interviews with patients to identify the best set of questions, and to maximize patient comprehension and ease of administration. The construct validity and measurement properties of FLU-PRO have been evaluated in various viral respiratory infections including influenza, respiratory syncytial virus, rhinovirus, enteroviruses, and seasonal coronavirus infections [15, 17, 18]; however, there are limited data on the performance of FLU-PRO Plus in COVID-19. The objective of this study was to assess the performance properties of FLU-PRO Plus among inpatients and outpatients presenting as part of a prospective observational cohort study with laboratory-confirmed COVID-19 in the US Military Health System (MHS).

MATERIALS AND METHODS

Study Design

The Epidemiology, Immunology, and Clinical Characteristics of Pandemic Infectious Diseases (EPICC) study is an ongoing prospective observational cohort study that includes hospitalized and outpatient MHS beneficiaries tested for COVID-19, COVID-19–like illness, or those with exposure to confirmed cases at 10 Military Treatment Facilities (Brooke Army Medical Center, Carl R. Darnall Army Medical Center, Fort Belvoir Community Hospital, Madigan Army Medical Center, Naval Medical Center Portsmouth, Naval Medical Center San Diego, Tripler Army Medical Center, William Beaumont Army Medical Center, Walter Reed National Military Medical Center, and Womack Army Medical Center) [19]. Participants consent to providing information directly to the study through coordinator interview, medical record review, and/or surveys including FLU-PRO Plus, as well as having their electronic medical record information abstracted from the MHS database. Standard of care laboratory test data are captured, and research respiratory swabs and blood specimens are collected at defined timepoints. To evaluate FLU-PRO Plus in a SARS-CoV-2–positive population, we excluded participants who did not have a clinical or research SARS-CoV-2–positive nasal swab or serology.

Upon enrollment in the study, participants provided information about their comorbidities, demographics, and COVID-19 symptoms, if any. If symptoms were present at the time of enrollment, participants were asked to fill out a daily FLU-PRO Plus questionnaire for 2 weeks. Although we sought to enroll participants as early in their disease course as possible, participants could be enrolled at any time (up to weeks) after development of symptoms; therefore, there was variability in the timing of the FLU-PRO Plus administration relative to symptom onset. We analyzed data from SARS-CoV-2–positive participants who were enrolled from 24 March 2020 through 25 June 2021 and had at least 1 FLU-PRO (March to April 2020) or FLU-PRO Plus (May 2020 onward) survey that was completed within 1 week of symptom onset and in which at least 1 symptom was reported as “a little bit” or worse.

Patient Consent Statement

The EPICC cohort study was conducted in accordance with the Declaration of Helsinki and good clinical practice guidelines. Informed consent was obtained from all individual participants in the study. This study was approved by the Uniformed Services University Institutional Review Board (IDCRP-085).

Instruments: Patient-Reported Outcomes

FLU-PRO Plus

In FLU-PRO Plus, participants are asked to rate the extent to which they have experienced symptoms during the past 24 hours, using a 5-point severity scale ranging from “not at all” (0) to “very much” (4). In addition, as in the original FLU-PRO, FLU-PRO Plus includes questions on the frequency of sneezing, coughing, coughing up mucus or phlegm, vomiting, and diarrhea in the past 24 hours. In the original instrument, symptoms are categorized into 6 domains (nose, throat, eyes, chest/respiratory, gastrointestinal, body/systemic). For FLU-PRO Plus, a seventh domain of senses (loss of taste and smell) was added on 1 May 2020, given evidence of the increased frequency of anosmia and ageusia in COVID-19. Responses are scored using the mean score for all symptoms within that domain. The total score was derived using the mean score for all symptoms, with and without the senses domain, as those questions were added after the study began.

The original FLU-PRO was previously evaluated for reliability, construct, and known-groups validity, and responsiveness in participants with influenza [17] and with influenza-like illness [18]; this analysis explored these characteristics for FLU-PRO and FLU-PRO Plus with respect to participants with COVID-19 symptoms.

Patient Global Assessments

Participants were administered global assessments of severity, including an overall rating of disease severity, an assessment of interference in daily activities, and an assessment of physical health.

Return to “Usual” Health and Activities

Participants were asked daily to respond (yes/no) to whether they had returned to usual health and usual activities.

Statistical Analysis

Statistical tests were performed consistent with classical test theory to evaluate the psychometric testing properties in the EPICC cohort of participants for FLU-PRO Plus, including reliability, construct and known-groups validity, and responsiveness of scores to change in patients’ health status over time [20].

Reliability (Internal and Test-Retest)

Internal consistency of the domain and total scores on day 1 was measured using Cronbach coefficient α (<.0.4=poor, 0.4–0.7=moderate, >0.7=good internal consistency). Test-retest reliability of FLU-PRO Plus for COVID-19 symptoms was assessed in participants who reported no change in the PGA from one day to the next during the first week postenrollment. Among those with no change in patient global assessment (PGA), we calculated intraclass correlation coefficients (ICCs) and used paired t tests to compare changes in domain scores from one day to the next, with the expectation that ICCs would be >0.60 and mean differences in scores would be small.

Construct Validity

We used Spearman correlations to assess the relationship between the domain-specific/total scores and the 3 global ratings (disease severity, physical health, and interference in daily activities) on day 1.

Known-Groups Validity

FLU-PRO Plus domain and total scores were compared across 3 categories of severity on the PGA (“none” or “mild”; “moderate”; and “severe” or “very severe”) using analysis of variance and pairwise t tests.

Responsiveness

FLU-PRO Plus scores at day 6/7 in those who reported returning to usual health (or activity) defined as “responders” were compared with scores in those who did not report returning to usual health or activity (“nonresponders”) using analysis of covariance (ANCOVA), adjusting for day 1 scores. Participants who had 6 to 7 days of FLU-PRO Plus data and did not report returning to usual health or activities on day 1 were included in this subanalysis.

RESULTS

Sample

The study enrolled 1879 participants during the time frame covered by this analysis. Among the cohort enrolled, 226 participants filled out a FLU-PRO or FLU-PRO Plus survey within 1 week of symptom onset and reported at least 1 specific symptom (Supplementary Figure 1). More than half of the participants were 18–44 years of age, and 14% were hospitalized (Table 1). The median days post–symptom onset that participants began filling out the FLU-PRO Plus was 6 days (25th, 75th percentiles: 4.25, 7). Many participants reported no (33%) or little (19%) interference with daily activities, and at least good physical health (53%) on day 1.

Evaluation of Psychometric Properties

Descriptive Statistics of FLU-PRO Plus Domain and Total Scores

The average domain scores ranged from 0.6 (eyes, gastrointestinal symptoms domains) to 1.5 (senses domain), with an average total score of 0.9 (Figure 1; Supplementary Table 1). Half of the scores were 0 (floor) for the eyes domain. There were few domain scores of 4 (ceiling), almost all of which were in the senses domain, where 17% were the maximum value of 4.

Figure 1.

Mean inFLUenza Patient-Reported Outcome Plus (FLU-PRO Plus) domain and total scores, by day, in the first 2 weeks participants were enrolled in the Epidemiology, Immunology, and Clinical Characteristics of Pandemic Infectious Diseases (EPICC) cohort. Abbreviation: GI, gastrointestinal.

Open in new tabDownload slide

Reliability

For all domains, Cronbach α was high as a measure of internal consistency (nose=0.88, throat=0.88, eyes=0.87, systemic=0.86, gastrointestinal=0.88, respiratory=0.86, senses=0.93, total=0.86). Among those who reported no change in their symptoms from the prior day, the mean differences in the domain scores from one day to the next were small (ranging from 0.07 [eyes, gastrointestinal domains] to 0.14 [nose domain], from day 1 to day 2), while the ICCs ranged between 0.68 (nose domain) to 0.94 (senses domain) (also from day 1 to day 2) (Supplementary Table 2). Similar relationships were seen for the other day to day changes within the first week of FLU-PRO Plus surveys.

Construct Validity

We observed moderate to high correlation between the PGA rating of disease severity and many of the symptom domains on day 1, with the highest correlations in the systemic, respiratory, and gastrointestinal domains, in addition to the total scores (Table 2). The domain that had the lowest correlation with the PGAs was the senses domain, followed by the nose domain.

Known-Groups Validity

Overall, mean values of domain and total scores increased with increasing severity rating from mild to severe (Table 3). Mean (SD) total scores were lowest in the no/mild symptoms group (0.6 [0.5]), followed by the moderate group (1.4 [0.6]) and the severe group (1.9 [0.5]). Similar patterns were observed for the domain scores. The differences between the mild group and the moderate group were highly statistically significant (P<.0001) in the total scores and many of the domains. Similarly, the differences between the mild and the severe group were also statistically significant for many of the domains. There were fewer differences between the moderate and the severe group.

Responsiveness

Changes in FLU-PRO Plus total and domain scores over time are shown in Figure 1, demonstrating decreasing scores over time from days 1 to 14. Comparing those reporting return to usual health (“responders”) with those who did not (“nonresponders”), nonresponders often had higher baseline scores than responders (Table 4), whereas no such differences were observed in those who reported returning to usual activities. Supporting responsiveness to change, responders often had greater absolute differences in scores over time compared to nonresponders, especially for the senses domain. Statistically significant differences between responders and nonresponders were observed for all domains using an ANCOVA analysis adjusting for baseline scores for return to usual health, and except for throat for return to usual activities.

DISCUSSION

Results of this study indicate that FLU-PRO Plus scores are reliable and reproducible, have construct and known-groups validity, and demonstrate responsiveness to change in patients with COVID-19. Internal consistency was high for both domain and total scores. Two-day test-retest reliability over 7 days was moderate to strong for both domain and total scores.

Standardized, comprehensive, and interpretable measurements of patients’ symptoms are important for accurate characterization of the impact of disease on patients’ health. Prior research shows that nonstandardized clinician-derived questions may miss important aspects of a disease, or inaccurately represent the frequency of symptoms [21, 22]. Clinicians and patients may differ on symptoms that each consider “important.” Research in other diseases, like human immunodeficiency virus, show that patients’ views of symptoms were more predictive of health outcomes than clinician-derived symptoms [23, 24]. Capturing symptoms in a comprehensive and standardized way, using words that are meaningful to patients, is important, both to accurately characterize illness and to be able to compare results across studies.

FLU-PRO Plus performed similarly in COVID-19 as in other studies evaluating FLU-PRO in influenza-positive and -negative patients [17, 18]. Total and domain scores were on average lower in this study. Symptoms were present in all domains, confirming the relevance and importance of all symptoms captured for COVID-19, like that published previously for other viral respiratory diseases. Studies that evaluate only respiratory symptoms or symptoms considered by clinicians to be most important may miss the impacts of multiple symptom domains on patients’ health in response to respiratory viral disease as well as fail to accurately measure treatment effects of interventions.

Supporting construct validity, FLU-PRO Plus scores were related to patient global impressions of disease severity, physical health, and interference in daily activities. As in a prior study evaluating FLU-PRO [17], correlations with domain/total scores were least in comparison to physical health. Scores were lowest in patients rating symptoms as mild and, reflecting increased symptom intensity, scores increased with increasing patient-reported symptom severity. The absence of statistical significance in some comparisons to patients with severe disease may be due to the small numbers of participants in the severe category.

A quarter of participants with altered/lost sense of taste and smell showed ceiling effects, demonstrating that patients may experience these symptoms as an all-or-nothing (present/absent), rather than graded, impact. However, anosmia and ageusia do not seem to affect overall return to usual health (low correlation with patient global impressions of disease), as patients may be able to function in their daily lives even with impaired senses of taste and smell. Further research is needed regarding whether the senses domain should be retained in the FLU-PRO Plus, as anosmia and ageusia may not directly relate to severity.

While these results demonstrate the validity of FLU-PRO Plus in COVID-19, there were several limitations of this study, including that many patients in this sample had mild or no symptoms affecting the resulting scores and absolute magnitude of change. The participants in this study began filling out the FLU-PRO Plus surveys a median of 6 days post–symptom onset; therefore, the symptom scores were lower compared to previously published FLU-PRO studies in which participants began filling out their surveys closer to the time of onset. In addition to symptom scores being low overall, there were few patients in the sample who regarded their disease as severe; therefore, some P values may not reach statistical significance due to small numbers of patients in those groups, and the low number of hospitalized participants precluded a subanalysis in that group. Regardless, the instrument was still able to detect change in symptoms over time.

To capture greatest symptom burden, analysis was limited to participants who started responses to FLU-PRO Plus within 1 week of symptom onset. However, this reduced the sample size, reflecting that many presented to receive care days to weeks after development of symptoms, likely due to concern about access or exposure in the overburdened healthcare system during the first months of the pandemic. The individuals who were enrolled later in their illness course had lower domain and total scores, as expected, since they were often enrolled after symptom resolution. We assessed the performance properties of FLU-PRO Plus in the subset of participants who presented for care within 1 week post–symptom onset in order to identify a subpopulation more comparable to the populations in which FLU-PRO characterizations have previously been used and in whom responsiveness to change over time could be most accurately evaluated; however, when we performed these analyses without the time post–symptom onset restriction, the relationships were very similar (data not shown).

The use of a standardized, well-validated PRO instrument will facilitate a better understanding of the clinical course of COVID-19 across various patient populations, help us understand various phenotypes of patient presentations, allow for the exploration of their relationship to short- and long-term patient outcomes, and facilitate clinical trials evaluating treatment and prevention of COVID-19. The results of this analysis demonstrate that FLU-PRO Plus is valid for use in evaluating symptoms in SARS-CoV-2 infection, as has been shown for a variety of other respiratory infections [15, 17, 18, 25], with excellent reliability, construct validity, known-groups validity, and responsiveness to change. FLU-PRO Plus is currently being used in multiple observational studies and clinical trials [26, 27].

In conclusion, the results presented here suggest that FLU-PRO Plus scores are reliable, valid, and responsive to change in patients with laboratory-confirmed COVID-19. This instrument is valid for use in studies of prevention and treatment of COVID-19 as well as other viral respiratory diseases.

Supplementary Data

Supplementary materials are available at Open Forum Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Acknowledgments. We appreciate the Epidemiology, Immunology, and Clinical Characteristics of Pandemic Infectious Diseases (EPICC) participants for their central role in this study. Many thanks to the Infectious Diseases Clinical Research Program (IDCRP) team at the clinical research sites—physician/clinical investigators, site managers, regulatory staff, clinical research coordinators, and laboratory personnel—for their support of this study and contributions to its success under very challenging circumstances. The authors would like to thank Marietta Grother for her editorial assistance.

Copyright statement. A. M. W. M., R. M., R. M. M., C. J. C., D. A. L., N. H., D. T. L., S. B., M. S., D. R. T., T. H. B., and J. H. P. are service members or employees of the US government. This work was prepared as part of their official duties. Title 17 U.S.C. §105 provides that “copyright protection under this title is not available for any work of the United States Government.” Title 17 U.S.C. §101 defines a US government work as a work prepared by a military service member or employee of the US government as part of that person’s official duties.

Disclaimer. The contents of this publication are the sole responsibility of the author(s) and do not necessarily reflect the views, opinions, or policies of the Uniformed Services University of the Health Sciences (USUHS); the Department of Defense (DoD); the Departments of the Army, Navy, or Air Force; Brooke Army Medical Center; Tripler Army Medical Center; Carl R. Darnall Army Medical Center, Fort Hood; Walter Reed National Military Medical Center; Naval Medical Center San Diego; Madigan Army Medical Center; William Beaumont Army Medical Center; Fort Belvoir Community Hospital; Naval Medical Center Portsmouth; the National Institutes of Health (NIH); or the Henry M. Jackson Foundation for the Advancement of Military Medicine, Inc (HJF). The investigators have adhered to the policies for protection of human subjects as prescribed in 45 Code of Federal Regulations 46. The content of this publication does not necessarily reflect the views or policies of the Department of Health and Human Services, nor does mention of trade names, commercial products, or organizations imply endorsement by the US government.

Financial support. This work was supported by awards from the Defense Health Program (award number HU00012020067) and the National Institute of Allergy and Infectious Diseases (NIAID) (award number HU00011920111). The protocol was executed by the IDCRP, a DoD program executed by the USUHS through a cooperative agreement by HJF. This project has been funded in part by the NIAID at the NIH, under interagency agreement Y1-AI-5072. This project was funded in whole or in part with federal funds from the National Cancer Institute, NIH (contract number HHSN261200800001E 75N910D00024, task order number 75N91019F00130).

Potential conflicts of interest. S. D. P., T. H. B., and D. R. T. report that the Uniformed Services University (USU) IDCRP, a US DoD institution, and HJF were funded under a cooperative research and development agreement to conduct an unrelated phase 3 COVID-19 monoclonal antibody immunoprophylaxis trial sponsored by AstraZeneca. The HJF, in support of the USU IDCRP, was funded by the DoD Joint Program Executive Office for Chemical, Biological, Radiological, and Nuclear Defense to augment the conduct of an unrelated phase 3 vaccine trial sponsored by AstraZeneca. Both of these trials were part of the US government COVID-19 response. Neither is related to the work presented here. All other authors report no potential conflicts of interest.

All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.

APPENDIX

We thank the members of the Epidemiology, Immunology, and Clinical Characteristics of Pandemic Infectious Diseases (EPICC) COVID-19 Cohort Study Group for their many contributions in conducting the study and ensuring effective protocol operations. The following members were all closely involved with the design, implementation, and/or oversight of the study and have met group authorship criteria for this manuscript:

Brooke Army Medical Center, Fort Sam Houston, Texas: Col J. Cowden; LTC M. Darling; T. Merritt; CPT T. Wellington.

Fort Belvoir Community Hospital, Fort Belvoir, Virginia: A. Rutt.

Madigan Army Medical Center, Joint Base Lewis McChord, Washington: S. Chambers; W. Robb-McGrath.

Naval Medical Center San Diego, San Diego, California: CDR C. Berjohn; N. Kirkland.

Uniformed Services University of the Health Sciences, Bethesda, Maryland: C. Broder; C. Byrne; M. Fritschlanski; COL P. Hickey; E. Laing; LTC J. Livezey; E. Parmelee; J. Rusiecki; A. Scher.

Womack Army Medical Center, Fort Bragg, North Carolina: B. Barton; LTC D. Hostler; LTC (Ret) J. Hostler; MAJ K. Lago; C. Maldonado.

William Beaumont Army Medical Center, El Paso, Texas: M. Wayman.

The authors wish to also acknowledge all who have contributed to the EPICC COVID-19 study:

Brooke Army Medical Center, Fort Sam Houston, Texas: Col J. Cowden; LTC M. Darling; S. DeLeon; Maj D. Lindholm; LTC A. Markelz; K. Mende; S. Merritt; T. Merritt; LTC N. Turner; CPT T. Wellington.

Carl R. Darnall Army Medical Center, Fort Hood, Texas: LTC S. Bazan; P. K. Love.

Fort Belvoir Community Hospital, Fort Belvoir, Virginia: N. Dimascio-Johnson; MAJ E. Ewers; LCDR K. Gallagher; LCDR D. Larson; A. Rutt.

Henry M. Jackson Foundation, Inc, Bethesda, Maryland: P. Blair; J. Chenoweth; D. Clark.

Madigan Army Medical Center, Joint Base Lewis McChord, Washington: S. Chambers; LTC C. J. Colombo; R. Colombo; CAPT C. Conlon; CAPT K. Everson; COL P. Faestel; COL T. Ferguson; MAJ L. Gordon; LTC S. Grogan; CAPT S. Lis; COL C. Mount; LTC D. Musfeldt; CPT D. Odineal; LTC M. Perreault; W. Robb-McGrath; MAJ R. Sainato; C. Schofield; COL C. Skinner; M. Stein; MAJ M. Switzer; MAJ M. Timlin; MAJ S. Wood.

Naval Medical Center Portsmouth, Portsmouth, Virginia: S. Banks; R. Carpenter; L. Kim; CAPT K. Kronmann; T. Lalani; LCDR T. Lee; LCDR A. Smith; R. Smith; R. Tant; T. Warkentien.

Naval Medical Center San Diego, San Diego, California: CDR C. Berjohn; S. Cammarata; N. Kirkland; CAPT (Ret) R. Maves; CAPT (Ret) G. Utz.

Tripler Army Medical Center, Honolulu, Hawaii: S. Chi; LTC R. Flanagan; MAJ M. Jones; C. Lucas; LTC C. Madar; K. Miyasato; C. Uyehara.

Uniformed Services University of the Health Sciences, Bethesda, Maryland: B. Agan; L. Andronescu; A. Austin; C. Broder; CAPT T. Burgess; C. Byrne; COL K Chung; J. Davies; C. English; N. Epsi; C. Fox; M. Fritschlanski; M. Grother; A. Hadley; COL P. Hickey; E. Laing; LTC C. Lanteri; LTC J. Livezey; A. Malloy; R. Mohammed; C. Morales; P. Nwachukwu; C. Olsen; E. Parmelee; S. Pollett; S. Richard; J. Rozman; J. Rusiecki; E. Samuels; M. Sanchez; A. Scher; CDR M. Simons; A. Snow; K. Telu; D. Tribble; L. Ulomi.

United States Air Force School of Aerospace Medicine, Dayton, Ohio: Sgt T. Chao; R. Chapleau; A. Fries; C. Harrington; S. Huntsberger; S. Purves; K. Reynolds; J. Rodriguez; C. Starr.

Womack Army Medical Center, Fort Bragg, North Carolina: B. Barton; LTC D. Hostler; LTC (Ret) J. Hostler; MAJ K. Lago; C. Maldonado; J. Mehrer.

William Beaumont Army Medical Center, El Paso, Texas: MAJ T. Hunter; J. Mejia; R. Mody; R. Resendez; P. Sandoval; M. Wayman.

Walter Reed National Military Medical Center, Bethesda, Maryland: I. Barahona; A. Baya; A. Ganesan; MAJ N. Huprikar; B. Johnson.

Walter Reed Army Institute of Research, Silver Spring, Maryland: S. Peel.

References