Utility of the Fitbit Flex to Evaluate Sleep in Major Depressive Disorder: A comparison against polysomnography and wrist-worn actigraphy

1. Introduction

Sleep disturbance is very common in patients with Major Depressive Disorder (MDD). It has been estimated that up to 90% of individuals with MDD experience reduced sleep quality during a depressive episode (Tsuno et al., 2005). Depression can be accompanied by a diverse range of sleep disturbances including insomnia (difficulty falling asleep, maintaining sleep, or waking up too early) and/or hypersomnolence (excessive daytime sleepiness and/or sleep duration (Soehner et al., 2014). Objective changes in sleep continuity and duration in MDD, as measured by polysomnography, are robust physiological indicators of sleep disturbance in the disorder (Benca et al., 1992; Steiger and Kimura, 2010; Pillai et al., 2011; Plante et al., 2017). Sleep disturbance is also associated with treatment resistance, symptomatic relapse, suicidality, and impaired daytime function, underscoring its importance in the course of affective illness (Baglioni et al., 2011; McCall et al., 2010; Nadorff et al., 2013; Perlis et al., 1997; Riemann and Voderholzer, 2003; Szklo-Coxe et al., 2010). Thus, the ability to quantify sleep duration and continuity in patients with MDD is of potentially high value in the assessment and treatment of patients with mood disorders.

Polysomnography (PSG) is considered the gold standard for objective sleep measurement, however its widespread applicability is limited by its time-intensiveness, high cost, and intrusiveness (Meltzer et al., 2015; Montgomery-Downs et al., 2012). Furthermore, PSG is typically unable to provide information on longitudinal sleep-wake patterns over a multiple night assessment period (Meltzer et al., 2015). The use of validated actigraphs that utilize wrist-worn accelerometry to quantify movement as a surrogate measure for sleep and wake can circumvent some of these shortcomings of PSG due to their relatively low-cost, nonintrusiveness, and ambulatory capabilities (de Souza et al., 2003; Montgomery-Downs et al., 2012). Although actigraphy has been validated in its ability to identify sleep/wake times and patterns in adult populations (Morgenthaler et al., 2007) actigraphic devices may tend to overestimate total sleep time (de Souza et al., 2003; Montgomery-Downs et al., 2012). This deficiency stems largely from an inability to correctly identify wake epochs (Marino et al., 2013) as these devices rely upon an accelerometer (movement detector) as the sole measurement for sleep/wake designation, which inherently leaves them vulnerable to classifying periods of inactivity as sleep regardless of vigilance state. In addition, most validated actigraphs used in clinical and research settings generally require patients to return the device periodically (typically 2–4 weeks) in order for data to be retrieved for evaluation, making their use over more extended periods cumbersome.

The rise of commercially-available fitness activity trackers has provided another low-cost, field-based, and user-friendly alternative that may prove useful in evaluating sleep for both clinical and research purposes (de Zambotti et al., 2015; Meltzer et al., 2015; Montgomery-Downs et al., 2012). These mass-marketed devices are gaining a broader acceptance in both general and patient populations, and practitioners have begun to integrate their use in the assessment and treatment of affective disorders, despite limited research evaluating their use in patients with psychiatric disorders (Vahia and Sewell, 2016). Besides their low cost, these devices typically leverage direct-to-consumer cloud-based platforms and/or mobile technologies to allow for continuous data collection and retrieval over time. Considering the widespread availability of these devices and their potential impact on the management of psychiatric illness, comparison of their performance in estimating sleep against gold standard PSG and other validated actigraphs is a vital area of inquiry.

In a previous investigation conducted in healthy adults, a fitness activity tracker (the inaugural version of the Fitbit ) overestimated total sleep time and congruently demonstrated an inability to correctly identify wake epochs when compared against a commonly used brand of actigraphy (Actiwatch 64) and PSG (Montgomery-Downs et al., 2012). Although not many validation studies have been performed on fitness activity trackers in the domain of sleep, this demonstration of an overestimation for total sleep time and inability to accurately identify wake epochs has been corroborated by studies on women with insomnia (de Zambotti et al., 2015) and adolescents referred for clinical PSG (Meltzer et al., 2015). Contrary to the results of these investigations, one study in healthy, young adults demonstrated comparable results in estimation of total sleep time for multiple fitness activity trackers, relative to PSG (Mantua et al., 2016). However, because epoch-by-epoch comparisons were not performed, the full performance characteristics of the fitness trackers utilized in this study, relative to PSG, could not be determined. The inconsistent results of existing validation studies - particularly in regards to the estimation of total sleep time and identification of wake periods - suggests a need to further investigate the true capabilities of these devices, with an emphasis on elucidating their utility within specific disorders. To our knowledge, no prior research has evaluated the validity of a commercially-available fitness activity tracker in persons with affective illness.

Thus, to further extend this line of inquiry into patients with affective illness, the primary aim of this investigation was to examine the utility of a commercially-available fitness activity tracking device, the Fitbit Flex (FBF, Fitbit Inc.; San Francisco, CA), against both PSG and validated actigraphy, the Actiwatch 2 (AW-2; Phillips Respironics), in a well-characterized cohort of adult patients with MDD.

2. Methods

3. Results

4. Discussion

To our knowledge, this is the first investigation to examine the utility of the Fitbit Flex as a sleep measurement device in adult patients with major depressive disorder, and as such may significantly impact both clinical care and research in affective illness. As it becomes more commonplace in clinical practice for patients to report data obtained from commercially available fitness activity trackers (Meltzer et al., 2015; Vahia and Sewell, 2016), elucidating the capabilities and shortcomings of these devices is increasingly important in the delivery of care. Furthermore, from a research perspective, clarifying the potential for these devices to accurately estimate sleep is a vital step in interpreting longitudinal, field-based assessments of sleep-wake patterns in large cohorts of patients with psychiatric disorders that may employ these devices. The results of our investigation suggest that the FBF has some utility in quantifying sleep and wake in patients with MDD; however, there are several limitations of the device, particularly when compared to gold standard polysomnography.

The primary results of our study indicate that the FBF, under the normal (non-sensitive) settings, when compared to PSG, significantly overestimates sleep duration and efficiency, while underestimating wake after sleep onset. Furthermore, the FBF-N displays an inability to accurately identify wake epochs relative to PSG staging (specificity = 0.35). Interestingly, the FBF, under normal setting, performed comparably to a validated actigraph, the AW-2, in assessing total sleep time. Both devices significantly overestimated total sleep time when compared against PSG, but did not significantly differ when evaluated against one another, with a difference of only approximately 5 minutes between devices. Similarities in performance between the FBF-N and AW-2 were further substantiated by epoch-by-epoch comparisons relative to PSG, where the FBF-N and AW-2 displayed similar sensitivity, specificity, and accuracy.

It is noteworthy that the specificity of the FBF-N and AW-2 observed in our investigation were relatively similar (0.35 and 0.31, respectively). These results differ from prior studies that utilized the inaugural version of the Fitbit and Actiwatch-64 and demonstrated poorer specificities for Fitbit compared to standard actigraphy (0.198 and 0.389, respectively) (Montgomery-Downs et al., 2012). Since other investigations have demonstrated similarly poor specificity for standard actigraphs relative to PSG, frequently below 0.5 (Blood et al., 1997; Sivertsen et al., 2006; de Souza et al., 2003; Paquet et al., 2007), our findings suggest the specificity of the FBF (used under the normal setting) may be improved relative to older models of the device, potentially due to changes in the device algorithm or the hardware itself. However, the interpretation and generalizability of these findings must be considered in the context of the patient population studied, as well as the use of ad libitum overnight recordings, which could theoretically increase the potential for misidentification of wake-epochs due to longer evaluation periods with increased wake after sleep onset. Regardless, these results call into question the necessity of using more costly standard actigraphs over this commercially-available fitness activity tracker for field-based estimates of total sleep time in patients with affective disorders.

Despite these similarities between FBF-N and AW-2, there were important differences in their abilities to assess sleep onset latency and wake after sleep onset. The FBF-N more closely approximated sleep onset latency in comparison to PSG, but the AW-2 significantly overestimated this variable. Estimates of wake after sleep onset for both devices were significantly lower than PSG, but AW-2 estimates were closer to PSG than FBF-N. These inter-device differences are noteworthy as they may impact device selection for clinical assessment or research depending on the primary sleep variable of interest. For instance, if a patient’s main sleep complaint is difficulty falling asleep, then the FBF-N would theoretically be more optimal than the AW-2 given its stronger performance in assessing sleep onset latency. Ultimately, our data suggest neither device can replace PSG as a measure of nocturnal sleep; however, both devices make similar estimates of sleep duration to each other, with more variable estimates of sleep onset and continuity.

Notably, relative performance similarities between FBF and AW-2 only apply to the FBF in the normal (non-sensitive setting). FBF-S had markedly different performance relative to PSG, with large underestimates of sleep duration and efficiency, as well as over estimates of SOL and WASO. Our results are similar to a recent investigation in children with sleep disordered breathing that demonstrated the sensitive setting significantly underestimated sleep duration and efficiency in comparison to PSG (Meltzer et al., 2015). Thus, based on our results, the sensitive setting of the FBF should not be utilized either in the clinical treatment of MDD or research studies in these patients, as estimates of sleep are not accurate, particularly relative to the normal setting.

There are limitations of our study that merit discussion. Participants were predominantly young to middle-aged, and thus our results may not extend to other age groups with MDD, such as pediatric or geriatric populations. Second, our findings may not generalize equally to both sexes, as participants in this study were predominantly female. Moreover, findings cannot be directly extended to other mood and/or sleep disorders as our study specifically examined outpatients with unipolar MDD that was generally mild to moderate in severity. Also, due to logistical and security issues, collection and analysis of analysis of accelerometer-based data was performed on a computer that was not identical to that used to record polysomnography, which despite our best efforts to synchronize the devices within the local network, may have introduced error into epoch by epoch comparisons. Finally, our results cannot be extended to other fitness trackers, or more current generations of the same model (i.e. Fitbit Flex 2), as these devices may have different performance characteristics.

The rapid evolution of fitness activity trackers presents a major complication for the use of these devices in clinical and scientific contexts, and creates a necessity for frequent reevaluation of the validity of the most current generations of these devices, as underlying software and algorithmic improvements may yield more accurate estimations of sleep and wake. A major hurdle towards accomplishing this goal will be obtaining sufficient funding that is free from commercial influence to conduct research that informs personal consumers, clinical practitioners, and scientific researchers about the capabilities of these devices. In addition, the validity of a given device to measure sleep and wake remains continuously in question if the algorithm used to define vigilance states is proprietary and can be altered without notice to consumers. Thus, before these devices are adopted for scientific research on a broad scale, these issues will need to be addressed to ensure adequate scientific rigor of results obtained, which may include open access to user data and algorithmic transparency.

Similar to prior validation studies comparing PSG to wrist-worn accelerometry to measure sleep, only a single night of PSG was utilized in analysis. This research design, while highly pragmatic and cost-effective, leaves in question the true capabilities of the FBF as a long-term, longitudinal sleep measurement device. Previous research studies utilizing fitness activity trackers as a longitudinal sleep assessment tool have encountered data acquisition loss stemming mostly from user implementation error (Baroni et al., 2015), which may be an unavoidable limitation of these devices. Since commercial fitness trackers may be particularly useful for field-based, epidemiological research studies, further research is warranted to determine the reliability of these devices assessed over multiple nights, and develop implementation methods that might improve reliability.

In summary, this study demonstrates that the Fitbit Flex cannot fully serve as a proxy for gold standard polysomnography in the quantification of sleep and wake in major depressive disorder. However, this commercially-available fitness tracker does demonstrate similar performance characteristics to a standard actigraph, particularly in the estimation of total sleep duration, when used in the normal (non-sensitive) mode. Clinicians and researchers should consider the capabilities, limitations, and settings of these devices when interpreting data and/or treating patients, as it will likely impact treatment decisions and interpretation of empiric results. To provide a more substantial evidence base for the application of fitness trackers in clinical and research settings will require ongoing investigation to evaluate the utility of an ever-expanding number of these devices across the spectrum of psychiatric illness.