Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2018 Nov 20;37(26):3776-3788.
doi: 10.1002/sim.7844. Epub 2018 Jun 5.

Augmented outcome-weighted learning for estimating optimal dynamic treatment regimens

Ying Liu  1 , Yuanjia Wang  2 , Michael R Kosorok  3 , Yingqi Zhao  4 , Donglin Zeng  3
Affiliations

Augmented outcome-weighted learning for estimating optimal dynamic treatment regimens

Ying Liu et al. Stat Med. .

Abstract

Dynamic treatment regimens (DTRs) are sequential treatment decisions tailored by patient's evolving features and intermediate outcomes at each treatment stage. Patient heterogeneity and the complexity and chronicity of many diseases call for learning optimal DTRs that can best tailor treatment according to each individual's time-varying characteristics (eg, intermediate response over time). In this paper, we propose a robust and efficient approach referred to as Augmented Outcome-weighted Learning (AOL) to identify optimal DTRs from sequential multiple assignment randomized trials. We improve previously proposed outcome-weighted learning to allow for negative weights. Furthermore, to reduce the variability of weights for numeric stability and improve estimation accuracy, in AOL, we propose a robust augmentation to the weights by making use of predicted pseudooutcomes from regression models for Q-functions. We show that AOL still yields Fisher-consistent DTRs even if the regression models are misspecified and that an appropriate choice of the augmentation guarantees smaller stochastic errors in value function estimation for AOL than the previous outcome-weighted learning. Finally, we establish the convergence rates for AOL. The comparative advantage of AOL over existing methods is demonstrated through extensive simulation studies and an application to a sequential multiple assignment randomized trial for major depressive disorder.

Keywords: Q-learning; SMARTs; adaptive intervention; individualized treatment rule; machine learning; outcome-weighted learning; personalized medicine.

PubMed Disclaimer

Figures

FIGURE 1
FIGURE 1
Simulation setting 1 with four-stage design (optimal value = 10.1). AOL, Augmented Outcome-weighted Learning. OWL, outcome-weighted learning [Colour figure can be viewed at wileyonlinelibrary.com]
FIGURE 2
FIGURE 2
Simulation setting 2 with four-stage design (optimal value = 4.0). AOL, Augmented Outcome-weighted Learning; OWL, outcome-weighted learning [Colour figure can be viewed at wileyonlinelibrary.com]
FIGURE 3
FIGURE 3
Mean and standard error of the value function (depression symptom score, Quick Inventory of Depressive Symptomatology) based on 1000 repetitions of two-fold cross validation for Sequenced Treatment Alternatives to Relieve Depression data (lower score desirable)
FIGURE 4
FIGURE 4
Normalized coefficients of the stage 1 tailoring variables (left panel) and stage 2 tailoring variables (right panel) obtained by Augmented Outcome-weighted Learning. QIDS, Quick Inventory of Depressive Symptomatology
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Lavori PW, Dawson R. A design for testing clinical strategies: biased adaptive within-subject randomization. J Royal Stat Soc Ser A Stat Soc. 2000;163(1):29–38.
    1. Thall PF, Sung HG, Estey EH. Selecting therapeutic strategies based on efficacy and death in multicourse clinical trials. J Am Stat Assoc. 2002;97(457):29–39.
    1. Lunceford JK, Davidian M, Tsiatis AA. Estimation of survival distributions of treatment policies in two-stage randomization designs in clinical trials. Biometrics. 2002;58(1):48–57. - PubMed
    1. Rush AJ, Fava M, Wisniewski SR, et al. Sequenced treatment alternatives to relieve depression (STAR* D): rationale and design. Contemp Clin Trials. 2004;25(1):119–142. - PubMed
    1. Murphy SA. An experimental design for the development of adaptive treatment strategies. Stat Med. 2005;24(10):1455–1481. - PubMed

Publication types

LinkOut - more resources