title: "Adaptive Prespecification - Vignette"

author: "Laura B. Balzer (laura.balzer@berkeley.edu) "

date: "`r Sys.Date()`"

output: rmarkdown::html_vignette

vignette: >

```{r setup, include = FALSE}

Here we provide worked examples of using Adaptive Prespecification (APS) for empirical efficiency maximization in randomized trials. From a pre-specified set, APS is used within TMLE to data-adaptively select the optimal combination of estimators of the *outcome regression* (i.e., conditional expectation of the outcome, given the randomized intervention and candidate covariates) and of the known *propensity score* (i.e., conditional probability of the intervention, given the candidate covariates) to minimize the cross-validated variance estimate.

Key methods references include

- Balzer et al., [Adaptive pre-specification](https://pubmed.ncbi.nlm.nih.gov/27436797/) in randomized trials with and without pair-matching, *Statistics in Medicine*, 2016

- Balzer et al., [Two-Stage TMLE](https://pubmed.ncbi.nlm.nih.gov/34939083/) to reduce bias and improve efficiency in cluster randomized trials, *Biostatistics*, 2021

- Balzer et al., [Adaptive Selection](https://arxiv.org/abs/2210.17453) of the Optimal Strategy to Improve Precision and Power in Randomized Trials, *arXiv*, 2022

Example applications include

- Havlir et al., [HIV Testing and Treatment](https://pubmed.ncbi.nlm.nih.gov/31314966/) with the Use of a Community Health Approach in Rural Africa, *NEJM*, 2019 with corresponding Statistical Analysis Plan [(SAP)](https://arxiv.org/abs/1808.03231)

- Kakende et al., [A mid-level health manager intervention](https://pubmed.ncbi.nlm.nih.gov/35908553/) to promote uptake of isoniazid preventive therapy among people with HIV in Uganda: a cluster randomised trial, *LancetHIV*, 2022 with corresponding [SAP](https://arxiv.org/abs/2111.10467)

- Hickey et al., [Effect of a one-time financial incentive](https://pubmed.ncbi.nlm.nih.gov/36342940/) on linkage to chronic hypertension care in Kenya and Uganda: A randomized controlled trial, *PLoSOne*, 2022 (corresponding SAP included in article's supplementary materials)

For demonstration, we will use real data from the [AIDS Clinical Trials Group (ACTG) Study 175](https://pubmed.ncbi.nlm.nih.gov/8813038/). ACTG 175 was an individually randomized trial to evaluate the impact of monotherapy vs. combination therapy among persons with HIV. The data are publicly available through the [`speff2trial` package](https://CRAN.R-project.org/package=speff2trial) by Juraska. For demonstration, we are focusing on adults, aged 18+ years. After loading the data, we do a bit of pre-processing to create binary indicators of being "young" (aged 18-30years), having a baseline CD4 count >350 c/mm3, having a baseline CD8 count >350 c/mm3, and starting on antiretroviral therapy (ART) within 1-52 weeks of baseline.

```{r, message=F}

APS is applicable to both individually randomized trials and cluster randomized trials; therefore, we need to specify the independent unit with `id`. Additionally, we need to create a dummy indicator `U` equal to 1 for the unadjusted estimator, which is always included as a candidate. In cluster randomized trials, weights (`alpha`) can be included to target effect individual-level or cluster-level effects; see [Benitez et al. (2021)](https://arxiv.org/abs/2110.09633) for details. In this individually randomized trial, set `alpha=1`. Finally, we specify the treatment indicator `A`, where $A=1$ for the intervention and $A=0$ for the control. We will specify the outcome `Y` below.

As candidate adjustment variables, we consider demographic variables (e.g., age, gender), measures of disease severity (e.g., Karnofsky score, being symptomatic), history of ART use, and baseline measures of CD4 and CD8 counts. We refer to the help file (`help(ACTG175)`) for more information

These characteristics are summarized by arm and overall in the following Table. Continuous variables are shown as median [Q1, Q2] and binary variables as N (%).

```{r, echo=F, include=F, results='hide'}

```{r, echo=F}

We consider two implementations of APS:

1. ["Small APS":](https://pubmed.ncbi.nlm.nih.gov/27436797/) The candidate estimators of the outcome regression and propensity score are limited to "working" generalized linear models (GLMs) with at most one adjustment covariate. This approach is recommended for small sample size ($N<40$).

2. ["Large APS":](https://arxiv.org/abs/2210.17453) The candidate estimators now consider adjusting for multiple covariates. The algorithms currently coded are main terms (`glm`), stepwise regression (`stepwise`), LASSO (`lasso`), multivariate adaptive regression splines (MARS; `mars`), and MARS after screening based on pairwise correlations (`mars.corp`). The candidate estimators in the Small APS implementation (i.e., working GLMs with at most one adjustment covariate) are also included in the Large APS implementation.

The `get.cand.adj` function generates the set of candidate learners. We input the candidate covariates (`all.cand`), estimators of the outcome regression (`cand.Qform.fancy`), and estimators of the propensity score (`cand.gform.fancy`). Setting `cand.Qform.fancy` and `cand.gform.fancy` to `NULL` will return working GLMs with at most 1 adjustment variable.

We first demonstrate implementation with a continuous outcome: CD4 count at 20 +/- 5 weeks and for estimation of the effect on the difference scale (i.e., ATE).

For comparison, we first consider the unadjusted estimator.

```{r, warning=F}

The `Stage2` returns point estimates, 95% confidence intervals (CIs), and standard error estimates arm-specific endpoints (denoted `Txt` for intervention and `Con` for control) as well as the intervention effect on the inputted scale. The p-value for null hypothesis testing is also generated, and `reject` indicates if the relevant null hypothesis was rejected at the selected significance level.

Additional output used for simulation studies when we know the true value of the effect (`psi`) include `bias` and `cover` (indicating the 95%CI include the true value). Finally, `QAdj` and `Qform` indicate the selection of the adjustment variables and their form for the outcome regression, while `gAdj` and `gform` indicate the selection of the adjustment variables and their form for the propensity score.

Here, `QAdj=gAdj=1` and `Qform=gform=glm` indicate adjusting for the dummy variable $U$ as a main term in a working regression; this is equivalent to the unadusted estimator.

Also for comparison, we consider a TMLE with fixed adjustment for `age` in the outcome regression and for `gender` in the propensity score.

```{r, warning=F}

The output will return `QAdj=-99` and `gAdj=-99` if fixed adjustment is being used.

We now consider the small sample implementation of APS in TMLE. We now set `do.data.adapt=T`, specify the number of folds in cross-validation (`V`), as well as the candidate adjustment variables and estimators:

```{r, warning=F}

The outcome at baseline `cd40` was selected for adjustment in both the outcome regression and propensity score. Let's examine the added benefit of collaborative estimation of the propensity score by setting `gAdj=NULL` and `gform=glm`. This will generate a TMLE only adjusting in the outcome regression. For demonstration, we will use the same approach for estimating the outcome regression that was selected previously.

```{r, warning=F}

We now consider the large sample implementation of APS in TMLE.

```{r, warning=F}

In the large sample implementation, the outcome regression was estimated with MARS, and, as before, the propensity score was estimated with working GLM adjusting for `cd40`. Let's examine the added benefit of adaptive adjustment in the propensity score by setting `gAdj=NULL` and `gform=glm`. This will generate a TMLE only adjusting in the outcome regression, which was previously selected through our adaptive approach.

```{r, warning=F}

Comparative results with a **continous outcome** for arm-specific outcomes (95%CI) and the intervention effect (95%CI).

+ `Rel.Var.` is the estimated variance of a given approach to that of the unadjusted approach.

+ `Savings` is the estimated reduction in sample size from using an adjusted approach, assuming negligible bias.

+ `Out.Reg` is the fixed or adaptively selected estimator for the outcome regression.

+ `PScore` is the fixed or adaptively selected estimator for the propensity score.

+ `Small TMLE` and `Large TMLE` refer to using APS only to select of the outcome regression estimator in the small-trial and large-trial implementation, respectively.

+ `Small CTMLE` and `Large CTMLE` refer to using APS for selection of the outcome regression estimator and **collaborative** selection of the known propensity score estimator in the small-trial and large-trial implementation, respectively.'

```{r, echo=F}

For demonstration with smaller sample sizes, as seen in subgroup analyses, we now examine effects defined within strata defined by baseline age group (18-30 years vs. 31+ years) and gender.

See `aps_wrapper()` function within `MakePretty_App.R` for the wrapper function used to generate estimates from the algorithms under consideration.

```{r, warning=F}

```{r, echo=F, include=F, results='hide'}

We now consider binary outcome that CD4 count at 20 week window is >350. For demonstration, we now do effect estimation on the relative scale (i.e., arithmetic risk ratio).

```{r, warning=F}

Comparative results with a **binary outcome** for arm-specific outcomes and intervention effect, overall and for select subgroups.

```{r, echo=F}

```{r, warning=F}

```{r, echo=F, include=F, results='hide'}

We now demonstrate out method's ability to preserve Type-1 error control. To do so, we permute the treatment indicator to make the null hypothesis true ($\psi=0$ for the continuous outcome and $\psi=1$ for the binary outcome). Then we implement each of the estimators and repeat 5000 times.

```{r, results='hide', eval=F}

```{r, echo=F, include=F, results='hide'}