Clinical trials with anti-tau drugs will need to target individuals at risk of accumulating tau. Our objective was to identify variables available in a research setting that predict future rates of tau PET accumulation separately among individuals who were either cognitively unimpaired or cognitively impaired. All 337 participants had: a baseline study visit with MRI, amyloid PET, and tau PET exams, at least one follow-up tau PET exam; and met clinical criteria for membership in one of two clinical diagnostic groups: cognitively unimpaired (n = 203); or cognitively impaired (n = 134, a combined group of participants with either mild cognitive impairment or dementia with Alzheimer's clinical syndrome). Our primary analyses were in these two clinical groups; however, we also evaluated subgroups dividing the unimpaired group by normal/abnormal amyloid PET and the impaired group by clinical phenotype (mild cognitive impairment, amnestic dementia, and non-amnestic dementia). Linear mixed effects models were used to estimate associations between age, sex, education, APOE genotype, amyloid and tau PET standardized uptake value ratio (SUVR), cognitive performance, cortical thickness, and white matter hyperintensity volume at baseline, and the rate of subsequent tau PET accumulation. Log-transformed tau PET SUVR was used as the response and rates were summarized as annual per cent change. A temporal lobe tau PET meta-region of interest was used. In the cognitively unimpaired group, only higher baseline amyloid PET was a significant independent predictor of higher tau accumulation rates (P < 0.001). Higher rates of tau accumulation were associated with faster rates of cognitive decline in the cognitively unimpaired subgroup with abnormal amyloid PET (P = 0.03), but among the subgroup with normal amyloid PET. In the cognitively impaired group, younger age (P = 0.02), higher baseline amyloid PET (P = 0.05), APOE ε4 (P = 0.05), and better cognitive performance (P = 0.05) were significant independent predictors of higher tau accumulation rates. Among impaired individuals, faster cognitive decline was associated with faster rates of tau accumulation (P = 0.01). While we examined many possible predictor variables, our results indicate that screening of unimpaired individuals for potential inclusion in anti-tau trials may be straightforward because the only independent predictor of high tau rates was amyloidosis. In cognitively impaired individuals, imaging and clinical variables consistent with early onset Alzheimer's disease phenotype were associated with higher rates of tau PET accumulation suggesting this may be a highly advantageous group in which to conduct proof-of-concept clinical trials that target tau-related mechanisms. The nature of the dementia phenotype (amnestic versus non-amnestic) did not affect this conclusion.