Beyond initial discovery of a pathogenic variant, establishing that a variant is recurrently associated with disease is important for understanding clinical impact and disease etiology. Disappointingly, our ability to characterize pathogenicity under varied circumstances is limited. Here we discuss the role of genetic and environmental background and how it affects variant penetrance and outcomes. Specifically, genetic and environmental settings determine penetrance, and we should expect lower penetrance where contexts are diverse. For example, when over 5000 ClinVar pathogenic and loss-of-function variants were assessed in two large biobanks, UK Biobank and BioMe, the mean penetrance was only 7%. This indicates that the participants in the family-based, clinical, and case-control studies that identified these variants were more homogenous and enriched for etiologic co-factors, and the winner’s curse was at play. We also emphasize that the outcome of interest can vary across conditions. The variant that causes hemoglobin S can increase the risk of death from sickling, lower the risk of death from malaria, or increase the risk of kidney disease, depending on the presence of other variants, the endemicity of malaria, and a suite of other factors. Overall, annotation on a single continuum from benign to pathogenic attempts to shoehorn a complex phenomenon into an overly simplistic framework. Variant effects often vary by context, and thus it is critical to assess potential pathogenicity in different settings. There is no panacea or easy fix, but we offer two recommendations for consideration. First, we need to routinely evaluate contexts such as sex and genetic ancestry by conducting stratified analyses and developing methods that can detect heterogenous effects (e.g. female-to-male allele proportion ratios). Second, we need to consistently document what we know about effect modifiers in our annotation databases. These are not the only possible approaches, but they begin to provide means to create robust annotations of pathogenicity.