Curated variation benchmarks for challenging medically relevant autosomal genes

doi:10.1038/s41587-021-01158-1

. 2022 May;40(5):672-680.

doi: 10.1038/s41587-021-01158-1. Epub 2022 Feb 7.

Curated variation benchmarks for challenging medically relevant autosomal genes

Justin Wagner¹, Nathan D Olson¹, Lindsay Harris¹, Jennifer McDaniel¹, Haoyu Cheng², Arkarachai Fungtammasan³, Yih-Chii Hwang³, Richa Gupta³, Aaron M Wenger⁴, William J Rowell⁴, Ziad M Khan⁵, Jesse Farek⁵, Yiming Zhu⁵, Aishwarya Pisupati⁵, Medhat Mahmoud⁵, Chunlin Xiao⁶, Byunggil Yoo⁷, Sayed Mohammad Ebrahim Sahraeian⁸, Danny E Miller^{9

10}, David Jáspez¹¹, José M Lorenzo-Salazar¹¹, Adrián Muñoz-Barrera¹¹, Luis A Rubio-Rodríguez¹¹, Carlos Flores^{11

12

13}, Giuseppe Narzisi¹⁴, Uday Shanker Evani¹⁴, Wayne E Clarke¹⁴, Joyce Lee¹⁵, Christopher E Mason¹⁶, Stephen E Lincoln¹⁷, Karen H Miga¹⁸, Mark T W Ebbert^{19

20

21}, Alaina Shumate^{22

23}, Heng Li², Chen-Shan Chin^#²⁴, Justin M Zook^#²⁵, Fritz J Sedlazeck^#²⁶

Affiliations

¹ Material Measurement Laboratory, National Institute of Standards and Technology, Gaithersburg, MD, USA.
² Department of Data Science, Dana-Farber Cancer Institute, Boston, MA, USA.
³ DNAnexus, Inc., Mountain View, CA, USA.
⁴ Pacific Biosciences, Menlo Park, CA, USA.
⁵ Human Genome Sequencing Center, Baylor College of Medicine, Houston, TX, USA.
⁶ National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA.
⁷ Genomic Medicine Center, Children's Mercy Kansas City, Kansas City, MO, USA.
⁸ Roche Sequencing Solutions, Santa Clara, CA, USA.
⁹ Department of Pediatrics, Division of Genetic Medicine, University of Washington and Seattle Children's Hospital, Seattle, WA, USA.
¹⁰ Department of Genome Sciences, University of Washington, Seattle, WA, USA.
¹¹ Genomics Division, Instituto Tecnológico y de Energías Renovables (ITER), Santa Cruz de Tenerife, Spain.
¹² CIBER de Enfermedades Respiratorias, Instituto de Salud Carlos III, Madrid, Spain.
¹³ Research Unit, Hospital Universitario N.S. de Candelaria, Santa Cruz de Tenerife, Spain.
¹⁴ New York Genome Center, New York, NY, USA.
¹⁵ Bionano Genomics, San Diego, CA, USA.
¹⁶ Department of Physiology and Biophysics, Weill Cornell Medicine, New York, NY, USA.
¹⁷ Invitae, San Francisco, CA, USA.
¹⁸ UC Santa Cruz Genomics Institute, University of California, Santa Cruz, Santa Cruz, CA, USA.
¹⁹ Sanders-Brown Center on Aging, University of Kentucky, Lexington, KY, USA.
²⁰ Department of Internal Medicine, Division of Biomedical Informatics, University of Kentucky, Lexington, KY, USA.
²¹ Department of Neuroscience, University of Kentucky, Lexington, KY, USA.
²² Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA.
²³ Center for Computational Biology, Whiting School of Engineering, Johns Hopkins University, Baltimore, MD, USA.
²⁴ DNAnexus, Inc., Mountain View, CA, USA. jchin@dnanexus.com.
²⁵ Material Measurement Laboratory, National Institute of Standards and Technology, Gaithersburg, MD, USA. jzook@nist.gov.
²⁶ Human Genome Sequencing Center, Baylor College of Medicine, Houston, TX, USA. Fritz.Sedlazeck@bcm.edu.

^# Contributed equally.

PMID: 35132260
PMCID: PMC9117392
DOI: 10.1038/s41587-021-01158-1

Curated variation benchmarks for challenging medically relevant autosomal genes

Justin Wagner et al. Nat Biotechnol. 2022 May.

. 2022 May;40(5):672-680.

doi: 10.1038/s41587-021-01158-1. Epub 2022 Feb 7.

Authors

Affiliations

¹ Material Measurement Laboratory, National Institute of Standards and Technology, Gaithersburg, MD, USA.
² Department of Data Science, Dana-Farber Cancer Institute, Boston, MA, USA.
³ DNAnexus, Inc., Mountain View, CA, USA.
⁴ Pacific Biosciences, Menlo Park, CA, USA.
⁵ Human Genome Sequencing Center, Baylor College of Medicine, Houston, TX, USA.
⁶ National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA.
⁷ Genomic Medicine Center, Children's Mercy Kansas City, Kansas City, MO, USA.
⁸ Roche Sequencing Solutions, Santa Clara, CA, USA.
⁹ Department of Pediatrics, Division of Genetic Medicine, University of Washington and Seattle Children's Hospital, Seattle, WA, USA.
¹⁰ Department of Genome Sciences, University of Washington, Seattle, WA, USA.
¹¹ Genomics Division, Instituto Tecnológico y de Energías Renovables (ITER), Santa Cruz de Tenerife, Spain.
¹² CIBER de Enfermedades Respiratorias, Instituto de Salud Carlos III, Madrid, Spain.
¹³ Research Unit, Hospital Universitario N.S. de Candelaria, Santa Cruz de Tenerife, Spain.
¹⁴ New York Genome Center, New York, NY, USA.
¹⁵ Bionano Genomics, San Diego, CA, USA.
¹⁶ Department of Physiology and Biophysics, Weill Cornell Medicine, New York, NY, USA.
¹⁷ Invitae, San Francisco, CA, USA.
¹⁸ UC Santa Cruz Genomics Institute, University of California, Santa Cruz, Santa Cruz, CA, USA.
¹⁹ Sanders-Brown Center on Aging, University of Kentucky, Lexington, KY, USA.
²⁰ Department of Internal Medicine, Division of Biomedical Informatics, University of Kentucky, Lexington, KY, USA.
²¹ Department of Neuroscience, University of Kentucky, Lexington, KY, USA.
²² Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA.
²³ Center for Computational Biology, Whiting School of Engineering, Johns Hopkins University, Baltimore, MD, USA.
²⁴ DNAnexus, Inc., Mountain View, CA, USA. jchin@dnanexus.com.
²⁵ Material Measurement Laboratory, National Institute of Standards and Technology, Gaithersburg, MD, USA. jzook@nist.gov.
²⁶ Human Genome Sequencing Center, Baylor College of Medicine, Houston, TX, USA. Fritz.Sedlazeck@bcm.edu.

^# Contributed equally.

PMID: 35132260
PMCID: PMC9117392
DOI: 10.1038/s41587-021-01158-1

Abstract

The repetitive nature and complexity of some medically relevant genes poses a challenge for their accurate analysis in a clinical setting. The Genome in a Bottle Consortium has provided variant benchmark sets, but these exclude nearly 400 medically relevant genes due to their repetitiveness or polymorphic complexity. Here, we characterize 273 of these 395 challenging autosomal genes using a haplotype-resolved whole-genome assembly. This curated benchmark reports over 17,000 single-nucleotide variations, 3,600 insertions and deletions and 200 structural variations each for human genome reference GRCh37 and GRCh38 across HG002. We show that false duplications in either GRCh37 or GRCh38 result in reference-specific, missed variants for short- and long-read technologies in medically relevant genes, including CBS, CRYAA and KCNE1. When masking these false duplications, variant recall can improve from 8% to 100%. Forming benchmarks from a haplotype-resolved whole-genome assembly may become a prototype for future benchmarks covering the whole genome.

PubMed Disclaimer

Conflict of interest statement

Competing Interests

AMW and WJR are employees and shareholders of Pacific Biosciences. AF and CSC are employees and shareholders of DNAnexus. SMES is an employee of Roche. JL is an employee of Bionano Genomics. SEL was an employee of Invitae. FJS has sponsored travel from Pacific Biosciences and Oxford Nanopore. The remaining authors declare no competing interests.

Figures

**Figure 1:**
GIAB developed a process to create new phased small variant and structural variant benchmarks for 273 challenging, medically relevant genes. (A) We developed a list of 4,701 autosomal potentially medically relevant genes. We generated a new benchmark for 273 of the 4,701 genes that were completely resolved by our hifiasm haplotype-resolveddiploid assembly and <=90% included in the v4.2.1 GIAB small variant benchmark for HG002 (V4.2.1 Regions). (B) We required that the entire gene region (pink) and 20 kb flanking sequence on each side (blue) were completely resolved by both haplotypes in the assembly (hifiasm Hap1 and hifiasm Hap2), indicated with the hifiasm Dipcall Bed track. In addition, we required that any segmental duplications overlapping the gene were completely resolved. From the small variant benchmark regions (CMRG Small Variant blue bars), we excluded SVs and any tandem repeats or homopolymers overlapping SVs (right TR and Homopol. region in brown). The left TR and Homopol. region in brown is excluded from the small variant benchmark regions because the larger tandem repeat contains an imperfect homopolymer longer than 20 bp, which we exclude because long homopolymers have a higher error rate in the assembly. All regions of this gene were included in the SV benchmark regions (CMRG Structural Variant blue bar). The vertical red lines in CMRG Small Variant and CMRG Structural Variant indicate locations of benchmark small variants and SVs, respectively. Finally, we evaluated the small variant and structural variant benchmarks with manual curation and long range PCR, and also ensured they accurately identify false positives and false negatives after excluding errors found during curation.

**Figure 2.**
The new CMRG benchmark contains more challenging variants and regions than previous benchmarks. (A) Fraction of each gene region (blue) and exonic regions (red) included in the new CMRG small variant or SV benchmark regions. (B) Comparison of fraction of challenging sequences and variants for genes included in the new CMRG benchmark vs. the previous v4.2.1 HG002 benchmark vs. genes excluded from both benchmarks. 99% of CMRG benchmark genes have at least 15% of the gene region with challenging sequences or variants. The catalog of repetitive challenging sequences comes from GIAB and the Global Alliance for Genomics and Health (see text). Challenging variants for HG002 are defined as complex variants (i.e., more than one variant within 10 bp) as well as putative SVs and putative duplications excluded from the HG002 v4.2.1 benchmark regions. C) Size distribution of INDELs in the small variant benchmark, which includes some larger INDELs in introns (light blue) and exons (dark blue). D) Size distribution of large insertions and deletions in the SV benchmark in introns (light blue) and exons (dark blue).

**Figure 3:**
The new benchmark covers the gene SMN1, which was previously excluded due to mapping challenges for all technologies in the highly identical segmental duplication. (A) Dotplot of GRCh38 against GRCh38 in the SMA region, showing a complex set of inverted repeats that make it challenging to assemble. (B) IGV view showing that only a small portion of SMN1 was included in v4.2.1, and that all technologies have challenges mapping in the region, but 10x Genomics and ultralong ONT reads support the variants called in the new CMRG benchmark. For the CMRG and v4.2.1 benchmarks, thick blue bars indicate regions included by each benchmark and orange and light blue lines indicate positions of homozygous and heterozygous benchmark variants, respectively. CMRG variants were called from the trio-based hifiasm assembly of paternal and maternal haplotypes (Hifiasm-pat and Hifiasm-mat, respectively). Coverage tracks are show for 60x PCR-free Illumina 2×150 bp reads (Illumina-60x), 10x Genomics linked reads (10X Genomics), 50x PacBio HiFi 15 kbp and 20 kbp reads (PB Hifi-50x), and 60x Oxford Nanopore ultralong reads (ONT-UL-60x).

**Figure 4:**
(A) The benchmark resolves the gene CBS, which has a highly homologous gene CBSL due to a false duplication in GRCh38 that is not in HG002 or GRCh37. The duplication in GRCh38 causes Illumina and PacBio HiFi reads from one haplotype to mismap to CBSL instead of CBS. The ultralong ONT reads, 10x Genomics linked reads, and assembled PacBio HiFi contigs map properly to this region for both haplotypes because they contain sufficient flanking sequence. When the falsely duplicated sequence is masked using our new version of GRCh38, variant calls from a standard Illumina-GATK pipeline (ILMN-GATK w/ Mask VCF) are completely concordant with the new benchmark. Pink shaded box indicates CMRG benchmark regions, only variants within the benchmark regions are included in the benchmark. (B) Comparison of variant accuracy for GRCh38 before and after masking false duplications on chromosome 21. The new benchmark demonstrates decreases in false negative and false positive errors for 3 callsets in the falsely duplicated genes CBS, CRYAA, and KCNE1 when mapping to the masked GRCh38.

**Figure 5:**
The new CMRG small variant benchmark includes more challenging variants and identifies more false negatives in a standard short-read callset (Illumina-bwamem-GATK) than the previous v4.2.1 benchmark in these challenging genes. While the false negative rate (circles) is similar in easier regions (purple “Not In All Difficult” points), the false negative rate is much higher overall (green “All CMRG Benchmark Regions” points). The fraction of variants excluded from the benchmark regions (triangles) is much higher for the v4.2.1 benchmark in all stratifications. This information is also presented in “summary stats NYGC” in Supplementary Data 4.

See this image and copyright information in PMC

Cited by

Semi-automated assembly of high-quality diploid human reference genomes.
Jarvis ED, Formenti G, Rhie A, Guarracino A, Yang C, Wood J, Tracey A, Thibaud-Nissen F, Vollger MR, Porubsky D, Cheng H, Asri M, Logsdon GA, Carnevali P, Chaisson MJP, Chin CS, Cody S, Collins J, Ebert P, Escalona M, Fedrigo O, Fulton RS, Fulton LL, Garg S, Gerton JL, Ghurye J, Granat A, Green RE, Harvey W, Hasenfeld P, Hastie A, Haukness M, Jaeger EB, Jain M, Kirsche M, Kolmogorov M, Korbel JO, Koren S, Korlach J, Lee J, Li D, Lindsay T, Lucas J, Luo F, Marschall T, Mitchell MW, McDaniel J, Nie F, Olsen HE, Olson ND, Pesout T, Potapova T, Puiu D, Regier A, Ruan J, Salzberg SL, Sanders AD, Schatz MC, Schmitt A, Schneider VA, Selvaraj S, Shafin K, Shumate A, Stitziel NO, Stober C, Torrance J, Wagner J, Wang J, Wenger A, Xiao C, Zimin AV, Zhang G, Wang T, Li H, Garrison E, Haussler D, Hall I, Zook JM, Eichler EE, Phillippy AM, Paten B, Howe K, Miga KH; Human Pangenome Reference Consortium. Jarvis ED, et al. Nature. 2022 Nov;611(7936):519-531. doi: 10.1038/s41586-022-05325-5. Epub 2022 Oct 19. Nature. 2022. PMID: 36261518 Free PMC article.
Resolving intra-repeat variation in medically relevant VNTRs from short-read sequencing data using the cardiovascular risk gene LPA as a model.
Di Maio S, Zöscher P, Weissensteiner H, Forer L, Schachtl-Riess JF, Amstler S, Streiter G, Pfurtscheller C, Paulweber B, Kronenberg F, Coassin S, Schönherr S. Di Maio S, et al. Genome Biol. 2024 Jun 26;25(1):167. doi: 10.1186/s13059-024-03316-5. Genome Biol. 2024. PMID: 38926899 Free PMC article.
Pangenomic genotyping with the marker array.
Mun T, Vaddadi NSK, Langmead B. Mun T, et al. Algorithms Mol Biol. 2023 May 5;18(1):2. doi: 10.1186/s13015-023-00225-3. Algorithms Mol Biol. 2023. PMID: 37147657 Free PMC article.
A multilocus approach for accurate variant calling in low-copy repeats using whole-genome sequencing.
Prodanov T, Bansal V. Prodanov T, et al. Bioinformatics. 2023 Jun 30;39(39 Suppl 1):i279-i287. doi: 10.1093/bioinformatics/btad268. Bioinformatics. 2023. PMID: 37387146 Free PMC article.
Pangenome graph construction from genome alignments with Minigraph-Cactus.
Hickey G, Monlong J, Ebler J, Novak AM, Eizenga JM, Gao Y; Human Pangenome Reference Consortium; Marschall T, Li H, Paten B. Hickey G, et al. Nat Biotechnol. 2024 Apr;42(4):663-673. doi: 10.1038/s41587-023-01793-w. Epub 2023 May 10. Nat Biotechnol. 2024. PMID: 37165083 Free PMC article.

See all "Cited by" articles

References

1. Wenger AM et al. Accurate circular consensus long-read sequencing improves variant detection and assembly of a human genome. Nat. Biotechnol 37, 1155–1162 (2019). - PMC - PubMed
1. Cheng H, Concepcion GT, Feng X, Zhang H & Li H Haplotype-resolved de novo assembly using phased assembly graphs with hifiasm. Nat. Methods 18, 170–175 (2021). - PMC - PubMed
1. Nurk S et al. HiCanu: accurate assembly of segmental duplications, satellites, and allelic variants from high-fidelity long reads. Genome Res. 30, 1291–1305 (2020). - PMC - PubMed
1. Shafin K et al. Nanopore sequencing and the Shasta toolkit enable efficient de novo assembly of eleven human genomes. Nat. Biotechnol 38, 1044–1053 (2020). - PMC - PubMed
1. Mahmoud M et al. Structural variant calling: the long and the short of it. Genome Biol. 20, 246 (2019). - PMC - PubMed

Publication types

Actions
Actions
Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources
Other Literature Sources
- JRNLclub
Research Materials
- Coriell Cell Repositories

[1] Wenger AM et al. Accurate circular consensus long-read sequencing improves variant detection and assembly of a human genome. Nat. Biotechnol 37, 1155–1162 (2019). - PMC - PubMed

[2] Wenger AM et al. Accurate circular consensus long-read sequencing improves variant detection and assembly of a human genome. Nat. Biotechnol 37, 1155–1162 (2019). - PMC - PubMed

[3] Cheng H, Concepcion GT, Feng X, Zhang H & Li H Haplotype-resolved de novo assembly using phased assembly graphs with hifiasm. Nat. Methods 18, 170–175 (2021). - PMC - PubMed

[4] Cheng H, Concepcion GT, Feng X, Zhang H & Li H Haplotype-resolved de novo assembly using phased assembly graphs with hifiasm. Nat. Methods 18, 170–175 (2021). - PMC - PubMed

[5] Nurk S et al. HiCanu: accurate assembly of segmental duplications, satellites, and allelic variants from high-fidelity long reads. Genome Res. 30, 1291–1305 (2020). - PMC - PubMed

[6] Nurk S et al. HiCanu: accurate assembly of segmental duplications, satellites, and allelic variants from high-fidelity long reads. Genome Res. 30, 1291–1305 (2020). - PMC - PubMed

[7] Shafin K et al. Nanopore sequencing and the Shasta toolkit enable efficient de novo assembly of eleven human genomes. Nat. Biotechnol 38, 1044–1053 (2020). - PMC - PubMed

[8] Shafin K et al. Nanopore sequencing and the Shasta toolkit enable efficient de novo assembly of eleven human genomes. Nat. Biotechnol 38, 1044–1053 (2020). - PMC - PubMed

[9] Mahmoud M et al. Structural variant calling: the long and the short of it. Genome Biol. 20, 246 (2019). - PMC - PubMed

[10] Mahmoud M et al. Structural variant calling: the long and the short of it. Genome Biol. 20, 246 (2019). - PMC - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Curated variation benchmarks for challenging medically relevant autosomal genes

Affiliations

Curated variation benchmarks for challenging medically relevant autosomal genes

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources

Research Materials

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Publication types

MeSH terms

Related information

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources

Research Materials