Benchmarking challenging small variants with linked and long reads

doi:10.1016/j.xgen.2022.100128

. 2022 May;2(5):100128.

doi: 10.1016/j.xgen.2022.100128.

Benchmarking challenging small variants with linked and long reads

Justin Wagner¹, Nathan D Olson¹, Lindsay Harris¹, Ziad Khan², Jesse Farek², Medhat Mahmoud², Ana Stankovic³, Vladimir Kovacevic³, Byunggil Yoo⁴, Neil Miller⁴, Jeffrey A Rosenfeld⁵, Bohan Ni⁶, Samantha Zarate⁶, Melanie Kirsche⁶, Sergey Aganezov⁶, Michael C Schatz⁶, Giuseppe Narzisi⁷, Marta Byrska-Bishop⁷, Wayne Clarke⁷, Uday S Evani⁷, Charles Markello⁸, Kishwar Shafin⁸, Xin Zhou⁹, Arend Sidow^{10

11}, Vikas Bansal¹², Peter Ebert¹³, Tobias Marschall¹³, Peter Lansdorp¹³, Vincent Hanlon¹⁴, Carl-Adam Mattsson¹⁴, Alvaro Martinez Barrio¹⁵, Ian T Fiddes¹⁵, Chunlin Xiao¹⁶, Arkarachai Fungtammasan¹⁷, Chen-Shan Chin¹⁷, Aaron M Wenger¹⁸, William J Rowell¹⁸, Fritz J Sedlazeck², Andrew Carroll¹⁹, Marc Salit^{20

21}, Justin M Zook^{1

21

22}

Affiliations

¹ Material Measurement Laboratory, National Institute of Standards and Technology, 100 Bureau Dr, MS8312, Gaithersburg, MD 20899, USA.
² Human Genome Sequencing Center, Baylor College of Medicine, One Baylor Plaza, Houston, TX 77030, USA.
³ Seven Bridges, Omladinskih brigada 90g, 11070 Belgrade, Republic of Serbia.
⁴ Children's Mercy Kansas City, Kansas City, MO, USA.
⁵ Rutgers Cancer Institute of New Jersey, New Brunswick, NJ, USA.
⁶ Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA.
⁷ New York Genome Center, 101 Avenue of the Americas, New York, NY, USA.
⁸ University of California at Santa Cruz Genomics Institute, 1156 High Street, Santa Cruz, CA, USA.
⁹ Department of Computer Science, Stanford University, Stanford, CA 94305, USA.
¹⁰ Department of Pathology, Stanford University, Stanford, CA 94305, USA.
¹¹ Department of Genetics, Stanford University, Stanford, CA 94305, USA.
¹² Department of Pediatrics, University of California, San Diego, La Jolla, CA 92093, USA.
¹³ Institute of Medical Biometry and Bioinformatics, Medical Faculty, Heinrich Heine University Düsseldorf, 40225 Düsseldorf, Germany.
¹⁴ Terry Fox Laboratory, BC Cancer Research Institute and Department of Medical Genetics, University of British Columbia, Vancouver, BC, Canada.
¹⁵ 10X Genomics, Pleasanton, CA 94588, USA.
¹⁶ National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, 8600 Rockville Pike, Bethesda, MD 20894, USA.
¹⁷ DNAnexus, Inc., Mountain View, CA 94040, USA.
¹⁸ Pacific Biosciences, Menlo Park, CA 94025, USA.
¹⁹ Google Inc., 1600 Amphitheatre Pkwy., Mountain View, CA 94040, USA.
²⁰ Joint Initiative for Metrology in Biology, SLAC National Laboratory, Stanford, CA, USA.
²¹ Senior author.
²² Lead contact.

PMID: 36452119
PMCID: PMC9706577
DOI: 10.1016/j.xgen.2022.100128

Benchmarking challenging small variants with linked and long reads

Justin Wagner et al. Cell Genom. 2022 May.

. 2022 May;2(5):100128.

doi: 10.1016/j.xgen.2022.100128.

Authors

Affiliations

¹ Material Measurement Laboratory, National Institute of Standards and Technology, 100 Bureau Dr, MS8312, Gaithersburg, MD 20899, USA.
² Human Genome Sequencing Center, Baylor College of Medicine, One Baylor Plaza, Houston, TX 77030, USA.
³ Seven Bridges, Omladinskih brigada 90g, 11070 Belgrade, Republic of Serbia.
⁴ Children's Mercy Kansas City, Kansas City, MO, USA.
⁵ Rutgers Cancer Institute of New Jersey, New Brunswick, NJ, USA.
⁶ Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA.
⁷ New York Genome Center, 101 Avenue of the Americas, New York, NY, USA.
⁸ University of California at Santa Cruz Genomics Institute, 1156 High Street, Santa Cruz, CA, USA.
⁹ Department of Computer Science, Stanford University, Stanford, CA 94305, USA.
¹⁰ Department of Pathology, Stanford University, Stanford, CA 94305, USA.
¹¹ Department of Genetics, Stanford University, Stanford, CA 94305, USA.
¹² Department of Pediatrics, University of California, San Diego, La Jolla, CA 92093, USA.
¹³ Institute of Medical Biometry and Bioinformatics, Medical Faculty, Heinrich Heine University Düsseldorf, 40225 Düsseldorf, Germany.
¹⁴ Terry Fox Laboratory, BC Cancer Research Institute and Department of Medical Genetics, University of British Columbia, Vancouver, BC, Canada.
¹⁵ 10X Genomics, Pleasanton, CA 94588, USA.
¹⁶ National Center for Biotechnology Information, National Library of Medicine, National Institutes of Health, 8600 Rockville Pike, Bethesda, MD 20894, USA.
¹⁷ DNAnexus, Inc., Mountain View, CA 94040, USA.
¹⁸ Pacific Biosciences, Menlo Park, CA 94025, USA.
¹⁹ Google Inc., 1600 Amphitheatre Pkwy., Mountain View, CA 94040, USA.
²⁰ Joint Initiative for Metrology in Biology, SLAC National Laboratory, Stanford, CA, USA.
²¹ Senior author.
²² Lead contact.

PMID: 36452119
PMCID: PMC9706577
DOI: 10.1016/j.xgen.2022.100128

Abstract

Genome in a Bottle benchmarks are widely used to help validate clinical sequencing pipelines and develop variant calling and sequencing methods. Here we use accurate linked and long reads to expand benchmarks in 7 samples to include difficult-to-map regions and segmental duplications that are challenging for short reads. These benchmarks add more than 300,000 SNVs and 50,000 insertions or deletions (indels) and include 16% more exonic variants, many in challenging, clinically relevant genes not covered previously, such as PMS2. For HG002, we include 92% of the autosomal GRCh38 assembly while excluding regions problematic for benchmarking small variants, such as copy number variants, that should not have been in the previous version, which included 85% of GRCh38. It identifies eight times more false negatives in a short read variant call set relative to our previous benchmark. We demonstrate that this benchmark reliably identifies false positives and false negatives across technologies, enabling ongoing methods development.

PubMed Disclaimer

Conflict of interest statement

A.M.W. and W.J.R. are employees and shareholders of Pacific Biosciences. A.M.B. and I.T.F. were employees and shareholders of 10X Genomics. F.J.S. has received sponsored travel from Oxford Nanopore and Pacific Biosciences and a 2018 sequencing grant from Pacific Biosciences. A.S. and V.K. are employees of Seven Bridges. A.C. is an employee of Google Inc. and a former employee of DNAnexus. A.F. and C.-S.C. are employees of DNAnexus.

Figures

**Figure 1**
The new benchmark set includes more of the reference genome and more variants (A) Percentage of the genomic region that is included by HG002 v.3.3.2 and v.4.2.1 of all non-gap, autosomal GRCh38 bases; the MHC; low-mappability regions and segmental duplications; and 159 difficult-to-map, medically relevant genes described previously. (B) The number of unique SNVs by genomic context. Circle size indicates the total number of SNVs in the union of v.3.3.2 and v.4.2.1. Circles above the diagonal indicate a net gain of SNVs in the newer benchmark, and circles below the diagonal indicate a net loss of SNVs in the newer benchmark.

**Figure 2**
v.4.2.1 includes many more difficult-to-map, medically relevant genes (A) Cumulative distribution of the percentage of each gene included in HG002 v.4.2.1 benchmark regions for 159 autosomal difficult-to-map, medically relevant genes. Dashed lines indicate that the number of genes included more than 90% increased from 19 in v.3.3.2 to 110 in v.4.2.1. (B) Pairwise comparison of difficult-to-map, medically relevant gene inclusion in the benchmark set. Genes falling on the dashed line are similarly included by both benchmark sets, whereas genes above (red fill) or below (blue fill) the dashed line are included more by the v.4.2.1 or v.3.3.2 benchmark set, respectively. The genes included more by v.4.2.1 tend to be in segmental duplications, and the smaller number of genes included more by v.3.3.2 are mostly genes duplicated in HG002 relative to GRCh38 and should be excluded.

**Figure 3**
Genes in the KIR locus are excluded in v.4.2.1 because of duplication in HG002 Medically relevant genes in the KIR locus, such as *KIR2DL1*, were partially included in v.3.3.2 with many erroneous variants but are correctly excluded by v.4.2.1 because of a likely duplication and other structural variation. Thick blue bars indicate regions included by each benchmark, and orange and light blue lines indicate positions of homozygous and heterozygous benchmark variants, respectively. A duplication of part of this region, which is common in the population, is supported by higher-than-normal coverage and high variant density across all technologies as well as alignment of multiple contigs from the maternal trio-based HG002 Hifiasm assembly (Hifiasm-maternal). The region is very challenging to characterize and assemble accurately because of high variability and copy number polymorphisms in the population as well as segmental duplications (shaded regions).

**Figure 4**
The difficult-to-map, medically relevant gene *PMS2* is better included in v.4.2.1 The medically relevant gene *PMS2* is 85.6% included in the v.4.2.1 benchmark regions, whereas it is 25.9% included in v.3.3.2 because segmental duplications (shaded regions) were largely excluded in previous benchmark versions. Thick blue bars indicate regions included by each benchmark, and orange and light blue lines indicate positions of homozygous and heterozygous benchmark variants, respectively. This region is challenging for assembly-based approaches, and an extra contig from the maternal trio-based HG002 Hifiasm assembly (Hifiasm-maternal) aligned to the left half of the gene because of misalignment or misassembly.

**Figure 5**
Summary of manual curations from the evaluation of the v.4.1 benchmark, demonstrating that it reliably identifies FPs and FNs in 13 call sets from different technologies and variant callers (A) For each call set, we curated 20 FPs and 20 FNs, and this shows the proportion of curated FP and FN variants where the benchmark set was correct, and the query call set was incorrect. The dashed black line indicates the desired majority threshold, 50%. Half of the curated variants were from GRCh37, and half were from GRCh38. (B) Breakdown of the total number of variants by category determined during manual curation, where the benchmark curation bar indicates whether the benchmark variant and genotype were determined to be correct and the query curations color indicates whether the query variant and genotype were determined to be correct. (A and B) Excluded in (B) are variants from (A), where the benchmark was deemed correct and the query incorrect and shows that most of these sites were difficult to curate. (C) Benchmark-unsure variants by call set. ONT, Oxford Nanopore; PB, PacBio HiFi; Ill, Illumina PCR-free; 10X, 10X Genomics.

See this image and copyright information in PMC

Cited by

xAtlas: scalable small variant calling across heterogeneous next-generation sequencing experiments.
Farek J, Hughes D, Salerno W, Zhu Y, Pisupati A, Mansfield A, Krasheninina O, English AC, Metcalf G, Boerwinkle E, Muzny DM, Gibbs R, Khan Z, Sedlazeck FJ. Farek J, et al. Gigascience. 2022 Dec 28;12:giac125. doi: 10.1093/gigascience/giac125. Epub 2023 Jan 16. Gigascience. 2022. PMID: 36644891 Free PMC article.
Towards population-scale long-read sequencing.
De Coster W, Weissensteiner MH, Sedlazeck FJ. De Coster W, et al. Nat Rev Genet. 2021 Sep;22(9):572-587. doi: 10.1038/s41576-021-00367-3. Epub 2021 May 28. Nat Rev Genet. 2021. PMID: 34050336 Free PMC article. Review.
A genome sequencing system for universal newborn screening, diagnosis, and precision medicine for severe genetic diseases.
Kingsmore SF, Smith LD, Kunard CM, Bainbridge M, Batalov S, Benson W, Blincow E, Caylor S, Chambers C, Del Angel G, Dimmock DP, Ding Y, Ellsworth K, Feigenbaum A, Frise E, Green RC, Guidugli L, Hall KP, Hansen C, Hobbs CA, Kahn SD, Kiel M, Van Der Kraan L, Krilow C, Kwon YH, Madhavrao L, Le J, Lefebvre S, Mardach R, Mowrey WR, Oh D, Owen MJ, Powley G, Scharer G, Shelnutt S, Tokita M, Mehtalia SS, Oriol A, Papadopoulos S, Perry J, Rosales E, Sanford E, Schwartz S, Tran D, Reese MG, Wright M, Veeraraghavan N, Wigby K, Willis MJ, Wolen AR, Defay T. Kingsmore SF, et al. Am J Hum Genet. 2022 Sep 1;109(9):1605-1619. doi: 10.1016/j.ajhg.2022.08.003. Epub 2022 Aug 24. Am J Hum Genet. 2022. PMID: 36007526 Free PMC article. Review.
WarpSTR: determining tandem repeat lengths using raw nanopore signals.
Sitarčík J, Vinař T, Brejová B, Krampl W, Budiš J, Radvánszky J, Lucká M. Sitarčík J, et al. Bioinformatics. 2023 Jun 1;39(6):btad388. doi: 10.1093/bioinformatics/btad388. Bioinformatics. 2023. PMID: 37326967 Free PMC article.
Multiscale analysis of pangenomes enables improved representation of genomic diversity for repetitive and clinically relevant genes.
Chin CS, Behera S, Khalak A, Sedlazeck FJ, Sudmant PH, Wagner J, Zook JM. Chin CS, et al. Nat Methods. 2023 Aug;20(8):1213-1221. doi: 10.1038/s41592-023-01914-y. Epub 2023 Jun 26. Nat Methods. 2023. PMID: 37365340 Free PMC article.

See all "Cited by" articles

References

1. Zook J.M., Catoe D., McDaniel J., Vang L., Spies N., Sidow A., Weng Z., Liu Y., Mason C.E., Alexander N., et al. Extensive sequencing of seven human genomes to characterize benchmark reference materials. Sci. Data. 2016;3:160025. - PMC - PubMed
1. Zook J.M., Chapman B., Wang J., Mittelman D., Hofmann O., Hide W., Salit M. Integrating human sequence data sets provides a resource of benchmark SNP and indel genotype calls. Nat. Biotechnol. 2014;32:246–251. - PubMed
1. Zook J.M., Hansen N.F., Olson N.D., Chapman L., Mullikin J.C., Xiao C., Sherry S., Koren S., Phillippy A.M., Boutros P.C., et al. A robust benchmark for detection of germline large deletions and insertions. Nat. Biotechnol. 2020;38:1347–1355. - PMC - PubMed
1. Zook J.M., McDaniel J., Olson N.D., Wagner J., Parikh H., Heaton H., Irvine S.A., Trigg L., Truty R., McLean C.Y., et al. An open resource for accurately benchmarking small variant and reference calls. Nat. Biotechnol. 2019;37:561–566. - PMC - PubMed
1. Krusche P., Trigg L., Boutros P.C., Mason C.E., De La Vega F.M., Moore B.L., Gonzalez-Porta M., Eberle M.A., Tezak Z., Lababidi S., et al. Best practices for benchmarking germline small-variant calls in human genomes. Nat. Biotechnol. 2019;37:555–560. - PMC - PubMed

Grants and funding

LinkOut - more resources

Full Text Sources
Miscellaneous
- NCI CPTAC Assay Portal

[1] Zook J.M., Catoe D., McDaniel J., Vang L., Spies N., Sidow A., Weng Z., Liu Y., Mason C.E., Alexander N., et al. Extensive sequencing of seven human genomes to characterize benchmark reference materials. Sci. Data. 2016;3:160025. - PMC - PubMed

[2] Zook J.M., Catoe D., McDaniel J., Vang L., Spies N., Sidow A., Weng Z., Liu Y., Mason C.E., Alexander N., et al. Extensive sequencing of seven human genomes to characterize benchmark reference materials. Sci. Data. 2016;3:160025. - PMC - PubMed

[3] Zook J.M., Chapman B., Wang J., Mittelman D., Hofmann O., Hide W., Salit M. Integrating human sequence data sets provides a resource of benchmark SNP and indel genotype calls. Nat. Biotechnol. 2014;32:246–251. - PubMed

[4] Zook J.M., Chapman B., Wang J., Mittelman D., Hofmann O., Hide W., Salit M. Integrating human sequence data sets provides a resource of benchmark SNP and indel genotype calls. Nat. Biotechnol. 2014;32:246–251. - PubMed

[5] Zook J.M., Hansen N.F., Olson N.D., Chapman L., Mullikin J.C., Xiao C., Sherry S., Koren S., Phillippy A.M., Boutros P.C., et al. A robust benchmark for detection of germline large deletions and insertions. Nat. Biotechnol. 2020;38:1347–1355. - PMC - PubMed

[6] Zook J.M., Hansen N.F., Olson N.D., Chapman L., Mullikin J.C., Xiao C., Sherry S., Koren S., Phillippy A.M., Boutros P.C., et al. A robust benchmark for detection of germline large deletions and insertions. Nat. Biotechnol. 2020;38:1347–1355. - PMC - PubMed

[7] Zook J.M., McDaniel J., Olson N.D., Wagner J., Parikh H., Heaton H., Irvine S.A., Trigg L., Truty R., McLean C.Y., et al. An open resource for accurately benchmarking small variant and reference calls. Nat. Biotechnol. 2019;37:561–566. - PMC - PubMed

[8] Zook J.M., McDaniel J., Olson N.D., Wagner J., Parikh H., Heaton H., Irvine S.A., Trigg L., Truty R., McLean C.Y., et al. An open resource for accurately benchmarking small variant and reference calls. Nat. Biotechnol. 2019;37:561–566. - PMC - PubMed

[9] Krusche P., Trigg L., Boutros P.C., Mason C.E., De La Vega F.M., Moore B.L., Gonzalez-Porta M., Eberle M.A., Tezak Z., Lababidi S., et al. Best practices for benchmarking germline small-variant calls in human genomes. Nat. Biotechnol. 2019;37:555–560. - PMC - PubMed

[10] Krusche P., Trigg L., Boutros P.C., Mason C.E., De La Vega F.M., Moore B.L., Gonzalez-Porta M., Eberle M.A., Tezak Z., Lababidi S., et al. Best practices for benchmarking germline small-variant calls in human genomes. Nat. Biotechnol. 2019;37:555–560. - PMC - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Benchmarking challenging small variants with linked and long reads

Affiliations

Benchmarking challenging small variants with linked and long reads

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

Grants and funding

LinkOut - more resources

Full Text Sources

Miscellaneous