Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 May 11;2(5):100129.
doi: 10.1016/j.xgen.2022.100129. Epub 2022 Apr 27.

PrecisionFDA Truth Challenge V2: Calling variants from short and long reads in difficult-to-map regions

Nathan D Olson  1   2 , Justin Wagner  1 , Jennifer McDaniel  1 , Sarah H Stephens  3 , Samuel T Westreich  4 , Anish G Prasanna  3 , Elaine Johanson  5 , Emily Boja  5 , Ezekiel J Maier  3 , Omar Serang  4 , David Jáspez  6 , José M Lorenzo-Salazar  6 , Adrián Muñoz-Barrera  6 , Luis A Rubio-Rodríguez  6 , Carlos Flores  6   7   8   9 , Konstantinos Kyriakidis  10   11 , Andigoni Malousi  11   12 , Kishwar Shafin  13 , Trevor Pesout  13 , Miten Jain  13 , Benedict Paten  13 , Pi-Chuan Chang  14 , Alexey Kolesnikov  14 , Maria Nattestad  14 , Gunjan Baid  14 , Sidharth Goel  14 , Howard Yang  14 , Andrew Carroll  14 , Robert Eveleigh  15 , Mathieu Bourgey  15 , Guillaume Bourque  15 , Gen Li  16 , ChouXian Ma  16 , LinQi Tang  16 , YuanPing Du  16 , ShaoWei Zhang  16 , Jordi Morata  17   18 , Raúl Tonda  17   18 , Genís Parra  17   18 , Jean-Rémi Trotta  17   18 , Christian Brueffer  19 , Sinem Demirkaya-Budak  20 , Duygu Kabakci-Zorlu  20 , Deniz Turgut  20 , Özem Kalay  20 , Gungor Budak  20 , Kübra Narcı  20 , Elif Arslan  20 , Richard Brown  20 , Ivan J Johnson  20 , Alexey Dolgoborodov  20 , Vladimir Semenyuk  20 , Amit Jain  20 , H Serhat Tetikol  20 , Varun Jain  21 , Mike Ruehle  21 , Bryan Lajoie  21 , Cooper Roddey  21 , Severine Catreux  21 , Rami Mehio  21 , Mian Umair Ahsan  22 , Qian Liu  22 , Kai Wang  22   23 , Sayed Mohammad Ebrahim Sahraeian  24 , Li Tai Fang  24 , Marghoob Mohiyuddin  24 , Calvin Hung  25 , Chirag Jain  26 , Hanying Feng  27 , Zhipan Li  27 , Luoqi Chen  27 , Fritz J Sedlazeck  28 , Justin M Zook  1
Affiliations

PrecisionFDA Truth Challenge V2: Calling variants from short and long reads in difficult-to-map regions

Nathan D Olson et al. Cell Genom. .

Abstract

The precisionFDA Truth Challenge V2 aimed to assess the state of the art of variant calling in challenging genomic regions. Starting with FASTQs, 20 challenge participants applied their variant-calling pipelines and submitted 64 variant call sets for one or more sequencing technologies (Illumina, PacBio HiFi, and Oxford Nanopore Technologies). Submissions were evaluated following best practices for benchmarking small variants with updated Genome in a Bottle benchmark sets and genome stratifications. Challenge submissions included numerous innovative methods, with graph-based and machine learning methods scoring best for short-read and long-read datasets, respectively. With machine learning approaches, combining multiple sequencing technologies performed particularly well. Recent developments in sequencing and variant calling have enabled benchmarking variants in challenging genomic regions, paving the way for the identification of previously unknown clinically relevant variants.

PubMed Disclaimer

Conflict of interest statement

DECLARATION OF INTERESTS C.B. is an employee and shareholder of SAGA Diagnostics AB. A.C., P.-C.C., A.K., M.N., G.B., S.G., and H.Y. are employees of Google, and A.C. is a shareholder. S.D.-B., D.K.-Z., D.T., Ö.K., G.B., K.N., E.A., R.B., I.J.J., A.D., V.S., A.J., and H.S.T. are employees of Seven Bridges Genomics. O.S. and S. T.W. are employees of DNAnexus. G.L., C.M., L.T.F., Y.D., and S.Z. are employees of Genetalks. V.J., M.R., B.L., C.R., S.C., and R.M. are employees of Illumina. S.M.E.S. and M.M. are employees of Roche. C.H. is an employee of Wasai Technology. H.F., Z.L., and L.C. are employees of Sentieon.

Figures

None
Graphical abstract
Figure 1
Figure 1
Truth challenge V2 structure Participants were provided sequencing reads (FASTQ files) from Illumina, PacBio HiFi, and ONT for the GIAB Ashkenazi trio (HG002, HG003, and HG004). Participants uploaded VCF files for each individual before the end of the challenge, and then the new benchmarks for HG003 and HG004 were made public.
Figure 2
Figure 2
Challenge submission breakdown and performance overview (A) Challenge submission breakdown by technology and type of variant caller used. Deep-learning methods use either a convolutional neural network or a recurrent neural network architecture for learning the variant-calling task, while non-deep-learning methods use techniques that broadly arise from statistical techniques (e.g., Bayesian and Gaussian mixture models) or other ML techniques (e.g., random forest) to differentiate variant and non-variant loci based on expert-designed features of the sequencing data. (B and C) Overall performance (B) and submission rank (C) varied by technology and stratification (log scale). Generally, submissions that used multiple technologies (MULTI) outperformed single-technology submissions for all three genomic context categories. (B) A histogram of F1 percentage (higher is better) for the three genomic stratifications evaluated. Submission counts across technologies are indicated by light gray bars and individual technologies by colored bars. (C) Individual submission performance. Data points represent submission performance for the three stratifications (difficult-to-map regions, all benchmark regions, MHC), and lines connect submissions. Category top performers are indicated by diamonds with Ws and labeled with team names. F1 is plotted on a phred scale with axes labels and ticks indicating F1 percentage values.
Figure 3
Figure 3
Submission performance comparison for F1 metric between MHC, all benchmark regions, and difficult-to-map regions F1 is plotted on a phred scale with axis labels and ticks indicating F1 percentage values. Points above the diagonal black line perform better in MHC relative to all benchmark regions or the difficult-to-map regions. Submissions with the largest difference in performance between MHC and difficult-to-map or all benchmark regions for each subplot are labeled. Seven Bridges is a graph-based short-read variant caller. ONT ensemble is an ensemble of ONT variant callers; NanoCaller, Clair, and Medaka. PEPPER-DV is the ONT PEPPER-DeepVariant haplotype-aware ML variant-calling pipeline.
Figure 4
Figure 4
Performance comparisons by sample, benchmark version, and challenges Ratio of error rates using semi-blinded parents’ benchmark versus public son’s (HG002) benchmark. (A) Submissions ranked by error-rate ratio. (B) Comparison of error-rate ratio with the overall performance for the parents (F1 in all benchmarking regions, as defined in Equation 1). Error rate defined as 1 − F1. F1 is plotted on a phred scale with axis labels and ticks indicating F1 percentage values. (C and D) Comparison of benchmarking performance for (C) different benchmark sets and (D) challenges. (C) The 2016 (V1) Truth Challenge top performers F1 performance metric for SNVs and INDELs benchmarked against the V3.2 benchmark set (used to evaluate the first challenge) and V4.2 benchmark set (used to evaluate the second challenge). Performance metrics for the same variant calls decrease substantially versus the V4.2 benchmark set because it includes more challenging regions. (D) Performance of V1 challenge top performers (using 50X Illumina sequencing) compared with V2 submissions (using only 35X Illumina sequencing) for the harmonic mean of the parents’ F1 scores for combined SNVs and INDELs and the V4.2 benchmark set used to evaluate the second truth challenge. The black horizontal lines represent the performance for the overall top performer, regardless of technology used, for each stratification. For the first challenge, variant call sets for the blinded HG002 against GRCh37 were used to evaluate performance, and, for the second challenge, variant calls for the semi-blinded HG003 and HG004 against GRCh38 were used to evaluate performance. F1 is plotted on a phred scale with axes labels and ticks indicating F1 percentage values.
Figure 5
Figure 5
Comparison of ONT PEPPER-DeepVariant variant call set performance with Illumina DeepVariant by genomic context F1 is plotted on a phred scale with axis labels and ticks indicating F1 percentage values. Points above and below the diagonal line indicate stratifications where ONT PEPPER-DeepVariant submission performance metric was higher than the Illumina DeepVariant submission. The points are colored by stratification category.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Ewing A.D., Houlahan K.E., Hu Y., Ellrott K., Caloian C., Yamaguchi T.N., Bare J.C., P’ng C., Waggott D., Sabelnykova V.Y., et al. Combining tumor genome simulation with crowdsourcing to benchmark somatic single-nucleotide-variant detection. Nat. Methods. 2015;12:623–630. - PMC - PubMed
    1. Lee A.Y., Ewing A.D., Ellrott K., Hu Y., Houlahan K.E., Bare J.C., Espiritu S.M.G., Huang V., Dang K., Chong Z., et al. Combining accurate tumor genome simulation with crowdsourcing to benchmark somatic structural variant detection. Genome Biol. 2018;19:188. - PMC - PubMed
    1. Salcedo A., Tarabichi M., Espiritu S.M.G., Deshwar A.G., David M., Wilson N.M., Dentro S., Wintersinger J.A., Liu L.Y., Ko M., et al. A community effort to create standards for evaluating tumor subclonal reconstruction. Nat. Biotechnol. 2020;38:97–107. - PMC - PubMed
    1. Andreoletti G., Pal L.R., Moult J., Brenner S.E. Reports from the fifth edition of CAGI: the critical assessment of genome interpretation. Hum. Mutat. 2019;40:1197–1201. - PMC - PubMed
    1. Hoskins R.A., Repo S., Barsky D., Andreoletti G., Moult J., Brenner S.E. Reports from CAGI: the critical assessment of genome interpretation. Hum. Mutat. 2017;38:1039–1041. - PMC - PubMed