Skip to main page content
U.S. flag

An official website of the United States government

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2011;79 Suppl 10(S10):107-18.
doi: 10.1002/prot.23161. Epub 2011 Sep 16.

Evaluation of disorder predictions in CASP9

Bohdan Monastyrskyy  1 , Krzysztof Fidelis, John Moult, Anna Tramontano, Andriy Kryshtafovych
Affiliations

Evaluation of disorder predictions in CASP9

Bohdan Monastyrskyy et al. Proteins. 2011.

Abstract

Lack of stable three-dimensional structure, or intrinsic disorder, is a common phenomenon in proteins. Naturally, unstructured regions are proven to be essential for carrying function by many proteins, and therefore identification of such regions is an important issue. CASP has been assessing the state of the art in predicting disorder regions from amino acid sequence since 2002. Here, we present the results of the evaluation of the disorder predictions submitted to CASP9. The assessment is based on the evaluation measures and procedures used in previous CASPs. The balanced accuracy and the Matthews correlation coefficient were chosen as basic measures for evaluating the correctness of binary classifications. The area under the receiver operating characteristic curve was the measure of choice for evaluating probability-based predictions of disorder. The CASP9 methods are shown to perform slightly better than the CASP7 methods but not better than the methods in CASP8. It was also shown that capability of most CASP9 methods to predict disorder decreases with increasing minimum disorder segment length.

Keywords: CASP; assessment of disorder prediction; intrinsically disordered proteins; rediction of disordered regions; unstructured proteins.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Length distribution of disordered regions in CASP9 target proteins.
Figure 2
Figure 2
Performance of DR groups according to three evaluation scores: AUC (black bars), Acc (grey bars) and MCC (light grey bars). The groups are sorted according to decreasing AUC score. The error bars on the plot indicate boundaries of the 95% confidence intervals for each measure.
Figure 3
Figure 3
ROC curves of disordered region predictions for all CASP9 groups. Legends are shown for the best 12 groups according to the AUC. There are four non-regular ROCs corresponding to poorly performing groups, two of which misinterpreted DR format (G193 used only a single value for ordered residues and G114 did not use continuous scale but rather 5 different numbers).
Figure 4
Figure 4
Comparison of prediction performance across four different minimum disorder segment length thresholds. Different panels show scores for different evaluation measures (Acc, MCC and AUC). Each group is marked with a different color; groups in the legend are sorted according to the AUC score (across and then down); the artificial average group (‘AVG’, black thicker line) is added to the graph as a point of reference.
Figure 5
Figure 5
Comparison of the performance of the best 12 groups in the latest 3 CASPs. Groups in each CASP are sorted according to the MCC score. CASP8 results are evaluated for both the full set of targets and the set without target T0500, a long, completely unfolded protein considerably influencing the scores. The reduced target set is marked with an asterisk in the legend. The scores in CASP9 are higher than in CASP7 but lower than in CASP8. The CASP8–CASP9 drop in scores may be attributed to the greater difficulty of targets in CASP9.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Wu H. Studies on denaturation of proteins. XIII. A theory of denaturation. Chin J Physiol. 1931;1:219–234. - PubMed
    1. Anfinsen CB. Principles that govern the folding of protein chains. Science. 1973;181(96):223–230. - PubMed
    1. Wright PE, Dyson HJ. Intrinsically unstructured proteins: re-assessing the protein structure-function paradigm. J Mol Biol. 1999;293(2):321–331. - PubMed
    1. Dunker AK, Obradovic Z, Romero P, Garner EC, Brown CJ. Intrinsic protein disorder in complete genomes. Genome Inform Ser Workshop Genome Inform. 2000;11:161–171. - PubMed
    1. Tompa P. Intrinsically unstructured proteins. Trends Biochem Sci. 2002;27(10):527–533. - PubMed

Publication types

LinkOut - more resources