Poor correlation between T-cell activation assays and HLA-DR binding prediction algorithms in an immunogenic fragment of Pseudomonas exotoxin A

2.1 Normalization of experimental data

Previous analysis of the experimental data (Mazor et al., 2014) showed that many peptides do not cause T-cell responses among many donors. The large number of such unresponsive peptide-donor pairs provides a firm background level, which should help in detecting and accurately quantifying real responses. To take advantage of the large number of non-responsive data, we used the Z-score to normalize the ELISpot counts in this study (see Materials and methods).

The peptide that had a Z-score above a cutoff value for a donor was considered an epitope to the donor. We chose 6.0 as the cutoff value for this purpose because the epitopes that resulted from this choice agreed most closely with those published (Mazor et al., 2014). Figure 1A shows the normalized donor responses and Figure 1B shows the binarized data using the Z-score cutoff value of 6.0. The horizontal bars above the chart indicate the peptides that are epitopes to four or more donors.

2.2. HLA allele coverage in the donor cohort

Since the prediction programs use specific MHC allele, we typed the donor DRB molecules (DRB1, 3, 4 and 5) with high resolution (see results in Supporting Information Table 1). We concentrated on DR because prediction programs use very few DP or DQ alleles. There are four different MHC DRB loci in a human, one pair from each parent, each of which is highly polymorphic. Not counting those that could not be typed for various reasons, there were 36 different DRB alleles in our cohort of 50 donors.

The DRA alleles were not considered because the DRA gene is conserved and allelic variations do not affect the peptide-binding site (Trowsdale et al., 1985; Marshall et al., 1994).

2.3. Evaluation of prediction for individual donors

Each prediction method works for only a given set of HLA alleles. The immune epitope database (IEDB) “recommended” method covers 33 of the 36 donor alleles and IEDB “consensus” method covers 21. All other methods have much lower coverage. We, therefore, used only the IEDB “recommended” and IEDB “consensus” methods for comparison with the experimental data.

The prediction score for a peptide-allele pair is given in terms of the percentile rank of the predicted binding strength of the given peptide among a large number of randomly selected 15mer peptides; the lower the rank the better the binding. For each of the 111 PE38 peptides, each donor can have up to four different covered alleles and the corresponding number of different predictions. To assign one score per each donor for each peptide, we recorded either the minimum or the average of the percentile ranks for the donor. Figure 2A shows the heat map of the minimum percentile rank of each peptide for each donor predicted by the IEDB “recommended” method. Several regions such as peptides 13-15, 67-68 and 75-80 (peptide ID is indicated on the X-axis) are predicted to be very immunogenic, against which almost every donor has a strong predicted response (black in the heat map). On the other hand, the same method predicted that no donor would respond to peptides 22-24 and 35-40. This is consistent with experimental observation.

Figure 2B shows the predicted donor responses by the IEDB “recommended” method, if one considers only those for which the percentile rank is better than 2.84% as responses. Percentile rank cutoff value of 2.84% gives the False Positive Rate (FPR) of 10% when compared with binarized experimental Z-scores. Figure 2C shows the false positives in white and false negatives in black when the predictions using this particular cutoff value is compared with the experimental T-cell responses given in Figure 1B. The FPR (the white area in Fig. 2C as a fraction of the white area in Fig. 1B) using this cutoff value is 10%.

Figure 2D shows the receiver operating characteristic (ROC) curves, which are the plots of True Positive Rate (TPR) versus FPR as the percentile rank cutoff value is varied from low to high, for both the minimum and the average percentile ranks predicted by the two IEDB methods. Even though the IEDB “recommended” method covers more MHC alleles in our donor cohort than the IEDB “consensus” method, the two methods perform similarly. All have the AUC between 0.67 and 0.69. Table 1 gives the TPRs and the corresponding percentile rank cutoff values that yield a few samples of FPRs. It can be seen from Table 1 and the ROC curves that the TPR is below 50%, i.e. more than half of the epitopes are missed by the predictions, unless the percentile rank cutoff value is set so large that more than 10% of the many non-epitopes are predicted as epitopes. We note in passing that, for a given FPR, using minimum and average percentile ranks give similar TPRs, but the percentile rank cutoff values are much lower when the minimum is used than when the average is used.

2.4. Evaluation of prediction for the population using the 33 HLA-DRB alleles in our donor cohort

Generally, a donor was considered to be a responder to a peptide if the donor has the corresponding ELISpot Z-score higher than 6 and as a predicted responder if the donor has an HLA-DRB allele that is predicted by the IEDB “recommended” method to bind the peptide better than a certain threshold value, as measured by the binding percentile rank. The numbers of predicted responding donors for each peptide at the percentile rank threshold of 1%, 6% and 20% are shown in Figure 3 A, B and C. As expected, the more stringent the threshold, the fewer predicted number of responding donors. Peptides 13-15, and 75-79 were ranked as the top two epitope regions in our previous studies. Interestingly, they are also predicted to be epitopes to many donors at the very stringent 1% binding threshold (Fig. 3A). On the other hand, many experimentally observed frequent epitopes are not predicted at this binding threshold. More peptides are predicted as the percentile rank cutoff value is raised (Fig. 3B) but some frequent peptides such as peptides 4, 8, 9, 10, and 93 remain undetected (predicted to produce no or very few responses) even when the percentile rank cutoff value is raised to 20% (Fig. 3C and Table 2).

To quantitatively evaluate the population-wide prediction of the immunogenicity of a peptide, we concentrate on those peptides that produce a response in a large number of people. To this end, we define “frequent epitope” as the peptide that produces a response in at least a certain number of donors. After testing values 3, 4, and 5 using the experimental Z-score, this donor frequency cutoff value was chosen as 4, which gave the results closest to our published data (Mazor et al., 2014). This cutoff value was used both for the experimental Z-score data and also for the predicted percentile rank data. Figure 4A shows the number of peptides that are correctly or incorrectly predicted as frequent epitopes at binding threshold of 1% to 50%. Among the total 111 peptides, 27 produced responses from four or more donors experimentally and are colored in red (FN) or green (TP). The remaining 84 experimental non-frequent epitope peptides are colored orange (FP) or white (TN). At 2% percentile rank cut off, there are 26 peptides that are predicted to be frequent epitope (TP and FP combined), of which only 12, less than half, are correctly predicted, shown in green. At 6%, users would have to test 56 predicted frequent epitope peptides, but only 1/3, or 19 of them, are truly frequent epitopes. In order not to miss any frequent epitope, the binding percentile rank threshold would need to be at 35%, at which threshold, all 111 peptides are predicted to be frequent epitopes.

The TPR (TP/(FN+TP), FNR (FN/(FN+TP), FPR (FP/(FP+TN), Precision (TP/(TP+FP), and Accuracy (TP+TN)/(FN+TP+FP+TN) at thresholds of 1% to 20% percentile rank are plotted in Figure 4B. Although the TPR increases as the binding threshold is loosened, the Accuracy decreases because the number of false positives increases more rapidly. The Precision, true among predicted epitopes, drops below 50% when the binding cutoff is 2% or larger. This means that if we wanted to use the prediction algorithm to narrow down the number of peptides to test in vitro, more than 50% of the peptides would be FPs, even at a stringent threshold of 2%.

2.5. Evaluation of prediction for the population using HLA-DRB alleles in the worldwide population

To evaluate epitope prediction in the case where no cohort is selected, we predicted percentile rank from IEDB “recommended” method for the 15 reference HLA-DRB alleles that represent the world’s population (Greenbaum et al., 2011). Figure 5A shows the predicted percentile rank from IEDB “recommended” method for the 15 reference HLA-DRB alleles as a heat map. Two alleles, DRB1*0901 and DRB4*0101, are not in our cohort although it does include DRB4*0101.

To compare these predictions with the experimental data of our cohorts, we computed the correlation coefficient between the fractions of experimentally determined responding donors and the fractions of predicted responding alleles for all the frequent epitope peptides. This calculation requires two parameters; the percentile rank cutoff value and the allele frequency cutoff value (see Materials and methods). The correlation coefficient was calculated for each combination of these two parameter values. The results for the percentile rank cutoff values of 1% - 20% and the allele frequency cutoff value of 1-15 are given in Supporting Information Table 2. The correlation is generally poor, reaching the maximum value of 0.60 when the percentile rank and the allele frequency cutoff values are 6% and 5, respectively.

We chose the allele frequency cutoff value of 5, which gives the best correlation, to define the predicted frequent epitopes, which were then compared with the experimental frequent epitopes in the same manner as that used to evaluate the predictions using our own cohort HLA-DRB alleles. Figure 5B gives the numbers of correctly and incorrectly predicted frequent epitopes using the IEDB “recommended” method on the 15 reference HLA-DRB alleles. Compared to the predictions using our own 33 cohort HLA-DRB alleles (Fig. 4A), there are noticeably fewer peptides predicted to be frequent epitopes. FNR is still 30% (8 missed out of 27 frequent epitopes) at a very loose binding threshold of 20% (Fig 5E). At a very stringent, 1%, binding threshold, there are no peptides predicted to be a frequent epitope (Fig 5C). On the other hand, at binding cutoff of 6%, 9 out of 11 predicted frequent epitopes are correct which gives the best Precision of 82%, but the Miss Rate (FNR) is still high at 67%. The nine TP frequent epitopes at this cutoff level are peptides 12-15, 67-68, 76-78 (Fig. 5D and Table 2), which are the previously identified epitopes 1, 2A, 2B and 5 (Mazor et al., 2014). This indicates that highly immunogenic peptides can be correctly predicted at this 6% binding percentile rank cutoff and predicted responding allele frequency cutoff of 5 out of 15 reference alleles. However, the remaining 18 peptides that were found to be frequent epitopes experimentally would not have been identified.

2.6. Number of donors and alleles predicted to respond to the experimentally determined frequent epitopes

Table 2 gives the 27 experimentally determined frequent epitopes, their peptide sequences and the number of responding donors, along with the predicted number of responding donors and reference alleles at different percentile rank cutoffs for these peptides. Since the peptides overlap by 12 amino acids, one 9-mer binding core is included in two or three overlapping peptides. In some cases the epitope is spread over a number of neighboring peptides representing different cores in the same region of sequence. For these reasons, frequent epitopes often occur over several neighboring peptides. In Table 2 the peptides are grouped into 9 regions of consecutive frequent epitopes centered around a peak number of donors, of which the regions are sorted in descending order. The shaded entries indicate the number of responding donors and alleles that are above the cutoff values (4 and 5, respectively) to make the peptide a frequent epitope. Overall, 5 or 6 of the 9 regions are predicted to be epitopes independent of percentile rank between 6% and 20%. Four peptide regions I, II, V and IX are predicted to be frequent epitopes even when the binding percentile cutoff is as stringent as 1% if our cohort is used. On the other hand, epitope regions III, IV, VI and VII would be missed. Interestingly, peptide 8 has 10 responding donors (20% of the 50 in the cohort) indicating it is an important epitope. However, the analysis does not predict a single response even at 20% cutoff. Examination of the HLA alleles of the 10 responding donors for this peptide, given in Supporting Information Table 3, reveals that all of their DRB1 are covered by IEDB “recommended” method and that 9/10 donors have at least one allele included in the 15 reference HLA most common worldwide.

IEDB “recommended” method includes more than 200 DRB alleles, and only 5 and 6 DP and DQ alleles, respectively. This low selection did not cover most of our DP and DQ alleles in our cohort. Nevertheless, in order to examine if additional prediction for those alleles can improve the inability of the algorithm to predict epitope 3, we computed the binding of peptide 8 to the 5 DP and 6 DQ alleles covered by IEDB consensus. We found that both peptides got high percentile rank (>30) indicating that incorporation of DP and DQ binding predictions to the analysis would not have improved the prediction.

3.1. Comparison strategies

In this study we compared the promiscuous epitopes, which are those that produce a response in many donors that were identified experimentally to predicted epitopes using two separate DRB HLA populations. The first population includes the DRB alleles in our cohort that provide a good coverage of the world population (Mazor et al., 2012). The second population includes 15 reference HLA alleles that were shown to offer optimal coverage (Greenbaum et al., 2011) of the world’s population.

The use of cohort alleles will be useful to predict responses of specific patient populations that have unique HLA alleles or bias to certain alleles like DRB1*11 in well differentiated thyroid carcinoma and hairy cell leukemia patients (Juhasz et al., 2005; Arons et al., 2014), DRB1*0301and DRB1*1001 in ovarian carcinoma patient (Kubler et al., 2006). On the other hand, the use of a limited number of reference alleles to predict T-cell epitopes is commonly used in the field of in silico T-cell epitope predictions (Oseroff et al., 2012; Schulten et al., 2013; Li et al., 2014; Salvat et al., 2014) and would be useful for researchers that do not have any cohort. Some investigators use a stringent threshold (low percentile rank), and identify the top binders as predicted epitopes (Cantor et al., 2011; Li et al., 2014). Others use a more forgiving threshold that provides a large number of positive candidate peptides, which can then be further analyzed using experimental methods (Iwai et al., 2003; Fonseca et al., 2006; Oseroff et al., 2012; Schulten et al., 2013).

While we assume that most end-users do not have a known cohort and would use a 15-allele list for epitope promiscuity prediction, we found that this approach gives a lower correlation to the experimental data and a much higher rate of FN compared with the HLA alleles in our cohort. Two of the experimentally positive peptides could not be identified even with a threshold cutoff of 50% percentile rank that accepts 90% of all peptides as positive. This could be attributed to the fact that some of the HLA alleles that contribute to the experimental response are not included in the list of 15 alleles.

Our analysis uses a list of 15 reference alleles and treats each of them with equal importance following a prediction strategy that is considered to be state of the art (Iwai et al., 2003; Tangri et al., 2005; Koren et al., 2007; Cantor et al., 2011; Abdel-Hady et al., 2014; King et al., 2014; Li et al., 2014; Salvat et al., 2014; Salvat et al., 2015); however, it may introduce a bias because those alleles actually have different frequencies in the population. For example, if a peptide is predicted positive to DRB1*0701 (which is the most frequent in the world population), but negative to DRB1*0401 (which is much less frequent in the world population (Gonzalez-Galarza et al., 2011)), those predictions are considered equally important in the in silico analysis. In fact, their impact on the immunogenicity of the end population will not be the same, and experimental T-cell epitope mapping with a cohort that represents the world population will indicate that the peptide is more immunogenic than the prediction.

3.2. Possible reasons for FP and FN predictions

We found a high rate of FP epitopes. This finding is not surprising because the MHC binding algorithms only cover a small fraction of the factors that affect the immune response. It does not consider the processing of the protein in APC, and therefore the algorithm may predict peptides as epitopes that are not created naturally. It also does not consider the transport of the peptide to the APC surface and most importantly, the lack of ability to discriminate between tolerated and non-tolerated proteins, namely the availability of a T-cell receptor to recognize a bound peptide. There are other factors that add additional complexity like regulatory components of the immune system that can down regulate the immune response, though these aspects are also difficult to predict using experimental methods. Alternatively, we cannot rule out the possibility that the experimental method “missed” some epitopes due to low frequency of antigen-specific naïve T cells (Delluc et al., 2011) or due to a stringent threshold (of 4/50 donors), and those epitopes can be mistakenly classified as “FP” predictions. However, we do not suspect those are important epitopes, because the epitopes in this study were confirmed by epitope mapping of samples from immunized patients that had epitope spreading and clonal expansion of T cells and we did not observe new major epitopes.

It is noteworthy that the two strongest experimental epitopes (epitopes I and II in table 2) were consistently predicted correctly and were predicted to be in the top 5 epitopes throughout most of the analyses, whereas weaker epitopes (epitopes III and IV in Table 2) were considered FN in most of the analyses. A similar observation was reported by Schwaiger et al. (2014) who found very good agreement between the strong epitopes and their predictions and a poorer correlation for the weaker epitopes.

There are also surprising examples of FNs, which in hindsight does not seem so surprising. In this comparison we did not include predictions for DP or DQ molecules because there is not yet sufficient prediction data for those molecules (Wang et al., 2010). IEDB “recommended” method includes more than 200 DRB alleles, and only 5 and 6 DP and DQ alleles, respectively. This low selection did not cover most of our DP and DQ alleles in our cohort. Furthermore, previous restriction assays we have done for epitope 1 (peptide 15) using 10 donors that had a positive response to that epitope showed that this epitope is restricted by DR alone (Mazor et al., 2012). Moreover, focus on DR binding prediction is considered highly common in the art of immunogenicity prediction (Iwai et al., 2003; Tangri et al., 2005; Koren et al., 2007; Cantor et al., 2011; Abdel-Hady et al., 2014; King et al., 2014; Li et al., 2014; Salvat et al., 2014; Salvat et al., 2015) and it has been recently shown that prediction of DP and DQ had very low correlation with experimental data.(Paul et al., 2015) Therefore, we chose to focus only on DR. In our comparison study we found four epitopes that were missed by the prediction algorithm (FN) in most thresholds. It is plausible that the FN peptides are ones that cause immune response through presentation by DP or DQ (or both) and not by DR and thus could not be predicted without a prediction of DP and DQ. The inability to handle most of DP and DQ alleles is potentially a serious drawback of current prediction software.

3.3. Individual donor prediction

One of the possible applications for HLA binding predictions is the ability to individually predict immunogenicity responses for each patient based on his HLA. For example, whether a certain protein therapeutic will be immunogenic to an individual, a certain vaccine will work well or even develop specific immunotherapy tolerance for treating allergy patients. For a given person, it is unknown which HLA allele contributes to the presentation of a specific peptide or how much each different allele contributes to the activation of the T-cell population recognizing a specific peptide. Therefore, we included two types of analyses to evaluate the ability of the binding algorithm to predict the immunogenicity response of individuals. The minimum percentile rank represents the case where a single high affinity peptide-HLA allele complex is sufficient to elicit a response, while the average predicted percentile rank assumes that four presentation molecules, each with only a moderate binding, can elicit a strong immunogenicity response in combination. Interestingly, both types of analyses gave very similar TPRs at a given FPR and the AUC, except that a lower binding percentile cut off should be used if minimum percentile rank is used for analysis. Overall, the best AUC achieved for individual prediction was only 0.69.

Examination of the heat map reveals a good correlation between the experimental and the predicted responses in the most immunogenic peptides and the least immunogenic ones. The majority of the FN and FP are in the moderate responding epitopes. The presence of FP is expected as prediction algorithms are compelled to a high rate of false positives because peptide binding to an HLA molecule represents only a small and critical step in T cell activation, however the presence of FN as apparent in peptides 8-9 and others is surprising and indicates the shortcomings of immunogenicity prediction by current software (see above).

3.4. Previous efforts to map PE38 T-cell epitopes

A previous study to map the T-cell epitopes in PE38 used activation of PBMCs from donors using a similar peptide library. Peptide presentation was done using dendritic cells that were loaded with peptides and there was no stimulation with the whole protein. (Harding, 2008). This study identified three epitopes; the major one corresponding to peptides 75 and 76, and two others corresponding to peptides 15 and 65. Peptides 75, 76, and 15 were identified in our experimental assay (ranked 1^st and 2^nd). As shown in Table 2, they were predicted to be among the most frequent epitopes using all thresholds and analysis strategies. This supports the observation that the prediction algorithm predicts the strongest and most promiscuous epitopes. The third epitope identified by Harding (Harding, 2008) is peptide 65. This peptide was not found positive in our experimental results. However it was predicted to be relatively immunogenic (ranked 20^th, 24^th and 40^th using 1%, 6% and 20% thresholds, respectively). We consider that the discrepancy between the experimental methods is a result of antigen processing and presentation of the full protein, such that peptide 65 is non-immunogenic in a native setting. This is supported by an analysis of the sequence of peptide 65 using protein processing algorithms PeptideCutter (Wilkins et al., 1999), which predicted this peptide to be cleaved by common proteases at position 8 of the peptide.

3.5. Current uses for algorithms

One of the common uses of algorithm based T-cell epitope prediction is to narrow the peptide population to be examined in vitro (Fonseca et al., 2006; Oseroff et al., 2012; Paul et al., 2013). This approach assumes that the peptides that have a low binding (high percentile score) will not bind to HLA molecules and thus there is no chance for those peptides to be epitopes. This approach is based on the assumption that algorithms always over-predict and rarely miss. We found a high rate of FNs in our analysis and 5 peptides that would not have been considered positive unless the entire 111-peptide library was considered positive. Furthermore, a promiscuity prediction holds the risk of missing peptides that bind at high affinity to only one or two HLA molecules and not be considered a promiscuous epitope.

When we had completed our study, an analysis comparing the computed and experimental responses of more than 95 donors and peptides derived from 30 proteins was published (Paul et al., 2015). This work focused solely on the false positive rate of the predictions, mainly due to the nature of the experimental work that did not allow identification of false negative predicted peptides. Nevertheless, the correlation and conclusions from this work strongly agree with the high rate of false positive responses observed in our study.

In conclusion, T-cell epitope prediction using HLA binding algorithms provides a convenient and easy to use tool. Current prediction tools offer very good binding data for DRB molecules but are hindered by the lack of knowledge of DP and DQ binding. Based on the rates of FN peptides in this analysis we conclude that current HLA binding algorithms are not yet reliable as a standalone strategy and should be accompanied with experimental work.

4.1. Description of the experimental procedure

The experimental method that was used included a step of in vitro expansion of PBMC from 50 donors with the whole protein for 14 days, followed by re-stimulation with a library of 15-mer peptides spanning the whole sequence of PE38 with 12 residues overlap between neighboring peptides. Assessment of the T-cell activation response was done by counting the number of activated cells using IL-2 ELISpot. The detailed steps were previously described (Mazor et al., 2012). The epitopes identified in this study were further corroborated using samples of clinical patients that were previously treated with the protein and had a neutralization response to it. Epitope mapping using those samples gave the same epitopes (Mazor et al., 2014). The initial stimulation with the whole protein and in vitro expansion step reduces the rate of experimental FP (compared to other assays that only stimulate the cells with peptides) because it eliminates positive response from peptides that are not naturally processed. It also improves the sensitivity and reproducibility of the T-cell epitope mapping.

4.2. HLA-DRB alleles used within our cohort

Human PBMCs from naive donors were collected under research protocols approved by the National Institutes of Health (NIH) Review Board (99-CC-016). HLA typing was performed either by the HLA typing unit at NIH or by Thermo Fisher transplant diagnostic service using sequence specific primer extension and sequence based typing (Bunce et al., 1995; McGinnis et al., 1995).

4.3. Analysis of ELISpot experimental data

To distinguish the positive signals from noise for each donor, the ELISpot count of each peptide of each donor was normalized by calculating the Z-score as

where E_i is the average ELISpot count of four replicas of each peptide i for each donor, $\stackrel{‒}{E}$ is the average ELISpot count of that donor toward all PE38 peptides, and S is the standard deviation about the average. The ELISpot counts from the negative control, no peptide added, were not included in the mean and standard deviation calculations. The histogram of the Z-score does not show normal distribution, with the peak shifted to the left and a long tail extended to the right. There would still be many signals hidden in the background within 3 sigmas. To better differentiate the signals from noise, Z-scores were calculated twice: The first Z-score calculation used responses toward all 111 PE38 peptides to calculate the average and the standard deviation. In the second calculation, the responses that had the first Z-score greater than or equal to 1 were omitted when calculating the mean and standard deviation.

The Z-scores were then “binarized” using a cutoff score, i.e. turned into 1 or 0 depending on whether the value was greater or less than the Z-score cutoff value, respectively. A peptide was considered immunogenic, or to be an epitope, to a donor if the corresponding binarized Z-score was 1. The Z-score cutoff value chosen was 6, which produced epitopes that matched best with the epitopes previously identified (Mazor et al., 2014).

4.4. Prediction programs and analysis of prediction data

The prediction programs used in this study are from the IEDB MHC-II binding predictions tool (Zhang et al., 2008) which includes the following prediction methods: IEDB “recommended”, IEDB “consensus” (Wang et al., 2008), Combinatorial library (Comlib) (Sturniolo et al., 1999), NN-align (netMHCII-2.2) (Nielsen and Lund, 2009), SMM-align (netMHCII-1.1) (Nielsen et al., 2007), Sturniolo (Sturniolo et al., 1999) and NetMHCIIpan (Nielsen et al., 2008). The sequences for the 111 peptides spanning PE38 and a set of HLA-alleles (see below) were input to the website. The prediction results are given in terms of the percentile rank of the score of each input peptide-allele pair against the scores of five million random 15mer peptides from the SWISSPROT database for each allele. The smaller the percentile rank, the higher the affinity.

The prediction results were also “binarized” using a percentile rank cutoff value. When the prediction result was considered for a donor, who may have up to four different HLA-alleles (two DRB1 and two DRB3/4/5), the binarization was done either using the average percentile rank or on the basis of the best (the smallest) percentile rank over all the alleles of the donor (see below).

4.5. Evaluation by counting correctly predicted donor-peptide pairs; ROC curves

We predicted the epitopes for individual donors by computing the binding scores of 111 peptides to four HLA DRB alleles of each donor (Supporting Information Table 1). The responses were binarized using a percentile rank cutoff value.

To evaluate the performance of the predictions, we calculated the rate of TPs and FPs using the binarized data. At a given percentile rank threshold, TP for example, is the peptide-donor pair for which the predicted response is below the percentile rank cutoff (the lower the percentile rank, the stronger the binding) and the ELISpot Z-score is above the Z-score cutoff value of 6. The TPR is the fraction of TPs among all experimentally positive donor-peptide pairs. The ROC curve (Hanley and McNeil, 1982) was made by plotting the TPR against the FPR as the percentile rank cutoff value was varied and the AUC was calculated.

4.6. Evaluation by counting the correctly predicted “frequent epitopes”

A peptide was considered to be a “frequent epitope” if it was an epitope for four or more donors among our cohort of 50 donors. Since the IEDB “recommended” method has greater allele coverage and the performance is similar to IEDB “consensus” method, only the predictions from IEDB “recommended” method were analyzed. Also, for this evaluation, we used the best binding allele among up to four different alleles to represent each donor because the previous ROC curve analysis indicated that prediction accuracy was slightly better by using the best prediction score (lowest percentile rank) rather than the average.

The TPs, FPs, TNs, and FPs were computed by comparing the predicted frequent epitopes to the experimentally determined frequent epitopes. Following quantities were calculated to measure the performance:

Sensitivity = TPR = TP/(TP+FN) = fraction correctly predicted as frequent epitope among all experimentally determined frequent epitopes.

Precision = positive predictive value = TP/(TP+FP) = fraction of experimentally determined frequent epitopes among all that are predicted as frequent epitopes.

Accuracy = (TP+TN)/(TP+FN+FP+TN) = fraction correctly predicted, both frequent and not frequent, among all peptides.

Miss rate = FNR = FN/(FN+TP) = fraction of peptides not predicted as frequent epitope among all experimentally determined frequent epitopes.

Fall-out = FPR = FP/(FP+TN) = fraction of peptides predicted to be frequent epitopes among those non-epitope peptides determined experimentally.

4.7. Evaluation using HLA-DRB alleles in the worldwide population

To evaluate the ability of the prediction programs to predict epitopes without donor cohort information, 15 reference alleles that provide maximal population coverage (Greenbaum et al., 2011) were submitted to IEDB “recommended” method to obtain the predicted binding percentile rank for each allele-peptide pair. These would be useful in cases where the HLA sequences of the human subjects are not known.

As before, a peptide is a frequent epitope experimentally if four or more donors have Z-scores above 6 for the peptide. A predicted (as opposed to experimentally determined) frequent epitope in this case is the peptide that is an epitope (percentile rank below a cutoff value) to more than a certain number, the allele frequency cutoff, of alleles. The frequent epitopes predicted and experimentally determined in this manner were compared by calculating the TPs, FPs, TNs and FNs in the same way as described before. In addition, for each percentile rank cutoff value and each allele frequency cutoff value, the Pearson Correlation Coefficient was computed between the fractions of predicted responding alleles and the fractions of the experimentally determined responding donors for all peptides.

4.8. Statistical analysis

All the ROC curves, HEAT maps and Pearson correlation coefficients were calculated and generated with MATLAB software.