High-Throughput Analysis of Antimalarial Susceptibility Data by the WorldWide Antimalarial Resistance Network (WWARN) In Vitro Analysis and Reporting Tool

Data sets used.

Primary data sets describing the growth of P. falciparum in culture at varying drug concentrations in individual wells of 96-well plates were collated, allowing the comparison of data obtained using various assay methods (microscopic assessment, radiolabeled hypoxanthine uptake inhibition, HRP2 and Plasmodium LDH [pLDH] enzyme-linked immunosorbent assays [ELISA], and SYBR green). Three groups of drugs were studied: (i) artemisinins found in ACTs, consisting of dihydroartemisinin (DHA), artemether (AM), and artesunate (AS); (ii) ACT partner drugs (or active metabolites) desethylamodiaquine (DQ), lumefantrine (LUM), mefloquine (MQ), and piperaquine (PIP); and (iii) chloroquine (CQ) and quinine (QN), drugs that are no longer recommended for first-line treatment of P. falciparum malaria. Data sets describing at least 40 parasite isolates were considered large enough to be included in this analysis. Fourteen data sets from 11 laboratories fulfilled these criteria (Table 1). The laboratory methodologies for many of these studies have been described previously (17–27). The primary growth outputs from assays were formatted as uniform 12-by-8 96-well plate layouts in spreadsheets to facilitate automated processing and analysis.

Analysis of constraints for curve fitting.

The levels of uninhibited and maximally inhibited growth are key parameters in IC₅₀ calculations. For example, in the sigmoid E_max model, these levels represent the upper and lower asymptotes of the concentration-inhibition curve, where E_max is defined as the difference between these two measures of growth. The concentration-inhibition curve can be left unconstrained at its upper and lower ends (i.e., a 4-parameter model), particularly with large numbers of points, but for antimalarial susceptibility studies, this is frequently not practical, since the small number of drug concentrations used (in many cases, only 7) can produce highly unstable estimates. The upper (baseline growth without the drug) and lower (minimum growth) values were therefore constrained prior to modeling. Given the variability in experimental design and plate layout in the data sets examined, the upper growth constraint for each set of drug concentration-growth data [i.e., the baseline level of uninhibited growth, G(C₀)] (see Table S1 in the supplemental material) was defined as the average growth in all drug-free wells on the same plate.

Assessment of an appropriate means of determining the lower growth constraint involved systematic examination of concentration-inhibition data from a range of data sets. This approach took into account the biological reality that growth at the highest drug concentration, G(C_max), does not always correspond to the maximum drug effect because of a paradoxical rise in the growth measured at very high drug concentrations (Fig. 1), a phenomenon that has been noted previously (15). To explore this issue further, we prospectively defined two measurements of growth reduction (see Table S1 in the supplemental material) providing distinct measures of drug efficacy: R_max1, calculated as G(C₀) − G(C_max), and the modified measure of efficacy R_max2, calculated as G(C₀) − G_min, where G_min is defined as the mean growth at the two concentrations ranked as having the lowest mean growth in the concentration-inhibition series.

A pooled analysis of possible factors associated with the occurrence of an apparent paradoxical increase in growth at high drug concentrations was undertaken; this effect was considered to be present when R_max1/R_max2 was <0.9 (a >10% rise in apparent growth over that at intermediate drug concentrations). The roles of the drug and the assay methodology were explored using a random-effects model (Stata, version 11.1; StataCorp), with the drug and the method as fixed effects and the site as a random effect (due to the heterogeneity between sites). Since it was suspected from initial observations that this phenomenon was associated with drugs that are relatively inactive against ring-stage parasites, DHA was used as the reference group for the drug; it was also the most commonly assayed antimalarial drug in current use (1,391 assays across 13 of the 14 data sets). Hypoxanthine incorporation was defined as the reference method.

High-throughput estimation of IC₅₀s.

Curve fitting was undertaken using a sigmoid E_max model. In its general form, this model has four parameters: the IC₅₀ (the 50% effective concentration [EC₅₀] for concentration-inhibition data), a measure of the curve's steepness at the IC₅₀ (the sigmoidicity factor, or gamma), and the levels of uninhibited growth and maximally inhibited growth (see above).

Code from ICEstimator (15), based on the nls algorithm of R, which performs successive fittings of a sigmoid E_max model to concentration-inhibition data, was adapted within a Google Web Toolkit (GWT) Java-based Web application to perform data transformation, standardized analysis, and reporting of IC₅₀s for each data set. The details of ICEstimator have been described elsewhere (15). Briefly, the primary growth data are first converted to a percentage scale, with baseline growth (no drug) representing 100% and minimum growth (maximum drug inhibition) representing 0%. Following this conversion, the model is constrained at its upper and lower ends to 100% and 0%, respectively, and therefore produces only two parameters: the IC₅₀ and the sigmoidicity factor (gamma). Initial values for the IC₅₀ and gamma are determined by the point at which growth first falls below 50% of control growth (15, 18), and iterations are then undertaken until the limit of improvement is reached. In case of nonfitting (because of a weak dose-response relationship or a paucity of intermediate data points between 100% and 0%, as seen with a very steep slope), curve fitting is attempted again with gamma fixed at 10, based on a previous sensitivity analysis showing that gamma values greater than 10 would not significantly alter IC₅₀s in steep curves (15).

The sigmoid E_max model is focused primarily on determining the IC₅₀ and the slope at this IC₅₀, and all other points on the modeled line are entirely determined by the IC₅₀ and gamma. Points toward the ends of the curve, such as the IC₉₀ and IC₉₅, frequently depart to some degree from the data observed, and for this reason, these values are potentially misleading and are not reported by IVART.

Assessment of criteria for defining a reliable subset of assays.

It is generally recognized that at least 30% of parasite isolates placed in short-term culture exhibit less than optimal growth due to preexposure to drugs or other factors contributing to reduced parasite viability (4). To detect assays that are less reliable due to such factors, IVART was set up to calculate the ratio of the upper and lower 95% confidence intervals of the IC₅₀ estimate, known as the confidence interval ratio (CIR). A threshold CIR of <3 was selected to define core assays of higher reliability for entry into pooled analyses and association studies. The CIR parameter is not useful in a subset of cases where modeling can be achieved only with a fixed gamma of 10, since this becomes a 1-parameter model, with inevitable narrowing of confidence intervals. For this subset of fixed-gamma assays, the growth ratio (uninhibited growth divided by maximally inhibited growth) was used to define core assays of higher reliability in accordance with previous recommendations (4). For each data set, the main subset of assays in which both the IC₅₀ and gamma were successfully obtained was examined, and the proportion of assays with tight confidence intervals (CIR, <3) was determined at four levels of the growth ratio: <2, 2 to 3, 3 to 5, and >5.

The effect of applying reliability criteria was explored by examining intraisolate Pearson correlations of IC₅₀s for drug pairs in the whole data set and repeating this procedure with increasingly strict criteria.

Relative efficacies of artemisinins and partner drugs.

Within the 12 nonmicroscopic data sets, the growth ratio (uninhibited growth divided by maximally inhibited growth) for each drug was compared to that for DHA by using the Mann-Whitney test (with Bonferroni's correction for the number of drugs). The relative proportions of growth inhibition by DHA and MQ for individual parasite isolates, as illustrated in Fig. 1, were assessed by the Wilcoxon signed-rank test.

Constraints for curve fitting.

The 14 data sets contained 7,350 individual drug inhibition assays. Analysis of growth inhibition characteristics revealed that 1,334 (18.1% of total) assays showed evidence of paradoxical growth at high concentrations (a rise in apparent growth of >10% over that at intermediate concentrations). Both the nature of the drug being tested and the readout methodology were clearly associated with this behavior. The four ACT partners (DQ, LUM, MQ, and PIP) were associated with a risk 2- to 3-fold greater (P, <0.001 in all cases) than that for the reference drug, DHA, while CQ and QN showed paradoxical effects at an intermediate level (P, <0.001 in both cases) (see Table S2 in the supplemental material). The phenomenon was seen less commonly with AS than with DHA. The assay method was also relevant; by using hypoxanthine incorporation as the reference group, a paradoxical increase in growth was most commonly encountered in assays based on LDH quantification by ELISA (odds ratio [OR], 1.27; P, 0.007). There was also a trend toward less-frequent occurrence in HRP2 and SYBR green assays that did not reach statistical significance. The issue was not encountered at all in assays assessing schizont maturation by microscopy.

These findings indicate that a paradoxical increase in growth measured at high drug concentrations reflects biological properties of antimalarial drugs rather than being simply an experimental artifact due to equipment or human error. For this reason, it was decided to amend the lower constraint of the sigmoid E_max model to G_min in order to provide a more accurate measure of maximum inhibition, avoiding underestimation of overall drug efficacy and spuriously low IC₅₀s.

Reliability criteria.

The default criteria for defining a subset of more-reliable assays consisted of one main criterion, an IC₅₀ confidence interval ratio (CIR) of <3. In the subset of curves with a fixed gamma (1,427 assays [19.4% of the total]), a growth ratio indicating a satisfactory signal/background ratio was used to assess reliability. As expected, there was a clear relationship between the growth ratio and the CIR in the 80.6% of assays where gamma was derived by modeling (i.e., 2-parameter models) (see Fig. S1 in the supplemental material). Growth ratios of 3 to 5 and >5 were associated with very high levels of assays with tight confidence interval ratios (across the 12 nonmicroscopic data sets, the median proportions with CIRs of <3 were 91.3% for a growth ratio of 3 to 5 and 95.3% for a growth ratio of >5). In contrast, a growth ratio of <2 was associated with relatively high proportions (median, 33.9%) of assays with uncertain IC₅₀ estimates (CIR, >3). A growth ratio of 2 to 3 was generally associated with high levels of assays with tight confidence intervals (median proportion, 86.8%), but there did appear to be greater potential for less-reliable IC₅₀ estimates to be accepted at this growth ratio level in hypoxanthine incorporation assays (see Fig. S1 in the supplemental material, data sets A and C).

The effect of applying reliability criteria was assessed by calculating the correlation between IC₅₀s for drug pairs within each data set, since it was predicted that application of increasingly strict criteria might improve correlation coefficients. MQ-QN and CQ-DQ provided the most powerful test cases given the number of data sets assessing each drug pair (11 and 8, respectively) and the strong, consistently documented associations between IC₅₀s for these drug pairs in field studies (20, 28–33). Exclusion of assays with CIRs of >3 and growth ratios of <2 (for assays with a fixed gamma) led to stronger correlations in 9/11 data sets for MQ-QN and in all 8 data sets for CQ-DQ (Fig. 2; see also Table S3 in the supplemental material). Statistical significance was generally maintained or strengthened in the more-reliable subset despite the reduction in sample size.

Increasingly strict classification of the fixed-gamma subset of assays, involving the exclusion of additional assays with growth ratios of <3 or <5 generally led to relatively small and inconsistent improvements in correlation (Fig. 2; see also Table S3 in the supplemental material). For hypoxanthine assays, where there was concern that unreliable assays might be accepted with growth ratios of 2 to 3 (see above), the mean proportion of additional isolates excluded in this range ranged from 1.8 to 5.1% in the four data sets (examining all drug pairs). The effects of various levels of exclusion criteria in hypoxanthine assays (drug pairs MQ-QN and CQ-DQ) are illustrated in Fig. S2 in the supplemental material.

IVART was therefore set to apply a default growth ratio of 2 to classify the subset of curves with a fixed gamma. When this default growth ratio was combined with the main criterion of a CIR of <3, 6,158 of 7,350 curves (83.8%) met the IVART core criteria.

Relative efficacies of artemisinins and partner drugs.

Using only data that conformed to the default IVART reliability criteria (see above), it was possible to discern clear patterns in terms of the growth ratio depending on the assay and the drug (Fig. 3). For example, hypoxanthine-based assays showed the highest growth ratios (typically 5- to 15-fold reductions in growth), while protein-based and SYBR green assays had substantially lower growth ratios (i.e., relatively higher background values). Furthermore, nonartemisinin drugs tended to show lower growth ratios than artemisinin derivatives, an effect more marked with ELISA-based assays.

This issue was explored in more detail by examining the proportion of growth that could be inhibited {[G(C₀) − G_min]/G(C₀)} (Fig. 1) by DHA compared to MQ for individual parasite isolates for which both assays passed the core criteria. In all 11 data sets with data for both drugs, MQ inhibited growth significantly less than DHA (P, <0.01 by the Wilcoxon signed-rank test), but the extent of this effect ranged from 1.0 to 30.2% (median values) across data sets (Fig. 4). ELISA-based assays showed substantially greater differences than those based on hypoxanthine incorporation or SYBR green fluorescence.

IVART online.

The modeling approaches, constraints, and core criteria described above were adopted within an online version of IVART that is now available for external use at http://www.wwarn.org/toolkit/data-management/ivart. After a one-time registration process, users have access to the tool via a personalized interface that provides secure upload and storage of primary data in a 96-well plate format. IVART incorporates a “Plate Assistant” function to verify data layout and drug concentration information and then undertakes a single-pass analysis that produces graphical (Fig. 5) and spreadsheet reports for each individual assay, along with summaries of reliable-assay subsets by drug and year of study.

The range of drug concentrations in tests can also affect the calculated IC₅₀. In some data sets, the lowest concentration of the drug tested demonstrated more than 50% inhibition, suggesting that the drug concentration range tested was too high. Such assays still provide useful information on drug sensitivity (as long as they pass core criteria) but are marked in IVART with a “Range High” warning, because reassessment of the range of concentrations used in future assays may be indicated. Similarly, in some assays, growth inhibition appears to be incomplete, even at the highest drug concentration used. A “Range Low” warning is therefore displayed when the level of inhibition at the highest concentration of a drug is >10% greater than that at the next lowest concentration. Cautious interpretation of such assay data is required, since the effect may be explained by technical factors, such as underdosing of the drug in the wells or hemolysis of red blood cells, but such assays may also hint at emerging drug resistance.

A standardized approach to modeling in vitro data.

Differences in laboratory and analytical practices complicate the comparison of data from antimalarial susceptibility assays obtained in different laboratories. This problem can potentially be reduced in a number of ways (2), including the use of validated reference clones (25, 34, 35) and quality-controlled drugs (36). IVART was developed to address a third source of variability by defining a single approach to the calculation of IC₅₀s that could be applied to primary data collected using a range of growth readout methods.

Examination of a wide range of data sets from 11 in vitro testing laboratories confirmed wide variations in experimental method and design. The number and identity of drugs being assessed differs across laboratories, presumably influenced by local patterns of clinical drug usage and susceptibility. This, in turn, affects the number and range of drug concentrations assessed, along with the number of no-drug control wells (critical for establishing a baseline for calculating IC₅₀s). In addition, some investigators select specific subsets of control wells from certain rows or columns for each drug, and control growth values may also be derived from wells containing low drug concentrations if these produce higher apparent growth than drug-free wells for any reason. Other potential sources of variation include the use of different models for curve fitting and manual removal of individual points considered to be outliers.

We began the process of standardizing the calculation of IC₅₀s by selecting the sigmoid E_max nonlinear regression model for curve fitting, since this does not involve subjective decisions regarding the form of the inhibition curve (a potential requirement if a polynomial curve is used). The upper and lower bounds of the model were constrained; given the range of plate layouts employed by different investigators, the upper constraint was defined as the mean growth in all wells on the plate with no drug present. Although edge and cross-plate changes in growth have been reported (37), the use of this larger number of drug-free control wells provides a statistical advantage in terms of greater numbers.

Several approaches to defining a lower constraint for curve fitting of antimalarial susceptibility data have been described. In hypoxanthine incorporation assays, the signal in uninfected red blood cells can be used, while the background in ELISA-based assays can be obtained by measuring the baseline antigen present at the start of incubation. However, in practice, these parameters are rarely recorded, and growth at the highest drug concentration, G(C_max), is commonly used; for example, G(C_max) was the method of choice in the original description of the LDH ELISA (9). However, in the data sets examined here, the assumption that the highest concentration of a drug defines its greatest level of inhibition proved incorrect, since nearly one-fifth of assays showed a paradoxical increase in apparent growth at very high drug concentrations. This effect did not appear to be due simply to noise or outlying values, since it was more marked with nonartemisinin drugs and ELISA-based readouts. One possible explanation for such paradoxical apparent growth at high drug concentrations is that drugs lacking a primary effect on the ring-stage parasite at standard pharmacological concentrations may nevertheless affect the ring-stage parasite in other ways, leading to altered transcriptional responses and increased protein production or nucleic acid uptake. Additional explanations include precipitation of the drug from the solution at high concentrations, plate edge effects (37), and mixed-clone infections (38). Whatever the mechanism of this higher apparent growth at high drug concentrations, the definition of maximum growth inhibition clearly needs to account for cases where maximum inhibition occurs at intermediate concentrations of a drug. Ranking of concentrations in terms of the degree of inhibition allowed the selection of a modified measure of maximal inhibition, based on average growth over the two concentrations with the lowest growth.

Systematic application of reliability criteria.

Since a proportion of parasites adapt poorly to in vitro culture and provide misleading signals, investigators usually define a core subset of more-reliable assays for use in association studies and summary outputs. The method for defining such assays is rarely described in publications, and when conducted at the level of the individual assay, the decision-making process is likely to be time-consuming and potentially subjective. A key aim of IVART was to promote an objective approach to be applied across whole data sets. IVART uses a confidence interval ratio (CIR) of the IC₅₀ estimate as its main method of defining core assays: a CIR of <3 is considered to indicate a reliable assay. However, the CIR is not useful in a subset of assays where initial 2-parameter modeling of the concentration-inhibition data fails and a fixed gamma value of 10 is used (around 20% of all assays); in this scenario, other means of defining reliable assays are required. Measures of goodness of fit were not chosen as IVART's default criteria because of the clear evidence that in a proportion of assays, the biological properties of drug inhibition produce data that naturally deviate from the classical sigmoid concentration-inhibition curve (see above). Such assays may be robust in terms of signal but nevertheless produce poor scores in goodness-of-fit assessments and would tend to be inappropriately rejected.

Historically, the overall level of signal to background (uninhibited to maximally inhibited growth, known as the growth ratio) has been recommended as a means of defining reliable curves (4). Examination of the relationship between the growth ratio and the confidence interval ratio across the data sets indicated that a threshold growth ratio of 2 would lead to acceptance of very few unreliable assays for ELISA- and SYBR green-based assays, and this was adopted within the default criteria of IVART. However, it was noted that there was a greater potential to accept less-reliable data in hypoxanthine-based assays, where the signal-to-background ratio is usually much higher than 2. This is also consistent with previous suggestions for reliability criteria in hypoxanthine-based studies, for which a growth ratio of 5 was proposed (4). In this study, when more-restrictive criteria were applied in hypoxanthine-based data sets, leading to the exclusion of assays with growth ratios of <3 or <5, relatively few additional isolates were excluded (since such growth ratios are rarely encountered in hypoxanthine-based data sets, and the growth ratio is applied only to the minority fixed-gamma subset). Accordingly, the correlation coefficients for MQ-QN and CQ-DQ did not, on the whole, improve with these more-stringent criteria.

IVART was not designed with assays based on microscopic assessment in mind, since the use of microscopy-based methods to assess growth is decreasing. For all the drugs described here, growth inhibition should be complete at high drug concentrations, so the issues of determining maximal inhibition and the use of the growth ratio (usually infinity in schizont maturation experiments) to define reliable assays under certain circumstances do not apply. Nevertheless, the tool may be useful to laboratories continuing to use this methodology provided these issues are appreciated.

Distinctive properties of artemisinin derivatives.

The high-throughput nature of IVART provided a unique opportunity to undertake a systematic examination of growth characteristics across a range of drugs and readout methods. As well as informing the design of IVART itself, this process provided additional biological insights informative for the future design and interpretation of in vitro antimalarial susceptibility studies. There was clear evidence that artemisinin derivatives show higher efficacy (i.e., inhibition of growth) than ACT partner drugs; for example, DHA inhibited a significantly greater proportion of growth than MQ in all sets of assays. This finding is consistent with the earlier onset of action of artemisinins, at the ring stage of parasite development (39–41), but the fact that this property is substantially greater in ELISA-based readouts had not been documented previously. The most likely reason for this is that both LDH and HRP2 are produced in significant quantities by ring-stage parasites (42), while hypoxanthine and SYBR green signals accumulate only at more-mature stages of asexual parasite development. There also appeared to be an effect of site, possibly reflecting the critical role of the timing of drug exposure in relation to the parasite stage. In locations with substantial delays between the removal of the sample from the patient and the setup of in vitro culture, parasites are more likely to first encounter the drug at mature stages, when they are susceptible to a wider range of compounds.

These observations prompt a reevaluation of how resistance is measured for different classes of antimalarial drugs. In the ACT era, assessment may require different approaches for artemisinins, which act rapidly against both the ring and mature stages, and ACT partner drugs, which act only against the more-mature stages of parasite development. The timing and duration of parasite-drug contact have been identified as important determinants of antimalarial susceptibility in the laboratory (2, 43), and specific methodologies and analyses for different applications are likely to provide more relevant information than a single method alone. Both the microscopic and hypoxanthine methods were developed in the era of slower-acting antimalarials with longer half-lives (CQ and MQ) (44); in these assays, ring-stage parasites contribute little signal (indeed, ring-stage growth is not assessed at all if hypoxanthine is added only after 24 h of incubation). In contrast, forms of artemisinin resistance reported from Southeast Asia (27, 45, 46) have been proposed to be confined to ring-stage parasites (47) and would not be predicted to influence susceptibility at the trophozoite or schizont stage. Short pulses of a drug during ring-stage growth have lasting growth-inhibitory effects (39–41, 48–52), and ring-stage pulse assays (in which relatively high concentrations of artemisinins are applied for relatively short periods) have been described recently (53, 54), providing the first clear view of ring-stage artemisinin resistance in parasites from western Cambodia (53). The need to remove a drug or to quantify ring-stage growth using a specific marker may present a challenge for widespread field use of this technique.

Summary and future work.

IVART provides high-throughput, rapid, single pass analysis of in vitro sensitivity data, avoiding a variety of manual and potentially subjective processing steps currently in use. The tool can be applied to data sets obtained by a variety of methods and defines a subgroup of core IC₅₀s of greater reliability for pooled analyses and association studies. Its advantages, therefore, relate to consistency of approach and convenience.

The criteria suggested for accurate identification of a reliable subset of assays for use in association studies appear to be well suited to the signal-to-background properties of data sets from ELISA and SYBR green assays (methods increasingly used by in vitro testing laboratories); this is evidenced by substantially improved correlation scores for interdrug comparisons upon the application of these criteria. Nevertheless, ongoing monitoring of the tool's operation will be important in order to confirm prospectively that the approaches described are appropriate for further data sets from a variety of laboratories and methods. The handling of assays with sparse data around the IC₅₀, and consequently steep falls in growth between two drug concentrations, is a particular challenge for high-throughput approaches. Future incorporation of an additional algorithm that is better able to fit 2-parameter models to such data may provide a further advance, although this will require careful validation on a similarly representative data set. Large data sets of the type described here may also be used to develop mixed-effects modeling approaches to the analysis of concentration-inhibition data, involving a Bayesian framework for assessing whether individual P. falciparum isolates are resistant to a given drug.