Research Diagnostic Criteria for Temporomandibular Disorders: Validity of Axis I Diagnoses

Reliability of the RDC/TMD Examination Protocol

The level of measurement reliability (agreement or measurement reproducibility) associated with any diagnostic instrument can be a limiting factor for its diagnostic validity (accuracy). Reliability results for the RDC/TMD algorithmic diagnoses are reported in the second paper in this series.⁽²⁾ It was concluded that the reliability of the RDC/TMD protocol is excellent when myofascial pain diagnoses are combined into one diagnosis. Reliability was good for all the common diagnoses, including myofascial pain, myofascial pain with limited opening, arthralgia, and disc displacement with reduction. Of the less common diagnoses, reliability was good for disc displacement without reduction with limited opening. However, reliability was fair to poor for disc displacement without reduction without limited opening, osteoarthritis, and osteoarthrosis. These estimates of reliability are consistent with the results of the most comprehensive study to date that assessed the reliability of the RDC/TMD Axis I TMD diagnoses.⁽³⁾

Validity of the RDC/TMD Examination Protocol

Ongoing efforts to investigate the validity of the RDC/TMD taxonomic system were anticipated at its inception in 1992, as it was not intended to be an end product.⁽¹⁾ However, the diagnostic accuracy of the RDC/TMD has not been tested comprehensively. Only limited comparisons of components of the RDC/TMD algorithms have been made to objective evidence or gold standard. The RDC/TMD diagnosis of arthralgia has been compared to magnetic resonance imaging (MRI) findings for disc displacement, joint effusions or osteoarthritis.^(4–6) The RDC/TMD diagnosis for disc displacement has been compared to MRI-detected disc displacement.^(7–11) A clinical diagnosis of disc displacement has also been compared to joint sound recordings.⁽¹²⁾ The aim of the latter study was to investigate the validity of the RDC/TMD criterion for disc displacement with reduction that stipulates a reciprocal joint click should occur at an interincisal mouth opening measurement that is 5 mm greater than for the closing click. However, no comprehensive assessment of the validity of the RDC/TMD has been performed.

Validity Testing

The definition of validity is the degree to which an index test correctly classifies the presence/absence of a disorder in individuals when compared with a reference standard, also referred to as a gold standard or criterion measure. For such a comparison, both the index test and the reference standard are measured in participants who are suspected of having the condition of interest.⁽¹³⁾ With use of a reference standard, the criterion validity of an index test is established. Validity can be evaluated in terms of sensitivity and specificity. Sensitivity is the proportion of participants with the target disorder who have a positive test outcome. Specificity is the proportion of participants without the target disorder who have a negative test outcome. The term “test” refers to any method used to obtain diagnostic information relevant to a patient’s health status. It may include information from history and physical examination, laboratory tests, imaging tests, function tests, and histopathology.⁽¹³⁾

The aim of this paper is to report the validity of the RDC/TMD Axis I algorithm-derived TMD diagnoses when compared with reference-standard diagnoses. The reference standards were established by 2 expert TMD and orofacial pain dentists who independently performed the standardized criterion examination protocol, rendered their diagnoses, and then came together for a consensus diagnosis. This procedure is described in the first paper of this series.⁽¹⁴⁾

Study Setting, Locations, and Examiners

Data collections were carried out at 3 sites: the University at Buffalo (UB), the University of Minnesota (UM), and the University of Washington (UW). Based on terminology recommended by the Standards for Reporting of Diagnostic Accuracy (STARD),⁽¹³⁾ these data collections were prospective in that all history, examination, and imaging data collections were planned before the index test (RDC/TMD procedures) and the criterion procedures for the reference standard were performed. No validation data measures were collected retrospectively. A total of 9 clinicians served as the examiners for the RDC/TMD Validation Project, including 2 criterion examiners (CEs) and 1 test examiner/dental hygienist (TE) for each study site. All 6 CEs were experienced TMD and orofacial pain dentists. Their training, experience, and reliability are reported in the first paper in this series.⁽¹⁴⁾ The 3 dental hygienists who served as the TEs were trained and calibrated to perform the RDC/TMD examination protocol.

Study Participants

During August 2003 to September 2006, the 3 participating study sites recruited a combined total of 628 TMD cases and 91 controls from clinic and community sources. The site principal investigator or co-investigator at each study site determined that the participants fully understood the study procedures and obtained consent. All procedures were reviewed and approved by the Institutional Review Boards overseeing each study site. Fourteen participants, distributed across the three sites and originally categorized as cases were excluded due to the presence of chondromatosis (n = 2), reported fibromyalgia (n = 9) or reported rheumatoid arthritis (n = 3). Persons with a documented medical diagnosis of fibromyalgia or rheumatoid arthritis were included in the study. Chondromatosis was excluded based on suspicion of the presence of the disorder as detected on MRI by the radiologist. It is possible that a small number of subjects may have had fibromyalgia or emerging symptoms of fibromyalgia but potential subjects reporting widespread pain as required in fibromyalgia were excluded to reduce the number of undiagnosed cases. A total of 614 cases remained for the primary analysis. The first paper of this series presents a complete description of the recruitment of the participants and their compensation, the inclusion and exclusion criteria used, the study sample demographics, and baseline clinical characteristics.⁽¹⁴⁾ In this first paper, we established that the participant population was appropriate for assessing the validity of the Axis I TMD diagnoses.

Index Test

The index test specified for this validation project was the published RDC/TMD Axis I diagnostic examination procedure that employs a set of standardized clinical and questionnaire items.⁽¹⁾ This procedure allows for assignment of TMD participants to any of 3 diagnostic groups that include 8 sub diagnoses:

• Group I Muscle Disorders: (Ia) myofascial pain; (Ib) myofascial pain with limited opening.

• Group II Disc Displacements: (IIa) disc displacement with reduction; (IIb) disc displacement without reduction with limited opening; (IIc) disc displacement without reduction without limited opening.

• Group III Arthralgia, Arthritis, Arthrosis: (IIIa) arthralgia; (IIIb) osteoarthritis; (IIIc) osteoarthrosis.

RDC/TMD nomenclature, especially for Group III is not universally employed. For purposes of this paper, the term “arthralgia” will be used to describe joint pain without osseous joint changes as determined with computed tomography (CT). The term “arthrosis” will be used to signify the presence of any osseous joint change, whether accompanied by joint pain (RDC/TMD osteoarthritis, IIIb) or without joint pain (RDC/TMD osteoarthrosis, IIIc).

Reference Standard (Gold Standard)

The reference-standard diagnoses were derived from the consensus diagnoses of 2 TMD experts who independently assessed all participants using the criterion examination protocol specified for this study. This criterion protocol incorporated a comprehensive history and examination, as well as imaging that included a panoramic radiograph, bilateral temporomandibular joint (TMJ) magnetic resonance imaging (MRI), and bilateral TMJ computed tomography (CT). All imaging was interpreted by a calibrated board-certified radiologist. These methods are described in detail in the first paper in this series.⁽¹⁴⁾

Clinical Examination Sequence

The formal validation of the RDC/TMD first required an assessment of each participant by a criterion examiner (CE) who performed the standardized criterion evaluation and ordered imaging that was completed at the second visit. At the third visit, within 1–2 weeks after visit 1, the TE completed the RDC/TMD examination protocol while blinded to the results of the first examination. This was followed on the same day by a second criterion examination performed by the second CE who was blinded to the results of the previous examinations. The reference-standard criterion diagnosis was then established by the 2 CEs coming together to determine their consensus diagnosis with the participant still present. The radiologist was also involved if the CEs’ interpretation of the TMJ MRIs or CTs differed from the radiologist’s interpretation. The radiologist rendered the final interpretation of the images following these consultations.

Primary Outcome Variables and Covariates

One of the primary outcomes for this project, and the primary focus of this report, is the validity assessment for the RDC/TMD Axis I TMD diagnoses. Validity was assessed in terms of the sensitivity and specificity of diagnostic outcomes based on the TE examination data collection. The diagnoses were not rendered directly by the TE, but instead were derived by applying the RDC/TMD algorithms to the examinations findings of the TE. Sensitivity and specificity of these diagnoses were determined relative to the reference-standard diagnoses rendered by the consensus of the 2 CEs. As is conventional, participants with a positive reference standard diagnosis for a given disorder were included in the sensitivity analyses and those with a negative reference standard diagnosis for a disorder were included in the specificity analyses.

We were interested not only in the sensitivity and specificity of the 8 RDC/TMD-specified diagnoses, but also 4 combinations of these diagnoses, including any Group I muscle pain (Ia or Ib), any Group II disc displacement (IIa, IIb, or IIc), any Group III joint pain (IIIa or IIIb), and any Group III arthrosis (IIIb or IIIc). The combined diagnoses for disc displacement, joint pain, and arthrosis were joint-specific within each participant.

Measurement error associated with the TE examination data collection was evaluated by comparing its agreement with the data collection performed the same day on the same study participant by the second criterion examiner. This comparison was facilitated because the criterion examination included all the examination items specified by the RDC/TMD, in addition to a much-expanded examination protocol described in the first paper in this series.⁽¹⁴⁾ Thus, the sensitivity and specificity of the RDC/TMD data collection that was embedded in the criterion data collection could be compared to the TE’s results.

The RDC/TMD diagnostic algorithms used for these analyses were published in 1992 as decision and classification tree models.⁽¹⁾ A classification tree is composed of nodes defined by a “split condition” that may consist of a single variable or a combination of variables. Each node in the tree is either satisfied or not satisfied using the data collection for a participant, thus allowing for this individual’s diagnostic status to be established by following out the tree to its terminal node. Such diagnostic structures are interpretable and intuitively consistent with theoretical constructs that describe the conditions, or disorders, being diagnosed.

Factors Influencing Diagnostic Sensitivity and Specificity

The planned analysis included reporting the observed variation across study sites in terms of their ranges of sensitivity and specificity estimates and any statistical differences among them, and the assessment of the influence on sensitivity and specificity estimates that might be associated with the number of concurrent TMD diagnoses, right- versus left-side differences within participants for Group II and III diagnoses, and other baseline variables of interest. The latter included age, gender, education,⁽¹⁾ income,⁽¹⁾ characteristic pain intensity,^(1,15) duration of TMD symptoms,⁽¹⁾ depression,^(1,16,17) nonspecific physical symptoms,^(16,17) pain-related disability,⁽¹⁵⁾ and current or recent treatment for TMDs. For purposes of this analysis, education with a range of 7 to 19 years was dichotomized by its median value, ≤ 15 years versus > 15. Income was dichotomized at its median value of $40,000, as was characteristic pain intensity (0–40 versus > 40), duration of TMD pain (≤ 72 months versus > 72 months), depression (0–0.3 versus > 0.3), and nonspecific physical symptoms (0–0.4 versus > 0.4). Pain-related disability was differentiated by Graded Chronic Pain Scale (GCPS) grades 0–II versus III–IV.⁽¹⁵⁾ Current treatment for TMDs was differentiated into 3 categories: none, during last 6 months, and prior to 6 months ago.

Validation Study Sample Size Calculations

Sample size requirements were determined as a function of the precision that was stipulated a priori for the sensitivity and specificity estimates in this project. The required precision specified that neither the upper or lower confidence bound should differ from the point estimate by more than 0.10. Assuming symmetrical confidence bounds, the half-width for each confidence interval is expressed as $2\sqrt{p(1-p)/N}$, where p is the estimated sensitivity or specificity, and N is the number of participants truly positive for a diagnosis as determined by the reference-standard diagnosis. This gave a conservative estimate for the precision for Group II and Group III diagnoses, because it assumed only one diagnosis per participant, whereas both joints in each participant were assessed for the validation study. Based on an observed sensitivity or specificity of 0.5 (when the binomial variance is the largest), and with the desired precision defined by upper and lower confidence bounds no greater than 0.10 for all sensitivity and specificity point estimates, 100 participants were required for each diagnosis.

This design resulted in the following recruitment methods:

• Participant recruitment was to include at least 100 cases for each of the 8 TMD diagnoses. Given that some participants would have multiple diagnoses for the various TMD subtypes, total case recruitment was planned to be 600. Initially, study participants were recruited consecutively with no attempt being made to selectively enrich the participant sample for the less common diagnoses (IIb, IIc, IIIb, IIIc). However, after three-fourths (approximately 550) of the participants had been enrolled, selective recruitment was implemented at all sites to ensure an adequate sample size for the less prevalent diagnoses.

• To rigorously test the RDC/TMD, we also planned to recruit 100 subthreshold cases, i.e., participants whose signs and symptoms did not qualify for an RDC/TMD diagnosis, but who nonetheless had at least 1 of the 3 cardinal symptoms or signs of TMDs: jaw pain, limited mouth opening, or TMJ noise. In addition, these participants were eligible if they had any of the additional TMD diagnoses listed in the Expanded TMD Taxonomy that was developed specifically for this study and reported in Table 3 in the first paper in this series.⁽¹⁴⁾

• Finally, we planned to recruit 100 controls, that is, participants without signs and symptoms of TMDs, to populate 4 age strata: 18 to 30, 31 to 40, 41 to 50, and 51 to 70 years of age. Equal distributions of controls in each age category was deemed unlikely, given the low prevalence of completely normal joints among older participants, but the goal was to recruit as many controls as possible in the 2 older age categories so as to be able to select a subset of control participants that reasonably matched the age distributions of the 8 TMD diagnostic groups. For example, we anticipated that the average age of participants with TMJ osteoarthrosis would be greater than that of participants without this diagnosis.

Statistical Procedures

The prevalence of the reference-standard diagnoses in the study sample and the overall percent agreement between the test and reference-standard diagnoses were computed. A logistic regression approach was used to estimate the sensitivity and specificity using generalized estimate equation (GEE) methodology to account for the multiple diagnoses per participant for Group II and Group III diagnoses when computing confidence intervals and statistical significance.⁽¹⁸⁾ To assess the influence of study site, number of concurrent TMD diagnoses, right versus left side differences for Group II and III diagnoses within participants, and other baseline variables, additional analyses were performed in which each covariate was added separately to the logistic regression model.

Cut-offs for Determinations of Validity

In the development process for the RDC/TMD, the desired goal for sensitivity was set at ≥ 0.70 and specificity at ≥ 0.95. In the original monograph, Dworkin showed that, when assuming a prevalence of 10% for TMDs, a positive predictive value of at least 0.75 for this diagnostic protocol would require specificity > 0.95 while sensitivity could be as low as 0.70.⁽¹⁾ Based on these expectations, we intended to declare as valid a given diagnosis -- either 1 of the 4 diagnostic groupings or 1 of the 8 subdiagnoses -- if its estimated sensitivity was at least 0.70 and its estimated specificity was at least 0.95. We planned to apply these validity thresholds to the estimated sensitivities and specificities, even though the lower 95% confidence bounds might include values that were less than the validity threshold.

Secondary Analyses

For the secondary analyses involving the 13 specified covariates, 16 tests were planned for each: 8 sensitivity estimates for the Axis I RDC/TMD diagnoses and 8 specificity estimates. No a priori hypotheses could be advanced for these tests based on previous studies. Given that these multiple analyses were purely exploratory, and with each covariate of independent interest, the P for significance was set at ≤ .005 to minimize Type I error, but also to allow for detection of associations of potential importance. All analyses were performed with SAS Version 9.1 statistical software (SAS Institute, Inc., Cary, NC, USA, 2003).

Statistical Effects of Covariates on Sensitivity and Specificity Estimates

The reason for an association between the female gender and lower specificity for Ib and IIIa pain diagnoses may relate to self-reported TMD pain in women being approximately twice that of men.⁽¹⁹⁾ We have not analyzed this finding further to see if other factors may have influenced these gender differences including longer pain duration, more sites of pain to palpation, higher characteristic pain intensity, depression, and nonspecific physical symptoms. These factors are related to higher sensitivity to noxious stimulation and may influence reporting of responses to physical pain measures. As for the impact of the number of concurrent diagnoses, the observed pattern for sensitivity and specificity estimates suggests that the rater was more likely to detect a true disc displacement (higher sensitivity) when more diagnoses were present. One might speculate that when a higher number of diagnoses are present, the participant would be more likely to have a Group II diagnosis, and that the higher diagnostic sensitivity for Group II could be a function of the increased opportunity for this diagnosis. We note, however, that sensitivity and specificity estimates are theoretically independent of the prevalence of a given condition,⁽²⁰⁾ and that the influence of the concurrent conditions on such higher sensitivity is not clear. The specificity for Ia, Ib, IIa, and IIIa was significantly greater in the presence of 2 or fewer diagnoses than when compared with 3 or more. The RDC/TMD was less likely to give a false-positive diagnosis when fewer diagnoses were present.

Duration of symptoms, characteristic pain intensity, and nonspecific physical symptoms all had a similar effect on the specificity of pain diagnoses (Ia, Ib and IIIa), with depression showing the same effect on specificity for Ib and IIIa. Lower levels were consistently associated with higher specificity. The same pattern, supported by a trend toward statistical significance, was observed for pain-related disability relative to its effect on specificity for IIIa. The higher levels of these covariates could simply be associated with more reports of pain to the TE vs. the CE’s. However, additional analyses to be completed may reveal more complex interactions associated with these Axis II covariates.

The statistical differences observed between study sites are more difficult to interpret. To render our estimates of validity more generalizable, we intentionally recruited participants from both coasts as well as the north-central area of the United States to obtain a more heterogeneous participant sample. Accordingly, we anticipated that there would be some differences in study populations recruited from widely separated locations across the United States. To study the site-specific differences in IIa and IIIa sensitivity estimates, preliminary adjusted analyses have been performed using a generalized estimate equation (GEE) multivariate model. When controlling for the effects of age, gender, number of concurrent diagnoses, left versus right side differences and recent treatment for TMD, the significance of site differences for IIa sensitivity did not remain (P > .005). In contrast, the significance of the site differences for IIIa sensitivity has been maintained throughout the analyses to date (P < .001). Our intention is to report the observed differences between sites at this time. A better understanding of these differences may be found after further data analyses using other regression models with different combinations of variables and assessment of their interactions. With respect to the observed site differences, we note the following:

• Using the RDC/TMD protocol, the diagnostic detection rate was low for the diagnoses IIa, IIb, IIIa and IIIc likely due to lack of imaging data in the RDC/TMD protocol. Low examiner detection rates due to the design of the diagnostic system result in greater variability for the estimates of agreement.

• Considerable variation has also been reported across the 10 sites in the International Consortium for RDC/TMD-based Research. The site-specific diagnostic agreement showed ICC ranges of 0.29–0.71 for IIa, 0.00–0.92 for IIb, 0.22–0.66 for IIIa, and 0.00–0.74 for IIIc.⁽³⁾

• None of the 3 sites for this study was consistently high or low as to diagnostic accuracy.

Generalizability of Study Estimates

The diagnostic validity estimates presented in this paper are both credible and generalizable. As reported in the first paper in this series, the demographics and characteristics of the study participants are comparable to other studies that have employed the RDC/TMD and appropriate for testing this examination protocol.⁽¹⁴⁾ The broad geographical distribution of the participants provides credibility for these estimates based on the heterogeneity of the study sample. In addition, the large sample size in this study made it the first to obtain adequately precise confidence intervals for its point estimates. Although this project was not designed to be a definitive assessment of the influence of covariates on sensitivity and specificity estimates, these findings should be viewed as preliminary data to be taken into consideration for future study designs. To the extent that these effects are not due to chance, small study samples may be more susceptible to chance-related perturbations that can affect estimates of validity for TMDs.

Validity Study Design Issues

Case definitions for musculoskeletal disorders are a set of diagnostic criteria that establish the boundaries by which a disorder can be considered to be present or absent. For TMDs, these criteria are determined by a clinical examination based on muscle and TMJ palpation, auscultation of the TMJ, and measurement of mandibular range of movement. This protocol may also be supplemented, as the 1992 RDC/TMD publication states, with imaging for detection of certain intra-articular disorders.⁽¹⁾ Some investigators may suggest that only an objective biological measurement can constitute a reference standard for the presence/absence of a disorder. In the absence of such an objective reference standard, however, the closest reference standard to date for TMDs, especially TMD pain, is the comprehensive clinical assessment that was specified for this study. The usefulness of a diagnostic protocol for TMDs that is necessarily based on clinical signs and symptoms is limited in the presence of competing diagnoses that share some or all of the same signs and symptoms (i.e., comorbid conditions such as fibromyalgia). Because this is the first study to comprehensively assess the TMD diagnoses with an adequate sample size, we followed Standards for Reporting of Diagnostic Accuracy (STARD) recommendations to compare participants presenting only with the target condition (TMDs) with healthy controls.⁽¹³⁾ We eliminated cases with potentially confounding comorbid conditions through specific exclusion criteria as reported in the first paper in this series.⁽¹⁴⁾

Parallels between Validity Testing in This Study and Other Areas of Medicine

Developing a credible reference standard to assess the construct of muscle and joint pain is challenging, given that pain is a subjective experience. Psychology and psychiatry have successfully dealt with this issue through structured interviews.⁽²¹⁾ In the case of fibromyalgia, the reference standard was a consensus diagnosis rendered by 2 rheumatologists using all available clinical tests. The resultant diagnostic criteria, however, were simplified to 11 of 18 points being reported by the participant as “tender” to palpation, with a concurrent self-report of widespread pain.⁽²²⁾ Other areas of medicine have used anesthetic blocks to make diagnostic decisions,^(23,24) although the placebo effect of blocks complicates their interpretation. Still other approaches have been to make diagnostic inference based on the results of interventions. Headache is one such disorder that is “confirmed” when a treatment mitigates the pain.⁽²⁵⁾ Such interventions can also have many nonspecific effects that limit the conclusions from these tests. Some diagnostic criteria for musculoskeletal pain include the construct of asking the participant if a pain-provocation test produces “familiar pain”.^{(23,24.26–32)} This method takes into consideration that many tests can produce pain, but that “familiar pain” allows the participant to identify the pain as reproducing the chief complaint. Other methods are based on the participant pointing to the location of his/her pain when location is informative in making a diagnosis. Because the criterion validity of the RDC/TMD Axis I diagnostic algorithms for myofascial pain and arthralgia had not been previously assessed, the reference standard specified in this project utilized many of these approaches.⁽¹⁴⁾ Relative to TMJ disc displacements and arthrosis, the standard in medicine for assessing intra-articular soft and hard tissue is MRI and CT, respectively. In the current project, the reference standard specified the use of these imaging techniques for assessing disc displacement and osteoarthritis, and required that experienced board-certified radiologists interpret these images.