Author's reply: The role of potential outcomes thinking in assessing mediation and interaction

I would like to thank the various commentators for their time and thoughts concerning my book ‘Explanation in Causal Inference: Methods for Mediation and Interaction’¹ and its accompanying synopsis.² Kaufman³ traces some of the history of the methodological developments on these topics; Oakes and Naimi⁴ discuss the relevance of the book’s content to social epidemiology; Pearce and Vandenbroucke⁵ raise various concerns principally about the scope of the book. Taken together, the commentaries provide a helpful overview of what the book does and does not accomplish, and of the extent and limits of the book’s scope.

There appear to be important tensions present when comparing the commentaries themselves. Oakes and Naimi speak of how important the phenomena of mediation, interaction, and spillover are within social epidemiology and how helpful the methods will be in assessing these; Pearce and Vandenbroucke, in contrast, repeatedly state that they believe the methods will only ever be applicable to a very narrow set of questions. Kaufman and Oakes and Naimi praise the book for its clarity of exposition of difficult ideas that will now be widely accessible; Pearce and Vandenbroucke state that it is too technical and esoteric (‘should not ‘try at home’’). Kaufman fears that the methods, having now been made accessible with easy-to-use software will result in an explosion of applications, some of which will be carried out without a thorough understanding of the assumptions and limitations. Pearce and Vandenbroucke believe that the methods ‘are unlikely to be applied routinely by researchers who use epidemiologic tools in their studies of public health or clinical problems.’ Some of the differences in perspective may reflect differences across the sub-disciplines of the commentators. Some of the differences may reflect something of a generational divide. Some are likely due to natural variation that is to be expected and experienced across different readers. As stated in the book’s preface, some of the material covered in the book is indeed challenging. I have attempted to make difficult concepts and methods as accessible as possible, but I acknowledged in the preface itself, my own limitations in my capacity to do so. Whether the methods are used, or prove useful within public health, and whether the book is found accessible or not, I suppose time will tell.

An important point touched upon by Kaufman and by Oakes and Naimi, but perhaps not entirely appreciated by Pearce and Vandenbroucke concerns the relation between the new methods for mediation analysis, and the traditional methods (sometimes referred to as the ‘difference method’, or the ‘Baron and Kenny method’,⁶ effectively fitting two separate regression models, alternately with and without the mediator). As noted in the book, and in my synopsis, the newer methods effectively subsume the more traditional ones. As noted by Kaufman, in the absence of interaction between the exposure and the mediator, the methods coincide. The assumptions (for example, about adequate control for mediator-outcome confounding, discussed by all commentators) also coincide. However there was often not an awareness of what those assumptions were in many applications of the traditional approaches. What the newer methods do then is relax the assumptions, about the absence of interaction, about functional forms, and, through sensitivity analysis,^7–15 about confounding and measurement error. Whenever the assumptions of the newer methods are not applicable, the assumptions of the traditional methods will not be either. This is an important point. It is not that the newer methods require stronger assumptions. It is that they point out the assumptions required by both the newer and the traditional methods (that have often been simply ignored when the traditional methods have been used). In the book and in my teaching I have emphasized the conditions under which the newer approaches are not needed and, in such cases, I have encouraged the use of the traditional approaches as they are easier to understand, interpret, and present. However, the advantages of the traditional approaches are simply ease of interpretation and use; they are not are making weaker assumptions nor are they sometimes applicable when the newer approaches are not. Again, whenever the newer methods are not applicable, the traditional methods will not be either.

Thus when Pearce and Vandenbroucke write that ‘In practice, very few studies will (even hypothetically) have data available of sufficient accuracy, and including sufficient variables, to enable us to use these methods with any confidence,’ this statement, contrary to what they seem to suggest elsewhere in their commentary, applies also to the traditional methods. In the context of mediation, it is effectively an abandonment of any attempt to statistically assess the importance of a mechanism. Indeed, in part because of the strengths of the assumptions required for mediation, this position of abandoning attempts at estimating direct and indirect effects has essentially been adopted by Donald Rubin.^1,16,17 I am not entirely unsympathetic to the view: the assumptions are strong. My own perspective, however, differs insofar as I have worked on at least a handful of examples in which I think we can draw at least reliable qualitative conclusions about the importance, or lack thereof, of a particular pathway.^1,18–20 What enables robust inference in these settings is sensitivity analysis to evaluate how conclusions might or might not be altered when assumptions are violated.

Pearce and Vandenbroucke repeatedly raise concerns about the scope of the book. They complain that it does not provide an overarching theory of causality, or of all types of scientific evidence, and of how to integrate this within epidemiology. However, it seems that they have misunderstood the intended scope of the text. The intended scope is relatively clearly laid out in the book’s preface: to address the topics of mediation and interaction from a counterfactual-based perspective. Much of their discussion is thus I think out of place. The book is not trying to present a grand theory of causality; the book is not intended as a comprehensive epidemiologic textbook; and the book is not intended as a broad philosophy of science. All of those undertakings would be important; material of that sort should be included in education; but none of this was the intent of the book. Their aphorism from Aquinas ‘Timeo hominem unius libri - I fear a man of one book’ is thus also out of place as a critique of the book.

They claim the book’s scope ought to have been broader on the grounds of its title: ‘Explanation in Causal Inference: Methods for Mediation and Interaction.’ They claim otherwise the book ought to have been titled: ‘Statistical Methods for Mediation Analysis and Interaction Assessment in Observational Studies which involve Exposures that are Potentially Manipulable.’ I do wonder what Oxford University Press’ marketing department would make of their proposal. Titles of books are often more pithy; one need only turn to the book’s preface for clarification of scope. I had in fact initially proposed as the title the even more compact ‘Methods for Mediation and Interaction’. Perhaps Pearce and Vandenbroucke would have thought this more accurate. One of the book’s initial reviewers suggested the expanded title on the grounds of placing the methods in the context of the proposed purpose of it all, and this indeed seemed reasonable; but again the second part of the title ‘Methods for Mediation and Interaction’, clarifying the scope, remained. What Pearce and Vandenbroucke would like to see, it seems, is a text that integrates different views of causality, comprehensive epidemiologic theory, and different forms of evidence, including animal models and qualitative research. I agree this would be desirable. It is an ambitious undertaking. It is not the task I set myself to undertake. Pearce and Vandenbroucke are better positioned to write such a text than I, and I very much hope that they will do so, and move beyond complaints that others have not supplied what they desire.

With regard to their comments on causality, Pearce and Vandenbroucke have, as also elsewhere,²¹ mis-represented my views, suggesting that I take the position that we can only speak of causes when something is subject to human manipulation. This is not a position I hold. Such a view is also contradicted in my book in my providing, in section 16.2, what I take to be a set of sufficient conditions for attributing causation or the truth of counterfactual claims, applicable even in the absence of human intervention (p. 450, 453, 454). Remarkably, this is in the very section from which they cherry pick their quotations to seemingly attribute to me a view I do not hold. It is not clear to me whether Pearce and Vandenbroucke have not carefully read the relevant section or have simply intentionally falsely represented my views. This was pointed out in a previous exchange,²² and still they persist in misrepresentation. I consider this poor scholarly practice in general and especially so for those of the intellectual stature of Pearce and Vandenbroucke.

Pearce and Vandenbroucke do raise important issues of the place of the potential outcomes framework within epidemiology. The potential outcomes approach is certainly only one tool among many that are important within epidemiology. There has been an entire discussion, to be published in the International Journal of Epidemiology, on the role and limits of potential outcomes in epidemiology. Both they and I have contributed to that exchange and so I will refer the interested reader to some of the relevant papers and commentaries^21–26 and will not repeat here my own views on the matter. I do think it is an important discussion. I do not think that there are easy answers. Apparently, my book offered another occasion for them to raise these issues again.

Pearce and Vandenbroucke do raise some important, more specific, points about the actual intended content of the text. One concerns measurement and misclassification. Even if the modern methods of mediation are applicable, they will only give estimates concerning the mechanistic importance of the actual measured construct. If smoking intensity is measured by average cigarettes per day then the mediated effect estimate, even if measurement error correction methods are employed, will only be with respect to average cigarettes smoking per day, not to other aspects of smoking intensity, such as depth of inhalation. This is an important point. It is acknowledged in the book (p. 45, 235), but it probably ought to have been given greater emphasis.

Pearce and Vandenbroucke suggest that problems of unmeasured mediator-outcome confounding may only generate substantial bias in quite extreme situations and question whether this really occurs in practice. Section 3.4 of the book gives a dramatic example of such bias in a randomized trial of cognitive behavioral therapy, dramatic bias that emerges if such issues are ignored in assessing direct and indirect effects. In the example, the qualitative conclusions are entirely incorrect due to such biases. Ignoring such mediator-outcome confounding has been a central problem in the assessment of mechanisms in epidemiology and in the social sciences. Whether the comments of Pearce and Vandenbroucke were an attempt to justify past practice that had ignored the issue I do not know, but this point on needing to control for mediator-outcome confounding when assessing mediation is what I have tried to most centrally emphasize in my writing and teaching. Ignoring it imperils scientific assessment of mechanisms with epidemiologic data.

This example also raises another important point. In their proposed alternative book title, ‘Statistical Methods for Mediation Analysis and Interaction Assessment in Observational Studies which involve Exposures that are Potentially Manipulable,’ there appears to be the suggestion that the methods are relevant only in the context of observational studies. This is not so. The methods are relevant in the context of randomized trials as well if questions of mediation and mechanisms are of interest. Moreover, when assessing mediation, the same problems of mediator-outcome confounding can plague estimates of direct and indirect effects as in observational studies, as clearly illustrated in the book in the randomized cognitive behavioral therapy example. In a randomized trial, the exposure has been randomized, but the mediator generally has not been.

Pearce and Vandenbroucke criticize the book for what they claim as placing more emphasis on the potential problems and assumptions of Mendelian randomization than with mediation analysis. I find this surprising. The strong assumptions required to carry out mediation analysis are pointed out repeatedly, and, as noted in the commentary by Kaufman, there is continual emphasis on the need to employ sensitivity analysis techniques to evaluate the robustness of conclusions to the assumptions. My hope, and one of the goals of the book, is to see mediation analysis, when it is carried out, done more rigorously. I would like to see the same with Mendelian randomization analyses. My issues are not with the idea of Mendelian randomization (I have seen some very fine applications), but with its typical use in practice. In actual application, the assumptions are sometimes not stated; when they are stated, there is often little discussion of their plausibility or as to what potential violations might be (such as those documented in the book); until recently, there has been little effort to carry out sensitivity analysis to the assumptions. Some of these same criticisms of course hold true with the use of mediation analysis techniques. All of this I would like to see changed. The book itself provides a number of tools for sensitivity analysis for mediation; the book also proposes some sensitivity analysis for Mendelian randomization.^1,27 A more recent proposal, subsequent to the book’s publication, for sensitivity analysis in Mendelian randomization, ²⁸ I believe holds considerable promise for improving the quality and the robustness of inferences made using Mendelian randomization approaches and has somewhat altered my outlook on the value of the approach. However, I still want to see assumptions stated, discussed in the context of the application, and potential violations explored through sensitivity analysis. I think that this needs to happen both for mediation analysis and for Mendelian randomization analyses. I do not see things differently across these two methodological approaches. The potential outcomes framework has helped clarify the assumptions underlying both.^1,29

More generally, the potential outcomes framework has brought tremendous clarity as to how to go about assessing more nuanced types of causal effects^1,30–43, including those related to mechanisms with observational and experimental data, and under what assumptions such inferences are warranted. I agree with Pearce and Vandenbroucke that the framework does not address, and is not particularly helpful, for answering numerous other important questions. But with regard to assessing mechanisms from epidemiologic data, I see no reason to return to unclear thinking, less accurate inferences, and the ignoring of assumptions. Progress has been made. One can debate how central or important this progress is to the overall project of epidemiology, but the book was intended only to address one particular subset of questions within the field, and to do this better than was done in the past.

My goal was to take a few topics in epidemiology – mediation, interaction, and spillover effects – that are often discussed but for which concepts and methodology had been inadequate, and provide a more formal foundation for these. Considered within the entire scope of epidemiology, it is a modest undertaking. It was not in any way intended to provide a comprehensive theory for the field, or for causation. Whether I have achieved my more modest goals, the reader can judge and, as I wrote in the synopsis², how useful any of this will be for the field depends on the extent to which the epidemiologic questions of greatest significance truly require quantitative assessment of mediation and interaction to address. Oakes and Naimi think often they will; in contrast, Pearce and Vandenbroucke believe that their usefulness will ‘often be limited.’ My hope is simply that when such mechanistic assessments are useful, they will now be done more carefully and more accurately.