Figures
Citation: Moher D, Schulz KF, Simera I, Altman DG (2010) Guidance for Developers of Health Research Reporting Guidelines. PLoS Med 7(2): e1000217. https://doi.org/10.1371/journal.pmed.1000217
Published: February 16, 2010
Copyright: © 2010 Moher et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: No specific funding was received to write this piece. DGA is supported by Cancer Research UK, DM by a University of Ottawa Research Chair, and KFS by Family Health International.
Competing interests: Competing interests: DM, KFS, IS, and DGA are members of the EQUATOR Network. DM, KFS, and DGA are members of the CONSORT executive. DM and DGA are members of the PRISMA executive. DM is on the Editorial Board of PLoS Medicine.
Abbreviations: CONSORT, CONsolidated Standards Of Reporting Trials; CONSORT for NPT, CONsolidated Standards Of Reporting Trials for Non-Pharmacological Treatment interventions; E&E, Explanation and Elaboration; EQUATOR, Enhancing the QUAlity and Transparency Of health Research; PRISMA, Preferred Reporting Items for Systematic reviews and Meta-Analyses; QUOROM, QUality Of Reporting Of Meta-analyses; RCT, randomized controlled trial; SPIRIT, Standard Protocol Items for RandomIzed Trials; STARD, STAndards for Reporting of Diagnostic accuracy; STREGA, STrengthening the REporting of Genetic Association Studies; STRICTA, STandards for Reporting Interventions in Controlled Trials of Acupuncture; STROBE, STrengthening the Reporting of OBservational studies in Epidemiology
Provenance: Not commissioned; externally peer reviewed.
Introduction
Publishing health research is a thriving, and increasing, enterprise. On any given month about 63,000 new articles are indexed in PubMed, the United States National Library of Medicine's public access portal for health-related publications. However, the quality of reporting in most health care journals remains inadequate. Glasziou and colleagues [1] assessed descriptions of given treatments in 80 trials and systematic reviews for which summaries were published during one year (October 2005 to October 2006) in Evidence-Based Medicine, a journal that is aimed at physicians working in primary care and general medicine. Treatment descriptions were inadequate in 41 of the original published articles, which made their use in clinical practice difficult if not impossible to replicate. This is just one of numerous examples of a large and disturbing literature indicating the general failure in the quality of reporting health research [2]–[6]. Many publications lack clarity, transparency, and completeness in how the authors actually carried out their research.
Inadequate reporting is problematic for several reasons. If authors do not provide sufficient details concerning the conduct of their study, readers are left with an incomplete picture of what was done. As such, they are not able to judge the reliability of the results and interpret them. There are also ethical and moral reasons for reporting research adequately [7].
The EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network is a new international initiative seeking to improve the quality of scientific publications by promoting transparent and accurate reporting [8]. The Network (http://www.equator-network.org) provides resources and training relating to the reporting of health research and assists in the development, dissemination, and implementation of reporting guidelines. As part of its initial resource development, the Network's Web site contains a comprehensive and up-to-date database of reporting guidelines relevant to heath research. A recent systematic review of 81 reporting guidelines found their development was often inadequate [9].
Reporting guidelines need to be differentiated from other efforts that produce a checklist or other guidance not specific to reporting research. We propose here a working definition of a reporting guideline: a checklist, flow diagram, or explicit text to guide authors in reporting a specific type of research, developed using explicit methodology. Some reporting guidelines recommend a flow diagram so that authors can clearly report information about sequential stages of their research project. A consensus process [10] should be a crucial characteristic of developing a reporting guideline.
The main motivation for the development of reporting guidelines is to help researchers improve the completeness and transparency of their research reports and limit the number of poorly reported studies. However, reporting guidelines can be also used by peer reviewers and editors to strengthen manuscript review. And research funders can benefit from introducing reporting guidelines into the research application system [11]. Ensuring clear and complete reporting of funded research through the use of reporting guidelines should facilitate more efficient use of the new findings and bring better returns on research investments. There are enormous potential benefits of good reporting. However, despite the impressive recent upsurge in the number and range of reporting guidelines, the literature on how individual guidelines were developed remains sparse [12],[13] and there is no generic guidance on how to develop one.
In this paper we update and expand upon an earlier effort to outline a strategy for developing reporting guidelines that was published only in Spanish [14]. We recognize that there is no single best or correct approach. However, this paper benefits from our collective experiences of helping to develop more than ten reporting guidelines over the last 16 years, over which period these ideas have evolved considerably. If reporting guidelines are to be useful and more widely disseminated, they need to be developed using robust and widely accepted methodologies.
This strategy assumes the involvement of an executive group to facilitate the guideline development and the expectation of having a face-to-face meeting as part of the reporting guideline development. We propose 18 steps to occur in five phases, which are outlined in Table 1.
Initial Steps
1. Identify the Need for a Guideline
Developing a reporting guideline is complex and time consuming, so a compelling rationale is needed. Most reporting guidelines have been developed because researchers are convinced of the need to improve the quality of reporting of a certain type of health research. For some study aspects there may be direct evidence that inadequate reporting is associated with biased reports or harmful consequences. At this early stage, the executive group needs to set out clearly and explicitly their objectives and consider the scope of recommendations. For example, an early decision of the STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) group was to restrict consideration to three main epidemiological study designs [15], leaving the way open to extension guidelines dealing with other designs, such as STREGA (STrengthening the REporting of Genetic Association Studies) [16].
Reporting guideline developers also need to consider whether a need exists to develop a new reporting guideline or to extend or implement an existing guideline.
1.1. Develop new guidance.
Developing a new reporting guideline assumes that there is no existing guideline on the topic under consideration. The development of STARD (STAndards for Reporting of Diagnostic accuracy), a reporting guideline for diagnostic accuracy studies, was undertaken because no guidance in this area existed previously and evidence suggested the need for one [17].
1.2. Extend existing guidance.
Sometimes there may be the view that an existing broad guideline can usefully be augmented by additional guidance for a specific set of studies. An example of such an extension is the recently published CONSORT (CONsolidated Standards Of Reporting Trials) for nonpharmacological treatments (CONSORT for NPT) [18]. Here we felt that there were specific issues for some types of trials (e.g., surgical) for which sufficient guidance was not offered in the original CONSORT publications. The NPT extension still considers the original 22-item CONSORT checklist as a minimum set of criteria to consider when reporting a randomized controlled trial (RCT), but it extends 11 of these items for further consideration when reporting a trial that has used an NPT intervention.
1.3. Implement existing guidance.
Most guidelines are defined by the study aims and methodology, leaving plenty of scope to illustrate their implementation in particular medical specialties. For example, recently a group implemented the CONSORT Statement for behavioral medicine [19], although this implementation was not done in collaboration with the CONSORT Group. Such implementations will generally include requests for additional topic-specific information. The distinction between an extension and an implementation can be unclear.
2. Review the Literature
2.1. Identify previous relevant guidance.
At this early stage, prospective guideline developers should search for any existing reporting guideline covering all or part of the area being considered. We developed a search strategy to identify and characterize reporting guidelines as part of an ongoing project. In addition to electronic searching, developers are encouraged to search the EQUATOR Network database of reporting guidelines (http://www.equator-network.org). We recommend that reporting guideline developers interested in extending or implementing an existing guideline contact the authors and discuss their plans.
2.2. Seek relevant evidence on the quality of reporting in published research articles.
We recommend searching for relevant evidence on the quality of reporting of published research articles within the domain of interest. Such reports provide an initial and important insight into the items to consider for inclusion in an eventual checklist and potential stakeholders to invite to a meeting. However, this literature can be elusive and appear in any journal. A more thorough process, such as a systematic review, would be useful here. The existence of such literature can be most readily ascertained by conducting comprehensive electronic searches of several databases. In preparing for the 2005 QUOROM (QUality Of Reporting Of Meta-analyses) meeting, among other activities we conducted a systematic review of studies that had reviewed the quality of reporting of systematic reviews. To help inform updating of the CONSORT Statement we search the literature every month for new studies of the reporting of randomized trials.
2.3. Identify key information related to the potential sources of bias in relevant studies.
For any guideline, both new and revised, there will be key pieces of information that must be included. For example, in a 1993 meeting that ultimately led to the development of the CONSORT Statement, we identified emerging empirical data on the importance of the methods and reporting of allocation concealment when describing the randomization process [20]. The 2010 update of the CONSORT Statement includes new checklist items on identifying any changes to trial outcomes after the trial commenced, and on trial registration. The new item on outcomes follows from empirical evidence [21] indicating that authors frequently do not provide analyses of outcomes in their published papers that were the pre-specified outcomes in their protocols (i.e., selective outcome reporting). The addition of a new trial registration item reflects current journal practice [22] stemming from some researchers hiding results of randomized trials [23].
3. Obtain Funding for the Guideline Initiative
To date, no published data inform the costs required to develop a reporting guideline. Even if funding is available, most developers limit their fiscal requests to cover only the main reporting guideline meeting. In our experience, considerable resources are required for pre-meeting activities, such as literature searching and conducting a Delphi exercise, and post-meeting activities, including the need for a small writing group to be able to meet in person, perhaps on several occasions, during the drafting of the guidance and development of an explanation and elaboration (E&E) document.
Depending on the extent of the pre-meeting activities we recommend allocating at least Can$10,000 for them. In our experience the main meeting costs are approximately Can$75,000. This covers the travel and accommodation costs of bringing together 20 to 30 participants. For new guidance in the form of a checklist, with a more detailed and extensive E&E, we recommend at least Can$35,000 for a small writing group to meet several times.
There is no obvious choice as to where reporting guideline developers can seek funding; we are unaware of any granting agency or other group that has a specific remit to provide funding to those interested in improving ways to report health research. Of the 30 respondents to our survey [24], 47% had received funds from a non-profit agency, 17% from the pharmaceutical industry, and 6% from government. We recommend that developers seek funds from all of these sources.
Pre-meeting Activities
4. Identify Participants
Most reporting guidelines have been developed by an international multidisciplinary group involving 22 (median) people [24], although not all may participate in meetings. The expertise of these individuals should reflect the particular guidance under consideration; participants will usually include statisticians, epidemiologists, methodologists, content experts, journal editors, and perhaps consumer representatives. The proportion of content experts needs to be at least a quarter, and perhaps larger depending on the content area under consideration. When developing the herbal extension of the CONSORT Statement we included pharmacognosy experts to provide input on several issues pertinent to reporting herbal interventions.
Well ahead of the proposed meeting date, a list of participants to invite should be developed. The CONSORT executive has typically invited participants in a two-stage process. First, we identify a small group of invitees whose participation we consider essential to hold a meeting. We invite this small group immediately. Once they have accepted our invitation and a date has been settled we invite the remaining participants. We also keep an additional smaller list of people to consider inviting should some second-wave invitees not be able to participate in the meeting. We recommend sending out the meeting invitations well in advance, ideally six months before the meeting date.
5. Conduct a Delphi Exercise
Not all potential meeting participants can be invited or will be able to come to the main guideline development meeting. These people can still be engaged in the guideline development process. The CONSORT group has started using a Delphi consensus method [25] to achieve this goal. The Delphi method is a structured process of obtaining information from a group of experts by means of a series of questionnaires, each one refined based on the feedback from respondents on a previous version [10].
To help develop CONSORT for RCT abstracts, we used a three-stage Delphi process. Journal editors, health care professionals, methodologists, clinical trialists, and others with expertise in the reporting of RCTs (n = 109) who were known to have an interest in the reporting of randomized trials, the structure of abstracts, or both were invited by e-mail to participate in a Web-based survey and rate the importance of 27 suggested checklist items selected from previous research. During three rounds of the survey, participants were asked about their views on the relative importance of the possible checklist items [25].
6. Generate a List of Items for Consideration at the Face-to-Face Meeting
There is no best way to generate the list of items for consideration, which will likely come from a number of sources, such as the Delphi process discussed above (see item 5). For example, the Delphi process for the SPIRIT (Standard Protocol Items for RandomIzed Trials) meeting resulted in 63 nominated items to consider for inclusion in the checklist discussed during the face-to-face consensus meeting. The executive may reduce the initial list of items to a more manageable number for discussion during the face-to-face meeting (see item 8).
7. Prepare for the Meeting
7.1. Decide size and duration of the face-to-face meeting.
Regardless of the available funding we recommend keeping the size of the meeting to no more than about 30 participants. Larger meetings can lose their spontaneity and limit interaction among the participants, both important attributes of a successful meeting. The CONSORT for herbal medicines meeting was small, involving 16 participants. In contrast, the CONSORT for NPT meeting was large, at 32 participants. Here we were interested in many types of interventions, such as surgery and psychotherapy, requiring a broader number of clinical content experts. It is valuable to have several people who have participated in previous similar meetings.
Reporting guideline meetings that we have been involved in have lasted from one to three days. We recommend a minimum of one day for developing a reporting guideline regardless of whether the guidance is new, an implementation, or an extension.
7.2. Develop meeting logistics.
The successful planning and implementation of any reporting guideline meeting requires thorough organization. If funding permits a coordinator should be hired to organize the meeting, including venue selection, meal plans, participant travel and accommodation, and finances.
7.3. Develop meeting agenda.
The executive must develop the form and structure of the meeting to ensure adequate time to discuss all of the agenda items. The most important outcome of the face-to-face meeting is an early version of the guidance checklist. We recommend that the following points be considered as part of developing the meeting agenda.
7.3.1. Consider presentations on relevant background topics, including summary of evidence. A useful way to set the stage and maximize participant dialogue is to devote at least a few hours to presentations on topics underpinning the reporting guideline development. Presentations might also be usefully devoted to specific items that are being proposed for the checklist and any relevant empirical evidence.
Ideally, by the conclusion of this session all participants should have up-to-date knowledge about the quality of reporting of the literature at which the guidance is aimed and the evidence relevant to considering the merits of including specific checklist items. Some of this material can be circulated in advance, but it is unlikely that people have time to read extensive materials. A major advantage of the Delphi exercise is that participants can think about some key issues before the meeting.
7.3.2. Plan to share results of Delphi exercise, if done. We have found that a Delphi exercise provides important information when developing a reporting guideline, regardless of whether the guidance is new, an extension, or implementation. The results of a Delphi exercise were presented to participants during the first day of the following three meetings: CONSORT for abstracts, CONSORT for NPT, and SPIRIT.
7.3.3. Invite session chairs. The main meeting will most likely be divided into sessions. While some of the chairs of sessions should be members of the executive group, other participants should also be invited to chair sessions, particularly if they have previous experience developing reporting guidelines and/or chairing meetings. It is essential that chairs have known ability to handle sessions effectively, to ensure that decisions are made.
7.4. Prepare materials to be sent to participants prior to meeting.
We recommend sending some materials to participants before the meeting, even though the same materials, and additional ones, should be readily available for each participant during the meeting. Sending the meeting agenda, participant list, one or two papers that might best highlight the quality of reporting of the content area, and the results of any Delphi exercise, if done, is a useful minimum and should be sent to the participants at least a week ahead of the meeting.
7.5. Arrange to record the meeting.
The options include audio (and possibly visual) recording of the entire event (this has proved valuable), hiring someone to take comprehensive minutes, or a combination of both. Depending on the meeting agenda, a more focused recording and/or minuting of certain parts of the meeting is another option. As a minimum we recommend comprehensive minuting of all discussions specifically related to the checklist development. Such minuting provides all participants with a record of events and decisions taken during the meeting.
The Face-to-Face Consensus Meeting Itself
The meeting should follow closely the pre-meeting plans, although timings should be flexible. It is unlikely that all the participants will know each other, so it is helpful for everyone to introduce themselves and indicate the relevance of their particular experience. One of the first tasks of the meeting is to review the objectives and the way the meeting will run, and to clarify any outstanding issues among the participants.
8. Present and Discuss Results of Pre-meeting Activities and Relevant Evidence
The substantive meeting begins with any formal presentations of background topics, empirical evidence from the literature, and results of any Delphi process (see item 5 above).
8.1. Discuss the rationale for including items in the checklist.
The most detailed and structured discussions at the meeting revolve around which checklist items to include in the guideline; these discussions should focus on information content and not get distracted by seeking agreed wording at this stage. We have always considered the items included in a checklist to be a minimum essential set of items that should be reported, and some discussion of this perspective is valuable before the detailed discussions begin.
The inclusion of each item is ideally supported by empirical evidence, when available. For example, the inclusion of allocation concealment as a checklist item of the original CONSORT Statement was based on an empirical study [20]. There are few such cases. More often there is a consensus that the information is methodologically important to assess in a study; there may also be good evidence that it is frequently not reported. Similarly, evidence and/or conceptual importance in one domain may be transferable from one guideline to another. For example, a specific checklist item on trial registration has been added to the CONSORT 2010 Statement. Similarly, during the 2005 QUOROM meeting we agreed to include a checklist item requesting that authors provide registration information about their systematic review, if available [26]. There may be other reasons for a focused discussion about the inclusion of an item. For example, the CONSORT Statement requests that the study be identified in the title as a randomized trial to aid searching for such studies.
The views expressed in a Delphi exercise, and perhaps also how similar issues were handled in other reporting guidelines, are also important. Ultimately the views of the meeting participants will usually converge, although it may occasionally be necessary to vote on some issues. We recommend considering a classification scheme for selecting items for inclusion in the checklist; we provide an example of this approach that we've used for including items in the CONSORT checklist (see Table 2).
8.2. Discuss the development of a flow diagram.
The majority of reporting guidelines have been limited to developing a checklist [24]; a few groups have also developed a flow diagram, the most well known of which is the CONSORT diagram. We recommend that the meeting agenda include discussion of the possibility of developing a diagram and, if appropriate, consideration of its content.
8.3. Discuss strategy for producing documents; identify who will be involved in which activities; discuss authorship.
The most important deliverable is the final reporting guideline. Commonly this will be in the form of a journal article, written after the meeting that introduces the checklist (and flow diagram) and summarises the processes used to develop it. Some groups such as CONSORT refer to this document as a “Statement” to distinguish it from other types of publication. A detailed explanatory paper, an E&E, may also need consideration. Sufficient time needs to be included in the meeting agenda to discuss these activities. Developing an E&E is very time consuming, and will require further input from the meeting participants (see item 10). To accomplish this task we have often asked meeting participants, all of whom would be considered for authorship, to volunteer to help draft particular sections. Time needs to be set aside to discuss these issues during the meeting and the authorship model needs to be agreed upon.
8.4. Discuss knowledge translation strategy.
One of the last major sessions of the meeting should be devoted to issues pertaining to disseminating the reporting guideline. There are several points to consider here, especially a publication strategy. Since the simultaneous publication in three journals of the 2001 CONSORT Statement, several reporting guidelines have been published in multiple journals. Ultimately reporting guideline developers want to positively influence the reporting of health research and will need to consider how best to ask journals and editorial groups (such as the International Committee of Medical Journal Editors), for help achieving these goals. To address these issues adequately the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) Group formed a three-member dissemination committee. How to maximize journal endorsement and adherence to the reporting guideline can also be discussed during the meeting. Additional points for discussion can include: whether and how the guidance will be formally evaluated; how any criticism will be handled; and whether to create a Web site and what it might contain.
Post-meeting Activities
Together with drafting and finalizing guideline documents the reporting guideline developers need to consider their implementation strategy, including publication issues (open access, copyright, peer review, and multiple and possibly simultaneous publications), website development, and seeking or monitoring endorsement of and adherence to the reporting guideline.
9. Develop the Guidance Statement
We recommend that initial efforts be focused on drafting the checklist for the proposed reporting guideline. Efforts aimed at extending or implementing existing reporting guidelines should make very clear which parts of their checklist remain the same as the original and which items have been modified or added. The process of developing the checklist is very likely to require several iterations. Initially, this entails taking the discussions from the face-to-face meeting (item 8.1 above) and crafting each item into a crisply and unambiguously worded checklist item. The most appropriate order for the checklist items also needs consideration.
For a new reporting guideline we recommend a short document of about 2,000 words reporting on the rationale for developing the guidance and the development process, including a brief description of the meeting and participants involved. The article will include the checklist and flow diagram, if developed. In our experience drafting the checklist and article is best done by a small writing group made up primarily of members of the executive team and perhaps one or two others. This group will usually constitute the guideline authors on behalf of the reporting guideline group, although group authorship may be preferred. For extending an existing reporting guideline a similar approach is recommended. Whether a new guideline is being developed or an existing one is being extended, the full guideline group will need to sign off on the document's content prior to journal submission.
9.1. Pilot test the checklist.
Pilot testing the checklist and flow diagram is worth considering. During the development of the QUOROM Statement, two members of the executive group independently asked two separate sets of students taking a systematic review course for their feedback on the checklist that was used to help report their systematic review. Their comments were incorporated into the checklist revisions.
10. Develop an Explanatory Document (E&E)
To date most reporting guidelines have not been accompanied by a detailed justification and explanation of the recommendations (e.g., STRICTA [STandards for Reporting Interventions in Controlled Trials of Acupuncture]). However, we believe that it is vitally important to provide an explanation of the rationale and evidence for a guideline item's inclusion and an elaboration on the details of an item, hence the title Explanation and Elaboration, or E&E, adopted by CONSORT and several other groups.
While there will undoubtedly be an urge to complete all of the guideline revisions rapidly and submit the paper for publication consideration, we recommend holding off doing so until an E&E has been prepared (see Box 1). The E&E will be considerably longer than the guideline statement (probably 10,000–20,000 words). As with the short guideline statement (see item 13) the full guideline group will need to sign off on the E&E's content before journal submission.
Box 1. The Rationale for Developing an Explanatory Document (E&E) to Support a Reporting Guideline
A reporting guideline gives a set of recommendations regarding the information that should be included in the report of a research study. It may also include guidance on how some information may best be presented (e.g., in a Table, or as absolute numbers).
Guideline developers aim to persuade editors and authors of the importance of adhering to their recommendations. To that end, simply providing a list of requirements, often as a checklist, is unlikely to be sufficient. Such a terse presentation may seem to make unsupported, even dogmatic statements without clear support, even when supported by relevant references to other publications. That concern underpinned the decision of the CONSORT Group to accompany the revised CONSORT Statement [44] with a detailed explanatory document, which was named “Explanation and Elaboration” (E&E) [45].
The E&E was intended to provide detailed rationales for all of the items in the CONSORT checklist and flow diagram. For each item, the paper included (a) an example of good reporting from a published paper, (b) the scientific background and rationale for including that information in a published article, (c) empirical evidence of bias associated with the way that aspect of a study is conducted or reported, and (d) any evidence relating to the extent of inadequate reporting of that information. In addition, some fundamental concepts that underpinned several of the items were discussed in boxes.
The novel format of the CONSORT E&E was emulated for several later reporting guidelines—STARD, STROBE, and PRISMA [46]–[48]. The broad format was retained with little modification, although the later examples have included rather more explanatory boxes; for example, the STROBE paper has eight boxes, including those addressing missing data, bias, interaction (effect modification), and how to use the paper. For some guidelines, notably some extensions of the CONSORT Statement, the explanatory information has been included with the reporting recommendations in a single article (e.g., CONSORT cluster [49], NPT [18]).
We feel that simultaneous publication of the statement along with an E&E is the most important way to disseminate the work of the reporting guideline group. For limited extensions a single article may best cover both roles, as was done, for example, for CONSORT for herbal interventions [27] and CONSORT for NPT [18].
11. Develop a Publication Strategy
We recommend that guideline developers negotiate a copyright agreement to retain the rights of the contents of the statement and related documents, such as the E&E. We have not found journals to be resistant to this idea. Similarly, we recommend developers to negotiate being able to put the reporting guideline on a dedicated website.
11.1. Consider multiple and simultaneous publications.
Starting with the revised CONSORT Statement in 2001, several developers have published their guidance in multiple journals simultaneously to enhance the uptake and dissemination of their reporting guideline (STARD, STROBE, STREGA, PRISMA). For example, the STARD Statement for reporting diagnostic accuracy studies is relevant to a broad spectrum of potential users, from clinical chemists to radiologists, all of whom might not be aware of each other's literature. Publishing in a radiology journal [28] and a clinical chemistry journal [29] likely increased the reach and potential influence of the guidance.
Our experience is that the journal submission and peer review processes for multiple simultaneous publications can be exhausting. Some steps can reduce the burden. For example, the guideline author group might want to nominate an author as corresponding author to facilitate the coordination process with the various journals. Before submission it might be useful for the reporting guideline authors to discuss with prospective journals their interest in publishing the guidance and the need for all journals to agree a common text. When submitting the final manuscript the journals can be asked about their willingness to have a common set of peer reviewers. Such a move enables the reporting guideline developers to respond to a single set of peer reviewers rather than separately to peer reviewers from each journal. Similarly, to ensure consistency across journals we believe that copy editing should be limited to spelling, layout, and punctuation, ensuring identical text across journals. This might be best achieved by having one lead journal take care of these issues and circulating the copy edited version across the other journals publishing the article. The PRISMA Statement, recently published in five journals [30]–[34], used these approaches; one journal volunteered to coordinate the whole editorial process and the other journals used their peer reviews and copyediting, thus making it more efficient for the developers and journals alike.
Post-publication Activities
12. Seek and Deal with Feedback and Criticism
It is important to seek feedback and criticism from all stakeholders regarding the reporting guideline, and we have outlined several stages by which this can be achieved in the development process. We recommend that guideline developers also encourage feedback at any time after publication, either directly or via the related Web site. Constructive criticism can help improve the reporting guideline if an update is prepared (see item 18, below). After the CONSORT Statement was originally published, Meinert [35] provided valuable suggestions on ways to improve the flow diagram and the clarity of some of the terminology initially used. These recommendations were incorporated into the 2001 revision of the CONSORT Statement.
13. Encourage Guideline Endorsement
Some journals, those whose editors initially were involved in the guideline development, will be eager to support the use of the reporting guideline. This support can be most readily achieved when a journal endorses the reporting guideline. However, some journals have used inconsistent language to describe their endorsement [36]–[38], and this vagueness likely diminishes serious efforts to improve the quality of reporting within journals.
When journals want to endorse a reporting guideline we recommend that they use strong, clear language of their expectations of authors, and include this information in their Instructions to Authors. For example, BioMed Central, the publisher of 199 open access journals, states, “We recommend authors refer to the EQUATOR network website for further information on the available reporting guidelines for health research, and the MIBBI Portal for prescriptive checklists for reporting biological and biomedical research where applicable. Authors are requested to make use of these when drafting their manuscript and peer reviewers will also be asked to refer to these checklists when evaluating these studies” (http://www.biomedcentral.com/bmcdermatol/ifora/).
Reporting guideline developers might also consider developing some brief text to help journals introduce their guidance to authors, incorporating the issues discussed above. We also recommend that when a journal endorses a reporting guideline they notify the guidance developers. This will help the developers to document and track all endorsements.
14. Support Adherence to the Guideline
Adherence is not part of developing a reporting guideline. However, it is central to whether reporting guidelines have their intended impact. We recommend that guideline developers consider issues of adherence regarding their reporting guideline, whether it is a new one or an extension or implementation of an existing one.
Journal endorsement, while encouraging for guideline developers, needs to be accompanied by a clear statement of how the journal expects authors to use the guideline and what level of adherence is required (e.g., authors of reports of randomized trials must submit a completed CONSORT checklist along with their submission). We recommend that journals consider ways to maximise adherence to reporting guidelines, such as by asking authors to submit completed checklists and by asking peer reviewers to use them as part of their review.
15. Evaluate the Impact of the Reporting Guidance
We recommend assessing the impact of any reporting guideline. Unfortunately, few guidelines appear to have been evaluated to date [24]. Although the guideline developers will likely want to evaluate their guidance, and we support such enthusiasm, other researchers should be also encouraged to conduct these assessments. There are several ways to design and carry out such an evaluation. Results from the systematic review on CONSORT evaluations suggest there is considerable room to improve the designs of these evaluations [39].
16. Develop Web Site
Another implementation strategy is to create a Web site dedicated to the reporting guideline. It makes the most sense to develop the Web site before publication so that the address can be included in the published articles. As a minimum we recommend including on the Web site the reporting guideline checklist, statement, ancillary documents such as an E&E, and any translated versions of the guideline. Also, the list of participants and funders should be included. The checklist and diagram should be available as both PDF and DOC files. Related unpublished documents can be published here too. The Web site is also a useful venue for alerting readers about emerging issues related to the guidance and perhaps inviting comments and discussion. We also recommend including a news section where new information about the guidance can be posted. Unfortunately, the lack of funding may impede such valuable additions.
We recommend including an endorser section for the names of journals and editorial groups endorsing the reporting guideline. The journals' Instructions to Authors can be linked to their endorsement.
The Web site can also provide policy documents regarding the use of the guidance. Issues pertaining to copyright, use of documents and, if relevant, the permitted use of the guidance logo if there is one, can be included in this section. Finally, we encourage guideline developers to consider linking their guideline with the EQUATOR Network (http://www.equator-network.org).
17. Translate Guideline
After publication of the reporting guideline it is possible that other researchers will want to translate the reporting guideline into different languages. The developers should seek to be actively involved to assure themselves that all translations are completed appropriately using robust methods including back translation.
18. Update Guideline
As yet few reporting guidelines have been updated, although most (83%) guideline developers recognize the need to do so [24]. The trigger to update is complex and likely similar to other areas of research [40]. Chief among these is the body of emerging literature to inform the currency of the checklist. If this literature is large and includes policy or empirical evidence it is likely important to consider updating. For example, recent empirical evidence [21] on outcome reporting bias led the CONSORT Group, when preparing CONSORT 2010, to add a checklist item specifically asking authors to describe any changes made to their study outcomes between the protocol and final analysis.
We recommend regular consideration of whether or not to update a reporting guideline and that reporting guideline groups maintain an executive group that can help make such decisions. Although it is possible to make frequent (e.g., annual) small updates, we believe this is not wise and it may even be counterproductive. Making occasional major updates, with clear version numbers, seems to be a better approach. We convened a one-day meeting to update the CONSORT Statement and two-day meeting to update QUOROM (subsequently published as the PRISMA Statement).
We assume that updating an existing reporting guideline will be completed by essentially the same team that initially developed the guidance, although some of the membership may change over time. That has been our experience from the CONSORT Statement updates. We also assume that several components of the initial guideline development process are still in place. For example, for updating the guideline, the developers would need to update their existing literature searches.
When considering an update to an existing reporting guideline, developers should give serious consideration to the seven essential items (denoted with an †) in Table 1. The development of an E&E deserves serious consideration when updating a reporting guideline if none existed before. If there is one, then it too will need updating (see Box 1). It is helpful to include in the publication a list of what was changed and why.
We also encourage reporting guideline developers to seriously consider how best to handle numbering and dates of updates. For example, the original version of CONSORT did not number the checklist items. We introduced numbering into the 2001 version, which meant that we have had to be careful in numbering new checklist items in the latest (2010) update. We have modified the numbering slightly. Updating a reporting guideline will influence checklist items of any published extensions and/or implementations, and due consideration is required as to how best to handle these issues.
Discussion
We hope this guidance on how to develop a reporting guideline, including an 18-step checklist, will fill a gap in the literature and be of help to potential and practicing developers. While some of the items are optional, there is a core set of steps we believe necessary to ensure adequate development of a reporting guideline. In the future, the EQUATOR Network might consider providing an appraisal grade for the reporting guidelines included on their database, reflecting the robustness of the guideline development process. We also encourage prospective guideline developers to contact the EQUATOR Network team to inform them about their work; this might prevent duplication of effort.
We will make the 18-step checklist available on the EQUATOR Network Web site. We will continue to monitor the literature to help maintain the guidance, particularly the checklist. Furthermore, we will annually review the need to update the checklist.
While there is increasing evidence that use of reporting guidelines is associated with improvements in the quality of reporting health research [37],[39],[41],[42], there is a growing anecdotal belief that reporting guidelines indirectly have a positive impact on how researchers design and conduct their research [43]. More formal research is required to substantiate these claims. Beyond the possible benefit in the design and conduct of health research, reporting guidelines are now starting to be used as an adjunct in developing educational courses in the design and conduct of health research.
Reporting guidelines are currently focused at the end of the knowledge generation cycle. However, we believe that investigators would benefit from the knowledge of key principles of health research reporting and relevant guidelines at the beginning of their research, having the end in mind. Indeed, guidelines are possibly equally useful earlier on in this process, and some granting agencies have acted upon this concept. The SPIRIT initiative is aimed at providing guidance for writing protocols of RCTs. Similarly, the UK National Institute of Health Research developed a research process flowchart (http://www.rdinfo.org.uk/) to guide researchers in how to start developing a research project. The flowchart includes reference to reporting guidelines and encourages researchers to “consult a relevant guideline in the early stages of research planning.”
Given the broad range of health research now covered by reporting guidelines, funders might start to require prospective applicants to use some parts of an “approved” reporting guideline when developing their research application and to include a completed checklist as part of this process. This might increase the overall quality of the applications to funders while increasing the potential return on their investment by emphasizing the importance of reporting much earlier in the knowledge generation cycle.
Reporting health research in a complete, accurate, transparent, and timely manner is a shared responsibility of all stakeholders involved in research funding, conduct, and publication. High-quality research reports contribute to more efficient translations of new research findings into clinical practice and help advance scientific knowledge and patient care. We will all benefit from these collective efforts.
Author Contributions
ICMJE criteria for authorship read and met: DM KFS IS DGA. Wrote the first draft of the paper: DM. Contributed to the writing of the paper: KFS IS DGA.
References
- 1. Glasziou P, Meats E, Heneghan C, Shepperd S (2008) What is missing from descriptions of treatment in trials and reviews? BMJ 336 (7659): 1472–4.
- 2. Zhang D, Yin P, Freemantle N, Jordan R, Zhong N, et al. (2008) An assessment of the quality of randomised controlled trials conducted in China. Trials 9: 22.
- 3. Scales CD, Norris RD, Preminger GM, Vieweg J, Peterson BL, et al. (2008) Evaluating the evidence: statistical methods in randomized controlled trials in the urological literature. J Urol 180: 1463–1467.
- 4. Lai TYY, Wong VWY, Lam RF, Cheng ACO, Lam DSC, et al. (2008) Quality of reporting of key methodological items of randomized controlled trials in clinical ophthalmic journals. Ophthalmic Epidemiol 14: 390–398.
- 5. Chan AW, Altman DG (2005) Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 365 (9465): 1159–62.
- 6. Moher D, Tetzlaff J, Tricco A, Sampson M, Altman DG (2007) Epidemiology and reporting characteristics of systematic reviews. PLoS Med 4: e78.
- 7. Moher D (2007) Reporting research results: a moral obligation for all researchers. Can J Anaesth 54: 331–335.
- 8. Altman DG, Simera I, Hoey J, Moher D, Schulz K (2008) EQUATOR: reporting guidelines for health research. Lancet 31: 1149–1150.
- 9.
Moher D, Simera I, Schulz K, Miller D, Grimshaw J, et al. Reporting Guidelines for Clinical Research: A Systematic Review. 6th International Congress on Peer Review and Biomedical Publication, Vancouver. Available: http://www.ama-assn.org/public/peer/abstracts-0912.pdf. Accessed 31 October 2009.
- 10. Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CFB, et al. (1998) Consensus development methods, and their use in clinical guideline development. Health Technol Assess 2: 1–88.
- 11. O'Toole LB (1997) MRC uses checklist similar to CONSORTs. BMJ 314: 1127.
- 12. Davidoff F, Batalden P, Stevens D, Ogrinc G, Mooney S (2008) Publication guidelines for improvement studies in health care: evolution of the SQUIRE Project. Ann Intern Med 149: 670–6.
- 13. Boutron I, Moher D, Tugwell P, Giraudeau B, Poiraudeau S, et al. (2005) Assessing the quality of randomized controlled trials: a checklist to evaluate a report of a nonpharmacological trial: (CLEAN NPT). J Clin Epidemiol 58: 123–1240.
- 14. Altman DG, Moher D (2005) Developing guidelines for reporting healthcare research: scientific rationale and procedures. (In Spanish) Med Clin (Barc) 125: Suppl 18–13.
- 15. von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, et al. (2007) The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med 147: 573–7.
- 16. Little J, Higgins J, Ioannidis JPA, Moher D, Gagnon F, et al. (2009) STROBE Extension to Genetic Association studies: STREGA (STrengthening the REporting of Genetic Association studies. PLoS Med 6: e22.
- 17. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, et al. (2003) for the STARD group. Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD initiative. Ann Intern Med 138: 40–44.
- 18. Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P, the CONSORT Group (2005) Reporting of nonpharmacologic treatment interventions: an extension of the CONSORT Statement. Ann Intern Med 148: W-60–W-66.
- 19. Davidson KW, Goldstein M, Kaplan RM, Kaufmann PG, Knatterud GL, et al. (2003) Evidence-based behavioral medicine: what is it and how do we achieve it? Ann Behav Med 26: 161–71.
- 20. Schulz KF, Chalmers I, Hayes RJ, Altman DG (1995) Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273: 408–412.
- 21. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, et al. (2008) Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3: e3081.
- 22. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, et al. (2004) Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Lancet 364: 911–912.
- 23. Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, et al. (2004) Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet 363: 1341–1345.
- 24. Simera I, Altman DG, Moher D, Schulz KF, Hoey J (2008) Guidelines for reporting health research: the EQUATOR Network's survey of guideline authors. PLoS Med 5: e139.
- 25. Hopewell S, Clarke M, Moher D, Wager E, Middleton P, et al. (2008) CONSORT for reporting randomized controlled trials in journal and conference abstracts. Lancet 371: 281–283.
- 26. Tricco AC, Pham B, Brehaut J, Tetroe J, Cappelli M, et al. (2009) An international survey suggested that unpublished systematic review exist. J Clin Epidemiol 62: 617–623.
- 27. Gagnier JJ, Boon H, Rochon P, Moher D, Barnes J, et al. (2006) Reporting Randomized, Controlled Trials of Herbal Interventions: An Elaborated CONSORT Statement. Ann Intern Med 144: 364–367.
- 28. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, et al. (2003) Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD initiative. Radiology 226: 24–28.
- 29. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, et al. (2003) Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD initiative. Clin Chem 49: 1–6.
- 30. Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. PLoS Med 6: e1000097.
- 31. Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. Ann Intern Med 151: 264–269.
- 32. Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. Open Med 3: 123–130.
- 33. Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. BMJ 339: 332–336.
- 34. Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. J Clin Epidemiol 62: 1006–12.
- 35. Meinert CL (1998) Beyond CONSORT: need for improved reporting standards for clinical trials. JAMA 279: 1487–1489.
- 36. Altman DG (2005) Endorsement of the CONSORT statement by high impact medical journals: survey of instructions for authors. BMJ 330: 1056–1057.
- 37. Smidt N, Rutjes AW, van der Windt DA, Ostelo RW, Bossuyt PM, et al. (2006) The quality of diagnostic accuracy studies since the STARD statement: has it improved? Neurology 67: 792–797.
- 38. Hopewell S, Altman DG, Moher D, Schulz KF (2008) Endorsement of the CONSORT Statement by high impact factor medical journals: a survey of journal editors and journal ‘Instructions to Authors’. Trials 9: 20.
- 39. Plint AC, Moher D, Morrison A, Schulz KF, Altman DG, et al. (2006) Does the CONSORT checklist improve the quality of reports of randomized controlled trials: a systematic review. Med J Aust 185: 263–267.
- 40. Shojania K, Sampson M, Ji J, Ansari MT, Doucette S, et al. (2007) How quickly do systematic reviews go out of date: A survival analysis. Ann Intern Med 147: 224–233.
- 41. Smith BA, Lee HJ, Lee JH, Choi M, Jones DE, et al. (2008) Quality of reporting randomized controlled trials (RCTs) in the nursing literature: application of the Consolidated Standards of Reporting Trials (CONSORT). Nurs Outlook 56: 31–37.
- 42. Prady SL, Richmond SJ, Morton VM, MacPherson H (2008) A Systematic Evaluation of the Impact of STRICTA and CONSORT Recommendations on Quality of Reporting for Acupuncture Trials. PLoS ONE 3: e1577.
- 43. Narahari SR, Ryan TJ, Aggithaya MG, Bose KS, Prasanna KS (2008) Evidence-based approaches for the ayurvedic traditional herbal formulations: towards an ayurvedic CONSORT model. J Altern Complement Med 14: 769–776.
- 44. Moher D, Schulz KF, Altman DG, for the CONSORT group (2001) The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. Lancet 357: 1191–1194.
- 45. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, et al. (2001) The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 134: 663–694.
- 46. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, et al. (2003) Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Explanation and Elaboration. Ann Intern Med 138: W1–W12.
- 47. Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow C, et al. (2007) The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. Epidemiology 18: 805–835.
- 48. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche P, et al. (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med 6: e1000100.
- 49. Campbell MK, Elbourne DR, Altman DG, for the CONSORT group (2004) CONSORT statement: extension to cluster randomised trials. BMJ 328: 702–708.