Ann Thorac Surg 2006;82:1173-1174
© 2006 The Society of Thoracic Surgeons
Editorial
"Surgical Research or Comic Opera" Redux
Robert M. Sade, MD*
Institute of Human Values in Health Care, Medical University of South Carolina, Charleston, South Carolina
* Address correspondence to Dr Sade, Department of Surgery, 96 Jonathan Lucas St, Suite 409, PO Box 250612, Charleston, SC 29425 (Email: sader{at}musc.edu).
 |
Introduction
|
|---|
The gold standard of clinical research is the randomized controlled trial (RCT). Yet, only 6–10% of studies published in the cardiothoracic surgery literature are RCTs, less than half the rate of RCTs published in medical journals [1, 2]. More than half of surgical research studies are retrospective chart reviews, mostly clinical series of operations, which have been decried by some of our medical colleagues as little better than a collection of anecdotal experiences, the weakest scientific evidence, or as hopelessly invalid because of uncontrolled bias and other confounding, potentially lethal flaws [3]. Some comments have gone beyond mere criticism to outright ridicule: "I should like to shame [surgeons] out of the comic opera performances which they suppose are statistics of operations" [3].
It is not often recognized, however, that RCTs can reach valid and useful conclusions only if they address procedures or treatments in which the study interventions can be easily standardized, the skill of the treating physician is of minimal importance, subjects are available in sufficient numbers to allow reliable statistical analysis, and blinding of relevant participants can be easily accomplished. Given those requirements, we can appreciate why RCTs are more common in medical research than in surgical, particularly surgical operations: placebo controls or no treatment controls are often either undesirable or unethical; blinding of all relevant participants is not practical or not possible because surgeons must know what they are doing (blinding of patients, assessors, and analysts often can be done); and patients needing specific procedures are often too few in number for reliable statistical analysis. Standardization of treatment is particularly difficult to accomplish because, unlike 20-milligram tablets, no two operations are the same. Surgical techniques vary among surgeons, and also vary in the same surgeon from one operation to the next; small incremental technical changes progressively improve outcomes as the learning curve moves to the right and upward. Moreover, new surgical techniques are often devised or adopted by technically skilled enthusiasts, so the results of randomized studies may not be generalizeable to all surgeons.
|
Quality of Surgical Research Reports
|
|---|
Editors of scientific journals are concerned with the quality and reliability of the reports they publish. Quality is not defined entirely by technical aspects of investigations, but also by the visibility to readers of potential, if subtle, biases that could make their way into a report. To address such issues, The Annals has for several years required disclosure of conflicts of interest by all of its authors, as well as assurance of "Freedom of Investigation," a signed statement that the study was not tainted by unwarranted outside influences. In pursuit of transparency, The Annals also requires disclosure of ghostwriters—writers not included in the list of authorship, usually employed by the manufacturer of a device or drug being studied. A special section of the journal, "New Technology," was introduced recently to encourage publication of independent evaluations of new devices and procedures.
The editors of The Annals recognize that the proportion of RCT reports in the cardiothoracic surgical literature is relatively small, but is not trivial, numbering about 30–40 a year. It is, therefore, important to ensure that they are well designed and clearly presented, so that readers can rely on their conclusions and recommendations. Two recent studies of the quality of RCTs in the cardiothoracic literature do not provide much comfort [1, 4]. Both Tiruvoipati and Anyanwu, with their coauthors, found the quality of reporting of RCTs in The Annals of Thoracic Surgery, The Journal of Thoracic and Cardiovascular Surgery, and the European Journal of Cardio-Thoracic Surgery to be disturbing, at best. For example, determination of sample sizes necessary for an adequate RCT should be based on a power calculation, but according to Tiruvoipati's and Anyanwu's reports, this was done only in 8% and 20%, respectively, of the published RCTs. Although adequate blinding of all participants is sometimes impossible, in those RCTs in which it could have been done, blinding of the subjects was described in only 33% and 21%, of the surgeon in 54% and 33%, and of the evaluators in 40% and 38%, respectively [1, 4]. Description of inclusion and exclusion criteria is critically important to reports of RCTs; yet these were absent in 25% and 34% of the published RCTs. Most studies (86%) used surrogate outcomes, such as laboratories measures, rather than relevant clinical outcomes as their end points [4]. Most, in fact, did not have enough detail to permit generalization of the findings nor to instill confidence in the validity of the conclusions. Design flaws and insufficient power render most RCTs in cardiothoracic surgical journals incapable of providing credible answers to the questions they address.
To provide valid, useful information to journal readers, RCTs must be well designed and accurately executed. Even when these are ideal, however, if the methods and results are inadequately described in the published paper, readers can assess neither the reliability of the conclusions nor the usefulness in their practices of recommendations that are based on the data.
|
Improving the Accuracy of RCT Reports: CONSORT
|
|---|
The relatively poor quality of RCT reports has been known to be widespread in the medical and surgical literature for many years. In an attempt to elevate the quality of RCT reports, an independent group of statisticians, epidemiologists, clinical trialists, and biomedical editors published a statement—Consolidated Standards of Reporting Trials (CONSORT)—a decade ago [5]. The statement comprises two components: a checklist of 22 items relating to details of design and process for RCTs and a flow diagram of the progress of the investigation's subjects through all phases of a trial, from eligibility assessment to final analysis. CONSORT immediately garnered the support or endorsement of a number of journals and editorial groups, including the Council of Science Editors, the International Committee of Medical Journal Editors, The Lancet, The New England Journal of Medicine, The Journal of the American Medical Association, and the Annals of Internal Medicine [6]. The group of supporters of this initiative has continued to grow over the last decade. At the time of this writing, 207 journals and 35 organizations support CONSORT. Among the journals, only 10 (5%) are surgical; this small subgroup includes The Journal of the American College of Surgeons, the Archives of Surgery, the Journal of Pediatric Surgery, and The Annals of Thoracic Surgery [7].
The checklist is divided into sections corresponding to the parts of a RCT: Title and Abstract, Introduction, Methods, Results, and Comment/Discussion. The 22 items on the checklist were not selected arbitrarily; they were specifically selected because 1) there is published evidence that failure to report the specified information introduces bias into estimates of the effects of treatment, or 2) the solicited information is required to permit the reader to judge the reliability of the conclusions and its relevance to the care they provide to their own patients.
The flow diagram tracks the number of patients at each stage of the trial, clearly displaying the structure of the trial and informing the reader of whether an intention-to-treat analysis has been performed.
The purpose of CONSORT was to improve the quality of RCT reports by increasing the transparency of the experimental process, exposing flaws and inadequacies, and allowing readers to evaluate the validity of the reported results for their own purposes. By repeatedly exposing authors of RCT reports to the detailed checklist, the creators of CONSORT hoped that it would serve to educate researchers, eventually resulting in progressively improved quality of the RCTs themselves as well as the papers describing them. Several studies have shown that this hope has been realized: the quality of RCT reporting has improved from before to after journals have adopted CONSORT [8–11]. Although progress has been documented, much remains to be improved.
The Annals believes that elevation of the quality of RCT reports in this journal will directly benefit our readers, and will have a ripple effect throughout our field. The journal's online Information for Authors document will contain the URL for the CONSORT website [6] as well as the specific pages for the checklist and the flow diagram [12]. The checklist is easy to use and informative because each listed standard links to an illustrative example, and a detailed explanation of what is meant by the standard. For example, item 11 on the checklist states: "Blinding (masking): Whether or not participants, those administering the interventions, and those assessing the outcomes were aware of group assignment. If not, how the success of masking was assessed." Linked to this item is an explanation of what the term "blinding" means, why it is important to blind the subjects, involved physicians, and assessors, as well as a description of methods to evaluate the success of blinding.
When submitting a report of an RCT to The Annals, authors will be required to accompany the manuscript with a completed CONSORT checklist and flow diagram. These items will not be published with the paper (although the authors may wish to include the flow diagram in the manuscript) but will be used by the editors and reviewers in evaluating the paper. We believe that this obligation will not place an undue burden on the authors of RCTs because the checklist and flow diagram are not difficult to complete, and there is neither a requirement nor an expectation that a submitted manuscript will meet every standard.
We anticipate that our adoption of CONSORT will have both short- and long-term beneficial effects: immediately, it will improve the clarity and transparency of RCT reports in this journal, and in the long term, its educational power will lead to gradual improvement not only in the reporting of RCTs, but also in their design and execution.
|
References
|
|---|
- Tiruvoipati R, Balasubramanian SP, Atturu G, Peek GJ, Elbourne D. Improving the quality of reporting randomized controlled trials in cardiothoracic surgery: the way forward J Thorac Cardiovasc Surg 2006;132:233-240.[Abstract/Free Full Text]
- Block MI, Khitin LM, Sade RM. Ethical process in human research published in thoracic surgery journals Ann Thorac Surg 2006;82:6-11discussion 11–2.[Abstract/Free Full Text]
- Horton R. Surgical research or comic opera: questions, but few answers Lancet 1996;347:984-985.[Medline]
- Anyanwu AC, Treasure T. Surgical research revisited: clinical trials in the cardiothoracic surgical literature Eur J Cardiothorac Surg 2004;25:299-303.[Abstract/Free Full Text]
- Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trialsThe CONSORT statement. JAMA 1996;276:637-639.[Medline]
- CONSORT-Group. CONSORT: strength in science, sound ethics. Available at: http://www.consort-statement.org/ (accessed June 16, 2006)..
- CONSORT-Group. CONSORT Supporters. Available at: http://www.consort-statement.org/Endorsements/endorsements.htm (accessed June 16, 2006)..
- Moher D, Jones A, Lepage L. Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation JAMA 2001;285:1992-1995.[Abstract/Free Full Text]
- Egger M, Juni P, Bartlett C. Value of flow diagrams in reports of randomized controlled trials JAMA 2001;285:1996-1999.[Abstract/Free Full Text]
- Ellis C, Hall JL, Khalil A, Hall JC. Evolution of methodological standards in surgical trials ANZ J Surg 2005;75:874-877.[Medline]
- Kober T, Trelle S, Engert A. Reporting of randomized controlled trials in Hodgkin lymphoma in biomedical journals J Natl Cancer Inst 2006;98:620-625.[Abstract/Free Full Text]
- CONSORT-Group. CONSORT Explanation. Available at: http://www.consort-statement.org/Explanation/explanation.htm (accessed June 16, 2006)..
Top
Introduction
Quality of Surgical Research...