Assessing ethics of trials in systematic reviews
BMJ 2004; 328 doi: https://doi.org/10.1136/bmj.328.7446.1013 (Published 22 April 2004) Cite this as: BMJ 2004;328:1013
- Michael A Weingarten, professor (weingml{at}post.tau.ac.il)1,
- Mical Paul, registrar2,
- Leonard Leibovici, head of department3
- 1Department of Family Medicine, Tel-Aviv University, Tel-Aviv, Israel
- 2Internal Medicine and Infectious Diseases Unit, Beilinson Hospital, Rabin Medical Centre, Petah Tikva, Israel
- 3Beilinson Campus, Rabin Medical Centre
- Correspondence to: M A Weingarten
- Accepted 30 January 2004
Routine assessment may improve ethical standards and overall quality of trials
Our awareness of the requirements for ethical clinical research has increased over the past century. Research ethics committees were set up after the Declaration of Helsinki to review research proposals. Many journals now require a statement that ethical approval has been obtained before they consider a research report for publication. Nevertheless, many published studies do not come up to standard, or at least do not report that they do. For example, 30 out of 37 consecutive studies published in five general paediatric journals did not report whether informed consent was obtained. Twenty four of them did not report whether the committee on research ethics had approved the study.1 We propose that systematic reviews of experimental clinical research on humans should also include information on the ethical standards of the trials.
Why include ethical information?
The main reason for including ethics in the checklist of systematic reviews is to increase awareness in the scientific community about the need for high ethical standards in research on humans. The proposal would also encourage reviewers to identify those occasional studies that were so unethical that there may be doubts about the morality of using the results. Although such trials are rare, history has given us too many real examples to allow us to be complacent.2 3 Opinions differ on whether it is justified to disseminate the results of such studies.4 Either way, a conscious decision should be made and revealed to readers of the review.
Issues around ethical quality overlap importantly with the central issues of the validity, reliability, and generalisability of research findings. They relate to some of the more subtle potential sources of bias in experimental clinical research. It is thus important to include ethical assessment in systematic reviews on prudential grounds as well as on moral grounds. The results of ethically sound trials may be more reliable than those from other trials. In a review of 767 randomised controlled trials published during 1993-5 in the New England Journal of Medicine, Lancet, BMJ, and JAMA, trials of higher methodological quality were more likely to provide information about their ethical aspects; so it seems that more reliable research is also more sensitive to the ethical requirements of research.5
Which issues should we assess?
Ethical issues at the level of a systematic review are different from those for original research, and it might be unfair to examine all trials under the same ethical loupe. Older trials will have been done when ethical standards of research had not been as clearly formulated as they are today. Ethical standards depend not only on the time the study was performed but also on place and social mores.
Nevertheless, we think that examination of the ethical standards of trials should form part of every review, but the ethical principles to be applied need to be defined. Among the approaches to research ethics, we find that proposed by Foster most attractive. She suggests examining the project from three perspectives: goals, duties, and rights.6 We used this model to devise a protocol to help assess the ethical quality of clinical trials included in systematic reviews (box).
Should systematic reviews merely delineate the ethical quality or attempt to integrate the observations into its conclusions? We believe that reviews should at least include a report of the ethical assessment and that the implications for further research should address the ethical gaps observed in existing studies.
How would the protocol work in practice?
We applied our protocol to a recently published systematic review of trials comparing β lactam monotherapy versus βlactam combined with aminoglycoside for febrile neutropenia.9 The review included 47 studies carried out during 1981-2000 and 7807 randomised patients.
Goal related considerations
Financial support—Twenty eight trials gave information on financial support: all had received funding from the company that manufactured the monotherapy antibiotic. Three trials declared research grant support as well as commercial support.
Conflicts of interest—None of the studies included a statement related to potential conflicts of interest.
Justification—Thirty seven studies did not report a sample size calculation. We calculated that an equivalence trial design with an overall success rate of 60% (as observed in this review), and permitting a 20% difference between the treatments, would require around 200 assessable patients to ensure an 80% chance of rejecting at α = 0.05 the null hypothesis of equivalence.10 Thirty eight trials had fewer than 200 patients. This requirement is modest given the broad difference permitted between the treatments. Compilation of all existing data in this review led us to believe that further trials comparing β lactam monotherapy with a narrower spectrum β lactam combined with an aminoglycoside would be unjustified. However, we could not identify a specific time at which this conclusion should have been reached.
Guide for ethical assessment of trials in systematic reviews
Goal related considerations
Is there a clear declaration on financial support in all trials?
Is there a statement that relates to potential conflicts of interest in all trials?
Justification—Could the results have been obtained by laboratory or animal experiments? Were any of the trials superfluous? Was the size of the study sufficient to achieve adequate statistical power?
Publication bias—How many of the identified trials remained unpublished? Is bias detectable by funnel plot analysis?
Duty related considerations
Were the comparators appropriate? If a placebo was used, was it justified?
Rights related considerations
Safety—Was the risk for participants appropriate to the importance of the research?7 Was appropriate follow up care assured?
Was informed consent obtained?
When participants had reduced competence, were appropriate measures taken to protect their best interests?8
Were adequate steps taken to prevent unauthorised access to personal and clinical data?
Global considerations
Was the study approved by a research ethics committee?
Publication bias—Five trials completed before 1990 (943 patients) remained unpublished or were published only as conference proceedings. The results of one study including 460 patients were never formally published.11 Although we included these studies in our review, the funnel plot showed that small studies favouring combination therapy might be missing.
Duty related considerations
Appropriate comparators—We considered antibiotic combinations advised by guidelines contemporary to the study years as adequate.12–14 Twenty trials used an inadequate comparator on this basis.
Rights related considerations
Safety—We considered monitoring aminoglycoside serum concentrations at any time during the study and creatinine at least twice weekly as minimal follow up measures to ensure patients' safety. Thirty one trials did not report aminoglycoside monitoring and 17 had inadequate creatinine follow up.
Informed consent—Twenty nine trials reported that informed consent had been obtained from adult patients.
Participants of reduced competence—Twenty five trials included patients younger than 16 years old, and in only five of these was consent obtained from parents or relatives. The trials covered a variety of patient populations, including some seriously ill patients with disseminated carcinoma and geriatric patients. Consent was sometimes given by the family rather than the patient, and in those cases it was not clear whether the family had any power of attorney to act on the patient's behalf.
Confidentiality—There was no information on steps taken to ensure confidentiality in the published reports.
Approval by research ethics committee—Twenty two studies stated that they had been approved by a research ethics committee. We noted an association between this approval and other ethical measures. All published studies with an adequate sample size reported approval, and an appropriate comparator was more commonly used in these studies than in studies that did not refer to the ethics committee (12/22 v 7/25 studies respectively).
Conclusions
Our proposed protocol showed several important ethical flaws in the studies on which we base our current management for cancer patients with febrile neutropenia. For most measures, the protocol enabled straightforward and systematic extraction of data on ethical issues. The details of how to assess several of the measures, such as the justification of studies or the appropriateness of the comparators, may need further consideration to allow for uniform evaluation in systematic reviews of different sorts.
Summary points
Many clinical trials do not report details of ethical issues
Including ethical assessment in systematic reviews will encourage researchers to conduct ethical research
Ethical research may provide more reliable results
Ethics can be assessed systematically by using the suggested protocol
We believe that including ethical considerations in systematic reviews will increase research workers' awareness of the need to conduct ethical research. It might also contribute to the interpretation of results. To encourage implementation of this proposal, we recommend that the next CONSORT statement should include ethical justification as an integral part of their protocol for reporting trials.15
Acknowledgments
We thank Iain Chalmers, Andrew Herxheimer, and Amanda Burls for their comments and suggestions.
Footnotes
-
Contributors and sources MAW was a founder member of the Cochrane Colaboration primary care field and teaches medical ethics. LL is a specialist in infectious disease and has written systematic reviews. MP wrote the systematic review that is used to illustrate this article
-
Competing interests None declared.