Article Text
PDF
Abstract
Background Medical research so flawed as to be retracted may put patients at risk by influencing treatments.
Objective To explore hypotheses that more patients are put at risk if a retracted paper appears in a journal with a high impact factor (IF) so that the paper is widely read; is written by a ‘repeat offender’ author who has produced other retracted research; or is a clinical trial.
Methods English language papers (n=788) retracted from the PubMed database between 2000 and 2010 were evaluated. Only those papers retracting research with humans or freshly derived human material were included; 180 retracted primary papers (22.8%) met inclusion criteria. Subjects enrolled and patients treated were tallied, both in the retracted primary studies and in 851 secondary studies that cited a retracted primary paper.
Results Retracted papers published in high-IF journals were cited more often (p=0.0004) than those in low-IF journals, but there was no difference between high- and low-IF papers in subjects enrolled or patients treated. Retracted papers published by ‘repeat offender’ authors did not enrol more subjects or treat more patients than papers by one-time offenders, nor was there a difference in number of citations. However, retracted clinical trials treated more patients (p=0.0002) and inspired secondary studies that put more patients at risk (p=0.0019) than did other kinds of medical research.
Conclusions If the goal is to minimise risk to patients, the appropriate focus is on clinical trials. Clinical trials form the foundation of evidence-based medicine; hence, the integrity of clinical trials must be protected.
- Data fabrication
- data falsification
- scientific fraud
- clinical trials
- applied and professional ethics
- professional misconduct
- scientific research
- clinical ethics
Statistics from Altmetric.com
- Data fabrication
- data falsification
- scientific fraud
- clinical trials
- applied and professional ethics
- professional misconduct
- scientific research
- clinical ethics
Introduction
It has been postulated that medical research so flawed that it is eventually retracted may put patients at risk of harm by influencing modes of treatment.1–4 Patients may be at risk in the context of a retracted (‘primary’) study, or in any secondary published study that drew ideas or inspiration from the primary study.4
To illustrate how risk can arise from a retracted paper, consider COOPERATE, a double-blind, randomised controlled trial (RCT) of medication for non-diabetic renal disease.5 Authors of COOPERATE claimed that 263 patients were randomly allocated to treatment with either an angiotensin-II receptor blocker, an ACE inhibitor, or a combination of both drugs at equivalent doses. COOPERATE concluded that combination therapy was better than monotherapy with either an ACE inhibitor or angiotensin-II receptor blocker alone.5 However, COOPERATE was eventually retracted because, “the trial had not been approved by the ethics committee… the involvement of a statistician could not be verified… [and] the trial was not a double-blind study, because [the principal investigator] knew the treatment allocation”.6
Despite early indications that the COOPERATE trial might be flawed, combination therapy was widely accepted by expert clinicians.7 Yet problems eventually cropped up; another RCT concluded that combination therapy can be associated with more frequent adverse events than monotherapy, with no increase in patient benefit.8 Combination therapy to prevent kidney failure may reduce proteinuria more than monotherapy, but may also worsen major renal outcomes.9 Diabetics with stage 1 hypertension fared worse with combination therapy than monotherapy,10 and patients with left ventricular dysfunction had an increased risk of adverse events on combination therapy, relative to ACE inhibitor therapy alone, with no survival benefit.11 Nevertheless, a recent population-based study found that elderly patients with no risk factors are still being prescribed combination therapy, though these patients may have an increased risk of adverse renal outcomes.12 The Wall Street Journal recently estimated that well over 100 000 patients have received combination therapy.13
Potential patient risk associated with the COOPERATE trial is thus fourfold: patients were enrolled in potentially dangerous secondary RCTs that were based on a fraudulent primary trial; time, effort and money were wasted by patients and clinicians; false information pervaded the medical literature; and combination therapy was accepted more quickly and used more widely than it might have been otherwise.4
How can patients be protected from the risks associated with retracted research? Clearly, the best thing would be to prevent publication of all fraudulent research, but that is not possible.14 The most practical thing may be to focus attention on those types of research that are most likely to put patients at risk. But what kinds of fraudulent research put the most patients at risk? To examine this question, we tally patients enrolled in primary and secondary studies as a surrogate measure of potential risk to patients.
We hypothesise that risk from clinical research is greater when a retracted paper appears in a journal with a high impact factor (IF) so that the paper is widely read and cited; is written by a ‘repeat offender’ author2 who has produced other retracted research; or is a clinical trial that enrols patients for treatment and inspires similar clinical trials in other patient populations.
Methods
The PubMed database was searched on 22 January 2010, to identify every research paper noted as retracted between 2000 and 2010.1–3 Using the limits of ‘items with abstracts, retracted publication, English,’ a total of 788 retracted papers were identified, all of which were exported from PubMed and saved as a text file (available upon request), as described.1
Data pertinent to each retracted paper were abstracted from the paper itself or the retraction notice for that paper.2 The retracted article itself was read to determine whether the research was a clinical study, defined as a study that worked with humans or freshly derived human material. ‘Freshly derived human material’ is simply recently harvested human cells or tissues.
We excluded from further analysis 468 basic science papers that did not report new data derived from study of humans or human material (figure 1). We also excluded 88 review articles which presented no new data and 22 case reports, since these are often a posteriori descriptions of a limited number of patients who were not treated in the context of a clinical trial. This left a total of 210 retracted medical papers for analysis. We reviewed retraction notices for all 210 papers and excluded from further consideration seven papers retracted because of journal error, since no evidence was presented that the authors had been at fault.
Flowchart showing articles analysed.
The remaining studies were dichotomised into medical research—any research that did not use competing treatments—and clinical trials—research involving humans who were allocated prospectively to competing treatments. The number of subjects enrolled in each retracted paper was then tallied using the following criteria4:
Total subjects: total number of patients and controls included, as stated in the results;
Patients at risk: patients with any illness or injury requiring treatment (if only healthy subjects were enrolled, this number could be zero);
Patients treated: patients who received a risky intervention (eg, medication, surgery) in a prospective study; blood draw or other minor intervention did not count to this total. Patients who received placebo or standard treatment also did not count. This is a subset of ‘patients at risk’ and is largely made up of patients enrolled in RCTs.
Every citation of a retracted primary paper was identified using ‘Web of Science (1955–present)’ a database within ISI Web of Knowledge (Thomson Reuters), as described.4 Web of Science was searched during the last weeks of November, 2010. Every secondary paper that cited a retracted primary paper was evaluated and the number of subjects enrolled in secondary studies was tallied as above, eliminating any secondary study that was cited more than once.4
ISI Web of Science was also used to track every citation of a retracted paper, as follows:
Total citations: raw count of all secondary papers citing a retracted primary paper;
Research-related: count of all new research papers citing a primary paper;
Post-retraction: research-related secondary papers published subsequent to the retraction of a primary paper; a subset of research-related papers;
Retraction-related: secondary papers discussing reasons for primary paper retraction;
Review: meta-analyses or systematic reviews which included the primary paper;
Patient study: prospective medical studies involving patients; a subset of research-related papers.
After tabulating data in an Excel spreadsheet, the database was sorted to test three hypotheses. Data were sorted by journal IF, to determine whether publication in a high-IF journal put more patients at risk, either in primary or secondary papers (table 1). Then data were sorted by whether the first author was a one-time offender or a ‘repeat offender’, to determine whether ‘repeat offender’ authors put more patients at risk (table 2). Finally, the data were sorted by type of primary study, to determine whether medical research or clinical trials put more patients at risk (table 3). Differences between groups of papers were tested for significance with the t test function of Excel, assuming two tails and equal variances and the p value for significance was Bonferroni-corrected for multiple tests (minimum significance, p=0.05/12 tests=0.00417).
Comparison of retracted studies from high- or low-IF journals
Comparison of retracted studies by one-time offenders or ‘repeat offenders’
Comparison of retracted studies that were either medical research or clinical trials
Results
Web of Science did not provide any information about the citation pattern of 23 papers, so these papers were excluded from evaluation.4 A total of 180 retracted papers (22.8% of all retractions) met inclusion criteria and were therefore evaluated (figure 1).
Retracted papers in high-IF journals were cited more often (p=0.0004) than retracted papers in low-IF journals (table 1). High-IF journals published primary papers with more research-related citations (p=0.0006) and more secondary patient studies were initiated as a result of retracted papers in high-IF journals (p=0.001). However, retraction from a high-IF journal was not discussed in the literature more than was retraction from a low-IF journal, which may explain why papers retracted from high-IF journals were cited more often after retraction than were papers retracted from low-IF journals (p<0.0001). Papers in high-IF journals did not enrol more subjects or treat more patients than papers in low-IF journals. The greater number of citations for papers in high-IF journals also did not result in more subjects being enrolled or patients being treated in secondary papers.
Retracted papers published by ‘repeat offender’ authors did not enrol more subjects or treat more patients than did papers published by one-time offenders (table 2). Neither did ‘repeat offender’ authors inspire more subjects to be enrolled or patients to be treated in secondary studies. In addition, papers by ‘repeat offenders’ were not cited more than papers by one-time offenders and seemed to have no disproportionate impact on the literature.
Retracted clinical trials treated more patients (p=0.0002) than did retracted medical research (table 3). In addition, retracted clinical trials inspired more patients to be treated in secondary studies (p=0.0019). There were 39 retracted clinical trials in this sample (table 3); 6373 patients were enrolled and 5735 patients were treated in these trials. In contrast, there were 141 retracted medical studies, which enrolled 11 410 patients and treated 3454 patients. Therefore, 35.8% of all patients studied and 62.4% of all patients treated were enrolled in a clinical trial. Although there were 3.6-fold more retracted medical studies than retracted clinical trials, retracted clinical trials still treated 66% more patients than did retracted medical studies.
Discussion
A large number of patients are enrolled and treated in research studies so flawed that they are eventually retracted (tables 1–3). Yet potential risk to patients does not entirely follow the patterns hypothesised. Although retracted papers in high-IF journals are cited frequently, they do not enrol or treat more patients than retracted papers from low-IF journals, nor do they inspire more secondary patient studies to be performed (table 1). Similarly, papers by ‘repeat offender’ authors do not endanger more patients than do papers by one-time offenders, nor are ‘repeat offender’ author papers cited more often than one-time offender papers (table 2). Instead, the source of greatest potential risk to patients seems to be clinical trials, which treat more patients and inspire more patients to be treated in secondary studies than do other types of medical research (table 3). This suggests that high-IF journals and ‘repeat-offender’ authors do not have a disproportionate impact on patient risk. However, clinical trials have a major impact on patient risk, no matter where those trials are published.
The idea that retracted clinical trials can put patients at risk will be controversial, though risk seems likely in at least some cases. Dr Joachim Boldt of the Klinikum Ludwigshafen in Germany was a leading proponent of hydroxyethyl starch (HES) for blood volume expansion in surgical and trauma patients.15 Boldt published an RCT favourable to HES in Anaesthesia & Analgesia, for which he claimed review and approval by an institutional review board, with written informed consent and a randomisation process for patients.16 Approval, consent and randomisation apparently did not actually happen and it is possible that the study was never done at all, so Anaesthesia & Analgesia retracted the RCT, noting extensive data fabrication.17 Subsequently, Boldt had an additional 22 papers retracted from Anaesthesia & Analgesia,18 and up to 89 papers with Boldt as a co-author may eventually be retracted.19
A recent Cochrane review evaluated 65 RCTs that compared colloids (such as HES) with crystalloids in critically ill patients who required blood volume expansion.20 This careful review, which included three RCTs by Boldt, found no evidence that resuscitation with colloids reduces the risk of death, compared with resuscitation with crystalloids, though colloids are more expensive than crystalloids. When Boldt's three studies were excluded from meta-analysis, the conclusions were unchanged. This review concluded that, because colloids are more expensive but not more effective, continued use of colloids cannot be supported.20
Several RCTs have shown that artificial colloids such as HES may cause troublesome renal side effects in patients with surgery or sepsis.21 There is meta-analytic evidence that HES is associated with an increased risk of death in patients with severe sepsis or septic shock.22 One meta-analysis of 22 RCTs included seven RCTs by Boldt—meaning that among 1865 patients evaluated, 590 (31.6%) may have been fabricated—and all but one of Boldt's studies supported HES.22 If Boldt's studies are excluded from meta-analysis,22 it is probable that safety concerns related to HES would be heightened.15 Recently, HES has been linked to serious side effects, including an increased risk of bleeding, heart failure and anaphylactic shock.23 No current studies of HES are adequate to assess long-term outcomes because studies are too small, too poorly designed, or too short-term to assess meaningful outcomes; there is thus no convincing evidence that HES is safe for use in critically ill patients.23 Nevertheless, a survey of European anaesthesiologists found that 82.1% of respondents believed colloids (± crystalloids) were preferable to crystalloids for haemodynamic management in septic shock and 57.9% of respondents preferred HES to other types of colloid.24
What do our results (tables 1–3) mean for medical research? Clinical trials are not more likely to be retracted than any other type of medical research; roughly one in 6070 clinical trials was retracted between 2000 and 2010, while roughly one in 6109 medical papers was retracted during the same period.1 Yet the number of patients put at risk in a flawed clinical trial—and the number of patients enrolled in secondary clinical trials—is higher than the number of patients put at risk in other types of medical research (table 3). Thus, if the goal is to prevent risk to patients, the appropriate focus is on clinical trials.
Little is known about the prevalence of research misconduct in clinical trials, but most reported misconduct cases do not involve clinical trials.25 Among 136 research misconduct reports from the Office of Research Integrity from 1992 to 2002, 13% were clinical trials.26 Yet only about 6% of all published medical research papers are RCTs,1 so RCTs are over-represented among studies investigated by the Office of Research Integrity. Clinical trials are open to a range of abuses25: selective reporting of positive results; omission of negative results or adverse events; improper emphasis on secondary outcomes or accidental findings; overgeneralisation of findings from small trials to large populations; selective reporting of only those findings that are significant; and failure to report any findings at all, if findings are negative or unfavourable.27
The research presented here is limited by several considerations. Patients in the clinic may also be put at risk if they are treated on the basis of retracted research, but it is not yet possible to characterise such risk, if treated patients are not described in a published paper.4 Patients enrolled in a clinical protocol that closed without a written report—or patients treated in a clinic without being enrolled in a protocol at all—would not affect our estimate of risk from retracted research. Furthermore, merely tracking citations does not fully document the potential extent of damage. False ideas can assume a life of their own28 and become difficult to expunge from the literature or from the thinking of clinicians.29
Do journal editors have a fiduciary responsibility to protect patients from harm that may arise from retracted research? Though the ethical responsibilities of journal editors are not clear at present, it is possible that medical malpractice suits could eventually be aimed at journals which publish a retracted paper. Journal culpability could arise in several ways. If a paper is retracted for obvious fraud or error, then the journal may be faulted for failing to expose the problem before publication. Even if the fraud or error is subtle, there may still be an effort to target journals for perceived shortcomings in the review process. After a paper is retracted, many journals fail to alert the naïve reader about that withdrawal; 32% of retracted research is never noted as such by the journal of publication.3 Thus, flawed work can have an impact on the literature long after retraction (tables 1–3). Journals may be held to blame for prolonging such adverse impact by inadequately publicising a retraction. Furthermore, many retraction notices are opaque as to the reasons for retraction; 8% of retracted papers were withdrawn for unstated reasons and another 10% were withdrawn for ambiguous reasons.3 Papers retracted for fraud (data fabrication or falsification) appear to put more patients at risk than papers retracted for error,4 which may mean that papers retracted for fraud represent a calculated effort to deceive.2 Yet journal retraction notices may be so poorly written that a reader cannot discern whether error or fraud was involved and thus whether deception is likely. These several journal failings could be perceived as creating legal liability and may eventually invite legal suit.
Scientific ethics and the scientific method do not merely coexist; they are interwoven to an extent that they cannot be separated.30 Ethical enquiry is the foundation of the scientific method. Biomedical research is currently experiencing a crisis in public opinion31 and the sense of goodwill that participants normally bring to clinical trials32 must not be taken for granted. It is imperative that measures be taken to reduce patient risk in clinical trials,14 since trials form the foundation of evidence-based medicine and will become increasingly important as a means to justify difficult clinical decisions.
References
Footnotes
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement I can share publication information and abstracts for the 788 papers evaluated for inclusion in this study.