Step 1: evaluating clinical promise using efficacy evidence

Preclinical efficacy studies set out to provide evidence of clinical promise. They do this by supporting inferences that a candidate intervention causes treatment effects in a model setting, and that these cause-and-effect relationships generalise to clinical settings. All else being equal, interventions that cause larger effects in animals have greater clinical promise. However, inferences about clinical promise can be threatened in three different ways, and evaluation of preclinical studies begins by determining the extent to which these threats have been addressed during study design and execution.

First, preclinical efficacy studies may introduce biases or random errors that lead to spurious causal inferences; these are called ‘threats to internal validity’. For instance, unless blinded to treatment allocation in an experiment, investigators committed to a hypothesis can unknowingly attend more closely to therapeutic responses in treated animals. Second, efficacy studies can fail clinical generalisation if investigators mischaracterise the relationship between a preclinical study and an ensuing trial. These mischaracterisations are called ‘construct validity threats’. A researcher might mischaracterise a drug's utility for chronic human disease if it was tested only on animals during acute illness. Third, efficacy studies can fail clinical generalisation because there is something idiosyncratic about test conditions that fail to carry over to trials. These artefacts represent ‘threats to external validity’. A drug might fail to generalise because of differences in animal physiology or laboratory practices that interact with treatment effects.

The first task in assessing the clinical promise begins with evaluators thoroughly retrieving all available and relevant efficacy studies on an intervention (though investigators often reference the preclinical evidence in trial brochures, this information can be abbreviated and selective). Evaluators should then determine the magnitude of treatment effects, and interrogate preclinical studies to determine the extent to which preclinical efficacy studies have addressed all three threats to valid clinical generalisation. We recently performed a systematic review of guidelines for designing clinically generalisable efficacy studies, and identified 26 guidelines pertaining to 11 broad disease areas, such as neurological and cerebrovascular diseases or cardiac and circulatory diseases. We also produced a minimal set of 14 consensus recommendations for the three types of validity threat.8 If a trial involves a disease area addressed by a focused guideline, evaluators can evaluate the clinical promise for a given intervention based on whether methods in the guideline(s) have been implemented. Alternatively or in combination with specific guidelines, reviewers can tap the collective wisdom of the preclinical research community and draw on our 14 consensus recommendations.

In particular, reviewers should assess the extent to which efficacy studies have minimised threats to internal validity by asking at least the following six questions:

(1) Was the sample size sufficient to exclude random variation as an explanation of effect sizes?

(2) Were animals randomly assigned to treatment?

(3) Were outcome assessors blinded?

(4) Do studies explain the flow of animals from inclusion through to analysis?

(5) Were appropriate controls included?

(6) Was a dose–response relationship demonstrated?

Reviewers can address the extent to which construct validity threats were addressed by at least examining whether, (1) animals were characterised at baseline and included based on prespecified eligibility criteria, (2) animal models used are widely believed to offer the most faithful and available representation of the human disease that will be tested in trials, (3) studies establish a mechanism of action, (4) outcomes used in studies are good representations of clinical outcomes used in trials and (5) animal age is matched to age of patients (often, it is possible to scale an animal's age to that of a human being, based on developmental milestones, longevity, etc; see, for instance, ref. 9).

The extent to which external validity threats have been addressed can be assessed by determining whether efficacy studies were replicated in (1) more than one model, (2) more than one laboratory and (3) more than one species. The importance of testing in more than one system is underscored by the fact that animal models are often flawed representations of human disease, and may capture a narrow set of disease phenomena. Observing activity in multiple models provides greater confidence that the activities are robust and address more fundamental disease processes.

Failure to implement any one of the above practices in the context of otherwise promising effect sizes should qualify a reviewer's confidence that activities in animal studies will generalise to patients.

Step 2: combining with reference class evidence

At this point, a reviewer will have formed an opinion about a drug's clinical promise using efficacy evidence alone. However, drug candidates backed by strong animal evidence frequently fail human trials; conversely, drug candidates backed by weak animal evidence may succeed in well-understood realms (eg, ‘me too drugs’). Often, this is because animal models or the circumstances of their use are poor representations of the clinical scenarios. The second step is to adjust assessments of clinical promise from efficacy studies by examining how other agents in the same class—and supported by similar evidence—have fared in clinical development. If most drugs in the same reference class have been successfully translated on similar evidence (a sign that animal models used are predictive), this provides grounds for confidence that responses in animals will be recapitulated in human beings. If most drugs in the same reference class have not demonstrated efficacy in trials (a sign that animal models used are not predictive), this provides grounds for caution in projecting clinical responses. For instance, imagine researchers are pursuing a cancer drug that inhibits a particular receptor. If several drugs targeting the same receptor were supported by similar preclinical evidence, but failed translation, reviewers should be more guarded in their estimation of clinical promise. A discussion of this step—and its relationship with models of disease—is described elsewhere.10

The next task is to feed estimates of clinical promise into evaluations of risk/benefit. A thorough account of this task is well beyond the scope of this paper, and has been discussed by other thoughtful commentators, including in early phase clinical development.11–13 We, nevertheless, offer four broad factors that should inform the moral evaluation of risk/benefit. The first is the moral justification for risks of an intervention. If an intervention implicates care obligations (eg, an intervention will be substituted for an established effective therapy), then the ethical permissibility of its risks is to be judged on whether the clinical promise is competitive with the standard of care. In the previous work, one of us has argued that the moral justification for exposing patients to novel interventions in early phase studies will generally rest on the value of the knowledge a study is expected to produce, not therapeutic value.14 This then prompts a second consideration: risk to subjects, which is related to subjects’ medical options and expected research burdens. A third consideration is the degree to which a research programme opens up new avenues for investigation. For highly novel intervention platforms or targets, trials can often turn up findings that are applicable across other translation efforts even if they fail to recapitulate preclinical studies. A last factor is the expected value to healthcare systems should the intervention translate successfully. Studies involving interventions that are believed to have transformative potential—or that address priority unmet health needs—permit greater subject risk than those pursuing incremental innovations.12 ,13 Our proposal for evaluating risk/benefit using efficacy studies is depicted in figure 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Process for assessing clinical promise and risk/benefit during trial evaluation.