Opportunistic evaluations

Done well and systematically, evaluative studies that are conducted alongside programmes that are happening anyway serve the interests of patients, the public and the healthcare system by producing systematic knowledge about the fate and impact of improvement and change programmes.4 We term these studies ‘opportunistic evaluations’, and define them as follows:

Opportunistic evaluations study programmes, initiatives, policies, and interventions that would occur whether or not there were any concurrent, coincident, or otherwise related research activities designed to produce knowledge from the programme. Opportunistic evaluations are systematic studies that make use of naturally arising opportunities to study the effects and costs of those programmes and how they work.

Opportunistic evaluations can make invaluable contributions to advancing knowledge about programme design, implementation, mechanisms of change and outcomes. For example, they can assist with determining effectiveness, identifying the influences on effectiveness, characterise the mechanisms of action, support replication and scaling of successful programmes, and reduce waste and costs associated with unsuccessful programmes by demonstrating what should not be supported in the future. It is especially important that initiatives and programmes be evaluated if they are novel, expensive, have the potential for significant impact, or there are plausible reasons to evaluate them in a new context.5 6 In such situations, there is a strong public interest argument for evaluation.

The Ottawa statement on the ethical design and conduct of cluster randomised trials,3 however, makes little of acknowledgement of a scenario where the researchers have not initiated the programme. For example, it draws on the principle of clinical equipoise to require researchers to ‘ensure that the study intervention is adequately justified.’ While this might be an appropriate safeguard when researchers design interventions for purposes of generating new knowledge, it is unduly restrictive when the intervention is wholly owned in the service. Indeed, the public interest in evaluating an intervention may arise because there is a lack of equipoise. For example, in the USA the Centers for Medicare and Medicaid Services recently invited states to establish work requirements for Medicaid benefits. A number of physician organisations and other groups have argued that these requirements are likely to lead to financial and health harms for patients.7 Prospectively examining the impact of this policy is essential to fully informing future debate.

The point that interventions may not be initiated by researchers but nonetheless warrant study is exemplified by the demand for evaluation from those who have designed, commissioned or implemented programmes and policies and the usefulness of the evidence such evaluations generate. As an example, 24 hospitals across the UK participated in the Safer Patients Programme, which was commissioned by the Health Foundation, a UK charitable foundation. Those hospitals took part in the programme in order to improve their systems, practices and culture—not to produce knowledge. An evaluation of the programme did not change anything that the organisations were going to do anyway, but it did show that, on average, those organisations did not improve more than organisations that were not participating in the programme, thus producing very useful learning about whether the programme was a suitable candidate for investment by the National Health Service (NHS).8 9

Another example is the evaluation of the reconfiguration of stroke services in two major UK cities: London and Manchester. The evaluation again was wrapped around a programme that was happening anyway, and produced important learning about centralised models, showing that they can reduce mortality and length of stay.10 The evaluation of the Mexican Progresa/Oportunidades programme, which included conditional cash transfers for certain health and educational outcomes, provides another illustration. Randomisation was built into the government’s implementation of the programme to enable experimental evaluation without affecting the policy itself or the eventual recipients. A great deal of valuable research has been published on the basis of this roll-out and subsequent follow-up studies on the effects of conditional cash transfer programmes, which now over 60 countries implement.11

A defining feature of opportunistic evaluations, in these kinds of scenarios, is that those studying the programme do not control the change under study: the evaluators do not conceive of, initiate, deliver, or affect the programme of interventions or other changes. Yet current governance and ethics requirements may be problematic or ill suited to the goals either of evaluation or to organisations-seeking improvement, since they typically assume that the evaluator is controlling many more aspects of the programme than is usually the case.

Study designs for opportunistic evaluations

In identifying the challenges associated with opportunistic evaluations, it is useful first to expand on what constitutes an ‘opportunistic evaluation’. When a programme is designed and led by researchers, even if implemented by policymakers, the researcher bears responsibility, and the normal rules of ethical engagement should apply. For instance, a service innovation to test an intervention to provide more patient-centred care12 clearly falls into this category, since the participating professionals and organisations would not have implemented it without a research project. Our concern is not with these researcher-led programmes, but where the researcher has not designed or initiated the programme. The distinction has analogies with the distinction between prospective and retrospective studies.

With a retrospective study of a programme not initiated by a researcher, the programme lies in the past and the researcher cannot be assumed to have responsibility for it. Nevertheless, these studies can take advantage of the opportunities afforded by how programmes were rolled out. For example, the lottery mechanism used to allocate limited sign-up opportunities for Medicaid in Oregon, USA, afforded researchers an opportunity to examine the effects of healthcare coverage that took advantage of a randomisation process that had been deployed for reasons other than research, and over which the researchers had no control.13 14 Likewise, allocation of permits by lottery to allow Fijians to emigrate to New Zealand allowed researchers to study the effects of migration on health and well-being many years later.15 16 Other studies may take the form of natural experiments: observational studies where a researcher ‘can make a credible claim that the assignment of the nonexperimental subjects to treatment and control conditions is ‘as if’ random.’17 For instance, a study in Canada of the impact of a pay-for-performance scheme, where some but not all of the physicians were exposed to the incentives, allowed a quasiexperimental evaluation where no aspect of the assignment of the physicians to the programme was controlled by the researchers.18

Retrospective evaluations of programmes that have already occurred clearly absolve researchers from responsibility for the programme itself, but they do not absolve researchers of ethical responsibility for the research activities they conduct. As for all studies, researchers must obtain ethics committee approval for all the things that they plan to do in their roles as researchers: data collection, analysis, protection of rights of participants, and so on. They have unassailable duties to protect anonymity, properly interpret data and avoid ‘over-claiming’. They must not fabricate or plagiarise data, and should place their work in the public domain.

These obligations apply equally to prospective studies of programmes that are not researcher initiated. Prospective study designs are often highly beneficial to science and learning. They may, for example, enable hypotheses to be specified a priori, thus mitigating the risks of data-driven comparisons. Evaluators can support augmentation of the information to be collected—for example, qualitative data to explicate implementation fidelity, or surveys of participants’ views, thus maximising the learning from the programme. A statistical analysis plan can also be specified, thus enhancing transparency and rigour. However, a number of ambiguities complicate clarity about the ethical duties of the evaluators in prospective opportunistic evaluations, which vary somewhat across two types of study.

In the first, the evaluator has no influence on the programme at all, but uses available data and the design of the programme to mount a study prospectively. As an example, an opportunistic evaluation of the 7 Day Services policy implemented across the NHS in England and Wales, which aimed to increase specialist availability in hospitals at the weekend, is examining a policy that was designed and implemented by the Department of Health and NHS England independently of the evaluation. The evaluation involves a review of patient case notes from before and after the policy implementation alongside qualitative studies during the implementation itself.19

The second study type is one where the programme is not initiated by researchers but they help to design the implementation (not the intervention) in a way that facilitates evaluation, for example, by proposing to programme leaders that study design such as wait-list, stepped-wedge or cluster randomised designs be adopted in situations where use of such strategies is consistent with the goals of the programme. This approach is often true when a programme cannot be rolled out to all possible sites at once, and much scientific value can be obtained by the order of roll-out to be determined at random.20 Importantly, though, this still meets our definition of an opportunistic evaluation in that the intervention would still occur without the involvement of researchers. Such was the case with the aforementioned Progresa/Oportunidades programme, a poverty alleviation programme in Mexico, which incorporated a randomised implementation to permit rigorous evaluation.11 Another example was the evaluation of the Matching Michigan programme in the English NHS, which incorporated a stepped (although non-random) roll-out.21

Very often, the most convenient and robust prospective study design for programmes involves allocation of interventions at the cluster level (eg, village, school, hospital ward) rather than the individual level. In the current literature, including the Ottawa declaration, cluster randomised studies are considered to pose distinctive ethical issues.22 However, many of these issues are not relevant to opportunistic evaluations, since there is little that is ethically fraught about randomising roll-out in a situation where every site is going to receive the programme anyway, where they cannot all receive it at once and where the role of the researchers is to support roll-out in a manner that optimises learning.

Ethical debates about cluster randomisation

The most influential guidance on the ethics of cluster randomisation is found in the Ottawa Statement on the design and conduct of cluster randomised trials.3 Its emphases are largely consistent with most current codifications of ethical practice,23–26 yet they make for a remarkably poor fit with the realities of opportunistic evaluations.

One evident problem is that the Ottawa statement calls on the researcher to justify the rationale for the intervention and demonstrate equipoise.3 But, for opportunistic evaluations, the researchers may not be well placed to supply the rationale (after all, it is not their choice), much less justify it, especially since their role may be one of surfacing the rationale and assessing its soundness.27

The Ottawa declaration also holds that the researcher is responsible for obtaining informed consent for the intervention from participants or if the intervention has minimal harm, obtain a waiver of consent. However, it should not be assumed that planned programmes are free of an a priori expectation of harm from the intervention. Consider again the Medicaid work requirements policy discussed above. A waiver of consent would not be granted for this intervention; indeed, it is unlikely participants would even consent to the intervention, and yet it would still occur regardless of the consent process for research. While the researcher can properly be held responsible for obtaining informed consent for any data collection for research purposes, this responsibility should not extend to the intervention itself, since it is implemented under a policy or leadership mandate—it would be strange indeed for a researcher to seek consent from members of a cluster for an intervention that the policymaker had decided to introduce regardless of any evaluation.

It has also been argued that researchers must justify the choice of control condition in cluster studies.3 22 While it is correct that the researcher, given a choice of possible controls, should select the controls to maximise their scientific value, the researcher cannot be held responsible for the fact that controls have not received the intervention, since it is the policymaker, not the researcher, who decides where and when to intervene (and where not to do so). As such, controls may not meet the high standard of a trial where the researcher has full control of all aspects of the intervention. Insisting that the intervention must receive ethical approval simply because it is part of an evaluative study does not solve the problem. It would be perverse if, by agreeing to an evaluation of an intervention, a government minister claiming a democratic mandate or a hospital chief acting on behalf of their board, then had to subject the intervention to a research ethics committee, when they could otherwise proceed unhindered: an example of the adage that ‘you can do anything you like, as long as you promise not to learn from it.’

The consequences of the poor alignment between ethical recommendations and the specificities of improvement evaluations are highly practical and far reaching. For instance, the assumption that the researcher controls any prospectively evaluated intervention is so strongly entrenched that it is difficult to register an evaluation of a policymaker’s intervention on ClinicalTrials.gov, the first, largest and most widely used clinical trials registry.28 If individuals are not ‘assigned by an investigator… to receive specific interventions,’ then the study must be classified as observational, according to their guidelines. Moreover, having entered ‘observational’ the study cannot then be classified as prospective, but must be classified as retrospective in order to be accepted by the system. Thus, a prospective stepped-wedge randomised controlled trial in which researcher involvement with the intervention and its implementation is limited only to randomising the order of units in the roll-out must be classified as a retrospective, observational study. The implication is that an investigator cannot mount a prospective study of an intervention implemented by a policymaker or programme leader. The pre-registration of a trial is an essential component to ensuring its validity. The impossibility of registering the type of study we describe here on the largest trials registry is indicative of the lack of consideration given to the ‘grey area’ of opportunistic evaluation and further serves as another hindrance to conducting valid studies with appropriate oversight.

Negotiating through these ambiguities leads to delays that may be highly consequential for real-time improvement programmes, to queries at organisation level about cost recovery (because of the difficulties of distinguishing programme activities from evaluation activity) and to inappropriate requirements for local collaborators/principal investigators. Before we suggest how this difficulty might be avoided, we need to examine the ethical principle on which we shall rely.

Ethics of opportunistic evaluations

Many of the problems we describe above are not resolved by classifying opportunistic evaluations as audit, quality improvement, or service evaluation, which the UK Health Research Authority specifies as comprising projects which are ‘designed and conducted solely to define or judge current care.’29 Such projects may not have sufficiently robust governance or ethical oversight and are subject to widely varying local arrangements. A further important problem is that where opportunistic evaluations are not classified as research, the ability to publish findings and accumulate knowledge is undermined, and the credibility and impact of the evaluation on practice and policy is weakened. Yet a significant percentage of evaluations are classified as ‘service evaluations’: Chen and Fawcett note that South East Scotland Research Ethics Service estimates it gave advice on 1300 studies between 2010 and 2015 of which 70% were classified as ‘not research’.30 This is not to say that all studies should be classified as research, merely that there is strong reason to believe that too many studies are classified as non-research in order to avoid certain regulatory or ethical barriers.

Once it is determined that an opportunistic evaluation counts as research, our basic principle is that a person can only assume (moral) responsibility, or be held accountable, for something that occurs as a result of their action (or absence of action).31 The responsibility for an intervention therefore lies with those whose action led to the intervention being designed and implemented. Clinical innovators, such as clinicians or pharmaceutical companies, are held accountable through strict ethical scrutiny, which grants authority to intervene.22 32 The authority of those who initiate improvement and change programmes is different, deriving, for example, from democratic mandate. In contrast, the researcher can only bear responsibility for, and hence requires scrutiny for, an intervention when she has participated in designing or implementing it. This position derives from the Kantian principle that ‘ought implies can’33: to be held accountable, a moral agent must know of the standards she is expected to meet, be charged with responsibility for meeting those standards, and have sufficient autonomy and capacity in her choice of actions, and access to resources, to be able to comply. When a researcher is unable to influence programme design or implementation, she is not accountable for it.

We propose that, for opportunistic evaluations, the policymaker/programme leader is responsible for the design and implementation of the programme: the evaluator is not (she has other responsibilities, but not for the programme). A conceptual litmus test would be to determine what would occur in the counterfactual scenario of no evaluation taking place. If, in the absence of evaluation, the implementation proceeds in the same way, it follows that the researcher should not be the responsible agent when it comes to programmes.

This is, of course, not to argue that a researcher qua researcher can simply abandon their obligations qua morally responsible agent. One challenge arises, for example, in circumstances when it might be morally indefensible for researchers to conduct an evaluation at all—for example, when a programme is highly likely or certain to cause significant or severe harm—because of the risk that the evaluation of a programme might vicariously endorse an immoral act. The question of whether a researcher ought to participate in a given evaluation is one that should be judged on a case-by-case basis, given that an evaluation might be ethical even if a programme is not (since the evaluation may, for instance, evidence harm that would otherwise remain concealed). The key point, for purposes of our argument, is that the current ethical guidance for cluster trials requires that the evaluation and intervention be justified on grounds of equipoise, minimal harm, and so forth, which would rule out many potentially beneficial evaluations of programmes. Given evidence that implementation of an untested policy based on intuition about what works may be less likely to invite objection from the public than rigorous evaluation of two or more otherwise unobjectionable policies,34 the risk is that policymakers or others continue to see imposing a programme on the whole population without evaluation as less problematic (and less trouble) than introducing it for half, and learning in the process. To safeguard the public interest, a new approach to the ethics of opportunistic evaluations is needed. We propose that the approvals necessary for a researcher should be limited only to factors over which the researcher has control. We provide an example of the prospective evaluation of an intervention deployed by policymakers in box 1.