The Adaptive designs CONSORT Extension (ACE) statement: a checklist with explanation and elaboration guideline for reporting randomised trials that use an adaptive design

Section 1. Title and abstract

• CONSORT 2010 item 1b: Structured summary of trial design, methods, results, and conclusions (for specific guidance see CONSORT for abstracts136137)

• ACE item 1b: Structured summary of trial design, methods, results, and conclusions (for specific guidance see ACE for abstracts, table 3)

Explanation—A well structured abstract summary encompassing trial design, methods, results, and conclusions is essential regardless of the type of design implemented.137 This allows readers to search for relevant studies of interest and to quickly judge if the reported trial is relevant to them for further reading. Furthermore, it helps readers to make instant judgements on key benefits and risks of study interventions. Table 3 presents minimum essential items authors should report in an AD randomised trial abstract. Authors should use this extension together with the CONSORT for journal and conference abstracts for additional details136137 and other relevant extensions where appropriate.

• CONSORT abstract item (Trial design): Description of the trial design (for example, parallel, cluster, non-inferiority)

• ACE abstract item (Trial design): Description of the trial design (for example, parallel, cluster, non-inferiority); include the word “adaptive” in the content or at least as a keyword

Explanation—AD randomised trials should be indexed properly to allow other researchers to easily retrieve them in literature searches. This is particularly important as trial design may influence interpretation of trial findings and the evidence synthesis approach used during meta-analyses. The MEDLINE database provides “Adaptive clinical trial” as a Medical Subject Heading (MeSH) topic to improve indexing.139 Authors may also like to state the type of the AD, including details of adaptations as covered under the new item 3b (table 3). See box 5 for exemplars.

• CONSORT/ACE abstract item (Outcome): Clearly defined primary outcome for this report

Explanation—In some AD randomised trials, the outcome used to inform adaptations (adaptation outcome) and the primary outcome of the study can differ (see item 6 of the main checklist for details). The necessity of reporting both of these outcomes and results in the abstract depends on the stage of reporting and whether the adaptation decisions made were critical to influencing the interpretation of the final results. For example, when a trial or at least a treatment group is stopped early, based on an adaptation outcome which is not the primary outcome, it becomes essential to adequately describe both outcomes in accordance with the CONSORT 2010 statement.34 Contrarily, only the description of the primary outcome in the abstract will be essential when non-terminal adaptation decisions are made (such as to change the sample size, update randomisation, or no dropping of treatments groups at interims) and when final (not interim) results are being reported. Furthermore, the results item (table 3) should be reported consistent with the stated primary and adaptation outcome(s), where necessary. See box 6 for exemplars.

• ACE abstract item (adaptation decisions made): Specify what trial adaptation decisions were made in light of the pre-planned decision-making criteria and observed accrued data

Explanation—A brief account of changes that were made to the trial, on what basis they were made, and when is important. The fact that the design allows for adaptations will influence interpretation of results, potentially due to operational and statistical biases. If changes should have been made, but were not, then this may further influence credibility of results. See the main checklist item 14c for details. See box 7 for exemplars.

Section 3: Methods (Trial design)

• ACE item 3b (new): Type of adaptive design used, with details of the pre-planned adaptations and the statistical information informing the adaptations

Explanation—A description of the type of AD indicates the underlying design concepts and the applicable adaptive statistical methods. Although there is an inconsistent use of nomenclature to classify ADs, together with growing related methodology,13 some currently used types of ADs are presented in table 1. A clear description will also improve the indexing of AD methods and for easy identification during literature reviews.

Specification of pre-planned opportunities for adaptations and their scope is essential to preserve the integrity of AD randomised trials22 and for regulatory assessments, regardless of whether they were triggered during the trial.14108109 Details of pre-planned adaptations enable readers to assess the appropriateness of statistical methods used to evaluate operating characteristics of the AD (item 7a) and for performing statistical inference (item 12b). Unfortunately, pre-planned adaptations are commonly insufficiently described.119 Authors are encouraged to explain the scientific rationale for choosing the considered pre-planned adaptations encapsulated under the CONSORT 2010 item “scientific background and explanation of rationale” (item 2a). This rationale should focus on the goals of the considered adaptations in line with the study objectives and hypotheses (item 2b).107108119123

Details of pre-planned adaptations with rationale should be documented in accessible study documents for readers to be able to evaluate what was planned and unplanned (such as protocol, interim and final SAP or dedicated trial document). Of note, any pre-planned adaptation that modifies eligibility criteria (such as in population enrichment ADs92146) should be clearly described.

Adaptive trials use accrued statistical information to make pre-planned adaptation(s) (item 14c) at interim analyses guided by pre-planned decision-making criteria and rules (item 7b). Reporting this statistical information for guiding adaptations and how it is gathered is paramount. Analytical derivations of statistical information guiding pre-planned adaptations using statistical models or formulae should be described to facilitate reproducibility and interpretation of results. The use of supplementary material or references to published literature is sufficient. For example, sample size re-assessment (SSR) can be performed using different methods with or without knowledge or use of treatment arm allocation.41424448 Around 43% (15/35) of regulatory submissions needed further clarifications because of failure to describe how a SSR would be performed.119 Early stopping of a trial or treatment group for futility can be evaluated based on statistical information to support lack of evidence of benefit that is derived and expressed in several ways. For example, conditional power,56147148149150 predictive power,55148151152153 the threshold of the treatment effect, posterior probability of the treatment effect,100 or some form of clinical utility that quantifies the balance between benefits against harms154155 or between patient and society perspectives on health outcomes.100 See box 8 for exemplars.

• CONSORT 2010 item 3b: Important changes to the design or methods after trial commencement (such as eligibility criteria), with reasons

• ACE item 3c (modification, renumbered): Important changes to the design or methods after trial commencement (such as eligibility criteria) outside the scope of the pre-planned adaptive design features, with reasons

Explanation—Unplanned changes to certain aspects of the design or methods in response to unexpected circumstances that occur during the trial are common and will need to be reported in AD randomised trials, as in fixed design trials. This may include deviations from pre-planned adaptations and decision rules,1570 as well as changes to timing and frequency of interim analyses. Traditionally, unplanned changes with explanation have been documented as protocol amendments and reported as discussed in the CONSORT 2010 statement.34 Unplanned changes, depending on what they are and why they were made, may introduce bias and compromise trial credibility. Some unplanned changes may render the planned adaptive statistical methods invalid or may complicate interpretation of results.22 It is therefore essential for authors to detail important changes that occurred outside the scope of the pre-planned adaptations and to explain why deviations from the planned adaptations were necessary. Furthermore, it should be clarified whether unplanned changes were made following access to key trial information such as interim data seen by treatment group or interim results. Such information will help readers assess potential sources of bias and implications for the interpretation of results. For ADs, it is essential to distinguish unplanned changes from pre-planned adaptations (item 3b).161 See box 9 for an exemplar.

Section 6. Outcomes

• CONSORT 2010 item 6a: Completely define pre-specified primary and secondary outcome measures, including how and when they were assessed

• ACE item 6a (modification): Completely define pre-specified primary and secondary outcome measures, including how and when they were assessed. Any other outcome measures used to inform pre-planned adaptations should be described with the rationale

Comment—Authors should also refer to the CONSORT 2010 statement34 for the original text when applying this item.

Explanation—It is paramount to provide a detailed description of pre-specified outcomes used to assess clinical objectives including how and when they were assessed. For operational feasibility, ADs often use outcomes that can be observed quickly and easily to inform pre-planned adaptations (adaptation outcomes). Thus, in some situations, adaptations may be based on early observed outcome(s)162 that are believed to be informative for the primary outcome even though different from the primary outcome. The adaptation outcome (such as a surrogate, biomarker, or an intermediate outcome) together with the primary outcome influences the adaptation process, operating characteristics of the AD, and interpretation and trustworthiness of trial results. Despite many potential advantages of using early observed outcomes to adapt a trial, they pose additional risks of making misleading inferences if they are unreliable.163 For example, a potentially beneficial treatment could be wrongly discarded, an ineffective treatment incorrectly declared effective or wrongly carried forward for further testing, or the randomisation updated based on unreliable information.

Authors should therefore clearly describe adaptation outcomes similar to the description of pre-specified primary and secondary outcomes in the CONSORT 2010 statement.34 Authors are encouraged to provide a clinical rationale supporting the use of an adaptation outcome that is different to the primary outcome in order to aid the clinical interpretation of results. For example, evidence supporting that the adaptation outcome can provide reliable information on the primary outcome will suffice. See box 10 for exemplars.

• CONSORT 2010 item 6b: Any changes to trial outcomes after the trial commenced, with reasons

• ACE item 6b (modification): Any unplanned changes to trial outcomes after the trial commenced, with reasons

Comment—Authors may wish to cross-reference the CONSORT 2010 statement34 for background details.

Explanation—Outcome reporting bias occurs when the selection of outcomes to report is influenced by the nature and direction of results. The prevalence of outcome reporting bias in medical research is well documented: discrepancies between pre-specified outcomes in protocols or registries and those published in reports12168169170171; outcomes that portray favourable beneficial effects of treatments and safety profiles being more likely to be reported169; some pre-specified primary or secondary outcomes modified or switched after trial commencement.170 Changes to trial outcomes may also include changes to how outcomes were assessed or measured, when they were assessed, or the order of importance to address objectives.171

Sometimes when planning trials, there is huge uncertainty around the magnitude of treatment effects on potential outcomes viewed acceptable as primary endpoints.36171 As a result, although uncommon, a pre-planned adaptation could include the choice of the primary endpoints or hypotheses for assessing the benefit-risk ratio. In such circumstances, the adaptive strategy should be clearly described as a pre-planned adaptation (item 3b). Authors should clearly report any additional changes to outcomes outside the scope of the pre-specified adaptations including an explanation of why such changes occurred in line with the CONSORT 2010 statement. This will enable readers to distinguish pre-planned trial adaptations of outcomes from unplanned changes, thereby allowing them to judge outcome reporting bias. See box 11 for an exemplar.

Section 7. Sample size and operating characteristics

• CONSORT 2010 item 7a: How sample size was determined

• ACE item 7a (modification): How sample size and operating characteristics were determined

Comments—This section heading was modified to reflect additional operating characteristics that may be required for some ADs in addition to the sample size. Items 3b, 7a, 7b, and 12b are connected so they should be cross-referenced when reporting.

Explanation—Operating characteristics, which relate to the statistical behaviour of a design, should be tailored to address trial objectives and hypotheses, factoring in logistical, ethical, and clinical considerations. These may encompass the maximum sample size, expected sample sizes under certain scenarios, probabilities of identifying beneficial treatments if they exist, and probabilities of making false positive claims of evidence.172173 Specifically, the predetermined sample size for ADs is influenced, among other things, by:

1. Type and scope of adaptations considered (item 3b);

2. Decision-making criteria used to inform adaptations (item 7b);

3. Criteria for claiming overall evidence (such as based on the probability of the treatment effect being above a certain value, targeted treatment effect of interest, and threshold for statistical significance174175);

4. Timing and frequency of the adaptations (item 7b);

5. Type of primary outcome(s) (item 6a) and nuisance parameters (such as outcome variance);

6. Method for claiming evidence on multiple key hypotheses (part of item 12b);

7. Desired operating characteristics (see box 2), such as statistical power and an acceptable level of making a false positive claim of benefit;

8. Adaptive statistical methods used for analysis (item 12b);

9. Statistical framework (frequentist or Bayesian) used to design and analyse the trial.

Information that guided estimation of sample size(s), including operating characteristics of the considered AD, should be described sufficiently to enable readers to reproduce the sample size calculation. The assumptions made concerning design parameters should be clearly stated and supported with evidence if possible. Any constraints imposed (for example, due to limited trial population) should be stated. It is good scientific practice to reference the statistical tools used (such as statistical software, program, or code) and to describe the use of statistical simulations when relevant (see item 24b discussion).

In a situation where changing the sample size is a pre-planned adaptation (item 3b), authors should report the initial sample sizes (at interim analyses before the expected change in sample size) and the maximum allowable sample size per group and in total if applicable. The planned sample sizes (or expected numbers of events for time-to-event data) at each interim analysis and final analysis should be reported by treatment group and overall. The timing of interim analyses can be specified as a fraction of information gathered rather than sample size. See box 12 for exemplars.

Box 12

Exemplars on reporting item 7a elements

• Example 1. MAMS AD; assumptions and adaptive methods; approach for claiming evidence or informing adaptations; statistical program

  • “The primary response (outcome) from each patient is the difference between the baseline HOMA-IR score and their HOMA-IR score at 24 weeks. The sample size calculation is based on a one-sided type I error of 5% and a power of 90%. If there is no difference between the mean response on any treatment and that on control, then a probability of 0.05 is set for the risk of erroneously ending the study with a recommendation that any treatment be tested further. For the power, we adopt a generalisation of this power requirement to multiple active treatments due to Dunnett.176 Effect sizes are specified as the percentage chance of a patient on active treatment achieving a greater reduction in HOMA-IR score than a patient on control as this specification does not require knowledge of the common SD, σ. The requirement is that, if a patient on the best active dose has a 65% chance of a better response than a patient on control, while patients on the other two active treatments have a 55% chance of showing a better response than a patient on control, then the best active dose should be recommended for further testing with 90% probability. A 55% chance of achieving a better response on active dose relative to control corresponds to a reduction in mean HOMA-IR score of about a sixth of an SD (0.178σ), while the clinically relevant effect of 65% corresponds to a reduction of about half an SD (0.545σ). The critical values for recommending that a treatment is taken to further testing at the interim and final analyses (2.782 and 2.086) have been chosen to guarantee these properties using a method described by Magirr et al,177 generalising the approach of Whitehead and Jaki.178 The maximum sample size of this study is 336 evaluable patients (84 per arm), although the use of the interim analysis may change the required sample size. The study will recruit additional patients to account for an anticipated 10% dropout rate (giving a total sample size of 370). An interim analysis will take place once the primary endpoint is available for at least 42 patients on each arm (i.e., total of 168, half of the planned maximum of 336 patients). Sample size calculation was performed using the MAMS package in R179.”57

• Example 2. 3-arm 2-stage AD with dose selection; group sequential approach; assumptions; adaptation decision-making criteria; stage 1 and 2 sample sizes; use of simulations

  • “Sample size calculations are based on the primary efficacy variable (composite of all-cause death or new MI through day 7), with the following assumptions: an event rate in the control group of 5.0%, based on event rates from the phase II study (24); a relative risk reduction (RRR) of 25%; a binomial 1-sided (α=0.025) superiority test for the comparison of 2 proportions with 88% power; and a 2-stage adaptive design with one interim analysis at the end of stage 1 data (35% information fraction) to select 1 otamixaban dose for continuation of the study at stage 2. Selection of the dose for continuation was based on the composite end point of all-cause death, Myocardial Infarction (MI), thrombotic complication, and the composite of Thrombosis in Myocardial Infarction (TIMI) major bleeding through day 7, with an assumed probability for selecting the “best” dose according to the primary endpoint (r=0.6), a group sequential approach with futility boundary of relative risk of otamixaban versus UFH plus eptifibatide ≥1.0, and efficacy boundary based on agamma (−10) α spending function.180 Based on the above assumptions, simulations (part of item 24b, see supplementary material) showed that 13 220 patients (a total of 5625 per group for the 2 remaining arms for the final analysis) are needed for this study.”181 See figure 1.

Fig 1

Adapted from Steg et al.197

• Download figure
• Open in new tab
• Download powerpoint

RETURN TO TEXT

• CONSORT 2010 item 7b: When applicable, explanation of any interim analyses and stopping guidelines

• ACE item 7b (replacement): Pre-planned interim decision-making criteria to guide the trial adaptation process; whether decision-making criteria were binding or non-binding; pre-planned and actual timing and frequency of interim data looks to inform trial adaptations

Comments—This item is a replacement so when reporting, the CONSORT 20103 item 7b content should be ignored. Items 7b and 8b overlap, but we intentionally reserved item 8b specifically to enhance complete reporting of ADs with randomisation updates as a pre-planned adaptation. Reporting of these items is also connected to items 3b and 12b.

Explanation—Transparency and complete reporting of pre-planned decision-making criteria (box 2) and how overall evidence is claimed are essential as they influence operating characteristics of the AD, credibility of the trial, and clinical interpretation of findings.2232182

A key feature of an AD is that interim decisions about the course of the trial are informed by observed interim data (element of item 3b) at one or more interim analyses guided by decision rules describing how and when the proposed adaptations will be activated (pre-planned adaptive decision-making criteria). Decision rules, as defined in box 2, may include, but are not limited to, rules for making adaptations described in table 1. Decision rules are often constructed with input of key stakeholders (such as clinical investigators, statisticians, patient groups, health economists, and regulators).183 For example, statistical methods for formulating early stopping decision rules of a trial or treatment group(s) exist.5152184185186187

Decision boundaries (for example, stopping boundaries), pre-specified limits or parameters used to determine adaptations to be made, and criteria for claiming overall evidence of benefit and/or harm (at an interim or final analysis) should be clearly stated. These are influenced by statistical information used to inform adaptations (item 3b). Decision trees or algorithms can aid the representation of complex adaptive decision-making criteria.

Allowing for trial adaptations too early in a trial with inadequate information severely undermines robustness of adaptive decision-making criteria and trustworthiness of trial results.188189 Furthermore, methods and results can only be reproducible when timing and frequency of interim analyses are adequately described. Therefore, authors should detail when and how often the interim analyses were planned to be implemented. The planned timing can be described in terms of information such as interim sample size or number of events relative to the maximum sample size or maximum number of events, respectively. For example, in circumstances when the pre-planned and actual timing or/and frequency of the interim analyses differ, reports should clearly state what actually happened (item 3c).

Clarification should be made on whether decision rules were binding or non-binding to help assess implications in the case when they were overruled or ignored. For example, when a binding futility boundary is overruled and a trial is continued, this would lead to a type I error inflation. Non-binding decision rules are those that can be overruled without having a negative effect on the control of the type I error rate. Use of non-binding futility boundaries is often advised.55 See box 13 for exemplars.

Additional examples on the use of non-binding futility boundaries and a cap on sample size following SSR and treatment selection are given in Appendix B.

Section 8. Randomisation (Sequence generation)

• CONSORT 2010 item 8b: Type of randomisation; details of any restriction (such as blocking and block size)

• ACE item 8b (modification): Type of randomisation; details of any restriction (such as blocking and block size); any changes to the allocation rule after trial adaptation decisions; any pre-planned allocation rule or algorithm to update randomisation with timing and frequency of updates

Comments—In applying this item, the reporting of randomisation aspects before activation of trial adaptations must adhere to CONSORT 2010 items 8a and 8b. This E&E document only addresses additional randomisation aspects that are essential when reporting any AD where the randomisation allocation changes. Note that the contents of extension items 7b and 8b overlap.

Explanation—In AD randomised trials, the allocation ratio(s) may remain fixed throughout or change during the trial as a consequence of pre-planned adaptations (for example, when modifying randomisation to favour treatments more likely to show benefits, after treatment selection, or upon introduction of a new arm to an ongoing trial).73 Unplanned changes may also change allocation ratios (for example, after early stopping of a treatment arm due to unforeseeable harms).

This reporting item is particularly important for response-adaptive randomisation (RAR) ADs as several factors influence their efficiency and operating characteristics, which in turn influence the trustworthiness of results and necessitate adequate reporting.13196197198199 For RAR ADs, authors should therefore detail the pre-planned:

1. Burn-in period before activating randomisation updates, including the period when the control group allocation ratio was fixed;

2. Type of randomisation method with allocation ratios per group during the burn-in period as detailed in the standard CONSORT 2010 item 8b;

3. Method or algorithm used to adapt or modify the randomisation allocations after the burn-in period;

4. Information used to inform the adaptive randomisation algorithm and how it was derived (item 3b). Specifically, when a Bayesian RAR is used, we encourage authors to provide details of statistical models and rationale for the prior distribution chosen;

5. Frequency of updating the allocation ratio (for example, after accrual of a certain number of participants with outcome data or defined regular time period) and;

6. Adaptive decision-making criteria to declare early evidence in favour or against certain treatment groups (part of item 7b).

In addition, any envisaged changes to the allocation ratio as a consequence of other trial adaptations (for example, early stopping of an arm or addition of a new arm) should be stated. See box 14 for exemplars.

Section 11. Randomisation (Blinding)

• ACE item 11c (new): Measures to safeguard the confidentiality of interim information and minimise potential operational bias during the trial

Explanation—Preventing or minimising bias is central for robust evaluation of the beneficial and harmful effects of interventions. Analysis of accumulating trial data brings challenges regarding how knowledge or leakage of information, or mere speculation about interim treatment effects, may influence behaviour of key stakeholders involved in the conduct of the trial.22122200 Such behavioural changes may include differential clinical management; reporting of harmful effects; clinical assessment of outcomes; and decision-making to favour one treatment group over the other. Inconsistencies in trial conduct before and after adaptations have wide implications that may affect trial validity and integrity.22 For example, use of statistical methods that combine data across stages may become questionable or may make overall results uninterpretable. AD randomised trials whose integrity was severely compromised by disclosure of interim results have resulted in regulators questioning the credibility of conclusions.201202 Most AD randomised trials, 76% (52/68)49 and 60% (151/251),112 did not disclose methods to minimise potential operational bias during interim analyses. The seriousness of this potential risk will depend on various trial characteristics, and the purpose of having disclosure is to enable readers to judge the risk of potential sources of bias, and thus judge how trustworthy they can assume results to be.

The literature covers processes and procedures which could be considered by researchers to preserve confidentiality of interim results to minimise potential operational bias.45123203 There is no universal approach that suits every situation due to factors such as feasibility; nature of the trial; and available resources and infrastructure. Some authors discuss roles and activities of independent committees in adaptive decision-making processes and control mechanisms for limiting access to interim information.203204205

Description of the process and procedures put in place to minimise the potential introduction of operational bias related to interim analyses and decision-making to inform adaptations is essential.22125203 Specifically, authors should give consideration to:

1. Who recommended or made adaptation decisions. The roles of the sponsor or funder, clinical investigators, and trial monitoring committees (for example, independent data monitoring committee or dedicated committee for adaptation) in the decision-making process should be clearly stated;

2. Who had access to interim data and performed interim analyses;

3. Safeguards which were in place to maintain confidentiality (for example, how the interim results were communicated and to whom and when).

See box 15 for exemplars.

Section 12. Statistical methods

• CONSORT 2010 item 12a: Statistical methods used to compare groups for primary and secondary outcomes

• ACE item 12a (modification): Statistical methods used to compare groups for primary and secondary outcomes, and any other outcomes used to make pre-planned adaptations

Comment—This item should be applied with reference to the detailed discussion in the CONSORT 2010 statement.34

Explanation—The CONSORT 2010 statement34 addresses the importance of detailing statistical methods to analyse primary and secondary outcomes at the end of the trial. This ACE modified item extends this to require similar description to be made of statistical methods used for interim analyses. Furthermore, statistical methods used to analyse any other adaptation outcomes (item 6) should be detailed to enhance reproducibility of the adaptation process and results. Authors should focus on complete description of statistical models and aspects of the estimand of interest206207 consistent with stated objectives and hypotheses (item 2b) and pre-planned adaptations (item 3b).

For Bayesian ADs, item 12b (paragraph 6) describes similar information that should be reported for Bayesian methods.

See box 16 for exemplars.

• ACE item 12b (new): For the implemented adaptive design features, statistical methods used to estimate treatment effects for key endpoints and to make inferences

Comments—Note that items 7a and 12b are connected. Key endpoints are all primary endpoints as well as other endpoints considered highly important, for example, an endpoint used for adaptation.

Explanation—A goal of every trial is to provide reliable estimates of the treatment effect for assessing benefits and risks to reach correct conclusions. Several statistical issues may arise when using an AD depending on its type and the scope of adaptations, the adaptive decision-making criteria and whether frequentist or Bayesian methods are used to design and analyse the trial.22 Conventional estimates of treatment effect based on fixed design methods may be unreliable when applied to ADs (for example, may exaggerate the patient benefit).96209210211212213 Precision around the estimated treatment effects may be incorrect (for example, the width of confidence intervals may be incorrect). Other methods available to summarise the level of evidence in hypothesis testing (for example, p-values) may give different answers. Some factors and conditions that influence the magnitude of estimation bias have been investigated and there are circumstances when it may not be of concern.209214215216217218 Secondary analyses (for example, health economic evaluation) may also be affected if appropriate adjustments are not made.219220 Cameron et al221 discuss methodological challenges in performing network meta-analysis when combining evidence from randomised trials with ADs and fixed designs. Statistical methods for estimating the treatment effect and its precision exist for some ADs68222223224225226227228229230231 and implementation tools are being developed.82232233234 However, these methods are rarely used or reported and the implications are unclear.49209235 Debate and research on inference for some ADs with complex adaptations is ongoing.

In addition to statistical methods for comparing outcomes between groups (item 12a), we specifically encourage authors to clearly describe statistical methods used to estimate measures of treatment effects with associated uncertainty (for example, confidence or credible intervals) and p-value (when appropriate); referencing relevant literature is sufficient. When conventional or naïve estimators derived from fixed design methods are used, it should be clearly stated. In situations where statistical simulations were used to either explore the extent of bias in estimation of the treatment effects (such as181236) or operating characteristics, it is good practice to mention this and provide supporting evidence (item 24c).

ADs tend to increase the risk of making misleading or unjustified claims of treatments effects if traditional methods that ignore trial adaptations are used. In general, this arises when selecting one or more hypothesis test results from a possible list in order to claim evidence of the desired conclusion. For instance, the risks may increase by testing the same hypothesis several times (for example, at interim and final analyses), hypothesis testing of multiple treatment comparisons, selecting an appropriate population from multiple target populations, adapting key outcomes, or a combination of these.22 A variety of adaptive statistical methods exist for controlling specific operating characteristics of the design (for example, type I error rate, power) depending on the nature of the repeated testing of hypotheses.51616282192237238239240241242

Authors should therefore state operating characteristics of the design that have been controlled and details of statistical methods used. The need for controlling a specific type of operating characteristic (for example, pairwise or familywise type I error rate) is context dependent (for example, based on regulatory considerations, objectives and setting) so clarification is encouraged to help interpretation. How evidence of benefit and/or risk is claimed (part of item 7a) and hypotheses being tested (item 2b) should be clear. In situations where statistical simulations were used, we encourage authors to provide a report, where possible (item 24b).

When data or statistical tests across independent stages are combined to make statistical inference, authors should clearly describe the combination test method (for example, Fisher’s combination method, inverse normal method or conditional error function)192240241243244 and weights used for each stage (when not obvious). This information is important because different methods and weights may produce results that lead to different conclusions. Bauer and Einfalt107 found low reporting quality of these methods.

Brard et al245 found evidence of poor reporting of Bayesian methods. To address this, when a Bayesian AD is used, authors should detail the model used for analysis to estimate the posterior probability distribution; the prior distribution used and rationale for its choice; whether the prior was updated in light of interim data and how; and clarify the stages when the prior information was used (interim or/and final analysis). If an informative prior was used, the source of data to inform this prior should be disclosed where applicable. Of note, part of the Bayesian community argue that it is not principled to control frequentist operating characteristics in Bayesian ADs,246 although these can be computed and presented.22154247

Typically, ADs require quickly observed adaptation outcomes relative to the expected length of the trial. In some ADs, randomised participants who have received the treatment may not have their outcome data available at the interim analysis (referred to as overrunning participants) for various reasons.248 These delayed responses may pose ethical dilemmas depending on the adaptive decisions taken, present logistical challenges, or diminish the efficiency of the AD depending on their prevalence and the objective of the adaptations.201 It is therefore useful for readers to understand how overrunning participants were dealt with at interim analyses especially after a terminal adaptation decision (for example, when a trial or treatment groups were stopped early for efficacy or futility). If outcome data of overrunning participants were collected, a description should be given of how these data were analysed and combined with interim results after the last interim decision was made. Some formal statistical methods to deal with accrued data from overrunning participants have been proposed.249

See box 17 for exemplars.

Section 13. Results (Participant flow)

• CONSORT 2010 item 13a: For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome

• ACE item 13a (modification): For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome and any other outcomes used to inform pre-planned adaptations, if applicable

Comments—Authors are referred to the CONSORT 2010 statement34 for detailed discussion. Here, we only address additional requirements for ADs.

Explanation—The CONSORT 2010 statement34 discusses why it is essential to describe participant flow adequately from screening to analysis. This applies to both interim and final analyses depending on the stage of reporting. The number of participants for each group with adaptation outcome data (that contributed to the interim analyses) should also be reported if different from the number of participants with primary outcome data. Furthermore, authors should report the number of randomised participants, for each group, that did not contribute to each interim analysis because of lack of mature outcome data at that interim look. For example, overrunning participants that were still being followed up when a terminal adaptation decision was made (for example, dropping of treatment groups or early trial termination). The presentation of participant flow should align with the key hypotheses (for example, subpopulation(s) and full study population) and treatment comparisons depending on the stage of results being reported.

See box 18 for exemplars.

Section 14. Results (Recruitment)

• CONSORT 2010 item 14a: Dates defining the periods of recruitment and follow-up

• ACE item 14a (modification): Dates defining the periods of recruitment and follow-up, for each group

Comment—Authors should refer to the CONSORT 2010 statement34 for the discussion.

Explanation—Consumers of research findings should be able to put trial results, study interventions, and comparators into context. Some ADs, such as those that evaluate multiple treatments allowing dropping of futile ones, selection of promising treatments, or addition of new treatments to an ongoing trial,19106258259 incorporate pre-planned adaptations to drop or add new treatment groups during the course of the trial. As a result, dates of recruitment and follow-up may differ across treatment groups. In addition, the comparator arm may also change with time and concurrent or non-concurrent controls may be used. There are statistical implications that include how analysis populations for particular treatment comparisons are defined at different stages. For each treatment group, authors should clearly state the exact dates defining recruitment and follow-up periods. It should be stated if all treatment groups were recruited and followed-up during the same period.

See box 19 for exemplars.

Box 19

Exemplars on reporting item 14a

• Example 1. MAMS platform AD

  • Figure 2 illustrates the graphical reporting of recruitment and follow-up periods for each treatment group including new arms that were added during the STAMPEDE trial. Corresponding comparator groups (controls) for treatment comparisons are indicated.

Fig 2

Redrawn from Gilson et al.260 Reused in accordance with the terms of Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). No changes to the original figure were made.

• Download figure
• Open in new tab
• Download powerpoint

• Example 2. Phase 2 Bayesian biomarker-targeted AD with RAR

  • “A total of 341 patients were enrolled in the BATTLE study between November 30, 2006, and October 28, 2009, with equally random assignments for the first 97 patients and adaptive randomization for the remaining 158.”256

RETURN TO TEXT

• CONSORT 2010/ACE item 14b (clarification): Why the trial ended or was stopped

Comment—This item should be applied without reference to the CONSORT 2010 statement.34

Explanation—Some clinical trials are stopped earlier than planned for reasons that will have implications for interpretation and generalisability of results. For example, poor recruitment is a common challenge.261 This may limit the inference drawn or complicate interpretation of results based on insufficient or truncated trial data. Thus, the reporting of reasons for stopping a trial early including circumstances leading to that decision could help readers to interpret results with relevant caveats.

The CONSORT 2010 statement,34 however, did not distinguish early stopping of a trial due to a pre-planned adaptation from an unplanned change. To address this and for consistency, we have now reserved this item for reporting of reasons why the trial or certain treatment arm(s) were stopped outside the scope of pre-planned adaptations, including those involved in deliberations leading to this decision (for example, sponsor, funder, or trial monitoring committee). We also introduced item 14c to capture aspects of adaptation decisions made in light of the accumulating data, such as stopping the trial or treatment arm because the decision-making criterion to do so has been met.

See box 20 for exemplars.

• ACE item 14c (new): Specify what trial adaptation decisions were made in light of the pre-planned decision-making criteria and observed accrued data

Explanation—ADs depend on adherence to pre-planned decision rules to inform adaptations. Thus, it is vital for research consumers to be able to assess whether the adaptation rules were adhered to as pre-specified in the decision-making criteria given the observed accrued data at the interim analyses. Failure to adhere to pre-planned decision rules may undermine the integrity of the results and validity of the design by affecting the operating characteristics (see item 7b for details on binding and non-binding decision rules).

Unforeseeable events can occur that may lead to deviations from some pre-planned adaptation decisions rules (for example, the overruling or ignoring of certain rules). It is therefore essential to adequately describe which pre-planned adaptations were enforced, which were pre-planned but were not enforced or overruled even though the interim analysis decision rules indicated an adaptation should be made, and which unplanned changes were made other than unplanned early stopping of the trial or treatment arm(s) covered by item 14b. Pre-planned adaptations that were not implemented are difficult to assess because the interim decisions made versus the pre-planned intended decisions are often poorly reported, and reasons are rarely given.115 The rationale for ignoring or overruling pre-planned adaptation decisions, or making unplanned decisions that affect the adaptations should be clearly stated and also who recommended or made such decisions (for example, the data monitoring committee or adaptation committee). This enables assessment of potential bias in the adaptation decision-making process, which is crucial for the credibility of the trial.

Authors should indicate the point at which the adaptation decisions were made (that is, stage of results) and any additional design changes that were made as a consequence of adaptation decisions (for example, change in allocation ratio).

See box 21 for exemplars.

Section 15. Results (Baseline data)

• CONSORT 2010 item 15: A table showing baseline demographic and clinical characteristics for each group

• ACE Item 15a «15 (clarification, renumbered): A table showing baseline demographic and clinical characteristics for each group

Comments—We renumbered the item to accommodate the new item 15b. This item should be applied with reference to the CONSORT 2010 statement,34 with additional requirements for specific ADs.

Explanation—The presentation of treatment group summaries of key characteristics and demographics of randomised participants who contributed to results influences interpretation and helps readers and medical practitioners to make judgements about which patients the results are applicable to. For some ADs, such as population (or biomarker or patient) enrichment,87146 when the study population is considered heterogeneous, a trial could be designed to evaluate if study treatments are effective in specific pre-specified subpopulations or a wider study population (full population). A pre-planned adaptation strategy may involve testing the effect of treatments in both pre-specified subpopulations of interest and the wider population in order to target patients likely to benefit the most. For such ADs, it is essential to provide summaries of characteristics of those who were randomised and who contributed to the results being reported (both interim or final), by treatment group for each subpopulation of interest and the full population consistent with hypotheses tested. These summaries should be reported without hypothesis testing of baseline differences in participants’ characteristics because it is illogical in randomised trials.263264265266 The CONSORT 2010 statement34 presents an example of how to summarise baseline characteristics.

In the presence of marked differences in the numbers of randomised participants and those included in the interim or final analyses, authors are encouraged to report baseline summaries by treatment group for these two populations. Readers will then be able to assess representativeness of the interim or final analysis population relative to those randomised and also the target population.

See box 22 for an exemplar.

• ACE item 15b (new): Summary of data to enable the assessment of similarity in the trial population between interim stages

Comment—This item is applicable for ADs conducted in distinct stages for which the trial has progressed beyond the first stage.

Explanation—Changes in trial conduct and other factors may introduce heterogeneity in the characteristics or standard management of patients before and after trial adaptations. Consequently, results may be inconsistent or heterogeneous between stages (interim parts) of the trial.201 For ADs, access to interim results or mere guesses based on interim decisions taken may influence behaviour of those directly involved in the conduct of the trial and thus introduce operational bias.22 Some trial adaptations may introduce intended changes to inclusion or exclusion criteria (for example, population enrichment92146). Unintended changes to characteristics of patients over time may occur (population drift).267 A concern is whether this could lead to a trial with a different study population that does not address the primary research objectives.268 This jeopardises validity, interpretability, and credibility of trial results. It may be difficult to determine whether differences in characteristics between stages occurred naturally due to chance, were an unintended consequence of pre-planned trial adaptations, represent operational bias introduced by knowledge or communication of interim results, or are for other reasons.269 However, details related to item 11c may help readers make informed judgements on whether any observed marked differences in characteristics between stages are potentially due to systematic bias or just chance. Therefore, it is essential to provide key summary data of participants included in the analysis (as discussed in item 15a) for each interim stage of the trial and overall. Authors are also encouraged to give summaries by stage and treatment group. This will help readers assess similarity in the trial population between stages and whether it is consistent across treatment groups.

See box 23 for an exemplar.

Section 16. Results (Numbers analysed)

• CONSORT 2010/ACE item 16 (clarification): For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups

Comments—The item should be used in reference to the CONSORT 2010 statement34 for original details and examples. Here, we give additional clarification for some specific requirements of certain ADs such as population enrichment.87146

Explanation—We clarify that the number of participants by treatment group should be reported for each analysis at both the interim analyses and final analysis whenever a comparative assessment is performed (for example, for efficacy, effectiveness, or safety). Most importantly, the presentation should reflect the key hypotheses considered to address the research questions. For example, population (or patient or biomarker) enrichment ADs can be reported by treatment group for each pre-specified subpopulation and full population depending on key hypotheses tested.

Section 17. Results (Outcomes and estimation)

• CONSORT 2010/ACE item 17a (clarification): For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval)

Comments—We expanded the explanatory text to address some specific requirements of certain ADs such as population enrichment.146 Therefore, the item should be used in reference to the CONSORT 20103 for original details and examples.

Explanation—In randomised trials, we analyse participant outcome data collected after study treatments are administered to address research questions about beneficial and/or harmful effects of these treatments. In principle, reported results should be in line with the pre-specified estimand(s) and compatible with the research questions or objectives.206207 The CONSORT 2010 statement34 addresses what authors should report depending on the outcome measures. These include group summary measures of effect, for both interim and final analyses, including the number of participants contributing to the analysis, appropriate measures of the treatment effects (for example, between group effects for a parallel group randomised trial) and associated uncertainty (such as credible or confidence intervals). Importantly, the presentation is influenced by how the key hypotheses are configured to address the research questions. For some ADs, such as population (or biomarker or patient) enrichment, key hypotheses often relate to whether the study treatments are effective in the whole target population of interest or in specific subpopulations of the target population classified by certain characteristics. In such ADs, reporting of results as detailed in the CONSORT 2010 should mirror hypotheses of interest. That is, we expect the outcome results to be presented for the subpopulations and full target population considered by treatment group. This is to help readers interpret results on whether the study treatments are beneficial to the target population as a whole or only to specific pre-specified subpopulations.

• ACE item 17c (new): Report interim results used to inform interim decision-making

Explanation—Adherence to pre-planned adaptations and decision rules including timing and frequency is essential in AD randomised trials. This can only be assessed when the pre-planned adaptations (item 3b), adaptive decision rules (item 7b), and results that are used to guide the trial adaptations are transparently and adequately reported.

Marked differences in treatment effects between stages may arise (for example, discussed in item 15b) making overall interpretation of their results difficult.92110267269270271272 The presence of heterogeneity questions the rationale for combining results from independent stages to produce overall evidence, as is also the case for combining individual studies in a meta-analysis.92273 Although this problem is not unique to AD randomised trials, consequences of trial adaptation may worsen the problem.269 Authors should at least report the relevant interim or stage results that were used to make each adaptation, consistent with items 3b and 7b; for example, interim treatment effects with uncertainty, interim conditional power or variability used for SSR, and trend in the probabilities of allocating participants to a particular treatment group as the trial progresses. Authors should report interim results of treatment groups or subpopulations that have been dropped due to lack of benefit or poor safety. This reduces the reporting bias caused by selective disclosure of treatments only showing beneficial and/or less harmful effects.

See box 24 for exemplars.

Box 24

Exemplars on reporting item 17 elements

• Example 1. Bayesian RAR; change in randomisation probabilities across arms throughout the trial; randomisation updates were made after every patient

  • Giles et al71 present a table of changes in allocation probabilities used to create figure 3 by treatment group including allocated treatment and primary outcome response for each participant.

Fig 3

Redrawn from Pallmann et al.22 Reused in accordance with the terms of Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). No changes to the original figure were made.

• Download figure
• Open in new tab
• Download powerpoint

• Example 2. Inferentially seamless phase 2/3 AD; stage 1 treatment selection results

  • Barnes et al141 clearly presented the results that led to the interim selection of the two indacaterol drug doses to progress to stage 2 of the study; 150 µg (the lowest dose that exceeded both pre-specified treatment selection criteria) and 300 µg (the next highest dose that met the same criteria). The interim difference in treatment effect compared to placebo with uncertainty per group for the two adaptation outcomes are displayed in figures 1 and 2 of the paper.

• Example 3. 2-stage GSD; stage 1 dose selection results

  • “At the interim analysis planned after at least 1969 patients had been randomized and reached day 7 follow-up in each group,181 the otamixaban dose for stage 2 of the trial was selected as described in eFigure 1 in the Supplement. At that time, the rates of the primary efficacy outcome in the higher-dose otamixaban group was xx (4.7%) (the one selected to go forward) and was xx (5.6%) in the UFH-pluseptifibatide group (adjusted RR, 0.848; 95% CI, 0.662-1.087) but the lower-dose group fulfilled the pre-specified criteria for futility with a RR of more than 1 (primary efficacy outcome, xx (6.3%); RR, 1.130; 95% CI, 0.906-1.408) and was discontinued.”144 xx are the corresponding number of participants with primary response that should have been stated.

• Example 4. Adapted from Khalil et al143; sequential-step AD. See table 6.

Table 6

Interim results

View this table:

• View popup
• View inline

RETURN TO TEXT

Section 20. Discussion (Limitations)

• CONSORT 2010/ACE item 20 (clarification): Trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses

Comments—No change in wording is made to this item so it should be applied with reference to the CONSORT 2010 statement34 for original details and examples. Here, we only address additional considerations for ADs.

Explanation—We expect authors to discuss the arguments for and against the implemented study design and its findings. Several journals have guidelines for structuring the discussion to prompt authors to discuss key limitations with possible explanations. The CONSORT 2010 statement34 addresses general aspects relating to potential sources of bias, imprecision, multiplicity of analyses and implications of unplanned changes to methods or design. For AD randomised trials, further discussion should include the implications of:

• Any deviations from the pre-planned adaptations (for example, decision rules that were not enforced or overruled and changes in timing or frequency of interim analyses);

• Interim analyses (for example, updating randomisation with inadequate burn-in period);

• Protocol amendments on the trial adaptations and results;

• Potential sources of bias introduced by interim analyses or decision-making;

• Potential bias and imprecision of the treatment effects if naïve estimation methods were used;

• Potential heterogeneity in patient characteristics and treatment effects between stages;

• Whether outcome data (for example, efficacy and safety data) were sufficient to robustly inform trial adaptations at interim analyses and;

• Using adaptation outcome(s) different from the primary outcome(s).

Additionally, it is encouraged to discuss the observed efficiencies of pre-planned adaptations in addressing the research questions and lessons learned about using the AD, both negative and positive. This is optional as it does not directly influence the interpretation of the results but enhances much-needed knowledge transfer of innovative trial designs. Therefore, authors have been encouraged to consider separate methodology publications in addition to trial results.58181

See box 25 for exemplars.

Section 21. Discussion (Generalisability)

• CONSORT 2010/ACE item 21 (clarification): Generalisability (external validity, applicability) of the trial findings

Comments—We have not changed the wording of this item so it should be considered in conjunction with the CONSORT 2010 statement.34 However, there are additional considerations that may influence the generalisability of results from AD randomised trials.

Explanation—Regardless of the trial design, authors should discuss how the results are generalisable to other settings or situations (external validity) and how the design and conduct of the trial minimised or mitigated potential sources of bias (internal validity).3 For ADs, there are many factors that may undermine both internal (see item 20 clarifications) and external validity. Trial adaptations are planned with a clear rationale to achieve research goals or objectives. Thus, the applicability of the results may be intentionally relevant to the target population enrolled or pre-specified subpopulation(s) with certain characteristics (subsets of the target population). Specifically, the implemented adaptations and other factors may cause unintended population drift or inconsistencies in the conduct of the trial. Authors should discuss the population to whom the results are applicable including any threats to internal and external validity which are trial dependent based on the implemented adaptations.

See box 26 for exemplars.

Section 24. Other information (Statistical analysis plan and other relevant trial documents)

• ACE item 24b (new): Where the full statistical analysis plan and other relevant trial documents can be accessed

Explanation—Pre-specifying details of statistical methods and their execution including documentation of amendments and when they occurred is good scientific practice that enhances trial credibility and reproducibility of methods, results and inference. The SAP is the principal technical document that details the statistical methods for the design of the study; analysis of the outcomes; aspects that influence the analysis approaches; and presentation of results consistent with the research questions/objectives and estimands206207 in line with the trial protocol (now item 24a). General guidance on statistical principles for clinical trials to consider with the aim to standardise research practice exists.274275276 AD trials tend to bring additional statistical complexities and considerations during the design and analyses depending on the trial adaptations considered. Access to the full SAP with amendments (if applicable) addressing interim and final analyses is essential. This can be achieved through the use of several platforms such as online supplementary material, online repositories, or referencing published material. This enables readers to access additional information relating to the statistical methods that may not be feasible to include in the main report.

Critical details of the trial adaptations (for example, the decision-making criteria or adaptation algorithm and rules) may be intentionally withheld from publicly accessible documents (for example, protocol) while the trial is ongoing.45203 These details may be documented in a formal document with restricted access and disclosed only when the trial is completed in order to minimise operational bias (item 11c). For this situation, authors should provide access to such details withheld with any amendments made for transparency and an audit trail of pre-planned AD aspects.

For some AD randomised trials, methods to derive statistical properties analytically may not be available. Thus, it becomes necessary to perform simulations under a wide range of plausible scenarios to investigate the operating characteristics of the design (item 7a), impact on estimation bias (item 12b), and appropriateness and consequences of decision-making criteria and rules.154277 In such cases, we encourage authors to reference accessible material used for this purpose (for example, simulation protocol and report, or published related material). Furthermore, it is good scientific practice to reference software, programs or code used for this task to facilitate reproducible research.

The operating characteristics of ADs heavily depend on following the pre-planned adaptations and adaptive decision-making criteria and rules. ADs often come with additional responsibilities for the traditional monitoring committees or require a specialised monitoring committee to provide independent oversight of the trial adaptations (for example, adaptive decision-making or adaptation committee). Thus, it is essential to be transparent about the adaptation decision-making process, roles and responsibilities of the delegated DMC(s), recommendations made by the committee and whether recommendations were adhered to. Authors are encouraged to provide supporting evidence (for example, DMC charter).

See box 27 for exemplars.