The limitations of a cross-sectional study design

In July 2017, the International Association of Athletics Federations (IAAF) was expected to return to the Court of Arbitration for Sport (CAS) with evidence to justify reinstatement of their controversial hyperandrogenism rule. CAS has granted IAAF a 2-month extension for their response, which was due by the end of September. CAS suspended the IAAF Hyperandrogenism Regulations1 2 years earlier following the successful appeal by the Indian runner Dutee Chand.2 IAAF advisors have published two recent articles in this journal3 4 to support their claims that women with high endogenous testosterone levels have such a significant performance advantage over women with lower levels that it is necessary to exclude them from competing in the female category.

Both papers have examined the relationship between blood testosterone concentration and athletic performance using a cross-sectional design. While such studies can show the presence of association, they do not prove causality and no mention is made of the possible importance of associated androgen insensitivity, while in some there may be a contribution from exogenous testosterone. The paper by Eklund et al 3 compared a spectrum of androgen-related endocrine variables in a group of 106 elite Swedish female athletes with 117 sedentary controls. They found no difference in testosterone concentrations between the two groups, and most importantly no correlation between testosterone concentration and indices of performance in the elite female athletes.

The paper by Bermon and Garnier4 (funded by the IAAF and WADA) examined the results of endocrine variables in 1332 blood samples from elite female athletes who took part in the 2011 and 2013 IAAF World Athletic Championships; 17.3% of these athletes were sampled at both events.

The hyperandrogenism rule concerns total endogenous blood testosterone, but Bermon and Garnier4 concentrated on the derived ‘free testosterone’ (fT) and found that there were five (of 21) events where athletes whose fT was in the highest tertile appeared to do better than those whose fT was in the lowest tertile. On the other hand, although not statistically significant, those with fT in the lowest tertile appeared to perform better than those in the highest tertile in nine events. Furthermore, for the total endogenous testosterone levels, only three running events showed significant differences in performance between high and low testosterone tertiles.

We view the comparison between the ‘tertile groups’ using the unpaired t-test as statistically inappropriate and the lack of adjustment for multiple comparisons a major flaw of the analysis. A better approach would have been to look for a direct and statistically significant correlation between endogenous testosterone and performance as in the study by Eklund et al 3; in that case there was no correlation. Correlation analysis is the appropriate analysis tool, particularly for a cross-sectional study such as this where causal inference is not possible.

In the absence of a statistically significant correlation between androgens and performance, the evidence that either total testosterone or free testosterone predicts performance in women is not supported. A further criticism of the paper is the considerable controversy and concerns about the validity of various methods of calculating (rather than measuring) fT.5 Both papers provide useful information about the association between androgens and athletic performance in women but do not come close to addressing the issue of causality. Furthermore, their findings cannot be extrapolated to the extraordinarily high but largely non-functional testosterone levels that are seen in women with the androgen insensitivity syndrome.

Initial burden: male typical advantage

The IAAF was tasked by CAS with providing sufficient evidence that female athletes with androgen levels in the so-called male range have a competitive advantage over their peers, comparable to that men have over women, previously identified as 10%–12% [2; Para 526]. These studies do not provide this evidence. Both papers provide some new data but do not directly tackle the issues articulated by the CAS.

As the CAS panel noted, a competitive advantage in the range of 1%–3% suggested by the IAAF advisors is a relatively marginal one ‘…given the many other relevant variables that also legitimately affect athletic performance’ (para 527). One of the difficulties in quantifying the performance advantage that any individual athlete (male or female) derives from high endogenous testosterone levels is that it fails to account for these other variables. The IAAF acknowledged this fact in their previous study to determine whether unusually high levels of testosterone in women provide a competitive advantage, which used the same data from the 2011 World Athletic Championships in Daegu.6 They stated that they were unable to exclude other variables that ‘…in some unknown way may bring an advantage to female athletes’ (see also Ferguson-Smith and Bavington7).

Ethicists (including several authors of this editorial) have argued that even if the IAAF were able to prove a performance advantage of a high endogenous testosterone in elite female athletes, this would not be unfair.8 9

In conclusion, even considering the methodical limitations, these two new papers present no new evidence to support the regulation beyond what the CAS reviewed before reaching their decision on the Dutee Chand case.