Objectives: The objective of this study is to investigate the relationship between a physician’s subjective mortality prediction and the level of confidence with which that mortality prediction is made.
Design and participants: The study is a prospective cohort of patients less than 18 years of age admitted to a tertiary Paediatric Intensive Care Unit (ICU) at a University Children’s Hospital with a minimum length of ICU stay of 10 h. Paediatric ICU attending physicians and fellows provided mortality risk predictions and the level of confidence associated with these predictions on consecutive patients at the time of multidisciplinary rounds within 24 hours of admission to the paediatric ICU. Median confidence levels were compared across different ranges of mortality risk predictions.
Results: Data were collected on 642 of 713 eligible patients (36 deaths, 5.6%). Mortality predictions greater than 5% and less than 95% were made with significantly less confidence than those predictions <5% and >95%. Experience was associated with greater confidence in prognostication.
Conclusions: We conclude that a physician’s subjective mortality prediction may be dependent on the level of confidence in the prognosis; that is, a physician less confident in his or her prognosis is more likely to state an intermediate survival prediction. Measuring the level of confidence associated with mortality risk predictions (or any prognostic assessment) may therefore be important because different levels of confidence may translate into differences in a physician’s therapeutic plans and their assessment of the patient’s future.
Statistics from Altmetric.com
The termination of technologic support, limitations of care, and the establishment of “do not resuscitate” orders entails that the treating physician can, with a degree of accuracy, predict patient outcomes and present these estimates to the patient or their surrogates. Using the statistical analyses of large patient databases to create mortality prediction models has improved our ability to predict the risk of death in critically ill children and adults.1,2 The majority of these models utilise the concept of physiologic instability to establish the person’s relative risk of dying; such that an increased degree of physiologic instability is associated with an increased risk of mortality. These population based models of mortality prediction leave a “gap” in our ability to prognosticate. This is the gap that occurs in applying population statistics to the individual. Models designed to assess outcome that are population estimates coupled with an individual practitioner’s experience increase the accuracy of the prediction.3
The technical dynamic of mortality prediction models, alone or coupled with the individual practitioner’s experience, are one aspect of treatment limitation discussions between physicians and families of critically ill children. Factors that influence the conviction with which a mortality prediction is delivered include the degree of confidence a health care provider has in his or her assessment of that mortality risk. Beyond survival, quality of life issues (such as: technology dependence and degree of neurological dysfunction) are additional elements that generate their own complexities of prognostication and certainty. Additionally, since a physician may deliver care differently depending upon his or her prognostic estimates, as well as the certainty he or she has in those estimates, the ability to generate and act upon accurate prognoses may have important implications for quality of care, length of stay, and the cost of hospitalisation.
Previous investigators have assessed the accuracy of physicians’ prognostic estimates of mortality, but few data exist on factors that may influence these estimates or the accuracy of these estimates.4–8 The primary purpose of this study was to investigate the relationship between a physician’s mortality prediction for a patient admitted to a paediatric intensive care unit (ICU) and the confidence with which that prediction is made. This work extends the previous work of the author (JM) in asking the question “Can nurses and physicians prognosticate as well as statistical models on individual patients admitted to the paediatric ICU and what are the factors influencing their predictions?”3,9 We hypothesised that the more extreme a physician’s prediction of death (near 0% or 100%), the surer the physician would be of his or her prediction. We anticipated that the less sure a physician is about their prognostic estimate the more likely he or she would predict a 50–50 chance of survival.
MATERIALS AND METHODS
The methods of data collection have been previously published.3,9 The study was conducted at the Children’s National Medical Center Paediatric Intensive Care Unit, a 16 bed paediatric ICU with approximately 1000 admissions per year in Washington DC, from April 1997 to November 1998. All consecutive patients who were admitted to the ICU for at least 10 h were included in the study. Data from paediatric ICU attending physicians (those physicians in charge of the ICU) and paediatric ICU fellows (those physicians in paediatric ICU training) were obtained between 10 h and 24 h after admission to the ICU, on morning or afternoon formal rounds, whichever came first.
Data were collected from 10 h to 24 h after admission for two reasons. Firstly, we wished to exclude patients who were admitted for very short stays (less than 10 h). This would include patients admitted for procedures and patients admitted in the process of imminently dying. Secondly, providing at least a 10 h observation period allowed the health team time to review all pertinent medical information, assess the patient’s clinical status, as well as time to discuss the patient’s current condition and treatment plan. Excluded patients were those who were discharged or died before the 10 h observation period, and patients admitted just prior to morning rounds who would not form part of the formal rounds during the 10 h to 24 h observation period.
Data included the physician’s exact estimate of mortality risk (from 0% to 100%) and the degree of confidence associated with the estimate on a continuous scale from 0 to 5, with 0 representing no confidence and 5 representing absolute confidence. The subjective ICU mortality risk prediction (that is, the probability that the child would not survive the ICU admission) and the confidence associated with the mortality risk prediction (that is, the confidence associated with prediction) were posed as separate questions. For example, a physician could predict a 20% chance of death, but may only be 40% confident that 20% is the correct probability to assign. The information sheet was confidentially completed by each of the providers just after rounds on that patient.
The Institutional Review Board approved the study with the condition that the data collection contain no identifiers to the participating subjects (patients and physicians), and informed consent was obtained from each participating physician.
Mortality predictions were grouped into ranges because the frequency distributions of physician mortality predictions were not uniform. The ranges were selected prior to the analyses and were based on frequency and convenience. Median confidence levels were compared after stratification using the Kruskal-Wallis test. Individual comparisons were made using the Mann-Whitney statistic with a Bonferroni correction. The κ coefficient was used to assess prediction accuracy. Likelihood ratios were calculated for predictions of mortality for the different ranges of certainty. For the κ coefficient and the likelihood ratios, a cut off value of an estimated mortality risk of 50% was used to differentiate subjective predictions of survival and death. Statistical analyses were preformed with Stata 6.0 (College Station, Texas).
Of the 713 consecutive patients, 642 met the eligibility criteria. Excluded from the study were patients who were discharged (n = 55) or died (n = 5) less than 10 h after admission. Also excluded were 11 patients (zero deaths) that were admitted just prior to morning rounds and therefore were not formally part of rounds between the 10 h to 24 h observation period. The number of responses from critical care fellows totalled 600, and responses from critical care attending physicians totalled 678. For every patient, a subjective mortality risk estimate was collected from at least one of each of the critical care attending physicians (n = 5) and critical care fellows (n = 9). In some instances, if more than one provider from each group was present on rounds (for example, an additional attending), predictions were collected from both. Data were collected on 96.4% and 93.8% of the eligible admissions from the attending physicians and fellows, respectively. None of the missing patients died.
Figures 1 and 2 illustrate the relationship between the mortality predictions and the confidence with which the predictions were made. There was a significant difference between confidence levels among the different ranges of mortality predictions (p<0.001) for both critical care attending physicians and fellows. Individual comparisons between the extremes of mortality prediction (0% to <5% and >95% to 100%) and the middle ranges of mortality risk prediction (5% to 95%) were significantly different (p<0.01) for both critical care attending physicians and fellows.
The likelihood of a child dying given that the attending physician’s mortality prediction was made with a high level of certainty (>4 to 5), is 2.75 times greater than if the same prediction had an average certainty (table 1). Also, the proportion of intermediate mortality predictions (>5% to <95%) was lower among attending physicians (11.36%) versus fellows (23.5%), p<0.001. Table 1 shows that the κ statistics and likelihood ratios increase as the certainty level associated with the mortality prediction increases.
Multiple authors have criticised the ability and/or willingness of a physician to prognosticate upon a patient’s disease or condition.10,11 Most notably, the work of Christakis has examined the barriers to prognostication that occur in current medical practice. Identified barriers to prognostication include a lack of emphasis on prognostication in medical texts, poor mentorship, the stress involved in predicting outcomes, a lack of experience in making such predictions and a recognition in our own limitations of prognosticating accurately.11–14 What is seen as a crisis of prognostication has raised the suggestion that prognostication is a fundamental duty of the physician, a duty that is not currently being met, which arises out of the proper execution of the doctrine of informed consent.10
Several studies have assessed the ability of health care providers to estimate mortality risk.4–8,15 In one study, Lynn et al analysed the ability of physicians and physiologic-based mortality prediction models to correctly identify the likelihood of survival the day and week before a patient expired. This study demonstrated that even on the day prior to a patient’s death physicians and mortality prediction models had a substantial number of patients predicted to live an additional two months, and one week prior to their deaths roughly 50% of patients were expected to survive an additional two months.16 This study points out the limitation of prognostication by the best means available. Database analyses, like PRISM and APACHE, inform us that all admissions to the ICU involve a potential risk of death. The “routine” asthmatic patient admitted to the ICU may carry a less than 1% chance of dying, but this prognosis is hardly useful to the patient or family despite being statistically accurate. Discussions of prognosis only have their true cogency when the risk of death is substantial.
The limitation or withdrawal of medical care in critically ill children is an issue that embraces questions of longevity, quality of life, and surrogate decision making, and requires the physician to forecast the patient’s course. In the authors’ experience questions about the certainty of these predictions often arise. Absolute certainty in medical care is a rarity, but decision makers are likely to factor in the degree of certainty expressed by the physician in end of life discussions. The study reported here demonstrates that a physician’s subjective mortality prediction is associated with the confidence with which the mortality prediction is made. Mortality predictions between 5% and 95% are correlated with significantly less confidence than predictions of mortality less than 5% and more than 95%. This relationship suggests that if a physician is unsure or lacks confidence in his or her assessment of mortality, it is perhaps more likely that he or she will predict an intermediate mortality prediction, say 50%. Therefore, mortality predictions may be as dependent on the level of expertise and certainty of the physician as on the actual medical risk of mortality. This hypothesis is supported by the fact that ICU attending physicians had more confidence in their mortality predictions and had a lower proportion of intermediate mortality predictions compared to ICU fellows.
Our data suggest that a physician’s mortality predictions are related to the confidence in the mortality prediction, and that perhaps physician experience, clinical skills, and personal self confidence may affect subjective mortality predictions. Physician mortality risk estimates have also been shown to depend upon the patients’ disease or severity of illness.17,18 Only our previously published data and this investigation consider physician confidence when assessing mortality predictions.3,9 Our analysis suggests that incorporating confidence in the assessment of physician’s subjective prediction performance improves accuracy, however the improvement in accuracy was only marginal.9 Measuring the level of confidence associated with mortality risk predictions (or any prognostic assessment) may therefore be important because different levels of confidence may translate into differences in the diagnostic work-up or therapeutic plans.
A limitation of this study is that the data were derived from a single institution with a limited number of critical care attending physicians and fellows whose predictions and levels of confidence may not be representative of the general population. In addition, there were only 36 deaths during the study period generating small numbers after stratification for confidence level. The mortality rate of 5.6% observed in this study is comparable to most paediatric ICUs in the United States.1 Because we did not have data specifying how many predictions were made by each physician, and for which patients these predictions were made, we could not eliminate over (or under) representation as a source of bias. We were also not able to evaluate “within” group variation to ascertain the heterogeneity of the attending and fellow predictions. A further limitation of this study is that it is assumed that physician certainty is a factor that patients, or their surrogates, use in decision making. This is the authors’ experience, but to our knowledge no study has examined the outcome of care conferences where the degree of certainty of the delivery of that conference is a variable. However, in a related topic, that of giving bad news to parents, physician confidence was rated as an important factor in parents’ assessment of receiving bad news.19
In summary, a crisis of prognostication may in fact exist. Statistical models of estimating patient mortality that encompass far more patient lives than any individual practitioner’s experience predict mortality with no greater accuracy than experienced physicians. The literature demonstrates that those who work in medical subspecialties and those with the greatest experience are best at predicting a patient’s course.14 If what is expected of prognostication is a prognosis delivered accurately, specifically, and with certainty then current knowledge and experience will not allow these predictions. If a guarantee is what is understood by prognostication, rather than a prediction that is provided in good faith by a competent practitioner, then our own expectations may be the most significant barrier to prognostication.
Supported, in part, by a grant from the Children’s Research Institute, Children’s National Medical Center, Washington DC.
U E Ruttimann died 25 March 1998.
Presented, in part, at the Society for Critical Care Medicine 30th International Educational and Scientific Symposium, San Francisco, CA; 11 February 2001.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.