Article Text

Download PDFPDF

Paper
Evaluation of clinical ethics support services and its normativity
  1. Jan Schildmann1,
  2. Bert Molewijk2,3,
  3. Lazare Benaroyo4,
  4. Reidun Forde5,
  5. Gerald Neitzke6
  1. 1Department of Medical Ethics, Institute of Medical Ethics and History of Medicine, RuhrUniversity Bochum, Bochum, Germany
  2. 2Department of Medical Humanities, EMGO+ Institute for Health and Care Research, VU Medical Center, Amsterdam, The Netherlands
  3. 3Center for Medical Ethics, University of Oslo, Oslo, Norway
  4. 4Ethics Unit, Faculty of Biology and Medicine and Interdisciplinary Ethics Platform, University of Lausanne, Lausanne, Switzerland
  5. 5Center for Medical Ethics, University of Oslo, Oslo, Norway
  6. 6Institut für Geschichte, Ethik und Philosophie der Medizin, Medizinische Hochschule Hannover, Hannover, Germany
  1. Correspondence to Dr Jan Schildmann, Department of Medical Ethics, Institute of Medical Ethics and History of Medicine, RuhrUniversity Bochum, Malakowturm Markstr. 258a, Bochum 44799, Germany; jan.schildmann{at}rub.de

Abstract

Evaluation of clinical ethics support services (CESS) has attracted considerable interest in recent decades. However, few evaluation studies are explicit about normative presuppositions which underlie the goals and the research design of CESS evaluation. In this paper, we provide an account of normative premises of different approaches to CESS evaluation and argue that normativity should be a focus of considerations when designing and conducting evaluation research of CESS. In a first step, we present three different approaches to CESS evaluation from published literature. Next to a brief sketch of the well-established approaches of ‘descriptive evaluation’ and ‘evaluation of outcomes’, we will give a more detailed description of a third approach to evaluation—‘reconstructing quality norms of CESS’—which is explicit about the normative presuppositions of its research (design). In the subsequent section, we will analyse the normative premises of each of the three approaches to CESS evaluation. We will conclude with a brief argument for more sensitivity towards the normativity of CESS and its evaluation research.

  • Clinical Ethics
  • Ethics Committees/Consultation

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

The evaluation of clinical ethics support services (CESS)i has attracted considerable interest in recent decades.1–7 CESS evaluationii as a tool for quality assessment has been widely supported by scholars as well as practitioners in the field of clinical ethics.8–10 The reasons for performing a CESS evaluation vary considerably.8 It may be, for example, that a hospital management is interested in the effects of CESS on treatment costs. Ethics committee members may be keen to get feedback on whether their CESS is accessible to potential clients, while healthcare professionals might be more interested whether CESS improves patient oriented outcome parameters. The aforementioned examples indicate that evaluation studies of CESS have a normative dimension. What exactly is a good CESS or a good result of CESS? Who determines which criteria are ‘relevant’ or ‘important’ in CESS evaluation? While these questions apply to evaluation research in general, the normative dimension of evaluation of CESS is of specific interest. This is because as part of CESS we deliberate about good actions, the question of what is morally good and how to determine what is morally good. Therefore, any evaluation research related to CESS is automatically confronted with normative questions that are addressed within CESS.

The normativity of CESS evaluation has received little attention in the literature. For the purpose of this paper we understand by (implicit) normativity of CESS evaluation ethically relevant premises which underlie evaluation with regards to design, method and evaluation criteria. Moreover, evaluation of CESS becomes an explicit normative project if data are used to establish moral norms/standards or to check to what extent certain norms/standards are met by CESS. Moral norms or standards require an agreement on parameters to distinguish ‘good’ ethics consultation from practices with lesser quality. While implicit normativity has been a topic in the context of the evaluation of outcomes of CESS,9 ,11 ,12 there is a scarcity of systematic analyses of CESS evaluation from an explicit ethico-normative perspective.13 This paper aims to provide an analysis of normative presupposition relevant to CESS evaluation. As a starting point for our analysis, we present different approaches to CESS evaluation. Subsequently, these approaches will be analysed with regard to normative presuppositions. We conclude with a brief argument for the need of a more normative, sensitive approach to CESS evaluation research.

Approaches to the evaluation of CESS

We distinguish three approaches to CESS evaluation in the following. The starting point of our work was a selective literature review conducted by all authors with the aim of gathering evaluation studies as well as literature reviews on CESS evaluation.7 ,9 ,13 In addition to unsystematic gathering of literature by the authors and further experts of the European Clinical Ethics Networkiii the literature review included results from an earlier systematic review on outcome evaluation of CESS conducted by the first author in PubMed. A detailed description of the search algorithm has been described elsewhere.7 In a next step, we characterised and distinguished evaluation studies according to criteria which are relevant from a normative perspective. The working process consisted of several stages in which one or two authors forwarded drafts of categorisation which were then critically discussed among the group of authors. The iterative process has been documented through draft versions of the categorisation, meeting reports and individual notes made during the discussions. In addition to the consensual processes among the authors, the categorisation of CESS evaluation studies was discussed at several meetings of the of the European Clinical Ethics Network14 and at a scientific session during the International Conference on Clinical Ethics and Consultation in Amsterdam in 2011.

The depiction of the first two approaches ‘descriptive evaluation’ and ‘evaluation of outcomes’ will be sketched rather briefly because a considerable number of evaluation studies of these types have been published (for reviews of these studies see7 ,9 ,13). The third approach, ‘Reconstructing quality norms of CESS’ will be presented in more detail because this approach is comparatively novel and furthermore from a normative perspective it is relevant that this approach explicitly acknowledges the normativity of evaluation criteria during the process of evaluation. The approaches outlined below are not mutually exclusive categories. Moreover, we hold the view that each of the approaches mentioned can contribute to the quality of CESS. The purpose of this section is to provide grounds for an analysis of normative presuppositions relevant to existing approaches to CESS evaluation. The distinction into three approaches might also function as a heuristic overview for discussion on and further development of CESS evaluation in the future.

Descriptive evaluation

Descriptive quantitative evaluation, to which we limit this approach, encompasses many of the earliest evaluation studies published on the structural aspects of ethics support services.13 This approach to evaluation provides numeric data on access, activities, structural features and further aspects according to which the set up and functioning of CESS can be characterised. Typical examples of descriptive evaluation are cross-sectional studies which inform about the professional background of ethics committee members, the number of case consultations per period of time or the topics of case consultations.12 ,15 ,16 Such research can stimulate further development of an ethics support service. An example would be if data from a descriptive study indicate that a specific group of professionals never requests ethics consultations: a result which may trigger further research to explore the reasons for this finding.

The results of descriptive evaluation are often used to substantiate judgements regarding the functioning of CESS. One example is the number of case consultations conducted per time, which, in the case of high numbers, is used to demonstrate the need and/or acceptance of CESS.9 However, such interpretation has its limitation. While an increasing number of case consultations might be interpreted as an indicator of satisfactory acceptance of the service, it could also be concluded that a declining number in a certain area indicates an educational effect of earlier CESS interventions on the moral competencies of the hospital staff involved. It should also be noted that purely quantitative descriptive evaluation research only gathers data which fit into predefined categories. This implies that potentially relevant, but unanticipated information will not be collected as part of such an evaluation. Qualitative exploration studies conducted prior to quantitative descriptive research can provide some remedies in this respect. This is, for example, because by means of semistructured interviews with stakeholders relevant criteria may be elicited, which have not been identified by the researcher before. Furthermore, a deliberation on possible evaluation criteria with CESS stakeholders may inform researchers regarding priorities of the various possible criteria as viewed from the perspectives of different stakeholders.

Evaluation of outcomes

The assessment of outcome parameters of ethics support services has received increasing attention in recent years. The essential characteristic of this type of research is that it focuses on outcomes in the sense of parameters for the quality of CESS. The categories used in outcome evaluation are predefined in two ways. First, and similar to descriptive evaluation, the categories in outcome evaluation are also predefined in terms of what data are gathered. Secondly, the evaluation categories are predefined also with regard to what consists of good or bad outcome. Thus, the research instruments are being designed after the determination of what is perceived as a desirable or undesirable outcome of CESS. Evaluation in this category can be divided into research which uses subjective or objective criteria. A frequently used subjective outcome criterion is satisfaction. Evaluation studies of this type are guided by the question: Are stakeholders who are involved in CESS (eg, those who requested the service, the administration of the hospital, the healthcare professionals or ethics consultants themselves) satisfied with the processes and outcomes of the services delivered? In addition to subjective criteria, objective outcome parameters have been used to demonstrate the effectiveness of CESS. Studies on the effects of CESS on days spent in intensive care units prior to death or costs of treatment are examples.17 The sources of information to assess outcomes include questionnaires or files, such as treatment documentation or hospital bills.

Evaluation of the outcomes of CESS has been widely advocated as a tool for the quality assurance of CESS. Such evaluation research may foster trust and confidence among patients, families and members of the healthcare team as a necessary prerequisite for the successful and sustained implementation of these services.10 Furthermore, evaluation of outcomes can inform the clinical ethicist and/or members of an ethics committee about the perceived quality of their work. A frequently used criterion in this respect is the ‘satisfaction’ of CESS participants. However, research on the satisfaction of CESS participants also indicates some of the challenges associated with this approach. While earlier evaluation studies focused on the ratings by physicians and showed affirmative results12 the satisfaction of patient representatives or relatives has repeatedly been shown to be less.18 Such findings trigger questions about whom we should ask as part of CESS evaluation and in which way satisfaction is relevant for a judgement on the quality of CESS. Furthermore, it should be noted that, similar to the descriptive approach, purely quantitative outcome research only gathers data which fit into predefined categories.

Reconstructing quality norms of CESS

The (re)construction of the quality norms of CESS has been more recently suggested as a different approach to CESS evaluation.19–23 This kind of evaluation research is an open process research in which the quality and outcome parameters of CESS are explicitly on the agenda of the evaluation itself. The first central characteristic of ‘reconstructing quality norms of CESS’ as an approach to CESS evaluation is the explicit reflective or deliberative focus on what is understood as the quality of CESS, how it is defined and on what basic and underlying presuppositions. In contrast to the aforementioned approaches to CESS evaluation, the criteria are not defined beforehand and those norms which are (implicitly) at hand are not taken for granted. Norms for good CESS are explicitly addressed and developed during the research process. As a consequence, this type of research can never start with a fixed set of evaluation parameters for CESS. A second central characteristic is that both the outcome parameters (of good CESS) and the research are developed in close cooperation with the various stakeholders involved in CESS practices. The stakeholders are co-owners of the research project. The participation and ownership of multiple stakeholders in evaluation research is increasingly being promoted22 and is inspired by the Fourth Generation Evaluation Research as developed by Guba and Lincoln24 (examples of this type of research are: responsive evaluation research, action research and community based participant research). Third, the determination and the final justification of the norms for quality are strongly connected with the contextual features in which CESS are performed. The purpose of this approach to CESS evaluation is to improve the quality of CESS by means of the evaluation research itself.

Responsive evaluation has been used to evaluate CESS on an institutional level in psychiatry.19–23 This type of research design is driven by democratic, participative and dialogical values. In responsive evaluation, the issues (eg, outcome parameters of CESS or good CESS) of all stakeholders are investigated to obtain a rich understanding of the practice evaluated from the insiders’ perspectives.25 ,26 Responsive evaluation, which shares features of Guba and Lincoln's ‘Fourth Generation Evaluation’,24 includes the voices of all stakeholders in the process evaluated, as information givers and as advisors and partners.23 ,27 Moreover, responsive evaluation meets well with the principles of hermeneutic ethics using dialogue as the main vehicle for learning, understanding and the reconstruction of norms.28

The explicit focus of this approach to CESS evaluation on what constitutes the quality of CESS corresponds with a fundamental feature of ethics: a continuous reflection upon quality and its determination. It is a critical de-construction and reconstruction of what the stakeholders involved (including the researchers) define as quality. The risk of misinterpretations or even abuse of evaluation data is comparably low since the actual meaning of the data is explicitly checked with the stakeholders. The ownership of the quality of CESS is contextualised. This implies that the results cannot lead to a universal score for quality measurement. Furthermore, the different stakeholders need to have trust in the ongoing creative process of evaluation, since there are no predefined norms regarding quality beforehand.

Normative presuppositions of CESS evaluation

Most of the CESS evaluation studies which have been published are not explicit about normative presuppositions. However, all of the approaches to CESS evaluation outlined above have normative presuppositions. To substantiate this claim, we will present a selection of these premises for each of the three evaluation approaches.

Descriptive (quantitative) evaluation of CESS (approach 1) reports data according to a predefined research design and selected criteria. From a normative perspective, this evaluation approach is associated with a number of value judgements. First of all, the decision in favour of gathering quantitative data encompasses decisions about priorities on what we want to know. While a quantitative research design is very much in line with the medical research paradigm, it also means that a number of interesting data from an ethical perspective (eg, information about the process of decision-making in case consultation which can be gathered by observation, video or scrutiny of reports) will be hard to elicit. Further decisions about priorities must also be made once a research design has been chosen: criteria for evaluation have to be selected, a decision especially important in the context of purely quantitative descriptive evaluation which will only elicit data within predefined criteria. Next to the selection of criteria, another priority decision refers to the source of the data. It is possible that the findings vary depending on the source of the information. Last but not least, it should be noted that there are further, more subtle but also normatively relevant issues, for example, with regard to the framing of questions when designing a research instrument for a descriptive evaluation.

The analysis above indicates that descriptive evaluation includes a number of decisions about priorities and further normatively relevant aspects. These value judgements in the context of descriptive evaluation are not wrongful but necessary and should, therefore, be put in the focus of any CESS evaluation.

The normative dimension of evaluating outcomes of CESS (approach 2) has already been pointed out already by a number of scholars.7 ,11 ,12 One focus of the debate with regard to outcome evaluation is whether general outcome criteria are useful at all because they cannot usually be applied to every single case under evaluation. An example which has been discussed in the literature is the outcome criterion of ‘non-beneficial treatment’. Schneiderman et al17 in their multi-centred randomised controlled trial showed that in the study group using ethics case consultation, fewer days were spent in intensive care prior to death compared with the control group. There was no statistically significant difference of mortality between the groups. One criticism that was sparked off by this study was the categorisation of medical treatment prior to death as ‘non-beneficial’. Looking at an individual case, it may well be that even life-sustaining treatment prior to death was associated with some benefit to this patient. Moreover, and of fundamental importance for CESS evaluation research, it is crucial to distinguish goals from outcomes. If a study reveals that the time spent in intensive care decreases due to CESS, this does not imply that CESS should decrease the duration of intensive care in the future. A second example of challenges related to outcome criteria for CESS evaluation which will be discussed here is ‘ethicality’, which has been defined as practice which complies with existing professional or legal standards.3 On the one hand, it seems plausible that the practice of CESS needs to be assessed against the background of existing normative frameworks. Neither healthcare professionals nor other parties involved in CESS should be kept in the dark, for example, about the legal consequences of their actions. However, on the other hand, in light of the nature of ethical dilemmas and the limitations of legal norms in solving such dilemmas, ethicality, in the sense of a simple reconfirmation of commonly accepted (moral, political, organisational) norms as part of CESS activity, is questionable. Such a conception of appropriate ethics consultation would collide with a core task of ethics, namely, to critically reflect on existing norms in ethically challenging cases.

In summary, the evaluation of predefined outcomes of CESS does very much match the medical paradigm of outcome measurement11 and contributes important information on the quality of CESS. However, as demonstrated above, evaluation according to predefined outcomes relies on strong assumptions of ‘good CESS’ and further normatively relevant presuppositions.

The third approach to the evaluation research of CESS, (re)constructing quality norms of CESS, focuses explicitly on the normativity of evaluation research in general and CESS in particular.19 ,29 This fact in itself does not make this kind of research better research; however, the transparency of its normativity makes it easier to understand, easier to agree or disagree with and also easier to reflect on the normative status of the outcomes. An important normative presupposition of this approach is that there are no universal norms (for the quality of CESS). Therefore, quality norms and outcomes are defined by the stakeholders in the practice in close cooperation with CESS experts and researchers. A concrete example is a study in which a hospital planned to implement moral case deliberation (MCD). The researchers were hired to both facilitate and evaluate the implementation process through responsive evaluation. During that research, an evaluation questionnaire about the goals of MCD was developed. Two questions were central to this questionnaire: (1) Do you consider X as a goal of MCD and (2) do you think that this goal was met during this MCD? The construction of the list of goals was a joint process in which the researchers collected goals from the literature and from the stakeholders in that specific hospital.5 Another normatively relevant premise of this approach is that CESS evaluation research aims at improving the respective CESS practice. The goal of improvements of CESS practices necessarily needs criteria for what counts as ‘improvement’ within that specific context. Another normative presupposition to be discussed in this section is that CESS evaluation becomes co-owned by the stakeholders involved. The moral aims of CESS and respective evaluation criteria are defined as a joint endeavour during the process of evaluation. There is a fundamental and normatively relevant distinction between those researchers or ethics consultants who have a normative and instrumental understanding of the moral aims of CESS (eg, ‘CESS should always ensure patient rights’), and those who perceive ethics inherently as a reflective and deliberative enterprise about the (re)construction of what is considered as morally desirable. While the first group may be able to define the moral outcomes of CESS beforehand, the second group views CESS to be in charge of always reflecting itself and to define evaluation criteria as part of the working process, and adjusted them to the specific context. This normative presupposition is inspired by hermeneutic ethics which stresses that the process of defining what is good (eg, CESS quality norms) is inherently related to the meaning of the ‘good’ in the specific experiences of those who work with CESS.30 Meaning giving and the dialogical process of developing norms are ongoing processes or redefinitions in real practices.

CESS evaluation and normativity: a plea for transparency

The preceding analysis demonstrates the variety of normative presumptions inherent to all three evaluation approaches. At the same time, the literature, apart from a few exceptions,2 ,10 ,11 rather ignores the topic of the normativity of CESS evaluation. In this concluding part, we argue that evaluators should be explicit about the normative presumptions concerning the goals, purposes and perspectives regarding CESS and the respective evaluation criteria. In other words, when evaluating CESS, it should be made explicit why, by whom and to what ends the approach to evaluation and its criteria are chosen.

Some theoretical and pragmatic reasons underpin this plea: First, we argue that only CESS evaluation which is sensitive to its normative dimensions has the potential of being high-quality evaluation. This is because ethics focuses on what constitutes (moral) quality. A further argument in favour of our claim is that reflecting the normative dimension of CESS evaluation promotes an approach which is consistent with the (often implicit) priorities and purposes of CESS. It would make little sense, for example, to conduct a sophisticated randomised controlled trial using quantifiable outcome measures, such as days spent on intensive care prior to death, if the primary goal of CESS was to provide a forum for moral deliberation for all parties involved in a morally challenging situation. Another argument is that being explicit about the normativity of evaluation research of CESS enables others to reflect and (dis)agree with the normative presuppositions. It stimulates the professional debate about what constitutes good CESS. It might also prevent a too narrow-minded interpretation or (ab)use of the empirical results. Last but not least, an ethically informed approach to CESS evaluation also seems relevant from the point of view of resource allocation. The resources for CESS are usually scarce and, therefore, it would be a waste of time, effort and money if the goals of CESS and the evaluation criteria chosen did not match.

It is beyond the scope of this paper to provide a detailed outline on how to put a normative reflective approach of CESS evaluation into practice. However, we suggest that a primary moral duty when planning a study is to be explicit about the (morally relevant) goals and purposes of the CESS and the (normative) rationale behind both the research design and the selection of criteria for evaluation and to discuss the strengths and weaknesses of the respective approach from a normative perspective. While a certain vagueness regarding the goals and purposes of CESS might be attractive to different stakeholders,8 in practice, such vagueness is contrary to most rational and morally responsible evaluation approaches. In being explicit about the normative premises of both CESS and the evaluation of CESS, we are sensitive to and consistent with the inherent characteristics of ethics (support) as a transparent, critical and deliberative professional domain.

Acknowledgments

This paper has been written by the working group on evaluation from the European Clinical Ethics Network (ECEN), founded in 2005 and consisting of 29 members from 14 countries. We would like to thank all ECEN members for their contributions to this paper through the numerous discussions during ECEN meetings.

References

Footnotes

  • Contributors All authors have contributed to the conception and analysis. They were all involved in the production of the drafts of the article and have given their approval of the submitted version to be published. There is no one else who fulfils the criteria of authorship.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • i The term ‘clinical ethics support services (CESS)’ encompasses all institutionalised services within healthcare organisations which support healthcare professionals and institutions in dealing with moral issues.

  • ii We define evaluation of CESS as the systematic gathering of data with empirical research methods for the purpose of acquiring knowledge about the structure, functioning, quality and results of CESS.

  • iii The European Clinical Ethics Network consists currently of clinical ethics experts from 14 countries. One of its aims is to further improve the quality and professionalism of CESS.

Linked Articles

  • The concise argument
    Kenneth Boyd

Other content recommended for you