Article Text

Download PDFPDF
Monitoring ‘monitoring’ and evaluating ‘evaluation’: an ethical framework for monitoring and evaluation in public health
  1. Vijayaprasad Gopichandran,
  2. Anil Kumar Indira Krishna
  1. School of Public Health, SRM University, Kattankulathur, Tamil Nadu, India
  1. Correspondence to Dr Vijayaprasad Gopichandran, School of Public Health, SRM University, Medical College Third Floor, Kattankulathur, Kanchipuram District, Tamil Nadu 603 203, India; vijay.gopichandran{at}gmail.com

Abstract

Monitoring and evaluation (M&E) is an essential part of public health programmes. Since M&E is the backbone of public health programmes, ethical considerations are important in their conduct. Some of the key ethical considerations are avoiding conflicts of interest, maintaining independence of judgement, maintaining fairness, transparency, full disclosure, privacy and confidentiality, respect, responsibility, accountability, empowerment and sustainability. There are several ethical frameworks in public health, but none focusing on the monitoring and evaluation process. There is a need to institutionalise the ethical review of M&E proposals. A theoretical framework for ethical considerations is proposed in this paper. This proposed theoretical framework can act as the blueprint for building the capacity of ethics committees to review M&E proposals. A case study is discussed in this context. After thorough field testing, this practical and field-based ethical framework can be widely used by donor agencies, M&E teams, institutional review boards and ethics committees.

  • Codes of/Position Statements on Professional Ethics
  • Public Health Ethics

Statistics from Altmetric.com

Introduction

Health improvement is the overall goal of all health systems. Public health interventions aim to achieve preventive, promotional and curative health with universal coverage. Considerable public resources are allotted for these interventions. One of the essential components of management of these public health interventions is monitoring and evaluation (M&E). M&E provides vital information about performance of interventions, which helps in the making of management decisions. It helps programmes to optimise utilisation of resources and directs focus towards achievement of intended goals.1 Whereas monitoring is ongoing close supervision and observation of programme activities to compare ‘what is happening’ with ‘what ought to happen’, evaluation is a process of episodic assessment of achievement against standard criteria.1 M&E starts as a process of monitoring of performance in quantum terms in the initial stages of project implementation and gradually metamorphoses into evaluation of impact of the programme.1 The fact that public health interventions largely depend on good M&E for their smooth performance emphasises the importance of high-quality standards. As well as quality measures, there is also a need to consider ethical issues in M&E. In this article, we discuss some critical issues in the ethics of M&E and present a comprehensive ethical framework for assessing M&E.

Critical analysis of the ethics of M&E

Is M&E research?

The two inter-related activities, M&E and surveillance, collect information from populations to help organise public health programmes and policies in an effective manner.1 Can M&E and public health surveillance be described as research in the pure sense of the word? The main difference between M&E and research is the intention of the activity. While research focuses on increasing the body of knowledge in any field, the main aim of M&E is to provide information specific to the functioning of the public health programme. Whereas findings of research may be applied widely, the findings of M&E are locally relevant.2 In M&E, no experimental or non-standard interventions are involved. The information is utilised only for the population from which it is obtained for a specific purpose, and the study is conducted as part of standard operating procedures.2 The International Network of Field Sites with Continuous Demographic Evaluation of Populations and their Health Guidelines give recommendations on ethical conduct of public health surveillance activities.3 The Oklahoma State University Institutional Review Board has suggested guidelines for determining when M&E becomes a research project.4

The following criteria have been laid down to classify M&E as research4:

  1. If the main objective of the study is to test a hypothesis

  2. If the study will benefit communities other than those on which it is carried out—that is, if it is intended that the findings of the study are generalised

  3. If the study is not a routine activity in the project

  4. If the data gatherers do not have regular contact with the community

  5. If the activity of the study alters the timing or frequency of the standard procedures

  6. If the party sponsoring the study is not the party conducting the public health intervention.

There is increased emphasis on the need for evidence-based public health policy making. M&E is an important source of evidence-based public health policy making. For any organisation, the results of M&E will be essential for influencing its internal policy.5 The problem of whether M&E is research arises when the evidence generated from the M&E is applied widely beyond the scope of the intended project population.5 Irrespective of whether the M&E qualifies as research, there is a definite need for ethical standards in its practice.

Need for an ethics framework for M&E

Several codes of conduct, best practices and ethical guidelines for M&E have been proposed by evaluation societies and development agencies.6–12 These are based on sound ethical principles and can be generalised to any public health programme. An academic paper by Smith highlights the important ethical issues to be considered in the practice of programme evaluation.13 In an internet-based survey of evaluation practitioners, the respondents felt that ethical considerations are of utmost importance.14 A survey by the Australasian Evaluation Society revealed that evaluation practitioners came across ethical issues and dealt with them by informal discussion with colleagues and friends, following local evaluation standards, or ignoring the issues.15 Newman and Brown explained an ethical decision-making framework for programme evaluation in 1996, which had intuitions, rules, principles, theories and personal values as its components.16 The framework is very simple and gives general directions, but does not assess the various ethical issues in detail. There is a need for continued awareness building and training of evaluation professionals on ethical decision making.17 Berends in her review paper concludes that evaluation societies should continue discussions about ethical issues in M&E, and capacity of research ethics committees should also be built to assess M&E proposals.17 Therefore there is a definite need for an updated, consolidated and comprehensive ethical framework for M&E.

Institutionalisation of ethical review of M&E

Evaluation professionals the world over have perceived the importance of institutionalising the process of ethical review of M&E.17 It is our opinion that institutionalisation of the ethical review of M&E will make the M&E process more accountable and responsible. The capacity of institutional review boards and ethics committees should be built to assess ethical performance of M&E.

In the following paragraphs, we discuss some key ethical issues in M&E and propose a theoretical ethical framework for practical application. This theoretical framework can serve as the blueprint for the institutionalisation of the process of ethical review of M&E.

Ethical considerations in M&E

The ethical issues in M&E can be considered step by step based on the activities in M&E.6–12

  1. Avoiding/disclosing conflicts of interest. There is the chance of conflict of interest at various levels in the M&E process. If the M&E is performed internally by programme staff, the financial or intellectual interests of the staff in the programme may be in conflict with the M&E. In the case of external evaluations also, there is a chance of conflicts of interest either in the form of financial stakes with the funding agency or other intellectual interests of the external evaluators. These conflicts of interest can impair the judgement of the person in charge of the M&E. Therefore the results may become unreliable.3

  2. Maintaining independence of judgement. Recommendations based on the findings of M&E have to be free of any pressures or external influences. The judgements made by the M&E team must be independent. There may be circumstances where members of the team, funding agency or other stakeholders can influence the judgements of the M&E team. This may be the result of financial, intellectual or other interests. Such influences should be removed or resisted.18

  3. Avoiding bias and being fair. Fairness, impartiality and avoiding bias are very important ethical aspects of M&E. The process of M&E as well as the results should be impartial to any group of the community or other stakeholders. Marginalised communities, women, persons with mental illnesses, persons with disabilities, children, migrants and other vulnerable persons must be given due importance and representation during the M&E process.18 ,19

  4. Transparency. From the process of engaging stakeholders to planning and setting priorities, all steps must be performed in a transparent manner. Such transparency fosters trust and active participation by the community in the M&E process. Transparency also ensures credibility of the results. As well as transparency of the process, transparency of funding and finances are also important.18 ,20

  5. Full disclosure. The importance of full disclosure or complete truth telling cannot be overemphasised in the M&E process. Often positive findings that are favourable to the programme are reported, whereas negative results, wrong doings and mishaps are concealed. This can be detrimental to the process and philosophy of M&E. It hampers learning from, and improvement of, the programme.18 ,19

  6. Privacy and confidentiality. As in any research, M&E involves entering into individual lives and exploring aspects of their participation (or non-participation) in the public health programme, their experiences and their opinions. It often involves gathering private information from participants. Measures should be taken to maintain the privacy and confidentiality of the information collected from the participants. Measures to ensure privacy and confidentiality not only increase the credibility of the process of M&E, but also allow the participants to share their opinions and details freely.18 ,19 The collected data should be stored with passwords and access available only to those directly involved in data analysis.

  7. Respect for individuals. Another important ethical principle in M&E is respect for the agency of the individual participants in the process. Their opinions must be treated with respect. The M&E team should have the attitude of learning from the individual who is participating in the M&E.2 Obtaining informed consent from the respondents for participating in the M&E is essential. The participant should be given enough information about the study that they are fully informed and can opt to participate or not. Another important issue is compensation for the time of the participants. The issue of whether compensation should be provided for participants’ time should be adequately discussed and decided on.

  8. Responsibility to community. The M&E team has a responsibility to the community in which they are conducting the process. They have the responsibility to do what is good for the community. During the process of M&E, they may come across problems in the community and/or pitfalls in the operation of the programme. It is the responsibility of the team to report, or take steps to rectify, these problems.18 ,21 ,22 The M&E team may often not be directly in contact with the community other than in the process of data collection. In such instances, the M&E team has a responsibility to engage the organisation in doing what is due.

  9. Empowerment. The M&E process should aim to empower all the stakeholders at all stages. By engaging the stakeholders to freely share the concerns and problems with the programme managers, it can empower them by giving voice to their views. Not only does this make the process of M&E relevant, it also empowers the community members. Ensuring prompt and appropriate action on the results of the M&E can also empower the community.18 ,23 This can best be achieved through an in-built management-response mechanism within the M&E system.

  10. Accountability. The M&E team is answerable to the community, stakeholders and the programme for the results of their evaluation. The concept of accountability is closely related to that of responsibility. The team should take responsibility for the credibility of its findings.18 ,19

  11. Sustainability. In the M&E process, although evaluation is a periodic event, monitoring is an ongoing one. Therefore there is a need to introduce measures to sustain the monitoring process. Engaging the community, empowerment of stakeholders, putting emphasis and importance on the process can help sustain the efforts and participation of the community in the M&E.18 ,24

  12. Hawthorne effect. The Hawthorne effect refers to a change in behaviour of the participants in a study or intervention, not purely by virtue of the intervention but due to the fact that they are being observed. It is a psychological behavioural effect. In many public health programmes, as long as the intervention lasts and close monitoring and supervision is being carried out, good outputs and outcomes are observed. However, once the funding for the programme runs out and the programme is withdrawn, the outcomes do not translate into long-term effects, as behaviours revert to the baseline state. This effect should be kept in mind while interpreting the results of the M&E. Measures should be taken to sustain the efforts over a long period of time.18 ,25–27 This is where processes gain importance. As mentioned above, if processes are given due importance, this may go a long way to strengthening the sustainability of the project.

These ethical considerations have an intrinsic value in the conduct of the M&E. They are different from ethical considerations of public health research in several ways. The immediate goal of a pure public health research project is generation of evidence and knowledge base. Therefore the responsibility of the researcher stops at ensuring beneficence, non-maleficence, justice and respect for individuals. On the other hand, in M&E there is a different ethical demand, which extends to ensuring that the findings translate into actions and interventions in an empowering and sustainable manner. Thus the ethics of M&E extend beyond research ethics. A theoretical framework for ethical assessment of M&E can be derived from these considerations.

Ethical framework for M&E

As discussed above, there is a need for an easy-to-use and practical ethical framework for application in M&E of public health programmes. Such an ethical framework is given in table 1. A brief case studyi from an M&E programme in India is presented, and the framework is applied to this programme to assess the ethical nature of this M&E.

Table 1

Ethical framework for monitoring and evaluation (M&E) of public health programmes

AI was involved in programme management and M&E of the ‘no scalpel vasectomy’ (NSV) intervention in a highly populated state of north India. As an initiative to promote the acceptance of NSV, which is a relatively easy and permanent method of male sterilisation, the district administrators and health officials were carrying out advocacy, communication, NSV camps and active promotion of NSV at the hospitals. One of the methods used by the district administration to increase the demand for the NSV intervention was to offer incentives such as free land registration for those who accepted NSV if they met other general eligibility criteria. Immediately after this announcement, there was a sudden increase in the number of acceptors of NSV from less than 50 in a year to more than 300. The author was part of the M&E team for this programme. As part of the ongoing monitoring, a monthly report of the number of acceptors of NSV was provided to the author. The monitoring unit was not allowed to interact with the clients during the programme implementation phase. Subsequently, the author went to evaluate this intervention as the head of an internal team. Appropriate stakeholder participation was ensured in the process of evaluation. Data were collected from the beneficiaries and non-beneficiaries in a respectful manner ensuring adequate privacy and confidentiality. The author observed that many of the acceptors of the NSV were men whose wives had already undergone permanent sterilisation by tubectomy. These men were not actually the targets for the NSV promotion campaign. Although the author reported this, following the principle of transparency, accountability and responsibility, the evaluation report was shelved and no action was taken on it. The information was also withheld from the stakeholders and the community. With time, this funding agency's priorities changed and the programme itself was withdrawn.

The first section in the ethical framework presented in table 1 explains the engagement of stakeholders in M&E. In this case, stakeholder engagement was ensured. However, it is noteworthy that the evaluation was performed by an internal team. Therefore potential conflicts of interest cannot be ruled out.

An external impartial evaluation team would have been ideal, but it is often not feasible. Moreover, in the monitoring phase, the team was not allowed to interact with the clients. The process of data collection during the evaluation was carried out in a respectful manner ensuring adequate privacy and confidentiality to the participants.

However, it is clearly evident from the case study that the process did not demonstrate responsibility to the community. A responsible action would have been to report the unnecessary NSV for men whose wives had already been sterilised and prevent further such unnecessary NSVs. When such important findings are not shared with the community, it prevents community empowerment. If the monitoring team had been allowed to interact with the clients during the early stages, such unnecessary surgical procedures could have been averted. At the stage of data analysis, interpretation and reporting, the author made a significant finding. The NSVs that seem to have increased were not in the target population, but in a group where there was actually no need for sterilisation. Although honest and complete data analysis was conducted, it was suppressed. Moreover, no action was taken on the basis of the findings. This shows lack of accountability. Most importantly, the findings of the M&E were not used effectively for the benefit of the community. Over time since the programme was withdrawn, some serious issues have been raised about sustainability of the intervention and the M&E process.

This briefly demonstrates how the M&E process can be retrospectively evaluated on the basis of the proposed ethical framework. Similarly, the framework can be used prospectively at the stage of planning the M&E, so that all important ethical issues can be identified. The M&E ethical framework can also be used during the process, for periodically assessing the ethical functioning of the process. While efforts are made to institutionalise the ethical review of M&E, this framework can be used as an effective aid and checklist. Further field testing and operational evaluation of this framework is necessary for wider use and dissemination. Once adequately field tested, this framework can be adopted by non-government organisations and governments for their respective M&Es.

In summary, there are several ethical codes of conduct and guidelines for M&E of public health programmes, but there is a definite need for a comprehensive ethical framework for M&E. This paper discusses some of the ethical considerations in the M&E process, presents an ethical framework, and discusses a case study applying the framework to it. Good codes of conduct for a proper M&E include avoiding conflicts of interest, transparency, full disclosure, accountability, empowerment of the community, independent judgement, impartiality, sustainability, respect, privacy, confidentiality, reporting wrong doing, due process and justice. Ultimately, ethical review of M&E should be institutionalised within organisations.

Acknowledgments

We acknowledge the significant intellectual input of Dr Amar Jesani (ethicist).

References

Footnotes

  • Contributors VG carried out the review of the literature and prepared the draft of the paper. AI added case studies, reviewed the draft, and revised subsequent versions of the manuscript.

  • Funding VG is supported by the INSPIRE Fellowship of the Department of Science and Technology, Government of India.

  • Competing interests None.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • i Owing to the politically sensitive nature of the case study, exact details of the place, time and administrative body responsible are not revealed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.