Strengthening Health System Responses to Gender-based Violence in Eastern Europe and Central Asia

A resource package

5.5. Evaluation approaches

Many approaches to monitoring and evaluation are based on linear cause-effect models. Such models aim to logically connect a flow of inputs and activities to outputs and outcomes and to attribute change to an intervention. One of the most prominent linear models is the logical framework approach (described below). The strength of the logical framework approach is in making the underlying assumptions of one’s intervention transparent. It also helps to support a joint understanding among all involved actors of the logic underlying an intervention (meaning the relationship between inputs, activities and results). However, such linear models also have limitations, in particular when it comes to topics like violence against women. Sometimes a logical framework is limited in its ability to reflect the complexities of the real world and the different contributions made to the intended change by different stakeholders.

Therefore, health sector professionals might want to consider alternative approaches when developing a monitoring and evaluation framework. Three prominent alternative approaches are the most significant change technique, outcome mapping, and the quality of life battery method. They focus on the contribution one has made to a change, but do not intend to link the change to one’s contribution only (attribution). Furthermore, they do not work with a pre-defined set of indicators, but tend to be more flexible in measuring the impacts achieved.

It is recommended to take into account the advantages and disadvantages of different approaches, and - where feasible - to adapt and combine the approaches to fit the context and suit the exact needs of the corresponding intervention and setting.

5.5.1 The logical frameworkapproach

The logical framework approach (LFA) is a management tool mainly used in the design, monitoring and evaluation of international development interventions. A logical framework brings together all key components of a project, programme, policy or other intervention.Having all key components of an intervention in a systematic, concise and coherent way helps to clarify and demonstrate how interventions are expected to work. More specifically, it can help to link activities to different levels of results (objectives, outcomes, outputs). The LFA can also clarify and separate these various levels of results from each other in order to clearly distinguish between inputs, activities, outputs and objectives (see figure 13). In order to gain such clarity with all stakeholders involved, it is strongly recommended to develop the logical framework through a participatory approach and discuss all the assumptions and risks involved. The document will then serve as a reference for all stakeholders and staff involved during the implementation phase.

Figure 13: The logical model

The logical framework takes the form of a four by four table. The four rows are used to describe four different types of results the intervention aims to achieve or to contribute to: the objectives, outcomes, outputs and activities (OECD-DAC 2009). Objectives are the overall results to which an intervention is expected to contribute, for example, enhanced services for battered women or lower re-victimization rates. Outcomes are the short-term and medium-term effects of an intervention’s outputs, such as increased know-how among health personnel. Outputs are products and services which result from an intervention, for example training curriculum or health sector protocols. Outcomes and outputs need to be clear, simple and contain only one idea. For tips on formulating results statements, see UNFPA 2011b.

The four columns provide different types of information about the results in each row. The first column is used to provide a description. The second column lists one or more objectively verifiable indicators (OVIs) of these interventions. The third column describes the means of verification (MoV), i.e. the source of information on the OVIs, such as national statistics, training evaluations, interviews. The fourth column lists the assumptions. Assumptions are external factors that are believed to influence (positively or negatively) the events described in the narrative column. The list of assumptions should include those factors that potentially impact the success of the intervention but which cannot be directly controlled by the managers of the intervention. A good design of an intervention should be able to substantiate its assumptions, especially those with a high likelihood of having a negative impact.

A logical framework is usually developed from the top to the bottom (direction of thinking) and then during the monitoring process assessed from the bottom to the top (direction of action).

Table 13: Exemplary logical framework for a sexual and reproductive health intervention targeting GBV

The following list provides examples of results for interventions to end GBV. While managers of interventions may wish to draw on these examples when formulating results, it should be kept in mind that planned results have to be formulated according to the identified needs.

Supporting legal and institutional framework:

  • Increased harmonization of the national legislation in the area of GBV with international human rights standards
  • Clarified procedures for national response mechanisms
  • National Referral Mechanisms for the treatment of survivors of violence in place
  • Improved structures for assistance and rehabilitation of survivors established
  • Budget for good quality health services for survivors of violence ensured

Prevention:

  • Increased awareness of violence against women among health workers, teachers and social workers
  • Proportion of people who have been exposed to preventive messages against violence against women
  • Increased knowledge about patterns of violence in the region among the general public
  • Hotline providing information on support services established
  • Proportion of girls who feel able to say no to sexual activity
  • Increase in early detentions of perpetrators

Protection and support:

  • Improved protection of survivors
  • Improved standards for identification of and dealing with survivors of violence
  • Legal framework for the protection of survivors strengthened
  • Proportion of health units with at least one service provider trained to care for and refer survivors of violence
  • Improved system for dealing with minors as survivors of violence
  • Increased number of support facilities for survivors
  • Enhanced and comprehensive services (medical, legal, social) for survivors of violence
  • Proportion of people who would assist a woman being beaten by her partner
  • Proportion of rape survivors who received comprehensive care

Prosecution:

  • Legal system for the prosecution of perpetrators strengthened
  • Increased number of cases of domestic violence handled in courts
  • Increased number of cases of violence reported to the relevant authorities.

In order to implement a logical framework approach, indicators are vital to help to determine whether the planned results have been achieved. Indicators should follow the logic of the intervention (outcomes, outputs). Further, they provide a specific, observable, and measurable benchmark to show the changes or progress an intervention is making toward achieving a specific outcome, e.g. reduction of cases of domestic violence. The choice of indicators will determine how data is collected, interpreted and reported on. Indicators need to be disaggregated (male/female, age, rural/urban, ethnicity…) to the greatest extent possible and is necessary. Even if the indicator itself is not disaggregated, data collection should be disaggregated, so that the assessment can be specified according to different population groups.

Certain quality criteria should be applied to indicators. It should be checked that indicators are SMART and SPICED:

Box 29: SMART and SPICED indicators

Generally, a mix of both qualitative and quantitative indicators help to reflect the intended changes most adequately and allow for a triangulation of data collected. The following indicators are examples, which could be used. However, the sources and baselines or percentages indicated are purely indicative and would need to be established and replaced with those valid for your context. Remember that indicators, no matter whether quantitative or qualitative, have to be related to the formulated outcomes.

Quantitative indicators are statistical measures. They help to measure results in terms of (ICMPD 2010):

  • Numbers… e.g.: The number of survivors of violence assisted through specialized health institutions annually has risen from 78 (baseline 2012) to 100 (at the end of 2015) (Source: Statistics from NGOs).
  • Percentages… e.g.: The number of calls received by the national hotline increased by 25% by the end of 2015 (Source: Statistics from hotline) Or: The number of cases of domestic violence brought to court increased by 20% in 2014 (Source: Statistics of the Ministry of Justice, established baseline in 2012) Or: Proportion of health units that have documented & adopted a protocol for the clinical management of VAW/G survivors (Source: Health Ministry Statistics)
  • Rates… e.g.: The rate of re-affected survivors decreased by 25% between 2012 and 2015 (Source and baseline 2012: Health and police statistics) or: Rate of women who were asked about physical and sexual violence during a visit to a health unit.

Qualitative indicatorsreflect people’s judgments, opinions, perceptions and attitudes towards a given situation or subject. They can measure changes in sensitivity, satisfaction, influence, awareness, understanding, attitudes, quality, perception, dialogue or sense of well-being. Qualitative indicators measure results in terms of (ICMPD 2010):

  • Compliance with… e.g. set international/national standards, legislation, procedurese.g.: The national legal framework concerning GBV complies with the undersigned conventions and standards by 2015 (Source: Shadow reports.)
  • Quality of…e.g.: 80 % of interviewed (potential) survivors assess the quality of health services provided as adequate and targeted to their gender and age specific needs (Source: Qualitative survey). Or: In 2017, 80 % of survivors were satisfied with the assistance rendered in the process by the different authorities involved (Source: Qualitative survey conducted). 80% of women in rehabilitation programs assessed the programs as targeted to their needs (Source: Questionnaire).
  • Level of…e.g.: The level of knowledge on GBV among the participants of trainings has increased (Source: Questionnaire). Or: The level of coordination among the relevant stakeholders (government and non-governmental actors) for the fight against violence has increased (Source: Schedule of regular meetings and minutes, survey amongst stakeholders). Or: The (level of) awareness of the general population on governments action in the fight against violence has increased (Source: Survey and downloads of monitoring reports).
  • Extent of…e.g.: The extent of regular joint analysis of monitoring results increased significantly by the end of 2011 (Source: Anonymous questionnaire to stakeholders). The extent to which standards for interviewing survivors are observed increased by the end of 2011 (Source: Police reports, Interview protocols, baseline 2010). The extent of NGO involvement in the national anti-trafficking response significantly increased by the end of 2011 (Source: survey amongst NGOs). For tips on formulating indicators, see UNFPA 2011b.

The following case study from Russia provides an example of using a combination of qualitative and quantitative indicators for evaluation purposes (box 30).

Box 30: A case study from “Vrachi Detyam” (Doctors for Children) in Russia

In St. Petersburg, different organizations used different qualitative and quantitative data for the evaluation of the effectiveness of their work. Consequently, a joint model of identifying survivors of domestic violence in health care settings was developed. Today, if a survivor of domestic violence is identified in the screening process, she will be referred to specialists and will then be provided with telephone numbers of local crisis centres and receive information on counselling services. In order to assess the effectiveness of the referral, both the numbers of identified survivors and the number of survivors who turned to a crisis centre after intervention of the appropriate information was provided, are collected.

The data is then analysed according to the following formula: E = (В/А) х 100%. 

  • E stands for the effectiveness of identification and intervention;
  • В is the number of female survivors identified in the course of screening;
  • А is the number of women out of the number of identified survivors that turned to the crisis centre for further assistance.

In this particular context, 70 per cent was identified as target for the referral to be considered satisfactory.

For example, 70 survivors of domestic violence were identified by health care providers within one month. Interventions were conducted and contact information of crisis centres was provided. In the same month, 50 survivors turned to the crisis centre after the interventions were done in the health care setting. Based on the above-mentioned formula, the effectiveness of the referral was assessed as follows:

Indicator

Quantity

А. Number of survivors identified

70

В. Number of survivors who turned to crisis centre following referral

50

E. Effectiveness: 50/70 x 100% = 71%.

 

As E exceeds 70%, the referral in this example would be considered effective.

 

Source: Information provided by ANNA – Center for the Prevention of Violence, Russia 2011

5.5.2 Most significant change technique

This sub-chapter provides an overview of the most significant change (MSC) technique. It would exceed the scope of the current publication to provide a comprehensive introduction to this approach. For further information, readers are encouraged to consult Davies/Dart 2005, which is also the main reference for this sub-chapter.

“The most significant change technique is a form of participatory monitoring and evaluation. It is participatory because many project stakeholders are involved both in deciding the sorts of change to be recorded and in analyzing the data. It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program. It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole”(Davies/Dart 2005).

Essentially, the process involves the collection of significant change (SC) stories emanating from the field. At the outset, the designated staff and stakeholders are asked to ‘search’ for the impact of an intervention. Once the changes have been captured, the relevant stakeholders and staff sit down together, read the stories aloud and have regular and often in-depth discussions about the value of these reported changes. Then, panels of designated stakeholders engage in a systematic selection of the most significant of these stories. When the technique is implemented successfully, teams of health personnel or social workers are focusing on the impacts of the intervention rather than on the mere inputs provided.

Different from the logical framework approach, this technique does not work with pre-defined indicators, especially the ones that have to be counted or measured. The technique uses the following core question: “When looking back over the past months, what do you think was the most significant change?”

The technique usually incorporates ten steps; however, adaptations can be made to suit the specific context:

  1. Start and raise interest amongst your key stakeholders
  2. Define the domains of change
  3. Define the reporting period
  4. Collect SC stories
  5.  Select the most significant stories
  6. Feedback the results of the selection process
  7. Verify stories
  8. Quantify stories
  9. Secondary analysis and meta-monitoring
  10. Revise the system

The first step in MSC generally involves introducing MSC to a range of stakeholders in order to generate interest and commitment to participate. The second step is to identify the domains of change to be monitored, in this case, changes resulting from interventions against violence against women. This involves selected stakeholders identifying broad domains (e.g. enhanced access to health services for survivors of violence or better referral mechanisms). The third step is to decide how frequently to monitor changes taking place in these domains.

SC stories are collectedfrom those most directly involved, such as participants and field staff. The stories are collected by asking a simple question such as: ‘During the last month, in your opinion, what was the most significant change that took place for participants in the intervention?’ It is initially up to respondents to allocate their stories to a domain category (e.g. increased awareness, better referral, enhanced services). In addition to this, respondents are encouraged to report why they consider a particular change to be the most significant one.

Then, stories are analyzed and filtered at the different levels of authority typically found within a given organization or intervention. Each level of the hierarchy reviews a series of stories sent to them by the level below and selects the single most significant account of change within each of the domains. Each group then sends the selected stories up to the next level of the hierarchy of the intervention, so that the number of stories is reduced through a systematic and transparent process. Every time stories are selected, the criteria used to select them are recorded and fed back to all interested stakeholders, so that each subsequent round of story collection and selection is informed by feedback from previous rounds. The organization is effectively recording and adjusting the direction of discussion and, consequently, implementation.

After this process has been used for some time, for example one year, a collection of the selected stories is compiled at the uppermost organizational level in each domain of change. The reasons for selecting the stories are also mentioned. The funders of the intervention are asked to review the stories and to select those that best represent the sort of outcomes they wish to fund. They are also asked to document the reasons for their choice. This information is fed back to the managers.

The selected stories can be verified by visiting the sites where the described events took place. The purpose of this is two-fold: to check that stories have been reported accurately and honestly and to provide an opportunity to gather more detailed information about events seen as especially significant. If conducted sometime after the event, a visit also offers a chance to see what has happened since the event was first documented.

The next step is quantification, which can take place at two stages. When an account of change is first described, it is possible to include quantitative (e.g.: how many persons indicated that there is increased collaboration between different stakeholders working with survivors of domestic violence?) as well as qualitative information (e.g.: What kinds of characteristics were mentioned to indicate such a change?). It is also possible to compare such data between different locations and settings. The next step after quantification is monitoring the monitoring system itself, which can include looking at who participated and how they affected the contents, and analyzing how often different types of changes are reported. The final step is to revise the design of the MSC process to take into account what has been learned as a direct result of using it and from analyzing its use.

5.5.3 Outcome mapping

This sub-chapter provides an overview of outcome mapping. It would exceed the scope of the current publication to provide a comprehensive introduction to this approach. For further information, readers are encouraged to consult Patton 2010, which is the main reference for this sub-chapter, as well as the online platform of Outcome Mapping Learning Community (www.outcomemapping.ca).

The technique of outcome mapping focuses on measuring changes in the behaviour of the people with whom given intervention works most closely. Examples for such changed behaviours can be enhanced collaboration mechanisms between health centres and crisis intervention centres or increased awareness of the different forms of violence amongst health personnel. The main difference between the logical framework approach and outcome mapping is that the first also looks at overall objectives (e.g. eliminating violence against women), while the latter focuses only on those outcomes that fall strictly within the sphere of influence of a particular intervention (e.g. improved skills of health professionals trained). Thus, outcome mapping considers only those activities where the intervention can claim that it has had a direct impact on. This technique addresses only one type of results: behavioural change; it is oriented towards social and organizational learning.

Outcome mapping works with boundary partners, which are those individuals, groups and organizations with whom the intervention interacts directly and with whom the intervention anticipates opportunities for influence. For example, a health intervention might support a health facility in order to enhance its services for survivors of violence. In order to provide also social and legal services, the health facility collaborates with a non-governmental organization and government stakeholders (partners). Following the approach of outcome mapping, these partners should assess their own services instead of the intervention doing it for them.

The basic assumption of outcome mapping is that the direct influence of an intervention decreases from each level of the intended change, as the graph below demonstrates (figure 14). Instead, the ownership of the other stakeholders increases. This increased ownership of local stakeholders is also regarded as a prerequisite for sustainability. For example, an intervention is targeting enhanced referral mechanisms. A hospital can provide information on services and refer a survivor to a shelter. However, the hospital will not be able to guarantee that the woman is actually using the recommended services nor that these services are provided according to the standards of the hospital - the latter is the responsibility of the service providers and shelters. Consequently, the hospital would measure its referrals, whereas the service provider would evaluate its services. As the name indicates, the focus of outcome mapping as a monitoring and evaluation approach lays on the outcome level.

Figure 14: Changes in influence of an intervention and local ownership at the different levels of the results chain

There are three stages of outcome mapping, which involve a total of twelve steps:

The first stage, intentional design,helps to establish consensus among the relevant stakeholders on the intended macro level changes of the intervention, and to plan the strategies to be used. To this end the following four questions are being asked:

  • Why?What is the vision to which the intervention wants to contribute?
  • Who?Who are the intervention's boundary partners?
  • What?What are the changes that are being sought?
  • How?How will the programme contribute to the change process?

The intended changes are described as outcomes, which are defined in progress markers. Progress markers identify as a set of changed behaviour by the boundary partners. These changes are described and measured on three levels (behaviour change we plan to see, behaviour change we like to see and behaviour change we love to see).

The second stage,outcome and performance monitoring, provides a framework for the on-going monitoring of the intervention’s actions and the boundary partners' progress toward the achievement of outcomes. It is based largely on systematized self-assessment. It provides the following data collection tools for the elements identified in the intentional design stage (vision, mission, boundary partners, progress markers): an "outcome journal"; a "strategy journal"; and a "performance journal." An outcome journal is used to monitor the intended outcomes and progress markers. A strategy journal documents changes in strategy and activities. The performance journal monitors organizational practices.

The third stage, evaluation planning, helps to identify evaluation priorities and develop an evaluation plan.

The following case study (box 31) illustrates the use of outcome mapping in an intervention to improve the health of women and girls in rural India.

Box 31: Improving the Health of Women and Girls in Rural India

Swayamsiddha is a 5-year project (2000-2005) to improve the health and empowerment of women and girls in rural India. The project involves nine partner organizations in six Indian states and is co-funded by the Canadian International Development Agency (CIDA) and the International Development Research Centre (IDRC). It is designed to reach about 75 villages and provide benefits to the female members of community-based organizations, as well as their families. The goal of Swayamsiddha is to bring about behavioural changes at different levels: within the community at large; among the women with whom the project is working directly; through the establishment and strengthening of community-based organizations; and among the implementing teams and within their organizations. The Swayamsiddha project team has been using outcome mapping identification of boundary partners, outcome challenges, and progress markers. The staff reflects: “Outcome mapping is a useful tool for seeing the process of change as a continuous chain of many smaller changesand states that“outcome mapping can improve planning and monitoring.”

Source: Earl undated

5.5.4 The Quality of Life Battery method

This sub-chapter provides an overview of the Quality of Life Battery method. It would exceed the scope of the current publication to provide a comprehensive introduction to this approach. For further information, readers are encouraged to consult Jones 2010, which is also the main reference for this sub-chapter.

The Quality of Life Battery Method is a simple approach using the metaphor of a full or empty battery, which explores what quality of life means to beneficiaries and identifies changes which the intervention has had on different areas of their lives. The methodology was developed by Clodagh Byrne to support partners in assessing changes in the life of the beneficiaries of the intervention and to increase participation of clients in the planning and monitoring of interventions. The method was originally piloted by Cambodia HIV/AIDS Education and Care (CHEC). It can be adapted when working with survivors of GBV. The information drawn from this evaluation methodology may be used by clients and program staff to review progress and identify future actions that can be taken by the client and within the program. Data analysis can also help to identify whether or not a holistic response is in place and if it is actually improving the quality of life of the beneficiaries.

Figure 15: Example of a Quality of Life Battery

This tool involves three main steps:

  1.  Participants identify key elements of a good quality of life and categorize them into domains (e.g. health, emotional happiness, human rights, livelihood security). Following the intervention, participants fill in two batteries: one to assess their quality of life at the end of the intervention and one to assess quality of life before the intervention.
  2. Using the image of batteries, participants assess changes in different areas of their lives during the programme and examine the reasons for these changes.
  3. The data is analyzed and recommendations on how the program can further improve the quality of life are collected.

Step 1: Identification of key elements

Quality of life means different things to different people and depends on personal views and the social environment in which people live. Factors that make one person happy and fulfilled may not have the same effect on other people. Common domains are health, psychosocial happiness, human rights and livelihood security. These categories should be adapted according to the context. The question is: “What elements do you need to have a full and happy life?”

Step 2: Definition of batteries fullness level and reflection of underlying reasons

To assess changes, this methodology requires participants to consider their quality of life at two or more points in time, using the image of full/empty batteries. This should ideally be done by carrying out the process at different stages of an intervention. Alternatively, it can be a retrospective process where the participant considers their quality of life at present and then thinks back to how it was at a previous point in time. Two or more sets of batteries are compared, one for each point in time. Each battery is divided into levels ranging from 1-10, representing low-high energy levels, to help measure and record their evaluation. It is important that participants consider their own situation and not that of their family members, partners or a person that they care for. Whilst assigning energy levels to their batteries, participants should reflect the reasons for the indicated energy level. An important part of the process lies in recording these reasons. This reflection not only helps individuals to think about actions they can take in their own lives to further improve their quality of life; the process can also be used to adapt future program practice.

The following case study illustrates how the Qualities of Life Batteries method has been used in the health context, albeit slightly adapted (box 32):

Box 32: Case study: Medical Missionaries of Mary, Addis Ababa, Ethiopia

Instead of using diagrams of the batteries, participants in Addis Ababa used drinking glasses to represent the batteries and coloured water to represent their energy levels. This was done in order to evaluate the impact of a health program on individual participants. The process was carried out looking back on how participants felt their quality of life had been before they joined the program. Participants filled their “now glass” and discussed the reasons for choosing this level. They then filled their “before” glass and discussed the reasons for changes between the levels. The facilitators used cards marked with 10 levels to measure the level of the liquid and record this for each domain. The level changes were then transferred by facilitators to the batteries diagram and these, along with the reasons for the changes, made were recorded on a record sheet.

Source: Jones 2010

Step 3: Interpreting the data

An important next step is the accurate collation and analysis of data after the “batteries process” (step 2) has been carried out. While the process provides a useful individual review and planning tool, it also has wider relevance for evaluating interventions when the data from a range of program clients is collected and analyzed. During the process, both quantitative (the batteries energy level scores) and qualitative (the reasons for these scores) data is collected by staff or evaluators. Looking at a number of program clients over a specific period of time enables program managers to identify potential trends where a program has influenced the quality of their lives. If these are not the intended changes, the program can be adapted. It is recommended to cross-check the analysis with respondents and to collect their recommendations on how the program can be improved.