Outcome mapping

Last updated

Outcome mapping is a project progress measurement system that was designed by the grant-making organisation International Development Research Centre (IDRC). It differs from traditional metrics in that it does not focus on measuring deliverables and its effects on primary beneficiaries but on behavioural change exhibited by secondary beneficiaries. The outcome mapping process consists of a lengthy design phase followed by a cyclical record-keeping phase. Outcome mapping is intended primarily for change focussed organizations that deal with complex systems and issues in changing environments, though it was originally designed for evaluating the impact of research in the developing world.

Contents

History

The outcome mapping manual was first published in 2001 by the Evaluation Unit of the IDRC. It is based on the outcome engineering model developed by Barry Kibel of the Pacific Institute for Research and Evaluation. Much of the terminology and procedures that occur in outcome engineering are also present in outcome mapping. [1]

Overview of method

The key difference between outcome mapping and most other project evaluation systems is its approach to the problem that a project's direct influence over a community only lasts for as long as the project is running, and developing agencies have difficulty in attributing resultant change in those communities directly to the actions of the project itself. The outcome mapping approach is to focus less on the direct deliverables of the project and to focus more on the behavioural changes in peripheral parties affected by the project team. Thus an outcome mapped project report will focus less on the project's actual progress and more on the project's influence (both deliberate and unintended) during the project's progression. [2]

Report-backs based on outcome mapping would tend to contain more anecdotal evidence than more traditional measurement approaches. Because of its deviation from traditional evaluation philosophy, outcome mapping may not be accepted by all donor organisations. [3]

Outcome mapping consists of two phases, namely a design phase and a record-keeping phase. During the design phase, project leaders identify metrics in terms of which records will be kept. [4]

Design phase

Writing two visionary essays

Project leaders begin the outcome map design by writing two short essays about what effect they would like to see their project have on the community and how they believe those influences would come about, written from a wishful thinking type of perspective, with the presupposition that their project is successful beyond their wildest dreams. The sole purpose of writing the essays is to facilitate a mind-shift among project leaders to the larger environment in which their project resides. [5] In outcome mapping terminology, these essays are termed a vision and mission statement, but they have little in common with vision and mission statements used in strategic planning.

Identifying boundary partners

The philosophy of outcome mapping is that anyone the project influences is in fact a partner in that project. For this reason, project leaders draw a schematic (with the project in the centre) about which parties will be influenced by the project in any way (direct boundary partners), and which parties will in turn be influenced by those parties (indirect boundary partners). Project leaders select three or four boundary partners upon which they will focus additional activities (called primary boundary partners). Typically the direct recipients or beneficiaries of the project's deliverables will be one of the primary boundary partners. All further processes and reports within the outcome mapping with focus on various aspects of the primary boundary partners. In IDRC sponsored projects, project leaders are encouraged to include policy makers (e.g. governments) in their list of indirect boundary partners.

Defining progress markers

For each primary boundary partner, project leaders write a statement of desired overall behavioural change (called an outcome challenge) and a list of eight to eleven specific behavioural changes or actions the project would like the boundary partner to exhibit by the end of the project (called progress markers).

There are three types of progress markers, namely expect-to-see, like-to-see and love-to-see. These progress markers can be defined according to two approaches, namely term duration and degree of realism. In the term duration approach, expect-to-see are immediate responses that will be expected during the project itself, like-to-see are long-term responses that one would expect by the end of the project, and love-to-see are responses that one might expect after a number of months or years after the project came to an end. In the degree of realism approach, expect-to-see are items that are brutally realistic, like-to-see are items that are somewhat idealistic, and love-to-see are items that are actually unrealistic.

Identify tactics for boundary partners

Outcome mapping defines six types of actions that project leaders can take to effect each individual progress marker. These actions are filled in into a grid of tactics called in outcome mapping terminology a strategy map. The cells on the grid are called I1, I2, I3, E1, E2, and E3. I-class tactics ("internal" or "individual") are aimed at specific groups and individuals. E-class tactics ("external" or "environment") are aimed at the surrounding situation that groups or individuals may find themselves in.

Items I1 and E1 are called causal, I2 and E2 are called persuasive, and I3 and E3 are called supportive. Causal actions are directly related to the desired outcome and have a single purpose, persuasive actions are indirectly related to the desired outcome or attempts to produce the desired outcome indirectly, and supportive actions relate to providing and fostering an environment that enables or encourages the desired outcome with very little direct bearing on it.

It is not required to define actions for all six tactics for each progress marker—some progress markers may be suited to only one or two of these types of tactics. It is these tactics or actions that project leaders will perform during the project (in addition to producing the project's actual deliverables) and which project leaders will report back on. Interviews, with 6 consultants, occurred in order to learn experiences of the individuals in providing OM training. Together, the interviews with the summary of the data that was collected on applications and practitioners represent a valuable advance in comparison to the data previously held by OMLC. Of the status quo of OM, however, this study is not a good representation. User experiences and support available and required makes it distinctive for the experimenters and individuals.

Identify tactics for the visionary essays

After writing the visionary essays, and typically only after the rest of the design phase is completed, project leaders write down answers to eight specific questions about the visionary essays. These items are called organisational practices in outcome mapping terminology. The answers to these eight questions will be amended as the project progresses. The eight questions are:

  1. Where or how will we find out about new opportunities or resources to achieve our vision and mission?
  2. Who can we ask (in addition to our project beneficiaries) what our project beneficiaries think of our project?
  3. How will we get support and approval from organisations' management or from government departments for what we do?
  4. What procedures will be follow to ensure that we re-evaluate our methods continuously?
  5. How will we determine from our boundary partners whether methods we intend to use, has been tried and found wanting?
  6. How and what will we share about what we've learnt during the project?
  7. What systems will we put in place to foster responsible experimentation with our limited resources?
  8. When will we have feedback and bonding sessions for project management team members?

Record-keeping phase

In outcome mapping, three types of records can be kept, and it is largely up to the project leaders or donor organisation to decide which of the three (or all three) types of records should be reported back on. The records related to the items of the design phase.

Performance journal

The report named a performance journal is essentially a collection of minutes of the meetings at which the project's progress with regard to the organisational practices.

Strategy journal

The report named a strategy journal is a record of actions taken in terms of the strategy map (or tactics grid) along with results of such actions. The strategy journal is kept up to date continuously but may be required to be submitted to the donor at intervals.

Outcome journal

The report named an outcome journal is an anecdotal record of any events that related directly or indirectly to the progress markers (the expect-to-see, like-to-see and love-to-see items). The outcome journal is most useful towards the end of the project in providing the donor with visible impact stemming from the expenditure of funds, but may also be submitted to the donor at intervals.

Criticism

Since outcome mapping is more concerned about contribution than attribution, it is said that it should not replace traditional metrics but merely complement it. [6] It has also been said that outcome mapping simply side-steps the attribution issue. [7] An issue which has been addressed at IDRC workshops is the objection that outcome mapping procedures are too long and complex. [8]

See also

Related Research Articles

Project management is the process of leading the work of a team to achieve goals and meet success criteria at a specified time. The primary challenge of project management is to achieve all of the project goals within the given constraints. This information is usually described in project documentation, created at the beginning of the development process. The primary constraints are scope, time, budget. The secondary challenge is to optimize the allocation of necessary inputs and apply them to meet pre-defined objectives.

Work breakdown structure

A work-breakdown structure (WBS) in project management and systems engineering, is a deliverable-oriented breakdown of a project into smaller components. A work breakdown structure is a key project deliverable that organizes the team's work into manageable sections. The Project Management Body of Knowledge defines the work-breakdown structure as a "hierarchical decomposition of the total scope of work to be carried out by the project team to accomplish the project objectives and create the required deliverables."

Program evaluation is a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency. In both the public and private sectors, stakeholders often want to know whether the programs they are funding, implementing, voting for, receiving or objecting to are producing the intended effect. While program evaluation first focuses around this definition, important considerations often include how much the program costs per participant, how the program could be improved, whether the program is worthwhile, whether there are better alternatives, if there are unintended outcomes, and whether the program goals are appropriate and useful. Evaluators help to answer these questions, but the best way to answer the questions is for the evaluation to be a joint project between evaluators and stakeholders.

PDCA is an iterative design and management method used in business for the control and continuous improvement of processes and products. It is also known as the Deming circle/cycle/wheel, the Shewhart cycle, the control circle/cycle, or plan–do–study–act (PDSA). Another version of this PDCA cycle is OPDCA. The added "O" stands for observation or as some versions say: "Observe the current condition." This emphasis on observation and current condition has currency with the literature on lean manufacturing and the Toyota Production System. The PDCA cycle, with Ishikawa’s changes, can be traced back to S. Mizuno of the Tokyo Institute of Technology in 1959.

A value network is a graphical illustration of social and technical resources within/between organizations and how they are utilized. The nodes in a value network represent people or, more abstractly, roles. The nodes are connected by interactions that represent deliverables. These deliverables can be objects, knowledge or money. Value networks record interdependence. They account for the worth of products and services. Companies have both internal and external value networks.

Nursing process

The nursing process is a modified scientific method. Nursing practise was first described as a four-stage nursing process by Ida Jean Orlando in 1958. It should not be confused with nursing theories or health informatics. The diagnosis phase was added later.

Capacity building Process by which individuals and organizations obtain, improve, and retain the skills and knowledge needed to do their jobs competently

Capacity building is the process by which individuals and organizations obtain, improve, and retain the skills, knowledge, tools, equipment, and other resources needed to do their jobs competently. It allows individuals and organizations to perform at a greater capacity. "Capacity building" and "Capacity development" are often used interchangeably. This term indexes a series of initiatives from the 1950s in which the active participation of local communities’ members in social and economic development was encouraged via national and subnational plans. The United Nations Sustainable Development Goal 17 advocates for enhanced international support for capacity building in developing countries to support national plans to implement the 2030 Agenda.

Logic model Method of depicting causal relationships

Logic models are hypothesized descriptions of the chain of causes and effects leading to an outcome of interest. While they can be in a narrative form, logic model usually take form in a graphical depiction of the "if-then" (causal) relationships between the various elements leading to the outcome. However, the logic model is more than the graphical depiction: it is also the theories, scientific evidences, assumptions and beliefs that support it and the various processes behind it.

Renewable Energy and Energy Efficiency Partnership Organization for renewable energy

The Renewable Energy and Energy Efficiency Partnership (REEEP) is a Vienna-based international organisation that advances markets for renewable energy and energy efficiency with a particular emphasis on the emerging markets and developing countries. Its primary focus is in de-risking and scaling up clean energy business models.

Impact evaluation assesses the changes that can be attributed to a particular intervention, such as a project, program or policy, both the intended ones, as well as ideally the unintended ones. In contrast to outcome monitoring, which examines whether targets have been achieved, impact evaluation is structured to answer the question: how would outcomes such as participants' well-being have changed if the intervention had not been undertaken? This involves counterfactual analysis, that is, "a comparison between what actually happened and what would have happened in the absence of the intervention." Impact evaluations seek to answer cause-and-effect questions. In other words, they look for the changes in outcome that are directly attributable to a program.

Humanitarian Accountability Partnership International

Humanitarian Accountability Partnership International, established in 2003, was the humanitarian sector's first international self-regulatory body. A multi-agency initiative working to improve the accountability of humanitarian action to people affected by disasters and other crises, HAP members ranged from organisations with a mandate for emergency relief and development activities to institutional donors. The organisation aimed to strengthen accountability towards those affected by crisis situations and to facilitate improved performance within the humanitarian sector. The ultimate goal of the organisation was to uphold the rights and the dignity of crisis-affected populations across the world.

Theory of change

Theory of Change (ToC) is a methodology for planning, participation, and evaluation that is used in companies, philanthropy, not-for-profit and government sectors to promote social change. Theory of Change defines long-term goals and then maps backward to identify necessary preconditions.

Small-scale project management is the specific type of project management of small-scale projects. These projects are characterised by factors such as short duration; low person hours; small team; size of the budget and the balance between the time committed to delivering the project itself and the time committed to managing the project. They are otherwise unique, time delineated and require the delivery of a final output in the same way as large-scale projects.

In organizational theory, organizational analysis or industrial analysis is the process of reviewing the development, work environment, personnel, and operation of a business or another type of association. This review is often performed in response to crisis, but may also be carried out as part of a demonstration project, in the process of taking a program to scale, or in the course of regular operations. Conducting a periodic detailed organizational analysis can be a useful way for management to identify problems or inefficiencies that have arisen in the organization but have yet to be addressed, and develop strategies for resolving them.

Participatory evaluation is an approach to program evaluation. It provides for the active involvement of stakeholder in the program: providers, partners, beneficiaries, and any other interested parties. All involved decide how to frame the questions used to evaluate the program, and all decide how to measure outcomes and impact. It is often used in international development.

Opportunity management (OM) has been defined as "a process to identify business and community development opportunities that could be implemented to sustain or improve the local economy".

The Enhanced Integrated Framework for Trade-Related Assistance for the Least Developed Countries is a global development program with the objective of supporting least developed countries (LDCs) to better integrate into the global trading system and to make trade a driver for development. The multi-donor program was launched on 1 January 2007 as the successor of the Integrated Framework for Trade-Related Technical Assistance to the Least-Developed Countries, which existed from October 1997 to December 2006. The second phase of the EIF has started on 1 January 2016 and will last for 7 years.

An adaptive clinical trial is a clinical trial that evaluates a medical device or treatment by observing participant outcomes on a prescribed schedule, and modifying parameters of the trial protocol in accord with those observations. The adaptation process generally continues throughout the trial, as prescribed in the trial protocol. Modifications may include dosage, sample size, drug undergoing trial, patient selection criteria and "cocktail" mix. In some cases, trials have become an ongoing process that regularly adds and drops therapies and patient groups as more information is gained. Importantly, the trial protocol is set before the trial begins; the protocol pre-specifies the adaptation schedule and processes.

Monitoring and Evaluation (M&E) is used to assess the performance of projects, institutions and programmes set up by governments, international organisations and NGOs. Its goal is to improve current and future management of outputs, outcomes and impact. Monitoring is a continuous assessment of programmes based on early detailed information on the progress or delay of the ongoing assessed activities. An evaluation is an examination concerning the relevance, effectiveness, efficiency and impact of activities in the light of specified objectives.

Constituent Voice is a performance management and measurement method designed as a continual improvement process developed by Keystone Accountability to enable organisations addressing social issues to improve their results by improving relationships with their constituents. Like customer satisfaction measurement, which it draws from, Constituent Voice treats measurement as an aspect of an intervention that not only provides metrics but contributes directly to outcomes by increasing the engagement of intended beneficiaries.

References

  1. ↑ "Knowledge Sharing Tools and Methods Toolkit - Outcome Mapping". Kstoolkit.org. Retrieved 2012-04-02.CS1 maint: discouraged parameter (link)
  2. ↑ Earl, Sarah; Carden, Fred; Smutylo, Terry (2001). Outcome Mapping. International Development Research Centre. p. 1. ISBN   0-88936-959-3.
  3. ↑ "Outcome Mapping". wikiadapt.org.
  4. ↑ "Outcome Mapping". Fmfi. 2007-12-21. Retrieved 2012-04-02.CS1 maint: discouraged parameter (link)
  5. ↑ "decathlon:outcomemapping · Translate Toolkit & Pootle". Translate.sourceforge.net. 2008-05-05. Retrieved 2012-04-02.CS1 maint: discouraged parameter (link)
  6. ↑ "Australian Development Gateway: Ask the Monitoring & Evaluation (M&E) specialist - Dr Paul Crawford => Evaluating the Outcome Mapping Methodology proposed by IDRC in Canada". Developmentgateway.com.au. Retrieved 2012-04-02.CS1 maint: discouraged parameter (link)
  7. ↑ "page 28" (PDF). Retrieved 2012-04-02.CS1 maint: discouraged parameter (link)
  8. ↑ "page 22 "The most common arguments for and against OM"" (PDF). Retrieved 2012-04-02.CS1 maint: discouraged parameter (link)
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.