Program process monitoring

Last updated

Program process monitoring is an assessment of the process of a program or intervention. Process monitoring falls under the overall evaluation of a program. Program evaluation involves answering questions about a social program in a systematic way. Examples of social programs include school feeding programs, job training in a community and out-patient services of a community health care facility. Questions about a social program can be asked by program sponsors, developers, policymakers and even taxpayers who want to determine whether or not a particular program is effective. [1] More specifically, purposes of social programs include identifying a programs’ strengths and weaknesses, assessing the impact of a program, justifying the need for additional resources and responding to attacks on a program, among others. [1]

Program evaluation is a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency. In both the public and private sectors, stakeholders often want to know whether the programs they are funding, implementing, voting for, receiving or objecting to are producing the intended effect. While program evaluation first focuses around this definition, important considerations often include how much the program costs per participant, how the program could be improved, whether the program is worthwhile, whether there are better alternatives, if there are unintended outcomes, and whether the program goals are appropriate and useful. Evaluators help to answer these questions, but the best way to answer the questions is for the evaluation to be a joint project between evaluators and stakeholders.

Taxpayer person or organization subject to a tax on income

A taxpayer is a person or organization subject to pay a tax. Taxpayers have an Identification Number, a reference number issued by a government to its citizens.

Contents

Process monitoring

Apart from measuring the needs, inputs and outcomes of a program, evaluations also monitor the process of a program. [1] According to [2] (p. 171), program process monitoring “is the systematic and continual documentation of [the] key aspects of program performance that assesses whether the program is performing as intended or according to some appropriate standard.” Therefore, program process monitoring involves a systematic and continuous way of monitoring certain aspects of a program’s process which would indicate how well the program is performing, if at all. These activities help to facilitate the effective management of the program because continuous assessment allows for regular feedback about the program’s performance. [2] to

Tools for data gathering

Management Information Systems (MIS) are tools often used by evaluators to monitor the performance of the program. An MIS is a compilation of data regarding the services that are routinely provided by the program, the staff that have provided the services, the treatments that have been provided and the costs involved in the program. Many evaluators use MISs as their major source of data when monitoring the process of a program because MISs contain all the relevant information about the program that is needed for an assessment. [2]

A management information system (MIS) is an information system used for decision-making, and for the coordination, control, analysis, and visualization of information in an organization.

Process monitoring from various perspectives

The specific activities that form part of the monitoring process are aimed at generating information that is appropriate from the perspective of the consumer group, who the monitoring is for. [2] Three key consumer groups are discussed below:

From an accountability perspective

Due to outside funding, program managers need to provide regular information to funders and program sponsors about the implementation of the program, problems that have been encountered and how these were handled. Program managers are accountable for and how the program is run, which leads to either the success or failure of the program. [2]

Program management or programme management is the process of managing several related projects, often with the intention of improving an organization's performance. In practice and in its aims, program management is often closely related to systems engineering, industrial engineering, change management, and business transformation. In the defense sector, it is the dominant approach to managing very large projects. Because major defense programs entail working with contractors, it is often called acquisition management, indicating that the government buyer acquires goods and services by means of contractors.

From a management perspective

From a management perspective, process monitoring is not only aimed at finding out how the program is going, but it is also aimed at putting in place corrective measures to ensure that the program performs as it should. In this case, process monitoring can take place during the pilot testing of the program in order to find ways of dealing with unexpected problems. Management-oriented monitoring can also take place in already-developed programs in order to obtain information about the performance of the program and whether or not the target population is being reached by the planned intervention. [2]

From an evaluators perspective

In order to evaluate the outcomes of a program, the evaluator first needs to monitor the process in order to assess the implementation of the intervention. The reason for this is that many program failures are due to failures in the implementation of the program. Therefore, in order to determine whether or not the planned outcomes have been reached, the evaluator needs to assess how the intervention was implemented. [2]

Program process evaluation

Program process evaluation is an assessment of how the program is performing in terms of service utilisation and program organisation. Service utilisation examines program coverage in terms of intended services for the intended target population whereas program organisation looks at whether services provided are the ones actually laid down in the program.

Monitoring service utilisation

According to [3] (p. 136), an objective of many program projects is to ensure that project services or benefits reach a certain target population - defined in terms of its geographic, economic, or demographic characteristics. What is critical in program process monitoring is determining whether intended targets actually receive program services intended for them in the plan. This is of paramount importance especially in programs that are voluntary, that involve behaviour change, taking of instructions or learning of new procedures. [2]

Coverage and bias

Coverage refers to the extent to which a program reaches its intended target population whereas bias is the extent to which subgroups of a target population are reached unequally by a program [2] (p. 200). The aim of all programs is total coverage but very few social programs ever achieve total coverage, making bias an issue.

Causes of bias are, among others,:

Some programs can experience overcoverage, whereby the program captures numbers far exceeding the intended targets (e.g. feeding schemes). Depending on the nature of the program, overcoverage can be costly and problematic (e.g. social grants). However, most programs fail to achieve high target participation, either because of bias in the way targets are recruited or retained or because potential clients are unaware of the program, are unable to use it, or reject it. [2] (p. 185).

Measuring and monitoring coverage

Overcoverage and undercoverage are basic concerns in program implementation and should be measured and monitored to ascertain the extent of their effect on service utilisation. Efficient service utilisation requires that the program serve as many people as possible who are in need and very few who are not in need. A drawback to this exercise is that people may distort information so as to be eligible for the project . [3] [2] identified three sources of information that can be used to assess the extent to which a program is serving the appropriate target population:

Assessing bias

In assessing bias, questions like: Do all eligible individuals participate in the program? Are there any dropouts? What is the dropout rate? What are the causes of non-participation or dropout?, could be asked. Information obtained from answers to these questions is valuable in judging the effectiveness, worth and possible modification of the program to suit the needs of the target population. [2] The same methods of data collection mentioned under 'measuring and monitoring coverage' could be used to assess bias.

Related Research Articles

Evaluation is a systematic determination of a subject's merit, worth and significance, using criteria governed by a set of standards. It can assist an organization, program, design, project or any other intervention or initiative to assess any aim, realisable concept/proposal, or any alternative, to help in decision-making; or to ascertain the degree of achievement or value in regard to the aim and objectives and results of any such action that has been completed. The primary purpose of evaluation, in addition to gaining insight into prior or existing initiatives, is to enable reflection and assist in the identification of future change.

Educational assessment or educational evaluation is the systematic process of documenting and using empirical data on the knowledge, skill, attitudes, and beliefs to refine programs and improve student learning. Assessment data can be obtained from directly examining student work to assess the achievement of learning outcomes or can be based on data from which one can make inferences about learning. Assessment is often used interchangeably with test, but not limited to tests. Assessment can focus on the individual learner, the learning community, a course, an academic program, the institution, or the educational system as a whole. The word 'assessment' came into use in an educational context after the Second World War.

School psychology is a field that applies principles of educational psychology, developmental psychology, clinical psychology, community psychology, and applied behavior analysis to meet children's and adolescents' behavioral health and learning needs in a collaborative manner with educators and parents. School psychologists are educated in psychology, child and adolescent development, child and adolescent psychopathology, education, family and parenting practices, learning theories, and personality theories. They are knowledgeable about effective instruction and effective schools. They are trained to carry out psychological testing and psychoeducational assessment, counseling, and consultation, and in the ethical, legal and administrative codes of their profession.

Policy analysis is a technique used in public administration to enable civil servants, activists, and others to examine and evaluate the available options to implement the goals of laws and elected officials. The process is also used in the administration of large organizations with complex policies. It has been defined as the process of "determining which of various policies will achieve a given set of goals in light of the relations between the policies and the goals." Policy analysis can be divided into two major fields:

In social work, a caseworker is a social worker who is employed by a government agency, non-profit organization, or another group to take on the cases of individuals and provide them with advocacy, information and solutions. They are required by law to have a license in almost all states. Also, in political arenas, caseworkers are employed as a type of legislative staffer by legislators to provide service to their constituents such as dealing with individual or family concerns and obtaining social services through licensed professionals. British MPs and members of the United States Congress often provide constituent services through caseworkers for better use of their allotted funds.

The Joint Committee on Standards for Educational Evaluation is an American/Canadian based Standards Developer Organization (SDO). The Joint Committee represents a coalition of major professional associations formed in 1975 to help improve the quality of standardized evaluation. The Committee has thus far published three sets of standards for evaluations. The Personnel Evaluation Standards was published in 1988 and updated in 2008, The Program Evaluation Standards was published in 1994, and The Student Evaluation Standards was published in 2003.

Career assessments are tools that are designed to help individuals understand how a variety of personal attributes, impact their potential success and satisfaction with different career options and work environments. Career assessments have played a critical role in career development and the economy in the last century. Assessments of some or all of these attributes are often used by individuals or organizations, such as university career service centers, career counselors, outplacement companies, corporate human resources staff, executive coaches, vocational rehabilitation counselors, and guidance counselors to help individuals make more informed career decisions.

A needs assessment is a systematic process for determining and addressing needs, or "gaps" between current conditions and desired conditions or "wants". The discrepancy between the current condition and wanted condition must be measured to appropriately identify the need. The need can be a desire to improve current performance or to correct a deficiency.

Impact evaluation assesses the changes that can be attributed to a particular intervention, such as a project, program or policy, both the intended ones, as well as ideally the unintended ones. In contrast to outcome monitoring, which examines whether targets have been achieved, impact evaluation is structured to answer the question: how would outcomes such as participants' well-being have changed if the intervention had not been undertaken? This involves counterfactual analysis, that is, "a comparison between what actually happened and what would have happened in the absence of the intervention." Impact evaluations seek to answer cause-and-effect questions. In other words, they look for the changes in outcome that are directly attributable to a program.

The Communities In Schools (CIS) is a national organization working within public and charter schools in 25 states within the District of Columbia. It aims to build relationships that empower at-risk students to stay and perform well in school and become good achievers not just academically but also in life. Working in 2,300 schools and community-based sites in the most challenged communities, Communities In Schools serves 1.5 million young people and their families every year.

In the United States education system, School Psychological Examiners assess the needs of students in schools for special education services or other interventions. The post requires a relevant postgraduate qualification and specialist training. This role is distinct within school psychology from that of the psychiatrist, clinical psychologist and psychometrist.

The Interagency Working Group on Youth Programs is a group within the executive branch of the U.S. government, and is responsible for promoting healthy outcomes for all youth, including disconnected youth and youth who are at-risk. The Working Group also engages with national, state, local and tribal agencies and organizations, schools, and faith-based and community organizations that serve youth.

Community-based monitoring (CBM) is a form of public oversight, ideally driven by local information needs and community values, to increase the accountability and quality of social services such as health, development aid, or to contribute to the management of natural resources. Within the CBM framework, members of a community affected by a social program or environmental change track this change and its local impacts, and generate demands, suggestions, critiques and data that they then act on, including by feeding back to the organization implementing the program or managing the environmental change. For a Toolkit on Community-Based Monitoring methodology with a focus on community oversight of infrastructure projects, see www.communitymonitoring.org. For a library of resources relating to community-based monitoring of tropical forests, see forestcompass.org/how/resources.

Social and behavior change communication

Social and behavior change communication (SBCC), often also only "BCC" or "Communication for Development (C4D)" is an interactive process of any intervention with individuals, group or community to develop communication strategies to promote positive behaviors which are appropriate to their settings and there by solve world's most pressing health problems. This in turn provides a supportive environment which will enable people to initiate, sustain and maintain positive and desirable behavior outcomes.

The PRECEDE–PROCEED model is a cost–benefit evaluation framework proposed in 1974 by Lawrence W. Green that can help health program planners, policy makers and other evaluators, analyze situations and design health programs efficiently. It provides a comprehensive structure for assessing health and quality of life needs, and for designing, implementing and evaluating health promotion and other public health programs to meet those needs. One purpose and guiding principle of the PRECEDE–PROCEED model is to direct initial attention to outcomes, rather than inputs. It guides planners through a process that starts with desired outcomes and then works backwards in the causal chain to identify a mix of strategies for achieving those objectives. A fundamental assumption of the model is the active participation of its intended audience — that is, that the participants ("consumers") will take an active part in defining their own problems, establishing their goals and developing their solutions.

Badrul Khan's eight-dimensional e-learning framework is a detailed self-assessment instrument for institutions to organize their evaluation of educational technology (e-learning) readiness and opportunities for growth. The framework provides a structure for systematically reviewing initiatives and programs, so that desired learning outcomes are achieved. The framework is composed of eight dimensions, each reviewed by practical checklists of 50 - 70 questions. The checklist does not include a scoring system, but serves as an instrument that verifies that each area is cultivated.

Monitoring and Evaluation (M&E) is a process that helps improve performance and achieve results. Its goal is to improve current and future management of outputs, outcomes and impact. It is mainly used to assess the performance of projects, institutions and programs set up by governments, international organisations and NGOs. It establishes links between the past, present and future actions.

Strategic environmental assessment (SEA) is a systematic decision support process, aiming to ensure that environmental and possibly other sustainability aspects are considered effectively in policy, plan and program making. In this context, following Fischer (2007) SEA may be seen as:

CIPP evaluation model is a Program evaluation model which was developed by Daniel Stufflebeam and colleagues in the 1960s. CIPP is an acronym for Context, Input, Process and Product. CIPP is an evaluation model that requires the evaluation of context, input, process and product in judging a programme’s value. CIPP is a decision-focused approach to evaluation and emphasises the systematic provision of information for programme management and operation.

References

  1. 1 2 3 O’Sullivan, R. G. (2004). Practicing Evaluation: A Collaborative Approach. London: Sage Publications
  2. 1 2 3 4 5 6 7 8 9 10 11 12 13 Rossi, P. H., Lipsey, M. W. & Freeman, H. E. (2004). Evaluation: A Systematic Approach (7th ed.) London: Sage Publications
  3. 1 2 Valadez, J., Bamberger, M. (1994). Monitoring and Evaluating Social Programs in Developing Countries: A Handbook for Policymakers, Managers, and Researchers EDI Development Studies, The World Bank: USA
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.