Theory-driven evaluation

Last updated

Theory-driven evaluation is an umbrella term for any approach to program evaluation that develops a theory of change and uses it to design, implement, analyze, and interpret findings from an evaluation. [1] [2] [3] More specifically, an evaluation is theory-driven if it: [4]

Contents

  1. formulates a theory of change using some combination of social science, beneficiary lived experience, and program-related professionals' expertise;
  2. develops and prioritizes evaluation questions using the theory;
  3. uses the theory to guide the design and implementation of the evaluation;
  4. uses the theory to operationalize contextual, process, and outcome variables; and
  5. provides a causal explanation of how and why outcomes were achieved, including whether the program worked and/or had any unintended consequences (desirable or harmful), and what moderates outcomes.

By investigating the mechanisms through which outcomes are achieved, theory-driven approaches facilitate learning to improve programs and how they are implemented, and help knowledge to accumulate across apparently different programs. [5] [6] This is in contrast to methods-driven "black box" evaluations, which focus on following the steps of a method (for instance, randomized experiment or focus group) and only assess whether a program leads to its intended outcomes. [7] Theory-driven approaches can also improve the validity of evaluations, for instance leading to more precise estimates of impact in randomized controlled trials. [8]

History

Theory-driven evaluation emerged in the 1970s and 80s in response to the limitations of methods-driven "black box" evaluations. The term theory-driven evaluation was coined by Huey T. Chen and Peter H. Rossi. [9] Chen (1990) [10] wrote the first comprehensive introduction to conducting theory-driven evaluations, for example explaining how to develop a program theory of change and the different types of design. Its origins have been traced [11] to a book by Carol Weiss (1972) [12] and a rarely-cited article by Carol Taylor Fitz-Gibbon and Lynn Lyons Morris (1975). [13] However, "the first published use of what we would recognize as program theory" was in an evaluation of training programs, by Don Kirkpatrick in 1959. [14]

Funnell and Rogers (2011, pp. 23–24) comment on the confused nomenclature of the field, enumerating 22 approaches such as theory-based evaluation and program theory-driven evaluation science that are equivalent to or overlap significantly with theory-driven evaluation. The first definition of theory-based evaluation, by Fitz-Gibbon and Morris (1975), is near-identical to theory-driven evaluation: [15]

A theory-based evaluation of a program is one in which the selection of program features to evaluate is determined by an explicit conceptualization of the program in terms of a theory […] which attempts to explain how the program produces the desired effects. The theory might be psychological […] or social psychological […] or philosophical […]. The essential characteristic is that the theory points out a causal relationship between a process A and an outcome B.

Consequently, the terms theory-driven and theory-based evaluation are often used interchangeably in the literature. [16] [17] [18] However, theory-based evaluation is sometimes interpreted more narrowly to mean qualitative or small-n case study-based evaluations conducted without a comparison group, for example using process tracing or qualitative comparative analysis. [19] [20]

What is meant by "theory"?

The theory of theory-driven evaluation seeks to be as close as possible to the proximal causes of a social problem and site of intervention rather than, for instance, a "grand" theory, that tries to provide an overarching understanding of society, or a metaphysical theory about the nature of social reality: [21]

It advances evaluation practice very little to adopt one or another of current global theories in attacking, say, the problem of juvenile delinquency, but it does help a great deal to understand the authority structure in schools and the mechanisms of peer group influence and parental discipline in designing and evaluating a program that is supposed to reduce disciplinary problems in schools. [...T]he theory-driven perspective is closer to what econometricians call "model specification" than are more complicated and more abstract and general theories.

A distinction is also drawn between normative theory, concerning what a program is supposed to do and how it should be implemented, and causal theory, which specifies how the program is thought to work. [22] There can then be two broad ways in which a program fails to lead to the desired outcomes: (1) a program may be implemented as intended according to the normative theory; however, the causal theory is incorrect; and (2) the causal theory is correct; however, the program was not implemented correctly. [23]

Chen's action model/change model schema

Chen's action model/change model schema [24] provides an example of how a program theory and its context are conceptualized. The elements of the schema are then completed for each particular program.

Chen's (2015) action model/change model schema Chen's Action Model - Change Model Schema.svg
Chen's (2015) action model/change model schema

The change model specifies how an intervention of a program leads to outcomes via determinants, also known as intermediate or mediating variables.

The action model specifies how staff and delivery organizations deliver the intervention to beneficiaries:

Theory-driven methods

The full-range of research methods has been argued to apply. For instance, Chen (2015) provides examples using randomized experiments, quasi-experimental designs, process and outcome monitoring, and qualitative methods. [25] Although proponents of theory-driven evaluation are critical of "black box" experiments, Chen and Rossi (1983, p. 292) [26] argue that theory-driven experiments are possible and desirable:

[A]dvocates of the black box experimental paradigm often neglect the fact that after randomization exogenous variables are still correlated with outcome variables. Knowing how such exogenous factors affect outcomes makes it possible to construct more precise estimates of experimental effects by controlling for such exogenous variables.

It has been argued that theory-driven evaluation focusses too much on statistical approaches, such as randomized experiments, quasi-experiments, and structural equation modelling; [27] however, a case has also been made for the importance of qualitative methods, particularly when developing program theories and understanding implementation. [28]

There is also methodological debate concerning whether realist evaluations, considered a particular kind of theory-driven approach, may include randomized controlled trials in any form. Some evaluators think they may and conduct what they call "realist trials". [29] [30] [31] Others argue that a realist trial is an "oxymoron", and recommend instead calling them "theory-oriented trials". [32] A 2023 review of purported realist trials concluded that whether they are really realist depends on "ontological and epistemological" commitments of evaluators and that differences "cannot be resolved" by reviewing studies conducted. [33]

Examples

Examples discussed in a 2011 systematic review of 45 theory-driven evaluations include: [34]

A 2014 review of theory-driven evaluation in school psychology [38] highlighted two illustrative examples:

Related Research Articles

Program evaluation is a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency.

<span class="mw-page-title-main">Field experiment</span> Experiment conducted outside the laboratory

Field experiments are experiments carried out outside of laboratory settings.

A school counselor is a certified/licensed professional that provides academic, career, college readiness, and social-emotional support for all students. There are school counselor positions within each level of schooling. By developing and following a school counseling program, school counselors are able to provide students of all ages with the appropriate support and guidance needed for overall success.

<span class="mw-page-title-main">Transtheoretical model</span> Integrative theory of therapy

The transtheoretical model of behavior change is an integrative theory of therapy that assesses an individual's readiness to act on a new healthier behavior, and provides strategies, or processes of change to guide the individual. The model is composed of constructs such as: stages of change, processes of change, levels of change, self-efficacy, and decisional balance.

The social norms approach, or social norms marketing, is an environmental strategy gaining ground in health campaigns. While conducting research in the mid-1980s, two researchers, H.W. Perkins and A.D. Berkowitz, reported that students at a small U.S. college held exaggerated beliefs about the normal frequency and consumption habits of other students with regard to alcohol. These inflated perceptions have been found in many educational institutions, with varying populations and locations. Despite the fact that college drinking is at elevated levels, the perceived amount almost always exceeds actual behavior. The social norms approach has shown signs of countering misperceptions, however research on changes in behavior resulting from changed perceptions varies between mixed to conclusively nonexistent.

Evidence-based policy is a concept in public policy that advocates for policy decisions to be grounded on, or influenced by, rigorously established objective evidence. This concept presents a stark contrast to policymaking predicated on ideology, 'common sense,' anecdotes, or personal intuitions. The methodology employed in evidence-based policy often includes comprehensive research methods such as randomized controlled trials (RCT). Good data, analytical skills, and political support to the use of scientific information are typically seen as the crucial elements of an evidence-based approach.

<span class="mw-page-title-main">Systematic review</span> Comprehensive review of research literature using systematic methods

A systematic review is a scholarly synthesis of the evidence on a clearly presented topic using critical methods to identify, define and assess research on the topic. A systematic review extracts and interprets data from published studies on the topic, then analyzes, describes, critically appraises and summarizes interpretations into a refined evidence-based conclusion. For example, a systematic review of randomized controlled trials is a way of summarizing and implementing evidence-based medicine.

In education, Response to Intervention is an approach used to provide early, systematic, and appropriately intensive supplemental instruction and academic support to children who are at risk for or already underperforming as compared to appropriate grade or age level standards. However, to better reflect the transition to a broader approach to intervention, there has been a shift in recent years from the terminology referring to RTI to MTSS, which stands for "Multi-Tiered System of Supports."

<span class="mw-page-title-main">Logic model</span> Method of depicting causal relationships

Logic models are hypothesized descriptions of the chain of causes and effects leading to an outcome of interest. While they can be in a narrative form, logic model usually take form in a graphical depiction of the "if-then" (causal) relationships between the various elements leading to the outcome. However, the logic model is more than the graphical depiction: it is also the theories, scientific evidences, assumptions and beliefs that support it and the various processes behind it.

Impact evaluation assesses the changes that can be attributed to a particular intervention, such as a project, program or policy, both the intended ones, as well as ideally the unintended ones. In contrast to outcome monitoring, which examines whether targets have been achieved, impact evaluation is structured to answer the question: how would outcomes such as participants' well-being have changed if the intervention had not been undertaken? This involves counterfactual analysis, that is, "a comparison between what actually happened and what would have happened in the absence of the intervention." Impact evaluations seek to answer cause-and-effect questions. In other words, they look for the changes in outcome that are directly attributable to a program.

The Normalization process model is a sociological model, developed by Carl R. May, that describes the adoption of new technologies in health care. The model provides framework for process evaluation using three components – actors, objects, and contexts – that are compared across four constructs: Interactional workability, relational integration, skill-set workability, and contextual integration. This model helped build the Normalization process theory.

Normalization process theory (NPT) is a sociological theory, generally used in the fields of science and technology studies (STS), Implementation Science, and healthcare system research. The theory deals with the adoption of technological and organizational innovations into systems, recent studies have utilized this theory in evaluating new practices in social care and education settings. It was developed out of the normalization process model.

<span class="mw-page-title-main">Communities That Care</span>

Communities That Care (CTC) is a program of the Center for Substance Abuse Prevention (CSAP) in the office of the United States Government's Substance Abuse and Mental Health Services Administration (SAMHSA). CTC is a coalition-based prevention operating system that uses a public health approach to prevent youth problem behaviors such as violence, delinquency, school drop out and substance abuse. Using strategic consultation, training, and research-based tools, CTC is designed to help community stakeholders and decision makers understand and apply information about risk and protective factors, and programs that are proven to make a difference in promoting healthy youth development, in order to most effectively address the specific issues facing their community's youth.

<span class="mw-page-title-main">Theory of Change</span> Methodology for social impact

Theory of Change (ToC) is a methodology or a criterion for planning, participation, adaptive management, and evaluation that is used in companies, philanthropy, not-for-profit, international development, research, and government sectors to promote social change. A Theory of Change of a social program defines its long-term goals and then maps backward to identify necessary preconditions.

The PRECEDE–PROCEED model is a cost–benefit evaluation framework proposed in 1974 by Lawrence W. Green that can help health program planners, policy makers and other evaluators, analyze situations and design health programs efficiently. It provides a comprehensive structure for assessing health and quality of life needs, and for designing, implementing and evaluating health promotion and other public health programs to meet those needs. One purpose and guiding principle of the PRECEDE–PROCEED model is to direct initial attention to outcomes, rather than inputs. It guides planners through a process that starts with desired outcomes and then works backwards in the causal chain to identify a mix of strategies for achieving those objectives. A fundamental assumption of the model is the active participation of its intended audience — that is, that the participants ("consumers") will take an active part in defining their own problems, establishing their goals and developing their solutions.

A behavior change method, or behavior change technique, is a theory-based method for changing one or several determinants of behavior such as a person's attitude or self-efficacy. Such behavior change methods are used in behavior change interventions. Although of course attempts to influence people's attitude and other psychological determinants were much older, especially the definition developed in the late nineties yielded useful insights, in particular four important benefits:

  1. It developed a generic, abstract vocabulary that facilitated discussion of the active ingredients of an intervention
  2. It emphasized the distinction between behavior change methods and practical applications of these methods
  3. It included the concept of 'parameters for effectiveness', important conditions for effectiveness often neglected
  4. It drew attention to the fact that behavior change methods influence specific determinants.

Intervention mapping is a protocol for developing theory-based and evidence-based health promotion programs. Intervention Mapping describes the process of health promotion program planning in six steps:

  1. the needs assessment based on the PRECEDE-PROCEED model
  2. the definition of performance and change objectives based upon scientific analyses of health problems and problem causing factors;
  3. the selection of theory-based intervention methods and practical applications to change health-related behavior;
  4. the production of program components, design and production;
  5. the anticipation of program adoption, implementation and sustainability; and
  6. the anticipation of process and effect evaluation.

Realist evaluation or realist review is a type of theory-driven evaluation method used in evaluating social programmes. It is based on the epistemological foundations of critical realism, though one of the originators of realist evaluation, Ray Pawson, who was "initially impressed" by how critical realism explains generative causation in experimental science, later criticised its "philosophical grandstanding" and "explain-all Marxism". Based on specific theories, realist evaluation provides an alternative lens to empiricist evaluation techniques for the study and understanding of programmes and policies. This technique assumes that knowledge is a social and historical product, thus the social and political context as well as theoretical mechanisms, need consideration in analysis of programme or policy effectiveness.

<span class="mw-page-title-main">Critical realism (philosophy of the social sciences)</span> Philosophical approach to understanding science

Critical realism is a philosophical approach to understanding science, and in particular social science, initially developed by Roy Bhaskar (1944–2014). It specifically opposes forms of empiricism and positivism by viewing science as concerned with identifying causal mechanisms. In the last decades of the twentieth century it also stood against various forms of postmodernism and poststructuralism by insisting on the reality of objective existence. In contrast to positivism's methodological foundation, and poststructuralism's epistemological foundation, critical realism insists that (social) science should be built from an explicit ontology. Critical realism is one of a range of types of philosophical realism, as well as forms of realism advocated within social science such as analytic realism and subtle realism.

Huey-tsyh Chen is a Taiwanese American sociologist and scholar of program evaluation. He is Professor in the Department of Public Health and Director of the Center for Evaluation and Applied Research at Mercer University.

References

  1. ↑ Chen, H.-T., & Rossi, P. H. (1980). The Multi-Goal, Theory-Driven Approach to Evaluation: A Model Linking Basic and Applied Social Science. Social Forces, 59, 106–122.
  2. ↑ Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011, p. 201). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199–226.
  3. ↑ Donaldson, S. I. (2022, p. 9). Introduction to Theory-Driven Program Evaluation (2nd ed.). Routledge.
  4. ↑ Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011, pp. 203–205). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199–226.
  5. ↑ Chen, H. T. (2012). Theory-driven evaluation: Conceptual framework, application and advancement. In R. Strobl, O. Lobermeier, & W. Heitmeyer (Eds.), Evaluation von Programmen und Projekten für eine demokratische Kultur (pp. 17–40). Springer Fachmedien Wiesbaden. https://doi.org/10.1007/978-3-531-19009-9_2
  6. ↑ Weiss, C. H. (1995). Nothing as Practical as Good Theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families. In J. P. Connell, A. C. Kublsch, L. B. Schorr, & C. H. Weiss (Eds.), New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts (Issue 7, pp. 65–92). The Aspen Institute.
  7. ↑ Chen, H.-T., & Rossi, P. H. (1980). The Multi-Goal, Theory-Driven Approach to Evaluation: A Model Linking Basic and Applied Social Science. Social Forces, 59, 106–122.
  8. ↑ Chen, H.-T., & Rossi, P. H. (1983). Evaluating With Sense: The Theory-Driven Approach. Evaluation Review, 7(3), 283–302.
  9. ↑ Chen, H.-T., & Rossi, P. H. (1980). The Multi-Goal, Theory-Driven Approach to Evaluation: A Model Linking Basic and Applied Social Science. Social Forces, 59, 106–122.
  10. ↑ Chen, H. T. (1990). Theory-driven evaluations. Thousand Oaks, CA: Sage.
  11. ↑ Worthen, B. R. (1996). Editor’s Note: The Origins of Theory-Based Evaluation. Evaluation Practice, 17(2), 169–171.
  12. ↑ Weiss, C. H. (1972). Evaluation research: Methods for assessing program effectiveness. Englewood Cliffs, NJ: Prentice-Hall.
  13. ↑ Fitz-Gibbon, C. T., & Morris, L. L. (1975). Theory-based evaluation. Evaluation Comment, 5(1), 1–4. Reprinted in Fitz-Gibbon, C. T., & Morris, L. L. (1996). Theory-based evaluation. Evaluation Practice, 17(2), 177–184.
  14. ↑ See p. 16, Funnell, S. C., Rogers, P. J. (2011). Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Jossey-Bass.
  15. ↑ Fitz-Gibbon, C. T., & Morris, L. L. (1975, p. 1).
  16. ↑ Birckmayer, J. D., & Weiss, C. H. (2000). Theory-based evaluation in practice: what do we learn? Evaluation review, 24(4), 407-431.
  17. ↑ Matta, Corrado; Lindvall, Jannika; Ryve, Andreas (2024). "The Mechanistic Rewards of Data and Theory Integration for Theory-Based Evaluation". American Journal of Evaluation. 45 (1): 110–132. doi:10.1177/10982140221122764.
  18. ↑ Dahler-Larsen, P. (2018, p. 9). Theory-Based Evaluation Meets Ambiguity: The Role of Janus Variables. American Journal of Evaluation, 39(1), 6–23.
  19. ↑ Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B. (2012). Broadening the range of designs and methods for impact evaluations. Institute for Development Studies.
  20. ↑ HM Treasury (2020). The Magenta Book.
  21. ↑ Chen, H.-T., & Rossi, P. H. (1983). Evaluating With Sense: The Theory-Driven Approach. Evaluation Review, 7(3), 283–302.
  22. ↑ Chen, H. T. (1989). The conceptual framework of the theory-driven perspective. Evaluation and Program Planning, 12(4), 391-396.
  23. ↑ Weiss, C. H. (1972, p. 38). Evaluation research: Methods for assessing program effectiveness. Englewood Cliffs, NJ: Prentice-Hall.
  24. ↑ Chen, H. T. (2015, Chapter 3). Practical Program Evaluation: Theory-Driven Evaluation and the Integrated Evaluation Perspective. SAGE Publications Ltd.
  25. ↑ Chen, H. T. (2015). Practical program evaluation: Theory-driven evaluation and the integrated evaluation perspective (2nd edition). Sage Publications.
  26. ↑ Chen, H.-T., & Rossi, P. H. (1983). Evaluating With Sense: The Theory-Driven Approach. Evaluation Review, 7(3), 283–302.
  27. ↑ Smith, N. L. (1994). Clarifying and Expanding the Application of Program Theory-driven Evaluations. Evaluation Practice, 15(1), 83–87.
  28. ↑ Chen, H. T. (1994). Theory-driven Evaluations: Need, Difficulties, and Options. American Journal of Evaluation, 15(1), 79–82. https://doi.org/10.1177/109821409401500109
  29. ↑ Martin P and Tannenbaum C (2017) A realist evaluation of patients’ decisions to deprescribe in the EMPOWER trial. BMJ Open 7(4): e015959.
  30. ↑ Bonell C, Fletcher A, Morton M, et al. (2012) Realist randomised controlled trials: A new approach to evaluating complex public health interventions. Social Science & Medicine 75(12): 2299–306.
  31. ↑ Bonell, C., Melendez-Torres, G. J., & Warren, E. (2024). Realist Trials and Systematic Reviews: Rigorous, Useful Evidence to Inform Health Policy. Cambridge: Cambridge University Press.
  32. ↑ Marchal, B., Westhorp, G., Wong, G., Van Belle, S., Greenhalgh, T., Kegels, G., & Pawson, R. (2013). Realist RCTs of complex interventions—An oxymoron. Social Science & Medicine, 94, 124–128.
  33. ↑ Nielsen, S. B., Jaspers, S. Ø., & Lemire, S. (2023). The curious case of the realist trial: Methodological oxymoron or unicorn? Evaluation.
  34. ↑ Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199–226. https://doi.org/10.1177/1098214010389321
  35. ↑ Bickman, L. (1996). The application of program theory to the evaluation of a managed mental health care system. Evaluation and Program Planning, 19, 111-119.
  36. ↑ Hense, J., Kriz, W. C., & Wolfe, J. (2009). Putting theory-oriented evaluation into practice: A logic model approach for evaluating SIMGAME. Simulation & Gaming, 40, 110-133.
  37. ↑ Chen, H. T., Weng, J. C. S., & Lin, L.-H. (1997). Evaluating the process and outcome of a garbage reduction program in Taiwan. Evaluation Review, 21, 27-42.
  38. ↑ Mercer, S. H., Idler, A. M., & Bartfai, J. M. (2014). Theory-Driven Evaluation in School Psychology Intervention Research: 2007–2012. School Psychology Review, 43(2), 119–131.
  39. ↑ Sheridan, S. M., Bovaird, J. A., Glover, T. A., Andrew Garbacz, S., Witte, A., & Kwon, K. (2012). A Randomized Trial Examining the Effects of Conjoint Behavioral Consultation and the Mediating Role of the Parent–Teacher Relationship. School Psychology Review, 41(1), 23–46.
  40. ↑ Hawkins, R. O., Hale, A., Sheeley, W., & Ling, S. (2011). Repeated reading and vocabulary‐previewing interventions to improve fluency and comprehension for struggling high‐school readers. Psychology in the Schools, 48(1), 59–77.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.