Realist Evaluation

Last updated

Realist evaluation or realist review (also realist synthesis) is a type of theory-driven evaluation used in evaluating social programmes. [1] It was originally based on the epistemological foundations of critical realism. Ray Pawson, one of the originators of realist evaluation was "initially impressed" by how critical realism explains generative causation in experimental science; however, he later criticised its "philosophical grandstanding" and "explain-all Marxism". [2] Based on specific theories, realist evaluation provides an alternative lens to empiricist evaluation techniques for the study and understanding of programmes and policies. This technique assumes that knowledge is a social and historical product, thus the social and political context as well as theoretical mechanisms, need consideration in analysis of programme or policy effectiveness. [3] [4] [5]

Realist evaluation techniques recognise that there are many interwoven variables operative at different levels in society, thus this evaluation method suits complex social interventions, rather than traditional cause-effect, non-contextual methods of analysis. This realist technique acknowledges that intervention programmes and policy changes do not necessarily work for everyone, since people are different and are embedded in different contexts. [6]

Realist evaluation was popularised by the work of Ray Pawson and Nick Tilley in 1997. [7] They described the procedure followed in the implementation of realist evaluation techniques in programme evaluation and emphasise that once hypotheses have been generated and data collected, the outcomes of the programme are explored, focusing on the groups that the programme benefitted and those who did not benefit. Effectiveness of a programme is thus not dependent on the outcomes alone (cause–effect), rather there is a consideration of the theoretical mechanisms that are applied, and the socio-historical context in which the programmes were implemented. Thus, the final explanation of a programme considers context-mechanism-outcome. [8]

All research methods are applicable in realist evaluations, according to Pawson and Tilley (1997): [9]

"... it is quite possible to carry out realistic evaluation using: strategies, quantitative and qualitative; timescales, contemporaneous or historical; viewpoints, cross-sectional or longitudinal; samples, large or small; goals, action-oriented or audit-centred; and so on and so forth."

A 2024 book argues that it is possible to run realist randomized controlled trials [10] and Gill Westhorp and Simon Feeny (2024) explain the relevance of surveys and regression models (including interaction terms and covariate adjustment) to testing Context-Mechanism-Outcome configurations. [11]

Related Research Articles

International relations theory is the study of international relations (IR) from a theoretical perspective. It seeks to explain behaviors and outcomes in international politics. The three most prominent schools of thought are realism, liberalism and constructivism. Whereas realism and liberalism make broad and specific predictions about international relations, constructivism and rational choice are methodological approaches that focus on certain types of social explanation for phenomena.

Legal realism is a naturalistic approach to law; it is the view that jurisprudence should emulate the methods of natural science; that is, it should rely on empirical evidence. Hypotheses must be tested against observations of the world.

Program evaluation is a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency.

<span class="mw-page-title-main">Methodology</span> Study of research methods

In its most common sense, methodology is the study of research methods. However, the term can also refer to the methods themselves or to the philosophical discussion of associated background assumptions. A method is a structured procedure for bringing about a certain goal, like acquiring knowledge or verifying knowledge claims. This normally involves various steps, like choosing a sample, collecting data from this sample, and interpreting the data. The study of methods concerns a detailed description and analysis of these processes. It includes evaluative aspects by comparing different methods. This way, it is assessed what advantages and disadvantages they have and for what research goals they may be used. These descriptions and evaluations depend on philosophical background assumptions. Examples are how to conceptualize the studied phenomena and what constitutes evidence for or against them. When understood in the widest sense, methodology also includes the discussion of these more abstract issues.

Ram Roy Bhaskar was an English philosopher of science who is best known as the initiator of the philosophical movement of critical realism (CR). Bhaskar argued that the task of science is "the production of the knowledge of those enduring and continually active mechanisms of nature that produce the phenomena of the world", rather than the discovery of quantitative laws, and that experimental science makes sense only if such mechanisms exist and operate outside the lab as well as inside it.

<span class="mw-page-title-main">Grounded theory</span> Qualitative research methodology

Grounded theory is a systematic methodology that has been largely applied to qualitative research conducted by social scientists. The methodology involves the construction of hypotheses and theories through the collecting and analysis of data. Grounded theory involves the application of inductive reasoning. The methodology contrasts with the hypothetico-deductive model used in traditional scientific research.

<span class="mw-page-title-main">Margaret Archer</span> English sociologist (1943–2023)

Margaret Scotford Archer was a British sociologist, who spent most of her academic career at the University of Warwick where she was for many years Professor of Sociology. She was also a professor at l'Ecole Polytechnique Fédérale de Lausanne, Switzerland. She is best known for coining the term elisionism in her 1995 book Realist Social Theory: The Morphogenetic Approach. On 14 April 2014, Archer was named by Pope Francis to succeed former Harvard law professor and US Ambassador to the Holy See Mary Ann Glendon as President of the Pontifical Academy of Social Sciences, and served in this position until her retirement on 27 March 2019.

<span class="mw-page-title-main">Systematic review</span> Comprehensive review of research literature using systematic methods

A systematic review is a scholarly synthesis of the evidence on a clearly presented topic using critical methods to identify, define and assess research on the topic. A systematic review extracts and interprets data from published studies on the topic, then analyzes, describes, critically appraises and summarizes interpretations into a refined evidence-based conclusion. For example, a systematic review of randomized controlled trials is a way of summarizing and implementing evidence-based medicine. Systematic reviews, sometimes along with meta-analyses, are generally considered the highest level of evidence in medical research.

<span class="mw-page-title-main">Left realism</span>

Left realism emerged in criminology from critical criminology as a reaction against what was perceived to be the left's failure to take a practical interest in everyday crime, allowing right realism to monopolize the political agenda on law and order. Left realism argues that crime disproportionately affects working-class people, but that solutions that only increase repression serve to make the crime problem worse. Instead they argue that the root causes of crime lie in relative deprivation, and that although preventive measures and policing are necessary, they should be placed under democratic control.

Literary realism is a literary genre, part of the broader realism in arts, that attempts to represent subject-matter truthfully, avoiding speculative fiction and supernatural elements. It originated with the realist art movement that began with mid-nineteenth-century French literature (Stendhal) and Russian literature. Literary realism attempts to represent familiar things as they are. Realist authors chose to depict every day and banal activities and experiences.

<span class="mw-page-title-main">Logic model</span> Method of depicting causal relationships

Logic models are hypothesized descriptions of the chain of causes and effects leading to an outcome of interest. While they can be in a narrative form, logic model usually take form in a graphical depiction of the "if-then" (causal) relationships between the various elements leading to the outcome. However, the logic model is more than the graphical depiction: it is also the theories, scientific evidences, assumptions and beliefs that support it and the various processes behind it.

Naked objects is an architectural pattern used in software engineering. It is defined by three principles:

Impact evaluation assesses the changes that can be attributed to a particular intervention, such as a project, program or policy, both the intended ones, as well as ideally the unintended ones. In contrast to outcome monitoring, which examines whether targets have been achieved, impact evaluation is structured to answer the question: how would outcomes such as participants' well-being have changed if the intervention had not been undertaken? This involves counterfactual analysis, that is, "a comparison between what actually happened and what would have happened in the absence of the intervention." Impact evaluations seek to answer cause-and-effect questions. In other words, they look for the changes in outcome that are directly attributable to a program.

Neoclassical realism is a theory of international relations and an approach to foreign policy analysis. Initially coined by Gideon Rose in a 1998 World Politics review article, it is a combination of classical realist and neorealist – particularly defensive realist – theories.

<span class="mw-page-title-main">Theory of Change</span> A theory of how a social policy or program is thought to work

A theory of change (ToC) is an explicit theory of how and why it is thought that a social policy or program activities lead to outcomes and impacts. ToCs are used in the design of programs and program evaluation, across a range of policy areas.

Ray Pawson is Emeritus Professor of Social Research Methodology in the School of Sociology and Social Policy at the University of Leeds.

Empowerment evaluation (EE) is an evaluation approach designed to help communities monitor and evaluate their own performance. It is used in comprehensive community initiatives as well as small-scale settings and is designed to help groups accomplish their goals. According to David Fetterman, "Empowerment evaluation is the use of evaluation concepts, techniques, and findings to foster improvement and self-determination". An expanded definition is: "Empowerment evaluation is an evaluation approach that aims to increase the likelihood that programs will achieve results by increasing the capacity of program stakeholders to plan, implement, and evaluate their own programs."

(R.) Andrew Sayer is Emeritus Professor of Social Theory and Political Economy at Lancaster University, UK. He is known for significant contributions to methodology and theory in the social sciences.

<span class="mw-page-title-main">Critical realism (philosophy of the social sciences)</span> Philosophical approach to understanding science

Critical realism is a philosophical approach to understanding science, and in particular social science, initially developed by Roy Bhaskar (1944–2014). It specifically opposes forms of empiricism and positivism by viewing science as concerned with identifying causal mechanisms. In the last decades of the twentieth century it also stood against various forms of postmodernism and poststructuralism by insisting on the reality of objective existence. In contrast to positivism's methodological foundation, and poststructuralism's epistemological foundation, critical realism insists that (social) science should be built from an explicit ontology. Critical realism is one of a range of types of philosophical realism, as well as forms of realism advocated within social science such as analytic realism and subtle realism.

Theory-driven evaluation is an umbrella term for any approach to program evaluation that develops a theory of change and uses it to design, implement, analyze, and interpret findings from an evaluation. More specifically, an evaluation is theory-driven if it:

  1. formulates a theory of change using some combination of social science, lived experience, and program-related professionals' expertise;
  2. develops and prioritizes evaluation questions using the theory;
  3. uses the theory to guide the design and implementation of the evaluation;
  4. uses the theory to operationalize contextual, process, and outcome variables; and
  5. provides a causal explanation of how and why outcomes were achieved, including whether the program worked and/or had any unintended consequences, and what moderates outcomes.

References

  1. ↑ Marchal, B (2012). "Is realist evaluation keeping its promise?A review of published empirical studies in the field of health systems research". Evaluation. 18 (2): 192–212. doi:10.1177/1356389012442444. S2CID   145299417.
  2. ↑ Pawson, R. (2016). The ersatz realism of critical realism: A reply to Porter. Evaluation, 22, 49–57.
  3. ↑ Secord, P; Secord M (1983). "Implications for psychology of the new philosophy of science". American Psychologist. 38 (4): 399–413. doi:10.1037/0003-066x.38.4.399.
  4. ↑ Dalkin, SM; Jones, D; Lhussier, M; Cunningham, B (2012). "Understanding integrated care pathways in palliative care using realist evaluation:a mixed methods study protocol". BMJ Open . 3 (4): 1–6. doi:10.1136/bmjopen-2012-001533. PMC   3391371 . PMID   22761292. Open Access logo PLoS transparent.svg
  5. ↑ Dada, Sara; Dalkin, Sonia; Gilmore, Brynne; Hunter, Rebecca; Mukumbang, Ferdinand C. (2023-03-14). "Applying and reporting relevance, richness and rigour in realist evidence appraisals: Advancing key concepts in realist reviews". Research Synthesis Methods: jrsm.1630. doi: 10.1002/jrsm.1630 . ISSN   1759-2879. PMID   36872619. S2CID   257363167.
  6. ↑ Hewitt, G; S. Sims; R. Harris (2012). "The realist approach to evaluation research:an introduction". International Journal of Therapy and Rehabilitation. 19 (5): 205–259. doi:10.12968/ijtr.2012.19.5.250.
  7. ↑ Pawson, Ray (1997). Realistic evaluation. Nick Tilley. London: Sage. ISBN   0-7619-5008-7. OCLC   37255786.
  8. ↑ R Pawson, N. Tilley, "Realistic Evaluation" (Sage), London.
  9. ↑ Pawson, R., & Tilley, N. (1997, p. 85). Realistic Evaluation. SAGE Publications Ltd.
  10. ↑ Bonell, C., Melendez-Torres, G. J., Warren, E. (2024). Realist trials and systematic reviews: rigorous, useful evidence to inform health policy. Cambridge University Press. ISBN   978-1-00-945660-9.
  11. ↑ Westhorp, Gill; Feeny, Simon (2024-10-14). "Using surveys in realist evaluation". Evaluation Journal of Australasia. doi: 10.1177/1035719X241292083 . ISSN   1035-719X.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.