Goal-free evaluation

Last updated

Goal-free evaluation (Often shortened to GFE) is any evaluation in which the evaluator conducts the evaluation without particular knowledge of or reference to stated or predetermined goals and objectives. [1] [2] This external evaluation model typically consists of an independent evaluator who is intentionally screened from the program's stated goals and objectives in hopes of reducing potential goal-related tunnel vision. [3] [4] According to Scriven, [5] the logic behind avoiding stated goals and objectives has to do with "finding out what the program is actually doing without being cued as to what it is trying to do. If the program is achieving its stated goals and objectives, then these achievements should show up; if not, it is argued, they are irrelevant" . The goal-free evaluator attempts to observe and measure all actual outcomes, effects, or impacts, intended or unintended, all without being cued to the program's intentions. As Popham [6] analogizes, “As you can learn from any baseball pitcher who has set out in the first inning to pitch a shutout, the game’s final score is the thing that counts, not good intentions.”

Contents

Youker and Ingraham [7] [8] summarize GFE's benefits which are based on:

Development

Michael Scriven [9] first introduced the concept of GFE in the 1970s. According to Youker, Ford, and Bayer, following GFE's introduction, there was moderate interest use [10] [11] [12] [13] [14] [15] [16] [17] . [18] However, enthusiasm and interest waned, leaving few practitioners. Then in the 2000s, there was a small resurgence of interest in GFE. In a literature search of GFE from 1972 to 2012, Youker and Ingraham (2013) uncovered roughly a score of evaluators who purport to have conducted a GFE or others identified them as having conducted a GFE.

Description

Goal-free evaluation is not a comprehensive stand-alone evaluation model; rather it is considered either a perspective or position concerning an evaluator's goal orientation throughout an evaluation, or a technique or tool for evaluating without referencing goals. Scriven [19] claims GFE is methodologically neutral, which means that it can be used or adapted for use with several other evaluation approaches, models, and methods as long as the other approaches do not mandate goal orientation.

There are really only two methodological requirements of GFE. The first is that the goal-free evaluator be external from and independent of the program and its upstream stakeholders (program funders, designers, administrators, managers, staff, volunteers, vendors, etc.); [20] the second is that someone be appointed a goal screener. A screener is an impartial party (i.e., someone who is not assigned to GFE design or data collection), such as an administrative assistant, a third party, or even the evaluation client. [21] The screener intervenes between the evaluator and the program people to eliminate goal-oriented communications and documents before they reach the goal-free evaluator. The screener does not require extensive training; the screener should, however, have a basic understanding of GFE's purpose and methodology and be relatively familiar with the organization and its program to be able to identify program goals and objectives.

There are two types of GFE evaluations [22] that are used. The first type being when an evaluator intentionally avoids the stated goal and objectives by taking special precautions. For example, the evaluation may take special precautions to hire goal-screeners who play the role as a liaison between the evaluator and the client. This helps with reducing bias because it allows the outcomes speak for themselves. The second type is goal-dismissive. A goal dismissive evaluator simply disregards the goals and objectives, instead of using special precautions to avoid it.

Full and partial GFE

GFE can be full and partial. During a full GFE, the evaluator is goal-free throughout the duration of the evaluation. [23]

A partial GFE begins goal-free but at some point, during the evaluation, the evaluator learns the stated goals and objectives.

Principles

In a recent attempt to further articulate general principles for guiding the goal-free evaluator, Youker [24] proposed the following principles:

  1. Identify relevant effects to examine without referencing goals and objectives.
  2. Identify what occurred without the prompting of goals and objectives.
  3. Determine if what occurred can logically be attributed to the program or intervention.
  4. Determine the degree to which the effects are positive, negative, or neutral.

GFE in practice

Youker [25] introduced GFE dos and don'ts which are included below.

GFE dos

GFE don’ts

Limitations

GFE has identified limitations. [26] First, GFE is not appropriate in every circumstance. [27] For example, when a client is solely interested in examining goal attainment. Second, it is advocated as a standalone evaluation approach. [28] GFE is used as a supplement to goal-based evaluations strategies. Third, GFE may be efficient in assessing outcomes, as the evaluator casts a broad net in search for outcomes. [29] Fourth GFE disregards the opinions and goals of the certain one stakeholder group, [30] namely the administrators, designers, funders, etc.

Related Research Articles

<span class="mw-page-title-main">Industrial and organizational psychology</span> Branch of psychology

Industrial and organizational psychology, an applied discipline within psychology, is the science of human behavior in the workplace. Depending on the country or region of the world, I-O psychology is also known as occupational psychology in the United Kingdom, organisational psychology in Australia and New Zealand, and work and organizational (WO) psychology throughout Europe and Brazil. Industrial, work, and organizational (IWO) psychology is the broader, more global term for the science and profession.

<span class="mw-page-title-main">Risk management</span> Identification, evaluation, and prioritization of risks

Risk management is the identification, evaluation, and prioritization of risks followed by coordinated and economical application of resources to minimize, monitor, and control the probability or impact of unfortunate events or to maximize the realization of opportunities.

Evaluation is a systematic determination and assessment of a subject's merit, worth and significance, using criteria governed by a set of standards. It can assist an organization, program, design, project or any other intervention or initiative to assess any aim, realisable concept/proposal, or any alternative, to help in decision-making; or to ascertain the degree of achievement or value in regard to the aim and objectives and results of any such action that has been completed. The primary purpose of evaluation, in addition to gaining insight into prior or existing initiatives, is to enable reflection and assist in the identification of future change. Evaluation is often used to characterize and appraise subjects of interest in a wide range of human enterprises, including the arts, criminal justice, foundations, non-profit organizations, government, health care, and other human services. It is long term and done at the end of a period of time.

Instructional design (ID), also known as instructional systems design (ISD), is the practice of systematically designing, developing and delivering instructional materials and experiences, both digital and physical, in a consistent and reliable fashion toward an efficient, effective, appealing, engaging and inspiring acquisition of knowledge. The process consists broadly of determining the state and needs of the learner, defining the end goal of instruction, and creating some "intervention" to assist in the transition. The outcome of this instruction may be directly observable and scientifically measured or completely hidden and assumed. There are many instructional design models but many are based on the ADDIE model with the five phases: analysis, design, development, implementation, and evaluation.

A project is any undertaking, carried out individually or collaboratively and possibly involving research or design, that is carefully planned to achieve a particular aim.

Strategic planning is an organization's process of defining its strategy or direction, and making decisions on allocating its resources to attain strategic goals.

Educational assessment or educational evaluation is the systematic process of documenting and using empirical data on the knowledge, skill, attitudes, aptitude and beliefs to refine programs and improve student learning. Assessment data can be obtained from directly examining student work to assess the achievement of learning outcomes or can be based on data from which one can make inferences about learning. Assessment is often used interchangeably with test, but not limited to tests. Assessment can focus on the individual learner, the learning community, a course, an academic program, the institution, or the educational system as a whole. The word 'assessment' came into use in an educational context after the Second World War.

<span class="mw-page-title-main">Personality test</span> Method of assessing human personality constructs

A personality test is a method of assessing human personality constructs. Most personality assessment instruments are in fact introspective self-report questionnaire measures or reports from life records (L-data) such as rating scales. Attempts to construct actual performance tests of personality have been very limited even though Raymond Cattell with his colleague Frank Warburton compiled a list of over 2000 separate objective tests that could be used in constructing objective personality tests. One exception however, was the Objective-Analytic Test Battery, a performance test designed to quantitatively measure 10 factor-analytically discerned personality trait dimensions. A major problem with both L-data and Q-data methods is that because of item transparency, rating scales and self-report questionnaires are highly susceptible to motivational and response distortion ranging all the way from lack of adequate self-insight to downright dissimulation depending on the reason/motivation for the assessment being undertaken.

Program evaluation is a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency. In both the public and private sectors, stakeholders often want to know whether the programs they are funding, implementing, voting for, receiving or objecting to are producing the intended effect. While program evaluation first focuses around this definition, important considerations often include how much the program costs per participant, how the program could be improved, whether the program is worthwhile, whether there are better alternatives, if there are unintended outcomes, and whether the program goals are appropriate and useful. Evaluators help to answer these questions, but the best way to answer the questions is for the evaluation to be a joint project between evaluators and stakeholders.

Policy analysis is a technique used in public administration to enable civil servants, activists, and others to examine and evaluate the available options to implement the goals of laws and elected officials. The process is also used in the administration of large organizations with complex policies. It has been defined as the process of "determining which of various policies will achieve a given set of goals in light of the relations between the policies and the goals."

PDCA is an iterative design and management method used in business for the control and continual improvement of processes and products. It is also known as the Deming circle/cycle/wheel, the Shewhart cycle, the control circle/cycle, or plan–do–study–act (PDSA). Another version of this PDCA cycle is OPDCA. The added "O" stands for observation or as some versions say: "Observe the current condition." This emphasis on observation and current condition has currency with the literature on lean manufacturing and the Toyota Production System. The PDCA cycle, with Ishikawa's changes, can be traced back to S. Mizuno of the Tokyo Institute of Technology in 1959.

<span class="mw-page-title-main">Narrative therapy</span> Form of psychotherapy

Narrative therapy is a form of psychotherapy that seeks to help patients identify their values and the skills associated with them. It provides the patient with knowledge of their ability to live these values so they can effectively confront current and future problems. The therapist seeks to help the patient co-author a new narrative about themselves by investigating the history of those values. Narrative therapy claims to be a social justice approach to therapeutic conversations, seeking to challenge dominant discourses that it claims shape people's lives in destructive ways. While narrative work is typically located within the field of family therapy, many authors and practitioners report using these ideas and practices in community work, schools and higher education. Narrative therapy has come to be associated with collaborative as well as person-centered therapy.

<span class="mw-page-title-main">Advertising management</span> Part of the advertising industry

Advertising management is a planned managerial process designed to oversee and control the various advertising activities involved in a program to communicate with a firm's target market and which is ultimately designed to influence the consumer's purchase decisions. Advertising is just one element in a company's promotional mix and as such, must be integrated with the overall marketing communications program. Advertising is, however, the most expensive of all the promotional elements and therefore must be managed with care and accountability. Advertising management process also helps in defining the outline of the media campaign and in deciding which type of advertising would be used before the launch of a product.

Michael John Scriven is a British-born Australian polymath and academic philosopher, best known for his contributions to the theory and practice of evaluation.

The rational planning model is a model of the planning process involving a number of rational actions or steps. Taylor (1998) outlines five steps, as follows:

<span class="mw-page-title-main">Logic model</span> Method of depicting causal relationships

Logic models are hypothesized descriptions of the chain of causes and effects leading to an outcome of interest. While they can be in a narrative form, logic model usually take form in a graphical depiction of the "if-then" (causal) relationships between the various elements leading to the outcome. However, the logic model is more than the graphical depiction: it is also the theories, scientific evidences, assumptions and beliefs that support it and the various processes behind it.

Social competence consists of social, emotional, cognitive and behavioral skills needed for successful social adaptation. Social competence also reflects having an ability to take another's perspective concerning a situation, learn from past experiences, and apply that learning to the changes in social interactions.

Empowerment evaluation (EE) is an evaluation approach designed to help communities monitor and evaluate their own performance. It is used in comprehensive community initiatives as well as small-scale settings and is designed to help groups accomplish their goals. According to David Fetterman, "Empowerment evaluation is the use of evaluation concepts, techniques, and findings to foster improvement and self-determination". An expanded definition is: "Empowerment evaluation is an evaluation approach that aims to increase the likelihood that programs will achieve results by increasing the capacity of program stakeholders to plan, implement, and evaluate their own programs."

<span class="mw-page-title-main">Robert F. Mager</span> American psychologist and author (1923–2020)

Robert "Bob" Frank Mager [meɪgɜ:] was an American psychologist and author. Concerned with understanding and improving human performance, he is known for developing a framework for preparing learning objectives, and criterion referenced instruction (CRI), as well as addressing areas of goal orientation, student evaluation, student motivation, classroom environment, educational change, performance technology, and instructional design.

In statistics, in particular in the design of experiments, a multi-valued treatment is a treatment that can take on more than two values. It is related to the dose-response model in the medical literature.

References

  1. ↑ Youker, Brandon. "Goal-Free Evaluation: A Potential Model for the Evaluation of Social Work Programs". Social Work Research. Archived from the original on 2014-08-13.
  2. ↑ Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN   9780803943643.
  3. ↑ Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN   9780803943643.
  4. ↑ Youker, Brandon; Ingraham, Allyssa; Bayer, Nick (2014). "An assessment of goal-free evaluation: case studies of four goal-free evaluations". Evaluation and Program Planning. 46: 10–6. doi:10.1016/j.evalprogplan.2014.05.002. PMID   24863956.
  5. ↑ Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. p. 180. ISBN   9780803943643.
  6. ↑ Popham, James (1974). Evaluation in Education. McCutchan Publishing Corporation. p.  58. ISBN   082111512X.
  7. ↑ Youker, Brandon; Ingraham, Allyssa (2013). "Goal-free evaluation: An orientation for foundations' evaluations". The Foundation Review. 5 (4).
  8. ↑ Youker, Brandon; Ingraham, Allyssa; Bayer, Nick (2014). "An assessment of goal-free evaluation: case studies of four goal-free evaluations". Evaluation and Program Planning. 46: 10–6. doi:10.1016/j.evalprogplan.2014.05.002. PMID   24863956.
  9. ↑ Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN   9780803943643.
  10. ↑ Alkin, M.C. (1972). "Wider context goals and goal-based evaluators". Evaluation Comment: The Journal of Education Evaluation.
  11. ↑ Evers, J.W. (1980). "A field of goal-based and goal-free evaluation techniques". Kalamazoo: Western Michigan University Unpublished Doctoral Dissertation.
  12. ↑ House, E.R. (1980). Evaluating Validity. SAGE Publications, Inc.
  13. ↑ House, E.R. (1991). "Confessions of a responsive, goal-free evaluation". Evaluation Practice. 12: 109–114. doi:10.1177/109821409101200118. S2CID   220434776.
  14. ↑ Salasin, S. (1974). "Exploring goal-free evaluation: An interview with Michael Scriven". Evaluation. 2: 9–16.
  15. ↑ Scriven, M. (1973). "Goal-free evaluation". School Evaluation: 319–328.
  16. ↑ Scriven, M. (1974). "Pros and cons about goal-free evaluations". Evaluation in Education:Current Applications: 34–67.
  17. ↑ Thiagarajan, S. (1975). "Goal-free evaluation of media". Educational Technology: 38–40.
  18. ↑ Welch, W.W. "Goal-free evaluation report for St. Mary's Junior College (Unpublished report)".{{cite journal}}: Cite journal requires |journal= (help)
  19. ↑ Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN   9780803943643.
  20. ↑ Davidson, E.J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. SAGE Publications, Inc.
  21. ↑ Youker, Brandon (2005). "Ethnography and evaluation: Their relationship and three anthropological models of evaluation". Journal of MultiDisciplinary Evaluation. 3: 113–132.
  22. ↑ Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN   9780803943643.
  23. ↑ Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN   9780803943643.
  24. ↑ Youker, Brandon. "Goal-Free Evaluation: A Potential Model for the Evaluation of Social Work Programs". Social Work Research. p. 434. Archived from the original on 2014-08-13.
  25. ↑ Youker, Brandon. "Goal-Free Evaluation: A Potential Model for the Evaluation of Social Work Programs". Social Work Research. Archived from the original on 2014-08-13.
  26. ↑ Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN   9780803943643.
  27. ↑ Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN   9780803943643.
  28. ↑ Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN   9780803943643.
  29. ↑ Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN   9780803943643.
  30. ↑ Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN   9780803943643.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.