Design-focused evaluation

Last updated December 09, 2023

Design-focused evaluation (DFE) is an approach to the evaluation of educational quality.

In DFE, the evaluation specifically focuses on the effectiveness of the constructive alignment in the course (subject, paper, unit) design. DFE was developed for use in the higher education sector. However, it has relevance in training environments also, and may be broadly applicable in any educational environment.^[1]

History

DFE was developed by Calvin Smith, an educational researcher in Australia, to address the fact that most evaluation systems involve a focus on either inputs (the quality of the teaching, the quality of the resources, etc.) or outputs/outcomes (e.g. learning outcomes). Where both inputs and outcomes are included in an approach the items are separate (e.g. The Course Experience Questionnaire^[2] which has scales on inputs and a scale on generic skill development as outcomes). There was at the time no approach that focused on the alignment of the learning activities and the learning objectives.

Uses

DFE is primarily to be used to gather students' perceptions of the efficacy of the alignment (of teaching and learning activities with learning objectives) in the design of a course or unit of study (or part thereof). The reason for doing this is that it is considered a high-quality design practice that learning activities and learning objectives are aligned.^[3] Therefore, 'efficacy' here means 'the degree to which the teaching and learning activities supported the development of the learning outcomes'.

Approach

DFE is a survey-based approach to gathering evaluative data, therefore it generates quantitative data. The approach relies heavily on the way items are written for the surveys used. There is a clear articulation of how items can be written for the DFE approach.^[4]

In short, each question is composed in two parts conjoined by a grammatical structure that does the work of '...helped me to learn...'. The first part of the question indexes the teaching and learning activity (e.g. 'the lab session on dissection...') and the second part indexes the learning objective ('...how to dissect the human torso.').

One sophistication in this process is that on both sides of the item structure there can be varying degrees of specificity-generality (or granularity). Compare the following three items all of which are stable on the right hand side, but vary on the left hand side:

The whole 20-lecture series of lectures helped me to learn the rules of logic
The three guest lectures helped me to learn the rules of logic
The guest lecture by Dr Jeeves helped me to learn the rules of logic.

These items become increasingly granular, or more specific, moving from reference to all the lectures right down to a specific lecture session. The same issue of degrees of granularity/specificity can apply on the learning objective side of the item construction. Consider the following three items:

the textbook helped me to learn the rules of logic
the textbook helped me to learn argument structures and forms
the textbook helped me to learn modus tollens.

Again, the degree of specificity increases with each step.

This ability means the DFE approach is extremely flexible and allows the course designer to target specific aspects of the course design. This allows a more strategic approach to evaluation design, and allows for the selective and high-leverage use of the limited 'real estate' in any survey instrument.

A further observation about the scope of each side of a DFE item is that, as well as being more or less generic or specific, it can make an "implicit" reference to the teaching and learning activities or the learning objectives (respectively). An example of this is when reference is made to 'the course' or 'learning'. Consider the following: 'the course helped me to learn'. In this case the reference to the course is an implicit reference to all of the teaching and learning activities the student experienced. Similarly, the reference to 'learning' in this case implies all of the learning objectives in the course. In one sense implicit references to either the learning objectives or the teaching and learning activities is the most generic level of reference, but a distinction is made in DFE between generic or general references to either TLAs or LOBs and implicit references.

With these three categories a cross-tabulation of the combinations of left-hand- and right-hand-side components of DFE items can be devised. This is useful in helping novice DFE survey designers develop items (Table 1).

Table 1 - Nine combinations of specific-generic-implicit

	S	G	I
S	SS	SG	SI
G	GS	GG	GI
I	IS	IG	II

In the table S=Specific G=Generic I=Implicit

Some examples of follow:

SS: The demonstration of pivot tables (S) helped me to learn how to set up the pivot for my project (S)
II: This course (I) was excellent (I)
GG: The lectures (G) helped me to understand the essential concepts of Marxisim (G)
SI: The demonstration of pivot tables (S) was excellent (I)

Related Research Articles

An instructional theory is "a theory that offers explicit guidance on how to better help people learn and develop." It provides insights about what is likely to happen and why with respect to different kinds of teaching and learning activities while helping indicate approaches for their evaluation. Instructional designers focus on how to best structure material and instructional behavior to facilitate learning.

Outcome-based education or outcomes-based education (OBE) is an educational theory that bases each part of an educational system around goals (outcomes). By the end of the educational experience, each student should have achieved the goal. There is no single specified style of teaching or assessment in OBE; instead, classes, opportunities, and assessments should all help students achieve the specified outcomes. The role of the faculty adapts into instructor, trainer, facilitator, and/or mentor based on the outcomes targeted.

In common usage, evaluation is a systematic determination and assessment of a subject's merit, worth and significance, using criteria governed by a set of standards. It can assist an organization, program, design, project or any other intervention or initiative to assess any aim, realisable concept/proposal, or any alternative, to help in decision-making; or to generate the degree of achievement or value in regard to the aim and objectives and results of any such action that has been completed.

Instructional design (ID), also known as instructional systems design and originally known as instructional systems development (ISD), is the practice of systematically designing, developing and delivering instructional materials and experiences, both digital and physical, in a consistent and reliable fashion toward an efficient, effective, appealing, engaging and inspiring acquisition of knowledge. The process consists broadly of determining the state and needs of the learner, defining the end goal of instruction, and creating some "intervention" to assist in the transition. The outcome of this instruction may be directly observable and scientifically measured or completely hidden and assumed. There are many instructional design models but many are based on the ADDIE model with the five phases: analysis, design, development, implementation, and evaluation.

Problem-based learning (PBL) is a student-centered pedagogy in which students learn about a subject through the experience of solving an open-ended problem found in trigger material. The PBL process does not focus on problem solving with a defined solution, but it allows for the development of other desirable skills and attributes. This includes knowledge acquisition, enhanced group collaboration and communication.

<span class="mw-page-title-main">Active learning</span> Educational technique

Active learning is "a method of learning in which students are actively or experientially involved in the learning process and where there are different levels of active learning, depending on student involvement." Bonwell & Eison (1991) states that "students participate [in active learning] when they are doing something besides passively listening." According to Hanson and Moser (2003) using active teaching techniques in the classroom can create better academic outcomes for students. Scheyvens, Griffin, Jocoy, Liu, & Bradford (2008) further noted that "by utilizing learning strategies that can include small-group work, role-play and simulations, data collection and analysis, active learning is purported to increase student interest and motivation and to build students ‘critical thinking, problem-solving and social skills". In a report from the Association for the Study of Higher Education, authors discuss a variety of methodologies for promoting active learning. They cite literature that indicates students must do more than just listen in order to learn. They must read, write, discuss, and be engaged in solving problems. This process relates to the three learning domains referred to as knowledge, skills and attitudes (KSA). This taxonomy of learning behaviors can be thought of as "the goals of the learning process." In particular, students must engage in such higher-order thinking tasks as analysis, synthesis, and evaluation.

Educational assessment or educational evaluation is the systematic process of documenting and using empirical data on the knowledge, skill, attitudes, aptitude and beliefs to refine programs and improve student learning. Assessment data can be obtained from directly examining student work to assess the achievement of learning outcomes or can be based on data from which one can make inferences about learning. Assessment is often used interchangeably with test, but not limited to tests. Assessment can focus on the individual learner, the learning community, a course, an academic program, the institution, or the educational system as a whole. The word 'assessment' came into use in an educational context after the Second World War.

In education, a curriculum is broadly defined as the totality of student experiences that occur in the educational process. The term often refers specifically to a planned sequence of instruction, or to a view of the student's experiences in terms of the educator's or school's instructional goals. A curriculum may incorporate the planned interaction of pupils with instructional content, materials, resources, and processes for evaluating the attainment of educational objectives. Curricula are split into several categories: the explicit, the implicit, the excluded, and the extracurricular.

Program evaluation is a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency.

A lesson plan is a teacher's detailed description of the course of instruction or "learning trajectory" for a lesson. A daily lesson plan is developed by a teacher to guide class learning. Details will vary depending on the preference of the teacher, subject being covered, and the needs of the students. There may be requirements mandated by the school system regarding the plan. A lesson plan is the teacher's guide for running a particular lesson, and it includes the goal, how the goal will be reached and a way of measuring how well the goal was reached.

Mastery learning is an instructional strategy and educational philosophy, first formally proposed by Benjamin Bloom in 1968. Mastery learning maintains that students must achieve a level of mastery in prerequisite knowledge before moving forward to learn subsequent information. If a student does not achieve mastery on the test, they are given additional support in learning and reviewing the information and then tested again. This cycle continues until the learner accomplishes mastery, and they may then move on to the next stage.

Organizational effectiveness is a concept organizations use to gauge how effective they are at reaching intended outcomes. Organizational effectiveness is both a powerful and problematic term. The strength of it is that it may be used to critically evaluate and improve organisational activities. It's problematic since it means various things to different individuals. And there are other alternative methods for measuring organizational performance. Organizational effectiveness embodies the degree to which firms achieve the goals they have decided upon, a question that draws on several different factors. Among those are talent management, leadership development, organization design and structure, design of measurements and scorecards, implementation of change and transformation, deploying smart processes and smart technology to manage the firm's human capital and the formulation of the broader Human Resources agenda.

This glossary of education-related terms is based on how they commonly are used in Wikipedia articles. This article contains terms starting with G – L. Select a letter from the table of contents to find terms on other articles.

Formative assessment, formative evaluation, formative feedback, or assessment for learning, including diagnostic testing, is a range of formal and informal assessment procedures conducted by teachers during the learning process in order to modify teaching and learning activities to improve student attainment. The goal of a formative assessment is to monitor student learning to provide ongoing feedback that can help students identify their strengths and weaknesses and target areas that need work. It also helps faculty recognize where students are struggling and address problems immediately. It typically involves qualitative feedback for both student and teacher that focuses on the details of content and performance. It is commonly contrasted with summative assessment, which seeks to monitor educational outcomes, often for purposes of external accountability.

Constructive alignment is a principle used for devising teaching and learning activities, and assessment tasks, that directly address the intended learning outcomes (ILOs) in a way not typically achieved in traditional lectures, tutorial classes and examinations. Constructive alignment was devised by Professor John B. Biggs, and represents a marriage between a constructivist understanding of the nature of learning, and an aligned design for outcomes-based teaching education.

Backward design is a method of designing an educational curriculum by setting goals before choosing instructional methods and forms of assessment. Backward design of curriculum typically involves three stages:

Identify the results desired
Determine acceptable levels of evidence that support that the desired results have occurred
Design activities that will make desired results happen

Transformative assessment is a form of assessment that uses “institution-wide assessment strategies that are based on institutional goals and implemented in an integrated way for all levels to systematically transform teaching and learning.” Transformative assessment is focused on the quality of the assessment instruments and how well the assessment measures achieving of a goal. "The classic approach is to say, if you want more of something, measure it"

Language pedagogy is the discipline concerned with the theories and techniques of teaching language. It has been described as a type of teaching wherein the teacher draws from their own prior knowledge and actual experience in teaching language. The approach is distinguished from research-based methodologies.

Theory of Change (ToC) is a methodology or a criterion for planning, participation, adaptive management, and evaluation that is used in companies, philanthropy, not-for-profit, international development, research, and government sectors to promote social change. Theory of Change defines long-term goals and then maps backward to identify necessary preconditions.

The Wisconsin Collaboratory for Enhanced Learning, WisCEL, is a new program at the University of Wisconsin-Madison, USA, initiated by faculty from various departments. WisCEL's goal is use classroom innovation to lead all students to academic success. There are currently two WisCEL Centers located on UW-Madion's campus, at Helen C. White College Library and in Kurt F. Wendt Commons.

References

↑ Patton, Michael Quinn (2011-08-22). Essentials of Utilization-Focused Evaluation. SAGE Publications. ISBN 978-1-4833-0697-1.
↑ Ramsden, P. (1991) A performance indicator of teaching quality in higher education: the Course Experience Questionnaire, Studies in Higher Education 16(2), pp. 129–150.
↑ Biggs, J. B. and Tang, C. (2007). Teaching for quality learning at university. Open University Press/Mc Graw-Hill Education.
↑ Smith, C. D. (2008). Design Focused Evaluation. Assessment & Evaluation in Higher Education, 33(6), 631-645.

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.

[1] Patton, Michael Quinn (2011-08-22). Essentials of Utilization-Focused Evaluation. SAGE Publications. ISBN 978-1-4833-0697-1.

[2] Ramsden, P. (1991) A performance indicator of teaching quality in higher education: the Course Experience Questionnaire, Studies in Higher Education 16(2), pp. 129–150.

[3] Biggs, J. B. and Tang, C. (2007). Teaching for quality learning at university. Open University Press/Mc Graw-Hill Education.

[4] Smith, C. D. (2008). Design Focused Evaluation. Assessment & Evaluation in Higher Education, 33(6), 631-645.

[1]

[2]

[3]

[4]