Concept inventory

Last updated

A concept inventory is a criterion-referenced test designed to help determine whether a student has an accurate working knowledge of a specific set of concepts. Historically, concept inventories have been in the form of multiple-choice tests in order to aid interpretability and facilitate administration in large classes. Unlike a typical, teacher-authored multiple-choice test, questions and response choices on concept inventories are the subject of extensive research. The aims of the research include ascertaining (a) the range of what individuals think a particular question is asking and (b) the most common responses to the questions. Concept inventories are evaluated to ensure test reliability and validity. In its final form, each question includes one correct answer and several distractors.

Contents

Ideally, a score on a criterion-referenced test reflects the amount of content knowledge a student has mastered. Criterion-referenced tests differ from norm-referenced tests in that (in theory) the former is not used to compare an individual's score to the scores of the group. Ordinarily, the purpose of a criterion-referenced test is to ascertain whether a student mastered a predetermined amount of content knowledge; upon obtaining a test score that is at or above a cutoff score, the student can move on to study a body of content knowledge that follows next in a learning sequence. In general, item difficulty values ranging between 30% and 70% are best able to provide information about student understanding.

The distractors are incorrect or irrelevant answers that are usually (but not always) based on students' commonly held misconceptions. [1] Test developers often research student misconceptions by examining students' responses to open-ended essay questions and conducting "think-aloud" interviews with students. The distractors chosen by students help researchers understand student thinking and give instructors insights into students' prior knowledge (and, sometimes, firmly held beliefs). This foundation in research underlies instrument construction and design, and plays a role in helping educators obtain clues about students' ideas, scientific misconceptions, and didaskalogenic ("teacher-induced" or "teaching-induced") confusions and conceptual lacunae that interfere with learning.

Concept inventories in use

Concept inventories are education-related diagnostic tests. [2] In 1985 Halloun and Hestenes introduced a "multiple-choice mechanics diagnostic test" to examine students' concepts about motion. [3] It evaluates student understanding of basic concepts in classical (macroscopic) mechanics. A little later, the Force Concept Inventory (FCI), another concept inventory, was developed. [3] [4] [5] The FCI was designed to assess student understanding of the Newtonian concepts of force. Hestenes (1998) found that while "nearly 80% of the [students completing introductory college physics courses] could state Newton's Third Law at the beginning of the course, FCI data showed that less than 15% of them fully understood it at the end". These results have been replicated in a number of studies involving students at a range of institutions (see sources section below). That said, there remain questions as what exactly the FCI measures. [6] Results using the FCI have led to greater recognition in the science education community of the importance of students' "interactive engagement" with the materials to be mastered. [7]

This is a schematic of the Hake plot (see Redish page ) Hake Plot.JPG
This is a schematic of the Hake plot (see Redish page )

Since the development of the FCI, other physics instruments have been developed. These include the Force and Motion Conceptual Evaluation concept [9] and the Brief Electricity and Magnetism Assessment. [10] For a discussion of how a number of concept inventories were developed see Beichner. [11]

In addition to physics, concept inventories have been developed in statistics, [12] chemistry, [13] [14] astronomy, [15] basic biology, [16] [17] [18] [19] natural selection, [20] [21] [22] genetics, [23] engineering, [24] geoscience. [25] and computer science. [26]

In many areas, foundational scientific concepts transcend disciplinary boundaries. An example of an inventory that assesses knowledge of such concepts is an instrument developed by Odom and Barrow (1995) to evaluate understanding of diffusion and osmosis. [27] In addition, there are non-multiple choice conceptual instruments, such as the essay-based approach [14] and the essay and oral exams concept to measure student understanding of Lewis structures in chemistry. [21] [28]

Caveats associated with concept inventory use

Some concept inventories are problematic. The concepts tested may not be fundamental or important in a particular discipline, the concepts involved may not be explicitly taught in a class or curriculum, or answering a question correctly may require only a superficial understanding of a topic. It is therefore possible to either over-estimate or under-estimate student content mastery. While concept inventories designed to identify trends in student thinking may not be useful in monitoring learning gains as a result of pedagogical interventions, disciplinary mastery may not be the variable measured by a particular instrument. Users should be careful to ensure that concept inventories are actually testing conceptual understanding, rather than test-taking ability, language skills, or other abilities that can influence test performance.

The use of multiple-choice exams as concept inventories is not without controversy. The very structure of multiple-choice type concept inventories raises questions involving the extent to which complex, and often nuanced situations and ideas must be simplified or clarified to produce unambiguous responses. For example, a multiple-choice exam designed to assess knowledge of key concepts in natural selection [20] does not meet a number of standards of quality control. [22] One problem with the exam is that the two members of each of several pairs of parallel items, with each pair designed to measure exactly one key concept in natural selection, sometimes have very different levels of difficulty. [21] Another problem is that the multiple-choice exam overestimates knowledge of natural selection as reflected in student performance on a diagnostic essay exam and a diagnostic oral exam, two instruments with reasonably good construct validity. [21] Although scoring concept inventories in the form of essay or oral exams is labor-intensive, costly, and difficult to implement with large numbers of students, such exams can offer a more realistic appraisal of the actual levels of students' conceptual mastery as well as their misconceptions. [14] [21] Recently, however, computer technology has been developed that can score essay responses on concept inventories in biology and other domains, [29] promising to facilitate the scoring of concept inventories organized as (transcribed) oral exams as well as essays.

See also

Related Research Articles

Science education is the teaching and learning of science to school children, college students, or adults within the general public. The field of science education includes work in science content, science process, some social science, and some teaching pedagogy. The standards for science education provide expectations for the development of understanding for students through the entire course of their K-12 education and beyond. The traditional subjects included in the standards are physical, life, earth, space, and human sciences.

<span class="mw-page-title-main">Active learning</span> Educational technique

Active learning is "a method of learning in which students are actively or experientially involved in the learning process and where there are different levels of active learning, depending on student involvement." Bonwell & Eison (1991) states that "students participate [in active learning] when they are doing something besides passively listening." According to Hanson and Moser (2003) using active teaching techniques in the classroom can create better academic outcomes for students. Scheyvens, Griffin, Jocoy, Liu, & Bradford (2008) further noted that "by utilizing learning strategies that can include small-group work, role-play and simulations, data collection and analysis, active learning is purported to increase student interest and motivation and to build students ‘critical thinking, problem-solving and social skills". In a report from the Association for the Study of Higher Education, authors discuss a variety of methodologies for promoting active learning. They cite literature that indicates students must do more than just listen in order to learn. They must read, write, discuss, and be engaged in solving problems. This process relates to the three learning domains referred to as knowledge, skills and attitudes (KSA). This taxonomy of learning behaviors can be thought of as "the goals of the learning process." In particular, students must engage in such higher-order thinking tasks as analysis, synthesis, and evaluation.

Educational assessment or educational evaluation is the systematic process of documenting and using empirical data on the knowledge, skill, attitudes, aptitude and beliefs to refine programs and improve student learning. Assessment data can be obtained from directly examining student work to assess the achievement of learning outcomes or can be based on data from which one can make inferences about learning. Assessment is often used interchangeably with test, but not limited to tests. Assessment can focus on the individual learner, the learning community, a course, an academic program, the institution, or the educational system as a whole. The word 'assessment' came into use in an educational context after the Second World War.

<span class="mw-page-title-main">David Hestenes</span> American physicist and science educator

David Orlin Hestenes is a theoretical physicist and science educator. He is best known as chief architect of geometric algebra as a unified language for mathematics and physics, and as founder of Modelling Instruction, a research-based program to reform K–12 Science, Technology, Engineering, and Mathematics (STEM) education.

Physics education or physics teaching refers to the education methods currently used to teach physics. The occupation is called physics educator or physics teacher. Physics education research refers to an area of pedagogical research that seeks to improve those methods. Historically, physics has been taught at the high school and college level primarily by the lecture method together with laboratory exercises aimed at verifying concepts taught in the lectures. These concepts are better understood when lectures are accompanied with demonstration, hand-on experiments, and questions that require students to ponder what will happen in an experiment and why. Students who participate in active learning for example with hands-on experiments learn through self-discovery. By trial and error they learn to change their preconceptions about phenomena in physics and discover the underlying concepts. Physics education is part of the broader area of science education.

Formative assessment, formative evaluation, formative feedback, or assessment for learning, including diagnostic testing, is a range of formal and informal assessment procedures conducted by teachers during the learning process in order to modify teaching and learning activities to improve student attainment. The goal of a formative assessment is to monitor student learning to provide ongoing feedback that can help students identify their strengths and weaknesses and target areas that need work. It also helps faculty recognize where students are struggling and address problems immediately. It typically involves qualitative feedback for both student and teacher that focuses on the details of content and performance. It is commonly contrasted with summative assessment, which seeks to monitor educational outcomes, often for purposes of external accountability.

David Edward Pritchard is a professor at the Massachusetts Institute of Technology (MIT), working on atomic physics and educational research.

<span class="mw-page-title-main">Scientific misconceptions</span> False beliefs about science

Scientific misconceptions are commonly held beliefs about science that have no basis in actual scientific fact. Scientific misconceptions can also refer to preconceived notions based on religious and/or cultural influences. Many scientific misconceptions occur because of faulty teaching styles and the sometimes distancing nature of true scientific texts. Because students' prior knowledge and misconceptions are important factors for learning science, science teachers should be able to identify and address these conceptions.

Inquiry-based learning is a form of active learning that starts by posing questions, problems or scenarios. It contrasts with traditional education, which generally relies on the teacher presenting facts and their knowledge about the subject. Inquiry-based learning is often assisted by a facilitator rather than a lecturer. Inquirers will identify and research issues and questions to develop knowledge or solutions. Inquiry-based learning includes problem-based learning, and is generally used in small-scale investigations and projects, as well as research. The inquiry-based instruction is principally very closely related to the development and practice of thinking and problem-solving skills.

Scientific literacy or science literacy encompasses written, numerical, and digital literacy as they pertain to understanding science, its methodology, observations, and theories. Scientific literacy is chiefly concerned with an understanding of the scientific method, units and methods of measurement, empiricism and understanding of statistics in particular correlations and qualitative versus quantitative observations and aggregate statistics, as well as a basic understanding of core scientific fields, such as physics, chemistry, biology, ecology, geology and computation.

Threshold knowledge is a term in the study of higher education used to describe core concepts—or threshold concepts—which, once understood, transform perception of a given subject, phenomenon, or experience.

Continuous assessment is a form of educational examination that evaluates a student's progress throughout a prescribed course. It is often used as an alternative to the final examination system. Proponents of continuous assessment argue that the approach allows tracking of progress and has a chance of offering students more support, guidance, and opportunities to improve during the course or programme.

Peer instruction is an evidence-based, interactive teaching method popularized by Harvard Professor Eric Mazur in the early 1990s. Originally used in many schools, including introductory undergraduate physics classes at Harvard University, peer instruction is used in various disciplines and institutions around the globe. It is a student-centered approach that involves flipping the traditional classroom by moving information transfer out and moving information assimilation, or application of learning, into the classroom. There is some research that supports the effectiveness of peer instruction over more traditional teaching methods, such as traditional lecture.

Just-in-time teaching is a pedagogical strategy that uses feedback between classroom activities and work that students do at home, in preparation for the classroom meeting. The goals are to increase learning during classroom time, to enhance student motivation, to encourage students to prepare for class, and to allow the instructor to fine-tune the classroom activities to best meet students' needs. This should not be confused with just-in-time learning, which itself focuses on immediate connections between learners and the content that is needed at that moment.

<span class="mw-page-title-main">Medical College Admission Test</span> Standardized examination for prospective medical students in the United States and Canada

The Medical College Admission Test is a computer-based standardized examination for prospective medical students in the United States, Australia, Canada, and the Caribbean Islands. It is designed to assess problem solving, critical thinking, written analysis and knowledge of scientific concepts and principles. Before 2007, the exam was a paper-and-pencil test; since 2007, all administrations of the exam have been computer-based.

Confidence weighting (CW) is concerned with measuring two variables: (1) what a respondent believes is a correct answer to a question and (2) what degree of certainty the respondent has toward the correctness of this belief.

The Force Concept Inventory is a test measuring mastery of concepts commonly taught in a first semester of physics developed by Hestenes, Halloun, Wells, and Swackhamer (1985). It was the first such "concept inventory" and several others have been developed since for a variety of topics. The FCI was designed to assess student understanding of the Newtonian concepts of force. Hestenes (1998) found that while "nearly 80% of the [students completing introductory college physics courses] could state Newton's Third Law at the beginning of the course, FCI data showed that less than 15% of them fully understood it at the end". These results have been replicated in a number of studies involving students at a range of institutions, and have led to greater recognition in the physics education research community of the importance of students' "active engagement" with the materials to be mastered.

<span class="mw-page-title-main">PhET Interactive Simulations</span> Company

PhET Interactive Simulations, a project at the University of Colorado Boulder, is a non-profit open educational resource project that creates and hosts explorable explanations. It was founded in 2002 by Nobel Laureate Carl Wieman. PhET began with Wieman's vision to improve the way science is taught and learned. Their stated mission is "To advance science and math literacy and education worldwide through free interactive simulations."

Conceptual questions or conceptual problems in science, technology, engineering, and mathematics (STEM) education are questions that can be answered based only on the knowledge of relevant concepts, rather than performing extensive calculations. They contrast with most homework and exam problems in science and engineering that typically require plugging in numerical values into previously discussed formulas. Such "plug-and-chug" numerical problems can often be solved correctly by just matching the pattern of the problem to a previously discussed problem and changing the numerical inputs, which requires significant amounts of time to perform the calculations but does not test or deepen the understanding of how the concepts and formulas should work together. Conceptual questions, therefore, provide a good complement to conventional numerical problems because they need minimal or no calculations and instead encourage the students to engage more deeply with the underlying concepts and how they relate to formulas.

Laura Mary O'Dwyer is a professor of Measurement, Evaluation, and Statistics at Boston College known for her work on examining the impact of technology in education, especially science education, and for quantifying outcomes for K-12 student success.

References

  1. ↑ "Development and Validation of Instruments to Measure Learning of Expert-Like Thinking." W. K. Adams & C. E. Wieman, 2010. International Journal of Science Education, 1-24. iFirst, doi : 10.1080/09500693.2010.512369
  2. ↑ Treagust, David F. (1988). "Development and use of diagnostic tests to evaluate students' misconceptions in science". International Journal of Science Education. Informa UK Limited. 10 (2): 159–169. Bibcode:1988IJSEd..10..159T. doi:10.1080/0950069880100204. ISSN   0950-0693.
  3. 1 2 Hallouin, I. A., & Hestenes, D. Common sense concepts about motion (1985). American Journal of Physics, 53, 1043-1055
  4. ↑ Hestenes, David; Wells, Malcolm; Swackhamer, Gregg (1992). "Force concept inventory" (PDF). The Physics Teacher. American Association of Physics Teachers (AAPT). 30 (3): 141–158. Bibcode:1992PhTea..30..141H. doi:10.1119/1.2343497. ISSN   0031-921X. S2CID   12311835.
  5. ↑ Hestenes, David (1998). "Who needs physics education research!?". American Journal of Physics. American Association of Physics Teachers (AAPT). 66 (6): 465–467. Bibcode:1998AmJPh..66..465H. doi:10.1119/1.18898. ISSN   0002-9505.
  6. ↑ Huffman, Douglas; Heller, Patricia (1995). "What does the force concept inventory actually measure?" (PDF). The Physics Teacher. American Association of Physics Teachers (AAPT). 33 (3): 138–143. Bibcode:1995PhTea..33..138H. doi:10.1119/1.2344171. ISSN   0031-921X.
  7. ↑ Hake, Richard R. (1998). "Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses". American Journal of Physics. American Association of Physics Teachers (AAPT). 66 (1): 64–74. Bibcode:1998AmJPh..66...64H. doi:10.1119/1.18809. ISSN   0002-9505. S2CID   14835931.
  8. ↑ Redish page. Visited Feb. 14, 2011
  9. ↑ Thornton, Ronald K.; Sokoloff, David R. (1998). "Assessing student learning of Newton's laws: The Force and Motion Conceptual Evaluation and the Evaluation of Active Learning Laboratory and Lecture Curricula". American Journal of Physics. American Association of Physics Teachers (AAPT). 66 (4): 338–352. Bibcode:1998AmJPh..66..338T. doi:10.1119/1.18863. ISSN   0002-9505.
  10. ↑ Ding, L, Chabay, R, Sherwood, B, & Beichner, R (2006). Evaluating an electricity and magnetism assessment tool: Brief electricity and magnetism assessment Brief Electricity and Magnetism Assessment (BEMA). Phys. Rev. ST Physics Ed. Research 2, 7 pages. Ding, Lin; Chabay, Ruth; Sherwood, Bruce; Beichner, Robert (2006). "Evaluating an electricity and magnetism assessment tool: Brief electricity and magnetism assessment". Physical Review Special Topics - Physics Education Research. 2 (1): 010105. Bibcode:2006PRPER...2a0105D. doi: 10.1103/PhysRevSTPER.2.010105 .
  11. ↑ Beichner, Robert J. (1994). "Testing student interpretation of kinematics graphs". American Journal of Physics. American Association of Physics Teachers (AAPT). 62 (8): 750–762. Bibcode:1994AmJPh..62..750B. doi:10.1119/1.17449. ISSN   0002-9505.
  12. ↑ Allen, K (2006) The Statistics Concept Inventory: Development and Analysis of a Cognitive Assessment Instrument in Statistics. Doctoral dissertation, The University of Oklahoma.
  13. ↑ "The Chemical Concepts Inventory. Visited Feb. 14, 2011". Archived from the original on 2007-07-18. Retrieved 2007-07-30.
  14. 1 2 3 Wampold, Bruce E.; Wright, John C.; Williams, Paul H.; Millar, Susan B.; Koscuik, Steve A.; Penberthy, Debra L. (1998). "A Novel Strategy for Assessing the Effects of Curriculum Reform on Student Competence" (PDF). Journal of Chemical Education. American Chemical Society (ACS). 75 (8): 986–992. Bibcode:1998JChEd..75..986W. doi:10.1021/ed075p986. ISSN   0021-9584.
  15. ↑ Astronomy Diagnostic Test (ADT) Version 2.0, visited Feb. 14, 2011
  16. ↑ Garvin-Doxas, Kathy; Klymkowsky, Michael W. (2008). Alberts, Bruce (ed.). "Understanding Randomness and its Impact on Student Learning: Lessons Learned from Building the Biology Concept Inventory (BCI)". CBE: Life Sciences Education. American Society for Cell Biology (ASCB). 7 (2): 227–233. doi:10.1187/cbe.07-08-0063. ISSN   1931-7913. PMC   2424310 . PMID   18519614.
  17. ↑ D'Avanzo, Charlene (2008). "Biology Concept Inventories: Overview, Status, and Next Steps". BioScience. Oxford University Press (OUP). 58 (11): 1079–1085. doi: 10.1641/b581111 . ISSN   1525-3244.
  18. ↑ D'Avanzo C, Anderson CW, Griffith A, Merrill J. 2010. Thinking like a biologist: Using diagnostic questions to help students reason with biological principles. (17 January 2010; www.biodqc.org/)
  19. ↑ Wilson, Christopher D.; Anderson, Charles W.; Heidemann, Merle; Merrill, John E.; Merritt, Brett W.; et al. (2006). "Assessing Students' Ability to Trace Matter in Dynamic Systems in Cell Biology". CBE: Life Sciences Education. American Society for Cell Biology (ASCB). 5 (4): 323–331. doi: 10.1187/cbe.06-02-0142 . ISSN   1931-7913. PMC   1681358 . PMID   17146039.
  20. 1 2 Anderson, Dianne L.; Fisher, Kathleen M.; Norman, Gregory J. (2002-11-14). "Development and evaluation of the conceptual inventory of natural selection". Journal of Research in Science Teaching. Wiley. 39 (10): 952–978. Bibcode:2002JRScT..39..952A. doi: 10.1002/tea.10053 . ISSN   0022-4308.
  21. 1 2 3 4 5 Nehm, Ross H.; Schonfeld, Irvin Sam (2008). "Measuring knowledge of natural selection: A comparison of the CINS, an open-response instrument, and an oral interview" (PDF). Journal of Research in Science Teaching. Wiley. 45 (10): 1131–1160. Bibcode:2008JRScT..45.1131N. doi:10.1002/tea.20251. ISSN   0022-4308. Archived from the original (PDF) on 2011-05-17.
  22. 1 2 Nehm R & Schonfeld IS (2010). The future of natural selection knowledge measurement: A reply to Anderson et al. (2010). Journal of Research in Science Teaching, 47, 358-362. Archived 2011-07-19 at the Wayback Machine
  23. ↑ Smith, Michelle K.; Wood, William B.; Knight, Jennifer K. (2008). Ebert-May, Diane (ed.). "The Genetics Concept Assessment: A New Concept Inventory for Gauging Student Understanding of Genetics". CBE: Life Sciences Education. American Society for Cell Biology (ASCB). 7 (4): 422–430. doi:10.1187/cbe.08-08-0045. ISSN   1931-7913. PMC   2592048 . PMID   19047428.
  24. ↑ Concept Inventory Assessment Instruments for Engineering Science. Visited Feb. 14, 2011.
  25. ↑ Libarkin, J.C., Ward, E.M.G., Anderson, S.W., Kortemeyer, G., Raeburn, S.P., 2011, Revisiting the Geoscience Concept Inventory: A call to the community: GSA Today, v. 21, n. 8, p. 26-28. Archived 2013-07-26 at the Wayback Machine
  26. ↑ Caceffo, R.; Wolfman, S.; Booth, K.; Azevedo, R. (2016). Developing a Computer Science Concept Inventory for Introductory Programming. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (SIGCSE '16). ACM, New York, NY, USA, 364-369. DOI=https://dx.doi.org/10.1145/2839509.2844559
  27. ↑ Odom AL, Barrow LH 1995 Development and application of a two-tier diagnostic test measuring college biology students' understanding of diffusion and osmosis after a course of instruction. Journal of Research In Science Teaching 32: 45-61.
  28. ↑ Cooper, Melanie M.; Underwood, Sonia M.; Hilley, Caleb Z. (2012). "Development and validation of the implicit information from Lewis structures instrument (IILSI): do students connect structures with properties?". Chem. Educ. Res. Pract. Royal Society of Chemistry (RSC). 13 (3): 195–200. doi:10.1039/c2rp00010e. ISSN   1109-4028.
  29. ↑ Nehm, RH; Ha, M; Mayfield, E (2012). "Transforming Biology Assessment with Machine Learning: Automated Scoring of Written Evolutionary Explanations". Journal of Science Education and Technology. 21 (1): 183–196. doi:10.1007/s10956-011-9300-9. S2CID   254747549.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.