Cognitive reflection test

Last updated • 4 min readFrom Wikipedia, The Free Encyclopedia

The cognitive reflection test (CRT) is a task designed to measure a person's tendency to override an incorrect "gut" response and engage in further reflection to find a correct answer. However, the validity of the assessment as a measure of "cognitive reflection" or "intuitive thinking" is under question. [1] It was first described in 2005 by psychologist Shane Frederick. The CRT has a moderate positive correlation with measures of intelligence, such as the IQ test, and it correlates highly with various measures of mental heuristics. [2] [3] [4] [5] Some researchers argue that the CRT is actually measuring cognitive abilities (colloquially known as intelligence). [6]

Contents

Later research has shown that the CRT is a multifaceted construct: many start their response with the correct answer, while others fail to solve the test even if they reflect on their intuitive first answer. It has also been argued that suppression of the first answer is not the only factor behind the successful performance on the CRT; numeracy and reflectivity both account for performance. [7]

Basis of test

According to Frederick, there are two general types of cognitive activity called "system 1" and "system 2" (these terms have been first used by Keith Stanovich and Richard West [8] ). System 1 is executed quickly without reflection, while system 2 requires conscious thought and effort. The cognitive reflection test has three questions that each have an obvious but incorrect response given by system 1. The correct response requires the activation of system 2. For system 2 to be activated, a person must note that their first answer is incorrect, which requires reflection on their own cognition. [2]

Correlating measures

The test has been found to correlate with many measures of economic thinking, such as numeracy, [7] temporal discounting, risk preference, and gambling preference. [2] It has also been correlated with measures of mental heuristics, such as the gambler's fallacy, understanding of regression to the mean, the sunk cost fallacy, and others. [3] [4] [5]

Keith Stanovich found that cognitive ability is not strongly correlated with CRT scores because it will only lead to better CRT performance under certain conditions. First, the test-taker must recognize the need to override their system 1 response, and then they must have available cognitive resources to carry out the override. If the test-taker does not need to inhibit system 1 for the override, then the system 2 response immediately follows. Otherwise, they must have the capacity to sustain inhibition of system 1 in order to engage the system 2 response. [9] Contrarily, some researchers have assessed the validity of the assessment, using an advanced item response theory method, and found that the CRT likely measures cognitive ability. [10] The authors of the study explain the validity of the CRT has been questioned due to the lack of validity studies and the lack of a psychometric approach.

Test questions and answers

The original test penned by Dr. Frederick contained only the three following questions: [2]

  1. A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?
  2. If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?
  3. In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?

The intuitive answers to these questions that "system 1" gives typically are: 10 cents, 100 minutes, and 24 days; while the correct solutions are: 5 cents, 5 minutes, and 47 days.

Limitations and alternatives

Studies have estimated that between 44 and 51% of research participants have previously been exposed to the CRT. [11] [12] Those participants that are familiar with the CRT tend to outscore those with no previous exposure, which raises questions about the validity of the measure in this population. [11] [12] In an effort to combat limitations associated with familiarity, researchers have developed a variety of alternative measures of cognitive reflection. [13] [14] [15] Recent research, however, suggests that the CRT is robust to multiple exposure, so that despite the raw score increases in experienced participants, its correlations with other variables remain unaffected. [16]

Another limitation is due to a lack of strong psychometric properties and scarcity of validity studies in the literature. [17] The CRT was not designed in a manner that aligns with standards of the industry such as the Standards for Educational and Psychological Testing which was developed by the American Educational Research Association, American Psychological Association, & National Council on Measurement in Education.

See also

Related Research Articles

Psychometrics is a field of study within psychology concerned with the theory and technique of measurement. Psychometrics generally covers specialized fields within psychology and education devoted to testing, measurement, assessment, and related activities. Psychometrics is concerned with the objective measurement of latent constructs that cannot be directly observed. Examples of latent constructs include intelligence, introversion, mental disorders, and educational achievement. The levels of individuals on nonobservable latent variables are inferred through mathematical modeling based on what is observed from individuals' responses to items on tests and scales.

<span class="mw-page-title-main">Cognitive bias</span> Systematic pattern of deviation from norm or rationality in judgment

A cognitive bias is a systematic pattern of deviation from norm or rationality in judgment. Individuals create their own "subjective reality" from their perception of the input. An individual's construction of reality, not the objective input, may dictate their behavior in the world. Thus, cognitive biases may sometimes lead to perceptual distortion, inaccurate judgment, illogical interpretation, and irrationality.

Emotional intelligence (EI), also known as emotional quotient (EQ), is the ability to perceive, use, understand, manage, and handle emotions. High emotional intelligence includes emotional recognition of emotions of the self and others, using emotional information to guide thinking and behavior, discerning between and labeling of different feelings, and adjusting emotions to adapt to environments.

Validity is the main extent to which a concept, conclusion, or measurement is well-founded and likely corresponds accurately to the real world. The word "valid" is derived from the Latin validus, meaning strong. The validity of a measurement tool is the degree to which the tool measures what it claims to measure. Validity is based on the strength of a collection of different types of evidence described in greater detail below.

Experimental psychology refers to work done by those who apply experimental methods to psychological study and the underlying processes. Experimental psychologists employ human participants and animal subjects to study a great many topics, including sensation, perception, memory, cognition, learning, motivation, emotion; developmental processes, social psychology, and the neural substrates of all of these.

In psychology, a projective test is a personality test designed to let a person respond to ambiguous stimuli, presumably revealing hidden emotions and internal conflicts projected by the person into the test. This is sometimes contrasted with a so-called "objective test" / "self-report test", which adopt a "structured" approach as responses are analyzed according to a presumed universal standard, and are limited to the content of the test. The responses to projective tests are content analyzed for meaning rather than being based on presuppositions about meaning, as is the case with objective tests. Projective tests have their origins in psychoanalysis, which argues that humans have conscious and unconscious attitudes and motivations that are beyond or hidden from conscious awareness.

The recognition heuristic, originally termed the recognition principle, has been used as a model in the psychology of judgment and decision making and as a heuristic in artificial intelligence. The goal is to make inferences about a criterion that is not directly accessible to the decision maker, based on recognition retrieved from memory. This is possible if recognition of alternatives has relevance to the criterion. For two alternatives, the heuristic is defined as:

If one of two objects is recognized and the other is not, then infer that the recognized object has the higher value with respect to the criterion.

<span class="mw-page-title-main">Metacognition</span> Self-awareness about thinking, higher-order thinking skills

Metacognition is an awareness of one's thought processes and an understanding of the patterns behind them. The term comes from the root word meta, meaning "beyond", or "on top of". Metacognition can take many forms, such as reflecting on one's ways of thinking, and knowing when and how oneself and others use particular strategies for problem-solving. There are generally two components of metacognition: (1) cognitive conceptions and (2) cognitive regulation system. Research has shown that both components of metacognition play key roles in metaconceptual knowledge and learning. Metamemory, defined as knowing about memory and mnemonic strategies, is an important aspect of metacognition.

Personnel selection is the methodical process used to hire individuals. Although the term can apply to all aspects of the process the most common meaning focuses on the selection of workers. In this respect, selected prospects are separated from rejected applicants with the intention of choosing the person who will be the most successful and make the most valuable contributions to the organization. Its effect on the group is discerned when the selected accomplish their desired impact to the group, through achievement or tenure. The procedure of selection takes after strategy to gather data around a person so as to figure out whether that individual ought to be utilized. The strategies used must be in compliance with the various laws in respect to work force selection.

<span class="mw-page-title-main">Keith Stanovich</span> American psychologist (born 1950)

Keith E. Stanovich is an American research scientist and psychologist. He is an Emeritus Professor of Applied Psychology and Human Development at the University of Toronto and former Canada Research Chair of Applied Cognitive Science. His primary research areas are the psychology of reasoning and the psychology of reading. Stanovich has been acknowledged by his peers as one of the most influential cognitive psychologists in the world. His 2009 book What Intelligence Tests Miss won the 2010 Grawemeyer Award in Education. In 2012, Stanovich received the E. L. Thorndike Career Achievement Award from the American Psychological Association (APA). He is a fellow at the APA, the American Psychological Society, and the Committee for Skeptical Inquiry.

Belief bias is the tendency to judge the strength of arguments based on the plausibility of their conclusion rather than how strongly they justify that conclusion. A person is more likely to accept an argument that supports a conclusion that aligns with their values, beliefs and prior knowledge, while rejecting counter arguments to the conclusion. Belief bias is an extremely common and therefore significant form of error; we can easily be blinded by our beliefs and reach the wrong conclusion. Belief bias has been found to influence various reasoning tasks, including conditional reasoning, relation reasoning and transitive reasoning.

<span class="mw-page-title-main">Executive functions</span> Cognitive processes necessary for control of behavior

In cognitive science and neuropsychology, executive functions are a set of cognitive processes that support goal-directed behavior, by regulating thoughts and actions through cognitive control, selecting and successfully monitoring actions that facilitate the attainment of chosen objectives. Executive functions include basic cognitive processes such as attentional control, cognitive inhibition, inhibitory control, working memory, and cognitive flexibility. Higher-order executive functions require the simultaneous use of multiple basic executive functions and include planning and fluid intelligence.

Cognitive slippage is considered a milder and sub-clinical presentation of formal thought disorder observed via unusual use of language. It is often identified when a person attempts to make tangential connections between concepts that are not immediately understandable to listeners. When observed repeatedly, this is taken as evidence for unusual, maladaptive or illogical thinking patterns.

In psychology, a dual process theory provides an account of how thought can arise in two different ways, or as a result of two different processes. Often, the two processes consist of an implicit (automatic), unconscious process and an explicit (controlled), conscious process. Verbalized explicit processes or attitudes and actions may change with persuasion or education; though implicit process or attitudes usually take a long amount of time to change with the forming of new habits. Dual process theories can be found in social, personality, cognitive, and clinical psychology. It has also been linked with economics via prospect theory and behavioral economics, and increasingly in sociology through cultural analysis.

Dysrationalia is defined as the inability to think and behave rationally despite adequate intelligence. It is a concept in educational psychology and is not a clinical disorder such as a thought disorder. Dysrationalia can be a resource to help explain why smart people fall for Ponzi schemes and other fraudulent encounters.

A situational judgement test (SJT), also known as a situational stress test (SStT) or situational stress inventory (SSI), is a type of psychological test that presents the test-taker with realistic, hypothetical scenarios. The test-taker is asked to identify the most appropriate response or to rank the responses in order of effectiveness. SJTs can be administered through various modalities, such as booklets, films, or audio recordings. These tests represent a distinct psychometric approach compared to the traditional knowledge-based multiple-choice items and are frequently utilized in industrial-organizational psychology applications, such as personnel selection.

In psychology, the human mind is considered to be a cognitive miser due to the tendency of humans to think and solve problems in simpler and less effortful ways rather than in more sophisticated and effortful ways, regardless of intelligence. Just as a miser seeks to avoid spending money, the human mind often seeks to avoid spending cognitive effort. The cognitive miser theory is an umbrella theory of cognition that brings together previous research on heuristics and attributional biases to explain when and why people are cognitive misers.

In cognitive psychology, the Eriksen flanker task is a set of response inhibition tests used to assess the ability to suppress responses that are inappropriate in a particular context. The target is flanked by non-target stimuli which correspond either to the same directional response as the target, to the opposite response, or to neither. The task is named for American psychologists Barbara. A. Eriksen & Charles W. Eriksen, who first published the task in 1974, and for the flanker stimuli that surround the target. In the tests, a directional response is assigned to a central target stimulus. Various forms of the task are used to measure information processing and selective attention.

Shane Frederick is a tenured professor at the Yale School of Management. He earlier worked at Massachusetts Institute of Technology. He is the creator of the cognitive reflection test, which has been found to be "predictive of the types of choices that feature prominently in tests of decision-making theories, like expected utility theory and prospect theory. People who score high on the CRT are less vulnerable to various biases, and show more patience in intertemporal choice tasks.

<span class="mw-page-title-main">Rumination (psychology)</span> Focused attention on ones mental distress

Rumination is the focused attention on the symptoms of one's mental distress. In 1998, Nolen-Hoeksema proposed the Response Styles Theory, which is the most widely used conceptualization model of rumination. However, other theories have proposed different definitions for rumination. For example, in the Goal Progress Theory, rumination is conceptualized not as a reaction to a mood state, but as a "response to failure to progress satisfactorily towards a goal". According to multiple studies, rumination is a mechanism that develops and sustains psychopathology conditions such as anxiety, depression, and other negative mental disorders. There are some defined models of rumination, mostly interpreted by the measurement tools. Multiple tools exist to measure ruminative thoughts. Treatments specifically addressing ruminative thought patterns are still in the early stages of development.

References

  1. Blacksmith, Nikki; Yang, Yongwei; Behrend, Tara S.; Ruark, Gregory A. (2019). "Assessing the validity of inferences from scores on the cognitive reflection test". Journal of Behavioral Decision Making. 32 (5): 599–612. doi:10.1002/bdm.2133. ISSN   1099-0771. S2CID   197706996.
  2. 1 2 3 4 Frederick, Shane (2005). "Cognitive Reflection and Decision Making". Journal of Economic Perspectives. 19 (4): 25–42. doi: 10.1257/089533005775196732 .
  3. 1 2 Oechssler, Jörg; Roider, Andreas; Schmitz, Patrick W. (2009). "Cognitive abilities and behavioral biases" (PDF). Journal of Economic Behavior & Organization. 72 (1): 147–152. doi:10.1016/j.jebo.2009.04.018. ISSN   0167-2681.
  4. 1 2 Hoppe, Eva I.; Kusterer, David J. (2011). "Behavioral biases and cognitive reflection". Economics Letters. 110 (2): 97–100. doi:10.1016/j.econlet.2010.11.015. ISSN   0165-1765.
  5. 1 2 Toplak, Maggie (4 May 2011). "The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks" (PDF). Memory and Cognition. 39 (7): 1275–1289. doi: 10.3758/s13421-011-0104-1 . PMID   21541821 . Retrieved 30 May 2014.
  6. Blacksmith, Nikki; Yang, Yongwei; Behrend, Tara S.; Ruark, Gregory A. (2019). "Assessing the validity of inferences from scores on the cognitive reflection test". Journal of Behavioral Decision Making. 32 (5): 599–612. doi:10.1002/bdm.2133. ISSN   1099-0771. S2CID   197706996.
  7. 1 2 Szaszi, B., Szollosi, A., Palfi, B., Aczél B., (2017) The cognitive reflection test revisited: exploring the ways individuals solve the test, Thinking and Reasoning, https://www.tandfonline.com/doi/abs/10.1080/13546783.2017.1292954
  8. Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences, 23(5), 645-665. https://doi.org/10.1017/s0140525x00003435
  9. Stanovich, Keith E., & West, Richard F. (2008). "On the relative independence of thinking biases and cognitive ability." Personality Processes and Individual Differences, 94(4), 672-695. doi:10.1037/0022-3514.94.4.672 http://psycnet.apa.org/journals/psp/94/4/672
  10. Blacksmith, Nikki; Yang, Yongwei; Behrend, Tara S.; Ruark, Gregory A. (2019). "Assessing the validity of inferences from scores on the cognitive reflection test". Journal of Behavioral Decision Making. 32 (5): 599–612. doi:10.1002/bdm.2133. ISSN   1099-0771. S2CID   197706996.
  11. 1 2 Haigh, Matthew (2016). "Has the Standard Cognitive Reflection Test Become a Victim of Its Own Success?". Advances in Cognitive Psychology. 12 (3): 145–149. doi:10.5709/acp-0193-5. PMC   5225989 . PMID   28115997.
  12. 1 2 Stieger, Stefan; Reips, Ulf-Dietrich (2016-09-06). "A limitation of the Cognitive Reflection Test: familiarity". PeerJ. 4: e2395. doi: 10.7717/peerj.2395 . ISSN   2167-8359. PMC   5018679 . PMID   27651989.
  13. Primi, Caterina; Morsanyi, Kinga; Chiesi, Francesca; Donati, Maria Anna; Hamilton, Jayne (2016-12-01). "The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT)". Journal of Behavioral Decision Making. 29 (5): 453–469. doi:10.1002/bdm.1883. hdl: 2158/1011727 . ISSN   1099-0771. S2CID   56252490.
  14. Toplak, Maggie E.; West, Richard F.; Stanovich, Keith E. (2014-04-03). "Assessing miserly information processing: An expansion of the Cognitive Reflection Test". Thinking & Reasoning. 20 (2): 147–168. doi:10.1080/13546783.2013.844729. ISSN   1354-6783. S2CID   53340418.
  15. Thomson, Keela S.; Oppenheimer, Daniel M. (2016). "Investigating an alternate form of the cognitive reflection test". Judgment and Decision Making. 11: 99–113. doi: 10.1017/S1930297500007622 . S2CID   146924609.
  16. Bialek, Michal; Pennycook, Gordon (2017-08-28). "The Cognitive Reflection Test is robust to multiple exposures". Behavior Research Methods. 50 (5): 1953–1959. doi: 10.3758/s13428-017-0963-x . PMID   28849403.
  17. Blacksmith, Nikki; Yang, Yongwei; Behrend, Tara S.; Ruark, Gregory A. (2019). "Assessing the validity of inferences from scores on the cognitive reflection test". Journal of Behavioral Decision Making. 32 (5): 599–612. doi:10.1002/bdm.2133. ISSN   1099-0771. S2CID   197706996.