Cognitive reflection test

Last updated

The cognitive reflection test (CRT) is a task designed to measure a person's tendency to override an incorrect "gut" response and engage in further reflection to find a correct answer; however, the validity of the assessment as a measure of "cognitive reflection" or "intuitive thinking" is under question. [1] It was first described in 2005 by psychologist Shane Frederick. The CRT has a moderate positive correlation with measures of intelligence, such as the Intelligence Quotient test, and it correlates highly with various measures of mental heuristics. [2] [3] [4] [5] Some research argue that the CRT is actually measuring cognitive abilities (colloquially known as intelligence). [6]

Contents

Later research showed that the CRT is a multifaceted construct: many start their response with the correct answer, while others fail to solve the test even if they reflect on their intuitive first answer. It has also been argued that suppression of the first answer is not the only factor behind the successful performance on the CRT: numeracy and reflectivity both account for performance. [7]

Basis of test

According to Frederick, there are two general types of cognitive activity called "system 1" and "system 2" (these terms have been first used by Keith Stanovich and Richard West [8] ). System 1 is executed quickly without reflection, while system 2 requires conscious thought and effort. The cognitive reflection test has three questions that each have an obvious but incorrect response given by system 1. The correct response requires the activation of system 2. For system 2 to be activated, a person must note that their first answer is incorrect, which requires reflection on their own cognition. [2]

Correlating measures

The test has been found to correlate with many measures of economic thinking, such as numeracy, [7] temporal discounting, risk preference, and gambling preference. [2] It has also been correlated with measures of mental heuristics, such as the gambler's fallacy, understanding of regression to the mean, the sunk cost fallacy, and others. [3] [4] [5]

Keith Stanovich found that cognitive ability is not strongly correlated with CRT scores because it will only lead to better CRT performance under certain conditions. First, the test-taker must recognize the need to override their system 1 response, and then they must have available cognitive resources to carry out the override. If the test-taker does not need to inhibit system 1 for the override, then the system 2 response immediately follows. Otherwise, they must have the capacity to sustain inhibition of system 1 in order to engage the system 2 response. [9] Contrarily, some researchers have assessed the validity of the assessment, using an advanced item response theory method, and found that the CRT likely measures cognitive ability. [10] The authors of the study explain the validity of the CRT has been questioned due to the lack of validity studies and the lack of a psychometric approach.

Test questions and answers

The original test penned by Dr. Frederick contained only the three following questions: [2]

  1. A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?
  2. If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?
  3. In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?

The intuitive answers to these questions that "system 1" gives typically are: 10 cents, 100 minutes, and 24 days; while the correct solutions are: 5 cents, 5 minutes, and 47 days.

Limitations and alternatives

Studies have estimated that between 44 and 51% of research participants have previously been exposed to the CRT. [11] [12] Those participants that are familiar with the CRT tend to outscore those with no previous exposure, which raises questions about the validity of the measure in this population. [11] [12] In an effort to combat limitations associated with familiarity, researchers have developed a variety of alternative measures of cognitive reflection. [13] [14] [15] Recent research, however, suggests that the CRT is robust to multiple exposure, so that despite the raw score increases in experienced participants, its correlations with other variables remain unaffected. [16]

Another limitation is due to a lack of strong psychometric properties and scarcity of validity studies in the literature. [17] The CRT was not designed in a manner that aligns with standards of the industry such as the Standards for Educational and Psychological Testing which was developed by the American Educational Research Association, American Psychological Association, & National Council on Measurement in Education.

See also

Related Research Articles

Psychometrics is a field of study within psychology concerned with the theory and technique of measurement. Psychometrics generally refers to specialized fields within psychology and education devoted to testing, measurement, assessment, and related activities. Psychometrics is concerned with the objective measurement of latent constructs that cannot be directly observed. Examples of latent constructs include intelligence, introversion, mental disorders, and educational achievement. The levels of individuals on nonobservable latent variables are inferred through mathematical modeling based on what is observed from individuals' responses to items on tests and scales.

<span class="mw-page-title-main">Cognitive bias</span> Systematic pattern of deviation from norm or rationality in judgment

A cognitive bias is a systematic pattern of deviation from norm or rationality in judgment. Individuals create their own "subjective reality" from their perception of the input. An individual's construction of reality, not the objective input, may dictate their behavior in the world. Thus, cognitive biases may sometimes lead to perceptual distortion, inaccurate judgment, illogical interpretation, and irrationality.

Emotional intelligence (EI) is most often defined as the ability to perceive, use, understand, manage, and handle emotions. People with high emotional intelligence can recognize their own emotions and those of others, use emotional information to guide thinking and behavior, discern between different feelings and label them appropriately, and adjust emotions to adapt to environments.

Validity is the main extent to which a concept, conclusion, or measurement is well-founded and likely corresponds accurately to the real world. The word "valid" is derived from the Latin validus, meaning strong. The validity of a measurement tool is the degree to which the tool measures what it claims to measure. Validity is based on the strength of a collection of different types of evidence described in greater detail below.

Critical thinking is the analysis of available facts, evidence, observations, and arguments in order to form a judgement by the application of rational, skeptical, and unbiased analyses and evaluation. The application of critical thinking includes self-directed, self-disciplined, self-monitored, and self-corrective habits of the mind, thus a critical thinker is a person who practices the skills of critical thinking or has been trained and educated in its disciplines. Richard W. Paul said that the mind of a critical thinker engages the person's intellectual abilities and personality traits. Critical thinking presupposes assent to rigorous standards of excellence and mindful command of their use in effective communication and problem solving, and a commitment to overcome egocentrism and sociocentrism.

Experimental psychology refers to work done by those who apply experimental methods to psychological study and the underlying processes. Experimental psychologists employ human participants and animal subjects to study a great many topics, including sensation, perception, memory, cognition, learning, motivation, emotion; developmental processes, social psychology, and the neural substrates of all of these.

In psychology, a projective test is a personality test designed to let a person respond to ambiguous stimuli, presumably revealing hidden emotions and internal conflicts projected by the person into the test. This is sometimes contrasted with a so-called "objective test" / "self-report test", which adopt a "structured" approach as responses are analyzed according to a presumed universal standard, and are limited to the content of the test. The responses to projective tests are content analyzed for meaning rather than being based on presuppositions about meaning, as is the case with objective tests. Projective tests have their origins in psychoanalysis, which argues that humans have conscious and unconscious attitudes and motivations that are beyond or hidden from conscious awareness.

The recognition heuristic, originally termed the recognition principle, has been used as a model in the psychology of judgment and decision making and as a heuristic in artificial intelligence. The goal is to make inferences about a criterion that is not directly accessible to the decision maker, based on recognition retrieved from memory. This is possible if recognition of alternatives has relevance to the criterion. For two alternatives, the heuristic is defined as:

If one of two objects is recognized and the other is not, then infer that the recognized object has the higher value with respect to the criterion.

Personnel selection is the methodical process used to hire individuals. Although the term can apply to all aspects of the process the most common meaning focuses on the selection of workers. In this respect, selected prospects are separated from rejected applicants with the intention of choosing the person who will be the most successful and make the most valuable contributions to the organization. Its effect on the group is discerned when the selected accomplish their desired impact to the group, through achievement or tenure. The procedure of selection takes after strategy to gather data around a person so as to figure out whether that individual ought to be utilized. The strategies used must be in compliance with the various laws in respect to work force selection.

Belief bias is the tendency to judge the strength of arguments based on the plausibility of their conclusion rather than how strongly they support that conclusion. A person is more likely to accept an argument that supports a conclusion that aligns with their values, beliefs and prior knowledge, while rejecting counter arguments to the conclusion. Belief bias is an extremely common and therefore significant form of error; we can easily be blinded by our beliefs and reach the wrong conclusion. Belief bias has been found to influence various reasoning tasks, including conditional reasoning, relation reasoning and transitive reasoning.

<span class="mw-page-title-main">Executive functions</span> Cognitive processes necessary for control of behavior

In cognitive science and neuropsychology, executive functions are a set of cognitive processes that are necessary for the cognitive control of behavior: selecting and successfully monitoring behaviors that facilitate the attainment of chosen goals. Executive functions include basic cognitive processes such as attentional control, cognitive inhibition, inhibitory control, working memory, and cognitive flexibility. Higher-order executive functions require the simultaneous use of multiple basic executive functions and include planning and fluid intelligence.

Cognitive slippage is considered a milder and sub-clinical presentation of formal thought disorder observed via unusual use of language. It is often identified when a person attempts to make tangential connections between concepts that are not immediately understandable to listeners. When observed repeatedly, this is taken as evidence for unusual, maladaptive or illogical thinking patterns.

In psychology, a dual process theory provides an account of how thought can arise in two different ways, or as a result of two different processes. Often, the two processes consist of an implicit (automatic), unconscious process and an explicit (controlled), conscious process. Verbalized explicit processes or attitudes and actions may change with persuasion or education; though implicit process or attitudes usually take a long amount of time to change with the forming of new habits. Dual process theories can be found in social, personality, cognitive, and clinical psychology. It has also been linked with economics via prospect theory and behavioral economics, and increasingly in sociology through cultural analysis.

Acquiescence bias, also known as agreement bias, is a category of response bias common to survey research in which respondents have a tendency to select a positive response option or indicate a positive connotation disproportionately more frequently. Respondents do so without considering the content of the question or their 'true' preference. Acquiescence is sometimes referred to as "yea-saying" and is the tendency of a respondent to agree with a statement when in doubt. Questions affected by acquiescence bias take the following format: a stimulus in the form of a statement is presented, followed by 'agree/disagree,' 'yes/no' or 'true/false' response options. For example, a respondent might be presented with the statement "gardening makes me feel happy," and would then be expected to select either 'agree' or 'disagree.' Such question formats are favoured by both survey designers and respondents because they are straightforward to produce and respond to. The bias is particularly prevalent in the case of surveys or questionnaires that employ truisms as the stimuli, such as: "It is better to give than to receive" or "Never a lender nor a borrower be". Acquiescence bias can introduce systematic errors that affect the validity of research by confounding attitudes and behaviours with the general tendency to agree, which can result in misguided inference. Research suggests that the proportion of respondents who carry out this behaviour is between 10% and 20%.

Dysrationalia is defined as the inability to think and behave rationally despite adequate intelligence. It is a concept in educational psychology and is not a clinical disorder such as a thought disorder. Dysrationalia can be a resource to help explain why smart people fall for Ponzi schemes and other fraudulent encounters.

The Hayling and Brixton tests are neuropsychological tests of executive function created by psychologists Paul W. Burgess and Tim Shallice. It is composed of two tests, the Hayling Sentence Completion Test and the Brixton Spatial Awareness Test.

A situational judgement test (SJT), or situational stress test (SStT) or inventory (SSI) is a type of psychological test which presents the test-taker with realistic, hypothetical scenarios and ask them to identify the most appropriate response or to rank the responses in the order they feel is most effective. SJTs can be presented to test-takers through a variety of modalities, such as booklets, films, or audio recordings. SJTs represent a distinct psychometric approach from the common knowledge-based multiple choice item. They are often used in industrial-organizational psychology applications such as personnel selection. Situational judgement tests tend to determine behavioral tendencies, assessing how an individual will behave in a certain situation, and knowledge instruction, which evaluates the effectiveness of possible responses. Situational judgement tests could also reinforce the status quo with an organization.

In psychology, the human mind is considered to be a cognitive miser due to the tendency of humans to think and solve problems in simpler and less effortful ways rather than in more sophisticated and effortful ways, regardless of intelligence. Just as a miser seeks to avoid spending money, the human mind often seeks to avoid spending cognitive effort. The cognitive miser theory is an umbrella theory of cognition that brings together previous research on heuristics and attributional biases to explain when and why people are cognitive misers.

Shane Frederick is a tenured professor at the Yale School of Management. He earlier worked at Massachusetts Institute of Technology. He is the creator of the cognitive reflection test, which has been found to be "predictive of the types of choices that feature prominently in tests of decision-making theories, like expected utility theory and prospect theory. People who score high on the CRT are less vulnerable to various biases, and show more patience in intertemporal choice tasks.

<span class="mw-page-title-main">Rumination (psychology)</span> Focused attention

Rumination is the focused attention on the symptoms of one's mental distress, and on its possible causes and consequences, as opposed to its solutions, according to the Response Styles Theory proposed by Nolen-Hoeksema (1998).

References

  1. ↑ Blacksmith, Nikki; Yang, Yongwei; Behrend, Tara S.; Ruark, Gregory A. (2019). "Assessing the validity of inferences from scores on the cognitive reflection test". Journal of Behavioral Decision Making. 32 (5): 599–612. doi:10.1002/bdm.2133. ISSN   1099-0771. S2CID   197706996.
  2. 1 2 3 4 Frederick, Shane (2005). "Cognitive Reflection and Decision Making". Journal of Economic Perspectives. 19 (4): 25–42. doi: 10.1257/089533005775196732 .
  3. 1 2 Oechssler, Jörg; Roider, Andreas; Schmitz, Patrick W. (2009). "Cognitive abilities and behavioral biases" (PDF). Journal of Economic Behavior & Organization. 72 (1): 147–152. doi:10.1016/j.jebo.2009.04.018. ISSN   0167-2681.
  4. 1 2 Hoppe, Eva I.; Kusterer, David J. (2011). "Behavioral biases and cognitive reflection". Economics Letters. 110 (2): 97–100. doi:10.1016/j.econlet.2010.11.015. ISSN   0165-1765.
  5. 1 2 Toplak, Maggie (4 May 2011). "The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks" (PDF). Memory and Cognition. 39 (7): 1275–1289. doi: 10.3758/s13421-011-0104-1 . PMID   21541821 . Retrieved 30 May 2014.
  6. ↑ Blacksmith, Nikki; Yang, Yongwei; Behrend, Tara S.; Ruark, Gregory A. (2019). "Assessing the validity of inferences from scores on the cognitive reflection test". Journal of Behavioral Decision Making. 32 (5): 599–612. doi:10.1002/bdm.2133. ISSN   1099-0771. S2CID   197706996.
  7. 1 2 Szaszi, B., Szollosi, A., Palfi, B., Aczél B., (2017) The cognitive reflection test revisited: exploring the ways individuals solve the test, Thinking and Reasoning, https://www.tandfonline.com/doi/abs/10.1080/13546783.2017.1292954
  8. ↑ Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences, 23(5), 645-665. https://doi.org/10.1017/s0140525x00003435
  9. ↑ Stanovich, Keith E., & West, Richard F. (2008). "On the relative independence of thinking biases and cognitive ability." Personality Processes and Individual Differences, 94(4), 672-695. doi:10.1037/0022-3514.94.4.672 http://psycnet.apa.org/journals/psp/94/4/672
  10. ↑ Blacksmith, Nikki; Yang, Yongwei; Behrend, Tara S.; Ruark, Gregory A. (2019). "Assessing the validity of inferences from scores on the cognitive reflection test". Journal of Behavioral Decision Making. 32 (5): 599–612. doi:10.1002/bdm.2133. ISSN   1099-0771. S2CID   197706996.
  11. 1 2 Haigh, Matthew (2016). "Has the Standard Cognitive Reflection Test Become a Victim of Its Own Success?". Advances in Cognitive Psychology. 12 (3): 145–149. doi:10.5709/acp-0193-5. PMC   5225989 . PMID   28115997.
  12. 1 2 Stieger, Stefan; Reips, Ulf-Dietrich (2016-09-06). "A limitation of the Cognitive Reflection Test: familiarity". PeerJ. 4: e2395. doi:10.7717/peerj.2395. ISSN   2167-8359. PMC   5018679 . PMID   27651989.
  13. ↑ Primi, Caterina; Morsanyi, Kinga; Chiesi, Francesca; Donati, Maria Anna; Hamilton, Jayne (2016-12-01). "The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT)". Journal of Behavioral Decision Making. 29 (5): 453–469. doi:10.1002/bdm.1883. ISSN   1099-0771. S2CID   56252490.
  14. ↑ Toplak, Maggie E.; West, Richard F.; Stanovich, Keith E. (2014-04-03). "Assessing miserly information processing: An expansion of the Cognitive Reflection Test". Thinking & Reasoning. 20 (2): 147–168. doi:10.1080/13546783.2013.844729. ISSN   1354-6783. S2CID   53340418.
  15. ↑ Thomson, Keela S.; Oppenheimer, Daniel M. (2016). "Investigating an alternate form of the cognitive reflection test". Judgment and Decision Making. 11: 99–113. doi: 10.1017/S1930297500007622 . S2CID   146924609.
  16. ↑ Bialek, Michal; Pennycook, Gordon (2017-08-28). "The Cognitive Reflection Test is robust to multiple exposures". Behavior Research Methods. 50 (5): 1953–1959. doi: 10.3758/s13428-017-0963-x . PMID   28849403.
  17. ↑ Blacksmith, Nikki; Yang, Yongwei; Behrend, Tara S.; Ruark, Gregory A. (2019). "Assessing the validity of inferences from scores on the cognitive reflection test". Journal of Behavioral Decision Making. 32 (5): 599–612. doi:10.1002/bdm.2133. ISSN   1099-0771. S2CID   197706996.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.