Partial concurrent thinking aloud

Last updated

Partial concurrent thinking aloud (or partial concurrent think-aloud, or PCTA) is a method used to gather data in usability testing with screen reader users. It is a particular kind of think aloud protocol (or TAP) created by Stefano Federici and Simone Borsci [1] at the Interuniversity Center for Research on Cognitive Processing in Natural and Artificial Systems [2] of University of Rome "La Sapienza". The partial concurrent thinking aloud is built up in order to create a specific usability assessment technique for blind users, allowing them to maintain the advantages of concurrent and retrospective thinking aloud while overcoming their limits. Using PCTA blind users' verbalizations of problems could be more pertinent and comparable to those given by sighted people who use a concurrent protocol. In the usability evaluation with blind people, the retrospective thinking aloud is often adopted as a functional solution to overcome the structural interference due to thinking aloud and hearing the screen reader imposed by the classic thinking aloud technique. Such a solution has yet to relapse in the evaluation method, because the concurrent and the retrospective protocols measure usability from different points of view, one mediated by navigation experience (retrospective) and one more direct and pertinent (concurrent). [3] The use of PCTA could be widened to both summative and formative usability evaluations with mixed panels of users, thus extending the number of problems' verbalizations according to disabled users' divergent navigation processes and problem solving strategies.

Contents

Cognitive assumptions

In general, in the usability evaluation both retrospective and concurrent TAP could be used according to the aims and goals of the study. Nevertheless, when a usability evaluation is carried out with blind people several studies propose to use the retrospective TAP: indeed, using a screen reader and talking about the way of interacting with the computer implies a structural interference between action and verbalization. Undoubtedly, cognitive studies provide a lot of evidence supporting the idea that individuals can listen, verbalize, manipulate, and rescue information in multiple task conditions. As Colin Cherry [4] showed, subjects, when listening to two different messages from a single loudspeaker, can separate sounds from background noise, recognize the gender of the speaker, the direction, and the pitch (cocktail party effect). At the same time, subjects that must verbalize the content of a message (attended message) while listening to two different messages simultaneously (attended and unattended message) have a reduced ability to report the content of the attended message, while they are unable to report the content of the unattended message. Moreover, K. Anders Ericsson and Walter Kintsch [5] showed that, in a multiple task condition, subjects' ability of rescuing information is not compromised by an interruption of the action flow (as it happens in the concurrent thinking aloud technique), thanks to the “Long Term Working Memory mechanism” of information retrieval (Working Memory section Ericsson and Kintsch). Even if users can listen, recognize, and verbalize multiple messages in a multiple task condition and they can stop and restart actions without losing any information, other cognitive studies underlined that the overlap of activities in a multiple task condition have an effect on the goal achievement: Kemper, Herman and Lian, [6] analysing the users' abilities to verbalize actions in a multiple task condition, showed that the fluency of a user's conversation is influenced by the overlap of actions. Adults are likely to continue to talk as they navigate in a complex physical environment. However, the fluency of their conversation is likely to change: Older adults are likely to speak more slowly than they would if resting; Young adults continue to speak just as rapidly while walking as while resting, but they adopt a further set of speech accommodations, reducing sentence length, grammatical complexity, and propositional density. Just by reducing length, complexity, and propositional density adults free up working memory resources. We do not know how and how much the content of verbalizations could be influenced by the strategy of verbalization (i.e. the modification of fluency and the complexity in a multiple task condition). Anyway, we well know that users in the concurrent thinking aloud verbalize the problems in a more accurate and pertinent way (i.e. more focused on the problems directly perceived during the interaction) then in the retrospective one. [7] [8] The pertinence is granted to the user by the proximity of action-verbalization-next action; this multiple task proximity compels the subject to apply a strategy of verbalization that reduce the overload of the working memory. However, for blind users this time proximity between action and verbalization is lost: the use of the screen reader, in fact, increase the time for verbalization (i.e. in order to verbalize, blind users must first stop the [screen reader] and then restart it).

Protocol

PCTA method is composed of two sections, one concurrent and one retrospective:

The first section is a modified concurrent protocol built up according to the three concurrent verbal protocols criteria described by K. Anders Ericsson and Herbert A. Simon: [9] [10]

The first criterion
Subjects should be talking about the task at hand, not about an unrelated issue. In order to respect this rule, the time between problem retrieval, thinking and verbalization must be minimized to avoid the influence of a long perceptual reworking and the consequent verbalization of unrelated issues. Blind participants, using a screen reader, increase the time latency between identification and verbalization of a problem. To minimize this latency, users are trained to ring a desk-bell that stops both time and navigation. During this suspension, users can create a memory sign (i.e. ring the bell) and restart immediately the navigation. This setting modification allows to avoid the cognitive limitation problem and the influence of perceptual reworking, also creating a memory sign for the retrospective analysis.
The second criterion
To be pertinent, verbalizations should be logically consistent with the verbalizations that just preceded them. For any kind of user it is hard to be pertinent and consistent in a concurrent verbal protocol. Therefore, the practitioners could generally interrupt the navigation and ask for a clarification or stimulate the users to verbalize in a pertinent way. In order to do so and stop navigation to screen reader users, we propose to negotiate a specific physical sign with them: The practitioner, sitting behind the user, will put his hand on the user's shoulder. This physical sign grants the verbalization pertinence and consistence.
The third criterion
A subset of the information needed during the task performance should be remembered. The concurrent model is based on the link between working memory and time latency. The proximity between the occurrence of a thought and its verbal report allows users to verbalize on the basis of their working memory.

The second PCTA section is a retrospective one in which users analyse those problems previously verbalized in a concurrent way. The memory signs, created by users ringing the desk-bell, overcome the limits of classic retrospective analysis; indeed, these signs allow the users to be pertinent and consistent with their concurrent verbalization, thus avoiding the influence of long term memory and perceptual reworking.

See also

Related Research Articles

Cognitive psychology is the scientific study of mental processes such as attention, language use, memory, perception, problem solving, creativity, and reasoning.

Visual thinking, also called visual or spatial learning or picture thinking, is the phenomenon of thinking through visual processing. Visual thinking has been described as seeing words as a series of pictures. It is common in approximately 60–65% of the general population. "Real picture thinkers", those who use visual thinking almost to the exclusion of other kinds of thinking, make up a smaller percentage of the population. Research by child development theorist Linda Kreger Silverman suggests that less than 30% of the population strongly uses visual/spatial thinking, another 45% uses both visual/spatial thinking and thinking in the form of words, and 25% thinks exclusively in words. According to Kreger Silverman, of the 30% of the general population who use visual/spatial thinking, only a small percentage would use this style over and above all other forms of thinking, and can be said to be true "picture thinkers".

A think-aloudprotocol is a method used to gather data in usability testing in product design and development, in psychology and a range of social sciences.

<span class="mw-page-title-main">Usability</span> Capacity of a system for its users to perform tasks

Usability can be described as the capacity of a system to provide a condition for its users to perform the tasks safely, effectively, and efficiently while enjoying the experience. In software engineering, usability is the degree to which a software can be used by specified consumers to achieve quantified objectives with effectiveness, efficiency, and satisfaction in a quantified context of use.

A heuristic evaluation is a usability inspection method for computer software that helps to identify usability problems in the user interface design. It specifically involves evaluators examining the interface and judging its compliance with recognized usability principles. These evaluation methods are now widely taught and practiced in the new media sector, where user interfaces are often designed in a short space of time on a budget that may restrict the amount of money available to provide for other types of interface testing.

<span class="mw-page-title-main">Screen reader</span> Assistive technology that converts text or images to speech or Braille

A screen reader is a form of assistive technology (AT) that renders text and image content as speech or braille output. Screen readers are essential to people who are blind, and are useful to people who are visually impaired, illiterate, or have a learning disability. Screen readers are software applications that attempt to convey what people with normal eyesight see on a display to their users via non-visual means, like text-to-speech, sound icons, or a braille device. They do this by applying a wide variety of techniques that include, for example, interacting with dedicated accessibility APIs, using various operating system features, and employing hooking techniques.

Cognitive ergonomics is a scientific discipline that studies, evaluates, and designs tasks, jobs, products, environments and systems and how they interact with humans and their cognitive abilities. It is defined by the International Ergonomics Association as "concerned with mental processes, such as perception, memory, reasoning, and motor response, as they affect interactions among humans and other elements of a system. Cognitive ergonomics is responsible for how work is done in the mind, meaning, the quality of work is dependent on the persons understanding of situations. Situations could include the goals, means, and constraints of work. The relevant topics include mental workload, decision-making, skilled performance, human-computer interaction, human reliability, work stress and training as these may relate to human-system design." Cognitive ergonomics studies cognition in work and operational settings, in order to optimize human well-being and system performance. It is a subset of the larger field of human factors and ergonomics.

In psychology, a dual process theory provides an account of how thought can arise in two different ways, or as a result of two different processes. Often, the two processes consist of an implicit (automatic), unconscious process and an explicit (controlled), conscious process. Verbalized explicit processes or attitudes and actions may change with persuasion or education; though implicit process or attitudes usually take a long amount of time to change with the forming of new habits. Dual process theories can be found in social, personality, cognitive, and clinical psychology. It has also been linked with economics via prospect theory and behavioral economics, and increasingly in sociology through cultural analysis.

Protocol analysis is a psychological research method that elicits verbal reports from research participants. Protocol analysis is used to study thinking in cognitive psychology, cognitive science, and behavior analysis. It has found further application in the design of surveys and interviews, usability testing, educational psychology and design research. With the introduction of video- and audio-based based surveys, the scale and scope of verbal report collection is increased dramatically compared to in-person verbal report recording.

The pluralistic walkthrough is a usability inspection method used to identify usability issues in a piece of software or website in an effort to create a maximally usable human-computer interface. The method centers on recruiting a group of users, developers and usability professionals to step through a task scenario, discussing usability issues associated with dialog elements involved in the scenario steps. The group of experts used is asked to assume the role of typical users in the testing.

K. Anders Ericsson was a Swedish psychologist and Conradi Eminent Scholar and Professor of Psychology at Florida State University who was internationally recognized as a researcher in the psychological nature of expertise and human performance.

<span class="mw-page-title-main">Sony Ericsson P1</span> Mobile phone model

The Sony Ericsson P1 is a mobile phone and the successor of the P990. It was the last of the Sony Ericsson "P" Smartphone series, introduced in 2002 with the Sony Ericsson P800 and it integrates many of the hardware features of its predecessor the P990 in the form factor of the M600. It was announced on 8 May 2007. There is a Chinese version of P1 called P1c. Compare with P1/ P1i, P1c lacks of 3G, thereby using EDGE which is much slower but more available especially in the US and parts of Europe.

Usability testing methods aim to evaluate the ease of use of a software product by its users. As existing methods are subjective and open to interpretation, scholars have been studying the efficacy of each method and their adequacy to different subjects, comparing which one may be the most appropriate in fields like e-learning, e-commerce, or mobile applications.

A verbal fluency test is a kind of psychological test in which a participant is asked to produce as many words as possible from a category in a given time. This category can be semantic, including objects such as animals or fruits, or phonemic, including words beginning with a specified letter, such as p, for example. The semantic fluency test is sometimes described as the category fluency test or simply as "freelisting", while letter fluency is also referred to as phonemic test fluency. The Controlled Oral Word Association Test (COWAT) is the most employed phonemic variant. Although the most common performance measure is the total number of words, other analyses such as number of repetitions, number and length of clusters of words from the same semantic or phonemic subcategory, or number of switches to other categories can be carried out.

Retrospective memory is the memory of people, words, and events encountered or experienced in the past. It includes all other types of memory including episodic, semantic and procedural. It can be either implicit or explicit. In contrast, prospective memory involves remembering something or remembering to do something after a delay, such as buying groceries on the way home from work. However, it is very closely linked to retrospective memory, since certain aspects of retrospective memory are required for prospective memory.

Metamemory or Socratic awareness, a type of metacognition, is both the introspective knowledge of one's own memory capabilities and the processes involved in memory self-monitoring. This self-awareness of memory has important implications for how people learn and use memories. When studying, for example, students make judgments of whether they have successfully learned the assigned material and use these decisions, known as "judgments of learning", to allocate study time.

Cognitive skills, also called cognitive functions, cognitive abilities or cognitive capacities, are skills of the mind, as opposed to other types of skills such as motor skills. Some examples of cognitive skills are literacy, self-reflection, logical reasoning, abstract thinking, critical thinking, introspection and mental arithmetic. Cognitive skills vary in processing complexity, and can range from more fundamental processes such as perception and various memory functions, to more sophisticated processes such as decision making, problem solving and metacognition.

<span class="mw-page-title-main">Mental operations</span>

Mental operations are operations that affect mental contents. Initially, operations of reasoning have been the object of logic alone. Pierre Janet was one of the first to use the concept in psychology. Mental operations have been investigated at a developmental level by Jean Piaget, and from a psychometric perspective by J. P. Guilford. There is also a cognitive approach to the subject, as well as a systems view of it.

Verbal overshadowing is a phenomenon where giving a verbal description of sensory input impairs formation of memories of that input. This was first reported by Schooler and Engstler-Schooler (1990) where it was shown that the effects can be observed across multiple domains of cognition which are known to rely on non-verbal knowledge and perceptual expertise. One example of this is memory, which has been known to be influenced by language. Seminal work by Carmichael and collaborators (1932) demonstrated that when verbal labels are connected to non-verbal forms during an individual's encoding process, it could potentially bias the way those forms are reproduced. Because of this, memory performance relying on reportable aspects of memory that encode visual forms should be vulnerable to the effects of verbalization.

Seductive details are often used in textbooks, lectures, slideshows, and other forms of educational content to make a course more interesting or interactive. Seductive details can take the form of text, animations, photos, illustrations, sounds or music and are by definition: (1) interesting and (2) not directed toward the learning objectives of a lesson. John Dewey, in 1913, first referred to this as "fictitious inducements to attention." While illustrated text can enhance comprehension, illustrations that are not relevant can lead to poor learning outcomes. Since the late 1980s, many studies in the field of educational psychology have shown that the addition of seductive details results in poorer retention of information and transfer of learning. Thalheimer conducted a meta-analysis that found, overall, a negative impact for the inclusion of seductive details such as text, photos or illustrations, and sounds or music in learning content. More recently, a 2020 paper found a similar effect for decorative animations This reduction to learning is called the seductive details effect. There have been criticisms of this theory. Critics argue that seductive details do not always impede understanding and that seductive details can sometimes be motivating for learners.

References

  1. ↑ Borsci, S., & Federici, S. (2009). "The Partial Concurrent Thinking Aloud: A New Usability Evaluation Technique for Blind Users". In P. L. Emiliani; L. Burzagli; A. Como; F. Gabbanini; A. L. Salminen (eds.). Assistive technology from adapted equipment to inclusive environments. Vol. 25. IOS Press. pp. 421–425.{{cite book}}: CS1 maint: multiple names: authors list (link)
  2. ↑ "ECoNA - Home Page". Archived from the original on 2010-02-10. Retrieved 2010-02-04.
  3. ↑ Federici, S., Borsci, S., & Stamerra, G. (November 2009). "Web usability evaluation with screen reader users: implementation of the partial concurrent thinking aloud technique". Cognitive Processing. 11 (3): 263–72. doi:10.1007/s10339-009-0347-y. PMID   19916036. S2CID   2155123.{{cite journal}}: CS1 maint: multiple names: authors list (link)
  4. ↑ Cherry, E.C. (1953). "Some experiments on the recognition of speech, with one and with two ears". Journal of the Acoustical Society of America. 25 (5): 975–979. Bibcode:1953ASAJ...25..975C. doi:10.1121/1.1907229. hdl: 11858/00-001M-0000-002A-F750-3 .
  5. ↑ Ericsson, K.A., Kintsch, W. (1995). "Long-Term Working Memory". Psychological Review. 102 (2): 211–245. doi:10.1037/0033-295X.102.2.211. PMID   7740089.{{cite journal}}: CS1 maint: multiple names: authors list (link)
  6. ↑ Kemper, S., Herman, R.E., & Lian, C.H.T. (2003). "The Costs of Doing Two Things at Once for Young and Older Adults: Talking While Walking, Finger Tapping, and Ignoring Speech or Noise". Psychology and Aging. 18 (2): 181–192. doi:10.1037/0882-7974.18.2.181. hdl: 1808/8613 . PMID   12825768.{{cite journal}}: CS1 maint: multiple names: authors list (link)
  7. ↑ Bowers, V.A & Snyder, H.L. (2003). Concurrent versus retrospective verbal protocols for comparing window usability. Human Factors Society 34th Meeting, 8–12 October 1990 HFES, Santa Monica. pp. 1270–1274.
  8. ↑ Van den Haak, M.J. & De Jong, M.D.T. (2003). Exploring Two Methods of Usability Testing: Concurrent versus Retrospective Think-Aloud Protocols. IEEE International Professional Communication Conference Proceedings Piscataway, New Jersey.
  9. ↑ Ericsson, K.A., Simon, H.A. (1980). "Verbal reports as data". Psychological Review. 87 (3): 215–251. doi:10.1037/0033-295X.87.3.215.{{cite journal}}: CS1 maint: multiple names: authors list (link)
  10. ↑ Ericsson, K.A., Simon, H.A. (1993). Protocol analysis: Verbal reports as data (Revised ed.). MIT Press Cambridge.{{cite book}}: CS1 maint: multiple names: authors list (link)
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.