User experience evaluation

Last updated

User experience evaluation (UXE) or user experience assessment (UXA) refers to a collection of methods, skills and tools utilized to uncover how a person perceives a system (product, service, non-commercial item, or a combination of them) before, during and after interacting with it. It is non-trivial to assess user experience since user experience is subjective, context-dependent and dynamic over time. [1] For a UXA study to be successful, the researcher has to select the right dimensions, constructs, and methods and target the research for the specific area of interest such as game, transportation, mobile, etc.

Contents

Dimensions

There are many different dimensions to consider when choosing the best assessment approach:

Laboratory experiments may work well for studying a specific aspect of user experience, but holistic user experience is optimally studied over a longer period of time with real users in a natural environment.

Constructs

In all cases, however, there are certain aspects of user experience that researchers are interested in (measures), and certain procedures and techniques used for collecting the data (methods). There are many measures and some high-level constructs of user experience that can be used as the basis for defining the user experience measures, for example:

  1. Utility: Does the user perceive the functions in the system as useful and fit for the purpose?
  2. Usability: Does the user feel that it is easy and efficient to get things done with the system?
  3. Aesthetics: [2] Does the user see the system as visually attractive? Does it feel pleasurable in hand?
  4. Identification: Can I identify myself with the product? Do I look good when using it?
  5. Stimulation: Does the system give me inspiration? Or wow experiences?
  6. Value: Is the system important to me? What is its value for me?

To properly evaluate user experience, metrics and other factors surrounding a study need to be taken into account, for example:

Methods

An individual method can collect data about a set of specific constructs of user experience. For instance, usability testing is used to collect data about usability construct. [3] Methods also differ if they are to measure a momentary or episodic experience (i.e., assessing how a person feels about a specific interaction episode or after executing a task) or an experience over time, also known as an longitudinal experience. UXA methods can be classified in three categories: implicit, explicit and creative methods.

Implicit methods

Implicit methods of UX research focus not just only on what the users say, but also on what the user cannot express verbally. Many available tools can assist in the implicit evaluation, in particular to gather implicit or objective data. When available, UX researchers utilize state of the art equipment to uncover all aspects of the experience.

Examples of implicit evaluation methods and tools:

Explicit methods

Explicit methods of UX research explore what the user is consciously aware of getting them to reflect on their own feelings or thoughts, and gather their views and opinions. An important aspect of explicit methods includes usability testing and emotion evaluation.

Emotion assessment

When investigating momentary user experiences, we can evaluate the level of positive affect, negative affect, joy, surprise, frustration, etc. The measures for emotions are bound to the methods used for emotion assessment, but typical emotion measures are e.g. valence and arousal. Objective emotion data can be collected by psychophysiological measurements or by observing expressed emotions. Subjective emotional data can be collected by using self-report methods, which can be verbal or non-verbal.

The Geneva Emotion Wheel Geneva Emotion Wheel - English.png
The Geneva Emotion Wheel

Examples of emotion assessment methods:

  • Psychophysiological emotion measurements aim to identify emotions from physiological changes in muscles (e.g. face), pupils, skin, heart, brains, etc.
  • Expression
  • Think aloud protocol can be used for reporting emotions (real-time verbal self-report)
  • Positive and Negative Affect Schedule (PANAS) (retrospective verbal self-report)
  • Geneva emotion wheel [4] (retrospective verbal self-report)
  • Photographic Affect Meter (PAM) [5]
  • Emotion slider [6] (continuous non-verbal self-report)
  • Sensual evaluation instrument (SEI) [7] (snapshot non-verbal self-report)
  • PrEmo, a new version of EmoCards for assessing emotion [8] (snapshot non-verbal self-report)

Creative methods

Equally important to implicit and explicit methods are the creative methods that the user researcher can utilize in order to bring together the design team's view, as well as the target market's dreams, aspirations and ideas of optimal design. These activities are more open and allow people to either co-create with the engineers/designers, or to use their imagination to express their ideal system.

Examples of creative assessment methods

Longitudinal

In contrast to identifying a momentary emotion, longitudinal UXA investigates how a person feels about a system as a whole, after using it for a while.

Examples of longitudinal UXA methods (excluding traditional usability methods):

Areas of UXA research

Transportation

Automobiles have come a long way since their beginning in the late 19th century. One of the major things that have helped automobiles to provide more safety and convenience is electronics. With the advances in technology and electronics, car manufacturers have been able to offer a wide variety of services and conveniences. From the creation of the electronic fuel injection to the popular global positioning system found standard in many cars today, the auto industry has revolutionized the way people travel from place to place. Understanding how people interact with vehicles today, what contributes to a great driving experience, what is their current relationship with the car, what placement does it have in their lives, is key to the development of these technologies. This information ensures user-centered design practices to generate cohesive, predictive and desirable designs.

Once specific design concepts and ideas are on the table, UXA researchers further explore how people react to them regarding desirability, findability, usefulness, credibility, accessibility, usability and human factors metrics. Outcomes of this work includes user requirements, concept validation, and design guidelines. [15] Researchers have conducted intriguing research to answer questions such as: could an In-Vehicle Infotainment (IVI) system with a speech evoked personality change your relationship with your car?, [16] could an in-car system support unwinding after work?, [17] could in-car solutions address the special needs of children as passengers, and assist the parents with the task of driving? [18] and many others. Additionally, workshops and gatherings of researchers around the world take place to discuss current evaluation techniques and advance the field of experience research in the area of transportation. An important professional venue for this work is AutomotiveUI, the International Conference on Automotive User Interfaces and Interactive Vehicular Applications.

UXA methods for transportation

As with other UXA's the method chosen has a lot to do with the outcome desired and where the project is in its design cycle. Given that, methods are selected best suited to the research problem which most times ends up being a combination of implicit, explicit and creative. Some methods include:

  • Interviews: both structured and un-structured.
  • Diary studies [19]
  • Workload assessment questionnaires (i.e. DALI –Driving Activity Load Index adapted from NASA-TLX)
  • Subjective assessment of interfaces questionnaires (i.e. SASSI—Subjective Assessment of Speech System Interfaces [20] ) that can lead to design guidelines to speech interfaces [15]
  • Experience Probing (Prototypes, storytelling, storyboards)
  • Co-design activities
  • Observations (i.e. coding for frustration, delight and other non-verbal cues)

Video games

A relatively new pursuit in video game play-testing is UX and usability research. An increasing number of companies including some of the world's biggest publishers have begun outsourcing UX evaluation or opening their own in-house labs. [21] [22] [23] Researchers use a variety of HCI and psychological techniques to examine the effectiveness of the user experience of the games during the design process. [24]

There are also some companies starting to use biometrics to measure the relationship between in-game events and the player's emotions and feelings (the UX), such as Player Research and Serco ExperienceLab in the UK, [25] [26] and Valve, Electronic Arts, BoltPeters, and VMC Labs in the US and Canada. [27] [28] [29] [30] The interest in this area comes from both academia and industry, sometimes enabling collaborative work. [31] [32] Game UX work has been featured at professional venues, such as the Game Developers Conference (GDC). [33] [34]

Web design

User experience evaluation has become common practice in web design, especially within organizations implementing user-centered design practices. Through user testing, the user experience is constantly evaluated throughout the whole product design life-cycle.

See also

Related Research Articles

Psychophysiology is the branch of psychology that is concerned with the physiological bases of psychological processes. While psychophysiology was a general broad field of research in the 1960s and 1970s, it has now become quite specialized, based on methods, topic of studies and scientific traditions. Methods vary as combinations of electrophysiological methods, neuroimaging, and neurochemistry. Topics have branched into subspecializations such as social, sport, cognitive, cardiovascular, clinical and other branches of psychophysiology.

<span class="mw-page-title-main">Usability</span> Capacity of a system for its users to perform tasks

Usability can be described as the capacity of a system to provide a condition for its users to perform the tasks safely, effectively, and efficiently while enjoying the experience. In software engineering, usability is the degree to which a software can be used by specified consumers to achieve quantified objectives with effectiveness, efficiency, and satisfaction in a quantified context of use.

Gameplay is the specific way in which players interact with a game, and in particular with video games. Gameplay is the pattern defined through the game rules, connection between player and the game, challenges and overcoming them, plot and player's connection with it. Video game gameplay is distinct from graphics and audio elements. In card games, the equivalent term is play.

User experience (UX) is how a user interacts with and experiences a product, system or service. It includes a person's perceptions of utility, ease of use, and efficiency. Improving user experience is important to most companies, designers, and creators when creating and refining products because negative user experience can diminish the use of the product and, therefore, any desired positive impacts. Conversely, designing toward profitability as a main objective often conflicts with ethical user experience objectives and even causes harm. User experience is subjective. However, the attributes that make up the user experience are objective.

Kansei engineering aims at the development or improvement of products and services by translating the customer's psychological feelings and needs into the domain of product design. It was founded by Mitsuo Nagamachi, Professor Emeritus of Hiroshima University. Kansei engineering parametrically links the customer's emotional responses to the properties and characteristics of a product or service. In consequence, products can be designed to bring forward the intended feeling.

<span class="mw-page-title-main">User interface design</span> Planned operator–machine interaction

User interface (UI) design or user interface engineering is the design of user interfaces for machines and software, such as computers, home appliances, mobile devices, and other electronic devices, with the focus on maximizing usability and the user experience. In computer or software design, user interface (UI) design primarily focuses on information architecture. It is the process of building interfaces that clearly communicate to the user what's important. UI design refers to graphical user interfaces and other forms of interface design. The goal of user interface design is to make the user's interaction as simple and efficient as possible, in terms of accomplishing user goals.

Affective science is the scientific study of emotion or affect. This includes the study of emotion elicitation, emotional experience and the recognition of emotions in others. Of particular relevance are the nature of feeling, mood, emotionally-driven behaviour, decision-making, attention and self-regulation, as well as the underlying physiology and neuroscience of the emotions.

A pain scale measures a patient's pain intensity or other features. Pain scales are a common communication tool in medical contexts, and are used in a variety of medical settings. Pain scales are a necessity to assist with better assessment of pain and patient screening. Pain measurements help determine the severity, type, and duration of the pain, and are used to make an accurate diagnosis, determine a treatment plan, and evaluate the effectiveness of treatment. Pain scales are based on trust, cartoons (behavioral), or imaginary data, and are available for neonates, infants, children, adolescents, adults, seniors, and persons whose communication is impaired. Pain assessments are often regarded as "the 5th vital sign".

Situational awareness or situation awareness (SA) is the understanding of an environment, its elements, and how it changes with respect to time or other factors. Situational awareness is important for effective decision making in many environments. It is formally defined as:

“the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future”.

Quality of experience (QoE) is a measure of the delight or annoyance of a customer's experiences with a service. QoE focuses on the entire service experience; it is a holistic concept, similar to the field of user experience, but with its roots in telecommunication. QoE is an emerging multidisciplinary field based on social psychology, cognitive science, economics, and engineering science, focused on understanding overall human quality requirements.

Affective design describes the design of products, services, and user interfaces that aim to evoke intended emotional responses from consumers, ultimately improving customer satisfaction. It is often regarded within the domain of technology interaction and computing, in which emotional information is communicated to the computer from the user in a natural and comfortable way. The computer processes the emotional information and adapts or responds to try to improve the interaction in some way. The notion of affective design emerged from the field of human–computer interaction (HCI), specifically from the developing area of affective computing. Affective design serves an important role in user experience (UX) as it contributes to the improvement of the user's personal condition in relation to the computing system. Decision-making, brand loyalty, and consumer connections have all been associated with the integration of affective design. The goals of affective design focus on providing users with an optimal, proactive experience. Amongst overlap with several fields, applications of affective design include ambient intelligence, human–robot interaction, and video games.

Sentiment analysis is the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information. Sentiment analysis is widely applied to voice of the customer materials such as reviews and survey responses, online and social media, and healthcare materials for applications that range from marketing to customer service to clinical medicine. With the rise of deep language models, such as RoBERTa, also more difficult data domains can be analyzed, e.g., news texts where authors typically express their opinion/sentiment less explicitly.

User experience design defines the experience a user would go through when interacting with a company, its services, and its products. User experience design is a user centered design approach because it considers the user's experience when using a product or platform. Research, data analysis, and test results drive design decisions in UX design rather than aesthetic preferences and opinions. Unlike user interface design, which focuses solely on the design of a computer interface, UX design encompasses all aspects of a user's perceived experience with a product or website, such as its usability, usefulness, desirability, brand perception, and overall performance. UX design is also an element of the customer experience (CX), and encompasses all aspects and stages of a customer's experience and interaction with a company.

The self-perceived quality-of-life scale is a psychological assessment instrument which is based on a comprehensive theory of the self-perceived quality of life (SPQL) and provides a multi-faceted measurement of health-related and non-health-related aspects of well-being. The scale has become an instrument of choice for monitoring quality of life in some clinical populations, for example, it was adopted by the Positively Sound network for women living with HIV.

Peer assessment, or self-assessment, is a process whereby students or their peers grade assignments or tests based on a teacher's benchmarks. The practice is employed to save teachers time and improve students' understanding of course materials as well as improve their metacognitive skills. Rubrics are often used in conjunction with self- and peer-assessment.

The Questionnaire For User Interaction Satisfaction (QUIS) is a tool developed to assess users' subjective satisfaction with specific aspects of the human-computer interface. It was developed in 1987 by a multi-disciplinary team of researchers at the University of Maryland Human–Computer Interaction Lab. The QUIS is currently at Version 7.0 with demographic questionnaire, a measure of overall system satisfaction along 6 scales, and measures of 9 specific interface factors. These 9 factors are: screen factors, terminology and system feedback, learning factors, system capabilities, technical manuals, on-line tutorials, multimedia, teleconferencing, and software installation. Currently available in: German, Italian, Portuguese, and Spanish.

Subjective well-being (SWB) is a self-reported measure of well-being, typically obtained by questionnaire.

Diary studies is a research method that collects qualitative information by having participants record entries about their everyday lives in a log, diary or journal about the activity or experience being studied. This collection of data uses a longitudinal technique, meaning participants are studied over a period of time. This research tool, although not being able to provide results as detailed as a true field study, can still offer a vast amount of contextual information without the costs of a true field study. Diary studies are also known as experience sampling or ecological momentary assessment (EMA) methodology.

<span class="mw-page-title-main">Differential Emotions Scale</span>

The Differential Emotions Scale (DES) is a multidimensional self-report device for assessment of an individual's emotions. The DES helps measure mood based on Carroll Izard's differential emotions theory, The DES consists of thirty items, three for each of the ten fundamental emotions as visualized by Izard: interest, joy, surprise, sadness, anger, disgust, contempt, fear, shame/ shyness, and guilt, which are represented on 5-point Likert scale. There are currently four different versions of the scale. Despite the different versions, the basic idea of are very similar. Participants are asked to rate each of the emotions on a scale, and depending on the instructions given, they either rate their current feelings, feelings over the past week, or over long-term traits. The DES is similar to other scales such as the Multiple Affect Adjective Check List (MAACL) and the Multiple Affect Adjective Check List-Revised (MAACL-R) which are used to assess either the state or trait affect by varying the time of which instructions are given to the participants.

<span class="mw-page-title-main">EmojiGrid</span> Affective self-report tool

The EmojiGrid is an affective self-report tool consisting of a rectangular grid that is labelled with emojis. It is trademark of Kikkoman. The facial expressions of the emoji labels vary from disliking via neutral to liking along the x-axis, and gradually increase in intensity along the y-axis. To report their affective appraisal of a given stimulus, users mark the location inside the grid that best represents their impression. The EmojiGrid can either be used as a paper or computer-based response tool. The images needed to implement the EmojiGrid are freely available from the OSF repository.

References

  1. ↑ Law, E., Roto, V., Hassenzahl, M., Vermeeren, A., Kort, J.: Understanding, Scoping and Defining User Experience: A Survey Approach. In Proceedings of Human Factors in Computing Systems conference, CHI'09. 4–9 April 2009, Boston, MA, USA (2009)
  2. ↑ Moshagen, M. & Thielsch, M. T. (2010). Facets of visual aesthetics. In: International Journal of Human-Computer Studies, 68 (10), 689–709.
  3. ↑ Pelt, Mason (23 May 2016). "Stop overthinking UX and try the coffee shop test". venturebeat.com.
  4. ↑ Baenziger, T., Tran, V. and Scherer, K.R. (2005) ‘'The EmotionWheel. A Tool for the Verbal Report of Emotional Reactions, poster presented at the conference of the International Society of Research on Emotion, Bari, Italy.
  5. ↑ J. P., Adams, P., & Gay, G. (2011). PAM: a photographic affect meter for frequent, in situ measurement of affect. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 725–734). ACM.
  6. ↑ Laurans, G., Desmet, P.M.A., & Hekkert, P.P.M. (2009). The emotion slider: a self-report device for the continuous measurement of emotion. Proceedings of the 2009 International Conference on Affective Computing and Intelligent Interaction. Amsterdam, the Netherlands.
  7. ↑ Isbister, K., Höök, K., Sharp, M., and Laaksolahti, J. 2006. The sensual evaluation instrument: developing an affective evaluation tool. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada, 22–27 April 2006). CHI '06. ACM, New York, NY, 1163–1172
  8. ↑ Desmet, P.M.A., Overbeeke, C.J., Tax, S.J.E.T. (2001). Designing products with added emotional value: development and application of an approach for research through design. The Design Journal, 4(1), 32–47.
  9. ↑ Bolger, N., Davis, A., & Rafaeli, E. (2003). Diary methods: Capturing life as it is lived. Annual Review of Psychology, 54, 579–616.
  10. ↑ Csikszentmihalyi M, Larson R. (1987). Validity and reliability of the Experience-Sampling Method. Journal of Nervous and Mental Disease. Sep 1987;175(9):526–536.
  11. ↑ Kahneman, D., Krueger, A., Schkade, D., Schwarz, N., and Stone, A. (2004). A Survey Method for Characterizing Daily Life Experience: The Day Reconstruction Method. Science. 306:5702, pp. 1776–780.
  12. ↑ Hassenzahl, M., Burmester, M., & Koller, F. (2003). AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In J.Ziegler & G. Szwillus (Eds.), Mensch & Computer 2003. Interaktion in Bewegung (pp. 187–196). Stuttgart, Leipzig: B.G. Teubner.
  13. ↑ Laugwitz, B., Schrepp, M. & Held, T. (2008). Construction and evaluation of a user experience questionnaire. In: Holzinger, A. (Ed.): USAB 2008, LNCS 5298, S. 63-76.
  14. ↑ Toussaint, C., Ulrich, S., Toussaint, M. (2012). HUX - Measuring Holistic User Experience. In German UPA e.V., Usability Professionals 2012 - Tagungsband (pp. 90-94).
  15. 1 2 Areti Goulati and Dalila Szostak. 2011. User experience in speech recognition of navigation devices: an assessment. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '11). ACM, New York, NY, USA, 517-520. DOI=10.1145/2037373.2037451
  16. ↑ Jennifer Healey and Dalila Szostak. 2013. Relating to speech evoked car personalities. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (CHI EA '13). ACM, New York, NY, USA, 1653-1658. DOI=10.1145/2468356.2468652
  17. ↑ Zoë Terken, Roy Haex, Luuk Beursgens, Elvira Arslanova, Maria Vrachni, Jacques Terken, and Dalila Szostak. 2013. Unwinding after work: an in-car mood induction system for semi-autonomous driving. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI '13). ACM, New York, NY, USA, 246-249. DOI=10.1145/2516540.2516571
  18. ↑ Liang Hiah, Tatiana Sidorenkova, Lilia Perez Romero, Yu-Fang Teh, Ferdy van Varik, Jacques Terken, and Dalila Szostak. 2013. Engaging children in cars through a robot companion. In Proceedings of the 12th International Conference on Interaction Design and Children (IDC '13). ACM, New York, NY, USA, 384-387. DOI=10.1145/2485760.2485815
  19. ↑ Lallemand, C. (2012) Dear Diary: Using Diaries to Study User Experience
  20. ↑ Kate S. Hone and Robert Graham. (2000). Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI). Nat. Lang. Eng. 6, 3-4 (September 2000), 287-303. DOI=10.1017/S1351324900002497.
  21. ↑ Halo 3: How Microsoft Labs Invented a New Science of Play. Wired.com. Retrieved on 21 October 2011.
  22. ↑ Bolt, Nate. (22 January 2009) Researching Video Games the UX Way – Boxes and Arrows: The design behind the design. Boxes and Arrows. Retrieved on 21 October 2011.
  23. ↑ THQ Chooses The Guildhall at SMU to House New Usability Lab | games industry | MCV. Mcvuk.com. Retrieved on 21 October 2011.
  24. ↑ Hong, T. (2008) Shoot to Thrill: Bio-Sensory Reactions to 3D Shooting Games, Game Developer Magazine, October
  25. ↑ GamesIndustry.biz. Player Research. Retrieved on 16 March 2013
  26. ↑ Game usability testing. PlayableGames. Retrieved on 21 October 2011.
  27. ↑ Valve. Valvesoftware.com. Retrieved on 21 October 2011.
  28. ↑ EA Games – Electronic Arts Archived 22 May 2012 at the Wayback Machine . Ea.com. Retrieved on 21 October 2011.
  29. ↑ VMC Consulting – Tailored Solutions for Your Business. Vmc.com. Retrieved on 21 October 2011.
  30. ↑ Bolt | Peters | Research, design, and products. Boltpeters.com. Retrieved on 21 October 2011.
  31. ↑ Nacke, L., Ambinder, M., Canossa, A., Mandryk, R., Stach, T. (2009). "Game Metrics and Biometrics: The Future of Player Experience Research" Panel at Future Play 2009
  32. ↑ 8–9 April 2010, Seminar Presentation at Games Research Methods Seminar, "Using physiological measures in conjunction with other UX approaches for better understanding of the player's gameplay experiences", University of Tampere, Finland
  33. ↑ Ambinder, M. (2011) Biofeedback in Gameplay: How Valve Measures Physiology to Enhance Gaming Experience. Game Developers Conference 2011
  34. ↑ Zammitto, V. (2011) The Science of Play Testing: EA's Methods for User Research. Game Developers Conference 2011
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.