Nadine Social Robot

Last updated
Nadine
Nadine Robot.jpg
Year of creation2013

Nadine is a gynoid humanoid social robot that is modelled on Professor Nadia Magnenat Thalmann. [1] The robot has a strong human-likeness with a natural-looking skin and hair and realistic hands. Nadine is a socially intelligent robot which returns a greeting, makes eye contact, and can remember all the conversations had with it. It is able to answer questions autonomously in several languages, simulate emotions both in gestures and facially, depending on the content of the interaction with the user. [2] [3] [4] Nadine can recognise persons it has previously seen, and engage in flowing conversation. [5] [6] [7] Nadine has been programmed with a "personality", in that its demeanour can change according to what is said to it. [8] Nadine has a total of 27 degrees of freedom for facial expressions and upper body movements. With persons it has previously encountered, it remembers facts and events related to each person. [9] [10] It can assist people with special needs by reading stories, showing images, put on Skype sessions, send emails, and communicate with other members of the family. [11] [12] [13] [14] It can play the role of a receptionist in an office or be dedicated to be a personal coach. [15] [16]

Contents

Nadine interacted with more than 100,000 visitors at the ArtScience Museum in Singapore during the exhibition, "HUMAN+: The Future of our Species", that was held from May to October 2017. [17] [18] [19] Nadine has worked as a customer service agent in AIA Insurance Company in Singapore. [20] [21] [22] This is the first time in the world that a humanoid robot is used as a customer service agent.

History

Nadine is a next-generation humanoid robot that is a successor from Eva, [23] a humanoid robot head manufactured by Hanson Robotics in 2008. Eva's software platform was developed at MIRALab, [24] University of Geneva. Eva's head shows very realistic moods and emotions [25] and short term memory. [26] [27] Eva has also performed in a play in the Roten Fabrik Theatre at Zurich. [28]

Nadine has been created in 2013 by Kokoro, Japan and has been modelled after Professor Nadia Magnenat Thalmann. Nadine has a head and full body with a natural appearance. Nadine software platform which has been developed at the Institute for Media Innovation in Singapore's Nanyang Technological University is able to show emotions, speak naturally, understand some gestures, and remember and retrieve facts during dialogue sessions. [29] [30] Nadine also interacts with arm movements. Ongoing research provides the social robot with two articulated hands and natural grasping. [31] [32] [33] [34] Nadine is also linked to all kinds of databases such as its personal dataset, Wikipedia, weather channels, and many others.

Platform

Nadine (social robot) is built with a classic perception – processing/decision – interaction layer framework. The design of Nadine platform with objectives of maintaining human-like natural conduct even in complex situation, be generic to handle any kind of data and place of operation, multi-lingual support etc.

Nadine's functionalities are based on her understanding of environment and perception of users/people in front of her. Nadine's perception layer is focused on this task. Nadine uses a 3D depth cameras, webcam and microphone to pick up vision and audio inputs from her environment and users. Perception layer is composed of independent sub-modules that operate on different input streams of the above-mentioned devices to recognize faces, [35] emotions, [13] gestures, [36] [37] user location, intention, comportment etc. and other environmental attributes such as object recognition, [38] [39] location etc.

The processing layer functions as Nadine's brain that uses the perception outputs to gauge the situation and decide on how to act according to it. The main component of this layer is a behavior tree planner, Nadine's central processing unit allows to process all perceived inputs. Based on the inputs received from perception layer, the behavior tree planner updates the other sub-modules of processing layer, which include processing dialog between user and Nadine, affective system and memories of her interaction. To process dialog, generic chatbots [40] [41] have been built to handle different situations and questions. An online search based on Google Assistant is also integrated to answer questions outside the trained corpus. Based on the user's speech, emotion and Nadine's current emotion, Nadine can exhibit different human motion to user. [13] Nadine's memory model [42] also allows her to remember specific facts about the user and context of current conversation in order to provide appropriate responses. Upon understanding the user interaction and environment context, an appropriate verbal or non-verbal response is decided. For this purpose, Nadine's processing layer maps each perception layer stimuli to an activation and threshold. Based on the processing of each stimulus by each sub-module, the activation levels are varied. When thresholds are reached, each winning action is passed on to interaction layer to show the various responses in Nadine.

The interaction layer or Nadine controller is responsible for executing each of the responses received from processing layer to show it in Nadine's face or gesture. For example, based on user location modify Nadine's head to maintain eye gaze with user. Apart from this, the interaction layer is also responsible for controlling her motors to show different gestures and facial expressions. For verbal responses, it includes a speech synthesizer and lip societyion module. Based on the verbal response, corresponding phonemes and visemes are generated. The speech synthesizer also takes into account the tone of dialog (to show various emotions) while generating speech. The lorgahronization converts the visemes into corresponding facial motor position to move Nadine's lips according to her speech. Currently, Nadine can support six languages including English, German, French, Chinese, Hindi and Japanese.

Events

Nadine has participated in live demos on stage and engaged with people from all walks of life. Proclaimed as one of the world's most realistic humanoid robot, [43] Nadine made her first public appearance as a key highlight at the “Human+: The Future of Our Species” exhibition held in Singapore's ArtScience Museum. [44]

She has interacted with many people from corporate companies across various industries such as Dentsu Aegis Network (DAN), Credit Suisse [45] and Deutsche Bank. [46]

Nadine also interacted with Prime Minister of India, His Excellency Narendra Modi during his historic visit to NTU Singapore, on 1 June 2018, which was one of the innovations he took special interest in. [47] [48]

Nadine has worked as a customer service agent at AIA Singapore. [20] [21] [22] She has been trained to handle questions that are usually asked to AIA customer service agents. She also encourages AIA customers to sign up with AIA e-care registration portal. Customer service interactions were used to train a machine-learning based conversational dialog engine. A client-server architecture was also set up between our platform and AIA portal to allow fast and secure communication. [49]

In late 2020 and until April 2021, Nadine has spent 6 months at Bright Hill Evergreen [50] Home in Singapore to assist elderly in playing Bingo and interacting with them. With the ethical committee agreement of NTU, a thorough study has been done for the first time on the interaction of Nadine social robot with light dementia patients. [51] The film of this interaction can be seen on Nadine YouTube Channel. [52]

Related Research Articles

<span class="mw-page-title-main">Android (robot)</span> Robot resembling a human

An android is a humanoid robot or other artificial being often made from a flesh-like material. Historically, androids were completely within the domain of science fiction and frequently seen in film and television, but advances in robot technology now allow the design of functional and realistic humanoid robots.

<span class="mw-page-title-main">Humanoid robot</span> Body shape similar to a human

A humanoid robot is a robot resembling the human body in shape. The design may be for functional purposes, such as interacting with human tools and environments, for experimental purposes, such as the study of bipedal locomotion, or for other purposes. In general, humanoid robots have a torso, a head, two arms, and two legs, though some humanoid robots may replicate only part of the body, for example, from the waist up. Some humanoid robots also have heads designed to replicate human facial features such as eyes and mouths. Androids are humanoid robots built to aesthetically resemble humans.

Affective computing is the study and development of systems and devices that can recognize, interpret, process, and simulate human affects. It is an interdisciplinary field spanning computer science, psychology, and cognitive science. While some core ideas in the field may be traced as far back as to early philosophical inquiries into emotion, the more modern branch of computer science originated with Rosalind Picard's 1995 paper on affective computing and her book Affective Computing published by MIT Press. One of the motivations for the research is the ability to give machines emotional intelligence, including to simulate empathy. The machine should interpret the emotional state of humans and adapt its behavior to them, giving an appropriate response to those emotions.

<span class="mw-page-title-main">Social robot</span>

A social robot is an autonomous robot that interacts and communicates with humans or other autonomous physical agents by following social behaviors and rules attached to its role. Like other robots, a social robot is physically embodied Some synthetic social agents are designed with a screen to represent the head or 'face' to dynamically communicate with users. In these cases, the status as a social robot depends on the form of the 'body' of the social agent; if the body has and uses some physical motors and sensor abilities, then the system could be considered a robot.

<span class="mw-page-title-main">Crowd simulation</span> Model of movement

Crowd simulation is the process of simulating the movement of a large number of entities or characters. It is commonly used to create virtual scenes for visual media like films and video games, and is also used in crisis training, architecture and urban planning, and evacuation simulation.

<span class="mw-page-title-main">Gesture recognition</span> Topic in computer science and language technology

Gesture recognition is an area of research and development in computer science and language technology concerned with the recognition and interpretation of human gestures. A subdiscipline of computer vision, it employs mathematical algorithms to interpret gestures.

In artificial intelligence, an embodied agent, also sometimes referred to as an interface agent, is an intelligent agent that interacts with the environment through a physical body within that environment. Agents that are represented graphically with a body, for example a human or a cartoon animal, are also called embodied agents, although they have only virtual, not physical, embodiment. A branch of artificial intelligence focuses on empowering such agents to interact autonomously with human beings and the environment. Mobile robots are one example of physically embodied agents; Ananova and Microsoft Agent are examples of graphically embodied agents. Embodied conversational agents are embodied agents that are capable of engaging in conversation with one another and with humans employing the same verbal and nonverbal means that humans do.

Human–robot interaction (HRI) is the study of interactions between humans and robots. Human–robot interaction is a multidisciplinary field with contributions from human–computer interaction, artificial intelligence, robotics, natural language processing, design, and psychology. A subfield known as physical human–robot interaction (pHRI) has tended to focus on device design to enable people to safely interact with robotic systems.

<span class="mw-page-title-main">Dialogue system</span>

A dialogue system, or conversational agent (CA), is a computer system intended to converse with a human. Dialogue systems employed one or more of text, speech, graphics, haptics, gestures, and other modes for communication on both the input and output channel.

Affective design describes the design of user interfaces in which emotional information is communicated to the computer from the user in a natural and comfortable way. The computer processes the emotional information and adapts or responds to try to improve the interaction in some way. The notion of affective design emerged from the field of human–computer interaction (HCI), specifically from the developing area of affective computing. Affective design serves an important role in user experience (UX) as it contributes to the improvement of the user's personal condition in relation to the computing system. The goals of affective design focus on providing users with an optimal, proactive experience. Amongst overlap with several fields, applications of affective design include ambient intelligence, human–robot interaction, and video games.

<span class="mw-page-title-main">Robotics</span> Design, construction, use, and application of robots

Robotics is the interdisciplinary study and practice of the design, construction, operation, and use of robots.

<span class="mw-page-title-main">Human–computer interaction</span> Academic discipline studying the relationship between computer systems and their users

Human–computer interaction (HCI) is research in the design and the use of computer technology, which focuses on the interfaces between people (users) and computers. HCI researchers observe the ways humans interact with computers and design technologies that allow humans to interact with computers in novel ways. A device that allows interaction between human being and a computer is known as a "Human-computer Interface (HCI)".

Emotions in virtual communication are expressed and understood in a variety of different ways from those in face-to-face interactions. Virtual communication continues to evolve as technological advances emerge that give way to new possibilities in computer-mediated communication (CMC). The lack of typical auditory and visual cues associated with human emotion gives rise to alternative forms of emotional expression that are cohesive with many different virtual environments. Some environments provide only space for text based communication, where emotions can only be expressed using words. More newly developed forms of expression provide users the opportunity to portray their emotions using images.

<span class="mw-page-title-main">Nadia Magnenat Thalmann</span> Computer scientist

Nadia Magnenat Thalmann is a computer graphics scientist and robotician and is the founder and head of MIRALab at the University of Geneva. She has chaired the Institute for Media Innovation at Nanyang Technological University (NTU), Singapore from 2009 to 2021.

<span class="mw-page-title-main">Daniel Thalmann</span>

Prof. Daniel Thalmann is a Swiss and Canadian computer scientist and a pioneer in Virtual humans. He is currently Honorary Professor at EPFL, Switzerland and Director of Research Development at MIRALab Sarl in Geneva, Switzerland.

Rendez-vous in Montreal is a 1987 animated film that used advanced computer techniques to achieve such effects as modelling the film stars Marilyn Monroe and Humphrey Bogart. The film was directed by Nadia Magnenat Thalmann and Daniel Thalmann and produced with a team of 10 people. Specific interactive software [1] was developed that allowed designers to interactively use commands to generate the sequences. The main purpose of Rendez-vous in Montreal was to show that true synthetic actors can be created. This film represented a technological breakthrough both on the software side and the film itself.

Artificial empathy or computational empathy is the development of AI systems—such as companion robots or virtual agents—that can detect emotions and respond to them in an empathic way.

<span class="mw-page-title-main">Jacob O. Wobbrock</span> American computer scientist

Jacob O. Wobbrock is a Professor in the University of Washington Information School and, by courtesy, in the Paul G. Allen School of Computer Science & Engineering at the University of Washington. He is Director of the ACE Lab, Associate Director and founding Co-Director Emeritus of the CREATE research center, and a founding member of the DUB Group and the MHCI+D degree program.

<span class="mw-page-title-main">Aude Billard</span> Swiss physicist

Aude G. Billard is a Swiss physicist in the fields of machine learning and human-robot interactions. As a full professor at the School of Engineering at Swiss Federal Institute of Technology in Lausanne (EPFL), Billard’s research focuses on applying machine learning to support robot learning through human guidance. Billard’s work on human-robot interactions has been recognized numerous times by the Institute of Electrical and Electronics Engineers (IEEE) and she currently holds a leadership position on the executive committee of the IEEE Robotics and Automation Society (RAS) as the vice president of publication activities.

<span class="mw-page-title-main">Virtual human</span> Computer simulation of a person

A virtual human is a software fictional character or human being. Virtual human have been created as tools and artificial companions in simulation, video games, film production, human factors and ergonomic and usability studies in various industries, clothing industry, telecommunications (avatars), medicine, etc. These applications require domain-dependent simulation fidelity. A medical application might require an exact simulation of specific internal organs; film industry requires highest aesthetic standards, natural movements, and facial expressions; ergonomic studies require faithful body proportions for a particular population segment and realistic locomotion with constraints, etc.

References

  1. ↑ Gieriet, Marc (2016-10-09). "Nadia und Nadine". SRF Schweizer Radio und Fernsehen. Retrieved 2016-10-10.
  2. ↑ "Media coverage on Nadine exhibition". Archived from the original on 2018-11-28. Retrieved 2017-08-01.
  3. ↑ Mulligan, Gabriella (2017-03-17). "Is robotics a solution to the growing needs of the elderly?". BBC. Retrieved 2017-03-17.
  4. ↑ Lim, Paige (2016-03-07). "Now you're talking: human-like robot may one day care for dementia patients". Reuters. Retrieved 2016-03-09.
  5. ↑ Lee, Jocelyn (2017-05-27). "Chat With A Female Robot (Who's Made To Look Just Like Her Creator)". 8 Days. Archived from the original on 2018-11-28. Retrieved 2017-06-08.
  6. ↑ J Ren, X Jiang and J Yuan, Quantized Fuzzy LBP for Face Recognition, 40th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2015, Brisbane, Australia, 2015
  7. ↑ "NTU scientists unveil social and telepresence robots". Nanyang Technological University. 2015-12-29. Archived from the original on 2019-09-03. Retrieved 2016-01-14.
  8. ↑ Kochanowicz J, A. H. Tan and D. Thalmann, Modeling human-like non-rationality for social agents, Proceedings of the ACM 29th International Conference on Computer Animation and Social Agents (CASA 2016), pp. 11-20, Geneva, Switzerland, May 23–25, 2016
  9. ↑ J. Zhang J, N. Magnenat Thalmann and J. Zheng, Combining Memory and Emotion With Dialog on Social Companion: A Review, Proceedings of the ACM 29th International Conference on Computer Animation and Social Agents (CASA 2016), pp. 1-9, Geneva, Switzerland, May 23–25, 2016
  10. ↑ Berger, Sarah (2015-12-31). "Humanlike, Social Robot 'Nadine' Can Feel Emotions And Has A Good Memory, Scientists Claim". International Business Times. Retrieved 2016-01-12.
  11. ↑ A. Beck, Z. Zhang and N. Magnenat Thalmann, Motion Control for Social Behaviors, Context Aware Human-Robot and Human-Agent Interaction, Springer International Publishing, 237-256, 2015
  12. ↑ Z.P. Bian, J. Hou, L.P. Chau and N. Magnenat Thalmann, Fall Detection Based on Body Part Tracking Using a Depth Camera, IEEE Journal of Biomedical and Health Informatics, Vol. 19, No. 2, Pp. 430-439, 2015
  13. 1 2 3 J. Zhang, J. Zheng and N. Magnenat Thalmann, PCMD: personality‐characterized mood dynamics model toward personalized virtual characters, Computer Animation and Virtual Worlds, Vol. 26, Issue 3-4, Pp. 237-245, 2015
  14. ↑ J. Zhang, J. Zheng and N. Magnenat Thalmann, Modeling Personality, Mood, and Emotions, Context Aware Human-Robot and Human-Agent Interaction, Springer International Publishing, 211-236, 2015
  15. ↑ Y. Xiao, Z. Zhang, A. Beck, J. Yuan and D. Thalmann, Human-Robot Interaction by Understanding Upper Body Gestures, MIT Press Journals - Presence: Teleoperators and Virtual Environments, Vol. 23, No. 2, Pp. 133-154, 2014
  16. ↑ Z. Yumak, J. Ren, N. Magnenat Thalmann, and J. Yuan, Modelling Multi-Party Interactions among Virtual Characters, Robots, and Humans, MIT Press Journals - Presence: Teleoperators and Virtual Environments, Vol. 23, No. 2, Pp. 172-190, 2014
  17. ↑ Martin, Mayo (2017-05-19). "Singapore's receptionist robot makes her public debut at ArtScience Museum's futuristic show". Channel NewsAsia. Archived from the original on 2018-11-18. Retrieved 2017-05-19.
  18. ↑ Wong, Cara (2017-05-19). "Conversation with a humanoid robot". The Straits Times. Retrieved 2016-03-09.
  19. ↑ "Chat With A Female Robot (Who's Made To Look Just Like Her Creator)". Archived from the original on 2018-11-28. Retrieved 2018-01-29.
  20. 1 2 "AIA To Pioneer Adoption Of Artificial Intelligence In Singapore's Life Insurance Sector". AIA Singapore. 2018-10-12. Retrieved 2018-10-26.
  21. 1 2 Benjamin, Ang (2018-10-22). "Singapore: AIA transforms customer service with insurance industry's first humanoids". Asia Insurance Review. Archived from the original on 2019-02-02. Retrieved 2018-10-26.
  22. 1 2 Gabriel, Olano (2018-10-19). "AIA Singapore welcomes two new robotic hires". Insurance Business Asia. Retrieved 2018-10-20.
  23. ↑ Social Robotics
  24. ↑ MIRALab
  25. ↑ C. Cig, Z. Kasap, A. Egges and N. Magnenat Thalmann, Realistic Emotional Gaze and Head Behavior Generation Based on Arousal and Dominance Factors, The 3rd International Conference on Motion in Games 2010, Springer, 2010
  26. ↑ Z. Kasap and N. Magnenat Thalmann, Building long-term relationships with virtual and robotic characters: the role of remembering, The Visual Computer (IF: 1.073), vol. 28, no. 1, pp. 87-97, January 2012
  27. ↑ Z. Kasap, M. Ben Moussa, P. Chaudhuri and N. Magnenat Thalmann, Making Them Remember—Emotional Virtual Characters with Memory, IEEE Computer Graphics and Applications (IF: 1.116), vol. 29, no. 2, pp. 20-29, March 2009
  28. ↑ The robot EVA playing in the Roten Fabrik theatre in Zurich
  29. ↑ ColdFusion, The Most Realistic Robots! (2018)
  30. ↑ J. Zhang J, N. Magnenat Thalmann and J. Zheng, Combining Memory and Emotion With Dialog on Social Companion: A Review, Proceedings of the ACM 29th International Conference on Computer Animation and Social Agents (CASA 2016), pp. 1-9, Geneva, Switzerland, May 23–25, 2016
  31. ↑ L. Tian, N. Magnenat Thalmann, D. Thalmann, J. Zheng, A methodology to model and simulate customized human robotic realistic hand, Proceedings of the 35th Computer Graphics International (CGI 2018), ACM, Bintan, Indonesia, June 11–14, 2018
  32. ↑ L. Tian, N. Magnenat Thalmann, D. Thalmann, J. Zheng, The Making of a 3D-Printed, Cable-Driven, Single-Model, Lightweight Humanoid Robotic Hand, Frontiers in Robotics and AI, DOI: 10.3389/frobt.2017.00065, pp. 65, December 04, 2017
  33. ↑ N. Magnenat Thalmann, L. Tian and F. Yao, Nadine: A Social Robot that Can Localize Objects and Grasp Them in a Human Way, Frontiers in Electronic Technologies, Springer, pp. 1-23, 2017
  34. ↑ H. Liang, J. Yuan, D. Thalmann and N. Magnenat Thalmann, AR in Hand: Egocentric Palm Pose Tracking and Gesture Recognition for Augmented Reality Applications, ACM Multimedia Conference 2015 (ACMMM 2015), Brisbane, Australia, 2015
  35. ↑ J. Ren, X. Jiang, and J. Yuan, Face and Facial Expressions Recognition and Analysis, Context Aware Human-Robot and Human-Agent Interaction, Springer International Publishing, 3-29, 2015
  36. ↑ L. Ge, H. Liang, J. Yuan and D. Thalmann, Robust 3D Hand Pose Estimation in Single Depth Images: from Single-View CNN to Multi-View CNNs, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, USA, 24 June 2016
  37. ↑ Q. Ke, M. Bennamoun, S. An, F. Sohel and F. Boussaid, A New Representation of Skeleton Sequences for 3D Action Recognition, CVPR 2017
  38. ↑ Z. Fang, J. Yuan and N. Magnenat Thalmann, Understanding Human-Object Interaction in RGB-D videos for Human Robot Interaction, Proceedings of the 35th Computer Graphics International (CGI 2018), ACM, Indonesia, June 11–14, 2018
  39. ↑ J. Redmon and A. Farhadi, YOLO9000: Better, Faster, Stronger, arXiv preprint, 2017
  40. ↑ R.S Wallace, The Anatomy of A.L.I.C.E., In Parsing the Turing Test (pp. 181-210). Springer, Dordrecht, 2009
  41. ↑ Chatterbot
  42. ↑ J. Zhang, J. Zheng and N. Magnenat Thalmann, MCAEM: Mixed-Correlation-Analysis based Episodic Memory for Companion-User Interactions, The Visual Computer, DOI: 10.1007/s00371-018-1537-3, Vol 34, Issue 6-8, pp. 1129-1141, May 10, 2018
  43. ↑ Aloysius, Low (2017-05-19). "'Human+' exhibit explores the future of humanity". CNET. Retrieved 2017-06-08.
  44. ↑ "Human+: The Future of Our Species". Archived from the original on 2018-11-28. Retrieved 2018-11-28.
  45. ↑ Credit Suisse Global Megatrends Conference 2018
  46. ↑ 9th Annual deAccess Asia 2018 Conference Highlights
  47. ↑ "NTU marks historic visit by Indian Prime Minister". Archived from the original on 2018-11-28. Retrieved 2018-11-28.
  48. ↑ PM Modi Interacts With Robot In Nanyang Tech University In Singapore
  49. ↑ NTU Singapore’s social robot Nadine starts work as a receptionist
  50. ↑ Bright Hill Evergreen
  51. ↑ Gauri Tulsulkar, Nadia Magnenat Thalmann et al., Can a humanoid social robot stimulate the interactivity of cognitively impaired elderly? A thorough study based on computer vision methods, The Visual Computer, vol. 37, pp. 3019-3038, July 2021
  52. ↑ Nadine Social Robot @ Brighthill Evergreen Home
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.