Mixed reality

Last updated

Mixed reality (MR), sometimes referred to as hybrid reality, [1] is the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time. Mixed reality takes place not only in the physical world or the virtual world, [1] but is a mix of reality and virtual reality, encompassing both augmented reality and augmented virtuality [2] via immersive technology. The first immersive mixed reality system, providing enveloping sight, sound, and touch was the Virtual Fixtures platform developed at the U.S. Air Force's Armstrong Laboratories in the early 1990s. In a study published in 1992, the Virtual Fixtures project at the U.S. Air Force demonstrated for the first time that human performance could be significantly amplified by the introduction of spatially registered virtual objects overlaid on top of a person's direct view of a real physical environment. [3]

A virtual world is a computer-based simulated environment which may be populated by many users who can create a personal avatar, and simultaneously and independently explore the virtual world, participate in its activities and communicate with others. These avatars can be textual, two or three-dimensional graphical representations, or live video avatars with auditory and touch sensations. In general, virtual worlds allow for multiple users but single player computer games, such as Skyrim, can also be considered a type of virtual world.

Reality is the sum or aggregate of all that is real or existent, as opposed to that which is merely imaginary. The term is also used to refer to the ontological status of things, indicating their existence. In physical terms, reality is the totality of the universe, known and unknown. Philosophical questions about the nature of reality or existence or being are considered under the rubric of ontology, which is a major branch of metaphysics in the Western philosophical tradition. Ontological questions also feature in diverse branches of philosophy, including the philosophy of science, philosophy of religion, philosophy of mathematics, and philosophical logic. These include questions about whether only physical objects are real, whether reality is fundamentally immaterial, whether hypothetical unobservable entities posited by scientific theories exist, whether God exists, whether numbers and other abstract objects exist, and whether possible worlds exist.

Virtual reality Computer-simulated environment simulating physical presence in real or imagined worlds

Virtual reality (VR) is an interactive computer-generated experience taking place within a simulated environment. It incorporates mainly auditory and visual feedback, but may also allow other types of sensory feedback. This immersive environment can be similar to the real world or it can be fantastical.


Virtual reality (VR) versus augmented reality (AR) versus mixed reality (MR)

The definitions in the modern contemporary economy makes the distinction between VR, AR and MR very clear:

An Example Mixed Reality: Virtual characters mixed into a live video stream of the real world. Mrfops.jpg
An Example Mixed Reality: Virtual characters mixed into a live video stream of the real world.


Virtuality continuum and mediality continuum

In 1994 Paul Milgram and Fumio Kishino defined a mixed reality as "...anywhere between the extrema of the virtuality continuum" (VC), [2] where the virtuality continuum extends from the completely real through to the completely virtual environment with augmented reality and augmented virtuality ranging between. The first fully immersive mixed reality system was the Virtual Fixtures platform developed at US Air Force, Armstrong Labs in 1992 by Louis Rosenberg to enable human users to control robots in real-world environments that included real physical objects and 3D virtual overlays called "fixtures" that were added enhance human performance of manipulation tasks. Published studies showed that by introducing virtual objects into the real world, significant performance increases could be achieved by human operators. [5] [6]

Augmented reality view of the real world with computer-generated supplementary features

Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real-world are "augmented" by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory. The overlaid sensory information can be constructive or destructive and is seamlessly interwoven with the physical world such that it is perceived as an immersive aspect of the real environment. In this way, augmented reality alters one's ongoing perception of a real-world environment, whereas virtual reality completely replaces the user's real-world environment with a simulated one. Augmented reality is related to two largely synonymous terms: mixed reality and computer-mediated reality.

A virtual fixture is an overlay of augmented sensory information upon a user's perception of a real environment in order to improve human performance in both direct and remotely manipulated tasks. Developed in the early 1990s by Louis Rosenberg at the U.S. Air Force Research Laboratory (AFRL), Virtual Fixtures was a pioneering platform in virtual reality and augmented reality technologies.

Armstrong Laboratory

Armstrong Laboratory was a research and development organization operated by the United States Air Force Materiel Command. In 1997, the Laboratory was merged into the Air Force Research Laboratory.

Mixed Reality (1992, US Air Force Laboratory, L Rosenberg) Virtual-Fixtures-USAF-AR.jpg
Mixed Reality (1992, US Air Force Laboratory, L Rosenberg)

The continuum of mixed reality is one of the two axes in Steve Mann's concept of mediated reality as implemented by various welding helmets and wearable computers and wearable photographic systems he created in the 1970s and early 1980s, [7] [8] [9] [10] [11] the second axis being the mediality continuum, which includes, for example, Diminished Reality (as implemented in a welding helmet or eyeglasses that can block out advertising or replace real-world ads with useful information) [12] [13]

Steve Mann Professor and wearable computing researcher

Steven Mann is a Canadian researcher and inventor best known for his work on augmented reality, computational photography, particularly wearable computing and high dynamic range imaging.

"The conventionally held view of a Virtual Reality (VR) environment is one in which the participant-observer is totally immersed in, and able to interact with, a completely synthetic world. Such a world may mimic the properties of some real-world environments, either existing or fictional; however, it can also exceed the bounds of physical reality by creating a world in which the physical laws ordinarily governing space, time, mechanics, material properties, etc. no longer hold. What may be overlooked in this view, however, is that the VR label is also frequently used in association with a variety of other environments, to which total immersion and complete synthesis do not necessarily pertain, but which fall somewhere along a virtuality continuum. In this paper we focus on a particular subclass of VR related technologies that involve the merging of real and virtual worlds, which we refer to generically as Mixed Reality (MR)."

Reality-Virtuality Continuum Reality-Virtuality Continuum.svg
Reality-Virtuality Continuum
Virtuality axis (left-to-right) and mediality axis (bottom to top) of the mediated reality continuum. Here four example points are shown: augmented reality, augmented virtuality, mediated reality, and mediated virtuality on the virtuality and mediality axes. This includes, for example, diminished reality (e.g. computerized welding helmets that filter out and diminish certain parts of a scene) Mediated reality continuum 2d.png
Virtuality axis (left-to-right) and mediality axis (bottom to top) of the mediated reality continuum. Here four example points are shown: augmented reality, augmented virtuality, mediated reality, and mediated virtuality on the virtuality and mediality axes. This includes, for example, diminished reality (e.g. computerized welding helmets that filter out and diminish certain parts of a scene)

Interreality physics

In a physics context, the term "interreality system" [14] refers to a virtual reality system coupled to its real-world counterpart. A paper in the May 2007 issue of Physical Review E [15] describes an interreality system comprising a real physical pendulum coupled to a pendulum that only exists in virtual reality. This system apparently has two stable states of motion: a "Dual Reality" state in which the motion of the two pendula are uncorrelated and a "Mixed Reality" state in which the pendula exhibit stable phase-locked motion which is highly correlated. The use of the terms "mixed reality" and "interreality" in the context of physics is clearly defined but may be slightly different in other fields.

Augmented virtuality

Someone in a virtual environment seen with a camera acting as a third person point of view. Mixed Reality with a Virtual Reality Headset.png
Someone in a virtual environment seen with a camera acting as a third person point of view.

Augmented virtuality (AV), is a subcategory of mixed reality which refers to the merging of real world objects into virtual worlds. [17]

As an intermediate case in the virtuality continuum, it refers to predominantly virtual spaces, where physical elements, e.g. physical objects or people, are dynamically integrated into, and can interact with, the virtual world in real time. This integration is achieved with the use of various techniques. Often streaming video from physical spaces (e.g., via webcam) [18] or using 3-dimensional digitalisation of physical objects. [19]

The use of real-world sensor information (e.g., gyroscopes) to control a virtual environment is an additional form of augmented virtuality, in which external inputs provide context for the virtual view.


A topic of much research, MR has found its way into a number of applications, evident in the arts and entertainment industries. However, MR is also branching out into the business and education worlds with systems such as these:

Moving from static product catalogs to interactive 3D smart digital replicas. Solution consists of application software products with scalable license model.

Moving from e-learning to s-learning—state of the art in knowledge transfer for education. Simulation/VR based training, interactive experiential learning. Software and display solutions with scalable licensed curriculum development model.

Combat reality is simulated and represented in complex layered data through HMD.

One of the possible applications mixed realities is for training military soldiers. Training solutions are often built on Commercial Off the Shelf (COTS) technologies. Examples of technologies used by the Army are Virtual Battlespace 3 and VirTra. As of 2018, the VirTra technology is being purchased by both the civilian and military law enforcement to train personnel in a variety of scenarios. These scenarios include active shooter; domestic violence; military traffic stops, etc. [20] [21]  Mixed reality technologies have been used by U.S. Army Research Laboratory scientists to study how this stress affects decision making. With mixed reality, researchers may safely study military service men and women in scenarios where soldiers would not likely survive. [22]

As of 2017, the U.S. Army was developing the Synthetic Training Environment (STE). STE is a collection of technologies for training purposes that has been estimated to include mixed reality. As of 2018, STE was still in development without a projected completion date. Some recorded goals of the simulation were to increase simulation training capabilities, and the availability of the environment to other systems, and to enhance realism. [23] It was claimed that training costs to be reduced with mixed reality environments like STE. [24] [25] For example, using mixed environments could reduce the amount of munition expended during training. [26] It was reported in 2018 that STE would include representation of any part of the world's terrain for training purposes. [27] STE would offer a variety of training opportunities for squad brigade and combat teams, including, but not limited to Stryker, armory, and infantry. [28] It is estimated that STE will eventually replace the Army's Live, Virtual, Constructive – Integrated Architecture (LVC-IA). [29]

Mixed reality allows a global workforce of remote teams to work together and tackle an organization's business challenges. No matter where they are physically located, an employee can strap on their headset and noise-canceling headphones and enter a collaborative, immersive virtual environment. Language barriers will become irrelevant as AR applications are able to accurately translate in real time. It also means a more flexible workforce. While many employers still use inflexible models of fixed working time and location, there is evidence that employees are more productive if they have greater autonomy over where, when and how they work. Some employees prefer loud work spaces, others need silence. Some work best in the morning, others at night. Employees also benefit from autonomy in how they work because everyone processes information differently. The classic VAK model for learning styles differentiates Visual, Auditory and Kinesthetic learners. [30]

Machine maintenance is also a subject that can be executed with the help of mixed reality. Larger companies that have multiple manufacturing locations with a lot of machinery can use mixed reality to educate and instruct their employee. The machines need regular checkups and have to be adjusted every now and then. These adjustments are mostly done by humans, so these employees need to be informed about every small adjustment that needs to be done. By using mixed reality, employees from multiple locations can put on a headset, and get live instructions about the changes. Instructors can operate the representation that every employee sees and can glide through the production area, zooming in to technical details and explain every change of a machine. It has shown that a five-minute training session with such a mixed reality program has the same results as the employees reading a 50-page training manual. [31]

Mixed reality is applied in the industrial field in order to build mockups that combine physical and digital elements. [32] With the use of simultaneous localization and mapping (SLAM), mockups can interact with the physical world to utilize features such as object permanence.[ citation needed ]

It has been hypothesised that a hybrid of mixed and virtual reality could pave the way for human consciousness to be transferred into digital form entirely - a concept known as Virternity, which would leverage blockchain to create its main platform. [33] [34]

Display technologies

Here are some more commonly used MR display technologies:

Notable examples

See also

Related Research Articles

Head-mounted display device used in virtual reality systems

A head-mounted display, both abbreviated HMD, is a display device, worn on the head or as part of a helmet, that has a small display optic in front of one or each eye. A HMD has many uses, including in gaming, aviation, engineering, and medicine lift. A head-mounted display is the primary component of virtual reality headsets.

Virtual reality therapy (VRT), also known as virtual reality immersion therapy (VRIT), simulation for therapy (SFT), virtual reality exposure therapy (VRET), and computerized CBT (CCBT), is the use of virtual reality technology for psychological or occupational therapy. Patients receiving virtual reality therapy navigate through digitally created environments and complete specially designed tasks often tailored to treat a specific ailment. Technology can range from a simple PC and keyboard setup, to a modern virtual reality headset. It is widely used as an alternative form of exposure therapy, in which patients interact with harmless virtual representations of traumatic stimuli in order to reduce fear responses. It has proven to be especially effective at treating PTSD. Virtual reality therapy has also been used to help stroke patients regain muscle control, to treat other disorders such as body dysmorphia, and to improve social skills in those diagnosed with autism.

Digital Life is a research and educational program about radically rethinking of the human-computer interactive experience. It integrates digital world and physical world. It makes interfaces more responsive and proactive

The virtuality continuum is a continuous scale ranging between the completely virtual, a virtuality, and the completely real, reality. The reality–virtuality continuum therefore encompasses all possible variations and compositions of real and virtual objects. It has been described as a concept in new media and computer science, but in fact it could be considered a matter of anthropology. The concept was first introduced by Paul Milgram.

In computing, 3D interaction is a form of human-machine interaction where users are able to move and perform interaction in 3D space. Both human and machine process information where the physical position of elements in the 3D space is relevant.

Immersion into virtual reality is a perception of being physically present in a non-physical world. The perception is created by surrounding the user of the VR system in images, sound or other stimuli that provide an engrossing total environment.

A projection augmented model is an element sometimes employed in virtual reality systems. It consists of a physical three-dimensional model onto which a computer image is projected to create a realistic looking object. Importantly, the physical model is the same geometric shape as the object that the PA model depicts.

Virtual Reality Cue Reactivity (VRCR) is a computer-enhanced methodology used to assess behavioral and physiological reactivity to drug and alcohol sensory cues. Studies indicate that cue reactivity—a response to the presentation of various visual, auditory, olfactory, and tactile cues—increases physiological excitement in addicts. VRCR utilizes virtual reality (VR) technology to stimulate cue reactivity in the most efficient and realistic environments possible; the intention being that coping skills can be taught in a contextual scenario that reflect a real world situation. While still in the early stages of development, studies have shown that VRCR is an effective means of generating a craving-inspiring environment that is tempting to a patient suffering from addiction.

Immersive technology

Immersive technology refers to technology that attempts to emulate a physical world through the means of a digital or simulated world, thereby creating a sense of immersion. Immersive technology enables mixed reality; in some uses, the term "immersive computing" is effectively synonymous with mixed reality as a user interface.

Visuo-haptic mixed reality (VHMR) is a branch of mixed reality that has the ability of merging visual and tactile perceptions of both virtual and real objects with a collocated approach. The first known system to overlay augmented haptic perceptions on direct views of the real world is the Virtual Fixtures system developed in 1992 at the US Air Force Research Laboratories. Like any emerging technology, the development of the VHMR systems is accompanied by challenges that, in this case, deal with the efforts to enhance the multi-modal human perception with the user-computer interface and interaction devices at the moment available. Visuo-haptic mixed reality (VHMR) consists of adding to a real scene the ability to see and touch virtual objects. It requires the use of see-through display technology for visually mixing real and virtual objects and haptic devices necessary to provide haptic stimuli to the user while interacting with the virtual objects. A VHMR setup allows the user to perceive visual and kinesthetic stimuli in a co-located manner, i.e., the user can see and touch virtual objects at the same spatial location. This setup overcomes the limits of the traditional one, i.e, display and haptic device, because the visuo-haptic co-location of the user's hand and a virtual tool improve the sensory integration of multimodal cues and makes the interaction more natural. But it also comes with technological challenges in order to improve the naturalness of the perceptual experience.

Windows Mixed Reality mixed reality computing platform by Microsoft

Windows Mixed Reality is a mixed reality platform introduced as part of the Windows 10 operating system, which provides holographic and mixed reality experiences with compatible head-mounted displays.

Extended reality (XR) is a term referring to all real-and-virtual combined environments and human-machine interactions generated by computer technology and wearables. It includes representative forms such as augmented reality (AR), augmented virtuality (AV) and virtual reality (VR). and the areas interpolated among them. The levels of virtuality range from partially sensory inputs to immersive virtuality, also called VR.

Industrial augmented reality Descrive the application of Augmented Reality in industry

Industrial augmented reality (IAR) is related to the application of augmented reality (AR) to support an industrial process. The use of IAR dates back to the 1990s with the work of Thomas Caudell and David Mizell about the application of AR at Boeing. Since then several applications of this technique over the years have been proposed showing its potential in supporting some industrial processes. Although there have been several advances in technology, IAR is still considered to be at an infant developmental stage.

Virtual reality in fiction describes fictional representations of the technological concept of virtual reality.

X Reality is defined as: a form of “mixed reality environment that comes from the fusion (union) of ... ubiquitous sensor/actuator networks and shared online virtual worlds....”. It encompasses a wide spectrum of hardware and software, including sensory interfaces, applications, and infrastructures, that enable content creation for virtual reality (VR), mixed reality (MR), augmented reality (AR), cinematic reality (CR). With these tools, users generate new forms of reality by bringing digital objects into the physical world and bringing physical world objects into the digital world.

Virtual reality (VR) is a computer application which allows users to experience immersive, three dimensional visual and audio simulations. According to Pinho (2004), virtual reality is characterized by immersion in the 3D world, interaction with virtual objects, and involvement in exploring the virtual environment. These facets of virtual reality have many applications within the primary education sphere in enhancing student learning, increasing engagement, and creating new opportunities for addressing learning preferences.

Applications of VR can be found in fields as diverse as entertainment, marketing, education, medicine, construction and road safety training and many others. They provide numerous possibilities for users to explore virtual realities for various purposes.


  1. 1 2 de Souza e Silva, Adriana; Sutko, Daniel M. (2009). Digital Cityscapes: merging digital and urban playspaces. New York: Peter Lang Publishing, Inc.
  2. 1 2 P. Milgram and A. F. Kishino (1994). "Taxonomy of Mixed Reality Visual Displays". IEICE Transactions on Information and Systems. pp. 1321–1329. Retrieved 2013-10-17.
  3. Rosenberg, Louis B. (1992). "The Use of Virtual Fixtures As Perceptual Overlays to Enhance Operator Performance in Remote Environments". Technical Report AL-TR-0089, USAF Armstrong Laboratory, Wright-Patterson AFB OH, 1992.
  4. R. Freeman, A. Steed and B. Zhou, Rapid Scene Modelling, Registration and Specification for Mixed Reality Systems Archived 2007-02-06 at the Wayback Machine Proceedings of ACM Virtual Reality Software and Technology, pp. 147-150, Monterey, California, November 2005.
  5. L. B. Rosenberg. The Use of Virtual Fixtures As Perceptual Overlays to Enhance Operator Performance in Remote Environments. Technical Report AL-TR-0089, USAF Armstrong Laboratory, Wright-Patterson AFB OH, 1992.
  6. Rosenberg, Louis B. (1993). "Virtual fixtures as tools to enhance operator performance in telepresence environments". Telemanipulator Technology and Space Telerobotics. 2057: 10–21. Bibcode:1993SPIE.2057...10R. doi:10.1117/12.164901.
  7. Steve Mann, "Campus Canada", ISSN 0823-4531, p55 Feb-Mar 1985, pp58-59 Apr-May 1986, p72 Sep-Oct 1986
  8. Impulse, Volume 12, Number 2, 1985
  9. Quantigraphic camera promises HDR eyesight from Father of AR, by Chris Davies, SlashGear, Sep 12th 2012
  10. IEEE Technology & Society 31(3)
  11. Through the Glass, Lightly, IEEE Technology & Society, Volume 31, Number 3, Fall 2012, pages 10-14
  12. Mann, S., & Fung, J. (2001). Videoorbits on EyeTap devices for deliberately diminished reality or altering the visual perception of rigid planar patches of a real world scene. Proceedings of the Second IEEE International Symposium on Mixed Reality, pp 48-55, March 14–15, 2001.
  13. 关于智能眼镜 (About Smart Glasses), 36KR, 2016-01-09
  14. J. van Kokswijk, Hum@n, Telecoms & Internet as Interface to Interreality Archived 2007-09-26 at the Wayback Machine (Bergboek, The Netherlands, 2003).
  15. V. Gintautas and A. W. Hubler, Experimental evidence for mixed reality states in an interreality system Phys. Rev. E 75, 057201 (2007).
  16. LIV. "Mixed Reality Studio - The LIV Cube". LIV. Retrieved 2017-04-08.
  17. P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321–1329, 1994.
  18. "The Dive Home Page". web.archive.org. 30 June 2012.
  19. "Introduction - Teleimmersion Lab". UC Berkeley.
  20. Inc., VirTra,. "VirTra's Police Training Simulators Chosen by Three of the Largest U.S. Law Enforcement Departments". GlobeNewswire News Room. Retrieved 2018-08-22.
  21. "How do police use VR? Very well | Police Foundation". www.policefoundation.org. 2017-08-14. Retrieved 2018-08-22.
  22. Patton, Debbie; Marusich, Laura (2015-03-09). 2015 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision. pp. 145–150. doi:10.1109/COGSIMA.2015.7108190. ISBN   978-1-4799-8015-4.
  23. Eagen, Andrew (June 2017). "Expanding Simulations as a Means of Tactical Training with Multinational Partners" (PDF). A thesis presented to the Faculty of the U.S. Army Command and General Staff College.[ dead link ]
  24. Bukhari, Hatim; Andreatta, Pamela; Goldiez, Brian; Rabelo, Luis (2017-01-01). "A Framework for Determining the Return on Investment of Simulation-Based Training in Health Care". INQUIRY: The Journal of Health Care Organization, Provision, and Financing. 54: 0046958016687176. doi:10.1177/0046958016687176. ISSN   0046-9580. PMC   5798742 . PMID   28133988.
  25. Smith, Roger (2010-02-01). "The Long History of Gaming in Military Training". Simulation & Gaming. 41 (1): 6–19. doi:10.1177/1046878109334330. ISSN   1046-8781.
  26. Shufelt, Jr., J.W. (2006) A Vision for Future Virtual Training. In Virtual Media for Military Applications (pp. KN2-1 – KN2-12). Meeting Proceedings RTO-MP-HFM-136, Keynote 2. Neuilly-sur-Seine, France: RTO. Available from: http://www.rto.nato.int/abstracts.asp {{Webarchive|url=https://web.archive.org/web/20070613170605/http://www.rto.nato.int/Abstracts.asp |date=2007-06-13 }}
  27. "STAND-TO!". www.army.mil. Retrieved 2018-08-22.
  28. "Augmented reality may revolutionize Army training | U.S. Army Research Laboratory". www.arl.army.mil. Retrieved 2018-08-22.
  29. "Army Shoots for Single Synthetic Training Environment". GovTechWorks. 2015-11-17. Retrieved 2018-08-22.
  30. Sena, Pete. "How The Growth Of Mixed Reality Will Change Communication, Collaboration And The Future Of The Workplace". TechCrunch. Retrieved 2017-05-16.
  31. The Manufacturer. "Manufacturers are successfully using mixed reality today". www.themanufacturer.com.
  32. Barbieri, L.; et al. (2013). Mixed prototyping with configurable physical archetype for usability evaluation of product interfaces. Computers in Industry. 64 (3). pp. 310–323. doi:10.1016/j.compind.2012.11.010.
  33. "Potential of Blockchain Technology as Protocol of Universal Virtual Reality". TurboFuture. Retrieved 2018-10-18.
  34. Shirazi, Sina. "Pairing Augmented Reality with Virtual Reality".

Further reading

Commons-logo.svg Media related to Mixed reality at Wikimedia Commons