Mixed reality

Last updated
An example of mixed reality, showing virtual characters mixed into a real-world live stream Mrfops.jpg
An example of mixed reality, showing virtual characters mixed into a real-world live stream

Mixed reality (MR) is the merging of real and virtual worlds to produce new environments and visualizations, where physical and digital objects co-exist and interact in real time. Mixed reality does not exclusively take place in either the physical or virtual world, but is a hybrid of reality and virtual reality, encompassing both augmented reality and augmented virtuality via immersive technology. [2]


The first immersive mixed reality system that provided enveloping sight, sound, and touch was the Virtual Fixtures platform, which was developed in 1992 at the Armstrong Laboratories of the United States Air Force. The project demonstrated that human performance could be significantly amplified, by overlaying spatially registered virtual objects on top of a person's direct view of a real physical environment. [3]

Differences in terminology

The reality-virtuality continuum Reality-Virtuality Continuum.svg
The reality-virtuality continuum

Mixed reality refers to a continuum that encompasses both virtual reality (VR) and augmented reality (AR):


Mediated reality continuum (horizontal axis: virtuality; vertical axis: mediality). Four points are shown for augmented reality, augmented virtuality, mediated reality, and mediated virtuality. Mediated reality continuum 2d.png
Mediated reality continuum (horizontal axis: virtuality; vertical axis: mediality). Four points are shown for augmented reality, augmented virtuality, mediated reality, and mediated virtuality.

Virtuality continuum and mediality continuum

In 1994, Paul Milgram and Fumio Kishino defined a mixed reality as "...anywhere between the extrema of the virtuality continuum" (VC), [2] where the virtuality continuum extends from the completely real through to the completely virtual environment, with augmented reality and augmented virtuality ranging between. The first fully immersive mixed reality system was the Virtual Fixtures platform, which was developed in 1992 by Louis Rosenberg at the Armstrong Laboratories of the United States Air Force. It enabled human users to control robots in real-world environments that included real physical objects and 3D virtual overlays ("fixtures") that were added enhance human performance of manipulation tasks. Published studies showed that by introducing virtual objects into the real world, significant performance increases could be achieved by human operators. [4] [5]

The continuum of mixed reality is one of the two axes in Steve Mann's concept of mediated reality as implemented by various welding helmets, wearable computers, and wearable photographic systems he created in the 1970s and early 1980s. [6] [7] [8] [9] [10] The second axis was the mediality continuum, which includes Diminished Reality, such as implemented in a welding helmet or eyeglasses that can block out advertising or replace real-world ads with useful information. [11] [12]

"The conventionally held view of a Virtual Reality (VR) environment is one in which the participant-observer is immersed in, and able to interact with, a completely synthetic world. Such a world may mimic the properties of some real-world environments, either existing or fictional; however, it can also exceed the bounds of physical reality by creating a world in which the physical laws ordinarily governing space, time, mechanics, material properties, etc. no longer hold. What may be overlooked in this view, however, is that the VR label is also frequently used in association with a variety of other environments, to which total immersion and complete synthesis do not necessarily pertain, but which fall somewhere along a virtuality continuum. In this paper, we focus on a particular subclass of VR related technologies that involve the merging of real and virtual worlds, which we refer to generically as Mixed Reality (MR)."

Interreality physics

In a physics context, the term "interreality system" refers to a virtual reality system coupled with its real-world counterpart. [13] A 2007 paper describes an interreality system comprising a real physical pendulum coupled to a pendulum that only exists in virtual reality. [14] This system has two stable states of motion: a "Dual Reality" state in which the motion of the two pendula are uncorrelated, and a "Mixed Reality" state in which the pendula exhibit stable phase-locked motion, which is highly correlated. The use of the terms "mixed reality" and "interreality" is clearly defined in the context of physics, but may be slightly different in other fields.

Mixed Reality

A person is in a virtual environment, with a camera acting as a third-person point of view. Mixed Reality with a Virtual Reality Headset.png
A person is in a virtual environment, with a camera acting as a third-person point of view.

Augmented virtuality (AV) is a subcategory of mixed reality that refers to the merging of real-world objects into virtual worlds. [16]

As an intermediate case in the virtuality continuum, it refers to predominantly virtual spaces, where physical elements (such as physical objects or people) are dynamically integrated into and can interact with the virtual world in real time. This integration is achieved with the use of various techniques, such as streaming video from physical spaces, like through a webcam, [17] or using the 3D digitalization of physical objects. [18]

The use of real-world sensor information, such as gyroscopes, to control a virtual environment is an additional form of augmented virtuality, in which external inputs provide context for the virtual view.


Mixed reality has been used in applications across fields including art, entertainment, and military training.

Interactive product content management (IPCM)

Moving from static product catalogs to interactive 3D smart digital replicas, this solution consists of application software products with scalable license models.[ clarification needed ]

Simulation-based learning (SBL)

Moving from e-learning to s-learning, simulation-based learning includes VR-based training and interactive, experiential learning. This also includes software and display solutions with scalable licensed curriculum development model.[ clarification needed ]

Military training

Combat reality is simulated and represented in complex, layered data through HMD.[ clarification needed ] Military training solutions are often built on commercial off-the-shelf (COTS) technologies, such as Virtual Battlespace 3 and VirTra, both of which are used by the United States Army. As of 2018, VirTra is being used by both civilian and military law enforcement to train personnel in a variety of scenarios, including active shooter, domestic violence, and military traffic stops. [19] [20]  Mixed reality technologies have been used by the United States Army Research Laboratory to study how this stress affects decision-making. With mixed reality, researchers may safely study military personnel in scenarios where soldiers would not likely survive. [21]

In 2017, the U.S. Army was developing the Synthetic Training Environment (STE), a collection of technologies for training purposes that was expected to include mixed reality. As of 2018, STE was still in development without a projected completion date. Some recorded goals of STE included enhancing realism and increasing simulation training capabilities and STE availability to other systems. [22]

It was claimed that mixed-reality environments like STE could reduce training costs, [23] [24] such as reducing the amount of ammunition expended during training. [25] In 2018, it was reported that STE would include representation of any part of the world's terrain for training purposes. [26] STE would offer a variety of training opportunities for squad brigade and combat teams, including Stryker, armory, and infantry teams. [27] STE is expected to eventually replace the U.S. Army's Live, Virtual, Constructive – Integrated Architecture (LVC-IA). [28]

Remote working

Mixed reality allows a global workforce of remote teams to work together and tackle an organization's business challenges. No matter where they are physically located, an employee can wear a headset and noise-canceling headphones and enter a collaborative, immersive virtual environment. As these applications can accurately translate in real time, language barriers become irrelevant. This process also increases flexibility. While many employers still use inflexible models of fixed working time and location, there is evidence that employees are more productive if they have greater autonomy over where, when, and how they work. Some employees prefer loud work environments, while others need silence. Some work best in the morning; others work best at night. Employees also benefit from autonomy in how they work because of different ways of processing information. The classic model for learning styles differentiates between Visual, Auditory, and Kinesthetic learners. [29]

Machine maintenance can also be executed with the help of mixed reality. Larger companies with multiple manufacturing locations and a lot of machinery can use mixed reality to educate and instruct their employees. The machines need regular checkups and have to be adjusted every now and then. These adjustments are mostly done by humans, so employees need to be informed about needed adjustments. By using mixed reality, employees from multiple locations can wear headsets and receive live instructions about the changes. Instructors can operate the representation that every employee sees, and can glide through the production area, zooming in to technical details and explaining every change needed. Employees completing a five-minute training session with such a mixed-reality program has been shown to yield the same results as reading a 50-page training manual. [30]

Functional mockup

Mixed reality can be used to build mockups that combine physical and digital elements. With the use of simultaneous localization and mapping (SLAM), mockups can interact with the physical world to utilize features like object permanence.[ citation needed ]


It has been hypothesized that a hybrid of mixed and virtual reality could pave the way for human consciousness to be transferred into a digital form entirely—a concept known as Virternity, which would leverage blockchain to create its main platform. [31] [32]


Mixed reality can combine smartglasses with surgical processes. [33]

Display technologies

Here are some commonly used MR display technologies:

Notable applications

See also

Related Research Articles

Virtual reality Computer-simulated environment simulating physical presence in real or imagined worlds

Virtual reality (VR) is a simulated experience that can be similar to or completely different from the real world. Applications of virtual reality can include entertainment and educational purposes. Other, distinct types of VR style technology include augmented reality and mixed reality.

Reality is the sum or aggregate of all that is real or existent within a system, as opposed to that which is only imaginary. The term is also used to refer to the ontological status of things, indicating their existence. In physical terms, reality is the totality of a system, known and unknown. Philosophical questions about the nature of reality or existence or being are considered under the rubric of ontology, which is a major branch of metaphysics in the Western philosophical tradition. Ontological questions also feature in diverse branches of philosophy, including the philosophy of science, philosophy of religion, philosophy of mathematics, and philosophical logic. These include questions about whether only physical objects are real, whether reality is fundamentally immaterial, whether hypothetical unobservable entities posited by scientific theories exist, whether God exists, whether numbers and other abstract objects exist, and whether possible worlds exist.

Augmented reality View of the real world with computer-generated supplementary features

Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory. AR can be defined as a system that fulfills three basic features: a combination of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects. The overlaid sensory information can be constructive, or destructive. This experience is seamlessly interwoven with the physical world such that it is perceived as an immersive aspect of the real environment. In this way, augmented reality alters one's ongoing perception of a real-world environment, whereas virtual reality completely replaces the user's real-world environment with a simulated one. Augmented reality is related to two largely synonymous terms: mixed reality and computer-mediated reality.

Head-mounted display device used in virtual reality systems

A head-mounted display (HMD) is a display device, worn on the head or as part of a helmet, that has a small display optic in front of one or each eye. An HMD has many uses including gaming, aviation, engineering, and medicine. Head-mounted Displays are the primary components of virtual reality headsets. There is also an optical head-mounted display (OHMD), which is a wearable display that can reflect projected images and allows a user to see through it.

Virtual reality therapy (VRT), also known as virtual reality immersion therapy (VRIT), simulation for therapy (SFT), virtual reality exposure therapy (VRET), and computerized CBT (CCBT), is the use of virtual reality technology for psychological or occupational therapy and in affecting virtual rehabilitation. Patients receiving virtual reality therapy navigate through digitally created environments and complete specially designed tasks often tailored to treat a specific ailment. Technology can range from a simple PC and keyboard setup, to a modern virtual reality headset. It is widely used as an alternative form of exposure therapy, in which patients interact with harmless virtual representations of traumatic stimuli in order to reduce fear responses. It has proven to be especially effective at treating PTSD. Virtual reality therapy has also been used to help stroke patients regain muscle control, to treat other disorders such as body dysmorphia, and to improve social skills in those diagnosed with autism.

Digital Life is a research and educational program about radically rethinking of the human-computer interactive experience. It integrates digital world and physical world. It makes interfaces more responsive and proactive

The virtuality continuum is a continuous scale ranging between the completely virtual, a virtuality, and the completely real, reality. The reality–virtuality continuum therefore encompasses all possible variations and compositions of real and virtual objects. It has been described as a concept in new media and computer science, but in fact it could be considered a matter of anthropology. The concept was first introduced by Paul Milgram.

In computing, 3D interaction is a form of human-machine interaction where users are able to move and perform interaction in 3D space. Both human and machine process information where the physical position of elements in the 3D space is relevant.

Immersion (virtual reality) perception of being physically present in a non-physical world

Immersion into virtual reality (VR) is a perception of being physically present in a non-physical world. The perception is created by surrounding the user of the VR system in images, sound or other stimuli that provide an engrossing total environment.

A projection augmented model is an element sometimes employed in virtual reality systems. It consists of a physical three-dimensional model onto which a computer image is projected to create a realistic looking object. Importantly, the physical model is the same geometric shape as the object that the PA model depicts.

Immersive technology

Immersive technology refers to technology that attempts to emulate a physical world through the means of a digital or simulated world by creating a surrounding sensory feeling, thereby creating a sense of immersion. Immersive technology enables mixed reality; which is a combination of Virtual reality and Augmented reality or a combination of physical and digital. in some uses, the term "immersive computing" is effectively synonymous with mixed reality as user interface.

Virtual reality sickness occurs when exposure to a virtual environment causes symptoms that are similar to motion sickness symptoms. The most common symptoms are general discomfort, headache, stomach awareness, nausea, vomiting, pallor, sweating, fatigue, drowsiness, disorientation, and apathy. Other symptoms include postural instability and retching. Virtual reality sickness is different from motion sickness in that it can be caused by the visually-induced perception of self-motion; real self-motion is not needed. It is also different from simulator sickness; non-virtual reality simulator sickness tends to be characterized by oculomotor disturbances, whereas virtual reality sickness tends to be characterized by disorientation.

Visuo-haptic mixed reality (VHMR) is a branch of mixed reality that has the ability of merging visual and tactile perceptions of both virtual and real objects with a collocated approach. The first known system to overlay augmented haptic perceptions on direct views of the real world is the Virtual Fixtures system developed in 1992 at the US Air Force Research Laboratories. Like any emerging technology, the development of the VHMR systems is accompanied by challenges that, in this case, deal with the efforts to enhance the multi-modal human perception with the user-computer interface and interaction devices at the moment available. Visuo-haptic mixed reality (VHMR) consists of adding to a real scene the ability to see and touch virtual objects. It requires the use of see-through display technology for visually mixing real and virtual objects and haptic devices necessary to provide haptic stimuli to the user while interacting with the virtual objects. A VHMR setup allows the user to perceive visual and kinesthetic stimuli in a co-located manner, i.e., the user can see and touch virtual objects at the same spatial location. This setup overcomes the limits of the traditional one, i.e, display and haptic device, because the visuo-haptic co-location of the user's hand and a virtual tool improve the sensory integration of multimodal cues and makes the interaction more natural. But it also comes with technological challenges in order to improve the naturalness of the perceptual experience.

Windows Mixed Reality mixed reality computing platform by Microsoft

Windows Mixed Reality is a mixed reality platform introduced as part of the Windows 10 operating system, which provides holographic and mixed reality experiences with compatible head-mounted displays.

Extended reality (XR) is a term referring to all real-and-virtual combined environments and human-machine interactions generated by computer technology and wearables. It includes representative forms such as augmented reality (AR), mixed reality (MR) and virtual reality (VR) and the areas interpolated among them. The levels of virtuality range from partially sensory inputs to immersive virtuality, also called VR.

Industrial augmented reality Descrive the application of Augmented Reality in industry

Industrial augmented reality (IAR) is related to the application of augmented reality (AR) to support an industrial process. The use of IAR dates back to the 1990s with the work of Thomas Caudell and David Mizell about the application of AR at Boeing. Since then several applications of this technique over the years have been proposed showing its potential in supporting some industrial processes. Although there have been several advances in technology, IAR is still considered to be at an infant developmental stage.

X Reality is defined as: a form of “mixed reality environment that comes from the fusion (union) of ... ubiquitous sensor/actuator networks and shared online virtual worlds....”. It encompasses a wide spectrum of hardware and software, including sensory interfaces, applications, and infrastructures, that enable content creation for virtual reality (VR), mixed reality (MR), augmented reality (AR), cinematic reality (CR). With these tools, users generate new forms of reality by bringing digital objects into the physical world and bringing physical world objects into the digital world.

Virtual reality (VR) is a computer application which allows users to experience immersive, three dimensional visual and audio simulations. According to Pinho (2004), virtual reality is characterized by immersion in the 3D world, interaction with virtual objects, and involvement in exploring the virtual environment. These facets of virtual reality have many applications within the primary education sphere in enhancing student learning, increasing engagement, and creating new opportunities for addressing learning preferences.

Virtual reality applications

Virtual reality applications are applications that make use of virtual reality (VR). VR is an immersive sensory experience that digitally simulates a remote environment. Applications have been developed in a variety of domains, such as education, architectural and urban design, digital marketing and activism, engineering and robotics, entertainment, fine arts, healthcare and clinical therapies, heritage and archaeology, occupational safety, social science and psychology.

The Augmented Reality Sandtable (ARES) is an interactive, digital sand table that uses augmented reality (AR) technology to create a 3D battlespace map. It was developed by the Human Research and Engineering Directorate (HRED) at the Army Research Laboratory (ARL) to combine the positive aspects of traditional military sand tables with the latest digital technologies to better support soldier training and offer new possibilities of learning. It uses a projector to display a topographical map on top of the sand in a regular sandbox as well as a motion sensor that keeps track of changes in the layout of the sand to appropriately adjust the computer-generated terrain display.


  1. R. Freeman, A. Steed and B. Zhou, Rapid Scene Modelling, Registration and Specification for Mixed Reality Systems Archived 2007-02-06 at the Wayback Machine Proceedings of ACM Virtual Reality Software and Technology, pp. 147-150, Monterey, California, November 2005.
  2. 1 2 P. Milgram and A. F. Kishino (1994). "Taxonomy of Mixed Reality Visual Displays". IEICE Transactions on Information and Systems. pp. 1321–1329. Retrieved 2013-10-17.
  3. Rosenberg, Louis B. (1992). "The Use of Virtual Fixtures As Perceptual Overlays to Enhance Operator Performance in Remote Environments". Technical Report AL-TR-0089, USAF Armstrong Laboratory, Wright-Patterson AFB OH, 1992.
  4. L. B. Rosenberg. The Use of Virtual Fixtures As Perceptual Overlays to Enhance Operator Performance in Remote Environments. Technical Report AL-TR-0089, USAF Armstrong Laboratory, Wright-Patterson AFB OH, 1992.
  5. Rosenberg, Louis B. (1993). "Virtual fixtures as tools to enhance operator performance in telepresence environments". Telemanipulator Technology and Space Telerobotics. 2057: 10–21. Bibcode:1993SPIE.2057...10R. doi:10.1117/12.164901.
  6. Steve Mann, "Campus Canada", ISSN 0823-4531, p55 Feb-Mar 1985, pp58-59 Apr-May 1986, p72 Sep-Oct 1986
  7. Impulse, Volume 12, Number 2, 1985
  8. Quantigraphic camera promises HDR eyesight from Father of AR, by Chris Davies, SlashGear, Sep 12th 2012
  9. IEEE Technology & Society 31(3)
  10. Through the Glass, Lightly, IEEE Technology & Society, Volume 31, Number 3, Fall 2012, pages 10-14
  11. Mann, S., & Fung, J. (2001). Video orbits on EyeTap devices for deliberately diminished reality or altering the visual perception of rigid planar patches of a real-world scene. Proceedings of the Second IEEE International Symposium on Mixed Reality, pp 48-55, March 14–15, 2001.
  12. 关于智能眼镜 (About Smart Glasses), 36KR, 2016-01-09
  13. J. van Kokswijk, Hum@n, Telecoms & Internet as Interface to Interreality Archived 2007-09-26 at the Wayback Machine (Bergboek, The Netherlands, 2003).
  14. V. Gintautas, and A. W. Hubler, Experimental evidence for mixed reality states in an interreality system Phys. Rev. E 75, 057201 (2007).
  15. LIV. "Mixed Reality Studio - The LIV Cube". LIV. Retrieved 2017-04-08.
  16. P. Milgram, and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321–1329, 1994.
  17. "The Dive Home Page". web.archive.org. 30 June 2012.
  18. "Introduction - Teleimmersion Lab". UC Berkeley.
  19. Inc., VirTra,. "VirTra's Police Training Simulators Chosen by Three of the Largest U.S. Law Enforcement Departments". GlobeNewswire News Room. Retrieved 2018-08-22.CS1 maint: extra punctuation (link)
  20. "How do police use VR? Very well | Police Foundation". www.policefoundation.org. 2017-08-14. Retrieved 2018-08-22.
  21. Patton, Debbie; Marusich, Laura (2015-03-09). 2015 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision. pp. 145–150. doi:10.1109/COGSIMA.2015.7108190. ISBN   978-1-4799-8015-4.
  22. Eagen, Andrew (June 2017). "Expanding Simulations as a Means of Tactical Training with Multinational Partners" (PDF). A thesis presented to the Faculty of the U.S. Army Command and General Staff College.[ dead link ]
  23. Bukhari, Hatim; Andreatta, Pamela; Goldiez, Brian; Rabelo, Luis (2017-01-01). "A Framework for Determining the Return on Investment of Simulation-Based Training in Health Care". INQUIRY: The Journal of Health Care Organization, Provision, and Financing. 54: 0046958016687176. doi:10.1177/0046958016687176. ISSN   0046-9580. PMC   5798742 . PMID   28133988.
  24. Smith, Roger (2010-02-01). "The Long History of Gaming in Military Training". Simulation & Gaming. 41 (1): 6–19. doi:10.1177/1046878109334330. ISSN   1046-8781.
  25. Shufelt, Jr., J.W. (2006) A Vision for Future Virtual Training. In Virtual Media for Military Applications (pp. KN2-1 – KN2-12). Meeting Proceedings RTO-MP-HFM-136, Keynote 2. Neuilly-sur-Seine, France: RTO. Available from: http://www.rto.nato.int/abstracts.asp {{Webarchive|url=https://web.archive.org/web/20070613170605/http://www.rto.nato.int/Abstracts.asp |date=2007-06-13 }}
  26. "STAND-TO!". www.army.mil. Retrieved 2018-08-22.
  27. "Augmented reality may revolutionize Army training | U.S. Army Research Laboratory". www.arl.army.mil. Retrieved 2018-08-22.
  28. "Army Shoots for Single Synthetic Training Environment". GovTechWorks. 2015-11-17. Retrieved 2018-08-22.
  29. Sena, Pete. "How The Growth Of Mixed Reality Will Change Communication, Collaboration And The Future Of The Workplace". TechCrunch. Retrieved 2017-05-16.
  30. The Manufacturer. "Manufacturers are successfully using mixed reality today". www.themanufacturer.com.
  31. "Potential of Blockchain Technology as Protocol of Universal Virtual Reality". TurboFuture. Retrieved 2018-10-18.
  32. Shirazi, Sina. "Pairing Augmented Reality with Virtual Reality".Cite journal requires |journal= (help)
  33. "Taipei hits highs in Medica 2017". healthcare-in-europe.com. Retrieved 2019-04-05.

Further reading

Commons-logo.svg Media related to Mixed reality at Wikimedia Commons