IllumiRoom is a Microsoft Research project that augments a television screen with images projected onto the wall and surrounding objects. The current proof-of-concept uses a Kinect sensor and video projector. The Kinect sensor captures the geometry and colors of the area of the room that surrounds the television, and the projector displays video around the television that corresponds to a video source on the television, such as a video game or movie.
IllumiRoom was first introduced at the 2013 Consumer Electronics Show. [1] At the show, Microsoft, with Samsung, showed a video presentation of the system. At CHI 2013, Microsoft presented more details of the system, including a paper written with a researcher at the University of Illinois at Urbana–Champaign. [2] [3]
The system prototype uses a wide field of view projector and a Kinect for Windows sensor. The Kinect captures the color and geometry of the room environment and the projector renders images onto the depth map acquired by the sensor.
The IllumiRoom concept is based on prior work and research using focus-plus-context screens and projection mapping. The focus-plus-context technology generally uses a high resolution screen surrounded by a lower resolution display. Microsoft's CHI 2013 research paper cites Philips' Ambilight as an example of a focus-plus-context display. In the case of IllumiRoom, the television represents the high resolution screen and the surrounding projection is the lower resolution display. The purpose of this technology is to provide the user with additional visual information in the visual periphery, both simulating and taking advantage of peripheral vision. While the center of a person's gaze is in high-resolution and is sensitive to color and detail, peripheral vision is less sensitive to color and detail, but highly sensitive to movement.
IllumiRoom combines the focus-plus-context concept with real-time projection mapping. This allows the system to be used in any room, not just one where a television is surrounded by flat, white wall.
The Kinect sensor is used to calibrate the system and projection. The projector displays a system of gray patterns and the Kinect camera reads the size of the pattern across the projection in order to map the 3D environment. Once calibrated, the Kinect sensor is no longer needed for the IllumiRoom system and can be used for gaming. [3] IllumiRoom was developed with the open-source first-person shooter Red Eclipse as prototype application. [4]
The system can currently display video game video in one of several modes. These modes require the system to have access to the game's rendering process: [3]
Without access to the game's rendering, several other projection modes are available: [3]
Although widely expected to be used in an Xbox application, the researchers have stated that the technology is, for now, only a research project and not ready for commercial use. [5]
RoomAlive, a related Microsoft Research project, also uses a depth camera and video projector in a projector-camera, or "procam" setup. It is a scalable system for dynamic, real-time interactive projection mapping in which multiple such procams can be used together in a room to generate an immersive unified projection mapping that is automatically adapted to the room environment, and which users can interact with physically. Unlike IllumiRoom, which implements focus-plus-context visual presentation centered on a television screen, RoomAlive focuses on spatial augmented reality applications. [6]
In April 2015, Microsoft released the RoomAlive Toolkit, an open-source MIT licensed [7] software development kit for calibrating a network of video projectors and Kinect sensors, which can be used to develop systems like those of the RoomAlive and IllumiRoom projects. [8] The source code is available is available in their public repository. [9]
Augmented reality (AR) is an interactive experience that combines the real world and computer-generated content. The content can span multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory. AR can be defined as a system that incorporates three basic features: a combination of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects. The overlaid sensory information can be constructive, or destructive. This experience is seamlessly interwoven with the physical world such that it is perceived as an immersive aspect of the real environment. In this way, augmented reality alters one's ongoing perception of a real-world environment, whereas virtual reality completely replaces the user's real-world environment with a simulated one.
A cave automatic virtual environment is an immersive virtual reality environment where projectors are directed to between three and six of the walls of a room-sized cube. The name is also a reference to the allegory of the Cave in Plato's Republic in which a philosopher contemplates perception, reality, and illusion.
A structured-light 3D scanner is a 3D scanning device for measuring the three-dimensional shape of an object using projected light patterns and a camera system.
Canesta was a fabless semiconductor company that was founded in April, 1999, by Cyrus Bamji, Abbas Rafii, and Nazim Kareemi.
Kinect is a line of motion sensing input devices produced by Microsoft and first released in 2010. The devices generally contain RGB cameras, and infrared projectors and detectors that map depth through either structured light or time of flight calculations, which can in turn be used to perform real-time gesture recognition and body skeletal detection, among other capabilities. They also contain microphones that can be used for speech recognition and voice control.
BigPark was a Canadian video game developer owned by Microsoft Studios.
A virtual touch screen (VTS) is a user interface system that augments virtual objects into reality either through a projector or optical display using sensors to track a person's interaction with the object. For instance, using a display and a rear projector system a person could create images that look three-dimensional and appear to float in midair. Some systems utilize an optical head-mounted display to augment the virtual objects onto the transparent display utilizing sensors to determine visual and physical interactions with the virtual objects projected.
Kinect Sports is a sports video game developed by Rare and published by Microsoft Game Studios for the Xbox 360. The game utilizes the Kinect motion-sensing peripheral and was released in North America on 4 November 2010 as a launch title for Kinect, then, a few days later, in Europe and Australia.
Kinect Adventures! is a sports video game released by Microsoft Game Studios for the Xbox 360. Released in 2010, it is a collection of five adventure and sports minigames and was developed by Good Science Studio, a subsidiary of Microsoft Game Studios. The game utilizes the Kinect motion camera and was offered as a pack-in game with the console. It was unveiled at the 2010 Electronic Entertainment Expo and went on to become the best-selling game on the Xbox 360, selling 24 million units worldwide.
PrimeSense was an Israeli 3D sensing company based in Tel Aviv. PrimeSense had offices in Israel, North America, Japan, Singapore, Korea, China and Taiwan. PrimeSense was bought by Apple Inc. for $360 million on November 24, 2013.
Projection mapping, similar to video mapping and spatial augmented reality, is a projection technique used to turn objects, often irregularly shaped, into display surfaces for video projection. The objects may be complex industrial landscapes, such as buildings, small indoor objects, or theatrical stages. Using specialized software, a two- or three-dimensional object is spatially mapped on the virtual program which mimics the real environment it is to be projected on. The software can then interact with a projector to fit any desired image onto the surface of that object. The technique is used by artists and advertisers who can add extra dimensions, optical illusions, and notions of movement onto previously static objects. The video is commonly combined with or triggered by audio to create an audiovisual narrative. In recent years the technique has also been widely used in the context of cultural heritage, as it has proved to be an excellent edutainment tool.
Kinect Sesame Street TV is a Sesame Street video game of an interactive television program for the Microsoft Windows & Xbox 360 consoles. It is based on the children's television series of the same name and is aimed at children. Because of its motion gesture features, the game requires the Xbox 360's Kinect sensor to get into action. It is developed by Microsoft Studios' Soho Productions & published by Microsoft Studios. This interactive television program was released on 18 September 2012.
Xbox is a video gaming brand created and owned by Microsoft. The brand consists of five video game consoles, as well as applications (games), streaming service Xbox Cloud Gaming, online services such as the Xbox network and Xbox Game Pass, and the development arm Xbox Game Studios. The brand was first introduced in the United States in November 2001, with the launch of the original Xbox console.
Project Digits is a Microsoft Research Project under Microsoft's computer science laboratory at the University of Cambridge; researchers from Newcastle University and University of Crete are also involved in this project. Project is led by David Kim a Microsoft Research PhD and also a PhD Student in computer science at Newcastle University. Digits is an input device which can be mounted on the wrist of human hand and it captures and displays a complete 3D graphical representation of the user's hand on screen without using any external sensing device or hand covering material like data gloves. This project aims to make gesture controlled interfaces completely hands free with greater mobility and accuracy. It allows user to interact with whatever hardware while moving from room to room or walking down the street without any line of sight connection with the hardware.
The Xbox One is a home video game console developed by Microsoft. Announced in May 2013, it is the successor to Xbox 360 and the third console in the Xbox series. It was first released in North America, parts of Europe, Australia, and South America in November 2013 and in Japan, China, and other European countries in September 2014. It is the first Xbox game console to be released in China, specifically in the Shanghai Free-Trade Zone. Microsoft marketed the device as an "all-in-one entertainment system", hence the name "Xbox One". An eighth-generation console, it mainly competed against Sony's PlayStation 4 and Nintendo's Wii U and later the Switch.
Lumitrack is a motion capture technology developed by Robert Xiao, Chris Harrison and Scott Hudson at Carnegie Mellon University. It combines projectors and sensors to provide high-fidelity motion-tracking. These types of sensors are used in video game controllers, such as Microsoft's Kinect, and in motion capture for movie and television production.
Windows Mixed Reality is a platform introduced as part of the Windows 10 and 11 operating system, which provides augmented reality and virtual reality experiences with compatible head-mounted displays.
Fru is a puzzle-platform game developed by Through Games. Fru was released for the Xbox One console on 13 July 2016.
The Augmented Reality Sandtable (ARES) is an interactive, digital sand table that uses augmented reality (AR) technology to create a 3D battlespace map. It was developed by the Human Research and Engineering Directorate (HRED) at the Army Research Laboratory (ARL) to combine the positive aspects of traditional military sand tables with the latest digital technologies to better support soldier training and offer new possibilities of learning. It uses a projector to display a topographical map on top of the sand in a regular sandbox as well as a motion sensor that keeps track of changes in the layout of the sand to appropriately adjust the computer-generated terrain display.
Red Eclipse is an open-source first-person shooter that is forked from Cube 2: Sauerbraten. Like the original Cube 2, it features multiplayer gameplay as well as in-game level editing, but with improved graphics and a focus on parkour movement. The game is free and open-source software, released under the zlib license, and developed by an open community of contributors. Its content is free, and released under a CC BY-SA license.
{{cite journal}}: Cite journal requires |journal= (help)| Main projects |
| ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MSR Labs applied research |
| ||||||||||||||
