Foveated rendering

Last updated

Foveated rendering is a rendering technique which uses an eye tracker integrated with a virtual reality headset to reduce the rendering workload by greatly reducing the image quality in the peripheral vision (outside of the zone gazed by the fovea). [1] [2]

Contents

A less sophisticated variant called fixed foveated rendering doesn't utilise eye tracking and instead assumes a fixed focal point. [3] [4]

History

Research into foveated rendering dates back at least to 1991. [5]

At Tech Crunch Disrupt SF 2014, Fove unveiled a headset featuring foveated rendering. [6] This was followed by a successful kickstarter in May 2015. [7]

At CES 2016, SensoMotoric Instruments (SMI) demoed a new 250 Hz eye tracking system and a working foveated rendering solution. It resulted from a partnership with camera sensor manufacturer Omnivision who provided the camera hardware for the new system. [8] [9]

In July 2016, Nvidia demonstrated during SIGGRAPH a new method of foveated rendering claimed to be invisible to users. [1] [10]

In February 2017, Qualcomm announced their Snapdragon 835 Virtual Reality Development Kit (VRDK) which includes foveated rendering support called Adreno Foveation. [11] [12]

During CES 2019 on January 7 HTC announced an upcoming virtual reality headset called Vive Pro Eye featuring eye-tracking and support for foveated rendering. [13] [14]

In December 2019, Facebook's Oculus Quest SDK gave developers access to dynamic fixed foveated rendering, allowing the variation in level of detail to be changed on the fly via an API. [15]

On January 4, 2022, Sony announced that their follow-up to PlayStation VR will include eye tracking and foveated rendering. [16]

On June 5, 2023, Apple announced that the Apple Vision Pro extended reality headset includes dynamic foveated rendering. [17]

Use

According to chief scientist Michael Abrash at Oculus, utilising foveated rendering in conjunction with sparse rendering and deep learning image reconstruction has the potential to require an order of magnitude fewer pixels to be rendered in comparison to a full image. [18] Later, these results have been demonstrated and published. [19]


Eye-tracked foveated rendering was first demonstrated [20] in a product in Sony PlayStation VR 2 headset.

See also

Related Research Articles

<span class="mw-page-title-main">Virtual reality</span> Computer-simulated experience

Virtual reality (VR) is a simulated experience that employs 3D near-eye displays and pose tracking to give the user an immersive feel of a virtual world. Applications of virtual reality include entertainment, education and business. VR is one of the key technologies in the reality-virtuality continuum. As such, it is different from other digital visualization solutions, such as augmented virtuality and augmented reality.

<span class="mw-page-title-main">Foveated imaging</span>

Foveated imaging is a digital image processing technique in which the image resolution, or amount of detail, varies across the image according to one or more "fixation points". A fixation point indicates the highest resolution region of the image and corresponds to the center of the eye's retina, the fovea.

<span class="mw-page-title-main">SensoMotoric Instruments</span>

SensoMotoric Instruments (SMI) was a German provider of dedicated computer vision applications with a major focus on eye-tracking technology. SMI was founded in 1991 as a spin-off from academic and medical research at the Free University of Berlin. The company has its headquarters in Teltow near Berlin, Germany, offices in Boston, Massachusetts and San Francisco, California, in the United States, and a worldwide distributor and partner network.

<span class="mw-page-title-main">HTC Vive</span> Virtual reality headset

HTC Vive is a line of virtual and mixed reality headsets produced by HTC Corporation. The brand currently encompasses headsets designed for use with personal computers as well as standalone headsets such as the Vive Focus line, Vive Flow glasses, and the Vive Elite XR mixed reality headset.

Oculus Touch is a line of motion controller systems used by Meta Platforms virtual reality headsets. The controller was first introduced in 2016 as a standalone accessory for the Oculus Rift CV1, and began to be bundled with the headset and all future Oculus products beginning in July 2017. Since their original release, Touch controllers have undergone revisions for later generations of Oculus/Meta hardware, including a switch to inside-out tracking, and other design changes.

FOVE is the first virtual reality headset to utilize built-in eye tracking technology. It was created by a Tokyo-based startup founded by Yuka Kojima (CEO) and Lochlainn Wilson (CTO). Announced in 2014, FOVE's technology uses infrared sensors within the headset to accurately track the user's pupils and eye movements with low latency, allowing the user to target and interact with objects by making eye contact with them.

<span class="mw-page-title-main">Virtual reality headset</span> Head-mounted device that provides virtual reality for the wearer

A virtual reality headset is a head-mounted device that uses 3D near-eye displays and positional tracking to provide a virtual reality environment for the user. VR headsets are widely used with VR video games, but they are also used in other applications, including simulators and trainers. VR headsets typically include a stereoscopic display, stereo sound, and sensors like accelerometers and gyroscopes for tracking the pose of the user's head to match the orientation of the virtual camera with the user's eye positions in the real world.

<i>Serious Sam VR: The Last Hope</i> 2017 video game

Serious Sam VR: The Last Hope is a 2017 first-person shooter game for virtual reality (VR) developed by Croteam VR and published by Devolver Digital. One or two players fight waves of enemies, including bosses, across five thematic planets. The enemies approach from a 180° field and the player can use dual-wielded weapons while moving across a limited space. To develop The Last Hope and experiment with other VR implementations, Croteam VR was established as a specialised division of Croteam. Devolver Digital announced the game at E3 in June 2016 and launched it in early access that October. After several updates, it was released in September 2017 for Windows with compatibility for the HTC Vive and Oculus Rift VR headsets. The Last Hope received mostly positive reviews, with post-release reception lauding the game's gameplay and visuals while criticising difficulty spikes and issues with the online multiplayer mode.

The Snapdragon Qualcomm VR820 is a virtual reality reference platform that was released in Q4 2016, with the first commercial devices based on the platform available shortly thereafter.

OpenXR is an open-source, royalty-free standard for access to virtual reality and augmented reality platforms and devices. It is developed by a working group managed by the Khronos Group consortium. OpenXR was announced by the Khronos Group on February 27, 2017, during GDC 2017. A provisional version of the standard was released on March 18, 2019, to enable developers and implementers to provide feedback on it. On July 29, 2019, OpenXR 1.0 was released to the public by Khronos Group at SIGGRAPH 2019.

<span class="mw-page-title-main">Pimax</span> Chinese virtual reality company

Pimax is a technology company specializing in virtual reality hardware products.

<span class="mw-page-title-main">Oculus Go</span> Untethered virtual reality headset by Oculus VR

The Oculus Go is a standalone virtual reality headset developed by Meta Reality Labs in partnership with Qualcomm and Xiaomi. It is in the first generation of Facebook Technologies' virtual reality headsets, and the company's first device in the category of standalone VR headsets, which was a new category at the time of the Go's release. The Oculus Go was unveiled on October 11, 2017 during the Oculus Connect developer conference, and released on May 1, 2018. Xiaomi launched their own version of the headset in China as the Mi VR Standalone on May 31, 2018.

<span class="mw-page-title-main">Oculus Quest</span> Virtual reality headset

The first-generation Oculus Quest is a discontinued virtual reality headset developed by Oculus, a brand of Facebook Inc., and released on May 21, 2019. Similar to its predecessor, Oculus Go, it is a standalone device, that can run games and software wirelessly under an Android-based operating system. It supports positional tracking with six degrees of freedom, using internal sensors and an array of cameras in the front of the headset rather than external sensors. The cameras are also used as part of the safety feature "Passthrough", which shows a view from the cameras when the user exits their designated boundary area known as "Guardian". A later software update added "Oculus Link", a feature that allows the Quest to be connected to a computer via USB, enabling use with Oculus Rift-compatible software and games.

<i>NeosVR</i> Virtual reality application

NeosVR is a free-to-play, massively multiplayer online, virtual reality application created by Solirax. It was released for free on Microsoft Windows via Steam on May 4, 2018, with support for several VR headsets.

<span class="mw-page-title-main">Quest 2</span> Virtual reality headset

The Quest 2 is a virtual reality (VR) headset developed by Reality Labs, a division of Meta Platforms. It was unveiled on September 16, 2020, and released on October 13, 2020 as the Oculus Quest 2. It was then rebranded as the Meta Quest 2 in 2022, as part of a company-wide phase-out of the Oculus brand following the rebranding of Facebook, Inc. as Meta.

<span class="mw-page-title-main">PlayStation VR2</span> Virtual reality headset developed by Sony Interactive Entertainment

The PlayStation VR2 is a virtual reality headset for the PC and PlayStation 5 home video game console developed by Sony Interactive Entertainment released on February 22, 2023.

The Meta Quest Pro is a mixed reality (MR) headset developed by Reality Labs, a division of Meta Platforms.

VIVE XR Elite is an extended reality (XR) headset developed and manufactured by HTC. Released in February 2023, the headset combines virtual reality (VR) and mixed reality (MR) functionalities.

<i>Synth Riders</i> Virtual reality rhythm game

Synth Riders is a virtual reality rhythm game developed and published by Kluge Interactive. It involves players touching spheres ('notes'), tracing lines ('rails'), and avoiding obstacles that move towards them.

References

  1. 1 2 Parrish, Kevin (2016-07-22). "Nvidia plans to prove that new method improves image quality in virtual reality". Digital Trends . Retrieved 2017-02-02.
  2. ↑ "Understanding Foveated Rendering". Sensics. 2016-04-11. Archived from the original on 2017-12-01. Retrieved 2017-02-04.
  3. ↑ Carbotte, Kevin (30 March 2018). "What Is Fixed Foveated Rendering, And Why Does It Matter?". Tom's Hardware. Retrieved 9 September 2019.
  4. ↑ Orland, Kyle (21 March 2016). "How Valve got passable VR running on a four-year-old graphics card". Ars Technica. Retrieved 9 September 2019.
  5. ↑ Wang, Lili; Shi, Xuehuai; Liu, Yi (2023). "Foveated rendering: A state-of-the-art survey". Computational Visual Media. 9 (2): 195–228. arXiv: 2211.07969 . doi:10.1007/s41095-022-0306-4. S2CID   253523241.
  6. ↑ "FOVE Uses Eye Tracking To Make Virtual Reality More Immersive". TechCrunch. 10 September 2014. Retrieved 2019-02-06.
  7. ↑ "FOVE: The World's First Eye Tracking Virtual Reality Headset". Kickstarter. Retrieved 2019-02-06.
  8. ↑ Mason, Will (2016-01-15). "SMI's 250Hz Eye Tracking and Foveated Rendering Are For Real, and the Cost May Surprise You". UploadVR . Retrieved 2017-02-02.
  9. ↑ Mason, Will (2016-01-15). "SMI's 250Hz Eye Tracking and Foveated Rendering Are For Real, and the Cost May Surprise You". UploadVR. Retrieved 2020-06-18.
  10. ↑ "NVIDIA Partners with SMI on Innovative Rendering Technique That Improves VR". Nvidia. 2016-01-21. Retrieved 2017-02-02.
  11. ↑ "Qualcomm Introduces Snapdragon 835 Virtual Reality Development Kit". Qualcomm. 2017-02-23. Retrieved 2021-06-01.
  12. ↑ "Foveation in Game Engines". vr.tobii.com. Retrieved 2021-06-01.
  13. ↑ Statt, Nick (2019-01-07). "HTC announces new Vive Pro Eye virtual reality headset with native eye tracking". The Verge. Retrieved 2019-01-14.
  14. ↑ Meng, Xiaoxu; Du, Ruofei; Zwicker, Matthias; Varshney, Amitabh (2018-07-01), "Kernel Foveated Rendering", Proceedings of the ACM on Computer Graphics and Interactive Techniques, 1: 1–20, doi:10.1145/3203199, S2CID   4899582 , retrieved 2018-07-01
  15. ↑ "Oculus Quest gets dynamic fixed foveated rendering". VentureBeat. 2019-12-22. Retrieved 2020-01-21.
  16. ↑ "PlayStation VR2 and PlayStation VR2 Sense controller: The next generation of VR gaming on PS5". 4 January 2022.
  17. ↑ "Discover visionOS". 7 June 2023.
  18. ↑ Oculus (2018-09-26), Oculus Connect 5 | Keynote Day 01 , retrieved 2018-09-30
  19. ↑ Kaplanyan, Anton (2020-05-15). "DeepFovea: AR/VR rendering, inspired by human vision". YouTube . Retrieved 2020-05-15.
  20. ↑ "PSVR 2 Foveated Rendering Provides 3.6x Faster Performance - Unity". UploadVR. 2022-04-27. Retrieved 2022-04-27.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.