Foveated imaging

Last updated
16:1 compression. Foveated image with fixation point at Stephen F. Austin statue. Texas state cemetery foveated1.png
16:1 compression. Foveated image with fixation point at Stephen F. Austin statue.

Foveated imaging is a digital image processing technique in which the image resolution, or amount of detail, varies across the image according to one or more "fixation points". A fixation point indicates the highest resolution region of the image and corresponds to the center of the eye's retina, the fovea.

Contents

The location of a fixation point may be specified in many ways. For example, when viewing an image on a computer monitor, one may specify a fixation using a pointing device, like a computer mouse. Eye trackers which precisely measure the eye's position and movement are also commonly used to determine fixation points in perception experiments. [1] [2] When the display is manipulated with the use of an eye tracker, this is known as a gaze contingent display. [3] Fixations may also be determined automatically using computer algorithms. [4] [5]

Some common applications of foveated imaging include imaging sensor hardware [6] and image compression. [7] For descriptions of these and other applications, see the list below. Miniaturized foveated imaging systems can be realized by high-resolution 3D printing of multi-lens objectives directly on a CMOS (Complementary metal-oxide-semiconductor) chip. [8]

Foveated imaging is also commonly referred to as space variant imaging or gaze contingent imaging.

Applications

Foveated imaging for progressive transmission ProgressiveTransmission.gif
Foveated imaging for progressive transmission

Compression

Contrast sensitivity falls off dramatically as one moves from the center of the retina to the periphery. [9] [10] In lossy image compression, one may take advantage of this fact in order to compactly encode images. If one knows the viewer's approximate point of gaze, one may reduce the amount of information contained in the image as the distance from the point of gaze increases. Because the fall-off in the eye's resolution is dramatic, the potential reduction in display information can be substantial. Also, foveation encoding may be applied to the image before other types of image compression are applied and therefore can result in a multiplicative reduction.

Foveated sensors

Foveated sensors are multiresolution hardware devices that allow image data to be collected with higher resolution concentrated at a fixation point. An advantage to using foveated sensor hardware is that the image collection and encoding can occur much faster than in a system that post-processes a high resolution image in software. [11]

Simulation

Foveated imaging has been used to simulate visual fields with arbitrary spatial resolution. For example, one may present video containing a blurred region representing a scotoma. By using an eye-tracker and holding the blurred region fixed relative to the viewer's gaze, the viewer will have a visual experience similar to that of a person with an actual scotoma.

Video gaming

Foveated rendering is a technique which uses an eye tracker integrated with a virtual reality headset to reduce the rendering workload by greatly reducing the image quality in the peripheral vision (outside of the zone gazed by the fovea). [12]

At the CES 2016, SensoMotoric Instruments (SMI) demoed a new 250 Hz eye tracking system and a working foveated rendering solution. It resulted from a partnership with camera sensor manufacturer Omnivision who provided the camera hardware for the new system. [13]

The Apple Vision Pro mixed reality headset features dynamic foveated rendering provided by its visionOS operating system. [14] [15]

Quality assessment

Foveated imaging may be useful in providing a subjective image quality measure. [16] Traditional image quality measures, such as peak signal-to-noise ratio, are typically performed on fixed resolution images and do not take into account some aspects of the human visual system, like the change in spatial resolution across the retina. A foveated quality index may therefore more accurately determine image quality as perceived by humans.

Image database retrieval

In databases that contain very high resolution images, such as a satellite image database, it may be desirable to interactively retrieve images in order to reduce retrieval time. Foveated imaging allows one to scan low resolution images and retrieve only high resolution portions as they are needed. This is sometimes called progressive transmission.

Example images

See also

Related Research Articles

<span class="mw-page-title-main">Saccade</span> Eye movement

A saccade is a quick, simultaneous movement of both eyes between two or more phases of fixation in the same direction. In contrast, in smooth-pursuit movements, the eyes move smoothly instead of in jumps. The phenomenon can be associated with a shift in frequency of an emitted signal or a movement of a body part or device. Controlled cortically by the frontal eye fields (FEF), or subcortically by the superior colliculus, saccades serve as a mechanism for fixation, rapid eye movement, and the fast phase of optokinetic nystagmus. The word appears to have been coined in the 1880s by French ophthalmologist Émile Javal, who used a mirror on one side of a page to observe eye movement in silent reading, and found that it involves a succession of discontinuous individual movements.

<span class="mw-page-title-main">Peripheral vision</span> Area of ones field of vision outside of the point of fixation

Peripheral vision, or indirect vision, is vision as it occurs outside the point of fixation, i.e. away from the center of gaze or, when viewed at large angles, in the "corner of one's eye". The vast majority of the area in the visual field is included in the notion of peripheral vision. "Far peripheral" vision refers to the area at the edges of the visual field, "mid-peripheral" vision refers to medium eccentricities, and "near-peripheral", sometimes referred to as "para-central" vision, exists adjacent to the center of gaze.

<span class="mw-page-title-main">Field of view</span> Extent of the observable world seen at any given moment

The field of view (FOV) is the angular extent of the observable world that is seen at any given moment. In the case of optical instruments or sensors, it is a solid angle through which a detector is sensitive to electromagnetic radiation. It is further relevant in photography.

<span class="mw-page-title-main">Scanning laser ophthalmoscopy</span>

Scanning laser ophthalmoscopy (SLO) is a method of examination of the eye. It uses the technique of confocal laser scanning microscopy for diagnostic imaging of the retina or cornea of the human eye.

<span class="mw-page-title-main">Fovea centralis</span> Small pit in the retina of the eye responsible for all central vision

The fovea centralis is a small, central pit composed of closely packed cones in the eye. It is located in the center of the macula lutea of the retina.

<span class="mw-page-title-main">Scotoma</span> Medical condition

A scotoma is an area of partial alteration in the field of vision consisting of a partially diminished or entirely degenerated visual acuity that is surrounded by a field of normal – or relatively well-preserved – vision.

The visual field is "that portion of space in which objects are visible at the same moment during steady fixation of the gaze in one direction"; in ophthalmology and neurology the emphasis is on the structure inside the visual field and it is then considered “the field of functional capacity obtained and recorded by means of perimetry”.

<span class="mw-page-title-main">Eye tracking</span> Measuring the point of gaze or motion of an eye relative to the head

Eye tracking is the process of measuring either the point of gaze or the motion of an eye relative to the head. An eye tracker is a device for measuring eye positions and eye movement. Eye trackers are used in research on the visual system, in psychology, in psycholinguistics, marketing, as an input device for human-computer interaction, and in product design. In addition, eye trackers are increasingly being used for assistive and rehabilitative applications such as controlling wheelchairs, robotic arms, and prostheses. Recently, eye tracking has been examined as a tool for the early detection of autism spectrum disorder. There are several methods for measuring eye movement, with the most popular variant using video images to extract eye position. Other methods use search coils or are based on the electrooculogram.

<span class="mw-page-title-main">Eye movement</span> Movement of the eyes

Eye movement includes the voluntary or involuntary movement of the eyes. Eye movements are used by a number of organisms to fixate, inspect and track visual objects of interests. A special type of eye movement, rapid eye movement, occurs during REM sleep.

<span class="mw-page-title-main">Fixation (visual)</span> Maintaining ones gaze on a single location

Fixation or visual fixation is the maintaining of the gaze on a single location. An animal can exhibit visual fixation if it possess a fovea in the anatomy of their eye. The fovea is typically located at the center of the retina and is the point of clearest vision. The species in which fixational eye movement has been verified thus far include humans, primates, cats, rabbits, turtles, salamanders, and owls. Regular eye movement alternates between saccades and visual fixations, the notable exception being in smooth pursuit, controlled by a different neural substrate that appears to have developed for hunting prey. The term "fixation" can either be used to refer to the point in time and space of focus or the act of fixating. Fixation, in the act of fixating, is the point between any two saccades, during which the eyes are relatively stationary and virtually all visual input occurs. In the absence of retinal jitter, a laboratory condition known as retinal stabilization, perceptions tend to rapidly fade away. To maintain visibility, the nervous system carries out a procedure called fixational eye movement, which continuously stimulates neurons in the early visual areas of the brain responding to transient stimuli. There are three categories of fixational eye movement: microsaccades, ocular drifts, and ocular microtremor. At small amplitudes the boundaries between categories become unclear, particularly between drift and tremor.

Within computer technology, the gaze-contingency paradigm is a general term for techniques allowing a computer screen display to change in function depending on where the viewer is looking. Gaze-contingent techniques are part of the eye movement field of study in psychology.

<span class="mw-page-title-main">Parafovea</span>

Parafovea or the parafoveal belt is a region in the retina that circumscribes the fovea and is part of the macula lutea. It is circumscribed by the perifovea.

<span class="mw-page-title-main">SensoMotoric Instruments</span>

SensoMotoric Instruments (SMI) was a German provider of dedicated computer vision applications with a major focus on eye-tracking technology. SMI was founded in 1991 as a spin-off from academic and medical research at the Free University of Berlin. The company has its headquarters in Teltow near Berlin, Germany, offices in Boston, Massachusetts and San Francisco, California, in the United States, and a worldwide distributor and partner network.

Microperimetry, sometimes called fundus-controlled perimetry, is a type of visual field test which uses one of several technologies to create a "retinal sensitivity map" of the quantity of light perceived in specific parts of the retina in people who have lost the ability to fixate on an object or light source. The main difference with traditional perimetry instruments is that, microperimetry includes a system to image the retina and an eye tracker to compensate eye movements during visual field testing.

Visual computing is a generic term for all computer science disciplines dealing with images and 3D models, such as computer graphics, image processing, visualization, computer vision, virtual and augmented reality and video processing. Visual computing also includes aspects of pattern recognition, human computer interaction, machine learning and digital libraries. The core challenges are the acquisition, processing, analysis and rendering of visual information. Application areas include industrial quality control, medical image processing and visualization, surveying, robotics, multimedia systems, virtual heritage, special effects in movies and television, and computer games.

The FourPrism Dioptre Reflex Test is an objective, non-dissociative test used to prove the alignment of both eyes by assessing motor fusion. Through the use of a 4 dioptre base out prism, diplopia is induced which is the driving force for the eyes to change fixation and therefore re-gain bifoveal fixation meaning, they overcome that amount of power.

Retinal birefringence scanning (RBS) is a method for detecting the central fixation of the eye. The method can be used in pediatric ophthalmology for screening purposes. By simultaneously measuring the central fixation of both eyes, small- and large-angle strabismus can be detected. The method is not invasive and requires little cooperation by the patient, so it can be used for detecting strabismus in young children. The method provides a reliable detection of strabismus and has also been used for detecting certain kinds of amblyopia. RBS uses the human eye's birefringent properties to identify the position of the fovea and the direction of gaze, and thereby to measure any binocular misalignment.

<span class="mw-page-title-main">Virtual reality headset</span> Head-mounted device that provides virtual reality for the wearer

A virtual reality headset is a head-mounted device that provides a virtual reality environment for the wearer. VR headsets are widely used with VR video games but they are also used in other applications, including simulators and trainers. VR headsets typically include a stereoscopic display, stereo sound, and sensors like accelerometers and gyroscopes for tracking the pose of the user's head to match the orientation of the virtual camera with the user's eye positions in the real world.

The Snapdragon Qualcomm VR820 is a virtual reality reference platform that was anticipated for release in Q4 2016, with the first commercial devices based on the platform expected to be available shortly thereafter.

Foveated rendering is a rendering technique which uses an eye tracker integrated with a virtual reality headset to reduce the rendering workload by greatly reducing the image quality in the peripheral vision.

References

  1. ↑ McConkie G W and Rayner K (1975) The span of the effective stimulus during a fixation in reading, Perception & Psychophysics, 17, 578–86
  2. ↑ Loschky, L.C. & Wolverton, G.S. (2007). How Late Can You Update Gaze-contingent Multi-resolutional Displays Without Detection? ACM Transactions on Multimedia Computing, Communications and Applications, 3(4):25, 1-10.
  3. ↑ Duchowski, A. T., Cournia, N., and Murphy, H. 2004. Gaze-Contingent displays: A review. Cyberpsychol. Behav. 7, 6, 621--634.
  4. ↑ Z. Wang, L. Lu and A. C. Bovik, "Foveation scalable video coding with automatic fixation selection," IEEE Trans. on Image Processing, Vol: 12 No: 2, February 2003.
  5. ↑ R. G. Raj, W. S. Geisler, R. A. Frazor, A. C. Bovik, "Contrast statistics for foveated visual systems: Fixation selection by minimizing contrast entropy" Journal of the Optical Society of America.
  6. ↑ J.A. Boluda, F. Pardo, T. Kayser, J.J. P'erez, and J. Pelechano. A new foveated space-variant camera for robotic applications. In IEEE, International Conference on Electronics Circuits And Systems, ICECS'96, Rodos, Greece, October 1996.
  7. ↑ Geisler, W.S. and Perry, J.S. (1998) A real-time foveated multi-resolution system for low-bandwidth video communication. In B. Rogowitz and T. Pappas (Eds.), Human Vision and Electronic Imaging, SPIE Proceedings, 3299, 294-305.
  8. ↑ Thiele, Simon; Arzenbacher, Kathrin; Gissibl, Timo; Giessen, Harald; Herkommer, Alois M. (2017-02-03). "3D-printed eagle eye: Compound microlens system for foveated imaging". Science Advances. 3 (2): e1602655. Bibcode:2017SciA....3E2655T. doi:10.1126/sciadv.1602655. ISSN   2375-2548. PMC   5310822 . PMID   28246646.
  9. ↑ Wandell, Brian A. (1995) Foundations of Vision. ISBN   0-87893-853-2 . p.236
  10. ↑ Barghout-Stein, Lauren. On differences between peripheral and foveal pattern masking. Diss. University of California, Berkeley, 1999.
  11. ↑ Marc Bolduc, Martin D. Levine. A real-time foveated sensor with overlapping receptive fields. June 1997, Real-Time Imaging, Volume 3 Issue 3
  12. ↑ Parrish, Kevin (2016-07-22). "Nvidia plans to prove that new method improves image quality in virtual reality". Digital Trends . Retrieved 2017-02-02.
  13. ↑ Mason, Will (2016-01-15). "SMI's 250Hz Eye Tracking and Foveated Rendering Are For Real, and the Cost May Surprise You". UploadVR . Retrieved 2017-02-02.
  14. ↑ "visionOS Overview". Apple Developer. Retrieved 2024-02-09.
  15. ↑ "📌 Official Support for visionOS Now Available". Unity Discussions. 2024-01-22. Retrieved 2024-02-09.
  16. ↑ Z. Wang, A. C. Bovik, L. Lu and J. Kouloheris, "Foveated wavelet image quality index," SPIE's 46th Annual Meeting, Proc. SPIE, Application of digital image processing XXIV, vol. 4472, July-Aug. 2001.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.