What is light field?

What is a light field display?

The light field describes the amount of energy, in visible wavelengths, traveling through every point and location in space. We see things in the world not because they exist, but because of the object’s surface properties (e.g., shiny or matte finish) and how light reflects off of the object and reaches your eyes. Surfaces reflect in countless directions, not just the one you currently see— this is why when it is light, you can see around objects and things like transparency or reflection, and why when it is dark, you can’t see any objects, but they are still there.  A Light Field Video Display (LFVD) projects objects that are undifferentiated from the real things.

A true LFVD must provide sufficient density to stimulate the eye into perceiving a real-world scene, providing for:

1. Binocular disparity without external accessories, head-mounted eyewear, or other peripherals;
2. Accurate motion parallax, occlusion and opacity throughout a determined viewing volume simultaneously for any number of viewers;
3. Visual focus through synchronous vergence, accommodation and miosis of the eye for all perceived rays of light; and
4. Converging energy wave propagation of sufficient density and resolution to exceed the visual acuity of the eye.

What is the difference between a light field and a volumetric display?

A Light Field display converges radial bundles of light in free-space from the display surface to create a holographic object. The converged bundles must incorporate multiple angular (q, j) color and intensity values for each single surface coordinate or feature. This allows for the projection of scene information that typically changes depending on the viewing angle and location, just like the real world. This (q, j) independence provides for the things that make light fields truly life-like, including reflections, refractions, etc. This is the core of a light field and the element that allows it to define how rays of light travel through space.

A volumetric display is similar to a traditional 2D display, but has the ability to show 2D pixels within a volume similar in concept to a point cloud. There are many ways these effects are achieved including time sequential volumetric slices, laser ionization, or high-speed rotating elements. However, none of these exhibit any holographic attributes, are limited to a small volume, are often dangerous to operate, are transparent, limit resolution and refresh rates, and cannot handle occlusion.

Synthetic 3D and volumetric images are now commonplace in computer games, movies and many other applications.  How is what you’re doing different?

Most applications supporting 3D views are not holographic and require a specific display (e.g., stereoscopic or VR/AR) that rely on a 2D right eye / left eye view to create the illusion of depth. Sometimes these displays are augmented with eye tracking or movement tracking in hopes of presenting images appropriate to the viewing angle for the application. However, the limited resolutions, narrow field of view, poor optical quality, lack of opacity handling (for AR), problematic motion latency, and the inability for the eye to truly focus freely about the volume.

A light field creates the full spray of light within the viewing volume that allows the eye to focus on the objects presented. No gear is required. With a light field display you can see behind objects when you move your head. Parallax is maintained, reflections and refractions behave correctly, and the brain concludes that objects are “real.” To achieve this holographic realism, each pixel must contain scene information that changes depending on the viewing angle and location— just like the real world.

Will these Immersive Experiences be distributed to the home?  How?

Lower quality experiences such as 360 video are already being delivered to the home from the network. However, in order to stream fully immersive experiences, we envision that the network itself can play a role in processing and computation. With the availability of light-field displays in the future, the network can be prepared deliver delightful immersive experiences using a multitude of displays.

IDEA will not only develop specifications for the Immersive Technologies Media Format, but it will also develop specifications to distribute ITMF over commercial networks, leveraging state-of-the-art IP networks (e.g. Cable 10G, WiFi 6, and 5G) for their speed, low-latency, and in-network compute functionalities.

What is a Scene Graph, and why is it helpful?

Scene graphs are used to structure a collection of nodes in a hierarchical “tree”. The node data can be based by vector-based graphics, point clouds, voxel maps or many other input sources.  Live photographic images can be mapped into a scene graph as well, using techniques originally drawn from photogrammetry and lumigraphs. Yes – we know this is rather complicated sounding, which is why IDEA will be producing educational seminars in order to provide background and information about these established techniques, and how the ITMF format can leverage current practices.

How is audio being handled?

Sound is an extremely important part of the immersive experience.  IDEA will leverage existing standards to represent a multi-dimensional soundfield associated with the image space, using technologies such as high-order ambisonics.  This will be accompanied by object-based audio files with associated metadata, which will allow the positioning of selected sounds in relation with the viewing position.  The ITMF framework includes the necessary position data to render this combination of ambisonic and object audio files to reinforce the immersive experience.