|
|
Next-Generation ITMF Streaming Demo Recap
In May, IDEA conducted a live demonstration of the latest advance in 3D streaming, showing a greatly advanced architecture and pipeline developed by engineers from IDEA members Charter Communications, OTOY, and CableLabs.
In the demo, hundreds of 3D objects were streamed in a complete 3D scene, creating a seamless 3D experience for the viewer. Now, the viewer will be able to watch and interact with longer-form 3D content on multiple immersive displays.
Both Unreal and Unity game engines were employed as clients. This is significant since over 90% of developers and content providers for immersive displays are using these game engines to create content, making this is a major step toward bringing 3D streaming to a wider audience.
For a more detailed description of the demo and its significance, see the article in the last issue of this newsletter here:
https://www.immersivealliance.org/2022/04/06/march-newsletter-2022/.
Posted July 22, 2022
Industry News
What Caught Our Eye
Is it just me, or does it seem like news about holograms, metaverses, 3D immersion, and the like is just everywhere? From medical breakthroughs to education, politics, sports, and even the art world, things happened in the past few months that pushed the use of technology a bit further. Truly exciting times.
Here’s a quick rundown of some of the stories out there.
Medical
In “Why Medical Holography Is Rapidly Gaining Ground,” the author explains: “Holographic 3D technology renders medical professionals with advanced diagnostic capabilities and surgical planning by allowing them to slice virtual tissue, organs, and other body parts at various angles. Holomedicine advances access to percutaneous procedures without invasive needle techniques. Surgeons across the world are now curious to explore medical holograms to enhance the efficiency of complex surgeries.” Read more in BioSpectrum here.
Other medical stories include “Dr. Hologram: London Surgeon Pioneers Shoulder-Replacement Technique” in The London Free Press here, and this YouTube video discusses an augmented clinical experience using 10G technology to deliver the best breast cancer care everywhere so underserved communities can get state-of-the-art treatments. Verified Market Research recently published their report “Medical Holography Market Size worth $10.7 Billion, Globally, by 2028.”
Space Exploration
The medical use of holographic technology is also taking doctors to space. Read about a “Houston Company Using Holographic Tech to Beam Doctors Into Space.”
Hospitality
Proto Inc. (formerly PORTL) is the world’s first two-way, interactive holographic communications platform available to hospitality businesses, designed to be used for speeches, event appearances, trade show booth displays, and much more. Read “Hotel and Convention Center Communications With Worldwide Holographic Technology” on HospitalityNet.
Education
Yet another medical application is taking place in the world of education with “The University of Kansas School of Nursing Immersive Education Technology Solution.” View on YouTube.
The University of South Florida addresses the age-old problem of needing to be in two places at once. Read “Two Places at Once? USF Leader-Turned Hologram Proves It’s Possible.”
Politics
The headline says it all in the France24 article: “France’s Mélenchon Looks to Magic Up More Leftist Votes With Hologram Campaign Rallies.”
Sports/Entertainment
Cricket is big in India, so when telco giant Airtel wanted to demo new tech, what better way than to recreate the 1983 Cricket World Cup? “From Immersive Video to Holograms: What Will the 5G Experience Be Like?”
Art World
Nothing seems to titillate more than a mysterious Leonardo painting that very few people have actually seen. Holography tech brings it alive, but is it real? Read these two stories to get to the bottom of the mystery: “Holo: A 3D Hologram Inside a Crystal Box, Sold Via Smart Contract” in Be[in]crypo and “NFT Twist Is Latest Development in Saga of Contested ‘Leonardo’ Painting Hidden in a Swiss Vault” in The Art Newspaper.
Business
The Harvard Business Review weighs in on “How the Metaverse Could Change Work.”
Books
Something to add to your reading list: Venture capitalist Matthew Ball just published (as of July 19) “The Metaverse: How It Will Revolutionize Everything.” Read The Guardian’s Q&A with the author here: “Exit the Internet, Enter the Metaverse – Your Online Future Is in 3D.” Available on Amazon here.
Advances in Science
Prehistoric creature inspires new camera tech. Based on the optical structure of the eye of a Cambrian-era trilobite, the “New Light Field Camera Can Focus Up-Close and Far Away Simultaneously” with unlimited depth of field.
Posted July 22, 2022
“Golden Ticket” by Jay Howse
At 65 Inches and 8K, This May Be the World’s Largest 3D Immersive Display
Perhaps the biggest news in the world of immersive imaging in recent months is the impressively large holographic display announced by IDEA member company Looking Glass Factory. The Looking Glass 65” is the largest, highest-resolution display of its type ever commercially introduced, at 65 inches with 8K resolution. To put it into perspective, the display is five times larger than typical 3D displays, and 50 times larger than other group-viewable 3D display currently available on the market.
At this size, the display is intended for viewing by as many as 50 people at once. Generation of up to 100 different perspectives of 3D content from 100 million points of light every 60th of a second means everything shown on the Looking Glass 65” feels perceptually real.
Prototypes of this new display are already in use by entertainment companies for both storytelling and marketing. For instance, Springbok Entertainment recently premiered its new film, “Zanzibar: Trouble in Paradise,” on the 65-inch display at Tribeca 2022. This groundbreaking production is not only the first holographic film or documentary on a Looking Glass display but also the first-ever holographic film or documentary in competition at the Tribeca Festival.
“The massive increase in the size promises 3D storytellers the ideal canvas to push the boundaries of immersive experiences,” says Brandon Zamel, CEO at Springbok Entertainment. “This display solidifies the mainstream opportunities and applications of the immersive medium, effectively providing a missing piece of the puzzle for the industry that, in turn, will accelerate its entire growth.”
Looking Glass CEO Shawn Frayne puts the significance of this news into more perspective, saying, “This is only the beginning. Similar to the shift from photographs to film, radio to television, and black & white to color over the past century, the Looking Glass 65” will usher in one of the monumental shifts in how media is consumed, from flat 2D media to deeply 3D. No headset or 3D glasses required.”
The announcement generated extensive coverage in the news media last month. Here are a just a couple of the stories:
- Forbes: https://www.forbes.com/sites/michellegreenwald/2022/06/29/will-holograms-be-the-next-big-media-format-theyre-not-the-holograms-you-remember/?sh=86a633731c4a
- Crunchbase: https://techcrunch.com/2022/06/07/looking-glass-factory-large-holographic-display/
Posted July 22, 2022
|
“Believe it or not the building blocks for holographic TV are emerging,” says Adrian Pennington in his article, “The Holodeck Emerges Into the Light,” published this month in IBC365. “Among them is a standard format for creating, processing and transmitting content which is being supported by IDEA (Immersive Digital Experiences Alliance),” a group including Light Field Labs, and other holographic developers such as Otoy and Looking Glass Factory, along with the cable network Charter and cable industry think tank CableLabs. Good summary of the work of IDEA and digs into Light Field Labs new SolidLight panels and the technology behind them.
Holographic TV is one thing, but there are some who are already working on 3D two-way calling. In “Meta filed a patent for ‘3D conversations’ — are holographic calls almost here?” (from thenextweb.com), the author asks, what does it mean that “Meta, then Facebook, filed a Patent Application in 2020 titled 3D Conversations in an Artificial Reality Environment. Specifically, the application outlines a pipeline for capturing and transmitting 3D representations of callers, ‘allowing conversation participants to appear as if face-to-face.’”
“Meta’s patent suggests it is interested in using realistic 3D depictions to communicate body language. But it’s that broadness that makes it interesting too. Right now, one of the biggest hurdles for Zuckerberg’s vision of the metaverse is the fragmentation between virtual and real spaces.”
Granted, Meta isn’t the only company working on 3D calls. Google has Starline, Cisco has Webex Hologram, and Microsoft has its Mesh platform ….
Speaking of Webex, read “When your colleague is in the same room … but isn’t” (in Webex Ahead) for details on Webex Hologram, a platform that uses augmented-reality headsets to enable a feeling of true copresence by delivering realistic, 3D, real-time holograms of actual people.
Much of the discussion about 3D vs. 2D visual communications is about the advantage 3D gives in enhancing the ability to read facial expressions. And on that note, The Korea Advance Institute of Science and Technology (KAIST) reports on the development of a technique for facial expression detection by merging near-infrared light-field camera techniques with artificial intelligence (AI) technology. You can find a technical deep dive into that technique here —AI light-field camera reads 3D facial expressions (in ScienceDaily) —
In which Professor Ki-Hun Jeong, one of the leaders of the research team pursuing it says, “It could be applied in various fields including mobile healthcare, field diagnosis, social cognition, and human-machine interactions.”
In the world of large seamless 3D displays, SciTechDaily reports that researchers for University of Cambridge and Disney Research have “developed a new method to display highly realistic holographic images using ‘holobricks’ that can be stacked together to generate large-scale holograms.”
And finally on the topic of 3D meetings for everyone, Matsuko has an app for your iPhone that claims to make remote video meetings feel more like IRL conversations by using 3D holograms. Sound too good to be true? Read Fastcompany’s article here, “This startup wants to replace your Zoom meetings with holograms.
Posted March 29, 2022
Within IDEA, most of our progress is made within our working groups. The association currently has three active working groups. Here is a quick report on their activities.
The Media Format Working Group continues to meet bi-weekly, and has just completed its 2022 Program of Work — a roadmap describing planned enhancements for the Immersive Technology Media Format (ITMF). This detailed report, recently approved by the IDEA board, spells out features to be added in version 2 and subsequent version 3 of the ITMF suite of specifications. In addition, extensions to the libitmf reference software are also underway. These will include adding parsing and creation of an ITMF Scene Graph to the current capability of libitmf to encode and decode the ITMF container. A summary of this 2022 roadmap can be found on the IDEA Website.
The Network Architecture Working Group has been very active these last few months preparing the second in our series of ITMF file format demonstrations. This demo will be presented by webcast on Wednesday, May 4, and will show several noteworthy extensions to ITMF streaming of 3D objects in scene. Details on the demo, and how to sign up for the webcast, are included in this newsletter.
The Promotions Working Group is preparing for the return of live industry conference events, following the two-year hiatus. We look forward to meeting more of you at in-person events later this year, answering your questions and learning about your specific needs and challenges in the fast-evolving space of immersive media.
Posted March 29, 2022
In October 2021, IDEA presented its first demonstration of 3D streaming using the Immersive Technologies Media Format, ITMF. In that demo we revealed the possibilities of taking an ITMF file, extracting the scene description and assets, then streaming to a client from a network-based server. That demo can be viewed here.
The October demo showcased an early prototype of the 3D streaming platform, focusing on a small number of 3D objects streamed to a single client with attached display. Since then, engineers from IDEA members Charter Communications, OTOY, and CableLabs have worked together to improve the architecture and the pipeline for streaming 3D, and now have prepared a demonstration that takes 3D streaming to a new level.
The new demo, which takes place May 4th, 2022, will involve the streaming of hundreds of 3D objects in a complete 3D scene, creating a seamless 3D experience for the viewer. Now, the viewer will be able to watch and interact with longer form 3D content on multiple immersive displays.
For this demo, we will employ both Unreal and Unity game engines as clients. Over 90% of developers and content providers creating content for immersive displays are using these game engines to create content, so this is a major step towards bringing 3D streaming to a wider audience. The use of Unreal and Unity runtimes paves the way for targeting current and future light field based holographic displays such as the recently announced SolidLight display from IDEA founding member Light Field Lab.
ITMF scenes, in this case designed using OTOY’s OctaneRender, are decoded using IDEA’s open source library, libitmf, and delivered over the network during the demo.
This method is unique in that our pipeline will demonstrate the possibility of authoring in popular 3D applications like Blender, OctaneRender, Cinema 4D, Autodesk 3D Max, and Maya, and then being able to translate the content to target Unreal or Unity runtime clients streamed across a network.
To enable this functionality, our client detects the GPU and CPU of the device and provides this information to the “network orchestrator,” which is our tool for sending the correct asset stream based on the hardware capabilities of the device.
This demonstration will show that there is a real pipeline for developers and content providers to stream 3D content to immersive displays using the ITMF format. As GPU’s continue to increase in capability across the industry on mobile and desktop devices, the ability to stream 3D content is now a reality.
In many cases, streaming 3D has multiple benefits over more traditional pixel based streaming methods, including allowing the 3D artist or developer to have more granular control of content based on device type and GPU rendering ability. For example, instead of using real-world cameras to capture and transmit information, we can use virtual cameras, which can provide either dynamic camera positions or hundreds of virtual camera positions.
The repurposing of 3D objects in multiple scenes is another example. The ability to leverage the power of GPUs to manipulate 3D content locally provides many benefits for both the user and the developer. In addition, this method also lowers input latency and bitrate, and allows reuse of assets, as well as several other advantages which will be demonstrated on May 4th.
Imagine being able to create content in a popular 3D application, export the project, and then be able to target multiple devices for lossless streaming across a network with each device receiving the optimal content for the GPU of each device.
To see the future of 3D interoperability, and streaming 3D from existing CDNs and future network delivery technologies via Edge, 5G and 10G, we invite you to attend the demonstration from IDEA on May 4th.
Posted March 29, 2022
Did you miss our October Virtual Event or maybe want to see the presentations again? Our event Streaming Immersive Content: Demonstration of ITMF over an Edge Network is now available on YouTube. Check it out.