|
The status of Project Starline, Google’s 3D video calling booths had been somewhat in doubt for the last year. The project had been wrapped up in a reorganization at Google a year ago, involving Google’s AR and VR initiatives along with the internal R&D group known as Area 120. Then in September, Google drastically cut Area 120 projects, leaving the state of Project Starline unknown. Even within the company, people weren’t sure if the project was still around.
Fast forward to October this year, and we find Google announcing expansion of real-world tests. An early access program will see Starline being used in the offices of various Google enterprise partners, including Salesforce, WeWork, T-Mobile and Hackensack Meridian Health.
Project Starline is basically a holographic video chat offering for one-on-one communications. Using 3D imagery, high-resolution cameras, custom depth sensors and a light field display, Starline creates a lifelike experience without requiring a headset.
While it’s still unclear how or when Google ever plans to commercialize the Starline technology, some observers think just the fact that Google has made a nod in this direction – i.e., no headsets, no avatars, and working to present people as they are – is a validation of sorts, a suggestion that this type of virtual meeting offering is really what the market wants.
Read more about it on TechCrunch here.
And here’s Googles own take from their Blog
As we near the close of 2022, the one thing that remains constant seems to be the increasing number of news stories involving holograms, 3D immersive tech, and the metaverse. Just about every sector of life seems to be in-play as far as being touched by this trend.
Here’s a sampling of what we spotted in the last few months.
- Sight and sound have been the key experiential elements we think of when we think about the virtual worlds, but what about enhanced interaction with the virtual world, what about touch and smell? Researchers at University of Glasgow are looking into just that. Read about “Touchable Holograms: Interacting With Virtual Words.”
- Marcus Cantu is the first baseball player to use holographic teleportation. He’s been teleported, holographically anyway, into the coach’s office where he can be coached remotely. It’s the same tech that put a doctor into the space station, but now available at a lower costs. Baseball Prodigy Uses Holographic Teleportation for Batting Practice.
- In the realm of virtual tourism, Airtel and Nokia are bringing to life the cultural splendor of India’s Kashi Vishwanath temple and the Statue of Unity, showcasing the architecture and design of these iconic monuments and replicating the physical experience through holographic images demonstrating the benefits of ultra-fast and low latency 5G network. Airtel, Nokia Partner to Bring Holographic Technology to Boost Virtual Tourism
- A new microscope developed by Korean scientists is able to “see through” the intact skull of a living mouse and produce high resolution 3D imaging of the neural network inside the mouse brain. The scientists have been able to overcome the problems associated with light scattering randomly when falling on living cells, and with the new microscope they were able to successfully achieve a high-resolution image of the neural network of the mouse brain present within the skull. Holographic Microscope to Observe the Brain’s Neural Network
- VisAR technology from Novarad uses augmented reality technology to superimpose a surgical navigation plan right over the patient’s body to provide guidance during spinal surgeries. The technology has just been cleared by the FDA in the U.S.FDA clears augmented reality system to help navigate spine surgery
- Medical holography in general has been a hot topic, and Fact.MR has released a report that projects that the global medical holography market will grow from US$ 447.8 million in 2019 to US$ 3.5 billion by 2026.
- When it comes to life in the metaverse or any virtual world, the ethical questions and human rights issues will inevitably have to be addressed. In Migration to the Metaverse: We Need Guaranteed Basic Immersive Rights the author identifies the issue that “metaverse providers will have unprecedented power to profile and influence their users,” and therefore suggests that it’s time to work toward guaranteeing a set of “immersive rights.”
- And just in time for the Holidays … the Immersive Nutcracker, A Winter Miracle is coming to Vegas. The 30-minute immersive experience retells the Christmas classic, encompassing visitors in over 500,000 cubic feet of projections composed of over 1 million frames of video.
Together with the Roddenberry Estate, IDEA member OTOY has made it possible for Star Trek fans to experience Gene Roddenberry’s original vision for the franchise in 1964. During this year’s Creation Las Vegas, they revealed the first previews of the Roddenberry Archive — a memorialization of the entirety of Gene Roddenberry’s life, legacy and creative works, including a real-time, fully interactive and immersive collection of ever-growing 3D datasets that are all presented at 1:1 scale.
In addition to physical assets filmed for real-world TV or film productions, the archive also includes in-universe conceptualizations for how items, ships or the show’s other props would behave inside the narrative universe. Perhaps the most striking of these is a life-size digital recreation of the Starship Enterprise as Gene Roddenberry had originally envisioned it in the show’s pilot, “The Cage.”
When entering or exiting the Enterprise, fans beam in and out using the ship’s teleporters, consequently transporting them some 234 years into the future. From there, the fully immersive and interactive 360-degree exhibition would take them through every filmed location in the show’s pilot, where they could interact with the ship’s interior as if they were on board. From sitting in Captain Pike’s chair to visiting his quarters, and from visiting different levels via the turbolift to flipping through Spock’s Talos IV presentation in the briefing room, the exhibit gives fans everything short of a full crew uniform.
Other displays at the show included a first-time presentation of scenes from Star Trek’s early TV, literary, and film history that were captured and recreated on holographs, intended for future volumetric viewing on real-life Holodeck panels built by Light Field Lab.
Additionally, virtual production technologies are combined with LightStage scans of the archive to put original Star Trek uniforms, props, and other materials on display in a new concept video “765874: Mem ory Wall.” The concept video is directed by Carlos Baena and Jules Urbach with collaboration from world-renowned Octane CG artists Aaron Westwood and BLR VFX.
OTOY and the Roddenberry Estate have collaborated for over a decade to unearth original materials from Gene Roddenberry’s legacy, including the Star Trek creator’s own personal notes, internal company documents, and prototypes of the series’ various iterations. Said Gene’s son and CEO of Roddenberry Entertainment, Rod Roddenberry, “We wanted to provide a behind-the- scenes personal story about Gene and the evolution of the series, which has made such a profound impact on the development of humanity in the 20th century, inspiring so many around the world.”
He added, “With new real time and holographic technologies, it is getting more and more possible to immerse new generations who didn’t experience Star Trek first-hand into my father’s universe, and we are thrilled to push forward on this new frontier in collaboration with 3D graphics pioneer, OTOY.”
Jules Urbach, CEO and Co-Founder of OTOY, spoke further about how a show of this scale best cap tures the genius of the franchise’s creator. He said, “Gene Roddenberry saw into the future and projected a vision of the future where remarkable technologies affirmed the human spirit and ethical powers. Technological advances today in holographic imaging and real time graphics are making recreating his vision ever more possible.”
Fortunately for those who missed the show, the archival videos and images—as well as behind-the scenes video logs — are available for public viewership online.
A year ago, CableLabs issued a challenge to individuals and organizations to think big – they launched the 10G Challenge, asking innovators to imagine technologies that the emerging 10G network will make possible. At SCTE Cable-Tec Expo 2022 they announced the winners, those with the outstanding ideas to leverage the four pillars of the multi-gigabit platform: faster internet speeds, better security, lower latency, and improved reliability. Notably, almost all the winning entries involved some kind of immersive experience. Here are the winners:
- Grand Prize Winner: MediViewXR leverages advanced virtual reality (VR) technology to help patients in rural areas access care from National Cancer Institute-designated cancer centers without having to leave their community.
- Live Category Winner:Xennial Digital is creating the future of medical simulation training using the reliability of the 10G broadband network.
- Work Category Winner: TractorClouddevelops hardware and software solutions that save farmers thousands of dollars and drive the future of tractor fleet management.
- Learn Category Winner: KC Digital Driveaims to utilize the 10G network to expand access to nursing education via digital twin classrooms.
- Play Category Winner:Dana Leung’s “Interacting Within Live-Action Video Scenes” technology hopes to enhance consumer experiences, whether shopping or watching movies, with real-time VR.
- People’s Choice Winner: T4 Movementhelps patients identify the root cause of their musculoskeletal issues and improve rehabilitation in a manner that is innovative, low cost, easy to execute and scalable.
Attendance at SCTE Cable-Tec Expo 2022 actually surpassed the attendance at the last Cable-Tec Expo in 2019, meaning it outperformed most other live shows in the last year — on average trade shows are only getting about 65% attendance compared to pre-pandemic years. By all accounts the show was hugely successful and included an opening day session in front of a packed house of thousands of attendees.
IDEA member company CableLabs was quite active, providing a glimpse into the future of content consumption at their booth, with a demo of the 3D holographic display from Looking Glass Factory, another IDEA member.
Also at the show, CableLabs software engineer, Austin Pahl, gave an in-depth presentation on motivations and algorithms behind “3D Streaming.” The concept of 3D streaming emerged from IDEA, which recognized a need for delivering immersive content to end displays with a simple, uniform methodology regardless of display type. With the ongoing growth of immersive media, particularly augmented reality (AR), virtual reality (VR), and light field displays, and the rapidly increasing data throughput requirements needed by these devices, IDEA believes it is more important than ever to build an architecture that streamlines distribution workflows.
Pahl’s presentation slides and paper will be made available for free on the Fall Technical Forum website later this year. https://expo.scte.org/fall-technical-forum/
“Believe it or not the building blocks for holographic TV are emerging,” says Adrian Pennington in his article, “The Holodeck Emerges Into the Light,” published this month in IBC365. “Among them is a standard format for creating, processing and transmitting content which is being supported by IDEA (Immersive Digital Experiences Alliance),” a group including Light Field Labs, and other holographic developers such as Otoy and Looking Glass Factory, along with the cable network Charter and cable industry think tank CableLabs. Good summary of the work of IDEA and digs into Light Field Labs new SolidLight panels and the technology behind them.
Holographic TV is one thing, but there are some who are already working on 3D two-way calling. In “Meta filed a patent for ‘3D conversations’ — are holographic calls almost here?” (from thenextweb.com), the author asks, what does it mean that “Meta, then Facebook, filed a Patent Application in 2020 titled 3D Conversations in an Artificial Reality Environment. Specifically, the application outlines a pipeline for capturing and transmitting 3D representations of callers, ‘allowing conversation participants to appear as if face-to-face.’”
“Meta’s patent suggests it is interested in using realistic 3D depictions to communicate body language. But it’s that broadness that makes it interesting too. Right now, one of the biggest hurdles for Zuckerberg’s vision of the metaverse is the fragmentation between virtual and real spaces.”
Granted, Meta isn’t the only company working on 3D calls. Google has Starline, Cisco has Webex Hologram, and Microsoft has its Mesh platform ….
Speaking of Webex, read “When your colleague is in the same room … but isn’t” (in Webex Ahead) for details on Webex Hologram, a platform that uses augmented-reality headsets to enable a feeling of true copresence by delivering realistic, 3D, real-time holograms of actual people.
Much of the discussion about 3D vs. 2D visual communications is about the advantage 3D gives in enhancing the ability to read facial expressions. And on that note, The Korea Advance Institute of Science and Technology (KAIST) reports on the development of a technique for facial expression detection by merging near-infrared light-field camera techniques with artificial intelligence (AI) technology. You can find a technical deep dive into that technique here —AI light-field camera reads 3D facial expressions (in ScienceDaily) —
In which Professor Ki-Hun Jeong, one of the leaders of the research team pursuing it says, “It could be applied in various fields including mobile healthcare, field diagnosis, social cognition, and human-machine interactions.”
In the world of large seamless 3D displays, SciTechDaily reports that researchers for University of Cambridge and Disney Research have “developed a new method to display highly realistic holographic images using ‘holobricks’ that can be stacked together to generate large-scale holograms.”
And finally on the topic of 3D meetings for everyone, Matsuko has an app for your iPhone that claims to make remote video meetings feel more like IRL conversations by using 3D holograms. Sound too good to be true? Read Fastcompany’s article here, “This startup wants to replace your Zoom meetings with holograms.
Posted March 29, 2022
Within IDEA, most of our progress is made within our working groups. The association currently has three active working groups. Here is a quick report on their activities.
The Media Format Working Group continues to meet bi-weekly, and has just completed its 2022 Program of Work — a roadmap describing planned enhancements for the Immersive Technology Media Format (ITMF). This detailed report, recently approved by the IDEA board, spells out features to be added in version 2 and subsequent version 3 of the ITMF suite of specifications. In addition, extensions to the libitmf reference software are also underway. These will include adding parsing and creation of an ITMF Scene Graph to the current capability of libitmf to encode and decode the ITMF container. A summary of this 2022 roadmap can be found on the IDEA Website.
The Network Architecture Working Group has been very active these last few months preparing the second in our series of ITMF file format demonstrations. This demo will be presented by webcast on Wednesday, May 4, and will show several noteworthy extensions to ITMF streaming of 3D objects in scene. Details on the demo, and how to sign up for the webcast, are included in this newsletter.
The Promotions Working Group is preparing for the return of live industry conference events, following the two-year hiatus. We look forward to meeting more of you at in-person events later this year, answering your questions and learning about your specific needs and challenges in the fast-evolving space of immersive media.
Posted March 29, 2022
In October 2021, IDEA presented its first demonstration of 3D streaming using the Immersive Technologies Media Format, ITMF. In that demo we revealed the possibilities of taking an ITMF file, extracting the scene description and assets, then streaming to a client from a network-based server. That demo can be viewed here.
The October demo showcased an early prototype of the 3D streaming platform, focusing on a small number of 3D objects streamed to a single client with attached display. Since then, engineers from IDEA members Charter Communications, OTOY, and CableLabs have worked together to improve the architecture and the pipeline for streaming 3D, and now have prepared a demonstration that takes 3D streaming to a new level.
The new demo, which takes place May 4th, 2022, will involve the streaming of hundreds of 3D objects in a complete 3D scene, creating a seamless 3D experience for the viewer. Now, the viewer will be able to watch and interact with longer form 3D content on multiple immersive displays.
For this demo, we will employ both Unreal and Unity game engines as clients. Over 90% of developers and content providers creating content for immersive displays are using these game engines to create content, so this is a major step towards bringing 3D streaming to a wider audience. The use of Unreal and Unity runtimes paves the way for targeting current and future light field based holographic displays such as the recently announced SolidLight display from IDEA founding member Light Field Lab.
ITMF scenes, in this case designed using OTOY’s OctaneRender, are decoded using IDEA’s open source library, libitmf, and delivered over the network during the demo.
This method is unique in that our pipeline will demonstrate the possibility of authoring in popular 3D applications like Blender, OctaneRender, Cinema 4D, Autodesk 3D Max, and Maya, and then being able to translate the content to target Unreal or Unity runtime clients streamed across a network.
To enable this functionality, our client detects the GPU and CPU of the device and provides this information to the “network orchestrator,” which is our tool for sending the correct asset stream based on the hardware capabilities of the device.
This demonstration will show that there is a real pipeline for developers and content providers to stream 3D content to immersive displays using the ITMF format. As GPU’s continue to increase in capability across the industry on mobile and desktop devices, the ability to stream 3D content is now a reality.
In many cases, streaming 3D has multiple benefits over more traditional pixel based streaming methods, including allowing the 3D artist or developer to have more granular control of content based on device type and GPU rendering ability. For example, instead of using real-world cameras to capture and transmit information, we can use virtual cameras, which can provide either dynamic camera positions or hundreds of virtual camera positions.
The repurposing of 3D objects in multiple scenes is another example. The ability to leverage the power of GPUs to manipulate 3D content locally provides many benefits for both the user and the developer. In addition, this method also lowers input latency and bitrate, and allows reuse of assets, as well as several other advantages which will be demonstrated on May 4th.
Imagine being able to create content in a popular 3D application, export the project, and then be able to target multiple devices for lossless streaming across a network with each device receiving the optimal content for the GPU of each device.
To see the future of 3D interoperability, and streaming 3D from existing CDNs and future network delivery technologies via Edge, 5G and 10G, we invite you to attend the demonstration from IDEA on May 4th.
Posted March 29, 2022
IDEA will be presenting at the SMPTE Hollywood Meeting.
Wed, June 19, 2019 | 6:30 PM – 9:30 PM PDT
Want to attend? Tickets available here: SMPTE Hollywood
Speakers will include:
Ryan Damm, CEO and Founder Visby
Dr. Arianne Hinds, Immersive Media Strategy at Cablelabs – Chairperson at IDEA
Jon Karafin, CEO and Founder of Light Field Lab.