Facebook and Instagram owner Meta and German car giant BMW revealed progress in their joint exploration of how augmented and virtual reality technology can work inside a fast-moving vehicle.
Meta is continuing on its metaverse path. Per BMW's announcement,
The #BMW Group Technology Office USA and Meta’s Reality Labs Research demonstrated, "for the first time, the ability to accurately display stable virtual reality (VR) and mixed reality (MR) content to passengers in a fast-moving car, even when making turns, going over speed bumps, and accelerating."
The goal of this joint research, announced in 2021, is to explore the way in which AR and VR could be integrated into smart vehicles in the future in order to improve the "passenger experience," Meta's announcement said.
And this is not just about your family's car. Per Meta,
"If we get it right, this technology could revolutionize travel in cars, trains, planes and beyond, unlocking new forms of hands-free communication, entertainment and utility — giving us far more value than the screens and instruments we’re used to seeing in vehicles today."
Claus Dorrer, Head of BMW Group Technology Office USA in Mountain View, called the research "promising," and said:
"It is too early to tell exactly how or when this technology will make it into customers’ hands, but we envision a number of potential use cases for XR devices in vehicles — from assisting the driver in locating their car in a crowded parking lot to alerting them to hazards on the road and surfacing important information about the vehicle’s condition."
However, in a moving vehicle, these sensors face a conflict. The headset's motion sensors think it is moving, but its cameras think it is standing still when looking at the car's interior.
But the two companies made "a huge feat," explaining that,
"To solve this problem, we [incorporated] IMU data from a BMW car’s sensor array in real time into the tracking system of our Project Aria research glasses. This additional information allows the system to calculate the glasses’ location relative to the car."
Transferring the tracking system to a Meta Quest Pro headset allowed the team to accurately anchor virtual objects to a moving car using a digital twin of that car.
This way, the team said they were able to "demo some compelling virtual and mixed reality passenger experiences."
The next step, said the partners, is to add the car’s location relative to the world in order to enable world-locked rendering.
"Access to the car’s precise 6DOF positioning system could allow us to render world-locked virtual content outside of the vehicle, like identifying landmarks and other points of interest."
Meta noted that it will continue to work with BMW and that it expects the capability they develop to be "invaluable" for AR glasses and personalized AI assistants in the days to come.
