How can we maintain a seamless social connection between the real and virtual world as AR and VR devices become an increasingly large part of our work and play? In other words, how do we maintain “social co-presence” in shared spaces between people who may or may not be involved in the same AR / VR experience?
This year, Facebook Reality Labs (FRL) Research is presenting a new concept for social co-presence with virtual reality headsets at SIGGRAPH: Reverse Passthrough VR under the direction of researcher Nathan Matsuda. Put simply, Reverse Passthrough is an experimental VR research demo that allows the eyes of someone wearing a headset to be seen by the outside world. This is in contrast to what Quest headsets can do today with Passthrough + and the experimental Passthrough API, which use outward-facing cameras so that users can easily see their outside environment while wearing the headset.
Over the years we’ve made strides to bring pass-through capabilities for Oculus to consumers and developers. In fact, Matsuda came up with the idea for this experimental reverse passthrough research after spending a day in the office wearing a Quest headset with passthrough to think about how to make mixed reality environments more seamless for social and work environments. With the headset with passthrough, he could see his colleagues and the room around him well. But his colleagues couldn’t see him without an external display. Every time he tried to talk to someone, they noticed how strange it was that he couldn’t make eye contact. So Matsuda asked the question, what if you could see his eyes – would that add something to the social dynamic?
When Matsuda demonstrated reverse passthrough for the first time in 2019 for FRL chief scientist Michael Abrash, Abrash was not convinced of the usefulness of this work. In the demo, Matsuda wore a custom-made Rift S headset with a front-mounted 3D display. On the screen, a floating 3D image of Matsuda’s face, roughly rendered by a game engine, reproduced his moment using signals from two eye-tracking cameras in the headset.
Research scientist Nathan Matsuda carries an early reverse passthrough prototype with outward-facing 2D displays. Right: The first fully functional reverse passthrough demo with 3D light field displays.
“My first reaction was that it was a silly idea, novelty at best,” Abrash said. “But I’m not telling the researchers what to do because without the freedom to try new things there is no innovation, and that’s a good thing because now it’s clearly a unique idea with real promise.”
Almost two years after the first demo, the 3D display technology and research prototype have advanced significantly, boasting specially designed optics, electronics, software and a range of assistive technologies to capture and display more realistic 3D faces. This progress is promising, but this research is clearly still experimental: Connected by many cables, it is anything but a stand-alone headset, and the representations of the eyes and faces are not yet entirely lifelike. However, it is a research prototype developed in the spirit of the core ethos of FRL Research to work with widely dispersed concepts that may seem a little bogus. While this work is far from being a product roadmap, it does provide a glimpse of how reverse passthrough could be used in collaborative spaces of the future – both real and virtual.
Left: A state-of-the-art VR headset with a deactivated external display. No glances are visible through the opaque headset housing. center: A VR headset with outward-facing 2D displays, as suggested in previous academic papers[1][2][3][4]. Some gaze cues are visible, but the wrong perspective limits the viewer’s ability to see the direction of the gaze. Right: Our most recent prototype uses 3D reverse passthrough displays that show the correct perspective for multiple external viewers.
Reverse passthrough
The essential component of a reverse passthrough headset is the outward-facing 3D display. You could just put a 2D display on the front of the headset and show a flat projection of the user’s face on it, but the offset from the actual face of the user to the front of the headset creates a visually annoying, unnatural effect that breaks anyone’s hope, reading correct eye contact. In developing the research prototype, it became clear that a 3D display was the better direction as the user’s eyes and face would appear in the correct position in space on the front of the headset. This representation helps maintain alignment as external viewers move relative to the 3D display.
There are several established ways to display 3D images. For this research we used a microlens array light field display because it is thin, easy to construct and based on existing consumer LCD technology. These displays use a tiny lenticular grid that emits light from different LCD pixels in different directions so that a viewer will see a different image when looking at the display from different directions. The perspective of the pictures naturally shifts so that any number of people in the room can look at the light field display and see the correct perspective for their location.
As with every early research prototype, this hardware still has considerable limitations: firstly, the viewing angle must not be too strong and, secondly, the prototype can only display objects in focus that are within a few centimeters of the physical screen surface. Conversations take place face to face, which of course limits the reverse angle of passage. And the wearer’s face is just inches away from the physical screen surface, so the technology works well for this case – and works even better when VR headsets continue to shrink using methods such as holographic optics.
Building the research prototype
FRL researchers used a Rift S for early reverse passthrough studies. As the concept evolved, the team began iterating on Half Dome 2 to build the research prototype presented at SIGGRAPH that year. Mechanical research engineer Joel Hegland dismantled the headset onto the bare display pod and provided an approximately 50 millimeter thick VR headset that served as the basis for the latest reverse passthrough demo. Then optician Brian Wheelwright designed a microlens array to be attached to the front.
The resulting headset contains two display pods that are mirror images of each other. They contain an LCD panel and a lens for the basic VR display. A ring of infrared LEDs illuminates the part of the face covered by the capsule. Between the lens and the screen there is a mirror that only reflects infrared light, so that two infrared cameras can almost see the eye from the front. All of this in the invisible infrared band prevents the eye imaging system from distracting the user from the VR display itself. Then the front of the pod has another LCD with the microlens array.
Left: A cutaway view of one of the prototype display pods. Right: The prototype display pod with driver electronics before being built into the complete headset prototype.
Show eyes and faces in 3D
Producing the nested 3D images for display on the light field display was a major challenge in and of itself. For this research prototype, Matsuda and his team decided to use a pair of stereo cameras to create a surface model of the face and then projected them Views of the eye on this surface. While the resulting projected eyes and face are not lifelike, this is only a short term fix to pave the way for future development.
Research on FRL’s codec avatars points to the next generation of this imaging. Codec avatars are realistic representations of the human face, facial expressions, voice and body that can be controlled in real time in a VR headset through deep learning from a compact set of measurements. These virtual avatars should be much more effective for reverse passthrough and enable a unified system of face rendering that works regardless of whether the viewer is local or remote.
The short video shown below shows a codec avatar from our Pittsburgh lab running on the prototype reverse pass-through headset. These images and their movement over time appear much more lifelike than those captured with the current stereo camera method, suggesting the kind of improvements such a system could offer when working with remote telepresence systems.
The reverse passthrough prototype showing a high fidelity codec avatar face reconstruction.
A way to social co-presence in VR
Totally immersive VR and AR glasses with display are fundamentally different technologies that are likely to serve different users in different scenarios in the long term. There will be situations where people need the true transparent optics of AR glasses and others where people prefer the image quality and immersion of VR. Facebook Reality Labs Research, led by Michael Abrash, has thrown a wide net in researching new technical concepts to move the ball forward across these two display architectures. Full exploration of this space will ensure that the laboratory grasps the full range of possibilities – and limitations – for future AR / VR devices and eventually puts those insights into practice in a way that human-computer interaction will work for most people Most places in the world are supported.
Reverse passthrough is representative of this type of work – an example of how ideas from the lab drive the usefulness of VR headsets. Later this year we’ll be giving a more holistic update on our research on display systems, showing how all of this work – from varifocal, holographic optics, eye tracking and distortion correction to reverse passthrough – comes together to help us make it through what we call the visual Turing test in VR.
Ultimately, these innovations and more will come together to create VR headsets that are compact, lightweight, and portable all day. mix the high quality virtual pictures with high quality real pictures; and that lets you be socially present with everyone in the world, whether they’re on the other side of the planet or standing next to you. To make this possible is our goal at Facebook Reality Labs Research.
[1] Liwei Chan and Kouta Minamizawa 2017. FrontFace: Facilitating communication between HMD users and outsiders using front-facing screen HMDs
[2] Kana Misawa and Jun Rekimoto, 2015. ChameleonMask: Embodied physical and social telepresence with human surrogate mothers
[3] Christian Mai, Lukas Rambold and Mohamed Khamis. 2017. TransparentHMD: Make the face of the HMD user visible to bystanders
[4] Jan Gugenheimer, Christian Mai, Mark McGill, Julie Williamson, Frank Steinicke and Ken Perlin. 2019. Challenges in using head-mounted displays in shared and social spaces
Comments are closed.