Co-experiences: Hand around in the digital lounge
In the spring of 2019 – a full year before the COVID-19 pandemic caused the world to turn our living rooms into offices, schools, and meeting places – Mark Zuckerberg shared a plan he wanted to create the digital equivalent of the living roomwhere people could connect and hang out together. This work was already underway earlier this year when countries started implementing measures at home. When people started moving online At one volume the internet could barely supportWe realized that the plan had to become a reality as soon as possible. We’ve postponed our plans to speed up work on co-experiences, a series of digital shared experiences that people can enjoy together.
What do we mean by co-experiences? Think of those moments in your own living room when you feel closer to friends and family. This probably includes a long conversation, but also experiences like watching sports or an emotional season finale on TV, sharing music with a new band you are in, or sharing old photos and the stories behind them. People connect through shared experiences, and we want to make video chat on Messenger and Instagram a place where people can share these types of experiences in their virtual living rooms. For this purpose we have already rolled out Messenger rooms, Cross-app communication between Messenger and Instagram as well as screen sharing, 360 backgrounds and AR lighting. Our newest addition is Watch together, which allows users to watch shows and videos together on their devices.
To achieve this, we had to work like a startup. While Messenger is an established product, we created a new experience from the ground up and needed to make sure we were developing something that people wanted and would actually use. It had to be valuable and make video chatting more fun. This meant optimization for learning, listening, and adapting. We questioned our assumptions early and often and made rapid progress. We also had to look at the video calls as a whole to really make this a hangout experience. We had to completely redesign the video calling experience to create a natural, comfortable place for people, making the user interface so simple and intuitive that anyone can quickly jump in and find the options they need.
Along the way, we’ve created new platforms and solved the tough technical issues associated with building synchronous experiences so everyone can see the same thing at the same time, regardless of what device or network they’re on. This presented additional technical challenges when done at the Facebook level. For Watch Together, we had to build a brand new platform for status syncing through our video call stack, tweaking audio to optimize mixing and reducing echo, and creating personalized video recommendation models, to name a few.
Product development challenges
We wanted to see Watch Together as a product in its own right and have a strong product / market customization – a product that people not only want to use but also want to share with others and use while hanging out. We knew we probably wouldn’t get things right the first time, so we were prepared for trial and error and had to learn quickly.
To optimize the features we created and to ensure it was the experience we wanted, we used both quantitative signals from our instrumentation and qualitative signals in the form of research feedback from real people with Watch Together: we did numerous research studies and one product -Influencer community with a large number of users. Thanks to feedback from these users, we’ve completely redesigned the video calling experience to create an intuitive space for the things people do together. We designed a new drawer. simplified the process for selecting videos; The ranking, the categories as well as the available groupings and content have been optimized and iterated. We also tested several different call layouts to optimize the viewing experience so that a person on the device sees a comfortable balance between the video they are watching and their view of the other participants. We also heard the importance of audio balance, echo cancellation, and independent volume control for the video that we each brought into the final experience.
We were able to rely on strong quantitative signals provided by tools to understand what is happening and how the product is being used, and to validate what is and what is not. We used tools like funnel analysis and drop rates, ranking performance, UX interactions, participation and retention J-curves to identify where Watch Together needs improvement. We carried out hundreds of individual experiments to arrive at today’s experiences. All of these UX patterns have been created generically so that we can use them for new experiences in the future.
Build a synchronous experience
One of the biggest challenges in the shared experience was enabling high-precision, real-time synchronization between different apps and devices to ensure everyone was seeing the same thing at the same time. We also needed to improve control reliability to ensure that playback controls such as play, pause, rewind, or fast-forward are done simultaneously across devices. We also had to sync the audio stream of videos and real-time calls well so people wouldn’t have to deal with lags and overlaps.
To remedy this, we created our own state sync platform that uses our multipath infrastructure (the servers on which we host individual and group calls) for real-time resolution of client-side status for all devices in a call. This is a significant improvement over previous technologies. Before status sync, engineers jumped through many frames to sync status between calls. The most common method has been to use data messaging, where information about the client-side resolution on the device is sent to all clients. This led to individual Android, iOS and web implementations for calculating the final state. However, this strategy often resulted in inconsistencies between customers and was not always as reliable as we would like. With status synchronization, we can support web, Android and iOS both via Messenger and Instagram and use our own call infrastructure.
We have defined thrift Structures for both input (the messages to be received on the server) and output (the expected resolved output structure from the server). From there, status synchronization can be broken down into three main parts: subscription, input, and resolution.
- To begin sending and receiving status, the client must first subscribe to the status synchronization topic with which it wants to interface. This ensures that we are not sending Watch Together data to a client that does not support it.
- In the input stage, we use the predefined Thrift structures to send formatted messages for resolution. We use a snapshot-based model so that the last input from a client is that used by the resolver in case multiple inputs are sent in quick succession.
- All clients in a call meet the same server-side resolver, centralizing all inputs through the same resolver logic and eliminating the need for multiple message processing units on each client. This is another great advantage of status synchronization: We save computing power on the clients and ensure consistency.
State Sync was developed as a new, generic platform designed so that we can use it for future experiences. Today it not only supports Watch Together, but also other scenarios, such as B. moderating screen sharing in messenger rooms and Room to live. The platform is also built with Cross-app communication Remember, friends and family can connect through Watch Together or one of the new devices Messenger experiences on Instagram.
The future of co-experiences
Thanks to the work we’ve done understanding Hangouts, building generic UX components that can be reused, and building our state-sync platform, we have a solid foundation on which to continue to experience like Watch Together, as, can expand and develop new co-experiences.
These tools allow us to scale over time to support more experiences and support them across all of our apps and services. In the future, we would love to see a world where people can use Messenger and Instagram as a virtual living room to talk, share, watch and much more. We are just getting started and looking forward to the journey ahead.