Core Information Science researchers talk about analysis award alternative in adaptive experimentation

On February 24th, Facebook launched a call for proposals (RFP) for Bayesian randomized sequential decision making, which ends on April 21st. With this call, the Facebook Core Data Science (CDS) team hopes to deepen its ties to the academic research community by finding innovative ideas and applications of Bayesian optimization that will further advance the field. For an inside look at the team behind the RFP, we reached out to Eytan Bakshy and Max Balandat, who lead the effort within CDS.
View RFPBakshy leads the Adaptive Experimentation team, which tries to improve the throughput of experiments using machine learning and statistical learning. Balandat supports the team’s modeling and optimization efforts, which mainly focus on probabilistic models and Bayesian optimization. In these Q&A, Bakshy and Balandat put the RFP in context by sharing more information on how their team’s work relates to the areas of interest for the call.

Q: What is the aim of this RFP?

A: First and foremost, we want to know more about the great research in this area. Conversely, we’re also able to share a number of really interesting real-world use cases that we hope can help stimulate additional applied research and increase interest and research activity in sample-efficient sequential Bayesian decisions. Ultimately, we want to further strengthen our ties to science and our collaboration with scientists who are at the forefront in this field.

We are both excited about the creative applications and approaches to Bayesian optimization that researchers bring to their proposals.

Q: What inspired you to start this RFP?

A: We publish quite a bit in top AI / ML venues and all of our articles are shaped by very practical issues that we face every day in our work. The need to explore large design spaces through experimentation on a budget is rife in Facebook, Instagram, and Facebook Reality Labs. Much of our team’s work is focused on applied problems to support the business and use-inspired basic research. However, it is clear that there are many ideas that can advance the field of sampling efficient sequential decision making such as Bayesian optimization and related techniques.

In science, it can sometimes be difficult to understand what exactly are the most relevant and powerful “real world” problems. Conversely, academics may find it easier to step back, look at the bigger picture, and do more exploratory research. With this RFP we hope to be able to contribute to closing this gap and to promoting increased cooperation and cross-pollination between industry and science.

Q: What is Bayesian Optimization and how is it applied on Facebook?

A: Bayesian optimization is a series of methods for exploring large design spaces on a budget. While Bayesian optimization is often used for hyperparameter optimization in machine learning (AutoML), our team’s work was originally motivated by the use of online experiments (A / B tests) to optimize software and recommendation systems.

Since then, the areas of application for Bayesian optimization have expanded enormously. Applications range from designing next-generation AR / VR hardware, to bridging the gap between simulations and real-world experiments, to efforts to provide affordable connectivity solutions to developing countries.

The main idea behind Bayesian optimization is to adapt a probabilistic substitute model to the “black box” function to be optimized and then to use this model to inform the new parameters for which the function is to be evaluated next. This enables a fundamental consideration of the reduction in uncertainty by exploring promising regions of the parameter space. As discussed earlier, we apply this approach to a wide variety of issues across different domains on Facebook.

Q: What is BoTorch and how does it relate to the RFP?

A: The adaptive experiment team has been investing in methodological development and tools for Bayesian optimization for over five years. A few years ago we found that our current tools were slowing researchers ‘ability to generate new ideas and engineers’ ability to scale use cases for Bayesian optimization.

This might interest you

BOTORCH: A Framework for Efficient Monte Carlo Bayesian Optimization

To address these issues, we developed BoTorch, a framework for Bayesian optimization research, and Ax, a robust platform for adaptive experimentation. BoTorch follows the same modular design philosophy as PyTorch, which makes it very easy for users to swap out or rearrange individual components to customize all aspects of their algorithm. This allows researchers to be on the cutting edge of research on modern Bayesian optimization methods. It’s also fast by leveraging modern parallel computing paradigms on both CPUs and GPUs.

BoTorch has fundamentally changed the way we approach Bayesian optimization research and accelerated our ability to tackle new problems. We hope that the RFP will spark wider interest in this area and raise awareness of our open source tools.

Q: Where can people stay up to date and learn more?

A: We actively work with researchers on Twitter. So follow @eytan and @maxbalandat for the latest research, and feel free to reach out to us via Twitter, email, or GitHub with any questions or ideas.

Check out our BoTorch and Ax open source projects for the latest and greatest of what we’re working on. It’s also helpful to keep an eye out for our contributions at machine learning conferences like NeurIPS, ICML, and AISTATS.

–

Proposals for the RFP for randomized sequential Bayesian decision making, which will be completed on April 21, 2021, and winners will be announced the following month. Subscribe to our RFP email list to be informed of new research pricing and appointment notification opportunities.