The way forward for Buck
When Buck was featured at Facebook Mobile DevCon in 2013, it was a fast Android build system. Since then, it has expanded to support application development in 15+ languages and targeting platforms from cell phones to servers to smart devices, VR headsets and more.
Since then, many users have found that Buck is a very powerful, fast build system that greatly improves their developer experience. We continue to develop innovative approaches to scaling builds, like our work with Android app compilations. The community has shared many stories about how they accelerated their own development using Buck, for example at Droidcon NYC 2016 and Devoxx Belgium 2017.
Over time, however, Buck has not kept pace with this growth in functionality and flexibility with the architectural changes and enhancements necessary to handle this complexity. As we develop new features inside and outside of Facebook, we’ve found that this has added significant implementation complexity and makes it increasingly difficult to make changes to the core and language rules. Although we started an initiative almost 4 years ago to incrementally make some of these overarching architecture improvements to Buck, these changes have been extremely difficult.
Shifting focus to our next generation build system
Since the beginning of 2020 we’ve been experimenting with what it would look like if we redesigned the build system from scratch. From the lessons we have learned over the past 7 years, how could we write a build system that is still scalable and expandable for the next 10 or 20 years?
There’s a lot that we think Buck did just right, and the next generation will stick with many of the core concepts. The new system will be fully downward compatible for many languages and workflows. Build files are still written in the Starlark language (using the Rust-Starlark interpreter we wrote about earlier). The rules (cxx_library, etc) are the same and mostly have the same behavior. The higher-level concepts such as rules, goals, actions are all adopted.
As for the big changes, our next generation build system is based on a single framework for incremental computations. This provides fine-grained computation dependency tracking and very high performance for incremental computations. It’s heavily inspired by research like Adapton and implementations like Salsa, Skip, and the Shake Build System.
The limited expandability of Buck v1 has always been a source of pain for our community. With this new build system we have focused on extensibility as the main goal and each build rule is written as a custom rule outside of the core build language.
While we were pondering this recast, we took the opportunity to experiment and validate Rust as a tech stack for the build system. Facebook is increasingly investing in the Rust programming language and is used in many projects with great success. We found the language to be a good fit for many reasons including:
- Rust’s async / await syntax makes writing asynchronous code really smooth, and Rust helps fix the complex details of concurrency. Any migration from Buck’s Java computations to concurrent computation has been a difficult transition for months, and there are still significant single-threaded bottlenecks.
- Rust has many high-level language features that make development easier and more enjoyable. These are things like enumerations, pattern matching, traits, procedural macros, and all the other features that make Rust developers love it so in general.
- Rust provides better control over memory allocations. Garbage collection languages (even with intergenerational collection) have challenges in dealing with incremental computations like the one Buck does.
- Rust is top performance. We have seen significant accelerations from translating things into Rust.
Share the next generation build system with the community
We are eager to share this new work and project with the community. The prep process is ongoing, but we have released part of it with the Rust Starlark library, and we plan to release other components as reusable libraries, such as our central incremental calculation engine and our “Super Console” textui framework . soon.
You may see less progress at Buck over the next few months, but rest assured that we will continue to work to bring the best build system to the community. We understand that an important part of the sharing process is defining a smooth transition for Buck’s users and making sure the community can move forward with us. We plan to make this publicly available by January 2022, when we will have more details on the transition for current Buck users.
What does this mean for buck users?
For the time being, not much will change from the current state. At the beginning of last year we announced that the active development of Buck was leaving the stable main branch. We will continue to support and maintain the stable branch, and we plan to continue doing so for at least a year after the new system is released. For most users, the transition should be seamless, and we provide support and tools for more complex cases.
Some may have noticed that even in Buck’s development branch, development has and will continue to slow down as we focus more on the new build system. The use of the dev branch in production is still discouraged and will have less support than the stable branch.
We anticipate we’ll be discussing more about this next generation build system later this year. You can always follow our work on Facebook’s open source blog and in our Github repositories.
To learn more about Facebook Open Source, visit our open source site, subscribe to our YouTube channel, or follow us on Twitter and Facebook.