Our open supply construct system
- Buck2, our new large scale open source build systemis now available on GitHub.
- Buck2 is an extensible and high-performance build system written in Rust and designed to make your build experience faster and more efficient.
- In our internal testing at Meta, we found that Buck2 completes builds twice as fast as Buck1.
Buck2, Meta’s large-scale, open-source build system, is now publicly available through the Buck2 website And the buck2 GitHub repository. While it shares some similarities with other build systems (like Buck1 and Bazel), Buck2 is a version written from scratch. Buck2 offers full separation of core and language-specific rules with increased concurrency, integration with remote execution and virtual file systems, and a redesigned console output. All of these changes aim to help engineers and developers spend less time waiting and more time iterating on their code.
Thousands of developers at Meta are already using Buck2 and performing millions of builds per day, completing builds twice as fast as Buck1. Our own internal analysis showed that engineers were able to produce significantly more code when their builds were run by Buck2, and we hope the wider industry will benefit as well.
Why rebuild Buck?
Build systems stand between a programmer and the execution of their code, so anything we can do to make the experience faster or more productive directly impacts how effective a developer can be. Buck2’s goal was to keep what we liked about Buck1 (the core concepts and workflows), be inspired by post-Buck1 innovations (including Basel, AdjustmentAnd shake) and focus on speed and enabling new experiences.
Buck2’s design is based on the following principles:
- The core build system has no language-specific rules. Separating the rules from the core means the rules are easier to change and understand. The core of Buck2 is written in Rust, and its language rules (e.g. how to build C++) are written in it Starlark. This separation is in contrast to Buck1 (where all rules are written in the core) and Bazel (where C++/Java are written in the core).
- The build system is supported by a single incremental dependency graph, avoiding any phases (unlike Buck1 or Bazel). This decision eliminates many types of bugs and increases concurrency.
- The Rules API is designed to include advanced capabilities, along with dynamic (or monadic) dependency functions for expressibility. At the same time, these features are carefully constrained to ensure that other properties (such as fast queries or hermeticity) are not compromised.
- The open source version is almost identical to our internal version. The only parts that are exchanged are the toolchains (which point to our compilers’ internal copies) and the remote execution (which points to our internal servers). — Both offer open source alternatives. We also publish all rules exactly as they are used internally. In addition, we have put some of the logical components in separate boxes (e.g Starlark, superconsole, allocative, pavilion) so they can be used outside of Buck2.
- Buck2 is written to integrate with remote execution, with the ability to perform actions on remote computers. We use the same API like Baseland have tested the remote execution with build barn And EngFlow. While not required (and not really expected for people starting with the open source version), we are able to efficiently compute recursive digests and efficiently send them to remote execution.
- Buck2 is written to integrate with virtual file systems, which doesn’t check out the entire repository but gets it on-demand when the files are accessed. In particular, we support Sapling-based file systems. For a good integration we pay attention to file notifications (with Guardian) and request both files and file digests without direct file operations. The benefit is that we can make virtual filesystems as fast as a full checkout, but with the benefits of a much faster checkout and much lower disk usage.
The key takeaway from all of these improvements is that we designed Buck2 to be fast. In practice, Buck2 is significantly faster than Buck1, depending on the build. If there are no source code changes, Buck2 is available almost immediately on subsequent builds. When busy, Buck2 starts execution faster and has greater concurrency. This increase in speed is a result of many of the factors mentioned above, as well as care and attention.
The user view
For end-users, Buck2 works much the same as Buck1 (which, to a first approximation, is quite similar to Bazel). A user defines goals in a DESIRE File:
rust_binary( name = “my_binary”, srcs = [“main.rs”]deps = [“:my_library”])
A user can then build with Build Buck2 //:my_binary. The value main.rs is a source file, and :my library is a dependency defined therein DESIRE File. It’s worth noting that Buck2 is mostly compatible with the DESIRE Files from Buck1.
In addition to the speed increase, there are two other important user-visible differences compared to Buck1.
First, the console output has been redesigned Super Console Library, which we developed specifically for Buck2. The console shows a few more details and feels a lot nicer to use:
Second, there is a persistent daemon that maintains a single dependency graph. If you change a DESIRE file, a dependency or a source file, we invalidate the appropriate things in the dependency graph and then request the output artifacts from the command line. There are several distinct dependency graphs in Buck1, leading to phases such as building the target graph, building the action graph, and then executing the action graph. There are also some operations that are not performed on the chart. When certain things change in Buck1, whole charts are thrown away instead of invalidating the minimum pieces. With a single dependency graph, Buck2 is simpler, avoids more redundant work, and avoids explicit phases. Everything in the dependency graph has a key (as it is identified) and a value, along with a function to compute the value from the key and other related keys (following the model in the paper “Build systems a la carte”).
The rule author’s view
While the user model follows Buck1 very closely, the approach to rules is completely different. At Buck, for example, there are many rules rust_binary used above. While a rule in Buck1 was a Java class baked into Buck1, a rule in Buck2 is completely decoupled. Buck2 also ships with a “prelude” of rules that implement most of the Buck1 rules.
Buck1 rules have been optimized over time, had many performance optimizations and powerful features like graph traversal, but these rules were also expected to obey many complex invariants—sometimes break those rules. For Buck2, the Rules API resides entirely in Starlark, which forced us to abstract these features as generic reusable APIs to make them safe, expressive, and powerful—a tricky balance. We will address two such examples.
The dependency structure of the OCaml library is difficult to express in Buck1. An OCaml library consists of a set of OCaml files. These must be compiled in dependency order—So if Aml Used B.mlyou need to compile B.ml First. Bazel requires dependency on Aml At B.ml write explicitly in the DESIRE File. Buck1 and Buck2 both leave this internal dependency implicit and run the tool ocamldep to infer what requires less maintenance as the structure changes. What Buck1 did is done ocamldep right after parsing the DESIRE File not really allowed and dependencies were not tracked. So if you changed the imports too much, Buck1 would result in spurious compilation errors. With Buck2 we can use those new primitive dynamic_outputwhich allows you to run a command, read the output of the file, and then wire the rest of the chart—Inserting the correct dependencies between the .ml files automatically.
C++ link dependencies
Consider the C++ linking model: To build a library, you typically need to link its build output along with transitively closing the build output of its dependencies. If you simply duplicate the set of dependencies at each level as you move up the diagram, you end up with At2) memory usage. In Buck1, many rules had custom code to capture this pattern, which relied on the ability to share Java values in memory and represent the dependencies in place within the rule structure (since there was no reified dependency graph) . There are much stronger abstraction boundaries in Buck2, so such reuse needs to be made more explicit. That’s why we introduced transitive sets (tsets) to capture this pattern of sets that represent a transitive closure. By making tsets more abstract, we were also able to connect the tset directly to the underlying dependency graph, meaning that this representation is both memory and computationally efficient.
Try Buck2 now
We really want people to try Buck2 and we would appreciate any feedback (GitHub issues are the best way). We assume that Buck2 will be most interesting for medium-sized multilingual projects. Visit the Buck2 Getting Started Page for more informations.
Comments are closed.