Thousands of developers at Meta work in repositories with millions of files every day. These developers need tools that help them at every stage of the workflow while working at extremely large scale. In this article, we’ll go through some of the tools in the development process. And as an added bonus, the ones we’re talking about below are open source, so you can try them out for yourself.
Sapling: Scaling version control
seedling is a version control system that can scale to large sizes, but also values ease of use. Sapling consists of three main components: a server, a client and a virtual file system.
The server stores all data and is a careful mix of clever storage formats, transmission protocols and algorithms. mostly implemented in Rust And designed to scale.
The client then communicates with this server and performs all known operations (check out, rebase, commit, modify, etc.). In addition, the client is also able to communicate with a Git server, so open source GitHub repositories can be edited with ours Open source Sapling release.
The last component is the virtual file system. When checking out a repository of our size, the disk I/O alone to write all the files can take a long time. A solution is sparse coffers, where a developer pre-specifies which subset of the repo they want to see. A more ergonomic alternative is EdenFS, which checks everything out in a matter of seconds, but then actually downloads the files from the server only when they’re accessed.
Buck2: build system
After changes are made, many developers at Meta Buck2 to compile the results and test your changes. Buck2 is designed for large-scale deployment and supports remote caching and execution, so developers can share each other’s compilations and a single developer can have access to thousands of machines to run compilations in parallel. Buck2 is also designed to support multiple programming languages simultaneously. So if you want your OCaml program to depend on a Rust library that uses a C++ library whose source code was generated by Erlang, that works fine.
Buck2 works well without Sapling, but requires special design considerations to enable Sapling and EdenFS. Buck2 uses Watchman to find out which files have changed, and Watchman supports EdenFS so it can seamlessly integrate with off-disk files. Buck2 can also use certain EdenFS operations to access the file without going through disk, optimizing performance on systems where virtual file systems can be slower.
Infer, RacerD and Jest: testing and static analysis
Handwritten tests and static analysis play an important role in ensuring that all of our code is working as intended. Working with the amount of code we create at Meta means we need tools that can deliver a high quality signal, and do it very quickly.
For general static analysis we use a platform called Closewhich is interprocedural and supports multiple languages including Java and C++. We also have other customized analysis tools such as RacerD, which detects Java concurrency bugs. RacerD played a big role in our project Move Facebook’s News Feed from Single-Threaded to Multi-Threaded on Android.
We also have language specific testing frameworks. For example, Is is our javascript testing framework. In 2022 We’ve officially committed Jest to the OpenJS Foundation to further support its growth across the industry.
Finally, there are tools that sit somewhere between static analysis and manual test cases. Our Sapience Toolfor example, automatically tests mobile apps by allowing developers to simulate the user experience to detect crashes and other potential problems.
Learn more about Meta’s developer workflow
In addition to our open source tools, our developers also use a number of proprietary tools in their daily workflows. For example, Phabricator (Phab for short), our CI and review tool, helps our developers review and submit batches of differences. For more information on these tools (along with those covered above), see the article on Meta developer workflow.
Comments are closed.