• Marking a major investment in Meta’s AI future, we are announcing two 24k GPU clusters. We are sharing details on the hardware, network, storage, design, performance, and software that help us extract high throughput and reliability for various AI workloads. We use this cluster design for Llama 3 training.
• We are strongly committed to open compute and open source. We built these clusters on top of Grand Teton, OpenRack, and PyTorch and continue to push open innovation across the industry.
• This announcement is one step in our ambitious infrastructure roadmap. By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100 GPUs as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s.

To lead in developing AI means leading investments in hardware infrastructure. Hardware infrastructure plays an important role in AI’s future. Today, we’re sharing details on two versions of our 24,576-GPU data center scale cluster at Meta. These clusters support our current and next generation AI models, including Llama 3, the successor to Llama 2, our publicly released LLM, as well as AI research and development across GenAI and other areas .

A peek into Meta’s large-scale AI clusters

Meta’s long-term vision is to build artificial general intelligence (AGI) that is open and built responsibly so that it can be widely available for everyone to benefit from. As we work towards AGI, we have also worked on scaling our clusters to power this ambition. The progress we make towards AGI creates new products, new AI features for our family of apps, and new AI-centric computing devices. 

While we’ve had a long history of building AI infrastructure, we first shared details on our AI Research SuperCluster (RSC), featuring 16,000 NVIDIA A100 GPUs, in 2022. RSC has accelerated our open and responsible AI research by helping us build our first generation of advanced AI models. It played and continues to play an important role in the development of Llama and Llama 2, as well as advanced AI models for applications ranging from computer vision, NLP, and speech recognition, to image generation, and even coding.

Under the hood

Our newer AI clusters build upon the successes and lessons learned from RSC. We focused on building end-to-end AI systems with a major emphasis on researcher and developer experience and productivity. The efficiency of the high-performance network fabrics within these clusters, some of the key storage decisions, combined with the 24,576 NVIDIA Tensor Core H100 GPUs in each, allow both cluster versions to support models larger and more complex than that could be supported in the RSC and pave the way for advancements in GenAI product development and AI research.

Network

At Meta, we handle hundreds of trillions of AI model executions per day. Delivering these services at a large scale requires a highly advanced and flexible infrastructure. Custom designing much of our own hardware, software, and network fabrics allows us to optimize the end-to-end experience for our AI researchers while ensuring our data centers operate efficiently. 

With this in mind, we built one cluster with a remote direct memory access (RDMA) over converged Ethernet (RoCE) network fabric solution based on the Arista 7800 with Wedge400 and Minipack2 OCP rack switches. The other cluster features an NVIDIA Quantum2 InfiniBand fabric. Both of these solutions interconnect 400 Gbps endpoints. With these two, we are able to assess the suitability and scalability of these different types of interconnect for large-scale training, giving us more insights that will help inform how we design and build even larger, scaled-up clusters in the future. Through careful co-design of the network, software, and model architectures, we have successfully used both RoCE and InfiniBand clusters for large, GenAI workloads (including our ongoing training of Llama 3 on our RoCE cluster) without any network bottlenecks.

Compute

Both clusters are built using Grand Teton, our in-house-designed, open GPU hardware platform that we’ve contributed to the Open Compute Project (OCP). Grand Teton builds on the many generations of AI systems that integrate power, control, compute, and fabric interfaces into a single chassis for better overall performance, signal integrity, and thermal performance. It provides rapid scalability and flexibility in a simplified design, allowing it to be quickly deployed into data center fleets and easily maintained and scaled. Combined with other in-house innovations like our Open Rack power and rack architecture, Grand Teton allows us to build new clusters in a way that is purpose-built for current and future applications at Meta.

We have been openly designing our GPU hardware platforms beginning with our Big Sur platform in 2015.

Storage

Storage plays an important role in AI training, and yet is one of the least talked-about aspects. As the GenAI training jobs become more multimodal over time, consuming large amounts of image, video, and text data, the need for data storage grows rapidly. The need to fit all that data storage into a performant, yet power-efficient footprint doesn’t go away though, which makes the problem more interesting.

Our storage deployment addresses the data and checkpointing needs of the AI clusters via a home-grown Linux Filesystem in Userspace (FUSE) API backed by a version of Meta’s ‘Tectonic’ distributed storage solution optimized for Flash media. This solution enables thousands of GPUs to save and load checkpoints in a synchronized fashion (a challenge for any storage solution) while also providing a flexible and high-throughput exabyte scale storage required for data loading.

We have also partnered with Hammerspace to co-develop and land a parallel network file system (NFS) deployment to meet the developer experience requirements for this AI cluster. Among other benefits, Hammerspace enables engineers to perform interactive debugging for jobs using thousands of GPUs as code changes are immediately accessible to all nodes within the environment. When paired together, the combination of our Tectonic distributed storage solution and Hammerspace enable fast iteration velocity without compromising on scale.     

The storage deployments in our GenAI clusters, both Tectonic- and Hammerspace-backed, are based on the YV3 Sierra Point server platform, upgraded with the latest high capacity E1.S SSD we can procure in the market today. Aside from the higher SSD capacity, the servers per rack was customized to achieve the right balance of throughput capacity per server, rack count reduction, and associated power efficiency. Utilizing the OCP servers as Lego-like building blocks, our storage layer is able to flexibly scale to future requirements in this cluster as well as in future, bigger AI clusters, while being fault-tolerant to day-to-day Infrastructure maintenance operations.

Performance

One of the principles we have in building our large-scale AI clusters is to maximize performance and ease of use simultaneously without compromising one for the other. This is an important principle in creating the best-in-class AI models. 

As we push the limits of AI systems, the best way we can test our ability to scale-up our designs is to simply build a system, optimize it, and actually test it (while simulators help, they only go so far). In this design journey, we compared the performance seen in our small clusters and with large clusters to see where our bottlenecks are. In the graph below, AllGather collective performance is shown (as normalized bandwidth on a 0-100 scale) when a large number of GPUs are communicating with each other at message sizes where roofline performance is expected. 

Our out-of-box performance for large clusters was initially poor and inconsistent, compared to optimized small cluster performance. To address this we made several changes to how our internal job scheduler schedules jobs with network topology awareness – this resulted in latency benefits and minimized the amount of traffic going to upper layers of the network. We also optimized our network routing strategy in combination with NVIDIA Collective Communications Library (NCCL) changes to achieve optimal network utilization. This helped push our large clusters to achieve great and expected performance just as our small clusters.

In the figure we see that small cluster performance (overall communication bandwidth and utilization) reaches 90%+ out of the box, but an unoptimized large cluster performance has very poor utilization, ranging from 10% to 90%. After we optimize the full system (software, network, etc.), we see large cluster performance return to the ideal 90%+ range.

In addition to software changes targeting our internal infrastructure, we worked closely with teams authoring training frameworks and models to adapt to our evolving infrastructure. For example, NVIDIA H100 GPUs open the possibility of leveraging new data types such as 8-bit floating point (FP8) for training. Fully utilizing larger clusters required investments in additional parallelization techniques and new storage solutions provided opportunities to highly optimize checkpointing across thousands of ranks to run in hundreds of milliseconds.

We also recognize debuggability as one of the major challenges in large-scale training. Identifying a problematic GPU that is stalling an entire training job becomes very difficult at a large scale. We’re building tools such as desync debug, or a distributed collective flight recorder, to expose the details of distributed training, and help identify issues in a much faster and easier way

Finally, we’re continuing to evolve PyTorch, the foundational AI framework powering our AI workloads, to make it ready for tens, or even hundreds, of thousands of GPU training. We have identified multiple bottlenecks for process group initialization, and reduced the startup time from sometimes hours down to minutes. 

Commitment to open AI innovation

Meta maintains its commitment to open innovation in AI software and hardware. We believe open-source hardware and software will always be a valuable tool to help the industry solve problems at large scale.

Today, we continue to support open hardware innovation as a founding member of OCP, where we make designs like Grand Teton and Open Rack available to the OCP community. We also continue to be the largest and primary contributor to PyTorch, the AI software framework that is powering a large chunk of the industry.

We also continue to be committed to open innovation in the AI research community. We’ve launched the Open Innovation AI Research Community, a partnership program for academic researchers to deepen our understanding of how to responsibly develop and share AI technologies – with a particular focus on LLMs.

An open approach to AI is not new for Meta. We’ve also launched the AI Alliance, a group of leading organizations across the AI industry focused on accelerating responsible innovation in AI within an open community. Our AI efforts are built on a philosophy of open science and cross-collaboration. An open ecosystem brings transparency, scrutiny, and trust to AI development and leads to innovations that everyone can benefit from that are built with safety and responsibility top of mind. 

The future of Meta’s AI infrastructure

These two AI training cluster designs are a part of our larger roadmap for the future of AI. By the end of 2024, we’re aiming to continue to grow our infrastructure build-out that will include 350,000 NVIDIA H100s as part of a portfolio that will feature compute power equivalent to nearly 600,000 H100s.

As we look to the future, we recognize that what worked yesterday or today may not be sufficient for tomorrow’s needs. That’s why we are constantly evaluating and improving every aspect of our infrastructure, from the physical and virtual layers to the software layer and beyond. Our goal is to create systems that are flexible and reliable to support the fast-evolving new models and research. 

By now you’re already aware that Python 3.12 has been released. But did you know that several of its new features were developed by Meta?

Meta engineer Pascal Hartig (@passy) is joined on the Meta Tech Podcast by Itamar Oren and Carl Meyer, two software engineers at Meta, to discuss their teams’ contributions to the latest Python release, including new hooks that allow for custom JITs like Cinder, Immortal Objects, improvements to the type system, faster comprehensions, and more.

Learn how and why they built these new features for Python and how they worked with and engaged with the Python community.

Download or listen to the episode below:

You can also find the episode wherever you get your podcasts, including:

Spotify
Apple Podcasts
PocketCasts
Castro
Overcast

The Meta Tech Podcast is a podcast, brought to you by Meta, where we highlight the work Meta’s engineers are doing at every level – from low-level frameworks to end-user features.

Send us feedback on Instagram, Threads, or X.

And if you’re interested in learning more about career opportunities at Meta visit the Meta Careers page.

• While deploying Precision Time Protocol (PTP) at Meta, we’ve developed a simplified version of the protocol (Simple Precision Time Protocol – SPTP), that can offer the same level of clock synchronization as unicast PTPv2 more reliably and with fewer resources.
• In our own tests, SPTP boasts comparable performance to PTP, but with significant improvements in CPU, memory, and network utilization.
• We’ve made the source code for the SPTP client and server available on GitHub.

We’ve previously spoken in great detail about how Precision Time Protocol is being deployed at Meta, including the protocol itself and Meta’s precision time architecture.

As we deployed PTP into one of our data centers, we were also evaluating and testing alternative PTP clients. In doing so, we soon realized that we could eliminate a lot of complexity in the PTP protocol itself that we experienced during data center deployments while still maintaining complete hardware compatibility with our existing equipment.

This is how the idea of Simple Precision Time Protocol (SPTP) was born. 

But before we dive under the hood of SPTP we should explore why the IEEE 1588 G8265.1 and G8275.2 unicast profiles (here, we just call them PTP) weren’t a perfect fit for our data center deployment.

PTP and its limitations

Excessive network communication

A typical IEEE 1588-2019 two-step PTPv2 unicast UDP flow consists of the following exchange:

Figure 1: Typical two-step PTPv2 exchange.

This sequence repeats either in full or in part depending on the negotiation result. The exchange shown is one of many possible combinations. It may involve additional steps such as grant cancellation, grand cancellation acknowledgements, and so on.

The frequency of these messages may vary depending on the implementation and configuration. After completing negotiation, the frequency of some messages can change dynamically.

This design allows for a lot of flexibility, especially for less powerful equipment where resources are limited. In combination with multicast, it allows us to support a relatively large number of clients using either very old or embedded devices. For example, a PTP server can reject the request or confirm a less frequent exchange if the resources are exhausted.

This design, however, leads to excessive network communication, which is particularly visible on a time appliance serving a large number of clients.

State machine

Due to the “subscription” model, both the PTP client and the server have to keep the state in memory. This approach comes with the tradeoffs such as:

• Excessive usage of resources such as memory and CPU.
• Strict capacity limits that mean multicast support is required for large numbers of clients.
• Code complexity.
• Fragile state transitions.

These issues can manifest, for example, in so-called abandoned syncs – situations where the work of a PTP client is interrupted (either forcefully stopped or crashed). Because the PTP server didn’t receive a cancellation signaling message it will keep sending sync and followup packets until the subscription expires (which may take hours). This leads to additional complexity and fragility in the system. 

 There are additional protocol design side effects such as:

• An almost infinite Denial of Service Attack (DoS) amplification factor.
• Server-driven communication with little control by the client.
• Complete trust in the validity of server timestamps.
• Asynchronous path delay calculations.

In data centers, where communication is typically driven by hundreds of thousands of clients and multicast is not supported, these tradeoffs are very limiting. 

SPTP

True to its name, SPTP significantly reduces the number of exchanges between a server and client, allowing for much more efficient network communication.

Exchange

Figure 2: Typical SPTP exchange.

In a typical SPTP exchange:

1. The client sends a delay request.
2. The server responds with a sync.
3. The server sends a followup/announce.

The number of network exchanges is drastically reduced. Instead of 11 different network exchanges as shown on Figure 1 and the requirement for client and server state machines for the duration of the subscription, there are only three packets exchanged and no state needs to be preserved on either side. In the simplified exchange, every packet has an important role:

Delay request

A delay request initiates the SPTP exchange. It’s interpreted by a server not only as a standard delay request containing the correction field (CF1) of the transparent clock, but also as a signal to respond with sync and followup packets. Just like in a two-step PTPv2 exchange, it generates T3 upon departure from the client side and T4 upon arrival on the server side.

To distinguish between a PTPv2 delay request and a SPTP delay request, the PTP profile Specific 1 flag must be set by the client.

Sync

In response to a delay request, a sync packet would be sent containing the T4 generated at an earlier stage. Just like in a regular two-step PTPv2 exchange, a sync packet will generate a T1 upon departure from the server side. While in transit, the correction field of the packet (CF2) is populated by the network equipment.

Followup/announce

Following the sync packet, an announce packet is immediately sent containing T1 generated at a previous stage. In addition, the correction filed from the Delay Request field is populated by the CF1 value collected at an earlier stage.  

The announce packet also contains typical PTPv2 information such as clock class, clock accuracy, and so on. On the client side, the arrival of the packet generates the T2 timestamp.

After a successful SPTP exchange, default two-step PTPv2 formulas for mean path delay and clock offset must be applied:

mean_path_delay = ((T4 – T3) + (T2-T1) – CF1 -CF2)/2

clock_offset = T2 – T1 – mean_path_delay

After every exchange the client has access to the announce message attributes such as time source, clock quality, etc., as well as the path delay and a calculated clock offset after every exchange with every server. And, because the exchange is client-driven, the offsets could be calculated at the exact same time. This avoids a situation where a client is following a faulty server and has no chance of detecting it.

Figure 3: Client following faulty Time Server 2 based on announce.

Reliability

We can also provide stronger reliability guarantees by using multi-clock reliance.

In our implementation for precision time synchronization, we provide time as well as a window of uncertainty (WOU) to the consumer application via the fbclock API. As we described in a previous blog post on how PTP is being deployed at Meta the WOU is based on the observation of time sync errors for the minimum duration to have stationarity of the state of the system. 

In addition, we’ve established a method based on a collection of clocks that each client can access for timing information that we call a clock ensemble. The clock ensemble operates in two modes, steady state and transient; where steady state is during normal operation and transient is in the case of holdover.

However, with a pool of N clocks, C, forming the clock ensemble, the question becomes which clocks to select for determining robustness and accurate timing information. Clocks that are not accurate are rejected (C_reject) and, thus, our ensemble size falls to N = C_total –  C_reject. We employ two stages, one that is based on each individual clock, and the second that acts on the collection of valid clocks in the ensemble. 

The first stage observes the previous measurements of each individual clock, where the main criteria is to reject outliers in the previous states of the clock. Once this criterion threshold is exceeded, the entire clock is rejected from the valid clock ensemble pool. This is based off Chauvenet’s criterion, where the criterion is a probability band that is centered on the mean of the clock outputs (assuming a normal distribution during steady state). Based on the stationarity tests, we use a sample size of 400 previous clock outputs and calculate a maximum allowable deviation. 

For example:

, where is the current clock output,  is the clock sample mean, and is the clock set standard deviation.

We find the probability that the current clock output is in disagreement with the previous 400 samples:

Based on a window size of 400 previous samples, the maximum allowed deviation is:

Now, the clock outputs are tested against this value. If they exceed the  they are rejected, an alert is raised, and a threshold counter is incremented. Once the rejection threshold is reached for an individual clock, this clock is entirely rejected.

Now, we enter the second stage of verifying the clock ensemble composed of the valid clocks. The second stage forms a weighted average of the non-rejected clocks in the valid clock ensemble, where each clock in the ensemble is reported as its sample size, mean, and variance. The average of the clocks’ means is the weighted average, where the weights are inversely proportional to the mean absolute deviations reported by each clock after applying Chauvenet’s criterion. 

Now we can report the mean and variance of the clock ensemble, ensuring the clocks contained therewith are valid and not providing erroneous values. The confidence interval is scaled with the number of good clocks in the ensemble, where the higher the number of valid clocks out of the total clocks provides greater reliability.

For a number of hosts, we show that the distribution of clocks falls within the following heatmap:

Figure 4: Offset distribution overlay of multiple clocks.

We calculate the variance, , of each individual clock’s observations, then we calculate a weighted mean, , taking into consideration the reciprocal of each clock’s variance as the weight.

Due to independence of clocks, the variance of the weighted sum, , is:

In summary, we collect samples from a number of clock sources that form our clock ensemble. The overall precision and reliability of the provided data by SPTP is a function of the number of reliable and in distribution clocks forming the clock ensemble.

A future post will focus on this specifically. 

SPTP’s performance

Let’s explore performance of the SPTP versus PTP.

Initial deployments to a single client confirmed no regression in the precision of the synchronization:

Figure 5: Clock offset after switching from ptp4l and SPTP.

Repeating the same measurement after migration to SPTP produces a very similar result, only marginally different due to a statistical error:

Figure 6: P99.99 offset collected from over 100000 SPTP clients.

With large-scale deployment of our implementations, we can confirm resource utilization improvements.

We noticed that due to the difference in multi-server support, the performance gains vary significantly depending on the number of tracked time servers.

For example, with just a single time appliance serving the entire network there are significant improvements across the board. Most notably over 40 percent CPU, 70 percent memory, and 50 percent network utilization improvements:

Figure 7: Packets per second with ptp4l (green) vs SPTP (blue).

The next steps for SPTP at Meta

Since SPTP can offer the exact same level of synchronization with a lot fewer resources consumed, we think it’s a reasonable alternative to the existing unicast PTP profiles.

In a large-scale data center deployment, it can help to combat frequently changing network paths and create savings in terms of network traffic, memory usage, and number of CPU cycles.

It will also eliminate a lot of complexity inherited from multicast PTP profiles, which is not necessarily useful in the trusted networks of the modern data centers.

It should be noted that SPTP may not be suitable for systems that still require subscription and authentication. But this could be solved by using PTP TLVs (type-length-value). 

Additionally, by removing the need for subscriptions, it’s possible to observe multiple clocks – which allows us to provide higher reliability by comparing the time sync from multiple sources at the end node.

SPTP can offer significantly simpler, faster, and more reliable synchronization. Similar to G.8265.1 and G.8275.2 it provides excellent synchronization quality using a different set of parameters. Simplification comes with certain tradeoffs, such as missing signaling messages, that users need to be aware of and decide which profile is the best for them.

Having it standardized and assigned a unicast profile identifier will encourage wider support, adoption, and popularization of PTP as a default precise time synchronization protocol.

The source code for the SPTP client and the server can be accessed on our GitHub page.

Acknowledgements

We would like to thank Alexander Bulimov, Vadim Fedorenko, and Mike Lambeta for their help implementing the code and the math for this article.

• We’ve open sourced DotSlash, a tool that makes large executables available in source control with a negligible impact on repository size, thus avoiding I/O-heavy clone operations.
• With DotSlash, a set of platform-specific executables is replaced with a single script containing descriptors for the supported platforms. DotSlash handles transparently fetching, decompressing, and verifying the appropriate remote artifact for the current operating system and CPU.
• At Meta, the overwhelming majority of DotSlash files are generated and committed to source control via automation, so we are also releasing a complementary GitHub Action to assemble a comparable setup outside of Meta.
• DotSlash is written in Rust for performance and is cross-platform.

At Meta, we have a vast array of first-party and third-party command line tools that need to be available across a diverse range of developer environments. Reliably getting the appropriate version of each tool to the right place can be a challenging task.

For example, the source code for many of our first-party tools lives alongside the projects that leverage them inside our massive monorepo. For such tools, the standard practice is to use buck2 run to build and run executables from source, as necessary. This has the advantage that tools and the projects that use them can be updated atomically in a single commit.

While we use extensive caching and remote execution to provide our developers with fast builds, there will always be cases where buck2 run is going to be considerably slower than running the prebuilt binary directly. While we leverage a virtual filesystem that reduces the drawbacks of checking large binaries into source control compared to a traditional physical filesystem, there are still pathological cases that are best avoided by keeping such files out of the repository in the first place. (This practice also eliminates a large class of code provenance issues.)

Further, not everything we use is built from source, nor do all of our tools live in source control. For example, there is the case of buck2 itself, which needs to be pre-built for developers and readily available on the $PATH for convenience. For core developer tools like Buck2 and Sapling, we use a Chef recipe to deploy new versions, installing them in /usr/local/bin (or somewhere within the appropriate %PATH$% on Windows) across a variety of developer environments.

While this approach is reasonable for commonly-used executables, it is not a great fit for the long tail of tools. That is, while it might be convenient to install everything a developer might need in /usr/local/bin by default, this could easily add up to tens or hundreds of gigabytes of disk, very little of which will end up being executed, in practice. In turn, this makes Chef runs more expensive and prone to failure.

Introducing DotSlash

DotSlash attempts to solve many of the problems described in the previous section. While we do not claim it is a silver bullet, we have found it to be the right solution for many of our internal use cases. At Meta, DotSlash is executed hundreds of millions of times per day to deliver a mix of first-party and third-party tools to end-user developers as well as hermetic build environments.

The idea is fairly simple: we replace the contents of a set of platform-specific, heavyweight executables with a single lightweight text file that can be read by the dotslash command line tool (which must be installed on the user’s $PATH). We call such a file a DotSlash file. It contains the information DotSlash needs to fetch and run the executable it replaces for the host platform. By convention, a DotSlash file maintains the name of the original file rather than calling attention to itself via a custom file extension. Instead, it aspires to be a transparent wrapper for the original executable. To that end, a DotSlash file is required to start with #!/usr/bin/env dotslash (even on Windows) to help maintain this illusion.

The following is a hypothetical DotSlash file named node that is designed to run v18.19.0 of Node.js. Note that users across x86 Linux, x86 macOS, and ARM macOS can all run the same DotSlash file, as DotSlash will take care of doing the work to select the appropriate executable for the host on which it is being run. In this way, DotSlash simplifies the work of cross-platform releases: 

In this example, the workflow DotSlash runs through when executing node looks like: 

See the How DotSlash Works documentation for details.

Because of how #! works on Mac and Linux, when a user runs ./node –version, the invocation effectively becomes dotslash ./node –version. DotSlash requires that its first argument is a file that starts with #!/usr/bin/env dotslash, as mentioned above. Once it verifies the header, it uses a lenient JSON parser to read the rest of the file. DotSlash finds the entry in the “platforms” section that corresponds to the host it is running on.

DotSlash uses the information in this entry and hashes it to compute a corresponding file path (that doubles as a key) in the user’s local DotSlash cache. DotSlash attempts to exec the corresponding file, replacing argv0 with the path to the DotSlash file and forwarding the remaining command line arguments (–version, in this example) to the exec invocation.

If the target executable is in the cache, the user immediately runs Node.js as originally intended. In the event of a cache miss (indicated by exec failing with ENOENT), DotSlash uses the information from the DotSlash file to determine the URL it should use to fetch the artifact containing the executable as well as the size and digest information it should use to verify the contents. If this succeeds, the verified artifact is atomically mv‘d into the appropriate location in the DotSlash cache and the exec invocation is performed again. Note that DotSlash uses advisory file locking to avoid making duplicate requests even if DotSlash files requiring the same artifact are run concurrently.

Note that it is common to have multiple DotSlash files refer to the same artifact, such as a .tar.zst file, while each DotSlash file maps to a distinct entry within the archive. For example, suppose node-v18.19.0-darwin-arm64.tar.gz is a compressed tar file that contains many entries, including node , npm , and npx. The DotSlash file for node would be as follows:

#!/usr/bin/env dotslash

{
“name”: “node-v18.19.0”,
“platforms”: {
“macos-aarch64”: {
“size”: 40660307,
“hash”: “blake3”,
“digest”: “6e2ca33951e586e7670016dd9e503d028454bf9249d5ff556347c3d98c347c34”,
// Note the difference from the previous example where “format”: “zst” has been
// replaced with “format”: “tar.gz”, which specifies what type of decompression
// logic to use as well as the path within the decompressed archive to run when
// this DotSlash file is executed.
“format”: “tar.gz”,
// Assuming node-v18.19.0-darwin-arm64.tar.gz contains node, npm, and npx in the
// node-v18.19.0-darwin-arm64/bin/ folder within the the archive, the following
// is the only line that has to change in the DotSlash file that represents
// those other executables.
“path”: “node-v18.19.0-darwin-arm64/bin/node”,
“providers”: [
{
“url”: “https://nodejs.org/dist/v18.19.0/node-v18.19.0-darwin-arm64.tar.gz”
}
]
},
/* other platforms omitted for brevity */
}
}

As noted in the comments, the only change in the DotSlash files for npm and npx would be the “path” entry. Because the artifact for all three DotSlash files would be the same, whichever DotSlash file was run first would fetch the artifact and put it in the cache whereas all subsequent runs of any of the three DotSlash files would leverage the cached entry.

This technique is often used to ensure that a set of complementary executables is released together. Further, because the archive will be decompressed in its own directory, it may also contain resource files (or library files, such as .dll files that need to live alongside .exe files on Windows) that will be unpacked using the directory structure specified by the archive. This also makes DotSlash a good fit for distributing executables that are not binaries, but trees of script files, which is common for Node.js or Python.

Generating DotSlash files

At Meta, most DotSlash files are produced as part of an automated build pipeline. Our continuous integration (CI) system supports special configuration for DotSlash jobs where a user must specify:

• A set of builds to run (these can span multiple platforms).
• The resulting generated artifacts to publish to an internal blobstore.
• The DotSlash files in source control to update with entries for the new artifacts.
• The conditions under which the job should be triggered (this is analogous to workflow triggers on GitHub).

The result of such a job is a proposed change to the codebase containing the updated DotSlash files. At Meta, we call such a change a “diff,” though on GitHub, this is known as a pull request. Just like an ordinary human-authored diff at Meta, putting it up for review triggers a number of jobs that include linters, automated tests, and other tools that provide signal on the proposed change. For a DotSlash diff, if all of the signals come back clean, the diff is automatically committed to the codebase without further human intervention.

See the Generating DotSlash Files at Meta documentation for details.

The script we use to generate DotSlash files injects metadata about the build job that makes it straightforward to trace the provenance of the underlying artifacts. The following is a hypothetical example of a generated DotSlash file for the CodeCompose LSP built from source at a specific commit in clang-opt mode. Note the “metadata” entries in the DotSlash file will be ignored by the dotslash CLI, but we include them as structured data so they can be parsed by other tools to facilitate programmatic audits:

#!/usr/bin/env dotslash

// @generated SignedSource<>
// https://yarnpkg.com/package?name=signedsource can be used to
// generate and verify the above signature to flag tampering
// in generated code.

{
“name”: “code-compose-lsp”,
// Added by automation.
“metadata”: {
“build-info”: {
“job-repo”: “fbsource”,
“job-src”: “dotslash/code-compose-lsp.star”,
// It is considered best practice to build the artifacts for
// all platforms from the same commit within a DotSlash file.
“commit”: {
“repo”: “fbsource”,
“scm”: “sapling”,
“hash”: “0f9e3d9e189bf393f7f9d0b6879361cd76fcdcd0”,
“date”: “2024-01-03 20:07:54 PST”,
“timestamp”: 1704341274
}
}
},
“platforms”: {
“linux-x86_64”: {
“size”: 2740736,
“hash”: “blake3”,
“digest”: “fc8a3ade56a97a6e73469ade1575e8f8e33fda99fbf6df429d555e480d6453d0”,
“format”: “zst”,
“providers”: [
{
“type”: “meta-cas”,
“key”: “fc8a3ade56a97a6e73469ade1575e8f8e33fda99fbf6df429d555e480d6453d0:2740736”
}
]
// Added by automation.
“metadata”: {
“build-command”: [
“buck2”,
“build”,
“–config-file”,
“//buildconfig/clang-opt”,
“//codecompose/lsp/cli:code-compose-lsp”
]
}
},
// additional platforms…
}
}

Without DotSlash, a developer would have to run buck2 build –config-file //buildconfig/clang-opt //codecompose/lsp/cli:code-compose-lsp to build and run the LSP from source, which could be a slow operation depending on the size of the build, the state of the build cache, etc. With DotSlash, the developer can run the optimized LSP as quickly as they can fetch and decompress it from the specified URL, which is likely much faster than doing a build.

Another thing you may have noticed about this example is that the “key” is not an ordinary URL, but an identifier that happens to be the concatenation of the BLAKE3 hash and the size of the specified artifact. This is because “type”: “meta-cas” indicates that this artifact must be fetched via a custom provider in DotSlash, which is specialized fetching logic built into DotSlash that has its own identifier scheme. In this case, the artifact would be fetched from Meta’s in-house content-addressable storage (CAS) system, which uses the artifact hash+size as a key.

While we do not provide the code for the meta-cas provider in the open source version of DotSlash, we do include one custom provider out-of-the-box beyond the default http provider.

Using DotSlash with GitHub releases

While DotSlash is generally useful for fetching an executable from an arbitrary URL and running it, we have found the combination of DotSlash and CI to be particularly powerful. To that end, we include custom tooling to facilitate generating DotSlash files for GitHub releases. To ensure DotSlash can fetch artifacts from private GitHub repositories as well as GitHub Enterprise instances, DotSlash includes a custom provider for GitHub releases that includes an appropriate authentication token when fetching artifacts.

For example, suppose you have existing workflows for building your release artifacts and publish them via gh release upload. For simplicity, let’s assume these are named linux-release, macos-release, and windows-release. To create a single DotSlash file that includes the artifacts from all three platforms you would introduce a new GitHub Action that leverages the workflow_run trigger so it fires whenever one of these release workflows succeeds. (Note that GitHub’s documentation states: “You can’t use workflow_run to chain together more than three levels of workflows,” so check the depth of your workflow graph if your workflow is not firing.)

The .yml file to define the new workflow would look like this:

name: Generate DotSlash File

on:
workflow_run:
# These must match the names of the workflows that publish
# artifacts to your GitHub Release.
workflows: [linux-release, macos-release, windows-release]
types:
– completed

jobs:
create-dotslash-file:
name: Generating DotSlash File
runs-on: ubuntu-latest
if: ${{ github.event.workflow_run.conclusion == ‘success’ }}
steps:
– uses: facebook/dotslash-publish-release@v1
env:
# This is necessary because the action uses
# `gh release upload` to publish the generated DotSlash file(s)
# as part of the release.
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
# Additional file that lives in your repo that defines
# how your DotSlash file(s) should be generated.
config: .github/workflows/dotslash-config.json
# Tag for the release to to target.
tag: ${{ github.event.workflow_run.head_branch }}

Because inputs to GitHub Actions are limited to string values, facebook/dotslash-publish-release takes config, which is a path to a JSON file in the repo that supports a rich set of configuration options for generating the DotSlash files. The other required input is the ID of the release, which in GitHub, is defined by a Git tag. When the action is run, it will check to see whether all of the artifacts specified in the config are present in the release, and if so, will generate the appropriate DotSlash files and add them to the release.

For example, consider an open source project like Hermes where a release includes a number of platform-specific .tar.gz files, each containing a handful of executables (hermes, hdb, etc.). To create a separate an individual DotSlash file for each executable, the JSON configuration for the action would be:

{
“outputs”: {

“hermes”: {
“platforms”: {
“macos-x86_64”: {
“regex”: “^hermes-cli-darwin-“,
“path”: “hermes”
},
“macos-aarch64”: {
“regex”: “^hermes-cli-darwin-“,
“path”: “hermes”
},
“linux-x86_64”: {
“regex”: “^hermes-cli-linux-“,
“path”: “hermes”
},
“windows-x86_64”: {
“regex”: “^hermes-cli-windows-“,
“path”: “hermes.exe”
}
}
},

“hdb”: {
“platforms”: {
“macos-x86_64”: {
“regex”: “^hermes-cli-darwin-“,
“path”: “hdb”
},
“macos-aarch64”: {
“regex”: “^hermes-cli-darwin-“,
“path”: “hdb”
},
“linux-x86_64”: {
“regex”: “^hermes-cli-linux-“,
“path”: “hdb”
},
“windows-x86_64”: {
“regex”: “^hermes-cli-windows-“,
“path”: “hdb.exe”
}
}
},

// Additional entries for hvm, hbcdump, and hermesc…

}
}’

Each entry in “outputs” corresponds to the name of a DotSlash file that will be added to the release. The “platforms” for each entry defines the “platforms” that should be present in the generated DotSlash file. The action uses the “regex” to identify the file in the GitHub release that should be used as the backing artifact for the entry. Assuming the artifact is an “archive” of some sort (.tar.gz, .tar.zst, etc.), the “path” indicates the path within the archive that the DotSlash file should run.

In this particular case, Hermes does not provide an ARM-specific binary for macOS, so the “macos-aarch64” entry is the same as the “macos-x86_64”one. Though if that changes in the future, a simple update to “regex” to distinguish the two binaries is all that is needed.

Note that the action will take responsibility for computing the digest for each binary. In this example, the resulting DotSlash file for hermes would be:

#!/usr/bin/env dotslash

{
“name”: “hermes”,
“platforms”: {
“linux-x86_64”: {
“size”: 47099598,
“hash”: “blake3”,
“digest”: “8d2c1bcefc2ce6e278167495810c2437e8050780ebb4da567811f1d754ad198c”,
“format”: “tar.gz”,
“path”: “hermes”,
“providers”: [
{
“url”: “https://github.com/facebook/hermes/releases/download/v0.12.0/hermes-cli-linux-v0.12.0.tar.gz”
},
{
“type”: “github-release”,
“repo”: “facebook/hermes”,
“tag”: “v0.12.0”,
“name”: “hermes-cli-linux-v0.12.0.tar.gz”
}
],
},
// additional platforms…
}
}

Note that there are two entries in the “providers” section for the Linux artifact. When DotSlash fetches an artifact, it will try the providers in order until one succeeds. Regardless of which provider is used, the downloaded binary will be verified against the specified “hash”, “digest”,  and “size” values.

In this case, the first provider is an ordinary, public URL that can be fetched using curl –location, but the second is an example of a custom provider discussed earlier. The “type”: “github-release” line indicates that the GitHub provider for DotSlash should be used, which shells out to the GitHub CLI (gh, which must be installed separately from DotSlash) to fetch the artifact instead of curl. Because facebook/hermes is a public GitHub repository, the first provider should be sufficient here. However, if the repository were private and the fetch required authentication, we would expect the first provider to fail and DotSlash would fallback to the GitHub provider. Assuming the user had run gh auth login in advance to configure credentials for the specified repo, DotSlash would be able to fetch the artifact using gh release download.

By publishing DotSlash files as part of GitHub releases, users can copy them to their own repositories to “vendor in” a specific version of your tool with minimal effect on their repository size, regardless of how large your releases might be.

Try DotSlash Today 

Visit the DotSlash site for more in-depth documentation and technical details. The site includes instructions on Installing DotSlash so you can start playing with it firsthand. 

We also encourage you to check out the DotSlash source code and provide feedback via GitHub issues. We look forward to hearing from you!

• At Meta, the quest for faster model training has yielded an exciting milestone: the adoption of Lazy Imports and the Python Cinder runtime.
• The outcome? Up to 40 percent time to first batch (TTFB) improvements, along with a 20 percent reduction in Jupyter kernel startup times.
• This advancement facilitates swifter experimentation capabilities and elevates the ML developer experience (DevX).

Time is of the essence in the realm of machine learning (ML) development. The milliseconds it takes for an ML model to transition from conceptualization to processing the initial training data can dramatically impact productivity and experimentation.

At Meta, we’ve been able to significantly improve our model training times, as well as our overall developer experience (DevX) by adopting Lazy Imports and the Python Cinder runtime. 

The time to first batch challenge

Batch processing has been a game changer in ML development. It handles large volumes of data in groups (or batches) and allows us to train models, optimize parameters, and perform inference more effectively and swiftly.

But ML training workloads are notorious for their sluggish starts. When we look to improve our batch processing speeds, time to first batch (TTFB) comes into focus. TTFB is the time elapsed from the moment you hit the “start” button on your ML model training to the point when the first batch of data enters the model for processing. It is a critical metric that determines the speed at which an ML model goes from idle to learning. TTFB can vary widely due to factors like infrastructure overhead and scheduling delays. But reducing TTFB means reducing the development waiting times that can often feel like an eternity to engineers – waiting periods that can quickly amass as expensive resource wastage.

In the pursuit of faster TTFB, Meta set its sights on reducing this overhead, and Lazy Imports with Cinder emerged as a promising solution.

The magic of Lazy Imports

Previously, ML developers explored alternatives like the standard LazyLoader in importlib or lazy-import`, to defer explicit imports until necessary. While promising, these approaches are limited by their much narrower scope, and the need to manually select which dependencies will be lazily imported (often with suboptimal results). Using these approaches demands meticulous codebase curation and a fair amount of code refactoring.

In contrast, Cinder’s Lazy Imports approach is a comprehensive and aggressive strategy that goes beyond the limitations of other libraries and delivers significant enhancements to the developer experience. Instead of painstakingly handpicking imports to become lazy, Cinder simplifies and accelerates the startup process by transparently deferring all imports as a default action, resulting in a much broader and more powerful deferral of imports until the exact moment they’re needed. Once in place, this method ensures that developers no longer have to navigate the maze of selective import choices. With it, developers can bid farewell to the need of typing-only imports and the use of TYPE_CHECKING. It allows a simple from __future__ import annotations declaration at the beginning of a file to delay type evaluation, while Lazy Imports defer the actual import statements until required. The combined effect of these optimizations reduced costly runtime imports and further streamlined the development workflow.

The Lazy Imports solution delivers. Meta’s initiative to enhance ML development has involved rolling out Cinder with Lazy Imports to several workloads, including our ML frameworks and Jupyter kernels, producing lightning-fast startup times, improved experimentation capabilities, reduced infrastructure overhead, and code that is a breeze to maintain. We’re pleased to share that Meta’s key AI workloads have experienced noteworthy improvements, with TTFB wins reaching up to 40 percent. Resulting time savings can vary from seconds to minutes per run.

These impressive results translate to a substantial boost in the efficiency of ML workflows, since they mean ML developers can get to the model training phase more swiftly.

The challenges of adopting Lazy Imports

While Lazy Imports’ approach significantly improved ML development, it was not all a bed of roses. We encountered several hurdles that tested our resolve and creativity.

Compatibility

One of the primary challenges we grappled with was the compatibility of existing libraries with Lazy Imports. Libraries such as PyTorch, Numba, NumPy, and SciPy, among others, did not seamlessly align with the deferred module loading approach. These libraries often rely on import side effects and other patterns that do not play well with Lazy Imports. The order in which Python imports could change or be postponed, often led to side effects failing to register classes, functions, and operations correctly. This required painstaking troubleshooting to identify and address import cycles and discrepancies.

Balancing performance versus dependability

We also had to strike the right balance between performance optimization and code dependability. While Lazy Imports significantly reduced TTFB and enhanced resource utilization, it also introduced a considerable semantic change in the way Python imports work that could make the codebase less intuitive. Achieving the perfect equilibrium was a constant consideration, and was ensured by limiting the impact of semantic changes to only the relevant parts that could be thoroughly tested.

Ensuring seamless interaction with the existing codebase required meticulous testing and adjustments. The task was particularly intricate when dealing with complex, multifaceted ML models, where the implications of deferred imports needed to be thoroughly considered. We ultimately opted for enabling Lazy Imports only during the startup and preparation phases and disabling it before the first batch started.

Learning curve

Adopting new paradigms like Lazy Imports can introduce a learning curve for the development team. Training ML engineers, infra engineers, and system engineers to adapt to the new approach, understand its nuances, and implement it effectively is a process in itself.

What is next for Lazy Imports at Meta?

The adoption of Lazy Imports and Cinder represented a meaningful enhancement in Meta’s AI key workloads. It came with its share of ups and downs, but ultimately demonstrated that Lazy Imports can be a game changer in expediting ML development. The TTFB wins, DevX improvements, and reduced kernel startup times are all tangible results of this initiative. With Lazy Imports, Meta’s ML developers are now equipped to work more efficiently, experiment more rapidly, and achieve results faster.

While we’ve achieved remarkable success with the adoption of Lazy Imports, our journey is far from over. So, what’s next for us? Here’s a glimpse into our future endeavors:

Streamlining developer onboarding

The learning curve associated with Lazy Imports can be a challenge for newcomers. We’re investing in educational resources and onboarding materials to make it easier for developers to embrace this game-changing approach. 

Enhancing tooling

Debugging code with deferred imports can be intricate. We’re working on developing tools and techniques that simplify the debugging and troubleshooting process, ensuring that developers can quickly identify and resolve issues.

Community collaboration

The power of Lazy Imports lies in its adaptability and versatility. We’re eager to collaborate with the Python community – sharing insights, best practices, and addressing challenges together. Building a robust community that helps supporting paradigms and patterns that play well with Lazy Imports is one of our future priorities.

What’s going on with generative AI (GenAI) at Meta? And what does the future have in store?

In this episode of the Meta Tech Podcast, Meta engineer Pascal Hartig (@passy) speaks with Devi Parikh, an AI research director at Meta. They cover a wide range of topics, including the history and future of GenAI and the most interesting research papers that have come out recently.

And, of course, they discuss some of Meta’s latest GenAI innovations, including:

• Audiobox, a foundational model for generating sound and soundscapes using natural language prompts.
• Emu, Meta’s first foundational model for image generation.
• Purple Llama, a suite of tools to help developers safely and responsibly deploy GenAI models.

Download or listen to the episode below:

You can also find the episode on various podcast platforms:

Spotify
PocketCasts
Apple Podcasts
Google Podcasts

The Meta Tech Podcast is a podcast, brought to you by Meta, where we highlight the work Meta’s engineers are doing at every level – from low-level frameworks to end-user features.

Send us feedback on Instagram, Threads, or X.

And if you’re interested in AI career opportunities at Meta visit the Meta Careers page.