Peering automation at Fb – Fb Engineering
Traffic on the Internet is carried over many different types of links. A quick and reliable way to exchange data traffic between different networks and service providers is through peering. First, we managed peering through a time-consuming manual process. Reliable peering is essential to Facebook and everyone’s internet usage. However, there is no industry standard for setting up a scalable, automatic peering management system. That’s why we developed a new automated method that enables faster self-service peering configuration. We’re sharing some best practices we’ve learned in automating our public peering in hopes of taking our approach into the internet community.
How does this work? Take Facebook, for example. Your friend just posted a video of an extremely cute cat and you are about to check it out. Let’s follow the path of the cat video before it reaches your device:
Option A: Often the slower, less reliable, higher latency route:
You see your friend’s post with a cute cat video and you click on it and can’t wait to watch it! Before the video can reach your device, Facebook needs to send it to your internet service provider using the most powerful and shortest route available. There may be many other networks (commonly known as transit networks) between Facebook and your ISP. You may be connected to each other in suboptimal locations with potential capacity constraints, so the awesome cat video is slowly reaching you. Nobody wants to see a buffering cat video!
Option B: Often the faster, more reliable and more direct way:
You clicked the cat video to see the cute cat! Even before the video reaches your device, the Facebook Traffic Controller recognizes that there is a quick and direct route to your ISP without other networks being in the middle. The cat video goes through a peering exchange, a common meeting point where many different types of networks are interconnected by facility Border Gateway Protocol (BGP) Sessions between their routers. These peek In meetings, they can directly exchange bits, including cat video, improving the quality, performance, latency, and reliability of the user experience.
Why we automated public peering
Manually configuring peering is known throughout the industry as a painfully slow, inefficient, and error-prone process. This challenge increases as networks connect to new Internet Exchanges (IX) and connect multiple routers to each IX.
Before we developed our automated system, we had the same battle. Peers would email us requesting peering sessions to be set up. Next, one of our Edge engineers would check the email and check our mutual traffic. To confirm that the traffic was adequate, this team member had to review numerous internal dashboards, reports, and rule books, as well as external resources such as their potential colleagues PeeringDB Recording. The team member would then use some internal tools to configure BGP sessions, respond to the peer, and wait for the peer to configure their side of the network.
This approach had several problems. First, there was no central location to display the incoming peering requests or the existing peering status. Inquiries can be received by email or through several other out-of-band systems. Edge engineers had to track, analyze, and manually review every request. Next, for each request, the team member had to manually start and monitor an internal tool for each peer, and then enter a response to each peering request when complete. We estimate that this process took over nine hours a week and wasted an entire day of each work week on an unnecessarily manual process.
We are happy to announce that peers can now request their own public peering sessions through our facebook.com/peering page.
PeeringDB is an open source database for networks and their peering network information that is reviewed and verified by PeeringDB administrators. We believe the PeeringDB database has value, and along with it many other In the industry, we support this through sponsorship. Since most peering networks already manage their PeeringDB records as a source of truth for other networks, we see PeeringDB’s OAuth service as a way to standardize an authentication method for peering management.
To ensure that the peering requests made on our peering page come from an authorized person, the requester must authenticate himself using his PeeringDB login and leverage PeeringDBs OAuth Service on behalf of the organization of your network. The peer does not need to provide any other authentication – no Facebook account is required. After authentication, the peer sees a list of all existing public peering sessions on his network with Facebook and can send new requests.
After requesting meetings, our internal process takes over. All the peer has to do is wait for our automated emails and configure their side of the network.
We have also set up a surveillance system to sort our Peering @ Mailbox. When the system detects a peering request, it automatically responds with instructions to direct the peer to our peering side. Of course, we continue to monitor the inbox to respond to inquiries or support requests. However, this new engine has significantly reduced the time it takes to search email and review requests.
Once received, the request is placed on a monitoring queue. If the request is approved, another service starts a workflow to set up peering. First, PeeringDB and our internal tables are queried in order to validate and collect the mutual peering information such as IP address and Max prefix settings. Next, the sessions are configured on the Facebook routers. It then sends an email to the peer to confirm that the BGP sessions on Facebook page are ready and waits for the peer to show up on its page. The workflow then checks daily that the sessions are set up. On the second, third, seventh and 13th day, an email is sent to the peer to remind it to configure the sessions. Once our workflow determines that all sessions have been established, our workflow sends a final confirmation email. At this point, our peer should be able to see the new sessions as active in the table on facebook.com/peering.
Create industry standard
Since launch, we’ve received more than 170 peering requests and approved 149 of them. As a result, we automatically ran more than 1,400 public peering sessions – saving more than eight hours a week.
With PeeringDB OAuth we can check the validity of peering request submissions and automate further steps in the peering turn-up process. Based on our experience with this system, we recommend using PeeringDB OAuth as the industry standard for other public peering automation applications and implementations.
Building on our success in public peering automation, we’re exploring ways to automate our private network connections (PNIs). Private peering is the larger equivalent of public peering, and we hope to have a self-service option later this year. We are also looking into the possibility of using PeeringDB OAuth as an alternate login service For other services we offer our network partners.