The Social Network™ releases its data networking code

Facebook has sent another shiver running up Cisco's spine, by releasing the code it uses for packet routing.

Open/R, its now-open source routing platform, runs Facebook's backbone and data centre networks. The Social Network™ first promised to release the platform in May 2017.

In the post that announced the release, Facebook said it began developing Open/R for its Terragraph wireless system, but since applied it to its global fibre network, adding: “we are even starting to roll it out into our data center fabrics, running inside FBOSS and on our Open Compute Project networking hardware like Wedge 100.”

Apart from Facebook's internal platforms, the company launched Open/R with support from third-party vendors including:

• Juniper – gRPC-based APIs integrate it with the QFX and PTX platforms; and
• Arista – integration with its open source EOS SDK;

Facebook said in the post that while working on Open/R with Terragraph, it decided it needed a distributed networking platform to get over the shortcomings of both Layer 2 protocols for large networks (lacking resiliency) and Layer 3 protocols (“poorly suited to highly variable and dynamic outdoor conditions”).

“Thus, we designed Open/R to be the distributed networking platform on top of the Terragraph network that could cope with its large scale and rapid changes, and allow for fast extensibility”, the post said.

Those features also made Open/R suitable to control traffic and routing across the Express Backbone it built in May.

“Instead of using a traditional routing protocol such as IS-IS, we decided to introduce Open/R into the Express Backbone as the distributed control plane for the network, where it would work together with a centralised controller. Here, Open/R provides basic connectivity in a network, fast reactions to network events, and a "distributed information bus" to export new information to the controller and agents on the network nodes.”

Open/R uses a database as the control plane rather than a routing protocol (as noted on slide 10 of this IETF presentation by Tom Herbert of Quantonium and Petr Lapukhov – a co-author of the announcement – describing a proposed IPv6 location identifier standard).

Open/R architecture

The image below shows the key architectural components.

Open/R architecture. Click to embiggen.

The KV-STORE – key-value store – is replicated around the network for distributed communications and state replication; Spark handles neighbour discovery and reporting.

LinkMonitor's role is probably self-explanatory, PrefixManager handles “automatic prefix suballocation”, Decision is the route information computer, FIB is the route programming proxy, and Platform “implements route programming and interface discovery logic for the target hardware platform.”

So engineers don't need to put live networks at risk, Open/R offers an emulation environment: a systemd-nspawn wrapper. This, Facebook wrote, “starts thousands of Linux containers and forms a mesh of tunnels among them with a single command”, so engineers can test the impact of latency, jitter, and packet loss.

The distributed state database lets an administrator view current network state from any node on the network, watching over activity like nodes connecting, adjacencies tearing down, or metric values changing.

For functions that need a centralised controller, the post said, “Open/R provides APIs allowing remote agents to learn the link state or subscribe to database updates, such as notifications of a link capacity change”.

Other features Facebook listed include:

• It uses IPv6 link-local addresses for “zero-touch configuration”;
• Native IPv4 routing;
• Ad hoc network prefix allocation and IP configuration for nodes;
• Live software updates with “graceful restart” so traffic forwarding isn't disrupted; and
• A controller integration API, and a Python library that interacts with “all the main Open/R processes”.

The post also said Open/R includes “many features similar to those found in IS-IS (intermediate system to intermediate system) and OSPF (open shortest path first).

Open/R's GitHub repository is here. ®