There are a litany of concepts that define a modern network, from simple principles to full feature sets.
In general, a next-generation data center design enables you to move to a widely distributed non-blocking fabric with uniform chipset, bandwidth, and buffering characteristics in a simple architecture.
In one example, to support these requirements, you would begin with a true three-tier Clos switching architecture with ToR, spine, and fabric layers to build a data center network. Each ToR would have access to multiple fabrics and have the ability to select a desired path based on application requirement or network availability.
Following the definition of a modern network from the introduction, here we lay out the general definition of the parts.
Here we will discuss the concepts that build an NGN.
SDNs can be defined in multiple ways. The general definition of a SDN is one which can be controlled as a singular unit instead of on a system-by-system basis. The control plane, which would normally be in the device and uses routing protocols, is replaced with a controller. SDNs can be built using many different technologies, including OpenFlow, overlay networks, and automation tools.
Within an SDN, you will have the concept of controllers. There are four controllers that we will talk about in this book:
- OpenDaylight and ONOS, which are OpenFlow-based open source controllers
- Application Policy Infrastructure Controller (APIC) from Cisco
- NSX from VMware
As we mentioned in the introduction, 20 years ago, NGN hardware would have been the Cisco GSR (officially introduced in 1997) or the Juniper M40 (officially released in 1998). Large Cisco and Juniper customers would have been working with the companies to help come up with the specifications and determining how to deploy the devices (possibly Alpha or Beta versions) in their networks:
Today, we can look at the hyperscale networking companies to see what a modern network looks like. A hyperscale network is one where the data stored, transferred, and updated on the network grows exponentially. Technology such as 100 Gb Ethernet, SDN, open networking equipment, and software are being deployed by hyperscale companies.
Open hardware has been around for about 10 years, first in the consumer space and more recently in the enterprise space. Enterprise open networking hardware companies such as Quanta and Accton provide a significant amount of the hardware currently utilized in networks today. Companies such as Google and Facebook have been building their own hardware for many years. Facebook's routers such as the Wedge 100 and Backpack are available publicly for end users to utilize.
Some examples of open networking hardware are as follows:
- Dell S6000-ON: This is a 32x40 G switch with 32 QSFP+ ports on the front
- Quanta LY8: This is a 48x10 G + 6x40 G switch with 48 SFP+ ports and 6 QSFP+ ports
- Facebook Wedge 100: This is a 32x100 G switch with 32 QSFP28 ports on the front
To use open networking hardware, you need an operating system. The operating system manages the system devices such as fans, power, LEDs, and temperature. On top of the operating system, you will run a forwarding agent. Examples of forwarding agents are Indigo, the open source OpenFlow daemon, and Quagga, an open source routing agent.
Cisco and Juniper are the leaders in the closed hardware and software space. Cisco produces switches such as the Nexus series (3000, 7000, and 9000) with 9000 programmable by ACI. Juniper provides the MX series (480, 960, and 2020) with 2020 being the highest end forwarding system they sell.
Cisco has multiple NOSes including IOS, NX-OS, and IOS-XR. All Cisco NOSes are closed source and proprietary to the system that they run on. Cisco has what the industry call an industry standard CLI, which is emulated by many other companies.
Juniper ships a single NOS, Junos, which can install on multiple different systems. Junos is a closed source BSD-based NOS. The Junos CLI is significantly different from IOS and is more focused on engineers who program.
Network virtualization, not to be confused with NFV, is the concept of recreating the hardware interfaces that exist in a traditional network in software. By creating a software counterpart to the hardware interfaces, you decouple the network forwarding from the hardware.
There are a few companies and software projects that allow the end user to enable network virtualization. The first one is NSX, which comes from the same team that developed Open vSwitch (OVS) called Nicira, which was acquired by VMware in 2012. Another project is Big Cloud Fabric by Big Switch Networks, which utilizes a heavily modified version of Indigo, an OpenFlow controller.
NFV can be summed up by the statement, due to recent network focused advancements in PC hardware, any service able to be delivered on proprietary, application specific hardware should be able to be done on a virtual machine, essentially, on routers, firewalls, load balancers, and other network devices, all running virtually on commodity hardware.
While Open-source software (OSS) has been around for decades, the concept of open networking hardware has not. By combining OSS with open networking hardware, end users are able to create their own network devices that provide the connectivity and services that are necessary for them.
One of the first companies to come out with open networking hardware was Quanta Computer. In 2009, Pronto started to provide open networking switches including the LB4G and LB9(A). These switches were used by the OpenFlow team at Stanford to develop OpenFlow on hardware. Sold under the Pronto Networks name (now, Pica8), these switches were used by companies such as Google for their SDN projects.
In 2011, Facebook started the Open Compute Project (OCP). The goal of the OCP was to provide a place where companies could share hardware and software designs. These designs are used by multiple hardware manufacturers to build OCP specification hardware. In 2013, the OCP introduced the networking project, where networking vendors could submit open hardware designs for network switches.
Companies such as Big Switch Networks, Cumulus Networks, and Pluribus Networks utilize open switching hardware built by companies such as Accton, Dell, and Quanta to create fully open and malleable networks. OSS projects such as Open Network Linux, OpenSwitch, and OS10 provide a open software base for these devices on which end users can build their own tools.
One of the most important software tools is Open Network Install Environment (ONIE), which is a small Linux image that allows end users to install a NOS onto a network device such as a switch.
Some examples of open source networking software are as follows:
- Facebook's FBOSS, a Thrift-based daemon that manages the forwarding of the switch by interacting with Broadcom's OpenNSL. FBOSS has no routing capabilities of its own and requires all information to be provided via a configuration file and Thrift API calls.
- Microsoft's Software for Open Networking in the Cloud (SONiC) uses a Quagga-based routing daemon talking to Switch Abstraction Interface (SAI) and runs on a few open hardware switches including the Dell S6000-ON and the Mellanox SN2700.
- Google have also designed their own switches since 2004, but have not released the designs or software information. In 2012, one of their switches was accidentally shipped to the wrong location and appeared on the internet.
Accton has been quite active in the open hardware space, providing a multitude of designs, including some designed by Facebook, such as the Wedge. The current generation Facebook Wedge is the Wedge 100, providing 32 ports of 100 G. Accton also has its own switches, such as the AS7716, that provide 32 ports of 100 G:
Most open networking hardware designs are based around switching ASICs from Broadcom, but over the past few years, other companies such as Barefoot Networks, Cavium, and Mellanox have brought out more open designs. Barefoot is a good example of a fully open design, where they utilize a specific language, called P4, to program the forwarding hardware.
The Facebook designed switches are focused on Facebook's own design needs. The Facebook Wedge 100 runs a standard Linux image with drivers for the Broadcom switching chips. On top of the software stack, Facebook uses an OSS project called FBOSS to control the switches via a Thrift API. This allows Facebook to manage their switches the same way they manage their servers.
Facebook contracted with both Accton and Quanta to build the Wedge 40. The Wedge 40 is built from commodity components and are reused from other Facebook systems. The CPU complex and Board Management Controllers (BMC) come from the Facebook servers.
The next generation networking devices that have come from the needs of hyperscale networking companies have a few commonalities:
- In general, the configuration and operation of these devices have been designed to be automated or managed from a central controller
- Automation is done via tools that use everything from screen scraping to utilizing Thrift or REST APIs
- Most of these hyperscale-focused next generation networking devices have one or more parts that are traditionally found on servers, such as a BMC, powerful Intel processors, and large solid state storage drives
The equipment used in hyperscale networks can be from established vendors, such as Cisco and Juniper, or from open networking companies, such as Edgecore and Quanta. Dell is a special case as they offer both closed and open versions of their switching hardware, designated with a -ON at the end, for example, the S6000-ON and a 32x40G switch. Mellanox, which started as a storage network vendor, has been building open networking switches, including the SN2700, a 32x100 G switch, and the SN2100, a 16x100 G switch.
Many open networking designs come out of specific needs of the hyperscale companies and some even come from the hyperscale companies. Facebook have open sourced five switches, all designed to meet their needs:
- Wedge 40: This is a 16x40 G switch with a BMC: running FBOSS
- 6-pack: This is a 128x40 G modular switch with multiple BMCs: running FBOSS
- Wedge 100: This is a 32x100 G switch with a BMC: running FBOSS
- Backpack: This is a 128x100 G modular switch with multiple BMCs: running SnapRoute
- Voyager: This is a open transponder for DWDM networks, which includes both 12x100 G Ethernet and 4x200 G DWDM ports: running an FBOSS-like daemon:
Software is the heart of any network: no matter how sophisticated hardware gets, software is necessary to utilize the hardware. Here we will discuss the software components behind open hardware initiatives.
In order to use open hardware, there needs to be an installation environment. Currently, ONIE is the standard. ONIE was developed by Cumulus Networks in coordination with Big Switch Networks, and it provides a GRUB or U-Boot installable miniature Linux environment from which NOS can be installed on the system.
The Bootloader is a small bootable software image that is flashed to Cisco hardware. The Bootloader initializes the system and brings up the devices necessary to load the main software, IOS.
Open source projects such as OpenBMC have been released to provide the software to run on the BMC and system processor.
Networking companies such as RTBrick and SnapRoute have been formed to provide API manageable networking stacks. SnapRoute provides an entire forwarding infrastructure including L2/L3 and forwarding chip drivers. SnapRoute have written their project in Go, Google's language of choice.
Software-defined products such as Cisco's Application Centric Infrastructure (ACI) and VMware's NSX have come out of large companies such as Cisco and VMware along with products such as Big Cloud Fabric (BCF) from Big Switch Networks.
Open NOS companies such as Cumulus Networks and Pica8 have released software for open switches. Pica8 also provides a full solution, selling switches with PicOS installed. Pica8 originally provided switches under the Pronto name, but now sells them under the Pica8 name, for example, the P-5401: 32x40 G switch.
Software from companies such as Cisco and Juniper are considered closed source as they do not include access to the source code. Cisco has multiple operating systems, including their original Internetwork Operating System (IOS), not to be confused with Apple's recent use for their iDevices.
IOS is a binary blob operating system that is loaded into memory on boot of Cisco devices. IOS is easy to upgrade since all of the configuration information is kept separate and the IOS filesystem is immutable. Installing a new IOS version simply requires that you upload it to the device and point the configuration to load it. IOS uses a CLI that is considered to be the standard interface and replicated by many other vendors.
Once you have your hardware picked out, you need an NOS. An NOS is what runs on your switch and allows it to forward packets. Companies such as Cumulus Networks and Pica8 sell full NOSes with L2 and L3 forwarding capability. In the OSS world, there are a few choices, including Open Network Linux (ONL), OpenSwitch, and SONiC.
ONL is a project started by Rob Sherwood, previously of Big Switch Networks and now at Facebook. The goal was to provide a simple, clean Linux-based open source network operating system. ONL provides a ONIE compatible installable NOS, on which the user can install their own forwarding agents. At the time of writing, ONL supported ~35 switches from Alpha Networks, Dell, DNI, Edgecore, Mellanox, Quanta, and others.
OpenSwitch is a project started by HP (now HPE) to provide a full NOS using Quagga as the base and creating a full layer 2 / layer 3 platform on top of a Linux base.
The project used a central database based on OVSDB and required all data to be exchanged through the database rather than directly between themselves. The design was complex and eventually supported ~3 switches directly, rebranded HP versions of Edgecore switches, and unofficially supported ~4 more that were ported by end user or vendors.
OpenSwitch started pivoting at the time of this book. The Quagga design with OVSDB was replaced with SnapRoute, an API-driven routing stack running on Dell's OS10 Open Edition, a Debian-based NOS.
SONiC is a project that Microsoft started to run inside their own network on white-box switches. It uses Quagga for forwarding and Redis as a database to store information and exchange data between processes. SONiC runs on top of Debian 8 and can be run on Dell OS10 or ONL.
At the time of writing, SONiC supported more switches than OpenSwitch, including some Edgecore, Arista, and Mellanox switches.
If you need to run a forwarding agent on a Linux-based NOS, there are a few options, of which most, other than Mellanox's SwitchDev implementation, have proprietary / closed source parts. Broadcom provides both OF-DPA and OpenNSL publicly as binary-only options for programming their ASICs. Cavium offers an SAI interface to their SDK, but it is not publicly available at the time of writing.
To quote the Linux kernel documentation, The Ethernet switch device driver model (switchdev) is an in-kernel driver model for switch devices which offload the forwarding (data) plane from the kernel. Essentially, it is an open Netlink listener that allows for the offloading of forwarding information to hardware. It can also be used for soft switches such as OVS and offload network interface cards using SR-IOV.
From the open switching side, currently, only Mellanox supports SwitchDev for their 10/25/40/50/100G switches. Broadcom only supports its consumer-grade switches such as those in access points and home routers.
Indigo is a OpenFlow-based forwarding agent that runs on the NOS to provide forwarding. Introduced in 2008 by Stanford University, Indigo is the base for Big Switch Networks OpenFlow daemon, the ON.Lab CORD project, and Indigo is integrated into OF-DPA, the Broadcom OpenFlow driver.
The concept of OpenFlow is simple, program forwarding tables in hardware and software switches using a standardized interface. We will cover OpenFlow in detail later in this book.
While specifically designed for the Facebook Wedge switches, FBOSS provides a Thrift-based API with integration into Broadcom's Open Network Switch Layer (OpenNSL). FBOSS allows for static configuration of interfaces and forwarding entries. FBOSS also has a Netlink listener available, which allows for the end user to run programs such as Quagga or FRR on the system.
SwitchD is the Cumulus Networks SDK programmer. It is available with the purchase of Cumulus Linux, a Linux-based NOS.
There are two types of software controllers. The first type uses OpenFlow to manage the hardware in the network. OpenFlow is an open standard that is easily utilized to manage the forwarding of traffic. The second type uses a proprietary API to manage the hardware in the network. The API can be open or closed, but will not be generic.
An OpenFlow controller manages multiple networking devices by programming switches using the OpenFlow protocol. OpenFlow-based network devices run a OpenFlow daemon such as Indigo, which translates the OpenFlow commands into switch forwarding data.
There are branded and unbranded versions of OpenDaylight available. Brocade makes the Brocade Virtual Controller (BVC). The OpenDaylight foundation releases unbranded versions of the software. You can also obtain OpenDaylight from Cisco.
The following diagram shows a test setup for OpenDaylight using two switches:
A good example of a open source, open API design is what the company called SnapRoute is doing with their product FlexSwitch. FlexSwitch uses a REST-based API to program the L2/L3 forwarding entries.