10 Gb Routing

10GbE, or 10-Gigabit Ethernet, has been gaining traction for a number of years now. Although the standard (IEEE 802.3ae) is nearly 20 years old, the first 10 years were dominated by the industry building out its 1GbE networks, due primarily to cost and cabling issues. Over the last 5-10 years however, its use is transforming enterprises, data centers, NAS (network-attached storage), and virtualized environments, and this trend will certainly continue, as its adoption drives more and more use cases and environments where 10GbE is rapidly becoming indispensable. To facilitate this technology, 10GbE routers are being installed in more and more locations. Let’s take a look at 10GbE routers, where and why they’re needed, and some considerations if you’re considering jumping into the 10GbE world. But first, a little trivia…

What U.S city was the first to offer 10GbE Internet connections? It must be in Silicon Valley, right? Wrong! Salisbury, North Carolina, population 33,000, became the first U.S. city to provide 10Gb internet back in 2015. A local private college was the first customer to receive the service. The fact that a municipality, along with a local college, were the first (wide-scale) implementation of 10GbE is interesting and informative. In general, 10GbE is not appropriate for individual consumers (at least, not yet although hardcore home networking aficionados are using 10G switches – but routers, not so much). But metropolitan areas and colleges are a great fit as 10GbE can be deployed across the MAN (metropolitan area network) and the CAN (campus area network) efficiently.

10GbE, built around Layer2 Ethernet technology, is still primarily concerned with treating pain points (aka bandwidth congestion and high latencies) associated with LANs, local area networks, and perhaps access links to WANs, but not WAN cores or backhauls in general. 10GbE operates in full-duplex mode only and supports data transfer rates of 10 gigabits per second for distances up to 300 meters on multimode fiber optic cables and up to 40 kilometers on single mode fiber optic cables.

On the 10Gb routing side of the equation, the use cases of 10Gb routers are:

• Core networks in general, the routers that make up the connectivity between various Internet islands.
• Telco and CableCo core networks, where service providers are aggregating traffic from end-users and connecting to their regional POPs (Point-of-Presence).
• Corporate edge networks, the gateways that aggregate the office network and pass it onto either an SD-WAN edge device for inter-office or cloud communication.
• Corporate data centers, where 10Gb routers can act as a gateway in and out of the data center.
• To connect geographically separated LANs over dedicated high-bandwidth SONET/SDH/TDM WAN networks. SD-WAN tunnels can also facilitate these point to point connectivity between geographically separated LAN segments.
• Support for future and developing IOT (internet of things) 5G applications that require throughput above 1 Gbps.

On the 10Gb switching side of the equation, there are many natural use cases for 10GbE:

• High-bandwidth, low-latency applications in the local networks, such as streaming video, VoIP, unified communications, medical imaging, telemedicine, high-end graphics, high-performance gaming, distance learning, telecommuting
• Upgrading, expanding, and/or extending existing Ethernet networks (generally LANs)
• Multi-user and high-tenancy networks
• Virtualization
• Remote backups and disaster recovery
• Storage on demand
• Provides greater bandwidth for server networks, storage area networks (SAN) and network- attached storage (NAS) applications
• In data centers to facilitate switch-to-switch and switch-to-server applications
• Wireless Access Points

Each of these scenarios/applications greatly benefits from 10GbE connectivity to handle the local traffic. Largely the routers will be located in data centers (physical or public/private/hybrid clouds) and SAN and NAS computing environments, where massive amounts of data from multiple users and applications converge. Let’s look a little more closely at a few of the above use cases.

Data Centers

Data centers have been a natural breeding ground for 10GbE. Two critical drivers of the requirement for a 10x improvement over legacy 1Gb networks have been large-scale adoption of virtualization, and cloud-based infrastructures. Servers have been getting faster and faster over the last decade (server CPU throughput has historically doubled every 18 months) which allows more virtual machines running on each server. Each VM competes for I/O on each server. More and more data must be processed, analyzed, shared, stored, and backed-up. Adoption of cloud-based services further increases the demands on VMs, servers, and data centers. 1GbE connectivity has been struggling to keep pace, and adopting 10GbE results in several immediate advantages:

• Overall Performance
  • 10x the bandwidth means 10x faster access to storage and backups, and faster network connections mean lower latency and reduced issues due to 1GbE bottlenecks
• Simplified, Unified Network Infrastructure
  • Upgrading 1GbE routers and switches to 10Gbe allow a 10-to-1 consolidation of much hardware – allowing 10Gb performance through a single interface instead of requiring 10 interfaces. This also significantly reduces cabling complexity (although it will require new cabling in general), and with the increased performance of the network as a whole, disparate traffic types can now be unified onto the network

Wireless Access Points

Wireless access points (WAPs) are another area where 10GbE has been critical. Wireless access points have exploded in use over recent years and the demand on their resources has overwhelmed 1GbE connectivity. There have been dramatic increases in the number of users per WAP and also the amount of data being shuffled back and forth per user. And with the relevant wireless spec, IEEE 802.11ac, stating a theoretical maximum throughput of 6.933 Gbps, 10GbE is required to achieve these speeds. While this may involve a significant effort in re-cabling, it is clearly necessary in the long run.

Desktop Computing

Another interesting space where 10GbE is gaining traction is the personal, desktop computer space. While 1Gb connectivity has been largely satisfactory for most of this market, there are certain, heavy users who could benefit from the 10x improvement afforded by 10GbE. And with the overall cost-of-ownership continuing to drop, it has become feasible for individual (power) users to upgrade their systems to 10GbE. Some of the fields which could really use the upgraded performance include:

• HD and UHD video editing/processing/transcoding
• CAD/CAM design, solid and surface modeling
• 3D rendering
• Financial/Scientific analysis
• Big data in general, both access (storage) as well as computation
• High-end Gaming

NAS and SAN Implementation

10GbE also plays a major role in NAS (network-attached storage) and SAN (storage area networks) implementations. Ethernet storage protocols dominate much of the new virtualized environments, including NAS protocols such as Network File System (NFS), Server Message Block (SMB), and Common Internet File System (CIFS), and also SAN protocols including Internet Small Computer System Interface (iSCSI), and Fibre Channel over Ethernet (FCoE). As physical servers continue to implement multiple VMs per box, the I/O bandwidth required increases accordingly.

10GbE Routers Interfaces

10 Gigabit Ethernet maintains backward compatibility with existing Ethernet standards, e.g. 10/100/1000BASE-T (1 Gigabit ethernet over twisted copper pairs). So these routers can be easily used to upgrade existing Ethernet networks incrementally, addressing choke points in order of importance. Incorporating 10GbE into your existing network requires 10Gb-capable NICs (network interface cards) on targeted servers or workstations, as well as 10Gb-capable switches.

As far as interfacing to the 10GbE router goes there are two choices:

• 10GBASE-T RJ-45 Copper
  • Advantages
    • Uniformity and backward compatibility with existing cabling (cat6a, cat7)
    • Cheaper for short-run connectivity (less than about 100 m)
    • Mature technology, completely standardized with many OTS solutions
    • Easy to find pre-made cables of virtually any length
    • Patch panels are supported
  • Disadvantages
    • More power consumption and heat generated
    • Higher latency
    • Latency increases with cable length
    • May not be suitable for larger distances

• 10GbE SFP+ Optical Fiber
  • Advantages
    • Less power consumption and heat generated
    • Extremely low latency that does not increase with cable length (speed of light is fast!)
    • Suitable and more cost effective for longer distances (up to 10-100 km using single-mode optical fiber)
  • Disadvantages
    • Will require new cabling
    • May not be backward compatible with 1 Gbps fiber
    • More expensive for short-distance connectivity (less than about 100 m)
    • May be difficult to find OTS custom-length, pre-made cables
    • Patch panels will need to use transceivers modules

Your particular deployment scenario will determine which interface option is best for you. And hybrid solutions may be a great approach, as you can upgrade your existing network, piece by piece.

Some general guidelines:

• 10GBASE-T RJ-45 Copper may be the better good choice:
  • If the cable run-distances are relatively short (less than ~100m) then using RJ-45 10GBASE-T may be a good solution in terms of cost and flexibility
  • For smaller enterprises who don’t want to re-cable their internal networking
• 10GbE SFP+ Optical Fiber may be the better choice:
  • If the physical separation between networks or network devices is relatively large (greater than ~100m), SFP+ is a better and more cost-effective solution
  • If lowest possible latency and/or power consumption is required, SFP+ is preferred

Salisbury, North Carolina Revisited

Back to why Salisbury, North Carolina is a good fit for a 10GbE upgrade. With a physical size of about 18 square miles, the city of Salisbury can easily deploy fiber optic cabling from a central location and reach the entire town. And by using a local college as a first client, the college could relatively easily and cheaply upgrade their multi-campus data centers and LANs without any new cabling required. The campuses themselves would then be linked using the underlying fiber-optic backbone from the service provider.

40GbE, 100GbE and Beyond

10GbE is currently transforming many aspects of our digital universe, including data centers, wireless access points, ethernet-based storage solutions, and many others. The 10-to-1 consolidation of resources combined with the 10x improvement in bandwidth and latency allow enterprises to keep pace with the surging demand for speed and low latency across many sectors. And while this is a huge improvement over 1Gb networks, there continues to be a demand for higher and higher speeds.

In 2010, the IEEE ratified IEEE 802.3ba-2010 which establishes standard for 40GbE and 100Gbe. Over the last few years the technology has started gaining traction and will continue to do so with increasing speed as the associated costs continue to drop. The IEEE task force emphasized that these increased bandwidths are needed for:  

• Switching, routing and aggregation in data centers
• Internet exchange points
• Service provider peering points
• High bandwidth applications like video on demand and high performance computing (HPC)

These ultra-high speeds are currently being used in service provider networks, and the technology is sometimes called “Carrier Ethernet” which emphasizes that widespread use among enterprises is still in its infancy, although its adoption in data centers is accelerating.

Rob Stone, Mushroom Networks, Inc. 

Mushroom Networks is the provider of Broadband Bonding appliances that put your networks on auto-pilot. Application flows are intelligently routed around network problems such as latency, jitter and packet loss. Network problems are solved even before you can notice.

https://www.mushroomnetworks.com