As Ethernet networks grow they invariably become more complex and difficult to manage, and due to the way they have to be configured they often suffer performance penalties as they grow. Ethernet networks also have to contend with an increasingly diverse set of applications, and now with iSCSI and AoE (as well as FCoE) we are seeing block storage being added to this mix. Different use cases have different requirements, and while web traffic is pretty latency tolerant the latest virtualised block storage requires very low latency and high bandwidth to maintain system performance.

In order to be able to combine and scale the requirements of the Enterprise network, Ethernet needs to evolve into a new form that is capable of meeting these demands – Ethernet Fabric.

Ethernet Fabric is borne out of the need to produce flatter, intelligent, more simple, scalable and efficient networks as demands on those networks increase. Traditional Ethernet networks were fine in the days of client/server, but as virtualisation develops into true heterogeneous cloud resource pools the way the network needs to tie these layers together needs to adapt to cope, if the network is not to become the bottleneck.

Ethernet Fabric was developed to address these requirements.

Flatter

Classic Ethernet networks are hierarchical with three or more tiers. Traffic has to move up and down this logical tree to be able to flow between server racks (left hand image above), which adds latency and creates congestion on inter-switch links (ISLs). This is because in order to prevent traffic loops only 1 path in a redundant connection architecture can be allowed to be active at any one time.

This is achieved with Spanning Tree Protocol (STP), which detects and disables any duplicate paths. This means that multiple paths between switches are prohibited and extra ones inactivated unless the designated primary link fails, in which case a lengthy process to activate a redundant link is initiated by the switches involved. This limits the bandwidth available across the network, and reduces the effectiveness of the redundant link architecture.

Enhancements to Ethernet, such as link aggregation, have been developed to address this limitation and Link Aggregation Groups (LAGs) allow multiple links between switches to increase bandwidth because they act like a single link as far as STP is concerned. However LAGs need to be manually configured and are not very flexible.

Ethernet Fabrics prevent loops without using STP. Flatter networks include self-aggregating ISL connections between switches, eliminating the requirement to define LAGs, and allow all ISLs to be active – which greatly increases the available bandwidth of the network (right hand image above).

Intelligent

Classic Ethernet switches require configuration of each switch port, including setting network policies such as QoS, security, VLANs, etc. When only physical servers accessed such a network this was sufficient, but as we progress to virtualisation and cloud systems the ability of a VM to migrate between physical host servers means the port configuration has to move to a new network port. This requires manual configuration of the switches when setting up the system to make sure this is taken into account. In a complex environment this can be an onerous task and has the potential for errors that may only come to bear when a VM migration fails.

Ethernet Fabrics have distributed intelligence, which allows common configuration parameters to be shared by all switch ports in the entire fabric. In the case of VMs migrating, because the network policies are known on every port on every switch, this does not require a lengthy manual configuration.

In an Ethernet Fabric, switches share configuration information and also know about each other. When a device connects to an edge port of the fabric all switches immediately know about that device, and as the devices send traffic the fabric determines the shortest loop-free path and forwards frames with the lowest possible latency.

Scalable

As previously discussed, classic Ethernet allows only one path between switches. LAGs can improve the amount of ports participating in the link to aggregate bandwidth but have to be manually configured on each switch. As a new switch is added this increases complexity.

Ethernet Fabrics overcome this, and when a new switch connects to the fabric it automatically participates with no manual configuration of links to the rest of the fabric. The new switch learns about all other switches in the fabric and the devices connected to those switches, and is then ready to participate simply by having devices connected to its ports.

If multiple inter-switch links are connected between two switches a logical trunk automatically forms. Traffic is load balanced so that utilisation is near line rate on every link for high efficiency. Should a link in a trunk go offline the traffic on the remaining links is not affected, and incoming frames are automatically distributed on the remaining links without disruption to the devices sending them.

This is in fact exactly how the AoE protocol manages multiple links from device to device, so the Ethernet Fabric therefore complements this feature by extending it across the entire network, making it easier to configure AoE storage at greater distances from accessing servers while still maintaining the important low latency paths it requires for maximum performance. One caveat to this would be that integrating AoE traffic in a unified network does come with security issues, as AoE does not have in built methods of protecting the data streams it carries, therefore strict VLAN separation is mandatory as a minimum.

Efficient

Classic Ethernet uses STP to define a loop-free path*, forming a logical hierarchical switch tree. Even when multiple links are connected for availability and scalability only one link or LAG can be active at any time, which lowers utilisation. When a new link is removed or added the entire network can halt traffic for tens of seconds or even minutes (!) while a new loop-free tree is configured. This is highly disruptive for storage traffic, and can lead to server crashes – hardly ideal in an always-on cloud environment.

Because Ethernet Fabrics do not use STP to remove loops they do not suffer this kind of disruption. They use link state routing with equal-cost multipath routes, which always take the shortest path through the network. When a link is added or removed in the fabric, traffic on other links continues to flow unaffected. Link resiliency is assured and full utilisation of all links between switches is automatic when the topology is changed, without any manual configuration required due to the change.

Simple

Simple is always best as it reduces the amount of management required to maintain a healthy network. In a classic Ethernet environment every switch has to be configured and each port on each switch has to be configured as well – a massive overhead on a complex network. As more server racks are needed more switches are added to the top of the rack, mid row and end of row, and the complexity increases.

Ethernet fabrics share configuration information among all switches in the fabric, and new switches automatically receive common information about other switches and devices when they join. This greatly simplifies the network, reduces management, the potential for mistakes, and cost.

Implications for Unified Networking

Because of the drive to converge the network in a virtualised/cloud environment there is a great deal of extra strain put on the network by the need to run storage as well as regular network traffic across the same network, in an attempt to reduce the even higher complexity introduced by using separate physical networks for storage. As can be seen above this can actually make matters worse and mistakes involving STP can be fatal to the entire network, with port shutdowns taking off entire banks of servers. If the ports handle storage traffic this can be very serious to the availability of the entire system.

Therefore it is vital that Ethernet evolves to be able to handle this issue, and to date Brocade have met this challenge with their VCS architecture and VDX switch products. This allows a migration from the classic Ethernet architecture to the new Ethernet fabric architecture without a massive redesign of the network or redeployment of equipment, as both classic and fabric architectures can co-exist until natural equipment refreshes deliver a new fabric network.

Use of TLV (Type, Length Value) in Ethernet frames can already distinguish iSCSI from other traffic and allow switches to automatically prioritise storage traffic based on this protocol. The simplicity of AoE, and its similarity in link utilisation to a fabric concept, would suggest it would benefit greatly from the use of TLV to allow it to be identified by such equipped switches.

Conclusions

Everything needs to progress and Ethernet is starting to fall behind the curve as we accelerate towards a unified environment where servers, networking and storage are all virtualised resource pools. The physical backbone to this virtual structure has to keep up so it is able to deliver the benefits in the most efficient manner possible, it is therefore important that networking doesn’t become the new bottleneck even as bandwidth potentials increase with 40Gbit and 100Gbit.

The Ethernet Fabric in the form of Brocade switches is already capturing the imagination of Enterprise users and they are grabbing onto this concept. They appreciate the performance and flexibility achieved by doing away with Spanning Tree and creating larger, flatter, layer 2 networks, along with being able to automatically migrate port profiles for VMs.

Nothing ever stands still in IT, and ironically it may be the concept of fabrics and DCB development, originally intended for fibre channel storage networks, that may end up being the enabler for Ethernet to rule the roost in a converged, unified network topology.

*If the AoE protocol is utilised on a separate physical network it does not need STP enabled because it works in the same way as Ethernet Fabric, only over regular Ethernet switches. However this is a device to device protocol, aimed primarily at storage, and therefore on a converged unified classic Ethernet network would be subject to STP issues.

Millennia® Is not a registered Brocade reseller or partner, and as such this article is an independent view and does not seek to indicate that no other vendor can provide features as described above.