Impact of virtualization on networking paradigms Siamack Ayandeh 3COM Corporation ABSTRACT Virtualization is emerging as an enabling technology to lower the

cost of large datacenters, repositories of data and applications that constitutes most of what is known as the World Wide Web. The entire infrastructure of datacenters, hosts/servers, storage, and network are taking advantage of virtualization technologies to enable rapid provisioning of resources, management of change, and increase utilization of resources, hence reducing the total cost of ownership within datacenters. Virtualization however impacts some basic premises of networking including basic connectivity, flow of traffic, security, and domains of management responsibility. These impacts are the focus of this paper. Some possible directions for future networks and standards to support virtualization are also discussed. INTRODUCTION Datacenter designs are evolving from silos of specialized dedicated equipment towards a more homogenous infrastructure which can be provisioned rapidly under control of software to satisfy the needs of new services and applications. Off the shelf compute, networking, and storage resources are provisioned under the control of what we generically refer to as virtualization manager software (VMS). By sharing large pools of resources amongst many users at higher utilization levels, the resulting economies of scale is used to lower 1 the total cost of ownership. Latter includes the need for electrical power which in recent years has been at a premium, as many datacenters have been operating at the limits of available power consumption. Also the burden of standing up new equipment, and managing the constantly changing environments has been a challenge. This is further amplified by the need for cooperation amongst specialized teams of operations staff each of which has a responsibility for their specific domain of expertise, being the server, network, storage, or security. The concept of virtualization is not new. Computer operating systems abstract the hardware resources from software processes and offer a virtual environment to applications. The concept is also used to help software portability in the form of virtual machines using software interpreters. Networks invoke virtualization at many levels. Packet communication is the vehicle that is used to create virtual connections amongst end points, virtual LANs limit the scope of connectivity over shared links, virtual circuits played an important role in evolution of networks; Time Division Multiplexing is yet another example, while Virtual Private Networks (VPN) have served privacy requirements of enterprise networks spread over shared public infrastructures. Similarly the idea behind a Storage Area Networks (SAN) is to share pools of disk space as Logical Units (LUNs) amongst SCSI initiators.

As such applications are no longer tied to the local disk of their host environment and can migrate to other hosts as long as they have access to the SAN. So what is different about virtualization manager software (VMS)? One way to consider virtualization managers is as a meta-operating system. In a host environment the VMS e.g. can sit on top of hardware and enable execution of multiple operating systems each with its own dedicated slice of CPU, memory, and network interface resources. As such a single host can be turned in to multiple virtual hosts each with its own operating system and applications. Applications continue to execute in their native OS environment using standard networking stacks e.g. TCP/IP. Linux and Windows can live side by side. The client/server architecture that has been used to build applications would persist despite the fact that several virtual hosts may be residing within the same physical host. Another view of VMS is as an operating system for the datacenter operations. Since physical resources are under control of VMS, many traditional management tasks within datacenter can be add on capabilities to the VMS. This includes: Ability to pool together CPU, memory, networking, and storage resources to satisfy silos of applications. Ability to control the usage levels of resources. Ability to control the backup and restore of such configurations and resulting data. Ability to migrate the applications in various forms from addition of more 2

resources, to physical migration both locally and to remote locations e.g. for disaster recovery applications. And many other day to day tasks of managing resources within a datacenter can benefit from potential for automation, as VMS offers an API for accessing such resources.

As such VMS helps create access to massive compute and storage resources made of discrete off the shelf components, running 3rd party operating systems such as Windows, Linux, and AIX, for most part transparent to applications. Figure-1 is an overview of the VMS environment.

Figure-1: VMS Environment In the remainder of this paper we focus on impact of VMS on operation of the network component within datacenters. IMPACT OF VIRTUALIZATION ON NETWORK

Several aspects of the network are impacted by VMS and we cover each in following subsections. A) Connectivity Traditionally the point of demarcation between the host and the network has been the Network Interface Card where the most common protocol has been IEEE 802.3 [1] Ethernet. The media access control (MAC) address of the host is burnt to the NIC in the factory. Ethernet then offers a plug and play environment where hosts are connected to the access layer of the switching infrastructure and uniquely indentified by their MAC address. A Virtual Machine maintains the same network protocol stack and concepts. Hence to operate a VM a MAC address and IP address are required. The underlying NIC resource is however abstracted away as a virtual NIC (vNIC) by VMS. The vNIC is assigned a MAC address by VMS once the VM is created. The IP address can also be statically assigned or dynamically obtained using the DHCP protocol. VMS shares the physical NIC amongst the vNICs including the capability to switch Ethernet frames amongst vNICs and virtual machines (VMs) without ever leaving the host. This capability of the VMS is referred to as a virtual switching (vSwitch) capability. Hence some of the capabilities of the standard access layer switching are also offered by the vSwitch. For example VLANs are supported by the vSwitch. Alternate approach is to simply aggregate the vNIC traffic to the edge access switch over the physical NIC. As such the host inter-VM traffic is hair pinned back over the same port. This involves a modification to the 802.1D bridging 3

protocol [2] which currently prevents such an action to prevent loops. Note that vSwitch or traffic aggregators only connect to vNICs and uplinks. There is no direct connection amongst these devices. This further prevents loops and removes the need for the spanning tree protocol [2]. The following are some of the issues with connectivity in a virtualized environment 1. Assignment of MAC addresses has to be such as to ensure uniqueness of the vNIC in native and adjacent layer-2 domains. 2. Assignment of the MAC and IP subnets has to account for the fact that virtual machines can move from one physical host to another both within and across physical datacenters (adjacent layer-2 domain). 3. Look up of MAC addresses in hardware switches often involves the use of Content Addressable Memory. Equivalent schemes in software have to be kept efficient in terms of CPU and memory resources. 4. Inter VM traffic does not leave the host if a vSwitch is used and as such all the services provided by the physical switch including security, Intrusion Prevention, and others need to be supplied in a different way (likely in software).

B) Transformation of Access The vSwitch or the traffic aggregator use a NICs uplinks to connect to the access layer switch ports. As such the switch port terminates an aggregation of virtual hosts. This implies that any services offered by the switch have to be on

a virtual port basis. The virtual ports may be identified by a MAC address or via a newly defined label that is tagged on the 802.3 frames (similar or a Q-tag). One or two levels of tagging are being discussed within the IEEE standards 802.1 working group. Protocols and procedures also need to be defined to assign virtual port IDs and associated capabilities across a link. Therefore both the physical switch and the NIC interface are likely to change to support the new additions to the frames on the wire. C) Configuration Complexity and Domains of Responsibility With the introduction of the virtual switch or aggregation device and vNICs, one question is where the management responsibility for these devices falls? Is it with the network or the host administration group? An ideal solution reduces the inter dependency between these teams. Therefore a current approach is to have two virtual switching devices, one for the inter-VM traffic configured by the host admin and one for the uplink traffic configured by the network admin. D) Traffic Flow Patterns Traffic flow patterns depend on the application architecture within the datacenter. For client/server web based applications traditionally North/South traffic flows have been a dominant component. Web servers, application servers, and the database server formed a three tier architecture where each tier handles a key component of the processing. With multiple VMs sharing hosts, it remains to be seen how these tiers are deployed. East West traffic flows will tend to grow in this new 4

environment. Also part of the traffic will never leave the host. New traffic types will also be sharing the switching fabric as described in the next section. To the mix one has to add creation of cloud compute infrastructures which can be deployed to enhance capacity on demand for certain applications. Latter encourage migration of applications away from the client hosts and towards large datacenter deployments (within the cloud). E) I/O Consolidation and Fabric Convergence Two technology trends tend to merge in this space. One is to use higher line speeds such as 10G Ethernet and soon to be 40 and 100G Ethernet [3]. Another is the convergence of Storage Area Network (SAN) and data traffic in the form of Fibre Channel over Ethernet (FCoE) ANSI standard [4]. Running servers at higher utilization levels, as well as merging the FC and data traffic over the same switching fabric necessitates 10G interfaces moving forward. Therefore low cost 10G ports is the lynch pin of a virtualized datacenter. A key component of a converged fabric is lossless Ethernet. Three new standards within IEEE 802.1 working group work together to achieve this capability. The first is 801.Qbb which offer priority based flow control to deal with transient congestion on a link by link basis [5]. The second is the 801.qaz which allows for scheduling of different traffic types [6]. The third is the 802.1Qau which throttles traffic at the source NIC/vNIC to prevent persistent congestion [7]. F) Security Finally switching traffic within a host hides the flow of traffic from the access layer switching

and service layers. This is where some of the security capabilities such as ACLs and perhaps firewalls and IPS may be deployed. Given that such security services are CPU intensive, providing equivalent services within a host would be a challenge. On the other hand the VMS controls all the activities of the virtual machines without being the target of client traffic which carries exploits that compromise vulnerabilities and infect systems. Worms and Viruses may have a harder time taking hold and hiding from the VMS. Security and virtual machine isolation remains a challenge for VMS moving forward. CONCLUSIONS We have articulated two views of the Virtualization Management Software (VMS), one as a meta-operating system on a physical host and another as an enabler of a datacenter wide operating system to ease the burden of managing complex infrastructure. Virtualization impacts many aspects of the network including connectivity, layers of access, domain of configuration responsibility, traffic flow patterns, need for higher speed NICs, and security. Many new standards have emerged to address these changes including modifications to the 802.1D bridging protocol, emergence of Q-tag like fields on the wire to identify virtual ports, and loss less Ethernet, as well as, ANSI FCoE. Migration of virtual machines to neighboring L2 domains over high latency WAN links is currently a challenge. An area which would challenge standardization would be VMS APIs. Such standards would help the growth of an eco-system around VMS. 5

This becomes especially important as applications take advantage of such APIs to take advantage of network based services. REFERENCES 1. IEEE 802.3-2008, IEEE Standard for Information technology-Specific requirements Parts 1 to 5. 2. IEEE P802.1D/D4, Draft Standard for Local and Metropolitan Area Networks: Media Access Control (MAC) Bridges, 2004. (Incorporates IEEE 802.1t-2001 and IEEE 802.1w). 3. IEEE P802.3ba 40Gb/s and 100Gb/s Ethernet Task Force 4. ANSI T11, Fibre Channel BB-5 Rev 1.07, May 2009. 5. IEEE 802.1Qbb, Priority Based Flow Control, draft 1.0, Sept 2009. 6. IEEE 802.1Qaz, Enhanced Transmission Selection, draft 0.4, Sept 2009. 7. IEEE 802.1Qau, Congestion Notification, draft 2.2, Sept 2009.