Table of contents Introduction 3 Requirements and support 3 Virtual Connect 3 Upstream Switch 4 FlexFabric CNAs 4 Storage Array 4 Virtual Connect Limitations 5 Dual-Hop FCoE with Nexus 5xxx Series ToR switch in FCF mode scenario 8 Requirements 9 Guidelines 9 Nexus configuration when using a single VC to Nexus link 10 Nexus configuration when using a port channel between VC and Nexus 13 Virtual Connect configuration 19 Converged Shared Uplink Sets (SUS) Details and Restrictions 20 Defining two Converged Shared Uplink Sets 21 QoS Configuration 25 Quality of Service configuration on the Nexus 5xxx series 25 Quality of Service configuration on the Virtual Connect modules 28 Nexus Troubleshooting commands 33 Useful Show commands 33 Per-Priority Flow Control commands 37 Issues with FCoE performance 38 Interface Status commands 43 FCoE Frames Analysis 44 Support and Other Resources 47
2 Contacting HP 47 Related documentation 48
3 Introduction Dual-Hop FCoE support is one of the main features of VC 4.01. This new feature allows the FCoE traffic to be propagated out of the enclosure to an external bridge which will handle the conversion of FCoE to FC traffic. Dual-Hop FCoE solution has several advantages when compared to a traditional FC + Ethernet design. With FCoE, the number of cable is reduced, this results in a simplification of the in-rack cable installation and consequently eliminates expensive equipment and reduces the overall solution cost. This feature is described as FCoE Dual Hop because there is two FCoE hops between the server and the storage the VC Module and the external FCoE ->FC bridge that connects the HP Virtual Connect modules to the storage. No additional external bridges are currently allowed in this configuration in order for Virtual Connect to guarantee the lossless of the FCoE connection.
Requirements and support Virtual Connect Dual-Hop FCoE with HP Virtual Connect is currently only supported with the following modules: HP Virtual Connect FlexFabric 10Gb/24-port Module HP Virtual Connect Flex-10/10D Ethernet Module
Firmware requirement Dual-Hop FCoE with HP Virtual Connect is supported with VC 4.01 or later.
FcoE-capable uplink ports VC FlexFabric module can only support FCoE on uplink port X1 to X4 VC Flex-10/10D module can support FCoE on ALL uplink ports (X1-X10)
X1 X4 Uplink ports available for FCoE connection X1 X10 Uplink ports available for FCoE connection VC FlexFabric 10Gb/24-port VC Flex-10/10D
FcoE-enabled uplink port limitation Long Range Ethernet Optical transceivers (LR SFP+) are not supported on Virtual Connect FCoE-enabled uplink port.
4 Upstream Switch Virtual Connect Dual-Hop FCoE is currently supported only with Cisco Nexus 5000, and Cisco Nexus 5500 product families (i.e. Nexus 5010, 5020, 5548UP and 5596UP). Supported Nexus configuration details are available in the C-Series FCoE Switch Connectivity stream document available on the following HP Single Point of Connectivity Knowledge (SPOCK) web page: http://h20272.www2.hp.com/Pages/spock2Html.aspx?htmlFile=hw_switches.html (Requires HP Passport; if you do not have an HP Passport account, follow the instructions on the webpage). This SPOCK document provides detailed information on Nexus switches, recommended NX-OS versions, supported transceivers, etc. Nexus support This cookbook does not intent to provide any detailed list of Cisco supported configurations, for information on Nexus FCoE supported configuration and design, refer to the Cisco web site. Nexus 5000 Series Design Guides http://www.cisco.com/en/US/products/ps9670/products_implementation_design_guides_list.html Data Center Design with Cisco Nexus Switches and Virtual PortChannel http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-572831- 00_Dsgn_Nexus_vPC_DG.pdf
FlexFabric CNAs SPP 2013.02 is recommended as a Baseline Release Set with the addition of Hotfix (10.70) - Linux Drivers and CNA Firmware for VC 4.01 compatibility (includes Emulex CNA FW 4.2.401.2215 and new drivers for RHEL 5.8/5.9 - RHEL 6.2/6.3 - SLES 10 SP4 - SLES 11 SP2). More details about FlexFabric CNA Firmware and FCoE Drivers are available in the Virtual Connect FCoE Modules section of the following SPOCK webpage: http://h20272.www2.hp.com/Pages/spock2Html.aspx?htmlFile=hw_virtual_connect.html (Requires HP Passport; if you do not have an HP Passport account, follow the instructions on the webpage). These Virtual Connect Connectivity Stream documents provide a detailed table of the FlexFabric CNA firmware and FCoE drivers that are supported by HP. Note: For more information about HP Service Pack for ProLiant and Hotfixes, see http://h18004.www1.hp.com/products/servers/service_packs/en/index.html
Storage Array SPOCK is the HP authoritative source of interoperability information for HP storage products. A particular configuration is supported if and only if, it is listed on SPOCK. HP Storage Arrays supported with Virtual Connect Dual-Hop configuration are listed in the Virtual Connect Connectivity Stream documents available on the following SPOCK webpage: http://h20272.www2.hp.com/Pages/spock2Html.aspx?htmlFile=hw_virtual_connect.html For information about Non-HP Storage support, firmware versions and specific supported configuration information, consult your storage equipment vendor.
5 Virtual Connect Limitations Only Dual-Hop FCoE is supported by HP at this time.
Figure 1: Dual-Hop FCoE Split FC & Ethernet at Top of Rack FCoE switch C7000 with Virtual Connect SAN LAN C7000 with Virtual Connect FCoE switch FCoE switch FCoE FCoE
Only one FCoE network can be associated with any single set of uplink ports.
FCoE-capable Shared Uplink Set (SUS) can be used to allow concurrent Ethernet and FCoE traffic, but only one of the networks in the SUS can be an FCoE Network.
Figure 2: Virtual Connect SUS configuration when using FCoE SUS-1 FCoE-1 VLAN-30 VLAN-20 VLAN-10 LACP FCoE Lossless Network VC Ethernet networks Converged-SUS defined to pass FCoE traffic and traditional Ethernet traffic
6 Virtual Connect requires the creation of an Active/Active Shared Uplink Set configuration for the FCoE traffic.
Figure 3: Active-Active FCoE Virtual Connect configuration FCoE ToR Switch 1 FCoE ToR Switch 2 FCoE FCoE VC Domain FCoE ToR Switch 1 FCoE ToR Switch 2 FCoE FCoE SUS-1 VC Domain SUS-2 SUS Lossless Ethernet Network Lossless Ethernet Network Active Active
FCoE-capable Shared Uplink Sets must always contain ports from a single VC module in order to maintain the SAN-A/B isolation.
Figure 4: FCoE-capable Shared Uplink Set configuration support FCoE ToR Switch 1 FCoE ToR Switch 2 FCoE FCoE VC Domain FCoE ToR Switch 1 FCoE ToR Switch 2 FCoE FCoE SUS-1 VC Domain SUS-2 SUS-1 SUS-2
7 FCoE-capable Shared Uplink Sets must always be connected to a single Top of Rack switch in order to maintain the SAN-A/B isolation.
Figure 5: FCoE-capable Shared Uplink Set LACP configuration support FCoE ToR Switch 1 FCoE ToR Switch 2 FCoE FCoE VC Domain FCoE ToR Switch 1 FCoE ToR Switch 2 FCoE FCoE SUS-1 VC Domain SUS-2 SUS-1 SUS-2 Active Active Lossless Ethernet Network Active Active Lossless Ethernet Network LACP LACP LACP LACP Ethernet forwarding through a link aggregation can break SAN A/B separation
8 Dual-Hop FCoE with Nexus 5xxx Series ToR switch in FCF mode scenario
In this scenario, the Nexus switches operate as Fiber Channel Forwarders (FCF). This is the default Cisco Nexus 5xxx Series switches mode, its also called the fabric mode. In this mode, the switch provides standard Fibre Channel switching capability and features.
Figure 6: Cisco Nexus 5xxx switches operating as FC Forwarder (FCF) E_Port N_Port VF_Port VN_Port Virtual Connect (FIP Snooping mode) FCF FCoE FC Native FC Lossless Ethernet network FC Storage Array F_Port N_Port Server (ENode) Cisco MDS FC Storage Array E_Port F_Port Native FC Dual-Hop FCoE Nexus 5xxx Series (Fabric Mode)
This cookbook provides only basic Cisco guidelines and configuration steps, for a more complete and detailed list of Nexus FCoE configuration guidelines, refer to the Cisco NX-OS Fibre Channel over Ethernet Configuration Guides: Cisco Nexus 5000 Series NX-OS Fibre Channel over Ethernet Configuration Guide, Release 5.2(1)N1(1) http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/fcoe/521n11/b_5k_FCoE_Config_521N1 1.pdf Cisco Nexus 5500 Series NX-OS Fibre Channel over Ethernet Configuration Guide, Release 6.0(2)N1(1) http://www.cisco.com/en/US/docs/switches/datacenter/nexus5500/sw/fcoe/602_N1_1/b_5500_FCoE_Config_6 02N11.pdf
9 Figure 7: Physical view FCoE FCoE FC FC FCoE HP Virtual Connect FlexFabric or Flex-10/10D modules FC-FCoE FC Storage Array Nexus 5xxx Series ToR Switch using Fabric Mode VC Domain LAN FC SAN B FC SAN A Cisco MDS Series SAN Switch
Requirements Minimum NX-OS version 5.2(1)N1(3) FCoE requires the Nexus Storage Protocols Services license (FC_FEATURES_PKG)
Note: The license is an option which is activated when the Nexus switch is shipped from Cisco.
Guidelines Nexus switches must either bridge to native FC infrastructure or directly connect to FC/FCoE-based Storage devices. Refer to the Cisco Nexus or Storage vendor interoperability Matrix. When bridged to a native FC infrastructure, it is mandatory to use Cisco MDS directors or fabric switches in order to provide interoperability between fabrics. To increase the FCoE traffic identification and to better control the span of this traffic over the Ethernet network, it is recommended to use different FCoE VLANs and VSANs numbers between the two fabrics. The FCoE VLAN should be dedicated to FCoE traffic (i.e. it should not carry IP traffic). The FCoE VLAN must not be configured as a native VLAN (the VLAN that carries untagged traffic on trunk ports, by default VLAN 1). Interfaces connecting to VC must be configured as trunk ports and STP edge ports. (STP does not run on FCoE VLANs between FCFs (VE_Ports) but does run on FCoE VLANs towards the host (VF_Ports)).
10 Nexus configuration when using a single VC to Nexus link
Details about the configuration: Interfaces eth1/5 are connected to the VC modules. Interfaces fc2/1 are directly connected to Cisco MDS 9148 switches. VLAN IDs 200 and 201 are used for the FCoE networks. VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-FCoE networks). The vfc interface (virtual Fibre Channel interface) binds to eth1/5.
Figure 8: Physical diagram HP BladeSystem c7000 fc2/1 LAN fc2/1 eth1/5 eth1/5 FCoE/DCB FC LAN X3 X3 eth1/17 eth1/17 FCoE VLAN 200 FCoE VLAN 201 VLAN 1,10,20 VLAN 1,10,20 VSAN 201 VSAN 200 VLAN 1,10,20 VLAN 1,10,20 vfc 2005 vfc 2005 FCF Switch A Nexus-5010 VC FlexFabric Bay 1 VC FlexFabric Bay 2 Blade Server FCoE NIC CNA Port 1 CNA Port 2 FCoE NIC WWN: 50:06:0b:00:00:c3:1a:22 WWN: 50:06:0b:00:00:c3:1a:20 3PAR Storage Array FCF Switch B Nexus-5010 FCoE / DCB Native Ethernet Native Fibre Channel WWN: 20:53:00:02:ac:00:15:9d WWN: 21:53:00:02:ac:00:15:9d SAN-A SAN-B Cisco MDS 9148
11 Nexus switch-A configuration Upgrade the first Nexus switch with minimum System version 5.2(1)N1(3) (enter: show version)
Enable FCoE on the switch (disabled by default): o configure terminal o feature fcoe
Note: The Nexus will require a reload.
Associate a VLAN with a VSAN for FCoE traffic: o vlan 200 o fcoe vsan 200
Create a virtual Fibre Channel interface to carry the FCoE traffic through eth1/5: o interface vfc 2005 o bind interface ethernet 1/5 o no shutdown
Create the VLANs for the IP traffic: o vlan 1,10,20
Create a trunk on the interface to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20): o interface Ethernet1/5 o description FCoE uplink to FlexFabric o switchport mode trunk o switchport trunk allowed vlan 1,10,20,200 o spanning-tree port type edge trunk
Assign the vfc interface and the Fibre Channel port to the FCoE VSAN: o vsan database o vsan 200 o vsan 200 interface fc2/1 o vsan 200 interface vfc 2005
Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20
Configuration of the zone: a. Create zones: o zone name fcoe-zone vsan 200 o member pwwn 21:53:00:02:ac:00:15:9d {This is the WWN of the first 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:20 {This is the WWN of the Blade FlexFabric Adapter port 1} b. Create zoneset: o zoneset name zoneset1 vsan 200 o member fcoe-zone c. Activate zoneset: o zoneset activate name zoneset1 vsan 200
12 Nexus switch-B configuration Upgrade the second Nexus switch with minimum System version 5.2(1)N1(3) (enter: show version)
Enable FCoE on the switch (disabled by default): o configure terminal o feature fcoe
Note: The Nexus will require a reload.
Associate a VLAN with a VSAN for FCoE traffic: o vlan 201 o fcoe vsan 201
Create a virtual Fibre Channel interface to carry the FCoE traffic through eth1/5: o interface vfc 2005 o bind interface ethernet 1/5 o no shutdown
Create the VLANs for the IP traffic: o vlan 1,10,20
Create a trunk on the interface to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20): o interface Ethernet1/5 o description FCoE uplink to FlexFabric o switchport mode trunk o switchport trunk allowed vlan 1,10,20,201 o spanning-tree port type edge trunk
Assign the vfc interface and the Fibre Channel port to the FCoE VSAN: o vsan database o vsan 201 o vsan 201 interface fc2/1 o vsan 201 interface vfc 2005
Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20
Configuration of the zone: a. Create zones: o zone name fcoe-zone vsan 201 o member pwwn 20:53:00:02:ac:00:15:9d {This is the WWN of the second 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:22 {This is the WWN of the Blade FlexFabric Adapter port 2} b. Create zoneset: o zoneset name zoneset1 vsan 201 o member fcoe-zone c. Activate zoneset: o zoneset activate name zoneset1 vsan 201
13 Nexus configuration when using a port channel between VC and Nexus
Current NX-OS limitations All ports in the FCoE port channel must belong the same Unified Port Controller (UPC) ASIC; you cannot configure port channels across UPCs. Nexus 5010/5020 switches can support a maximum of four ports per port-channel when carrying FCoE traffic. Nexus 5548/5596 switches can support a maximum of eight ports per port-channel when carrying FCoE traffic.
Note: These limitations will be removed in a future NX software release.
Details about the configuration: Interfaces eth1/5 and eth1/6 are connected to the VC modules. Interfaces fc2/1 are directly connected to Cisco MDS 9148 switches. VLAN IDs 200 and 201 are used for the FCoE networks. VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-FCoE networks). The vfc interface (virtual Fibre Channel interface) binds to the Port Channel 200 configured with eth1/5 and eth1/6.
Figure 9: Physical diagram HP BladeSystem c7000 3PAR Storage Array fc2/1 LAN fc2/1 eth1/5 eth1/6 eth1/5 eth1/6 FCoE/DCB FC LAN X3 - X4 X3 X4 eth1/17 eth1/17 VLAN 1,10,20 VLAN 1,10,20 VSAN 201 VSAN 200 FCoE VLAN 200 VLAN 1,10,20 FCoE VLAN 201 VLAN 1,10,20 vfc 2005 vfc 2005 FCF Switch A Nexus-5010 FCF Switch B Nexus-5010 VC FlexFabric Bay 1 VC FlexFabric Bay 2 Blade Server FCoE NIC CNA Port 1 CNA Port 2 FCoE NIC FCoE / DCB Native Ethernet Native Fibre Channel WWN: 50:06:0b:00:00:c3:1a:22 WWN: 20:53:00:02:ac:00:15:9d WWN: 21:53:00:02:ac:00:15:9d WWN: 50:06:0b:00:00:c3:1a:20 po200 po200 SAN-A SAN-B Cisco MDS 9148
14 Identifying ports that belong to the same Unified Port Controller It is necessary while configuring the FCoE port channel group to use interfaces that belong to the same UPC ASIC (e.g. Eth1/5 and Eth1/6 because both interfaces are mapped to the same ASIC, UPC#1). In order to identify ports that belong to same UPC, it is necessary to use the show hardware internal command. On Nexus 5010/5020 switches:
The Nexus 5010 and 5020 switches use Gatos Unified Port Controller ASIC (UPC Gen 1). Each UPC ASIC have 4 ports mapped to front panel ports. This can be seen with the following command:
Note: Column 1- Indicates the physical ports e.g., Eth1/1, Eth1/3, Eth1/9, Eth1/17, etc. Column 3- Indicates the UPC ASIC number. Column 4- Indicates the UPC ASIC interface number. 0-3 are the 4 port available on the Gen-1 ASIC.
Note: Only ports identified as mapped to the same UPC ASIC can be member of the same port channel. For example, Eth1/1 to Eth1/4 can be used in the same port channel because they are all mapped to UPC#0.
15 Figure 10: Interface Eth1/5 cannot be part of a same Port Channel as Eth1/1 Eth1/4 0 2 3 Eth1/1 . . . 10G Ethernet Interfaces Only these interfaces can be part of the same FCoE Port Channel UPC #0 xgb1/3 = Eth1/3 LACP 0 1 2 3 UPC #1 Eth1/2 Eth1/3 Eth1/6 Eth1/4 Eth1/5 1
On Nexus 5548/5596 switches:
The Nexus 5548 and 5596 switches use Carmel Unified Port Controller ASIC (UPC Gen 2). Each UPC ASIC have 8 ports mapped to front panel ports. This can be seen with the following command:
Note: Column 1- Indicates the physical ports e.g., Eth1/1, Eth1/3, Eth1/9, Eth1/17, etc. Column 3- Indicates the UPC ASIC number. Column 4- Indicates the UPC ASIC interface number. 0-7 are the 8 port available on the Gen-2 ASIC.
Note: Only ports identified as mapped to the same UPC ASIC can be member of the same port channel. For example, Eth1/1 to Eth1/8 can be used in the same port channel because they are all mapped to UPC#0.
16 Nexus switch-A configuration Upgrade the first Nexus switch with minimum System version 5.2(1)N1(3) (enter: show version)
Enable FCoE and LACP on the switch (disabled by default): o configure terminal o feature fcoe o feature lacp
Note: The Nexus can require a reload.
Associate a VLAN with a VSAN for FCoE traffic: o vlan 200 o fcoe vsan 200
Create a port channel with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010: o interface ethernet 1/5 o channel-group 200 mode active o lacp rate fast o interface ethernet 1/6 o channel-group 200 mode active o lacp rate fast
Create the VLANs for the IP traffic: o vlan 1,10,20
Create a trunk on the port channel interface to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20): o interface port-channel 200 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,200 o spanning-tree port type edge trunk
Create a virtual Fibre Channel interface to carry the FCoE traffic through the port channel: o interface vfc 2005 o bind interface port-channel 200 o no shutdown
Assign the vfc interface and the Fibre Channel port to the FCoE VSAN: o vsan database o vsan 200 o vsan 200 interface fc2/1 o vsan 200 interface vfc 2005
Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20
17 Configuration of the zone: a. Create zones: o zone name fcoe-zone vsan 200 o member pwwn 21:53:00:02:ac:00:15:9d {This is the WWN of the first 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:20 {This is the WWN of the Blade FlexFabric Adapter port 1} b. Create zoneset: o zoneset name zoneset1 vsan 200 o member fcoe-zone c. Activate zoneset: o zoneset activate name zoneset1 vsan 200
Nexus switch-B configuration Upgrade the first Nexus switch with minimum System version 5.2(1)N1(3) (enter: show version)
Enable FCoE and LACP on the switch (disabled by default): o configure terminal o feature fcoe o feature lacp
Note: The Nexus will require a reload.
Associate a VLAN with a VSAN for FCoE traffic: o vlan 201 o fcoe vsan 201
Create a port channel with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010: o interface ethernet 1/5 o channel-group 200 mode active o lacp rate fast o interface ethernet 1/6 o channel-group 200 mode active o lacp rate fast
Create the VLANs for the IP traffic: o vlan 1,10,20
Create a trunk on the port channel interface to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20): o interface port-channel 200 o switchport mode trunk o switchport trunk allowed vlan 1,10,20,201 o spanning-tree port type edge trunk
Create a virtual Fibre Channel interface to carry the FCoE traffic through the port channel: o interface vfc 2005 o bind interface port-channel 200 o no shutdown
18 Assign the vfc interface and the Fibre Channel port to the FCoE VSAN: o vsan database o vsan 201 o vsan 201 interface fc2/1 o vsan 201 interface vfc 2005
Configure the interface connected to the datacenter LAN: o interface eth 1/17 o switchport mode trunk o switchport trunk allowed vlan 1,10,20
Configuration of the zone: a. Create zones: o zone name fcoe-zone vsan 201 o member pwwn 20:53:00:02:ac:00:15:9d {This is the WWN of the second 3PAR controller port} o member pwwn 50:06:0b:00:00:c3:1a:22 {This is the WWN of the Blade FlexFabric Adapter port 2} b. Create zoneset: o zoneset name zoneset1 vsan 201 o member fcoe-zone c. Activate zoneset: o zoneset activate name zoneset1 vsan 201
19 Virtual Connect configuration Virtual Connect requires the creation of an Active/Active configuration of two converged Shared Uplink Sets (SUS) which contain one FCoE network and optionally associated non-FCoE network(s).
Figure 11: Virtual Connect Active-Active Converged Shared Uplink Sets scenario VC Module bay 2 Active FCF Mode FlexHBA FlexNIC FlexNIC FlexNIC VC Domain VC Module bay 1 FCF Mode Active FlexHBA FlexNIC FlexNIC FlexNIC vSwitch or NIC Teaming FlexFabric Adapter Port 1 FlexFabric Adapter Port 2 Multi Pathing Nexus 5xxx Series ToR Switch using Fabric Mode X1 X1 X2 X2 DCB-1 DCB-2 802.1q 802.3ad 802.1q 802.3ad Lossless Ethernet Network Converged Shared Uplink Sets Blade Server
20 An alternative scenario can be created using dedicated Share Uplink Sets for the FCoE traffic combined with Shared Uplink Sets for the standard Ethernet traffic since it is not mandatory to associate non-FCoE networks to a Converged Shared Uplink Set network.
A typical scenario for this is when the Virtual Connect Domain cannot be modified and is already configured with Shared Uplink Sets that are using either Non-FCoE capable VC uplink ports or Dual Homed LACP configurations (it is not supported to connect a FCoE-capable Shared Uplink Set to two Top of Rack switches).
This scenario is using additional uplink ports however these extra links used by the FCoE traffic can provide a better service level because the link is dedicated to the FCoE traffic and is not shared with any other types of network.
Figure 12: Alternative Virtual Connect Active-Active Converged Shared Uplink Sets scenario with Dedicated SUS for the FCoE traffic. FCF Mode FlexHBA FlexNIC FlexNIC FlexNIC VC Domain FCF Mode Blade Server FlexHBA FlexNIC FlexNIC FlexNIC vSwitch or NIC Teaming FlexFabric Adapter Port 1 FlexFabric Adapter Port 2 Multi Pathing Nexus 5xxx Series ToR Switch using Fabric Mode SUS-1 FCoE Shared Uplink Set (Non-FCoE traffic) SUS-FCoE-1 SUS-FCoE-2 FCoE SUS-2 Active Non-FCoE Active Non-FCoE vpc peer link VC Module bay 1 VC Module bay 2 Converged Shared Uplink Set (FCoE traffic) Lossless Ethernet Network Dual Homed LACP SUS configuration
Converged Shared Uplink Sets (SUS) Details and Restrictions FCoE-capable Shared Uplink Sets (SUS) can contain both the FCoE network and non-FCoE networks. FCoE-capable SUS can support port aggregation (802.3ad). FCoE-capable SUS can support only one FCoE network. FCoE-capable SUS must always contain ports from a single VC module.
Note: For Multi Enclosure Stacking (MES) environments, all corresponding ports in remote enclosures will be included in the same SUS. (e.g. selecting enc0:bay1:X1 means bay1:X1 in all remote enclosures is also included).
21 Defining two Converged Shared Uplink Sets
Create a SUS and name it DCB-1 On the Virtual Connect Manager screen, click Define, Shared Uplink Set to create a SUS.
Enter the Network Name of DCB-1. Select Add Port, then add one or more ports from Bay 1.
Note: In order to reduce the risk of congestion and increase the performance, make sure to use a suitable number of uplink to reach a good oversubscription ratio.
For Connection Mode, use Auto. For LACP Timer, leave the Domain Default, Short (1 sec). This setting controls the requested frequency of LACP control messages on a LACP capable interface. Using the "long" setting can help prevent loss of LAGs while performing in- service upgrades on upstream switch firmware. For Associated FCoE Network, click on Add then enter: o Network Name: FCoE-1. o VLAN ID: enter the VLAN ID used by the FCoE traffic, it should match with the FCoE VLAN ID used by the upstream switch. For Associated Networks, click on Add then enter all Ethernet networks that are required. o Select Smart Link for all defined networks.
22 Create a SUS and name it DCB-2 Enter the Network Name of DCB-2. Select Add Port, then add one or more ports from Bay 2.
Note: In order to reduce the risk of congestion and increase the performance, make sure to use a suitable number of uplink to reach a good oversubscription ratio.
For Connection Mode, use Auto. For LACP Timer, leave the Domain Default, Short (1 sec). This setting controls the requested frequency of LACP control messages on a LACP capable interface. Using the "long" setting can help prevent loss of LAGs while performing in- service upgrades on upstream switch firmware. For Associated FCoE Network, click on Add then enter: o Network Name: FCoE-2. o VLAN ID: enter the VLAN ID used by the FCoE traffic, it should match with the FCoE VLAN ID used by the upstream switch. For Associated Networks, click on Add then enter all Ethernet networks that are required. o Select Smart Link for all defined networks.
Note: For the removal of ports from a Converged Shared Uplink Set, it is recommended to first shutdown or remove a port from the vfc (interface vfc xx - no bind interface ethx/x) or channel group (interface ethx/x no channel-group xx mode active) on the FCF prior to removing the corresponding port from the Converged Shared Uplink Set in order to allow the FCF to properly halt traffic on the link. Failure to do so may result in loss of storage access.
23 FCoE Server traffic configuration Unifying storage traffic and network traffic on the same link can present some challenges and usually requires more careful bandwidth management than standard Ethernet environments. One easy solution is to maintain the server traffic to reasonable connection speed by using the VC Maximum connection speed option. This option limits the server traffic by providing a maximum connection speed option for defined networks. This option is recommended for high oversubscribed environment where the number of FCoE uplinks are limited which can increase the risk of congestion and could badly impact the network performance.
Note: Maximum bandwidth optimization requires the Flex-10 and FlexFabric adapter firmware and drivers to be updated to SPP version 2013.02.00 with Hotfix (10.70) - Linux Drivers and CNA Firmware for VC 4.01 compatibility, to take advantage of this enhancement. This feature excludes support for the following adapters: HP NC551i Dual Port FlexFabric 10Gb Converged Network Adapter HP NC551m Dual Port FlexFabric 10Gb Converged Network Adapter HP NC550m 10Gb 2-port PCIe x8 Flex-10 Ethernet Adapter
To configure a maximum bandwidth limit from Server for the FCoE traffic: Go to the Ethernet Networks and select the first FCoE network and click Edit.
Check the Advanced Network Settings option. Check Set Maximum Connection Speed. Select the best suitable speed for the maximum connection speed in order to meet your network design. Click Apply and Apply again to save the new FCoE network configuration.
This value will limit the maximum port speed from Server for the FlexHBA associated with this FCoE network.
24 Note: With 2 x 10Gb uplinks per VC modules connected to a Top of Rack switch using 50/50 bandwidth share and 16 servers using the FCoE network, it might be necessary to limit the maximum connection speed to get a reasonable FCoE over-subscription: o For a 2:1 oversubscription, use 1.3Gb o For a 4:1 oversubscription, use 2.5Gb o For a 8:1 oversubscription, use 5Gb o For a 16:1 oversubscription, use 10Gb
Repeat the same steps for the second FCoE network.
25 QoS Configuration The default Virtual Connect configuration provides lossless service of the FCoE traffic. PFC (Priority Flow Control) mechanisms are used to provide the FCoE lossless service and prevent congestion. It is therefore not necessary to enable any particular QoS settings in the VC Domain. Since Virtual Connect provides only best effort service for Ethernet traffic (i.e. flow control does not take place, in the event of congestion, packets get dropped) Administrator can decide to configure QoS in the Virtual Connect Domain and on the Nexus switches to improve the Ethernet traffic service level control. The bandwidth allocation for FCoE-capable SUS uplinks always allocates only half the link bandwidth to FCoE traffic, with the other half allocated to Ethernet traffic. With VC 4.01, this allocation is fixed and cannot be modified. Therefore it is important to configure the Nexus switches with the same Virtual Connect Domain QoS settings as follow: FCoE lossless traffic class 3. 50/50% of bandwidth sharing between regular and lossless traffic. Conflicting QoS parameter settings between the Virtual Connect Domain and the Nexus environment can badly impact the traffic and affect the performance.
Quality of Service configuration on the Nexus 5xxx series QoS on Nexus 5010 and 5020 On the Nexus 5010 and 5020 it is not necessary to modify the QoS settings because by default once the FCoE feature is configured (by entering feature fcoe) the default QoS template for shared links allocates the same shares and priorities than Virtual Connect. The following two classes of service are automatically configured: class-fcoe is configured to be no-drop with a COS priority 3 and a MTU of 2158 with an ingress/egress bandwidth allocation of 50%. class-default is configured to be drop with an MTU of 1500 with an ingress/egress bandwidth allocation of 50%.
26 To verify that the FCoE policy maps can be found in the running configuration, enter: show policy-map system
QoS on Nexus 5500 series On the Nexus 5500 series, enabling the FCoE feature does not create automatically no-drop policies like on the Nexus 5000 series, so before enabling FCoE, it is necessary to configure the QoS policies manually by including class-fcoe in each of the following policy types: Network-Qos Queuing Qos
The following is an example of a service policy that needs to be configured: o class-map type qos class-fcoe o class-map type queuing class-fcoe o match qos-group 1 o class-map type queuing class-all-flood o match qos-group 2
27 o class-map type queuing class-ip-multicast o match qos-group 2 o class-map type network-qos class-fcoe o match qos-group 1 o class-map type network-qos class-all-flood o match qos-group 2 o class-map type network-qos class-ip-multicast o match qos-group 2 o system qos o service-policy type qos input fcoe-default-in-policy o service-policy type queuing input fcoe-default-in-policy o service-policy type queuing output fcoe-default-out-policy o service-policy type network-qos fcoe-default-nq-policy
To verify the FCoE system class is active on the interface connected to VC, enter: show queuing interface Ethernet 1/5
For more detailed description of the QoS configuration on Nexus switches, see the Cisco Nexus 5000 Series NX-OS Quality of Service Configuration Guide at the following URL: http://www.cisco.com/en/US/products/ps9670/products_installation_and_configuration_guides_list.html
28 Quality of Service configuration on the Virtual Connect modules It is not necessary to enable any particular QoS settings in the VC Domain to provide Quality of Service for the FCoE traffic. The moment FCoE is configured, Virtual Connect provides lossless service for the FCoE traffic. By default, the VC QoS configuration mode is Passthrough. In this mode, two classes of service are provided, one for FCoE with lossless service and one for Ethernet without any form of flow control. To improve the Ethernet traffic service level control and make VC to take a specific action on a specified classified traffic, the VC QoS settings can be changed from Passthrough to Custom (with FCoE Lossless). In this mode, VC monitors the Ethernet traffic for Layer 2 802.1p Priority bits, or Layer 3 DSCP or ToS markings and places packets on the pre-defined egress queues. The Ethernet traffic is then prioritized based on the traffic classification.
Optional VCM QoS Configuration On the Virtual Connect Manager screen, click Configure then Quality of Service (QoS).
Change the default Pass-through mode to Custom (with FCoE Lossless).
29 In this mode, VC supports up to 8 configurable traffic classes: o 1 predefined system class for Best Effort. o 1 predefined system class for FCoE Lossless. o 6 user defined classes.
Note: The default 802.1p priority for the Lossless FCoE traffic class is 3.
Note: The FCoE Lossless traffic share is based on the VC server profile configuration. The MAX Share is based on the FCoE Fabric configuration.
For each user defined classes you want to use, check the Enabled box and enter the appropriate Share/Max share and 802.1p priority.
Then click Apply.
30 Connecting Servers to the FCoE networks
To connect Blade servers to FCoE networks that have been configured, go to the Server Profiles interface. Select a profile and click Edit.
Select the first FCoE HBA Connection and select the first FCoE network FCoE-1.
The server administrator can select the FCoE network like any other storage network when configuring profiles.
31 Select the second FCoE HBA Connection and select the second FCoE network FCoE-2.
The two Active/Active FCoE networks are now assigned to the server FCoE HBA.
To configure a Minimum Allocated Port Speed for the FCoE traffic: Select Custom from the Port Speed Type drop down menu of the first FCoE port.
Then select the port speed, this speed is the minimum guaranteed bandwidth. Maximum Connection Speed that had been defined for FCoE-1
Note: Make sure to select the best suitable Minimum Port Speed connection in order to meet your network design.
Click Ok.
Note: The Minimum Allocated Port Speed feature is an additional solution to overcome the challenges of network convergence discussed before. This Virtual Connect option guarantees that the FCoE traffic from server will get its minimum allocated bandwidth even if other FlexNICs on the same port generate too much traffic. Furthermore, if a congestion event occurs, the bandwidth is allocated based on the ratio of the Minimum setting. This option can really improve the Virtual connect bandwidth utilization and network performance.
32 Note: Minimum and maximum bandwidth optimization requires the Flex-10 and FlexFabric adapter firmware and drivers to be updated to SPP version 2013.02.00 with Hotfix (10.70) - Linux Drivers and CNA Firmware for VC 4.01 compatibility, to take advantage of this enhancement. This feature excludes support for the following adapters: o HP NC551i Dual Port FlexFabric 10Gb Converged Network Adapter o HP NC551m Dual Port FlexFabric 10Gb Converged Network Adapter o HP NC550m 10Gb 2-port PCIe x8 Flex-10 Ethernet Adapter
Repeat the same steps for the second FCoE port.
Click Apply to activate the profile configuration.
The Allocated Port Speed should now be displayed with the Min and Max that have been defined.
The next step is to configure under the Server Operating System the NIC teaming and the FC multi-pathing for failover.
33 Nexus Troubleshooting commands For more detailed information about Nexus FCoE troubleshooting guidance, see http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/troubleshooting/guide/n5K_ts_fcoe.html Disclaimer: we used the Cisco Nexus 5000 FCoE Troubleshooting Guide to create this section. Useful Show commands show flogi database
o This command displays all logins (server nodes and storage devices) which have logged into the Fabric using FC and FCoE. Each FCoE Adapter port logs into the Fabric using the same virtual FC interface. The vfc are listed along with the VC assigned WWN addresses, the VSAN numbers and the FCIDs assigned by the Fabric to every single server.
show fcoe database
o This command is the same as the previous one but only displays the FCoE logins. o It provides also the VC assigned MAC address of the FCoE adapter. o Interfaces are listed with their FCIDs assigned by the Fabric and their PWWNs and MAC addresses assigned by Virtual Connect.
show fcoe
o This command shows if FCoE is enabled. Server #1 Server #2 Server #3
34 o It displays the FCF-MAC and the FC-MAP of the Nexus device. This command is useful because every FCoE frame sent by servers always use this FCF-MAC address as their Destination MAC Address.
show fcdomain
show vsan 200 membership
35 show policy-map
o 2 traffic classes are defined, one for the FCoe traffic and one for Ethernet.
show class-map class-fcoe
o The FCoE traffic class is defined with priority 3.
Note: COS 3 is the default Industry standard value for the FCoE Lossless traffic class (IEEE 802.1p). The same priority is used by the Virtual Connect Domain.
36 show queuing interface ethernet 1/5
o This commands shows the ETS (Enhanced Transmission Selection - IEEE 802.1Qaz) configuration of an interface. o It shows two traffic classes, one for the FCoE traffic (no-drop, MTU 2240) and one for the Ethernet traffic (drop, MTU 1500). o The bandwidth allocation for each traffic class is 50% of the link bandwidth.
show logging last 20
o This command displays the last 20 system log messages of the device. o Always very helpful, the Nexus logs can provide useful messages to help the troubleshooting.
show logging | grep vfc o This command is useful to check the status of the virtual interface used for the FCoE traffic. o The vfc can be up, down, administratively down, etc. o A down but waiting for flogi can mean that the setup is ok but no blade server has been turned on yet. 50% of the link bandwidth is reserved to the FCoE traffic qos-group 1 defines the lossless type of service (FCoE).
37 Per-Priority Flow Control commands Make sure PFC (Per-priority Flow Control - 802.1Qbb) is enabled on the interface connected to VC. Flow Control must be disabled. PFC ensures the lossless behavior of the FCoE traffic.
Check the status of Flow Control; it must be turned off on interfaces connected to Virtual Connect. o show interface ethernet 1/5 flowcontrol
Check the status of the PFC. It must be turned on on interfaces connected to Virtual Connect. o show interface ethernet 1/5 priority-flow-control
If the command shows Mode as Off or On, it should be changed to Auto. To force PFC mode Auto on an interface, enter: interface ethernet 1/5 priority-flow-control mode auto
If the command shows Mode as Auto and Oper as Off or if the command returns nothing, you should check the setup of the Shared Uplink Set in VCM by making sure that an associated FCoE Network has been properly defined.
38 Issues with FCoE performance If FCoE throughput on servers is very low, this can be due to pause frames. It is necessary to check if the switch is sending pause frames or if it is getting paused.
Possible Cause: If the egress FC port is congested, then the switch will send PFC frames to the Virtual Connect uplink port. The PFC frames are sent to reduce its FCoE rate and avoid a drop. If the server is slow or congested, then the server will send PFC frames to the Virtual Connect downlink port. As a result Virtual Connect will eventually send PFC Frames to the Nexus interface.
To verify this situation, it is necessary to check the Nexus interface that is connected to the Virtual Connect modules.
When slow FCoE performance is not due to Pause Frames For a few iterations using the "show interface ethx/x priority-flow-control" command, the PFC frame count (RX/TX) is not incrementing:
For a few iterations using the "show queuing interface ethx/x" command, the pause status are detected for both RX and TX as Inactive:
39 Solution: The slow FCoE performance does not seem to be due to network congestion. Check the Nexus interface to find information about possible link issues: Use the show interface command to verify that the Nexus interface connected to VC is not showing errors, drops, discards, etc. Use the show interface command to verify that the vfc interface is not showing errors or discards. Use the show queuing interface to verify the number of discards.
When slow FCoE performance is due to Pause frames transmitted by the Nexus switch.
Figure 13: Nexus sending PFC Pause frames due to FC congestion.
FC Target LAN PPP Congestion Point Nexus interface connected to Virtual Connect that needs to be monitored
40 When the Nexus switch is sending Pause frames to VC, the Nexus interface that is connected to the Virtual Connect module is showing the following: For a few iterations using the "show queuing interface ethx/x" command, the pause status are detected for TX as Active:
For a few iterations using the "show interface ethx/x priority-flow-control" command, the pause TxPPP counter is incrementing:
Solution: Identify the source of the congestion and try to resolve it by increasing the FC bandwidth (or by adding additional Nexus FC ports to connect to the SAN environment).
41 When slow FCoE performance is due to Pause frames transmitted by VC/Servers.
Figure 14: Virtual Connect sending PFC Pause frames due to congestion.
FC Target LAN PPP Congestion Point PPP Congestion Point Nexus interface connected to Virtual Connect that needs to be monitored
When the Virtual Connect uplink is sending Pause frames to the Nexus, the Nexus interface that is connected to the Virtual Connect module is showing the following:.
For a few iterations using the "show queuing interface ethx/x" command, the pause status are detected for Rx as Active as shown below:
42
For a few iterations using the "show interface ethx/x priority-flow-control" command, the pause RxPPP counter is incrementing:
Solution: Identify the source of the congestion and try to resolve it with the following action: 1. Increase the FCoE connection speed (Min and Max) of every servers. 2. Increase the number of VC FCoE Uplink ports connected to the Nexus.
Note: If congestion is expected, then Pause is expected for FCoE traffic.
Note: PFC Pause frames are currently not available in the Virtual Connect statistics. Only the standard Ethernet Pause frames are displayed.
43 Interface Status commands To check the status of the port connected to VC, you can enter: show interface status err-disabled
VFC is usually down due to FIP solicitation failure and possible reasons are: o Fabric is not available for flogi. o Zoning or Storage presentation are misconfigured.
show interface vfc2005
show interface vfc2005 brief
44 FCoE Frames Analysis For network troubleshooting, capturing frames with a protocol analyzer can be very instructive but it is necessary to understand the way FCoE works.
FCoE traffic use a different MAC address than the MAC address of the FCoE Adapter, the physical MAC address of the FCoE adapter (i.e. the MAC displayed in the Virtual Connect interface) is only used during FIP negotiation (FCoE Initialization Protocol).
Figure 15: MAC address of the FCoE adapter found in the Server profile interface.
During the FIP negotiation, the FCF assigned a Fabric Provided MAC address (FPMA) to the server. This address is used for FCoE forwarding and is the one Wireshark will detect during FCoE network captures.
The FPMA is built by appending the FCID (assigned by the Fabric to the server) to the Nexus FC-MAP.
Figure 16: The way the FPMA is built. FC-MAPs 0E-FC-00 to 0E-FC-FF FC-ID E5.01.00 FC-MAP 0E-FC-00 FC_ID E5.01.00 24 Bits 24 Bits FPMA 48 bits 0E-FC-00-E5-01-00
45 FPMA addresses assigned to servers cannot be found from the Virtual Connect interface, it is necessary to launch the Nexus console and use the following commands: show fcoe show fcoe database
Figure 17: How to find Fabric Provided MAC Addresses.
FPMA of server 1 : Oe:fc:00:e5:01:00 FPMA of server 2 : Oe:fc:00:e5:03:00 FPMA of server 3 : Oe:fc:00:e5:02:00
FCoE Frames sent by servers: Destination MAC: the destination of FCoE frames sent by servers is always the MAC address of the Fibre Chanel Forwarder (i.e. the FCF-MAC address of the Nexus switch). Source MAC: the source of an FCoE frame is the Fabric Provided MAC Address (FPMA) assigned to the server by the FCF during the FIP negotiation.
FCoE Frames received by servers: Destination MAC: the destination of an FCoE frame is the Fabric Provided MAC Address (FPMA) assigned to the server by the FCF during the FIP negotiation. Source MAC: the source of FCoE frames sent by the FCF to a server is always the MAC address of the Fibre Chanel Forwarder (i.e. the FCF-MAC address of the Nexus switch).
46 Figure 18: Sample of an FCoE frame sent by a server FPMA (FC-MAP+FCID) FCID of the FCoE Server FCID of the Storage Target
47 Support and Other Resources Contacting HP Before you contact HP Be sure to have the following information available before you call contact HP: Technical support registration number (if applicable) Product serial number Product model name and number Product identification number Applicable error message Add-on boards or hardware Third-party hardware or software Operating system type and revision level
HP contact information For help with HP Virtual Connect, see the HP Virtual Connect webpage: http://ww.hp.com/go/virtualconnect For the name of the nearest HP authorized reseller: See the Contact HP worldwide (in English) webpage: http://www.hp.com/country/us/en/wwcontact.html For HP technical support: In the United States, for contact options see the Contact HP United States webpage: http://welcome.hp.com/country/us/en/contact_us.html To contact HP by phone: Call 1-800-HP-INVENT (1-800-474-6836). This service is available 24 hours a day, 7days a week. For continuous quality improvement, calls may be recorded or monitored. If you have purchased a Care Pack (service upgrade), call 1-800-633-3600. For more information about Care Packs, refer to the HP website: http://www.hp.com/hps In other locations, see the Contact HP worldwide (in English) webpage: http://welcome.hp.com/country/us/en/wwcontact.html
Subscription service HP recommends that you register your product at the Subscriber's Choice for Business website: http://www.hp.com/country/us/en/contact_us.html After registering, you will receive email notification of product enhancements, new driver versions, firmware updates, and other product resources.
48 Related documentation HP Virtual Connect Manager 4.01 Release Notes http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03801912/c03801912.pdf
HP Virtual Connect for c-Class BladeSystem Version 4.01 User Guide http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03791917/c03791917.pdf
HP Virtual Connect Version 4.01 CLI User Guide http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03790895/c03790895.pdf
HP Virtual Connect for c-Class BladeSystem Setup and Installation Guide Version 4.01 and later http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03801914/c03801914.pdf
HP Virtual Connect FlexFabric Cookbook http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02616817/c02616817.pdf
HP Virtual Connect Fibre Channel Networking Scenarios Cookbook http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01702940/c01702940.pdf
HP Virtual Connect with iSCSI Cookbook http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02533991/c02533991.pdf
HP Virtual Connect Multi-Enclosure Stacking Reference Guide http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02102153/c02102153.pdf
Converged Networks and Fibre Channel over Ethernet Technical Whitepaper http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03440617/c03440617.pdf
HP welcomes your feedback. To make comments and suggestions about product documentation, send a message to docsfeedback@hp.com. Include the document title and manufacturing part number. All submissions become the property of HP.
Get connected hp.com/go/getconnected Current HP driver, support, and security alerts delivered directly to your desktop
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and servi ces. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Trademark acknowledgments, if needed. 4AA4-xxxxENW, Created Month 2013
Get connected hp.com/go/getconnected Current HP driver, support, and security alerts delivered directly to your desktop
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and servi ces. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Trademark acknowledgments, if needed. c03808925, Created June 2013