Professional Documents
Culture Documents
Implementing Cisco
Storage Networking
Solutions
Volume 3
Version 3.0
Student Guide
Text Part Number: 67-2461-01
DISCLAIMER WARRANTY: THIS CONTENT IS BEING PROVIDED AS IS. CISCO MAKES AND YOU RECEIVE NO WARRANTIES IN
CONNECTION WITH THE CONTENT PROVIDED HEREUNDER, EXPRESS, IMPLIED, STATUTORY OR IN ANY OTHER PROVISION OF
THIS CONTENT OR COMMUNICATION BETWEEN CISCO AND YOU. CISCO SPECIFICALLY DISCLAIMS ALL IMPLIED
WARRANTIES, INCLUDING WARRANTIES OF MERCHANTABILITY, NON-INFRINGEMENT AND FITNESS FOR A PARTICULAR
PURPOSE, OR ARISING FROM A COURSE OF DEALING, USAGE OR TRADE PRACTICE. This learning product may contain early release
content, and while Cisco believes it to be accurate, it falls subject to the disclaimer above.
Table of Contents
Volume 3
Appendix A: The Fibre Channel Protocol
Overview
Module Objectives
FC Protocol Concepts
Overview
Objectives
Fibre Channel Overview
Fibre Channel: The Best of Both Worlds
Advantages of Serial Architecture
Fibre Channel Performance
Fibre Channel Topologies
What is the Point-to-Point Topology?
What is the Arbitrated Loop Topology?
What is the Switched Fabric Topology?
Fibre Channel Ports
Fibre Channel HBAs
Fibre Channel Classes of Service
Summary
FC Layers
Overview
Objectives
Fibre Channel Layers
FC-0: Physical Interface
FC-1: Encoding
FC-2: Framing and Flow Control
FC-3: Common Services
FC-4: Upper-Layer Protocol Interfaces
Fibre Channel Data Constructs
Fibre Channel Frames
Frame Headers
SCSI-FCP Operations
Link Services
Types of Link Services
Basic Link Services
Extended Link Services
Summary
FC Flow Control
Overview
Objectives
Fibre Channel Flow Control
Credit-Based Flow Control
Types of Flow Control
AA-1
AA-1
AA-1
AA-3
AA-3
AA-3
AA-4
AA-7
AA-10
AA-11
AA-13
AA-15
AA-18
AA-19
AA-20
AA-21
AA-21
AA-21
AA-22
AA-23
AA-24
AA-25
AA-29
AA-30
AA-31
AA-32
AA-34
AA-36
AA-37
AA-39
AA-41
AA-41
AA-41
AA-42
AA-44
AA-46
AA-49
AA-50
AA-51
AA-53
AA-54
AA-56
AA-58
AA-60
AA-61
AA-62
AA-63
AA-64
AA-65
AA-65
AA-65
AA-66
AA-67
AA-68
FC Login
Overview
Objectives
Fabric Login
Port Login
Port and Address Discovery
Process Login
Loop Initialization and Arbitration
The Loop Initialization Protocol
The Loop Arbitration Protocol
The Loop Port State Machine
Summary
FC Error Recovery
Overview
Objectives
FC-1 Errors
R_T_TOV
FC-2 Errors
E_D_TOV
Sequence Recovery
R_A_TOV
SCSI-FCP Error Recovery
Summary
FC Switched Fabric
Overview
Objectives
Fabric Configuration Overview
FSPF
FSPF Protocol Operations
Stage 1The Hello Protocol
Stage 2Initial Database Synchronization
Stage 3Database Maintenance
Stage 4Path Discovery
Stage 5Path Computation
Limitations of FSPF
The RSCN Process
Fabric State Changes
The RSCN Process
Standard Fabric Services
The Domain Manager
The Name Server
Name Server Operations
The Management Server
Well-Known Addresses
Summary
ii
AA-69
AA-71
AA-72
AA-73
AA-74
AA-75
AA-76
AA-77
AA-79
AA-79
AA-81
AA-82
AA-84
AA-85
AA-85
AA-85
AA-86
AA-94
AA-97
AA-98
AA-103
AA-103
AA-103
AA-105
AA-106
AA-107
AA-107
AA-107
AA-108
AA-109
AA-111
AA-113
AA-114
AA-117
AA-118
AA-122
AA-125
AA-125
AA-125
AA-126
AA-128
AA-130
AA-131
AA-133
AA-133
AA-135
AA-135
AA-137
AA-139
AA-139
AA-139
AA-143
AA-144
AA-145
AA-146
AA-147
AA-148
AA-149
2007 Cisco Systems, Inc.
AB-1
AB-1
AB-1
AB-3
AB-3
AB-4
AB-8
AB-12
AB-26
AB-30
AB-40
iii
iv
Appendix A
Module Objectives
Upon completing this module, you will be able to describe the SCSI and Fibre Channel
protocols. This includes being able to meet these objectives:
Describe the Fibre Channel layered model, data constructs, SCSI-FCP read and write
operations, and Link Services
AA-2
Lesson 1
Objectives
Upon completing this lesson, you will be able to describe the basic characteristics of the SCSI
protocol. This includes being able to meet these objectives:
Target
Requests
Application
Client
LUNs
Responses
Device
Server
Tasks
Delivery Subsystem
Parallel or FCP or IP
2007 Cisco Systems, Inc. All rights reserved.
ICSNS v3.07-4
The Small Computer System Interface (SCSI) performs the heavy lifting of passing commands,
status, and block data between platforms and storage devices. One function of operating
systems is to hide the complexity of the computing environment from the end user.
Management of system resources including , memory, peripheral devices, display, context
switching between concurrent applications, and son on, are generally concealed behind the user
interface. The internal operations of the OS must be robust, closely monitor changes of state,
ensure that transactions are completed within the allowable time frames, and automatically
initiate recovery or retires in the event of incomplete or failed procedures. For I/O operations
for peripheral devices such as disk, tape, optical storage, printers, and scanners, these functions
are provided by the SCSI protocol, typically embedded in a device driver or logic onboard a
host adapter.
Because the SCSI protocol layer sits between the operating system and the peripheral
resources, it has different functional components. Applications typically access data as files or
records. Although these may be ultimately stored on disk or tape media in the form of data
blocks, retrieval of the file requires a hierarchy of functions to assemble raw data blocks into a
coherent file that can be manipulated by an application.
SCSI architecture defines the relationship between initiators (hosts) and targets (for example,
disks) as a client/server exchange. The SCSI-3 application client resides in the host and
represents the upper layer application, file system, and operating system I/O requests. The
SCSI-3 device server sits in the target device, responding to requests.
AA-4
Files or Records
Files or Records
Operating System
Files or Records
Volume Management
Logical Drives
Block Transition
SCSI Mapping
Device Driver
Physical Layer
Interconnect
ICSNS v3.07-5
When a user application opens a file, a series of processes are launched that rely on lower SCSI
commands and controls to transport the appropriate data blocks from storage safely into
memory. A translation between file representation and block I/O thus occurs in the file system
layer.
Just as the file system presents an abstraction of data to the user application, the physical
storage devices are presented as an abstraction to the file system. An E: drive in Windows or a
/dev/dsk2 in UNIX may be a single disk, a partition on a larger disk, or a striped array of
multiple disks. The file system depends on a volume management function to present
sometimes diverse storage devices as coherent and easily addressable resources. Device
virtualization turns physical storage into logical storage, and assumes the intricate tasks
necessary for placement of data blocks on disks. This file/block translation and mapping
function can be as sophisticated as a separate volume management application or as
straightforward as an adaptor card device driver interface to an operating systems disk utility.
This hierarchy of logical abstractions descends to the physical world of actual SCSI devices
and their connectivity to the host system. Common access methods at the OS level allow
uniform treatment of SCSI devices regardless of their physical attachment. In saving a file, the
file system does not need to be concerned with whether the logical drive identifier fronts a
direct SCSI-attached unit, a Fibre Channel array or an IP storage device somewhere on the
Gigabit Ethernet network.
Regardless of the underlying plumbing, the operating systems view of the physical storage is
defined by the bus/target/LUN triad inherited from parallel SCSI technology. The mapping
between the bus/target/LUN designation and the logical drive identifier provides the portal
between physical devices and the upper layer file system. Because Fibre Channel and IP
storage are serial transports and have no bus component, the bus identifier is fabricated for
compatibility with the operating systems SCSI nomenclature. Two IP storage NICs in a single
server, for example, may have different bus designations to mimic SCSI adapter configuration.
AA-5
AA-6
Application
Layer
CAM
SBC
SSC
SES
SMC
SBP
SSP
FCP
IEEE
1394
SSA
FC-PH
Shared
Command Set
SPC
Transport
Protocol
SPI
Physical
Interconnect
ICSNS v3.07-7
Serial Storage Architecture (SSA) is a storage bus aimed primarily at the server
market.
SCSI-3 Parallel Interface (SPI) is the specification used for parallel SCSI buses.
2. The transport protocol layer defines the protocols used for session management:
Serial Storage Protocol (SSP) is the transport protocol used by SSA devices.
Serial Bus Protocol (SBP) is the transport protocol used by IEEE1394 devices.
3. The shared command set layer consists of command sets for accessing storage resources:
SCSI Block Commands (SBC) are used with block-oriented devices, such as disks.
SCSI Stream Commands (SBC) are used with stream-oriented devices, such as
tapes.
SCSI Media Changer Commands (SMC) are used to implement media changers,
such as robotic tape libraries and CD-ROM carousels.
AA-7
SCSI Enclosure Services (SES) defines commands used to monitor and manage
SCSI device enclosures, such as RAID arrays.
4. The SCSI Common Access Method (CAM) defines the SCSI device driver application
programming interface (API).
AA-8
SCSI
SCSIParallel
Parallel
Port
PortDriver
Driver
SCSI-3
SCSI-3FCP
FCP
Port
PortDriver
Driver
iSCSI
iSCSIIP
IP
Port
PortDriver
Driver
SAS
SAS
Port
PortDriver
Driver
SCSI
SCSISerial
SerialBus
Bus
Protocol
Protocol(SBP-2)
(SBP-2)
Port
PortDriver
Driver
SCSI
SCSIParallel
Parallel
Port
Port
Fibre
FibreChannel
Channel
Port
Port
Ethernet
Ethernet
Port
Port
SAS
SASSerial
Serial
Port
Port
IEEE-1394
IEEE-1394
(Firewire)
(Firewire)
Port
Port
Serial Attached
SCSI Interface
FC
SCSI Card
FC Card
Firewire
Interface
NIC
*SCSI-3: Separation of physical interface, transport protocols, and SCSI Command Set
2007 Cisco Systems, Inc. All rights reserved.
ICSNS v3.07-8
The SCSI_3 family of standards introduced several new variations of SCSI commands and a
protocol, including serial SCSI-3 and special command sets for streaming and media handling
required for tape. As shown in the diagram, the command layer is independent of the protocol
layer, which is required to carry SCSI-3 commands between devices. This enables more
flexibility in substituting different transports beneath the SCSI-3 command interface to the
operating system.
AA-9
Half-duplex
Parallel
Shared bus
ICSNS v3.07-10
The bus/target/LUN triad is defined from parallel SCSI technology. The bus represents one of
several potential SCSI interfaces installed in the host, each supporting a separate string of disks.
The target represents a single disk controller on the string. And the LUN designation allows for
additional disks governed by a controller for example, a RAID device.
The following are characteristics of parallel SCSI technology:
AA-10
SCSI uses a parallel architecture in which data is sent simultaneously over multiple wires.
On a SCSI bus, a device must assume exclusive control over the bus in order to
communicate. (SCSI is sometimes referred to as a simplex channel because only one
device can transmit at a time).
SCSI
Initiator
(I/O
Adapter)
Terminator
Data/Address Bus
Control Signals
Interface
Interface
ID
ID
ID
ID
ID
ID
Interface
ID
ID
Priority
2007 Cisco Systems, Inc. All rights reserved.
ICSNS v3.07-11
Data bits are sent in parallel on separate wires. Control signals are sent on a separate set of
wires.
Only one device at a time can transmita transmitting device has exclusive use of the bus.
A special circuit called a terminator must be installed at the end of the cable. The cable
must be terminated to prevent unwanted electrical effects from corrupting the signal.
Parallel transmission of data bits allows more data to be sent in a given time period but
complicates transmitter-receiver synchronization. The fact that control signals, such as
clock signals, are sent on a separate set of wires also makes synchronization more difficult.
It is an inefficient way to use the available bandwidth, because only one communication
session can exist at a time.
Termination circuits are built into most SCSI devices, but the administrator often has to set
a jumper on the device to enable termination.
Incorrect cable termination can cause either a severe failure or intermittent, difficult-totrace errors.
AA-11
Terminator
Data/Address Bus
SCSI
Initiator
(I/O
Adapter)
Control Signals
Interface
LUN 0
LUN 1
LUN 2
LUN 3
Interface
Interface
LUN 0
LUN 1
LUN 0
LUN 1
LUN 2
LUN 3
ICSNS v3.07-12
SCSI was designed to support a few devices at most, so its device addressing scheme is fairly
simpleand not very flexible. SCSI devices use hard addressing:
AA-12
Each device has a series of jumpers that determine the devices physical address, or SCSI
ID. The ID is software-configurable on some devices.
Each device must have a unique ID. Before adding a device to the cable, the administrator
must know the ID of every other device connected to the cable and choose a unique ID for
this new device.
The ID of each device determines its priority on the bus. For example, the SCSI target with
ID 7 always has a higher priority than the SCSI initiator with ID 6. Because each device
must have exclusive use of the bus while it is transmitting, ID 6 must wait until ID 7 has
finished transmitting. Fixed priority makes it more difficult for administrators to control
performance and quality-of-service.
SCSI Operation
This topic provides an overview of SCSI protocol operations.
SCSI Operation
SCSI includes three phases of operation:
Command send the required command and parameters via a
Command Descriptor Block (CDB)
Data Transfer data in accordance with the command
Status Receive confirmation of command execution
Initiator
Target
FC
FC
HBA
ICSNS v3.07-14
Every communication on the SCSI bus is formed by sequences of events called bus phases.
Each phase has a purpose and is linked to other phases to execute SCSI commands and transfer
data and messages back and forth.
The majority of the SCSI protocol is controlled by the target. The initiator only initiates a SCSI
task by selecting a target.
Once the target is selected, it (the target) controls the bus.
It does this by picking up the command from the initiator, executing it and delivering a status
back to the initiator.
AA-13
FC
HBA
FC
Connectio
n , Arbitra
te and sele
ct
SCSI Command:
SCSI Read
CMD (28h)
Command
Phase
e
Data In Phas
DATA
Status Phas
RSP
SCSI Message:
Send Command Complete message
Release the Bus
Disconnect
ICSNS v3.07-15
AA-14
Byte
0
1
2
3
4
5
6
7
8
9
Group Code
Command Code
Service Action
Reserved
Logical Block Address
MSB
Logical Block Address
Logical Block Address
Logical Block Address
Reserved
MSB
Transfer Length
Transfer Length
Control
LSB
Number of SCSI Blocks to be
transferred
LSB
Last Byte Control Byte
ICSNS v3.07-17
AA-15
N bytes of parameters
Control byte
The number of bytes of parameters (N) can be determined from the Operation Code byte
which is located in byte 0 of the Command Descriptor Block (CDB).
The Control Byte, which is located in the last byte of a Command Descriptor Block, contains
control bits that define the behavior of the command.
AA-16
Command Name
00h
03H
Request Sense
12h
Inquiry
15h
Mode Select
18h
Copy
1Ah
Mode Sense
1Ch
1Dh
Send Diagnostic
39h
Compare
3Ah
3Bh
Write Buffer
3Ch
Read Buffer
4Ch
Log Select
4Dh
Log Sense
55h
Mode Select
5Ah
Mode Sense
Hex
Status
00
Good
02
Check Condition
04
Condition Met
08
Busy
10
Intermediate
14
Intermediate-Condition Met
18
Reservation Conflict
28
30
ACA Active
..
ICSNS v3.07-18
SCSI defines commands for all devices as well as commands for specific devices. For example:
The OpCode for the Disk Command Write(6) is 0Ah, Write(10) is 2Ah.
The OpCode for the Disk Command Read(6) is 08h, Read(10) is 28h.
The numbers in the parenthesis (6) and (10) refer to the type of CDB utilized.
AA-17
SCSI Messages
This topic introduces the functions of SCSI messages.
SCSI Messages
SCSI messages are an additional way in which the
initiator and the target communicate with each other.
Some SCSI transmission parameters are not tied to a specific
command, but to the relationship between a specific initiator and
target.
Transfer speed
Data width
Other asynchronous events such as:
To abort a SCSI command that is currently executed by a
target
RESTORE POINTERS
ICSNS v3.07-20
SCSI messages are an additional way in which the initiator and the target communicate with
each other.
Some SCSI transmission parameters are not tied to a specific command, but to the relationship
between a specific initiator and target:
Transfer speed
Data width
AA-18
RESTORE POINTERS
ICSNS v3.07-21
Error Handling
Parallel SCSI is not as efficient in detecting transmission errors as, LAN protocols or SAN
protocols.
SCSI uses a parity bit. The receiving device calculates the parity and compares it with the
parity bit. If they dont match, a parity error has occurred. Consequently, the device that
detected the parity error sends a RESTORE POINTERS message that causes the data transfer
counter to be reset to the value at the last disconnect so that the transfer of data is repeated from
that point on.
AA-19
Summary
This topic summarizes the key points that were discussed in this lesson.
Summary
The SCSI protocol was originally based on parallel technology
and modeled after a bus topology.
The SCSI Architecture Model is a reference for the SCSI
functional layers and the SCSI Transport Interfaces.
To communicate, the SCSI protocol operates in phases.
The SCSI protocol has a set of command codes and status
codes.
SCSI messages are used for error handling.
AA-20
ICSNS v3.07-22
Lesson 2
FC Protocol Concepts
Overview
Fibre Channel (FC) has characteristics of both I/O channels and data networks, and this unique
blend of features is what makes FC ideal for storage area networks (SANs). This lesson takes a
close look at the features and capabilities of FC, and compares these features and capabilities
with those of traditional I/O channels such as SCSI, and data networks such as Ethernet and
ATM.
Objectives
Upon completing this lesson, you will be able to explain the role of Fibre Channel in a storage
environment. This includes being able to meet these objectives:
FC
HBA
FC
FC
HBA
IP
Network
Fibre Channel
Fabric
FC
FC
HBA
ICSNS v3.07-4
AA-22
FC
HBA
I/O Channel
Fibre Channel
Network
x Few devices
x Static
9Low latency
x Short distances
9Hardware-based
9Many devices
9Dynamic
9Low latency
9Long distances
9Hardware-based
9Many devices
9Dynamic
x High latency
9Long distances
x Software-based
delivery management
delivery management
delivery management
ICSNS v3.07-5
FCs serial architecture allows it to support many devices and flexible configurations.
Storage networks are generally somewhat simpler and more stable than data networks
(such as the Internet). Therefore, FC compromises between flexibility and efficiency in
order to minimize end-to-end latency.
AA-23
ICSNS v3.07-6
A serial architecture reduces the cost and complexity of cabling. Unlike SCSI, FC does not
require terminators, and it uses a network architecture (hubs and switches) rather than a
multidrop (single cable) architecture.
Serial networks can use either copper cables or optical fiber. This allows customers to
choose cheaper copper cables where distance is not a requirement, and to choose more
expensive optical cable when longer distances must be supported.
Because clock synchronization and data transmission are performed in one signal, rather
than on separate wires, synchronization can be more easily maintained at higher link rates
and longer distances.
Overall, a serial architecture simplifies product design, allowing faster evolution. For
example, the wire speed of SCSI doubles with each release, while the wire speed of
Ethernet and FC increases by a factor of 10 with each release.
One of the most significant advantages of serial networks is that serial networks can support
longer link distances. A single-mode fiber optic FC or Gigabit Ethernet link can support links
over 400 times as long as on the longest SCSI bus.
AA-24
Mode:
Maximum
# of Nodes:
Link Distance:
Up to 30 m/link copper
Up to 10 km/link optical
Reliability:
ICSNS v3.07-8
Bandwidth: 100, 200, 400, and 1000 MBps (sustained, each direction)
Maximum number of nodes: 126 for arbitrated loop, >16 million for switched fabric
Note that 100MBps, 200MBps, 400MBps, and 1000MBps are the half-duplex rates for Fibre
Channel, but Fibre Channel is actually a full-duplex technology. In other words, Fibre Channel
supports up to 1000MBps between two ports in both directions simultaneously.
AA-25
ICSNS v3.07-9
The Bit Error Ratio (BER) is calculated by dividing the number of erroneous bits by the total
number of bits transmitted, received, or processed over some stipulated period. For example,
2.5 erroneous bits out of 100,000 bits transmitted would be 2.5 divided by 100,000 or 2.5 105.
The minimum and maximum values of average received power range determine the input
power range required to maintain a BER less than 10-12. This value takes into account worst
case signal characteristics.
A BER of 10-12 corresponds to one error every 8 minutes at 2Gbps. This might seem like a
very low error rate, but due to some stringent applications, the industry is working on achieving
a BER of 10-15, which results in one error every 5.5 days at 2Gbps.
AA-26
Fibre Channel
Bandwidth:
Mode:
Half duplex,
parallel, shared bus
Full duplex,
serial, packet-based
16
1.525m
Up to 30m copper
Up to 10Km optical
Monolithic (SCSI)
Maximum
# of Nodes:
Link Distance:
Protocol Model:
2007 Cisco Systems, Inc. All rights reserved.
ICSNS v3.07-10
The table compares the characteristics of FC to those of SCSI. Significant differences between
FC and SCSI include:
Bandwidth: FC is capable of delivering the published data rates in a sustained manner. The
maximum SCSI bit rate is the peak rate, and cannot be sustained for long periods of time.
Mode: SCSI uses a parallel bus, with half duplex capability (transmission in one direction
at a time), while the FC serial connection has full duplex capability.
The SCSI cable length limitations results in a maximum link distance of 25 meters, while
FC, using optical cable, has a maximum link distance of 10 kilometers.
Note that the storage market typically measures data rates in megabytes-per-second (MBps),
whereas the network market typically measures data rates in megabits-per-second (Mbps) or
gigabits-per-second (Gbps). The Fibre Channel market measures data rates in both MBps and
Gbps, so you must be able to quickly translate between both units of measure. In Fibre
Channel, 100MBps equals 1Gbps. Note that this conversion assumes that each byte equals 10
bits. This is actually trueFibre Channel uses a bit encoding scheme in which each 8-bit byte
is encoded as 10 bits for transmission.
AA-27
Bandwidth:
Mode:
Average
Continuous
Data Flow:
Link Distance:
Protocol Model:
Gigabit Ethernet
Fibre Channel
Full duplex,
serial, packet-based
Full duplex,
serial, packet-based
~ 40%
~ 95%
100m copper
5Km optical
30m copper
10Km optical
Layered
Layered
ICSNS v3.07-11
AA-28
FC
FC
HBA
FC
Point-to-Point
HBA
FC
FC
FC
HBA
HBA
FC
FC
HBA
HBA
FC
HBA
HBA
FC
HBA
FC
HBA
FC
FC
FC
HBA
FC
HBA
Arbitrated Loop
Switched Fabric
ICSNS v3.07-13
Point-to-point
Arbitrated loop
Switched fabric
AA-29
FC
HBA
ICSNS v3.07-14
AA-30
Scalability:
FC
HBA
FC
HBA
Hub
Reliability:
If one device fails, the entire loop
can fail
FC
HBA
FC
HBA
FC
FC
HBA
ICSNS v3.07-15
I/O speeds depend upon the location of the initiator and target in the loop.
Loops suffer from poor performance. Because there is only one data path, only one pair of
devices can communicate at a time. This means that all the devices on the loop must share
the available bandwidth.
Loops have a higher latency than fabrics. Devices must negotiate for control of the loop.
Loops are not very scalable. Because loops were designed to support a small number of
devices, the FC-AL protocol provides only 127 unique addresses: 126 addresses are usable
for nodes (hosts and storage devices), and 1 address is reserved for attaching the loop to an
FC switched fabric.
In reality, only about a dozen devices can be connected in a loop before performance drops
below acceptable levels.
AA-31
FC
HBA
FC
FC
FC
HBA
FC
FC
HBA
FC
HBA
FC
HBA
FC
FC
FC
HBA
FC
FC
HBA
FC
FC
FC
HBA
FC
HBA
ICSNS v3.07-16
AA-32
ICSNS v3.07-17
The Fibre Channel Switched Fabric (FC-SW) protocol differs from the arbitrated loop topology
in several important ways:
The FC-SW device addressing scheme allows over 16,000,000 ports. Existing
implementations can support hundreds and even thousands of nodes using large directorclass switches.
The FC-SW protocol defines several management services that increase the scalability,
manageability, and security of the SAN.
Due to the limitations of the Arbitrated Loop topology, the majority of modern organizations
choose to implement a Switched Fabric topology because it offers greater scalability,
performance, reliability, and manageability.
AA-33
FC
FC
FC
HBA
Server
I/O Adapter
Switch
Array
controller
Tape device
Storage
ICSNS v3.07-19
In data networking terminology, ports are often thought of as just physical interfaces where you
plug in the cable. In FC, however, ports are intelligent interfaces, responsible for actively
performing critical network functions.
The preceding graphic contains several ports. There are ports in the host I/O adapter (host bus
adapter [HBA]), ports in the switch, and ports in the storage devices.
FC terminology differentiates between several different types of ports, each of which performs
a specific role on the SAN. You will encounter these terms often as you continue to learn about
FC, so it is important that you learn to recognize the different port types. In addition to the
common ports defined for FC, Cisco has developed some proprietary port types.
AA-34
FC
Hub
FC
HBA
Host
FL_Port
Storage
Array
Standard Ports
N_Port
F_Port
E_Ports
FC
FC
HBA
Host
B_Port
Storage
Array
E_Port
WAN Bridge
ICSNS v3.07-20
Logically, an FL_Port is considered part of both the fabric and the loop.
Note that FC hubs, although they obviously have physical interfaces, do not contain FC ports.
Hubs are basically just passive signal splitters and amplifiers. They do not actively participate
in the operation of the network. On an arbitrated loop, the node ports manage all FC operations.
Not all switches support FL_Port operation. For example, some McDATA switches do not
support FL_Port operation.
An NL_Port (Node Loop Port) is a port on a node that connects to another port in an arbitrated
loop topology. There are two types of NL_Ports:
Public NL_Ports can communicate with other loop ports and with N_Ports on an attached
fabric.
Note that the term L_Port (Loop Port) is sometimes used to refer to any port on an arbitrated
loop topology. L_Port can mean either FL_Port or NL_Port. In reality, there is no such
thing as an L_Port.
AA-35
Ethernet NIC
Flow Control
Sequencing
Segmentation
Error Correction
OS
OS
I/O Subsystem
I/O Subsystem
TCP Driver
FC Driver
HBA
Fibre Channel
HBA
Flow Control
Sequencing
Segmentation
Error Correction
ICSNS v3.07-22
HBAs are I/O adapters that are designed to maximize performance by performing protocol
processing functions in silicon. HBAs are roughly analogous to network interface cards, but
HBAs are optimized for storage networks, and provide features that are specific to storage.
The figure contrasts HBAs with NICs, illustrating that HBAs offload protocol processing
functions into silicon.
With NICs, protocol processing functions such as flow control, sequencing, segmentation and
reassembly, and error correction are performed by software drivers. HBAs offload these
protocol processing functions onto the HBA hardware itselfusually some combination of an
application-specific integrated circuit (ASIC) and firmware. Offloading these functions is
necessary to provide the performance required by storage networks.
AA-36
NICs can utilize over 80 percent of a servers CPU capacity (measured with a 1Ghz Intel
Pentium CPU) to deliver 50-80MBps on a Gigabit Ethernet link. I/O processing adds
considerable real cost to what may appear to be an inexpensive NIC.
HBAs manage I/O transactions with little or no involvement of the server CPU. FC HBAs
can provide throughput at nearly 95 percent of link speed with less than 10 percent server
CPU utilization.
Use
Class 1
Connection-oriented
Confirmed delivery
Class 2
Packet-switched
Confirmed delivery
Class 3
Packet-switched
No delivery confirmation
Class 4
Class 6
Connection-oriented multicast
Confirmed delivery
Class F
Packet-switched
Confirmed delivery
ICSNS v3.07-24
Class 3 is by far the most commonly used Class of Service on fabrics, and it is often the
only class supported on arbitrated loops. All FC SAN products support Class 3.
Note that Class 5 is not yet defined. Class 5 was intended to enable isochronous transactions by
multiple ports, but has not been completed. An isochronous connection is one in which
bandwidth and data delivery rate are guaranteed. Class 5 would be appropriate for video
delivery services.
AA-37
Connection Oriented
Yes
No
No
Yes
Yes
Packet-switched
No
Yes
Yes
No
No
Bandwidth Reserved
100%
No
No
Fractional
100%
Guaranteed Latency
Yes
No
No
Yes(QoS)
Yes
Yes
No
No
Yes
Yes
Delivery Confirmation
Yes
Yes
No
Yes
Yes
ICSNS v3.07-25
The preceding table summarizes the features of the Classes of Service. Although Classes 2 and
3 are the only options currently available in Fibre Channel products today, customers might
have specialized applications that call for the features of other classesand might be willing to
investigate specialized products that support those applications.
AA-38
Summary
This topic summarizes the key points that were discussed in this lesson.
Summary
Fibre Channel supports many devices, dynamic network
reconfiguration, low latency, long distances, and hardware-based
delivery management.
Fibre Channel currently supports 100, 200, 400 and 1000 MBps.
The Fibre Channel Protocol supports three topologies: Point to
Point, Arbitrated Loop and Switched Fabric.
Ports are intelligent interface points on the Fibre Channel
network.
Fibre Channel HBAs offload flow control, sequencing,
segmentation, and error correction into the HBA hardware,
increasing performance.
Fibre Channel has defined classes of service similar to the Class
of Service models in LAN networks, however the Fibre Channel
implementation is different.
ICSNS v3.07-26
AA-39
AA-40
Lesson 3
FC Layers
Overview
Like nearly all modern networks, Fibre Channel (FC) is designed with a modular, layered
architecture. This architecture is designed to carry other protocols, as well as new native
protocols. A layered architecture provides benefits for both vendors and users because it
enhances the clarity and flexibility of the architecture. This lesson describes the five layers of
the FC layered model, and the upper layer protocols that FC supports.
Objectives
Upon completing this lesson, you will be able to describe the Fibre Channel layered model,
data constructs, SCSI-FCP read and write operations, and Link Services. This includes being
able to meet these objectives:
Fibre Channel
NTFS CIFS
NFS DAFS ...
SCSI IP VI
HiPPI ...
Logical
Physical
Application
Presentation
Session
Transport
Network
Fibre Channel
HTTP FTP
SNMP ...
Data link
Physical
IP IPX ...
Ethernet
ICSNS v3.07-4
The OSI model defines seven layers of functionality for network protocols. While FC does not
map directly to the OSI model, it does use a layered model. FCs lower layers relate closely to
the lower layers of OSI:
FC defines the lower three layers (approximately) of the OSI model: Physical, Data link,
and Network
Other protocols, such as SCSI, are responsible for the upper layers
If you are familiar with data networking, you probably understand the difference between
physical-layer protocols, such as Ethernet, and logical-layer protocols, such as TCP and IP:
Protocols like TCP and IP define aspects of the network such as flow control and
addressing.
The preceding graphic shows that FC defines both the physical layer and part of the logical
layer, and then interfaces with ULPs that perform the functions of the upper layers of the OSI
model.
AA-42
Upper-layer
protocols
FC-4
FC-3 Common Services
FC-2 Framing and flow control
FC-FS
NTFS CIFS
NFS DAFS ...
FC-1 Encoding
FC-0 Physical interface
FC-PI
SCSI IP VI
HiPPI ...
Fibre Channel
ICSNS v3.07-5
The lower three layers (FC-0, FC-1, and FC-2) are collectively known as the FC Physical Layer
(FC-PH), even though they also implement logical functions such as framing and flow control.
The FC-3 layer provides a framework for implementing new SAN-wide services, while the FC4 layer interfaces with the ULPs and maps them to the FC.
The FC-PH specification was the original document that defined layers FC-0, FC-1, and FC-2.
The final version of the FC-PH specification was FC-PH-3. However, FC-PH was then
superceded by two additional documents:
Fibre Channel Framing and Signaling (FC-FS) defines FC-1 and FC-2
AA-43
FC-FS
FC-PI
ICSNS v3.07-6
AA-44
Specifying signaling protocols for transmitting and receiving a signal at different transfer
rates
33m
Coaxial
59m
30m100m
Multimode 62.5
Multimode 50
Single-mode 9
1Gb
2Gb
4Gb
Electrical Cables
300m
150m
70m
Optical Cables
500m
300m
150m
1Gb
2Gb
4Gb
10km
10km
10km
1Gb
2Gb
4Gb
30km
ICSNS v3.07-7
The current specification states a minimum 2m distance for optical fiber. This is to allow for a
build-up of photons that occur in the first 2m of cable after the laser fires, and, in multimode
cables, to eliminate problems associated with some modes of light which, due to their steep
angle of reflection, do not travel very far down the cable.
AA-45
FC-1: Encoding
FC-1 defines the bit encoding scheme:
Encoding and decoding of serial signals
Bit-level error detection
Clock synchronization
Link initialization and
recovery
ULPs
FC-4
FC-3 Common Services
FC-2 Framing and flow control
FC-1 Encoding
FC-0 Physical interface
FC-FS
FC-PI
ICSNS v3.07-8
FC-1: Encoding
The FC-1 layer specifies how data is encoded at the bit and byte levels for transmission across
the link. FC-1 is responsible for:
AA-46
Taking data from the transmitters I/O bus and encoding into a serial signal for
transmission
Taking a serialized signal and decoding it into a signal that can be sent to the receivers I/O
bus
8b/10b
Encoder
Tx Byte
Parallel/Serial
Converter
(SERDES)
Parallel Input
Transmitter
Serial Output
Media Output
ICSNS v3.07-9
The data from the transmitting I/O bus is encoded using the 8b/10b encoding scheme.
The receive end decodes the serial data and forwards it to the receivers I/O bus.
AA-47
0xDF
Bits
11011111
6
Encoded
Disparity
1010110110
0101000110
31 = D31.6
ICSNS v3.07-10
The 8b/10b scheme defines multiple transmission characters for each 8-bit data byte. Because
the encoder can choose between multiple 10-bit representations for each 8-bit byte, it can
balance the number of ones and zeros in the data stream. The imbalance between the number of
1s and 0sknown as the running disparityis continually reevaluated. To balance the number
of ones and zeros, every transmitted byte is encoded into one of two possible 10-bit
representations depending on the current running disparity.
In the FC-1 specification, every 10-bit character is represented using a special notation:
AA-48
Dxx.y: Used for data characters that map to 8-bit characters; xx is the decimal value of the
lowest 5 bits and y is the decimal value of the highest bits.
Kxx.y: Used for special control characters; xx and y are defined as for data characters.
ULPs
FC-4
FC-3 Common Services
FC-2 Framing and flow control
FC-1 Encoding
FC-0 Physical interface
FC-FS
FC-PI
ICSNS v3.07-11
To transport a stream of data from one port to another, the data must be packaged into
discrete packets, or frames. At the receiving end, the data must then be extracted from the
frames.
The flow of frames across the network must be also be controlled so that the sending port
does not send data faster than the receiving port can receive it. The FC-2 layer performs
this flow control function by initiating or preventing transfer of frames.
AA-49
Future services?
ULPs
Compression
FC-4
Encryption
Link multiplexing
FC-FS
FC-PI
ICSNS v3.07-12
AA-50
Generic services are functions that can span multiple ports and can be applied to multiple
upper-layer protocols.
FC defines generic fabric services that include a name server, a secure key distribution
server, a management server, and a time server.
Generic services use the Fibre Channel Common Transport (FC-CT) protocol to
communicate and distribute functions between switches.
FC-FS
FC-PI
ICSNS v3.07-13
The FC-4 layer receives the data from the ULP drivers and passes the data down to the FC3 layer.
The FC-3 layer performs any required manipulation on the data, such as compressing or
encrypting the data, or it simply passes the data directly to the FC-2 layer.
The FC-2 layer then packages the data into frames and passes it down to the FC-1 layer.
The FC-1 layer encodes each byte into 10-bit characters using the 8b/10b scheme and
passes the data down to the FC-0 layer.
The FC-0 layer encodes the data into physical signals and transmits those signals across the
link.
Only the FC-4 layer needs to know which ULP is used. The protocol-independence of the FC-3
through FC-0 layers allows FC to be easily adapted to new ULPs.
AA-51
FICON,
ESCON
FCPSCSI
IEEE
802.2 LLC
FC-SB-2
FC-LE
HiPPI
IP
FC-IP
FC-FP
ATM,
SONET
FC-BB-2
ICSNS v3.07-14
The Small Computer System Interface (SCSI) command set is widely used among storage
devices. Even though SCSI bus technology is not suitable for SANs, the SCSI command set is
well-suited for many types of storage applications.
The use of the SCSI command set enables the use of inexpensive SCSI disks and SCSI tape
drives in FC SAN storage devices. SCSI-FCP also enables compatibility with existing
operating systems and legacy storage applications. In fact, most operating systems and
applications are not aware of the FC SANFC devices appear to the host and its applications
as SCSI devices.
The mapping of the SCSI protocol to FC is called SCSI-Fibre Channel Protocol (SCSI-FCP), or
sometimes simply FCP. SCSI-FCP is the ULP command set used on most FC SANs. SCSIFCP provides the command set for reading and writing data to and from storage devices.
The fact that FC supports a wide range of protocols allows FC to meet the needs of diverse
applications and integrate with heterogeneous platforms. FC supports the following existing
ULP protocols:
AA-52
The IEEE 802.2 standard defines the generic logical link control (LLC) layer in the OSI
Reference Model. The FC-LE standard helps map IEEE 802.2-based protocols to FC.
IP is the protocol that drives the Internet. FC-IP allows FC to carry the IP protocol. Servers
can use IP to communicate with each other over the SAN.
High Performance Parallel Interface (HiPPI) connects devices at short distances and high
speeds. HiPPI is used primarily to connect supercomputers and to provide high-speed
backbones for LANs. The FC-FP standard maps HiPPI to FC.
The FC-BB-2 standard enables FC to exchange data with ATM and Synchronous Optical
Network (SONET) networks for long-haul transport of FC data.
FC
FC
HBA
Initiator
Target
ICSNS v3.07-16
The preceding graphic shows a transaction between a host (initiator) and a storage device
(target):
The smallest unit of data is a word. Words consist of 32 bits (4 bytes) of data that are
encoded into a 40-bit form by the 8b/10b encoding process.
A sequence is a series of frames sent from one node to another node. Sequences are
unidirectionalin other words, a sequence is a set of frames that are issued by one node.
An exchange is a series of sequences sent between tow nodes. The exchange is the
mechanism used by two ports to identify and manage a discrete transaction. The exchange
defines an entire transaction, such as a SCSI read or write request. An exchange is opened
whenever a transaction is started between two ports and is closed when the transaction
ends. An FC exchange is equivalent to a TCP session.
AA-53
1
4
S
O
F
6
24
Header
Optional
Headers
Words 016
Bytes 064
0528
02112
Payload
Data or commands
0512
02048
1
4
CRC
1 = 537
4 = 2148
E
O
F
Fill
Bytes
= 528
0-3 = 2112
ICSNS v3.07-17
A 24-byte header
A 4-byte (32-bit) CRC that is used to detect bit-level errors in the payload
AA-54
The first 64 bytes of the payload can be used to incorporate optional headers. This would
reduce the data payload size to 2048 bytes (2KB).
The payload ends with 1-3 fill bytes. This is necessary because the smallest unit of data
recognized by FC is a 4-byte word. However, the ULP is not aware of this FC requirement,
and the data payload for a frame might not end on a word boundary. FC therefore adds up
to 3 fill bytes to the end of the payloadas many as are needed to ensure that the payload
ends on a word boundary.
Frames (F)
ICSNS v3.07-18
The screen image displays an FC protocol trace. A single FC frameFabric Login (FLOGI)
is displayed in the right-hand window. Each word in the frame is depicted on a separate line,
beginning with the SOF Frame Delimiter (SOFi3) and ending with the EOF Frame Delimiter
(EOFt). The display shows the 6 words in the frame header, 29 words in the payload, and the
32-bit CRC.
AA-55
Frame Headers
S
O
F
Header
Bit
Word 31
0
1
2
3
4
5
24 23
R_CTL
CS_CTL
TYPE
SEQ_ID
Payload
16 15
CRC
8 7
E
O
F
D_ID
S_ID
F_CTL
DF_CTL
SEQ_CNT
OX_ID
RX_ID
Parameter
ICSNS v3.07-19
Frame Headers
These are the header fields of an FC frame:
AA-56
R_CTL (Routing Control, 8 bytes): Frame type and function; used by the switch to route
frames
CS_CTL (Class Specific Control, 8 bytes): Class specific control information for Class 1, 4
&6
TYPE (Data Structure Type, 8 bytes): Type of Information Unit & ULP carried by this
frame
F_CTL (Frame Control, 24 bytes): Specifies number of fill bytes and sequence control
information
SEQ_CNT (Sequence Count, 16 bytes): Frame count identifying each frame in the
sequence
Ordered Sets
Transmission Word
Ordered Set
Data Word
Primitive Signal
Frame Delimiter
Start-of-Frame
End-of-Frame
Fill Word
Idle
Arbitrate
Control Signal
Receiver Ready
Virtual Circuit Ready
Close
Open
Dynamic Half-Duplex
Mark
Synchronize
Primitive Sequence
Non-Operational State
Offline State
Link Reset
Link Reset Response
Loop Initialization
Loop Port Bypass
Loop Port Enable
ICSNS v3.07-20
Ordered Sets are FC words (5 bytes) that are used for link-level functions. They are used
because they are fast and light, and because commands sometimes need to be exchanged before
devices have been assigned FC addresses.
The first byte of an Ordered Set is always the K28.5 character, which defines the word as an
Ordered Set. The second byte identifies the Ordered Set type, and the last two bytes can be used
to transmit other parameters.
There are three types of Ordered Sets:
Frame Delimiters are used to mark the beginning and end of frames.
Primitive Signals are used to initiate, synchronize, and terminate communication sessions,
and to maintain synchronization when no other information is being transmitted on the link.
The two types of Primitive Signals are fill words and control signals.
Primitive Sequences are similar to Primitive Signals, but are transmitted repeatedly until a
response is received. They are used for link and loop initialization.
AA-57
SCSI-FCP Operations
This section provides a brief overview of SCSI-FCP protocol operations.
FCP_CMD
IU 1
Fabric
2
Target
Frame
Sequence 1
Frames
Exchange
Sequence 2
Frame
Sequence 3
IU 2
FCP_DATA
FCP_RSP
IU 3
ICSNS v3.07-22
AA-58
Initiator
1
FCP_CMD
IU 1
Frame
Frame
Sequence 2
Exchange
5
IU 3
FCP_DATA
Sequence 1
4
FCP_XFR_
IU 2
RDY
Frames
Sequence 3
Frame
Sequence 4
Target
FCP_RSP
IU 4
ICSNS v3.07-23
AA-59
Link Services
This section describes what a Link Services command is, the role of Link Services, and how
Link Services differ from Ordered Sets.
BA_ACC
N_Port
N_Port
ICSNS v3.07-25
Link Services are upper-layer protocol (ULP) independent FC commands. Link Services are
used to implement control functions used in session management, such as fabric and port login,
address resolution, and error recovery. Link Services are defined within the Fibre Channel
Common Transport (FC-CT) framework.
Link Services are transparent to ULPs. In other words, Link Services frames are generated by
the initiator N_Port, not by the ULP driver. Upon receiving a Link Services command, the
target N_Port processes and discards all Link Services frames.
The preceding diagram shows an example of a Link Services exchange:
The N_Port on the left has sent an ABTS Link Services command to attempt to terminate
the current FC sequence.
The N_Port on the right receives the ABTS request and responds with the BA_ACC Link
Services command to indicate that the N_Port has successfully processed the ABTS
request.
Note
AA-60
Ordered Sets are not Link Services. Ordered Sets are short, one-word (four-byte)
commands that can carry, at the most, two bytes of parameters, whereas Link Services
commands consist of one or more FC frames. Ordered Sets are typically used at the
physical layer to perform basic link management functions. Link Services comprise a higherlevel command set that is essential to performing session management and error recovery.
ICSNS v3.07-26
Basic Link Services implement just a handful of basic control functions. All Basic Link
Services must be supported by all FC devices. Each Basic Link Service command is
transmitted as a single frameeither as a single-frame sequence or as a frame inserted into
a longer sequence.
Extended Link Services are used to perform a variety of control functions between ports at
the FC-2 layer. Extended Link Services are added regularly by the ANSI committee to
provide additional control functions. Only a few Extended Link Services are required by
the FC specification; most are optional. The number and functionality of Extended Link
Services are constantly evolving. Extended Link Services are always sent as a separate
exchange.
AA-61
No Operation
ABTS
Abort Sequence
BA_ACC
BA_RJT
RMC
PRMT
ICSNS v3.07-27
AA-62
NOP performs no specific operation but can be used to carry control information in the
header, such as initiating a Class 1 or Class 6 connection.
ABTS is used to abort the current sequence or an entire exchange. Only the ABTS
command has a reply: BA_ACC if accepted and BA_RJT if rejected.
RMC is used to request an immediate Class 1 or 6 disconnection. The recipient will abort
any current sequences and exchanges resulting in possible frame loss and will send a final
ACK frame.
LS_RJT
FLOGI
Fabric Login
PLOGI
Port Login
LOGO
Port Logout
PRLI
Process Login
PRLO
Process Logout
SCN
RSCN
SCR
ECHO
Required by
Fibre Channel
standard
Supported by
all SAN
implementations
ICSNS v3.07-28
FLOGI is used by an N_Port to log in to an F_Port and obtain a Fibre Channel address.
PLOGI is used by one N_port to log in to another N_Port, open a session, and exchange
service parameters.
LOGO is used to terminate a login session and free its associated resources.
LS_ACC notifies the sender of an Extended Link Service command that the command has
been accepted.
LS_RJT notifies the sender of an Extended Link Service command that the command was
not accepted. The command might have been rejected because the port was busy at the
time, the command was invalid or malformed, or the command is not supported by the
receiver. The reason code in the payload gives a more precise reason for the rejection.
AA-63
Summary
This topic summarizes the key points that were discussed in this lesson.
Summary
Fibre Channel has a layered model that is similar in some
respects to the OSI model.
Fibre Channel constructs include words, frames, sequences and
exchanges.
The Fibre Channel protocol is broken down into control and data
units.
The control units in Fibre Channel are called Ordered Sets.
Link Services are additional communications from port to port
used to relay logins, state changes, error situations and other
administration messages.
AA-64
ICSNS v3.07-29
Lesson 4
FC Flow Control
Overview
Like any network protocol, Fibre Channel (FC) must define how the flow of data is managed.
FC defines two flow control processes that are used either individually or together. FC uses a
unique receiver-based flow control strategy that ensures that data is delivered efficiently and
with a minimum of delivery errors.
Objectives
Upon completing this lesson, you will be able to explain Fibre Channel flow control and
addressing. This includes being able to meet these objectives:
Tx
Data
Data
Rx
PAUSE
ICSNS v3.07-4
Flow control is a mechanism for ensuring that frames are sent only when there is somewhere
for them to go. Just as traffic lights are used to control the flow of traffic in cities, flow control
manages the data flow in an FC fabric.
Some data networks, such as Ethernet, use a flow-control strategy that can result in degraded
performance:
A transmitting port (Tx) can begin sending data packets at any time.
When the receiving ports (Rx) buffers are completely filled and cannot accept any more
packets, Rx tells Tx to stop or slow the flow of data.
After Rx has processed some data and has some buffers available to accept more packets, it
tells Tx to resume sending data.
This strategy results in lost packets when the receiving port is overloaded, because the
receiving port tells the transmitting port to stop sending data after it has already overflowed.
Lost packets must be retransmitted, which degrades performance. Performance degradation can
become severe under heavy traffic loads.
AA-66
1
0
free buffers
DATA
Tx
Rx
READY
ICSNS v3.07-5
The transmitting port (Tx) counts the number of free buffers at the receiving port (Rx).
Before Tx can send a frame, Rx must notify Tx that Rx has a free buffer and is ready to
accept a frame. When Tx receives the notification, it increments its count of the number of
free buffers at Rx.
AA-67
N_Port
F_Port
E_Port
E_Port
F_Port
N_Port
ICSNS v3.07-6
Buffer-to-buffer flow control takes place between two ports that are connected by a FC
link, such as an N_Port and an F_Port, or two E_Ports, or two L_Ports.
End-to-end flow control takes place between the source node and the destination node.
Note that buffer-to-buffer is performed between E_Ports in the fabric, but it is not performed
between the incoming and outgoing ports in a given switch. In other words, FC buffer-to-buffer
flow control is not used between two F_Ports or between an F_Port and an E_Port within a
switch. FC does not define how switches route frames across the switch.
Buffer-to-buffer flow control is used in the following situations:
Class 1 connection request frames use buffer-to-buffer flow control, but Class 1 data traffic
uses only end-to-end flow control.
AA-68
F_Port
R_RDY
Data
R_RDY
N_Port
A
1 N_Port
B
R_RDY
R_RDY
ACK
5
Buffer-to-buffer
flow control
Buffer-to-buffer
flow control
ICSNS v3.07-7
AA-69
When end-to-end flow control is used, the transmitting port is responsible for ensuring that all
frames are delivered. Only when the transmitting N_Port receives the last ACK frame in
response to a sequence of frames sent does it know that all frames have been delivered
correctly, and only then will it empty its ULP data buffers. If a returning ACK indicates that the
receiving port has detected an error, the transmitting N_Port has access to the ULP data buffers
and can resend all of the frames in the sequence.
AA-70
ICSNS v3.07-8
In the base credit management method, the transmitting port knows how many buffers the
receiving port has. The transmitting port can therefore begin sending frames immediately
after a session is established.
In the alternate credit management method, the transmitting port knows only how many
free buffers the receiving port can guarantee. The transmitting port must therefore wait for
the receiving port to send an R_RDY signal before sending a frame.
AA-71
R_RDY
ACC
PLOGI
DATA
Rx
ICSNS v3.07-9
When the transmitting port sends a port login request, the receiver responds with an accept
frame (ACC) that includes information on the size and number of frame buffers it has
(BB_Credit). The transmitting port stores the BB_Credit value in a table.
The transmitting port also stores another value called BB_Credit_CNT, which represents
the number of used buffer credits. BB_Credit_CNT is set to zero after the ports complete
the login process.
Upon receiving the frame, the receiver processes the frame and moves it to upper-layer
protocol (ULP) buffer space. The receiving port then sends an R_RDY acknowledgement
signal back to the transmitting port, informing it that a buffer is available.
When the transmitting port receives the R_RDY signal, it then decrements its
BB_Credit_CNT.
To prevent overrunning the receiving ports buffers, the transmitting port can never allow
BB_Credit_CNT (the count of frames which have not yet been acknowledged) to exceed
BB_Credit (the total number of buffers in the receiving port). In other words, if it cannot
confirm that the receiving port has a free buffer, it does not send any more frames.
AA-72
ICSNS v3.07-11
The number of buffer-to-buffers required for a link depends on the physical length of that link.
The number of credits required is calculated based on frame size, propagation delay (speed of
light in fiber), and the end-to-end latency of the link; of all of these factors, latency is the only
variable. On a pure FC link, latency is deterministic and depends primarily on the length of
the link and the number of hops. On an FC WAN link (such as FCIP), latency depends on the
characteristics of the WAN.
The default credit allocation on most vendors switches is generally sufficient for intradatacenter links. However, the credit allocation often must be increased for long-haul links. FC
WAN links, including FCIP, typically require additional buffer credits due to the increased
latency of the IP network.
Cisco 16-port switch modules support up to 255 credits per port, which provides ample credits
for most applications.
AA-73
(Round_Trip_Time + Processing_Time)
Serialization_Time
Initiator N_Port
2007 Cisco Systems, Inc. All rights reserved.
Target N_Port
ICSNS v3.07-12
You can calculate the number of credits required on a link to maintain optimal performance
using the following formula:
Credits = (Round_Trip_Time + Processing_Time) / Serialization_Time
Example
This diagram and the following two diagrams illustrate how the required number of BB_Credits
are calculated for a 10km, 1Gb/s FC link:
AA-74
At a link rate of 1.0625 Gb/s, the time required to serialize (transmit) each byte is 9.41ns.
(Note that each byte is 10 bits due to 8b/10b encoding.)
The maximum Fibre Channel frame size is 2048 bytes. The frame size used in an actual
customer environment would be based on the I/O characteristics of the customers
applications. You also need to account for the frame header, which is 36 bytes, and the
number of IDLEs between frames, which is usually 6 IDLEs, or 24 bytes. This gives a total
of 2108 bytes.
The total serialization time for a 2108-byte frame (including idles) is 19.84s, or
approximately 20s.
Processing time:
Assume same as deserialization time 20s
10Km
20s
20s
50s
Frame
Frame
Initiator N_Port
Target N_Port
ICSNS v3.07-13
The speed of light in a fiber optic cable is approximately 5s per kilometer, so each frame
will take about 50s to travel across the link.
The receiving port must then process the frame, free a buffer, and generate an R_RDY.
This processing time can varyfor example, if the receiver ULP driver is busy, the frame
might not be processed immediately. In this case, we can assume that the receiving port
will process the frame immediately, so the processing time is equal to the time it takes to
deserialize the frame. The deserialization time is equal to the serialization time: 20s
AA-75
20s
50s
Frame
Frame
R_RDY
50s
Initiator N_Port
AA-76
Target N_Port
ICSNS v3.07-14
The receiving port then transmits a credit (R_RDY) back across the link. This response
takes another 50s to reach the transmitter.
The total latency on the link is equal to the frame serialization time plus the round-trip time
across the link, or about 120s.
Initiator N_Port
Frame
Frame
Frame
Frame
Frame
Target N_Port
ICSNS v3.07-15
Given a frame serialization time of 20s, and a total round-trip latency of 120s, there
could be up to 6 frames on the link at one time. In other words, six buffer-to-buffer credits
are required to make full use of the bandwidth of the link.
AA-77
2Gb/s
Payload Size
Serialization
Time
Credits
Required
Serialization
Time
Credits
Required
2048 bytes
19.84s
9.93s
11
1024
10.20
11
5.11
21
512
5.38
20
2.69
38
256
2.97
35
1.49
68
128
1.77
58
0.89
114
64
1.17
87
0.58
172
32
0.87
116
0.43
232
ICSNS v3.07-16
AA-78
16 15
Domain
08 07
00
Port
Area
FC
Nodes
FC
HBA
FC
Hub
HBA
Switch
FC
FC
FC
HBA
FC
HBA
ICSNS v3.07-18
The FC point-to-point topology uses a 1-bit addressing scheme. One port assigns itself an
address of 000000 and then assigns the other port an address of 000001.
The FC Arbitrated Loop topology uses an 8-bit addressing scheme:
The Arbitrated Loop Physical Address (AL_PA) is an 8-bit address, which provides 256
potential addresses. However, only a subset of 127 addresses are available due to 8b/10b
encoding requirements.
One address is reserved for an FL_Port, so there are 126 addresses available for nodes.
The Domain ID is used to define a switch. Each switch receives a unique Domain ID.
The Area ID is used to identify groups of ports within a Domain. Areas can be used to
group port ports within a switch, and are also used to uniquely identify fabric-attached
arbitrated loops. Each fabric-attached loop receives a unique Area ID.
Although the Domain ID is an 8-bit field, only 239 Domains are available to the fabric:
AA-79
Each switch must have a unique Domain ID, so there can be no more than 239 switches in a
fabric. The largest director-class switch available today has 256 ports, so the practical limit on
the number of nodes that can be supported in a fabric is 61184 ports (239 domains x 256 ports).
With 16-port switches the total port count is reduced to 3824 (239 domains x 16 ports), minus
the number of ports used for ISLs. Note that these calculations do not take into account ports
consumed by inter-switch links (ISLs)which reduces the number of portsor the fact that an
arbitrated loop multiple L_Ports can be attached to a single FL_Portwhich increases the
potential number of ports.
AA-80
23
16 15
08 07
00
Fabric
Domain
Area
Port
Public Loop
Domain
Area
AL_PA
Private Loop
00000000 00000000
AL_PA
ICSNS v3.07-19
Public NL_Ports are assigned a full 24-bit fabric address when they log into the fabric.
There are 126 AL_PA addresses available to NL_Ports in an arbitrated loop; the AL_PA
0x00 is reserved for the FL_Port (which is logically part of both the fabric and the loop).
The Domain and Area fields are identical to those of the FL_Port to which the loop is
connected.
Private NL_Ports can communicate with each other based upon the AL_PA, which is
assigned to each port during loop initialization.
Private NL_ports are not assigned a 24-bit fabric address, and the Domain and Area
segments are not used.
AA-81
World-Wide Names
This section introduces WWNs, a second addressing scheme used on FC SANs.
World-Wide Names
Every Fibre Channel port and node has a hard-coded
address called a World Wide Name (WWN):
Allocated to manufacturer by IEEE
Coded into each device when manufactured
64 or 128 bits (128 bits most common today)
20:00:00:45:68:01:EF:25
Example
Example WWNs
WWNs from
from aa Dual-Ported
Dual-Ported Device
Device
WWNN
WWPN A
WWPN B
20:00:00:45:68:01:EF:25
21:00:00:45:68:01:EF:25
22:00:00:45:68:01:EF:25
ICSNS v3.07-21
WWNs are unique identifiers that are hard-coded into FC devices. Every FC port has at least
one WWN. Vendors buy blocks of WWNs from the IEEE and allocate them to devices in the
factory.
WWNs are important for enabling fabric services because they are:
These characteristics ensure that the fabric can reliably identify and locate devices, which is an
important consideration for fabric services. When a management service or application needs to
quickly locate a specific device:
1. The service or application queries the switch Name Server service with the WWN of the
target device
2. The Name Server looks up and returns the current port address that is associated with the
target WWN
3. The service or application communicates with the target device using the port address
AA-82
WWNNs uniquely identify devices. Every host bus adaptor (HBA), array controller,
switch, gateway, and FC disk drive has a single unique WWNN.
WWPNs uniquely identify each port in a device. A dual-ported HBA has three WWNs: one
WWNN and a WWPN for each port.
WWNNs and WWPNs are both needed because devices can have multiple ports. On singleported devices, the WWNN and WWPN are usually the same. On multi-ported devices,
however, the WWPN is used to uniquely identify each port. Ports must be uniquely identifiable
because each port participates in a unique data path. WWNNs are required because the node
itself must sometimes be uniquely identified. For example, path failover and multiplexing
software can detect redundant paths to a device by observing that the same WWNN is
associated with multiple WWPNs.
Cisco MDS switches use the following acronyms:
AA-83
Summary
This topic summarizes the key points that were discussed in this lesson.
Summary
Fibre Channel uses a credit-based strategy
Two types of flow control:
Buffer-to-buffer (port-to-port)
End-to-end (source-to-destination)
Credit requirements depend on frame size, RTT,
serialization, and processing time
FC addressing is a 24-bit number:
3 bytes represent: [ domain ] [ area ] [ port ]
Every Fibre Channel port and node has a hard-coded
address called a World Wide Name (WWN).
2007 Cisco Systems, Inc. All rights reserved.
AA-84
ICSNS v3.07-22
Lesson 5
FC Login
Overview
The Fabric Login, Port Login, and Process Login protocols define how fabric ports behave
when they are brought online and when they want to establish a communication session. This
lesson provides a detailed examination of each of the login protocols. It explains the role that
each login protocol serves, and identifies the commands that are exchanged during each phase
of each protocol.
Objectives
Upon completing this lesson, you will be able to describe the Fibre Channel device login
process. This includes being able to meet these objectives:
Fabric Login
This section provides an overview of the session establishment protocols that are performed by
N_Ports and F_Ports in a fabric topology.
Fabric Login
Fabric
Node
N_Port A
F_Port A
F_Port B
FLOGI
FLOGI
N_Port B
Node
FLOGI
FLOGI
PLOGI
PLOGI
Process
ProcessA
Process
PRLI
PRLI
Process B
ICSNS v3.07-4
Before an N_Port can begin exchanging data with other N_Ports, three processes must occur:
AA-86
The N_Port must log in to its attached F_Port. This process is known as Fabric Login
(FLOGI).
The N_Port must log in to its target N_Port. This process is known as Port Login (PLOGI).
The N_Port must exchange information about ULP support with its target N_Port to ensure
that the initiator and target process can communicate. This process is known as Process
Login (PRLI).
Hey! I am connected
to something I must
tell it I am here and in
an Off-Line State.
F_Port
Switch
N_Port
OLS
F_Port
NOS
F_Port
Node
N_Port
1
Node
F_Port
ICSNS v3.07-5
FLOGI is the initial bootstrap process that occurs when an N_Port is connected to an F_Port.
FLOGI is mandatory for N_Ports, and optional for NL_ports.The N_Port uses Fabric Login to
discover if a fabric is present. Communication with other N_Ports may not be attempted until
the Fabric Login process is complete.
The FLOGI protocol follows this process:
1. The F_Port sends a primitive sequence of NOS (Not Operational) to the N_Port.
2. When the N_Port receives the NOS, it responds with a primitive sequence of OLS (Offline
State) to begin link initialization.
AA-87
F_Port
N_Port
Switch
F_Port
LRR
F_Port
Node
N_Port
3
LR
Node
4
F_Port
ICSNS v3.07-6
3. After the N_Port begins the initialization process with by sending OLS, the F_Port tries to
reset the port by sending an LR (Link Reset) command.
4. The N_Port responds with an LRR (Link Reset Response) command.
AA-88
Switch
N_Port
IDLE
F_Port
IDLE
F_Port
Node
N_Port
F_Port
Node
F_Port
ICSNS v3.07-7
5. From this point on, the link is active and IDLE fill words flow in both directions on the
link.
6. Following link initialization, a new N_Port uses an S_ID of 000000 or 0000[AL_PA] to
indicate that the port is unidentified during FLOGI. An existing N_Port uses its existing
port address as its S_ID.
AA-89
N_Port
Switch
F_Port
FLOGI
F_Port
Login
Server
Node
N_Port
7
LS_ACC
Node
6
F_Port
ICSNS v3.07-8
7. After the N_Port has established a link to its F_Port, the N_Port obtains a port address by
sending a FLOGI Link Services command to the switch Login Server (at Well-Known
Address 0xFFFFFE).
8. The Login Server sends an ACC reply that contains the N_Port address in the D_ID field.
When an N_Port is performing FLOGI and receives ACC frame that indicates that the ACC
came from another N_Port, then the N_Port that is logging in assumes that it is in a point-topoint configuration. In this case, the N_Port immediately initiates PLOGI with the other N_Port
after completing FLOGI.
AA-90
F_Port
Switch
N_Port
PLOGI
F_Port
LS_ACC
F_Port
Name
Server
Node
N_Port
9
Node
8
F_Port
ICSNS v3.07-9
9. After receiving a port address, the N_Port logs into the Fabric Name Server at address
0xFFFFFC and transmits its service parameters, such as the number of buffer credits it
supports, its maximum payload size, and supported Classes of Service.
10. The Name Server responds with an LS_ACC frame.
AA-91
Word
0
1-4
5-6
7-8
9-12
13-17
18-21
22-25
26-29
Bits 31-24
Bits 23-16
Bits 15-8
Bits 7-0
Command 04
00
00
00
Common Service parameters (16 bytes)
N_Port Name (8 bytes)
Node Name (8 bytes)
Class 1 Service Parameters (16 bytes)
Class 2 Service Parameters (16 bytes)
Class 3 Service Parameters (16 bytes)
Class 4 Service Parameters (16 bytes)
Vendor Version Level (16 bytes)
ICSNS v3.07-10
AA-92
ICSNS v3.07-11
The preceding image shows an analyzer trace that displays part of a fabric login sequence. The
top of the trace shows the OLS-LR-LRR sequence that occurred while the link was being
initialized. The right-hand panel shows the contents of the FLOGI frame from the N_Port to the
F_Port (FFFFFE).
Useful information can be obtained by studying these analyzer traces:
Notice that at this time the N_Port does not yet have an address.
Notice also that the World Wide Port Name is the same as the World Wide Node Name.
This is common in single ported nodes.
The N_Port does not support Class 1, but it does support Classes 2 and 3.
The N_Port supports Alternate Buffer Credit Management Method and can guarantee 2
BB_Credits at its receiver port.
You can see that this is a single-frame Class 3 sequence because the Start of Frame is
SOFi3 and End of Frame is EOFt, meaning that this initial first frame is also the last one in
the sequence.
AA-93
Port Login
This section provides a description of the PLOGI protocol. Each command used during PLOGI
is identified; however, the parameters exchanged during each command are not described in
detail.
Port Login
PLOGI
N_Port
F_Port
PLOGI
F_Port
F_Port
Node
N_Port
Node
F_Port
ICSNS v3.07-13
After completing the FLOGI process, the N_Port can log in to another N_Port using the PLOGI
protocol. PLOGI must be completed before the nodes can perform any ULP operations.
The PLOGI protocol follows this process:
1. The initiator N_Port sends a PLOGI frame that contains the N_Ports operating parameters
encapsulated in the payload.
AA-94
LS_ACC
N_Port
F_Port
LS_ACC
F_Port
Node
N_Port
2
Node
F_Port
ICSNS v3.07-14
2. The target N_Port responds to the initiator N_Port by sending an ACC frame that specifies
the target N_Ports operating parameters. The operating system driver that manages the
initiator N_Port stores this information in a parameter block.
An N_Port can be logged into multiple N_Ports simultaneously. N_Ports typically perform Port
Logout (PLOGO) only when one of the nodes go offline.
AA-95
ICSNS v3.07-15
The image shows the N_Port logging in to another N_Port a PLOGI command. Note that the
N_Port has provided the same data that it provided when it logged in to the Name Server.
AA-96
ICSNS v3.07-16
Discover Address (ADISC) can be used to confirm another ports address or to discover
whether the other port has a hard-coded address.
Discovery Fabric Service Parameters (FDISC) can be used to verify fabric service
parameters.
Discover N_Port Service Parameters (PDISC) can be used to verify the service parameters
of another N_Port.
These commands allow ports to query and verify fabric and port parameters without performing
PLOGI and thus forcing logout of the current session.
AA-97
Process Login
This section provides a description of the PRLI protocol. Each command used during PRLI is
identified; however, the parameters exchanged during each command are not described in
detail.
Process Login
I am going to be using
the SCSI protocol. I
wonder if the target can
support the same
functions as I can.
PRLI
N_Port
F_Port
PRLI
F_Port
Node
N_Port
F_Port
Node
1
F_Port
ICSNS v3.07-18
After completing the PLOGI protocol, both N_Ports knows about the others Fibre Channel
(FC) operating parameters capabilities. At this point, the driver for the initiator port can open a
channel with the driver associated with the target port using the PRLI protocol. The PRLI
protocol is used to establish a session between two FC-4 level logical processes.
The PRLI protocol follows this process:
1. The initiator sends a PRLI frame that contains information about its ULP support.
AA-98
LS_ACC
N_Port
F_Port
LS_ACC
F_Port
Node
N_Port
2
Node
F_Port
ICSNS v3.07-19
2. The target port responds with an ACC frame that contains details about its ULP support. At
this point, a channel has been successfully opened and communication can take place. The
relationship between the initiator process and the target process is known as an image pair.
AA-99
LS_ACC
PRLO
N_Port
PRLO
4
F_Port
LS_ACC
F_Port
Node
N_Port
F_Port
Node
3
F_Port
ICSNS v3.07-20
3. When the initiator has finished exchanging data with the target, the initiator sends a Process
Logout (PRLO) frame.
4. The target responds with an ACC frame, and the image pair is then terminated.
At this point, the image pair must be established again before further communication can take
place.
AA-100
ICSNS v3.07-21
The image shows the N_Port performing process login (PRLI) with its target N_Port. The
payload data in a PRLI is relevant to the ULP, which in this case is SCSI-FCP. For example:
The ULP driver does not use the SCSI-3 XFER_RDY command during SCSI Read
operations.
AA-101
SP=0000001
2
ICSNS v3.07-22
The image shows the target N_Port responding to the PRLI command shown on the previous
page.
AA-102
Inform
Inform
all
all ports
ports
Arbitration
A
Arbitrate
Arbitrate
for
for ownership
ownership
Open
Open
aa channel
channel
Transfer
Transfer
data
data
Select
Select
B aa loop
loop master
master
C
Select
Select
AL_PAs
AL_PAs
Position
Position Map
Map
Reporting
Reporting
Position
Position Map
Map
Distribution
Distribution
Close
Close
the
the channel
channel
ICSNS v3.07-24
A. The port that has just come online informs the other ports that it is initiating the
initialization process.
B. The ports cooperatively select one port to manage the remainder of the process. This
port is known as the loop master.
D. The ports collect an address book of port addresses known as the position map or
position bitmap.
E. The ports circulate and save a copy of the completed position map.
AA-103
C. Transfer data
The Open-Transfer-Close process used on arbitrated loops is similar to the Select-TransferRelease process used by SCSI.
AA-104
LPSM
LPSM
Upper-layer
protocols
FC-4
FC-3 Common Services
FC-2 Framing & flow control
FC-FS
FC-1 Encoding
FC-0 Physical interface
2007 Cisco Systems, Inc. All rights reserved.
FC-PI
ICSNS v3.07-25
AA-105
Summary
This topic summarizes the key points that were discussed in this lesson.
Summary
Fabric Login (FLOGI) is performed by all connecting
devices to acquire an FCID
Port Login (PLOGI) is performed to pass identification
and capability to the name server
Process Login (PRLI) is performed between end
devices to exchange operating parameters capabilities
and establish a session
Loop initialization is performed by all device
connecting to a loop hub or device to acquire an
AL_PA (arbitrated loop physical address)
AA-106
ICSNS v3.07-26
Lesson 6
FC Error Recovery
Overview
Each Fibre Channel (FC) layer plays a role in error management. In this lesson, you will learn
about how each layer detects and recovers from errors. You will also learn about configuration
parameters that affect the way a FC SAN responds to error conditions.
Objectives
Upon completing this lesson, you will be able to explain how Fibre Channel recovers from
errors. This includes being able to meet these objectives:
FC-1 Errors
This section describes the error recognition protocol in the FC-1 layer of Fibre Channel.
FC-1 Errors
Four consecutive invalid transmission words trigger an
FC-0 loss-of-synchronization error
Synchronization Acquired
1
ICSNS v3.07-4
Four consecutive invalid transmission words must occur to trigger an FC-0 loss-ofsynchronization error. This requirement prevents transient errors from causing loss of
synchronization.
The preceding graphic shows the trigger conditions required to cause a loss-of-synchronization
error:
AA-108
After three consecutive invalid words, the system is in state 4. The next consecutive invalid
word will trigger a loss-of-synchronization error.
ICSNS v3.07-5
An expected Primitive Sequence is not received within a specified timeout period during
the link initialization, reset, and failure protocols.
R_T_TOV
The timeout period that governs both of these cases is the Receiver-Transmitter Timeout Value
(R_T_TOV). The default value of R_T_TOV is 100ms. R_T_TOV cannot be changed on MDS
switches.
R_T_TOV is an FC-1 layer timer. This timer is used to detect loss of synchronization between
the transmitter and receiver, and is also used to time link reset events. If R_T_TOV is too low,
the transmitter and receiver will experience repeated loss of synchronization and link reset
events. If R_T_TOV is too low for the link reset process to complete, the link will not come up.
AA-109
R_T_TOV (Cont.)
All ports in the fabric must have the same R_T_TOV
value
Fabric will segment if R_T_TOV is not consistent
Shorter R_T_TOV (100s) has been proposed to
provide faster error detection required for real-time
systems, such as avionics environments
ICSNS v3.07-6
R_T_TOV is a fabric-wide timeout value. All ports in the fabric must have the same value. If
R_T_TOV is not the same on two connected switches, the fabric will segment.
The default value of 100ms is acceptable in most situations. However, R_T_TOV might need
to be adjusted in some environments. Real-time environments like FC-AE require very fast
responses, fast error recovery, and low latency. For applications with these requirements,
100ms is a long time to wait; 5000 2KB frames could be sent in that time. (Each 2KB frame
takes approx 20s to serialize, so 5000 x 20s = 100ms.) Some FC developers have proposed
reducing the default R_T_TOV to 100s (1000 times shorter) for certain environments.
AA-110
FC-2 Errors
This section describes the error recognition protocol in the FC-2 layer of Fibre Channel.
FC-2 Errors
Frame Errors
Invalid D_ID
Invalid S_ID
Invalid SEQ_ID
Invalid OX_ID
Invalid RX_ID
Invalid SEQ_CNT
Invalid R_CTL
Invalid F_CTL
Invalid DF_CTL
Unsupported ULP
or invalid TYPE
Invalid Offset
Resource Errors
Too many
Sequences
Cannot establish
Exchange
Delimiter Errors
Delivery Errors
Invalid SOF/EOF
Missing frame
or ACK
Undeliverable
frames
Unsupported
Class of Service
Out-of-order
SEQ_CNT
Sequence or link
timeout
ICSNS v3.07-8
Frame Errors occur when any of the frame header fields are invalid, such as a frame with
an invalid D_ID or unsupported ULP.
Resource Errors occur when the sequence count exceeds the maximum number of
sequences within an exchange (256) or when a valid exchange cannot be established.
Delimiter Errors occur when either SOF or EOF are invalid or if a frame is received with
an unsupported Class of Service.
Delivery Errors occur when frames arrive out of sequence, are missing, or fail to arrive
within a specified time period.
AA-111
Sent by an N_Port
Frame is invalid
F_RJT
P_RJT
Receiver cannot
accept frame
F_BSY
P_BSY
ICSNS v3.07-9
In Classes 1, 2, 4, and F, which all provide acknowledged delivery, a RJT or BSY response will
be sent to the transmitting port when a frame is invalid or cannot be delivered:
The fabric will reply with F_BSY if the destination switch port had no free buffers.
The fabric will reply with F_RJT if the frame had an invalid D_ID or S_ID, or if the port is
unavailable.
The receiver will reply with P_BSY if the receiver port had no free buffers.
The receiver will reply with P_RJT if the requested Class of Service or ULP is not
supported.
In Class 3, frames will be discarded without notification if the receiver port has no buffers, is
unavailable, or does not support the requested Class of Service or ULP.
AA-112
E_D_TOV
E_D_T_V is an FC-2 layer timer:
Determines how long a receiver waits for a response before
declaring an error condition
ICSNS v3.07-10
E_D_TOV
E_D_TOV is an FC-2 layer timer. E_D_TOV determines how long a receiver waits for an
expected response before declaring an error condition. For example, if a frame arrives out of
sequence, the receiver waits E_D_TOV before it declares an error and aborts the sequence.
The default value of E_D_TOV is usually 2 seconds on FC switches. This value is always
sufficient for DWDM and almost always sufficient for SONET/SDH. On FCIP links, however,
you might need to increase E_D_TOV due to two factors:
There might be unusually high latency in the IP WAN, or there might be dropped packets
that need to be retransmitted. In either case, it is possiblealthough unlikely in a welldesigned IP networkfor the total round-trip latency to exceed 2 seconds.
The bandwidth of the IP link might be less than the bandwidth of the FC fabric, so frames
could pile up in the fabric if the IP link becomes congested.
AA-113
Sequence Recovery
Sequence Error
Detected
Retry ABTS
NO
Send ABTS
Reply BA_ACC
BA_ACC
received?
YES
BA_ACC
received?
NO
Implicit logout
of other port
YES
??
??
ICSNS v3.07-11
Sequence Recovery
This diagram illustrates the first part of the sequence recovery process:
When a sequence error occurs, the N_Port that detected the error sends the Abort Sequence
(ABTS) Extended Link Services command to abort the sequence. ABTS can be transmitted
as part of the current sequence or as a new sequence.
The other N_Port responds with the Basic Accept (BA_ACC) command..
If the N_Port that sent ABTS does not receive BA_ACC, it assumes that the other ports is
no longer available and performs an implicit port logout.
In some cases, the entire exchange is aborted with the Abort Exchange (ABTX) command, and
the entire exchange must be reestablished.
AA-114
ICSNS v3.07-12
When ABTS is issued to abort a sequence, the fabric must be purged of all frames in the
sequence before the sequence can be re-transmitted; otherwise, old frames could arrive out of
sequence. The receiver might not be able to differentiate between the old frames and the
retransmitted frames, and data errors could result at the ULP level.
Therefore, before the sequence is resent, the initiator waits for a specified period of time before
retransmitting the sequence. This time period is determined by the Resource Allocation
Timeout Value (R_A_TOV):
AA-115
Retry ABTS
NO
Send ABTS
Reply BA_ACC
BA_ACC
received?
YES
BA_ACC
received?
NO
Implicit logout
of other port
YES
Wait R_A_TOV
Send RRQ
ICSNS v3.07-13
AA-116
After BA_ACC is received, the originating N_Port waits for a time equal to R_A_TOV (10
seconds in a fabric).
After R_A_TOV has expired, the originating N_Port sends the Resource Recovery
Qualifier (RRQ) command.
After RRQ is sent, the ports can begin retransmission of the failed sequence or exchange.
R_A_TOV
R_A_TOV is an FC-2 layer timer:
Specifies how long a frame can be in transit
Used to determine how long a sender must wait before it can
begin resending an aborted sequence
ICSNS v3.07-14
R_A_TOV
R_A_TOV is an FC-2 layer timer. R_A_TOV specifies how long a FC frame can be in transit.
This value is used to determine how long a sender must wait before it can begin resending a
sequence after the sequence was aborted after an error occurred. The sender must wait for
R_A_TOV because if a sender begins to resend a sequence before the frames from the old
aborted sequence have been received, discarded or expired, frames from the old and new
sequences might arrive intermixed. Because the FC protocol provides no way for the receiver to
guarantee that the new sequence ID will be different than the old sequence ID, the sender waits
until there is no chance that frames could still be in transit.
The default value of R_A_TOV is usually 10 seconds on FC switches. This value is sufficient
for any type of WAN link except some IP links.
AA-117
ICSNS v3.07-16
By default, the SCSI-FCP protocol uses the Abort, Discard Multiple Sequences exchange
error policy, in which all sequences in the exchange are retransmitted. However, discarding the
entire exchange is often not the most desirable solution. The initiator must wait for the
R_A_TOV timeout period (10 seconds by default) to expire before retrying the aborted
exchange. In addition to reducing overall performance, this long wait time can have greater
impact in some situations. For example, if the failed operation is a backup application
streaming frames to a tape drive, then the tape buffer will empty and the drive will stop. When
the buffer begins to fill again, the tape will rewind, run up to speed, and continue streaming
from the last file mark.
AA-118
E_D_TOV
Initiator
Target
FCP_CMND
FCP_RSP
ABTS
R_A_TOV
BA_ACC
12 seconds
Retry FCP_CMND
ICSNS v3.07-17
Something goes wrong during the exchange and the FCP status sequence (FCP_RSP) does
not arrive.
The initiator waits for E_D_TOV (2 seconds) for the missing FCP_RSP to arrive.
The initiator then waits for R_A_TOV (10 seconds) for all frames to be purged from the
fabric before retrying the FCP command.
AA-119
ICSNS v3.07-18
Some ports are capable of using an enhanced recovery technique that allows nodes to recover
from sequence errors without having to abort the entire exchange. This enhanced recovery
technique is defined by FC-4 Link Services commands. FC-4 Link Services are similar to
Extended Link Services, but FC-4 Link Services are defined by the ULP, whereas Extended
Link Services are defined by FC-2.
The Read Exchange Concise (REC) Extended Link Service command allows the initiator to
ask the target to report the status of the exchange.
The Sequence Retransmission Request (SRR) Extended Link Service command requests
retransmission of the exchange beginning at a specific sequence.
The Read Exchange Concise Timeout Value (REC_TOV) determines how long the initiator
waits before sending the REC command. The default value of REC_TOV is equal to the value
of E_D_TOV (2 seconds) plus 1 second.
The REC and SRR commands are not defined in the FC-PH specification; rather, they are
defined in the FCP-2 ULP specification. Many vendors take advantage of this technique.
AA-120
REC_TOV
Initiator
Target
FCP_CMND
FCP_RSP
X
REC
3 seconds
FC-4_ACC
SRR
FC-4_ACC
Resend FCP_RSP
ICSNS v3.07-19
An FCP_CMND is issued by the initiator, but something goes wrong during the exchange.
The initiator waits REC_TOV (3 seconds) for the missing frame to arrive
The initiator sends REC to request information about the status of the exchange.
When the initiator receives FC-4_ACC, the initiator knows where in the exchange the
failure occurred. The initiator then sends SRR to request retransmission of the sequence.
A total of about 3 seconds (the default value of REC_TOV) elapses until the initiator sends
REC. This is about one quarter of the time elapsed during basic error recovery (12 seconds) in a
similar situation.
AA-121
Summary
This topic summarizes the key points that were discussed in this lesson.
Summary
At FC-1, all ports in the fabric must have the same R_T_TOV
value
Frame Errors
2.
Resource Errors
3.
Delimiter Errors
4.
Delivery Errors
AA-122
ICSNS v3.07-20
Summary (Cont.)
Receiver Transmitter Time-Out Value (R_T_TOV):
Short timer used to detect link-level failures
ICSNS v3.07-21
AA-123
AA-124
Lesson 7
FC Switched Fabric
Overview
This lesson explains three important protocols in a Fibre Channel switched fabric. The fabric
configuration protocol, the FSPF protocol, and the RSCN protocol. You will also learn about
fabric services and how they are addressed.
Objectives
Upon completing this lesson, you will be able to describe the Fibre Channel Switched Fabric
protocol. This includes being able to meet these objectives:
Word Sync
Acquired
E_Port Mode?
Exchange Link
Parameters (ELP)
YES
Exchange Switch
Capabilities (ESC)
NO
Exchange Fabric
Parameters (EFP)
Domain ID
Assigned?
YES
The diagram describes the steps taken, when a fabric is first initialized, a new switch is added
to an existing, or a link becomes active.
AA-126
A switch port detects a valid signal on its attached link and achieves word synchronization.
The switch port begins link initialization. If the port is capable of operating at more than
speed, it may perform speed negotiation.
The switch port determines the proper operating mode; FL_Port, F_Port or E_Port.
Exchange Link Parameters (ELP). When two E_Ports are connected and the link initialized,
the ports exchange link parameters. This is accomplished by using a set of switch internal
link service (SW_ILS) parameters called Exchange Link Parameters (ELP). The ELP is
sent from the Fabric Controller (xFFFFFD) in one switch to the Fabric Controller in the
neighbor switch using Class-F service.
Exchange Switch Capabilities (ESC). Next, ESC is sent between neighboring Fabric
Controllers to agree upon a common routing protocol.
Exchange Fabric Parameters (EFP). The principal switch is selected using the Exchange
Fabric Parameters (EFP) (SW_ILS). The EFT is sent between Fabric Controllers in
neighbor switches.
Domain ID Identifier (DIA). After a principal switch has been selected, Domain_IDs are
assigned to the switches. The Principal Switch assigns itself a Domain ID, then floods the
fabric with this information.
Request Domain Identifier (RDI). After a switch receives a Domain Identifier Assigned
(DIA) switch internal link service, it can request a Domain_ID from the principal switch by
sending a Request Domain identifier (RDI) to the principal switch.
Fabric Shortest Path First (FSPF). After the Domain_ID assignment phase is complete,
routing tables are built. The switch may use the standardized FSPF protocol or a vendorunique routing protocol
Build Routing Tables. Finally , each switch computes the paths it will use to deliver frames
to other switches.
AA-127
FSPF
This section provides an overview of the FSPF protocol.
FSPF
Fabric Shortest Path First
(FSPF):
FC
FC
HBA
HBA
FC
Frame 3
Frame 2
Frame 1
Single path
In-order delivery
2007 Cisco Systems, Inc. All rights reserved.
ICSNS v3.07-6
Routing using a single fixed path prevents looping of frames and, in a stable SAN, ensures inorder delivery. In other words, if routes are stable, frames always follow the same path.
However, if the least-cost route changes while a session is in progress, frames sent after the
route change might take the new route.
AA-128
FSPF (Cont.)
Three protocols used in FSPF:
The Hello protocol is used to establish communication between
two connected switches
Initial Link State Record (LSR) database synchronization
LSR database maintenance
ICSNS v3.07-7
The FSPF protocol maintains a Topology Database which is distributed to every switch in the
fabric. If a switch detects a lost connection, either to a Node or to another switch (ISL), it will
update the Topology Database and send a Link State Update frame to all other switches directly
connected to it. Each of these switches will update their Topology Database and pass the LSR
frame onto other switches. In this way the fabric is flooded with updates to the Topology
Database. Any LSR frames already received are discarded to stop duplicate LSRs from being
distributed throughout the fabric.
AA-129
ICSNS v3.07-8
AA-130
ICSNS v3.07-9
After a switch acquires a Domain_ID, it begins the process of building a routing table:
1. Does not know if neighbor switch has acquired a Domain_ID
2. Begins transmitting Hello messages to its neighbors on all initialized ISLs
3. Exchanges Domain_IDs with all neighbors
After two switches have exchanged Domain_IDs, the ISL is active and FSPF topology database
synchronization can begin.
The Hello Interval is the time in seconds between Hello messages sent by this port. Its
default value is 20 seconds.
The Dead Interval is the time in seconds this port will wait for a Hello message from the
attached port before removing the route to that port from the LSD. Its default value is 80
seconds.
AA-131
Note that the default values of these intervals mean that FSPF can take up to 100s to become
aware of a link failure. You can lower these values to promote faster recovery when a link fails,
but you should also keep in mind that Hello messages are flooded, so smaller Hello Interval
values increase congestion. The Hello Dead Interval should generally be set to 4 times the
Hello Interval to avoid triggering unnecessary FSPF route computation if Hello messages are
lost due to congestion.
AA-132
LSU(DB-A)
LSU(DB-B)
LSA(DBB)
LSA(DB-A)
LSU(LSR-A)
LSU(LSR-B)
LSA(LSRB)
LSA(LSR-A)
ICSNS v3.07-10
During topology database synchronization, each switch sends its entire LSD topology
database to its neighbor.
Switches synchronize databases by sending LSRs in a Links State Update (LSU) SW_ILS
extended link service command.
An LSU can contain one or more LSRs. An LSU with zero LSRs signals the end of the
database transmission.
When a switch receives an LSU, it compares each LSR in the LSU with its current
topology database. If the new LSR is not present in the switchs LSD, or if the new LSR is
newer than the existing LSR, the LSR is added to the database.
Each LSR is acknowledged with a Link State Acknowledgment (LSA) SW_ILS command
or with a newer instance of the LSR.
An ISL fails (or the switch associated with that ISL fails). A new LSR is transmitted to
remove the failed link(s) from the topology database.
An ISL reverts to the one-way communication state. A new LSR is transmitted to remove
the one-way link from the topology database.
AA-133
A new ISL completes link initialization (stage 1) and initial database synchronization (stage
2). One or more LSRs are transmitted to notify other switches to add the new information
to their databases.
This process by which LSRs are propagated through the fabric is known as reliable flooding.
When a switch receives an LSR, it retransmits the LSR on other links. After the LSR is
acknowledged, the switch stops transmitting that LSR on that link. The switch continues to
send the LSR on other links until acknowledgement is received on those links.
AA-134
Examples:
Default link cost for 1Gb/s link:
1 * (1.0625e12 / 1.0625e9) = 1000
Default link cost for 2Gb/s link:
1 * (1.0625e12 / 2.1250e9) = 500
ICSNS v3.07-11
AA-135
The calculation is performed on a link-by-link basis, so each link in a data path can be
advertised with a different cost. These costs are used by the path selection algorithm to
determine the most efficient paths. When a path contains multiple links, the costs of each link
are added up to determine the total cost of the path. In the case of two or more paths of equal
cost, the decision of which path to use is not specified and is determined by the switch vendor.
Note that FSPF only considers the ISLs along the data pathit does not consider the node-toswitch link at either end of the path. FSPF routes frames between domains only.
AA-136
Limitations of FSPF
FSPF algorithm does not account for traffic load
All frames in an exchange follow the same path
Path changes only in response to changes in the fabric
topology
Switch B
Host 2
Host 1
2Gb/s
Cost=500
2Gb/s
Cost=500
Storage 2
Storage 1 FC
FC
FC
HBA
FC
HBA
Switch A
2Gb/s
Cost=500
Switch C
ICSNS v3.07-12
Limitations of FSPF
The FSPF protocol supports load sharing, but it does not support load balancing. Load sharing
is significantly different than load balancing, and the distinction can have significant effects for
fabric design, especially when tuning performance:
Load balancing means that traffic load is balanced across multiple paths
FSPF does not account for actual path utilization. In other words, an unused path with a cost of
1000 will be disregarded in favor of an overutilized path with a cost of 500. All frames in an
exchange must follow the same path, and paths are recomputed only when the physical ISL
configuration changes.
The preceding diagram shows a simple SAN with two data paths:
AA-137
Host 2
Host 1
2Gb/s
Cost=500
2Gb/s
Cost=500
Storage 2
Storage 1 FC
FC
FC
HBA
FC
HBA
Switch A
1Gb/s
Cost=1000
Switch C
ICSNS v3.07-13
The FSPF least-cost path algorithm does not necessarily select the best path. For example, in
the preceding diagram, links AB and BC are 2Gb/s links, with a default cost of 500 per
link. Link AC is a 1Gb/s link with a default cost of 1000. (Note that this diagram differs from
the previous diagram only in that link AC is a 1Gb/s link in this diagram.)
There are two paths available from Switch A to Switch C:
Path ABC has a total cost of 1000 and supports 2Gb/s along the entire path
Path AC also has a total cost of 1000 but supports only 1Gb/s
FSPF will weight both paths identically. When a single pair of devices (Host 1 and Storage 1)
are attached to the SAN, FSPF might select path AC even though that path supports only half
the bandwidth of path ABC. (Path ABC does have greater latency than path AC,
but latency is a far less significant performance factor than bandwidth.)
When a second pair of devices (Host 2 and Storage 2) are attached to the same switches, the
switch will use the second equal-cost data path in an attempt to distribute the load evenly. In
other words, Host 1Storage 1 will be assigned one path, and Host 2Storage 2 will be
assigned the other path. Both data paths will be used.
In both situations, the administrator can force path selection by adjusting the administrative
weighting factor.
AA-138
RSCN
Registered State Change Notification
Switch
Path failure
FC
SCR
RSCN
FC
HBA
SCR
LS_ACC
Fabric
Controller
RSCN
Host
LS_ACC
Storage
ICSNS v3.07-15
Nodes register for notification by sending a State Change Registration (SCR) frame to the
Fabric Controller.
The Fabric Controller transmits RSCN commands to registered nodes when a fabric state
change event occurs. RSCNs are transmitted as unicast frames because multicast is an
optional service and is not supported by many switches.
Only nodes that might be affected by the state change are notified. For example, if the state
change occurs within Zone A, and Port X is not part of Zone A, then Port X will not
receive an RSCN.
AA-139
The RSCN message identifies the ports that were affected by the state change event, and it
identifies the general nature of the event. After receiving an RSCN, the node can then use
additional Link Services commands to obtain more information about the event. For example, if
the RSCN specifies that the status of Port Y has changed, the nodes that receive the RSCN can
attempt to verify the current (new) state of Port Y by querying the Name Server.
AA-140
ICSNS v3.07-16
The path between two Nx_Ports has changed (e.g., a change to the fabric routing tables that
affects the ability of the fabric to deliver frames in order, or an E_Port initialization or
failure)
An implicit fabric logout of an Nx_Port, including implicit logout resulting from loss-ofsignal, link failure, or when the fabric receives a FLOGI from a port that had already
completed FLOGI.
Loop initialization of an L_Port, and the L_bit was set in the LISA Sequence.
An Nx_Port can also issue a request to the Fabric Controller to generate an RSCN. For
example, if one port in a multi-ported node fails, another port in that node can send an
RSCN to notify the fabric about the failure.
AA-141
Reserved
Event Qualifier
Address
Format
Not very
informative
ICSNS v3.07-17
An RSCN frame payload contains one or more Port_ID Pages. Each Port_ID page is a 4-byte
page that describes a single state change that has occurred with respect to a single Nx_Port.
Each Port_ID page contains the following fields:
The Domain_ID, Area_ID, and Port_ID of the affected Nx_Port (bytes 1-3)
The Event Qualifier is a 4-bit code that specifies the general nature of the event:
0001 A Name Server object has changed; for example, a port came online or went
offline.
0010 A port attribute has changed; for example, the number of buffer credits assigned to
that port was changed.
0011 A fabric service object has changed; for example, an Alias_ID was added. In this
case, the Port_ID page will refer to the Well-Known Address of the affected fabric service.
0100 The switch configuration has changed; for example, a time-out value was changed.
Note that the Event Qualifiers do not communicate much information. For example, Event
Qualifier code 0001 indicates a change to a Name Server object. This could signify that a port
came online, went offline, or changed zones. The ports that receive the RSCN must then query
the Name Server to determine the specific change that occurred.
AA-142
Domain
Manager
Name
Server
Unzoned
Name Server
Zone
Server
Configuration
Server
Fabric
Management Server
Generic
ControllerServices
Alias
Server
Key
Server
Time
Server
Common Transport
Link Services
FC-1 Encoding
FC-0 Physical interface
2007 Cisco Systems, Inc. All rights reserved.
ICSNS v3.07-19
The FC-SW-2 specification defines several services that are required for fabric management.
These services include:
Name Server
Login Server
Address Manager
Alias Server
Fabric Controller
Management Server
Time Server
The FC-SW-2 specification does not require that switches implement all of these services;
some services can be implemented as an external server function. However, the services
discussed in this lesson are typically implemented in the switch, as in Cisco MDS 9000 Family
Switches.
AA-143
VSAN Manager
WWN Manager
Domain Manager
Fabric
Configuration
Principal Switch
Selection
FC_ID
Allocation
Domain ID
Allocation
FCID Database
and Cache
Port Manager
Login Server
ICSNS v3.07-20
Allocating domain IDs (requesting a domain ID, and assigning domain IDs to other
switches if this switch is the Principal Switch)
Performing the Fabric Build and Reconfiguration processes when the topology changes
The Domain Manager supports the Fabric Port Login Server, which is the service that N_Ports
use when logging in to the fabric. When an N_Port logs into the fabric, it sends a FLOGI
command to the Login Server. The Login Server then requests an FC_ID from the Domain
Manager and assigns the FC_ID the N_Port in its ACC reply to the FLOGI request.
The preceding diagram shows how the Domain Manager interacts with other fabric services:
AA-144
The VSAN Manager provides the Domain Manager with VSAN configuration and status
information.
The WWN Manager tells the Domain Manager what WWN is assigned to the VSAN.
The Port Manager provides the Domain Manager with information about the fabric
topology (a list of E_Ports) and notifies the Domain Manager about E_Port state changes.
The Login Server receives N_Port requests for FC_IDs during FLOGI.
The Domain Manager interacts with management services to allow administrators to view
and modify Domain Manager parameters.
ICSNS v3.07-21
FC_IDs
Enforces zoning by only providing information about nodes in the requestors zone
Is used by management applications that need to obtain information about the fabric
Each switch in a fabric contains its own resident name server, called a distributed Name Server
(dNS). Each dNS within a switch is responsible for the name entries associated with the domain
assigned to the switch. The dNS instances synchronize their databases using the RSCN process.
When a client Nx_Port wants to query the Name Service, it submits a request to its local via the
Well Known Address for the Name Server. If the required information is not available locally,
the dNS within the local switch responds to the request by making any necessary requests of
other dNS instances contained in the other switches. The communication between switches that
is performed to acquire the requested information is transparent to the original requesting
client.
Partial responses to dNS queries are allowed. If an entry switch sends a partial response back to
an Nx_Port, it must set the partial response bit in the CT header.
AA-145
Indexed Fields
Secondary Key
Indexed Fields
Port Name
Node Name
Node IP Address
Class of Service
FC-4 TYPEs
ICSNS v3.07-22
Register Object: Only one object at a time can be registered with the Name Server. A
Client registers information in the Name Server database by sending a registration request
containing a Port Identifier or Node Name.
Deregister Object: Only one global deregistration request is defined for the Name Server.
Name Server information is available, upon request, to other nodes, subject to zoning
restrictions. If zones exist within the fabric, the Name Server restricts access to information in
the Name Server database based on the zone configuration.
When a port logs out of a fabric, the Name Server deregisters all objects associated with that
port.
AA-146
ICSNS v3.07-23
The Fabric Configuration Service (FCS) supports configuration management of the fabric.
This service allows applications to discover the topology and attributes of the fabric.
The Zone Service provides zone information for the fabric to either management
applications or directly to clients.
The Unzoned Name Service provides access to provide information about the fabric
without regard to zones. This service allows management applications to see all the devices
on the entire fabric.
AA-147
Well-Known Addresses
Well-known addresses are the highest 16 addresses
in the 24-bit fabric address space
Broadcast Alias
FFFFFF
Mandatory
FFFFFE
Mandatory
Fabric Controller
FFFFFD
Mandatory
Name Server
FFFFFC
Optional
Time Server
FFFFFB
Optional
Management Server
FFFFFA
Optional
QoS Facilitator
FFFFF9
Optional
Alias Server
FFFFF8
Optional
FFFFF7
Optional
FFFFF6
Optional
Multicast Server
FFFFF5
Optional
Reserved
FFFFF4 FFFFF0
ICSNS v3.07-24
Well-Known Addresses
Well-known Addresses allow devices to reliably access switch services. All services are
addressed in the same way as an N_Port is addressed. Nodes communicate with services by
sending and receiving Extended Link Services commands (frames) to and from Well-Known
Addresses
Well-known addresses are the highest 16 addresses in the 24-bit fabric address space:
AA-148
FFFFF4FFFFF0 - Reserved
Summary
This topic summarizes the key points that were discussed in this lesson.
Summary
The FSPF protocol is the routing protocol used on FC SAN fabrics
Five stages of the FSPF protocol: Hello protocol; Initial topology
database synchronization; Topology database maintenance; Path
discovery; Path computation
FC Name Server is a database implemented by the switch that
stores information about each node
The Fabric Controller service provides a mechanism for state
change notification through the Registered State Change
Notification (RSCN) process
Well-known addresses are the highest 16 addresses in the 24-bit
fabric address space
ICSNS v3.07-25
AA-149
AA-150
Appendix B
Module Objectives
Upon completing this module, you will be able to describe installation and configuration
guidelines. This includes being able to meet these objectives:
AB-2
Lesson 1
Objectives
Upon completing this lesson, you will be able to explain the process used to install and power
up the switch. This includes being able to meet these objectives:
Describe the types of rack and cabinet installations that are compatible with the MDS 9000
platform
Describe the power supply configuration options for the MDS 9000 platform, and state the
power requirements of individual modules
Describe the characteristics and installation requirements of fan modules for the MDS 9000
Series
Describe the functions, interfaces, and installation requirements of MDS 9000 supervisor
modules
Installation Guidelines
This topic describes the installation guidelines for the MDS 9000 platform.
Installation Guidelines
Prepare the site:
Space evaluation
UPS type
Circuit sizes
Environmental evaluation
Power evaluation
Tools
Grounding evaluation
ESD straps
Grounding items
EMI evaluation
Gather network-related information
Hardware specifications
ICSNS v3.02-4
Installation Guidelines
Follow these guidelines when installing the Cisco MDS 9500 Series:
Plan your site configuration and prepare the site before installing the chassis. It is
recommended that you use the site planning tasks listed in the Cisco MDS Series Hardware
Installation Guide.
Ensure there is adequate space around the switch to allow for servicing the switch and for
adequate airflow.
Ensure the air conditioning meets the heat dissipation requirements listed in the Cisco MDS
Series Hardware Installation Guide.
Ensure the cabinet, or rack, meets the requirements listed in the Cisco MDS Series
Hardware Installation Guide.
Note
AB-4
Ensure the chassis is adequately grounded. Grounding the chassis is recommended in all
cases, and it is mandatory for Cisco MDS 9506 Directors that have a DC power supply
installed. If the switch is not mounted in a grounded rack or cabinet, it is recommend
connecting both the system ground on the chassis and the power supply ground to an earth
ground, regardless of whether the power supplies are AC or DC.
Ensure the site power meets the power requirements listed in the Cisco MDS Series
Hardware Installation Guide. If available, you can use an uninterruptible power supply
(UPS) to protect against power failures.
Note
Avoid UPS types that use ferroresonant technology. These UPS types can become unstable
with systems such as the Cisco MDS 9000 Family, which can have substantial current draw
fluctuations because of fluctuating data traffic patterns.
If you are using 200/240 VAC power sources in North America, the circuits must be
protected by two-pole circuit breakers.
Note
To prevent loss of input power, ensure that the total maximum loads on the circuits
supplying power are within the current ratings of the wiring and breakers.
Record your installation and configuration information as you work. See Site Planning
and Maintenance Records in the Cisco MDS Series Hardware Installation Guide.
Screw Torques
Use the following screw torques when installing the switch:
M3 screws: 4 in-lb
M4 screws: 12 in-lb
Required Equipment
Gather the following items before beginning the installation:
In addition to the grounding items provided in the accessory kit, you need the following items:
Grounding cable (6 AWG recommended), sized according to local and national installation
requirements; the required length depends on the proximity of the Cisco MDS 9500 to
proper grounding facilities.
Wire-stripping tool
For DC power supplies in a Cisco MDS 9506 Director, you need two 10-32 ring lugs for each
DC power supply.
AB-5
Front view
Rear view
Components:
6 KW power supply: 32.5 lbs
Po
we
C
Molock
du
les
Fab
ric
car
ds
rS
up
pli
es
Fa
n-
ICSNS v3.02-5
Installation of the Cisco MDS 9513 Director in a rack requires a mechanical lift to place the
chassis in the rack. Make sure you have access to the lift during the installation process. A fully
loaded 9513 can weigh about 300 pounds. Installations with three 9513s per rack should
include a floor-loading assessment to evaluate the static-load rating of the flooring as part of the
site evaluation process. For additional information about floor loading requirements, consult the
UL document GR-63-CORE, Network Equipment-Building System (NEBS) Requirements:
Physical Protection.
Module Weights
The following components are listed with their weights:
AB-6
AB-7
Installation Options
Standard telco rack (no side panels):
Not intended for use with the Cisco MDS 9513
Minimum of 6 inches (15.2 cm) of clearance between chassis is
recommended
Minimum of 2.5 inches (6.4 cm) of distance between the chassis air
vents and any walls is required
ICSNS v3.02-7
The Cisco MDS 9506 and MDS 9509 directors can be installed using the following methods:
The Telco and EIA Shelf Bracket Kit, optional and purchased separately, in addition to the
rack mount kit shipped with the switch.
The Telco and EIA Shelf Bracket Kit, optional and purchased separately, in addition to the
rack mount kit shipped with the switch.
The Telco and EIA Shelf Bracket Kit, optional and purchased separately, in addition to the
front brackets shipped with the switch.
The Cisco MDS 9509 Director can also be installed in a four-post nonthreaded cabinet or rack,
using the optional 9500 Shelf Bracket Kit.
The Cisco MDS 9513 Director can be installed in solid or perforated walled cabinets, but not in
two-post telco racks.
Note
AB-8
The Telco and EIA Shelf Bracket Kit is optional and is not provided with the switch. To order
the kit, contact your switch provider.
Note
The Telco and EIA Shelf Bracket Kit is not intended for use with a Cisco MDS 9509 Director
in a two-post telco rack. The MDS 9513 exceeds telco rack load ratings.
The front and rear doors must have at least a 60-percent open-area perforation pattern with
at least 15 square inches of open area per rack unit of door height.
The roof should be perforated with at least a 20-percent open-area perforation pattern.
A roof-mounted fan tray with bottom-to-top airflow that has a minimum of 500 cfm of
airflow exiting the cabinet roof through the fan tray.
Non-perforated (solid and sealed) front and back doors and side panels so that air travels
predictably from bottom to top.
A cabinet depth of 36 to 42 inches (91.4 to 106.7 cm) to allow the doors to close and
adequate airflow is recommended.
A minimum of 150 square inches (968 sq. cm) of open area must be at the floor air intake
of the cabinet.
The lowest piece of equipment should be installed a minimum of 1.75 inches (4.4 cm)
above the floor openings to prevent blocking the floor intake.
Minimum of 2.5 inches (6.4 cm) of distance between the chassis air vents and any walls is
required.
Not intended for use with the Cisco MDS 9513 director.
AB-9
ICSNS v3.02-8
Standard 19-inch four-post EIA cabinet, or rack, with mounting rails that conform to
English universal hole spacing per section 1 of ANSI/EIA-310-D-1992.
Standard two-post telco racks are not intended for use with the 9513.
The width between the rack-mounting rails must be at least 17.75 inches (45.1 cm). For fourpost EIA racks, this is the distance between the two front rails and rear rails.
The minimum spacing for four-post EIA cabinets (perforated or solid-walled) is as follows:
To ensure the minimum bend radius for fiber optic cables, the front mounting rails of the
cabinet should be offset from the front door by a minimum of 3 inches (7.6 cm) and a
minimum of 5 inches (12.7 cm) if cable management brackets are installed on the front of
the chassis.
A minimum of 2.5 inches (6.4 cm) of clear space between the side edge of the chassis and
the side wall of the cabinet; no sizeable flow obstructions should be immediately in the way
of the chassis air intake or exhaust vents.
Telco racks are not intended for use with the Cisco MDS 9513 Director. The MDS 9513
exceeds telco rack load ratings.
AB-10
ICSNS v3.02-9
If mounting the chassis in an open rack (no side panels or doors), ensure the rack meets two
requirements:
The minimum width between two front mounting rails must be 17.75 inches (45.1 cm).
For the Cisco MDS 9513 chassis 24.5 inches (62.2 cm), or 14 RU.
For the Cisco MDS 9509 chassis 24.5 inches (62.2 cm), or 14 RU.
For the Cisco MDS 9506 chassis 12.25 inches (31.1 cm), or 7 RU.
Note
The rack-mount support brackets provided with the Cisco MDS 9513 Director require an
additional height of 0.75 inches (1.9 cm). They are required for the installation of the Cisco
MDS 9513 Director and can not be removed.
Note
The side rail mount brackets provided with the Cisco MDS 9509 Director require an
additional height of 0.75 inches (1.9 cm). They are required only for the installation of the
Cisco MDS 9509 Director and can be removed, or left installed, after the front rack mount
brackets are securely fastened to the rack-mounting rails.
AB-11
ICSNS v3.02-11
To install an AC power supply in the Cisco MDS 9513 Director, follow these steps:
Step 1
Ensure that the system (earth) ground connection has been made.
Step 2
If a filler panel is installed, remove the filler panel from the power supply bay by
loosening the captive screw.
Step 3
Ensure that the power switch is in the off (0) position on the power supply that is
being installed.
Step 4
Grasp the power supply handles, one with each hand. Orient the power supply and
align it with the bay.
Note
AB-12
There is a handle at the top rear of the power supply you can also use to tilt the power
supply into the bay.
Step 5
Slide the power supply into the power supply bay. Ensure that the power supply is
fully seated in the bay.
Step 6
1. Power switch
2. Cable retention
ICSNS v3.02-12
Step 1
Plug the power cable into the power supply. Tighten the screw on the cable retention
device to ensure the cable can not be pulled out.
Step 2
Step 3
Turn the power switch to the on (1) position on the power supply.
Step 4
Verify power supply operation by checking that the power supply LEDs are in the
following states:
AB-13
ICSNS v3.02-13
AB-14
ICSNS v3.02-14
Turn the power switch on the power supply to the off (0) position. There is an
internal-lock mechanism that prevents you from removing the power supply if it is
not set to the off position.
Step 2
Step 3
Loosen the screw on the cable retention device, and disconnect the power cable from
the power supply.
Step 4
Loosen all four panel fasteners at the corners of the power supply.
Step 5
Grasp the power supply handles and slide the power supply partially out of the
chassis, about 4 to 5 inches.
Step 6
If the power supply is at your waist or chest level, place your other hand underneath
the power supply and slide the power supply completely out of the chassis.
Note
Step 7
To avoid damage to the panel fasteners, do not place the power supply down on the
perforated ends.
Install a filler panel over the opening. Tighten the captive screws if the power supply
bay is to remain empty.
AB-15
ICSNS v3.02-15
Remove the blank power-supply filler plate from the chassis power-supply bay
opening by loosening the captive installation screw, if necessary.
Step 2
Turn the power switch to the off (0) position on the power supply you are installing.
Step 3
Grasp the power supply handle with one hand. Place your other hand underneath the
power supply, as shown in the figure. Slide the power supply into the power supply
bay. Make sure that the power supply is completely seated in the bay.
Step 4
Step 5
Plug in the power cord to the power supply and tighten the screw on the cable
retention device.
Step 6
Turn the power switch to the on (1) position on the power supply you are installing.
Step 7
AB-16
Step 1
Remove the blank power-supply filler plate from the chassis power-supply bay
opening by loosening the captive installation screw, if necessary.
Step 2
Turn the power switch to the off (0) position on the power supply that you are
installing.
Step 3
Grasp the power supply handle with one hand. Place your other hand underneath the
power supply, as shown in the figure. Slide the power supply into the power supply
bay. Make sure that the power supply is fully seated in the bay.
Step 4
Step 5
Remove two screws securing the terminal block cover. Slide the cover off the
terminal block.
Step 6
Attach appropriate lugs to the DC-input wires. The maximum width of a lug is 0.300
inch (7.6 cm). The wire should be sized according to local and national installation
requirements.
Note
Step 7
Connect DC-input wires to the terminal block in the following order: (1) ground, (2)
negative (-), (3) positive (+).
Step 8
Turn the power switch to the off (0) position on the power supply that is being
installed.
Step 9
Note
For redundant or combined power requirements, the number and type of line cards and
supervisor modules determine the amount of power needed by the chassis. If each power
supply in the chassis is capable of supplying the total chassis power, then the power
supplies can be redundant. If each power supply is unable to supply the total chassis power,
then the power supplies are shared, and the loss of one supply results in some of the cards
being inoperable.
AB-17
The MDS 9500 series supports redundant hot swappable power supplies that support AC or DC
input voltages. Each power supply is capable of supplying sufficient power to the entire chassis
should one fail. The power supplies monitor their output voltage and provide status to the
supervisor module. To prevent the unexpected shutdown of an optional module, the power
management software only allows a module to power up if adequate power is available.
The power supplies can be configured to be redundant or combined. By default, they are
configured as redundant so that if one fails, the remaining power supply can still power the
entire system. Condition LEDs give visual indications of the installed modules and their
operation.
AB-18
1900 W AC or DC
1. Power supply LEDs
2. Captive screws
12 A maximum
Output:
1900 W at 200 to 240 VAC
1050 W at 100 to 120 VAC
ICSNS v3.02-17
As with the MDS 9509 power supplies, the MDS 9506 Director supports redundant AC hotswappable power supplies, each of which is capable of supplying sufficient power to the entire
chassis should one power supply fail. The power supplies monitor their output voltage and
provide status to the supervisor module. Also, they can be configured to be redundant or
combined. By default, they are configured as redundant. Condition LEDs are also available on
the power supply modules.
The 1900-watt supply provides full output capabilities when powered by 220 VAC; however,
that output is reduced to 1050 watts when powered by a 110 VAC input. It has a current rating
of 15 amps, but a maximum draw of 12 amps under normal conditions.
AB-19
1. Captive screws
2. PEM switch
3. AC power connection
ICSNS v3.02-18
AB-20
Step 2
Step 3
Loosen the captive screws on the DC PEM. Pull the PEM part way out of the chassis
to provide access to the PEM terminal block screws.
Step 4
The process for connecting the positive and negative DC cables to the DC PEM with
a 10-32 ring lug for each cable is as follows:
1. Identify the positive and negative DC cables and ensure that both are copper and sized
according to local and national installation requirements.
2. Strip the cable ends to allow for metal-to-metal contact. Insert each cable into a separate
ring lug. Crimp the lugs around the cables.
3. Insert each cable and lug into the appropriate hole in the front of the PEM. Fasten the lugs
to the appropriate terminal block screws in the following order: (1) negative (-), (2) positive
(+).
4. Secure the cables in place by tightening the terminal block screws.
AB-21
+5.0V at 16.2 A
ICSNS v3.02-19
The Cisco MDS 9216 switch supports dual hot swappable 845-watt AC power supplies. Each
supply is autoranging on the input voltage and can provide sufficient power to the entire chassis
should one of them fail. They also monitor their own output voltage and provide status to the
systems supervisor module. MDS 9216 power supplies can be configured to be redundant or
combined. By default, they are configured as redundant, so that if one fails, the remaining
power supply can still power the entire system.
The MDS 9216 power supplies are field-replaceable units (FRUs), are installed, and can be
removed easily from the rear of the chassis using pull handles. They also provide condition
LEDs for operational status.
The MDS 9216 supports AC voltage inputs only, not DC, ranging from 100 to 240 VAC. The
power supplies have a current rating of 15 amps for circuit breakers but draw a maximum of 12
amps at 110 VAC and only 5 amps at 220 VAC.
AB-22
Step 1
Ensure that the system (earth) ground connection has been made.
Step 2
If the power-supply bay has a filler panel, loosen the screws holding it on and
remove the panel.
Step 3
Verify that the power switch is in the off (0) position on the power supply you are
installing.
Step 4
Orient the power supply as shown in the figure. Hold it by the handle and slide the
power supply into the chassis power supply bay.
Step 5
Ensure that the power supply is completely seated in the bay. Tighten the power
supply captive screws.
Step 6
Plug the power cable into the power supply. Tighten the screw on the power cable
retainer to ensure that the cable can not be pulled out.
Step 7
Step 8
Turn the power switch to the on (1) position on the power supply.
Step 9
Verify power supply operation by checking that the power supply LEDs are in the
following states:
Note
In a system with dual power supplies, connect each power supply to a separate power
source. In case of a power source failure, the second source will most likely still be
available.
AB-23
ICSNS v3.02-20
Ensure that the system (earth) ground connection has been made.
Step 2
Make sure the power cord is disconnected before installing the power supply.
Step 3
Verify that the power switch is in the off (0) position on the power supply you are
installing.
Step 4
Slide the power supply into the power supply bay. Make sure that the power supply
is completely seated in the bay.
Step 5
Step 6
Step 7
Connect the other end of the power cord to an AC input power source.
Note
AB-24
Depending on the outlet receptacle on your power distribution unit, you might need the
optional jumper power cord to connect the Cisco MDS 9216 switch to your outlet receptacle.
Step 8
Turn the power switch to the on (1) position on the power supply.
Step 9
Verify the power supplys operation by checking that the power supply (P/S) LED in
the front panel is green.
Combined mode:
Is nonredundant
Twice the power capacity of the lower-capacity supply
Sufficient power might not be available in case of a power supply failure
System reset if power requirements exceed capacity
Only modules with sufficient power are powered up
If no reset, no modules down but no new modules up
Should not be used for director-class switches
ICSNS v3.02-21
Power supplies are configured in redundant mode by default, but they can also be configured in
a combined, or nonredundant, mode:
Redundant mode: The chassis uses the power capacity of the lower-capacity power supply
so that sufficient power is available in case of a single power supply failure.
Combined mode: The chassis uses twice the power capacity of the lower-capacity power
supply. Sufficient power might not be available in case of a power supply failure in this
mode. If there is a power supply failure and the real power requirements for the chassis
exceed the power capacity of the remaining power supply, the entire system is reset
automatically to prevent permanent damage to the power supply.
In both modes, power is reserved for the supervisor and fan assemblies. Each supervisor
module has roughly 220 watts in reserve, even if there is only one installed; and the fan module
has 210 watts in reserve. In the case of insufficient power, after supervisors and fans are
powered, line card modules are given power from the top of the chassis down.
After the reboot, only those modules that have sufficient power are powered up.
If the real power requirements do not trigger an automatic reset, no module is powered down.
Instead, no new module is powered up.
In all cases of power supply failure or removal, the following occur:
Note
AB-25
Air Flow
ICSNS v3.02-23
The MDS 9500 Series supports hot swappable fan modules that are easily installed or removed
from the from of the chassis. They provide 85 cfm of airflow per slot with 410 watts of power
dissipation per slot. The MDS 9506 has a fan module with 6 fans, the Cisco MDS 9509 has a
fan module with 9 fans, and the Cisco MDS 9513 has a fan module with 15 fans.
Sensors on the supervisor module monitor the internal air temperature. If the air temperature
exceeds a preset threshold, the environmental monitor displays warning messages. If one or
more fans within the module fails, the Fan Status LED turns red, and the module must be
replaced. When all fans are operating properly, the LED is green. If the fan LED is red, the fan
assembly might not be seated properly in the chassis. If this happens, remove the fan assembly
and reinstall. After reinstalling, if the LED is still red, then a failure on the fan assembly has
occurred. Fan LED status indication is provided on a per-module basis. If one fan fails, then the
module is considered failed.
The switch can continue to run when the fan module is removed for a maximum of 5 minutes if
the temperature thresholds are not exceeded. In this way, you can swap out a fan module
without having to bring the system down. The fan module is designed to be removed and
replaced while the system is operating without presenting an electrical hazard or damage to the
system, provided the replacement is performed promptly. Install the fan module in the front
chassis cavity with the status LED at the top. Push the fan module to ensure that the power
supply connector mates with the chassis. Tighten the captive installation screws. If the switch is
powered on, listen for the fans. You should hear them operating immediately.
No automated shutdown sequence is associated with the removal of the crossbar module fan
tray. Shutdown is initiated when temperature thresholds for the crossbar modules are exceeded.
Replacement of the crossbar module fan tray should be performed promptly.
AB-26
MDS 9513
2007 Cisco Systems, Inc. All rights reserved.
ICSNS v3.02-24
Hold the fan module so that the Fan Status LED is at the top.
Step 2
Place the fan module in the front chassis cavity so that it rests on the chassis. Lift the
fan module up slightly to align the top and bottom chassis guides.
Step 3
Push in the fan module to the chassis until it seats in the backplane and the captive
screws make contact with the chassis. The fan module snaps in.
Step 4
If the switch is powered on, listen for the fans. You should hear them operating
immediately.
Note
Step 5
If you do not hear the fans, ensure that the fan module is inserted completely in the chassis
and the outside surface is flush with the outside surface of the chassis.
Verify that the Fan Status LED is green. If the LED is not green, one or more fans
are faulty.
Push the button on the top fan-module latch to release the fan module from the
midplane.
Step 2
Step 3
Grasp the fan module with both hands and pull it outward. Rock it gently, if
necessary, to unseat the power connector from the backplane.
Step 4
AB-27
AB-28
If the fan module is removed from the chassis, a fan-removal sequence is started for the
following 3 minutes.
If the fan module is still not reinstalled within that 3-minute time frame, a system-shutdown
sequence is started for a period of 2 minutes. At the end of this 2-minute time frame, the
system is shut down.
If the fan module is reinstalled at any time in the total 5-minute sequence, the shutdown
sequences are stopped, and the system remains on.
ICSNS v3.02-25
Orient the crossbar module fan tray in the chassis by positioning the module in the
slot, and then sliding the module carefully into the slot until the fan tray is
completely inserted in the chassis.
Step 2
Tighten the two captive screws on the crossbar module fan tray to 8 in-lb screws.
Step 2
Hold the two captive screws and pull the fan tray out of the chassis with both hands.
Step 3
Take one hand and hold the face of the fan tray while supporting it with the other
hand.
Step 4
AB-29
1. Slot guides
2, 3. EMI gaskets
4. Ejector level (fully extended)
2007 Cisco Systems, Inc. All rights reserved.
ICSNS v3.02-27
In a Cisco MDS 9513 Director, slots 7 and 8 are reserved for the Supervisor-2 modules. In the
Cisco MDS 9506 and 9509 Directors, slots 5 and 6 are reserved for the supervisor modules. A
supervisor module should be installed before installing any switching modules.
AB-30
Step 1
Before installing any modules in the chassis, it is recommended that you install the
chassis in the rack.
Step 2
Step 3
Verify that the captive screws are tightened to 8 in-lb on all modules already
installed in the chassis. This ensures that the EMI gaskets are fully compressed and
maximizes the opening space for the module being installed.
Step 4
If a filler panel is installed, remove the two Phillips pan-head screws from the filler
panel and remove the panel.
Step 5
Slide the module carefully into the slot until the EMI gasket along the top edge of
the module contacts the module in the slot above it and both ejector levers close to
approximately 45 degrees with respect to the front of the module.
Step 2
Grasp the two ejector levers with the thumb and forefinger of each hand, and then
press down to create a small 0.040-inch (1-mm) gap between the module's EMI
gasket and the module above it.
Step 3
While pressing down, simultaneously close the left and right ejector levers to
completely seat the supervisor module or switching module in the backplane
connector. The ejector levers are completely closed when they are flush with the
front of the module.
Step 4
Tighten the two captive screws on the supervisor module or switching module to 8
in-lb.
AB-31
ICSNS v3.02-28
Note
Be sure the CompactFlash disk is seated properly on the supervisor module before installing
in the chassis.
Step 1
Connect the management port to the LAN using a Category 5 unshielded twistedpair (UTP) cable.
Step 2
Connect the supplied RJ-45 to DB-9 female adaptor to the computer serial port.
Note
Step 3
Then connect the console cable (a rollover RJ-45 to RJ-45 cable) to the console port
and to the RJ-45 to DB-9 adapter at the computer serial port. Configure the terminal
emulator program to match the following default port characteristics:
9600 baud
8 data bits
1 stop bit
No parity
Note
AB-32
It is recommend to use the adaptor and cable provided with the switch.
ICSNS v3.02-29
Ensure that the configured domain ID is the same as the current domain ID for every
VSAN on the switch.
Determine which Supervisor-1 module is standby with the show module command.
Take the standby Supervisor-1 module out of service with the out-of-service module
command.
Remove the standby Supervisor-1 module, and install the Supervisor-2 module in the
chassis.
Verify that the standby Supervisor-2 module is in the warm standby state using the show
system redundancy status command.
AB-33
AB-34
If necessary, copy the contents of the SSM NVRAM to the standby Supervisor-2 module.
Initiate a switchover on the active Supervisor-1 module to power it down and cause the
standby Supervisor-2 module to become the active supervisor module with the system
switchover command.
Run the install all command to update the image versions and boot variables.
ICSNS v3.02-30
This supervisor module is installed in the MDS 9500 Series chassis and has the following
interfaces:
Console port: RS-232 (RJ-45) for local command line interface (CLI) management.
10/100 Ethernet interface: Out-of-band (OOB) management access with integrated link and
activity LEDs.
COM1 serial port: DB-9 interface is an RS-232 port that you can use to connect to an
external serial communication device such as a modem.
CompactFlash LED: This LED is lit when a CompactFlash (CF) card is installed into slot 0.
Green: OK
Green: System OK
Green: Active
Orange: Standby
AB-35
Green: Good power, that is, sufficient power for all modules
Orange: Not enough power, that is, insufficient power for all modules
Connect the modem to the COM1 serial port with the adaptors and cables provided with the
accessory kit as follows:
AB-36
Step 1
Step 2
Step 3
Connect the adapters using the RJ-45 to RJ-45 rollover cable (or equivalent
crossover cable).
Fixed
interface
module
Connect to terminal
Connect VT100
terminal to the
console connector
Connect to Ethernet LAN
2007 Cisco Systems, Inc. All rights reserved.
ICSNS v3.02-31
Slot 1: This slot is reserved for the supervisor module with its integrated 16-port switching
module.
Slot 2: This slot can contain an optional 16- or 32-port switching module or a services
module such as an 8-port IP Storage Services (IPS) module.
AB-37
ICSNS v3.02-32
AB-38
Resetting modules should only be performed when necessary, for example if a software
upgrade or downgrade has failed.
Module Shutdown
Use the poweroff module command to power down a module
Verify status with the show module command
Remove module safely without shutting down entire switch
# conf t
(config)#
(config)#
Mod Ports
--- ----1
16
2
4
5
0
6
0
poweroff module 1
do show module
Module-Type
-------------------------1/2 Gbps FC Module
IP Storage Services Module
Supervisor/Fabric-1
Supervisor/Fabric-1
Caution
Model
--------------DS-X9016
DS-X9530-SF1-K9
DS-X9530-SF1-K9
Status
---------ok
powered-dn
active *
ha-standby
ICSNS v3.02-33
Even though you can hot-swap MDS 9000 modules, it is recommended that you shut down
a module before removal.
To shut down any module, use the poweroff module command in config mode:
<config># poweroff module 2
To verify the status of a module at any time, use the show module command in EXEC mode.
To view information on one module only, you can specify a module slot number.
Example
The command show module 1 returns the status information of only the module installed in slot
1.
# show module
Mod Ports Module-Type
Model
Status
--- ----- -------------------------- --------------- ---------1 16 1/2 Gbps FC Module
DS-X9016
ok
2 4
IP Storage Services Module
powered-dn
5 0
Supervisor/Fabric-1
DS-X9530-SF1-K9 active *
6 0
Supervisor/Fabric-1
DS-X9530-SF1-K9 ha-standby
AB-39
Summary
This topic summarizes the key points that were discussed in this lesson.
Summary
Installation guidelines include recommendations for evaluating
site preparedness, rack hardware, and power requirements.
MDS 9000 switches can be installed in a standard telco rack or
cabinets with solid panels. The MDS 9513 can not be used with a
telco rack.
Power supply installation and configuration should be carefully
considered to ensure high-availability.
MDS 9000 fan modules are hot swappable and provide for easy
installation and replacement.
Install at least one supervisor module before installing any line
card modules, and use the poweroff module command to shut
down individual line cards prior to removal.
AB-40
ICSNS v3.02-34