Introduction to MDS

and Storage Area

Networks (SAN)

2009 Cisco Systems - CAE.

Storage
Storage terms
Host Bus Adapter
Disks, JBODs, RAID, storage arrays

Storage protocols
SCSI
Storage transport protocols
Fibre Channel (FC), iSCSI, TCP/IP

Storage topologies
DAS, NAS, and SAN

2008 Cisco Systems - CAE.

Storage Terms
Host Bus Adapter (HBA)
Storage I/O devices
Small Computer Serial Interface (SCSI)
Converged Network Adapter (CNA FCoE)

2008 Cisco Systems - CAE.

Storage Terms
Host Bus Adapter (HBA)
Storage subsystem
Just a bunch of disks (JBOD)
Redundant array of independent disks (RAID)
Storage arrays

2008 Cisco Systems - CAE.

Storage Terms
Parallel SCSI copper interconnects
Optical direct connect
Fibre Channel switch

2008 Cisco Systems - CAE.

Storage Terms
I/O DevicesDisk Drives
Fundamental unit for data storage
Disk drive types
Parallel SCSI
Fibre channel
Advanced Technology
Attachment (ATA) or Integrated
Drive Electronics (IDE)
Serial ATA (SATA)

2008 Cisco Systems - CAE.

Storage Terms
I/O DevicesJBODs
iSCSI

Server
with SCSI

TRGT 3

TRGT 1

SCSI CABLE

SCSI BUS
TRGT 0

TRGT 2

TRGT 4

SCSIJBOD

TRGT 5

TRGT 4

iSCSI
SCSI CABLE

Server
with Fibre Channel
Arbitrated Loop
(FC-AL)
2008 Cisco Systems - CAE.

TRGT 3

FC-AL

TRGT 0

TRGT 1

TRGT 2

FC-ALJBOD
7

Storage Topologies

2009 Cisco Systems - CAE.

Storage Topologies
Direct Attached Storage
Clients

Direct Attached
Storage (DAS)
Storage is captive behind the
server, limited mobility
Limited scalability due
to limited devices
No storage sharing possible

LAN

Win2k Linux

Application
Servers

UNIX Win2k Linux UNIX

SCSI

FC

FC

Costly to scale;
complex to manage
Tape

Direct Attached
Storage
2008 Cisco Systems - CAE.

Storage Topologies
Network Attached Storage
Storage is accessed at a
file level via NFS or CIFS
Storage is accessed
over an IP network
Storage devices can
be shared between
servers files can be
shared between users
TCP can be tuned to optimize
for storage transport

LAN

Win2
k

Linux Unix

Application
Servers

Generic

Generic

NAS Appliances
or
NAS Head Ends

NAS = Optimized for FILE I/O

2008 Cisco Systems - CAE.

10

Storage Topologies
Storage Area Networks

Separation of Storage from the Server

Clients

Storage is accessed at
a block-level via SCSI
LAN

High performance interconnect

providing
high I/O throughput
Lower TCO relative to
direct attached storage, storage
can be shared
Limited vendor interoperability
and data access
Complex management

Database
Servers
Fibre
Channel
SAN
Block
Storage
Devices

Storage Area Network (SAN)

2008 Cisco Systems - CAE.

11

Data Center Landscape

LAN

SAN A

SAN B

InfiniBand
Access
Layer

Ethernet
Fibre Channel
High Performance
Computing

High Performance
Computing
2008 Cisco Systems - CAE.

Data Center
LAN

Data Center
SAN
12

Server Requirements

FC HBA
FC HBA
GE

Parallel LAN SAN infrastructure

Inefficient use of the network
infrastructure
5+ connections per server and higher
adapter and cabling costs:

GE

Adds downstream port costs;

CapEx and OpEx.

GE

Each connection adds additional points of failure in

the fabric

GE
10/100

Longer lead time for server provisioning

Multiple fault domains: Complex
diagnostics
Management complexity: Firmware,
driver patching, versioning

2008 Cisco Systems - CAE.

13

I/O Consolidation
Today
LAN

SAN A

I/O Consolidation with FCoE

SAN B

Enhanced Ethernet and FCoE

2008 Cisco Systems - CAE.

LAN

Ethernet

SAN A

SAN B

FC
14

Fibre Channel over Ethernet

10 Gb/s Ethernet
Lossless Ethernet:
Matches the lossless behavior guaranteed in Fibre Channel

Ethernet jumbo frames:

Max Fibre Channel frame = 2112 bytes
Normal Ethernet frame, Ethertype = FCoE

FCS

EOF

FC Payload

CRC

FC
Header

FCoE
Header

Ethernet
Header

Same as a physical Fibre Channel frame

Control information: Version, ordered sets (SOF, EOF)

2008 Cisco Systems - CAE.

15

Network Stack Comparison

SCSI

SCSI

SCSI

SCSI

SCSI

iSCSI

FCP

FCP

FCP

FC

FC

FC
Less Overhead
than FCIP, iSCSI

FCIP
TCP

TCP

IP

IP

FCoE

Ethernet

Ethernet

Ethernet

PHYSICAL WIRE
SCSI
2008 Cisco Systems - CAE.

iSCSI

FCIP

FCoE

FC
16

Data Center Ethernet Enhancements

A collection of IEEEbased enhancements to
classical Ethernet that
provide end-to-end QoS
Does not disrupt existing
infrastructure

Ethernet enhancements:
Priority groups: Virtualizes links and allocates
resources per traffic classes
Priority flow control by traffic class
End-to-end congestion management and
notification
Shortest-path bridging: Layer 2 multipathing

Benefits of Ethernet enhancements:

Eliminates transient and persistent congestion
Lossless fabric: No-drop storage links
Deterministic latency for HPC clusters
Enables a converged Ethernet fabric for
reduced cost and complexity

2008 Cisco Systems - CAE.

17

Storage Protocols

2009 Cisco Systems - CAE.

18

Storage Protocols
Small Computer System InterfaceSCSI
SCSI is a STANDARD that defines an interface
between an initiator (usually a computer) and
a target (usually a storage device such as a
hard disk)
INTERFACE refers to connectors, cables, electrical
signals, optical signals, and the command protocol that
allow initiators and targets to communicate

2008 Cisco Systems - CAE.

19

Storage Protocols
Small Computer System InterfaceSCSI
SCSI command protocol is the de facto standard that is
used extensively in high-performance storage
applications
The command part of SCSI can be:
Transported over a Fibre Channel storage area network
Encapsulated in IP and carried across IP networks

2008 Cisco Systems - CAE.

20

Storage Protocols
Storage Transport Protocols
Protocols used to transport SCSI
Fibre Channel
FCIP
FCoE
iSCSI

2008 Cisco Systems - CAE.

21

Storage Protocol
Fibre Channel Protocol (FCP) Essentials
FCP defines how SCSI-3 commands are transported
over a fibre channel network
Transport involves more than encapsulation
Individual SCSI commands are mapped to FC
sequences
The set of SCSI commands that form a SCSI transaction
are mapped to a FC exchange
Fibre Channel preserves the frame order

2008 Cisco Systems - CAE.

22

Storage Protocols
Storage ProtocolsiSCSI
SCSI data converted into an iSCSI by adding
a special header
iSCSI data is encapsulated into an IP packet
IP is the transport protocol
IP
TCP
iSCSI
SCSI
Data

2008 Cisco Systems - CAE.

23

Storage Protocols
Storage ProtocolsiSCSI
SCSI Applications (File Systems, Databases)
SCSI Block Commands

SCSI Stream
Commands

Other SCSI Commands

SCSI Commands, Data, and Status

Parallel SCSI
Interfaces

Fibre Channel

iSCSI
SCSI over IP
TCP
IP

Parallel SCSI
Interfaces
2008 Cisco Systems - CAE.

Fibre Channel

Ethernet

24

Storage Protocols
Storage ProtocolsFCIP
FCIP is a mechanism that allows SAN islands
to be interconnected over IP networks
Each interconnection is called a FCIP link and
can contain one (1) or more TCP connection(s)
Each end of a FCIP link is associated to a
Virtual E_Port (VE_Port)
VE_Ports communicate between themselves
just like normally interconnected E_Ports by
using SW_ILS: ELP, ESC, BF, RCF, FSPF, etc.

2008 Cisco Systems - CAE.

25

Storage Protocols
Storage ProtocolsFCIP
The FCIP Link carries encapsulated Fibre Channel
traffic between Link End Points (LEPs) over an IP
network by using TCP on port 3225
The result is a virtual ISL between VE_Ports
FC frame not changed
IP
TCP
FCIP
FC
SCSI
Data
2008 Cisco Systems - CAE.

26

Storage Protocols
Storage ProtocolsFCIP
Switch A
F
FC

Switch C

FC

IP

GE VE

FC

GE

E
FC

Switch B
VE GE
FC

F
FC

2008 Cisco Systems - CAE.

FCIP Link
Virtual ISL
FC = Fibre Channel
F
= F_Port
E
= E_Port
VE = Virtual E_Port
GE = Gigabit Ethernet
27

Brief Fibre Channel

Overview

2009 Cisco Systems - CAE.

28

Fibre Channel Communications Model

Fibre Channel Has Many Similarities to IP (TCP)
Point to point orientedfacilitated through device login
Similar to TCP session establishment

N_Port to N_Port connectionlogical node connection point

Similar to TCP/UDP sockets

Flow Controlledhop-by-hop and end-to-end basis

Similar to TCP flow controlDifferent mechanism (no drops)

Acknowledgedfor certain classes of traffic, none for others

Similar to TCP/UDP acknowledgement models

Multiple connections allowed per device

Similar to multiple TCP/UDP sockets

Node
N_Port

Node

Transmitter

Transmitter

Receiver

Receiver
Link

2008 Cisco Systems - CAE.

N_Port

29

Fibre Channel Port Types

N port

NL

FL

Node ports used to connect devices to switched fabric or point-to-point

configurations

F port
Fabric ports residing on switches connecting N port devices

L port
Loop ports are used in arbitrated loop configurations to build networks
without FC switches; these ports often also have N port capabilities
and are called NL ports

E port
Expansion ports are essentially trunk ports used to connect two Fibre
Channel switches

GL port
A generic port capable of operating as either an E or F port; its also
capable of acting in an FL port capacity; auto discovery

2008 Cisco Systems - CAE.

30

Inter-Switch Link (ISL)

The interconnection between switches is called the ISL
E_Port to E_Port (Expansion port)

Supports all classes of service

Class 1, 2, 3, and a special Class F (switch-to-switch)

FC-PH permits consecutive frames of a sequence to be routed

over different ISL links for maximum throughput
Ciscos implementation is to dedicate an FC_ID pair and/or a
given exchange to an ISL bundle member to guarantee in-order
delivery for exchange/sequence frames
Cisco Extended ISL (EISL, TE port)

2008 Cisco Systems - CAE.

31

Example of a Fabric Switch

Fabric Switch
Node
NL_Port

Node
NL_Port

Node
NL_Port

2008 Cisco Systems - CAE.

FL_Port

E_Port

E_Port

GL_Port

F_Port

N_Port

Node

GL_Port

F_Port

N_Port

Node

GL_Port

F_Port

N_Port

Node

32

FC_ID Address Model

FC_ID address models help speed up routing
Switches assign FC_ID addresses to N_Ports
Some addresses are reserved for fabric services
Private loop devices only understand 8-bit address (0x0000xx)
TL_Port can provide proxy service for private-to-public address translation
Maximum switch domains = 239 (based on standard)

Switch Topology
Model

8 bits

8 bits

8 bits

Switch
Domain

Area

Device

00

Arbitrated Loop
Physical Address
(AL_PA)

Area

Arbitrated Loop
Physical Address
(AL_PA)

Private Loop Device

Address Model

00

Public Loop Device

Address Model

Switch
Domain

2008 Cisco Systems - CAE.

33

Virtual SANs (VSANs)

A VSAN Provides a Method to Allocate
Ports within a Physical Fabric to Create
Virtual Fabrics

SAN Switches Enabled with

VSAN Services

Analogous to VLANs in Ethernet

Virtual fabrics created from larger costeffective redundant physical fabric
Reduces wasted ports of island approach
Fabric events are isolated per VSAN
maintains isolation for HA

Physical SAN
Islands Are
VIRTUALIZED onto
Common SAN
Infrastructure

Hardware-based isolationtraffic is
explicitly tagged across inter-switch links
with VSAN membership info
Statistics can be gathered per VSAN

2008 Cisco Systems - CAE.

34

VSANs and Zones: Complimentary

VSANs and Fabric Zoning Are Very Complimentary
Hierarchical relationship
First assign physical ports to VSANs
Then configure independent zones per VSAN

VSANs divide the physical infrastructure

Zones provide added security and allow sharing
of device ports

Relationship of VSANs to Zones

Physical Topology
VSAN 2
Disk2
ZoneA

Host1

Disk3
Disk1

Disk4

Host2

ZoneC

VSANs provide traffic statistics

VSANs only changed when ports needed per
virtual fabric

ZoneB

Zones can change frequently

(eg backup)

VSAN 3 ZoneD

Ports are added/removed non-disruptively to

VSANs

Host4
ZoneA
Disk5
Host3
Disk6

2008 Cisco Systems - CAE.

35

ISL Port Channeling

A Port Channel Is a Logical Bundling of
Identical Links
Criteria for forming a Port Channel
Same speed links
Same modes (auto, E, etc) and states
Between same two switches
Same VSAN membership

Treated as one logical ISL by upper layer protocols (FSPF)

Can use up to 16 links in a Port Channel (32 Gbps max)

8 Gbps
Port Channel

Can be formed from any ports on any modulesHA enabled

Exchange-based in-order load balancing
Mode 1: based on src/dst FC_IDs
Mode 2: based on src/dst FC_ID/OX_ID/RX_ID

Much faster recovery than FSPF-based balancing

Given logical interface name with aggregated bandwidth and derived
routing metric

2008 Cisco Systems - CAE.

36

VSAN-Based EISLs and TE_Ports

The VSANs Feature Introduces
Two New SAN Elements:

Cisco MDS 9500

Director with
VSAN Service

The Trunking E_Port (TE_Port)

Negotiated between MDS 9000 switchesdefault
Carries tagged frames from multiple VSANs
Can be optionally disabled to yield E_Port
Only understood by Cisco MDS 9000 switches
Also has a native VSAN assignment (for E_Port)
Trunk all VSANs (1-4093) by default
Also trunks VSAN 4094 used for ELP only (special)
Not to be confused with port aggregation (trunking)

The Enhanced ISL (EISL) link

The resultant link created by two connected TE_Ports
Superset of ISL functionality
Carry individual control protocol information per VSAN (e.g.
zoning updates)
Can be extended over distance (DWDM, FCIP, etc)

2008 Cisco Systems - CAE.

Enhanced ISL
(EISL) Trunk
Carries Tagged
Traffic From
Multiple VSANs

Trunking
E_Port
(TE_Port)

Trunking
E_Port
(TE_Port)
Trunking
E_Port
(TE_Port)

Cisco MDS
9216
Fabric With
VSAN Service

Enhanced ISL
(EISL) Trunk
Carries Tagged
Traffic from
Multiple VSANs
Notice: Blue VSAN
Doesnt Have to
Reside on Switch for It
to Traverse Switch
37

VSAN Number Space

Configured
VSANs

VSAN Numbering Rules

VSAN 10

VSAN 1 is the default VSAN

All ports are originally in VSAN1

VSAN 2 through 4093 can be assigned to

user VSANsVSAN 0, 4094, 4095 are
reserved
A maximum of 256 VSANs can be created from the
range of 2-4093

VSAN 4094 is a reserved

special VSAN

Cisco MDS 9000

Family with VSAN
Service
Enhanced ISL
(EISL) Trunk
Carries Tagged
Traffic from Multiple
VSANs

Called the isolated VSAN

Used to isolate ports whos port-VSAN has been
deleted

Port is in VSAN 4094

(Isolated VSAN)

Not propagated across switches

Always present, cant be deleted

2008 Cisco Systems - CAE.

Host Is Isolated
From the
Fabric

VSAN 20
VSAN 30
Trunking
E_Port
(TE_Port)
VSAN 30 Is Not
Propagated Across
EISL Due to
Nonexistence on
Remote Switch
Trunking
E_Port
(TE_Port)
VSAN 10
VSAN 20
VSAN 30

Configured
VSANs
38

VSANs and Domain IDs

Recall: Each VSAN Acts as a
Completely Independent Fabric
Each VSAN has its own principle switch
and domain_ID allocation policy (static or
dynamic)
Principle switches for different VSANs dont have to
reside on same physical switch

Each switch will have a separate

domain_ID for each active VSAN
These domain_IDs can overlap
between VSANs
All ports are originally in VSAN1

Domain 100 Domain 105

Domain 200 Domain 223

Domain Domain 153

126
Domain 173

Domain 110
Domain 153

Domain 112
Domain 171

Domain 156
Domain 102

Domain 113
Domain 180

Each VSAN can have a separate FC_ID

allocation policy (static or dynamic)
Domain 104
Domain Domain 170 Domain 201
Domain 204
Domain 215 Domain 162
157
Each VSAN can have its own fabric
Each Switch That Has End Ports in a Particular
timers (RA_TOV, ED_TOV)
VSAN Will Have a Domain_id Assigned to That
Particular VSAN; Core Switches That Trunk
These VSANs Will Also Have Assigned
Domain_ids in These VSANs

2008 Cisco Systems - CAE.

39

VSAN Best Practices

Use VSANs to isolate each application
Use IVR to allow resource sharing across VSANs
Place unused ports in VSAN 4094
Suspend VSAN 1:
Do not configure zones in VSAN 1
Set the default zone policy to deny
This prevents WWN spoofing on an unused port

2008 Cisco Systems - CAE.

40

VSANs and Non-Cisco Switches

The VSANs Feature Involves a Tagging Mechanism Which
Is Not Understood by 3rd Party Fabrics
Cisco Interoperability Mode (required)
is configured per-VSANno loss of
functionality in MDS 9000 switches
Cisco MDS 9000 switches
negotiate an E_Port with
non-Cisco switches
Cisco MDS 9000 E_Ports also
have a port VSAN
Therefore, the entire non-Cisco switch,
including all its ports, will reside in the
port VSAN of the connecting E_Port

Enhanced
ISL (EISL)
Trunks
Carrying
Numerous
VSANs

Simple
ISL
Links
E_Ports
Non-Cisco
Fabric
Switches
Each Non-Cisco Switch
Belongs to Only One VSAN

2008 Cisco Systems - CAE.

41

Flexible FSPF Routing

Fabric Shortest Path
First (FSPF)

Domain 100

0
25

Port Channels
Have Metric
Based on
Aggregated
Domain Bandwidth

22
5

10
0

50

112

15
0

75

Provides routing services within

a fabric
Routes between domainsseparate
Domain
calculations in both directions
110
Using summarized metric to weight multiple
paths to a domain
Provides load balancing between equal-cost Domain
156
paths
Recovery time from failed paths based on set
of FSPF timers
Outlined in ANSI T11 FC-SW-2 standard

20
0

Domain
113

Similar to IP OSPF routing

protocol with one area

2008 Cisco Systems - CAE.

20

Domain 104
Red PathMetric500 (Chosen)Load
Balanced
Blue PathMetric525
Purple PathMetric600
Yellow PathMetric500 (Chosen)
Load Balanced
42

Potential FCIP Environments

Wire-rate (1Gbps)
Relatively low latency
Sync or async
replication
Metro Ethernet offers
cost effective solution

Local
Datacenter

Remote
Datacenter
FCIP

Metro
Ethernet

FCIP

Short Distance ~ <= 60km

Typical OC-3/OC-12
Relatively low latency
Mainly asynchronous
Suitable for some
synchronous apps

Local
Datacenter

Remote
Datacenter
FCIP

SONET

FCIP

Medium Distance ~ <= 160km

Low speed (T1DS3)
Higher latency
Longer distance
Mainly asynchronous

Local
Datacenter

Remote
Datacenter
FCIP

IP Routed
WAN

FCIP

Long Distance > 160km

2008 Cisco Systems - CAE.

43