Professional Documents
Culture Documents
Technical Brief:
Cascaded FICON in a Brocade Environment
Cascaded FICON introduces the open systems SAN concept of the Inter-Switch
Links (ISLs). IBM now supports the flow of traffic from the processor through
two FICON directors connected via an ISL and on to the peripheral devices
such as disk and tape. This paper discusses the benefits and some technical
aspects of cascaded FICON in a Brocade environment.
MAINFRAME
Technical Brief
CONTENTS
Technical Brief: Cascaded FICON in a Brocade Environment...........................................................................................................................................................1
Contents.................................................................................................................................................................................................................................................................2
Introduction...........................................................................................................................................................................................................................................................3
The Evolution from ESCON to FICON Cascading.....................................................................................................................................................................................4
What is Cascaded FICON?.......................................................................................................................................................4
High Availability (HA), Disaster Recovery (DR), and Business Continuity (BC)......................................................................5
Benefits of FICON Cascading .........................................................................................................................................................................................................................7
Optimizing Use of Storage Resources.....................................................................................................................................8
Cascaded FICON Performance................................................................................................................................................9
Buffer-to-Buffer Credit Management ..........................................................................................................................................................................................................9
About BB Credits ....................................................................................................................................................................10
Packet Flow and Credits ........................................................................................................................................................................... 10
Buffer-to-Buffer Flow Control.................................................................................................................................................................... 10
Summary ...........................................................................................................................................................................................................................................................38
Appendix: Fibre Channel Class 4 Class of Service (CoS)..................................................................................................................................................................39
2 of 40
MAINFRAME
Technical Brief
INTRODUCTION
Prior to the introduction of support for cascaded FICON director connectivity on IBM zSeries mainframes in
January 2003, only a single level of FICON directors was supported for connectivity between a processor
and peripheral devices. Cascaded FICON introduced the open systems Storage Area Network (SAN) concept
of the Inter-Switch Links (ISLs). IBM now supports the flow of traffic from the processor through two FICON
directors connected via an ISL to the peripheral devices, such as disk and tape.
This paper starts with a brief discussion of cascaded FICON, its applications, and the benefits of a cascaded
FICON architecture. The next section provides a technical discussion of bufferto-buffer credits (BB credits),
open exchanges, and performance. The final section describes management of a cascaded FICON
architecture, including ISL trunking and the Traffic Isolation capabilities unique to Brocade.
FICON, like most technological advancements, evolved from the limitations of its predecessorthe IBM
Enterprise System Connection (ESCON) protocola successful storage network protocol for mainframe
systems considered the parent of the modern SAN. IBM Fiber Connection (FICON) was initially developed to
address the limitations of the ESCON protocol. In particular, FICON addresses ESCON addressing,
bandwidth and distance limitations. FICON has evolved rapidly since the initial FICON bridge mode (FCV)
implementations came to the data center, from FCV to single director FICON Native (FC) implementations,
to configurations that intermix Fibre Channel (FC) and open systems Fibre Channel (FCP), and now to
cascaded fabrics of FICON directors. FICON support of cascaded directors is available, has been supported
on the IBM zSeries since 2003, and is supported on the System z processors as well.
Cascaded FICON allows a FICON Native (FC) channel or a FICON CTC channel to connect a zSeries/System z
server to another similar server or peripheral device such as disk, tape library, or printer via two Brocade
FICON directors or switches. A FICON channel in FICON Native mode connects one or more processor
images to an FC link, which connects to the first FICON director, then dynamically through the first director
to one or more ports, and from there to a second cascaded FICON director. From the second director there
are Fibre Channel links to FICON Control Unit (CU) ports on attached devices. These FICON directors can be
geographically separate, providing greater flexibility and fiber cost savings. All FICON directors connected
together in a cascaded FICON architecture must be from the same vendor (such as Brocade). Initial support
by IBM is limited to a single hop between cascaded FICON directors; however, the directors can be
configured in a hub-star architecture with up to 24 directors in the fabric.
NOTE: In this paper the term switch is used to reference a Brocade hardware platform (switch, director, or
backbone) unless otherwise indicated.
Cascaded FICON allows Brocade customers tremendous flexibility and the potential for fabric cost savings
in their FICON architectures. It is extremely important for business continuity/disaster recovery
implementations. Customers looking at these types of implementations can realize significant potential
savings in their fiber infrastructure costs and channel adapters by reducing the number of channels for
connecting two geographically separate sites with high availability FICON connectivity at increased
distances.
Brocade (via the acquisitions of CNT/Inrange and McDATA) has a long and distinguished history of working
closely with IBM in the mainframe environment. This history includes manufacture of IBMs 9032 line of
ESCON directors, the CD/9000 ESCON directors, the FICON bridge cards, and the first FICON Native (FC)
directors with the McDATA ED-5000 and Inrange FC/9000. Brocades second generation of FICON
directors, the legacy McDATA Intrepid 6064, Brocade M6140, Brocade 24000, and Brocade Mi10K are the
foundation of many FICON storage networks. Brocade continues to lead the way in cascaded FICON with the
Brocade 48000 Director and the Brocade DCX Backbone.
3 of 40
MAINFRAME
Technical Brief
Increased distance
Initially, the FICON (FC-SB-2) architecture did not allow the connection of multiple FICON directors. (Neither
does ESCON except when static connections of chained ESCON directors were used to extend ESCON
distances.) Both ESCON and FICON defined a single byte for the link address, the link address being the
port attached to this director. This changed in January 2003. Now it is possible to have two-director
configurations and separate geographic sites. This is done by adding the domain field of the Fibre Channel
destination ID to the link address to specify the exiting director and the link address of that director.
The FICON directors themselves must be from the same vendor (that is, both should be from Brocade)
The mainframes must be zSeries machines or System z processors: z800, 890, 900, 990, z9 BC or z9
EC. Cascaded FICON requires 64-bit architecture to support the 2-byte addressing scheme. Cascaded
FICON is not supported on 9672 G5/G6 mainframes.
z/OS version 1.4 or greater, and/or z/OS version 1.3 with required PTFs/MCLs to support 2-byte link
addressing (DRV3g and MCL (J11206) or later)
The high integrity fabric feature for the FICON switch must be installed on all switches involved in the
cascaded architecture. For Brocade M-Series directors or switches, this is known as SANtegrity Binding,
and it requires M-EOS firmware version 4.0 or later. For the Brocade 5000 Switch and 24000 and
48000 Directors, this requires Secure Fabric OS (SFOS).
4 of 40
MAINFRAME
Technical Brief
High Availability (HA), Disaster Recovery (DR), and Business Continuity (BC)
The greater bandwidth of and distance capabilities of FICON over ESCON are starting to make it an
essential and cost-effective component in HA/DR/BC solutions, the primary reason mainframe installations
are adopting cascaded FICON architectures. Since Sept 11, 2001, more and more companies are bringing
DR/BC in-house (insourcing) and companies are building the mainframe component of their new DR/BC
data centers using FICON rather than ESCON. Until IBM released cascaded FICON, the FICON architecture
was limited to a single domain due to the single-byte addressing limitations inherited from ESCON. FICON
cascading allows the end user to have a greater maximum distance between sites (up to an unrepeated
distance of 36 km at 2 Gbit/sec bandwidth). For details, see Tables 1 and 2.
Following September 11, 2001, industry participants met with government agencies, including the United
States Securities and Exchange Commission (SEC), the Federal Reserve, the New York State Banking
Department, and the Office of the Comptroller of the Currency. These meetings were held specifically to
formulate and analyze the lessons learned from the events of September 11, 2001. These agencies
released an interagency white paper, and the SEC released its own paper, on best practices to strengthen
the IT resilience of the US financial system. These events underlined how critical it is for an enterprise to be
prepared for disastereven more for large enterprise mainframe customers. Disaster recovery is no longer
limited to problems such as fires or a small flood. Companies now need to consider and plan for the
possibility of the destruction of their entire data center and the people that work in it. A great many articles,
books and other publications have discussed the IT lessons learned from September 11, 2001:
The most successful DR/BC implementations are often based on as much automation as possible,
since key staff and skills may no longer be present after a disaster strikes.
Financial, government, military, and other enterprises now have critical RTOs measured in seconds and
minutes and not days and hours. For these end users it has become increasingly necessary to
implement insourced DR solution. This means that the facilities and equipment needed for the
HA/DR/BC solution are owned by the enterprise itself. In addition, cascaded FICON allows for
considerable cost savings compared with ESCON.
A regional disaster could cause multiple organizations to declare disasters and initiate recovery actions
simultaneously. This is highly likely to severely stress the capacity of business recovery services
(outsourced) in the vicinity of the regional disaster. Business continuity service companies typically
work on a first come, first served basis. So when a regional disaster occurs, these outsourcing
facilities can fill up quickly and be overwhelmed. Also, a companys contract with the BC/DR outsourcer
may stipulate that the customer has the use of the facility only for a limited time (for example, 45 days).
This may spur companies with BC/DR outsourcing contracts to a)consider changing outsourcing firms,
b) re-negotiate an existing contract, or c) study the requirements and feasibility for insourcing their
BC/DR and creating their own DR site. Depending on an organizations RTO and Recovery Point
Objective (RPO), option c) may be the best alternative.
The recovery site must have adequate hardware and the hardware at the recovery site must be
compatible with the hardware at the primary site. Organizations must plan for their recovery site to
have a) sufficient server processing capacity, b) sufficient storage capacity, and c) sufficient networking
and storage networking capacity to enable all business critical applications to be run from the recovery
site. The installed server capacity at the recovery site may be used to meet day-to-day needs (assuming
BC/DR is insourced). Fallback capacity may be provided via several means, including workload
prioritization (test, development, production, and data warehouse).
5 of 40
MAINFRAME
Technical Brief
Fallback capacity may also be provided via a capacity upgrade scheme based on changing a license
agreement versus installing additional capacity. IBM System z and zSeries servers have the Capacity
Backup Option (CBU). Unfortunately in the open systems world, this feature is not common. Many
organizations will take a calculated risk with open systems and not purchase two duplicate servers (one
for production at the primary data center and a second for the DR data center). Therefore, open
systems DR planning account for this possibility and pose the question What can I lose?
A robust BC/DR solution must be based on as much automation as possible. It is too risky to assume
that key personnel with critical skills will be available to restore IT services. Regional disasters impact
personal lives as well. Personal crises and the need to take care of families, friends, and loved ones will
take a priority for IT workers. Also, key personnel may not be able to travel and will be unable to get to
the recovery site. Mainframe installations are increasingly looking to automate switching resources
from one site to another. One way to do this in a mainframe environment is with a cascaded FICON
Geographically Dispersed Parallel Sysplex (GDPS).
Asynchronous remote mirroring becomes a more attractive option to organizations insourcing BC/DR
and/or increasing the distance between sites. While synchronous remote mirroring is popular, many
organizations are starting to give serious consideration to greater distances between sites and to a
strategy of asynchronous remote mirroring to allow further separation between their primary and
secondary sites.
HA/DR/BC implementations including GDPS, remote Direct-Attached Storage Device (DASD) mirroring,
electronic tape/virtual tape vaulting, and remote DR sites are all facilitated by cascaded FICON.
6 of 40
MAINFRAME
Technical Brief
7 of 40
MAINFRAME
Technical Brief
better the intersite cost savings from this consolidation. So, with 4 Gbit/sec FICON and 10 Gbit/sec FICON
available, the more attractive this option becomes.
Another benefit to this approach, especially over long distances, is that the Brocade FICON director typically
has many more buffer credits per port than do the processor and the disk or tape subsystem cards. More
buffer credits allow for a link to be extended to greater distances without significantly impacting response
times to the host.
The technology of DASD arrays places a limit on the number of CU ports inside, and there is a limit of
8 links per LCU. These 8 links can only perform so fast.
This also limits the I/O density (I/Os per GB per second) into and out of the frame, placing a cap on the
amount of disk space the frame can support and still supply reasonable I/O response times.
Cascaded FICON lets customers fully utilize their old disk arrays, preventing them from having to throttle
back I/O loads and make the most efficient use of technologies such as Parallel Access Volumes (PAVs).
Additionally, a cascaded FICON environment requires fewer fiber adapters on storage devices and
mainframes.
Cascaded FICON allows for Total Cost of Ownership (TCO) savings in an installations mainframe
tape/virtual tape environment. FICON platforms such as the Brocade 48000 and DCX are 5 nines
devices. The typical enterprise-class tape drive is only 2 or 3 nines at best due to all of the moving
mechanical parts. A FICON port on a Brocade DCX (or any FICON enterprise-class platform) typically costs
twice as much as a FICON port on a Brocade 5000 FICON switch. (The FICON switch is not a 5 nines
8 of 40
MAINFRAME
Technical Brief
device, while the FICON director is.) However, it may not make sense to connect 3 nines tape drives to 5
nines directors, when the best reliability achieved is that of the lowest common denominator (the tape
drive). Depending on your exact configuration, it can make more financial sense to connect tape drives to
Brocade 5000 FICON switches cascaded to a Brocade DCX (FICON), thus saving the more expensive
director ports for host and/or DASD connectivity.
The number of ISLs between the two cascaded FICON directors and the routing of traffic across ISLs
2.
The number of FICON/FICON Express channels whose traffic is being routed across the ISLs
3.
4.
5.
The nature of the I/O workload (I/O rates, block sizes, use of data chaining, and read/write ratio)
6.
The distances of the paths between the components of the configuration (the FICON channel links from
processor(s) to the first director, the ISLs between directors, and the links from the second director to
the storage control unit ports)
7.
The last factorthe number of buffer-to-buffer credits and the management of buffer to buffer credits
is typically the one examined most carefully, and the one that is most often misunderstood.
9 of 40
MAINFRAME
Technical Brief
About BB Credits
This section is an overview of BB credits; for a more detailed discussion, consult Robert Kembels
Fibre Channel Consultant series.
To prevent a target device (either host or storage) from being sent more frames than it has buffer
memory to store (overrun), the FC architecture provides a flow control mechanism based on a system
of credits. Each credit represents the ability of the receiver to accept a frame. Simply stated, a
transmitter cannot send more frames to a receiver than the receiver can store in its buffer memory.
Once the transmitter exhausts the frame count of the receiver, it must wait for the receiver to credit
back frames to the transmitter. A good analogy is a pre-paid calling card: there are a certain number
of minutes, and you can talk until there is no more time on the card.
Flow control exists at both the physical and logical level. The physical level is called buffer-to-buffer
flow control and manages the flow of frames between transmitters and receivers. The logical level
is called end-to-end flow control and it manages the flow of a logical operation between two end
nodes. It is important to note that a single end-to-end operation may have made multiple transmitterto-receiver pair hops (end-to-end frame transmissions) to reach its destination. However, the presence
of intervening directors and/or ISLs is transparent to end-to-end flow control. Buffer-to-buffer flow
control is the more crucial subject in a cascaded FICON environment.
10 of 40
MAINFRAME
Technical Brief
A BB credit can transport a 2,112-byte frame of data. The FICON FC-SB-2 and FC-SB-3 ULPs use 64 bytes
of this frame for addressing and control, leaving 2 K available for z/OS data. In the event that a 2 Gbit/sec
transmitter is sending full 2,112-byte frames, 1 credit is required for every 1 km of fiber between the sender
and receiver. Unfortunately, z/OS disk workloads rarely produce full credits. For a 4 K transfer, the average
frame size is 819 bytes. Therefore, 5 credits would be required per km of distance as a result of the
decreased average frame size. It is important to note that increasing the fiber speed increases the number
of credits required to support a given distance. In other words, every time the distance doubles, the number
of required BB credits doubles to avoid transmission delays for a specified distance.
BB credits are used by Class 2 and Class 3 service and rely on the receiver sending back receiver-readies
(R_RDY) to the transmitter. As was previously discussed, node pairs communicate their number of credits
available during FLOGI/PLOGI. This value is used by the transmitter to track the consumption of receive
buffers and pace transmissions if necessary. FICON directors track the available BB credits in the following
manner:
Before any data frames are sent, the transmitter sets a counter equal to the BB credit value
communicated by its receiver during FLOGI.
For each data frame sent by the transmitter, the counter is decremented by one.
Upon receipt of a data frame, the receiver sends a status frame (R_RDY) to the transmitter, indicating
that the data frame was received and that the buffer is ready to receive another data frame.
For each R_RDY received by the transmitter, the counter is incremented by one.
As long as the transmitter count is a non-zero value, the transmitter is free to continue sending data. This
mechanism allows for the transmitter to have a maximum number of data frames in transit equal to the
value of BB credit, with an inspection of the transmitter counter indicating the number of receive buffers.
The flow of frame transmission between adjacent ports is regulated by the receiving ports presentation
of R_RDYs; in other words, BB credits has no end-to end-component. The sender decrements the BB credit
by one for each R_RDY received. The initial value of the BB credit count must be non-zero. The rate of frame
transmission is regulated by the receiving port based on the availability of buffers to hold received frames.
It should be noted that the FC-FS specification allows the transmitter to be initialized at zero, or at the value
of the BB credit count and either count up or down on frame transmit. Different switch vendors can handle
this using either method, and the counting would be handled accordingly.
For write-intensive applications across an ISL (tape and disk replication) the BB credit value advertised
by the E_Port on the target gates performance. In other words, the number of BB credits on the target
cascaded FICON director is the major factor.
For read-intensive applications across an ISL (regular transactions) the BB credit value advertised
by the E_Port on the host gates performance. In other words, the number of BB credits at the local
location is the major factor.
Two ports do not negotiate BB credits down to the lowest common value. A receiver simply advertises
BB credits to a linked transmitter.
The depletion of BB credits at any point between an initiator and a target will gate overall throughput.
11 of 40
MAINFRAME
Technical Brief
12 of 40
MAINFRAME
Technical Brief
13 of 40
MAINFRAME
Technical Brief
Payload %
Payload Bytes
1 Gbit/sec
2 Gbit/sec
4 Gbit/sec
8 Gbit/sec
10 Gbit/sec
100%
2112
25
49
98
196
290
75%
1584
33
65
130
259
383
50%
1056
48
96
191
381
563
25%
528
91
181
362
723
1069
10%
211
197
393
785
1569
2318
5%
106
321
641
1281
2561
3784
1%
21
656
1312
2624
5248
7755
Keep in mind that tape workloads generally have larger payloads in a FICON frame, while DASD workloads
might have much smaller frame payloads. Some say the average payload size for DASD is often about 800
to 1500 bytes. By using the FICON Director Activity reports for your enterprise, you can gain an
understanding of your own average read and write frames sizes on a port-by-port basis.
To help you, columns five and six of the FICON Director Activity report in Figure 3 show the average read
frame size and the average write frame size for the frame traffic on each port. These columns are useful
when you are trying to figure out how many buffer credits will be needed for a long-distance link or possibly
to solve a local frame pacing delay issue.
14 of 40
MAINFRAME
Technical Brief
15 of 40
MAINFRAME
Technical Brief
a base address by WLM. The RMF 78-3 (I/O queuing) record has also been expanded. A similar
feature/functionality and interface between FICON switches and the z/OS IOS would be the ultimate in
BB credit allocation: true dynamic allocation of BB credits on an individual I/O basis.
This section has reviewed flow control, basics of BB credit theory, frame pacing delay, current BB credit
allocations methods and presented some proposals for a) counting BB credit usage and b) enhancing how
BB credits are allocated and managed.
Between the FICON channel card on the mainframe (known as an N_Port) and the FICON directors FC
adapter card (which is considered an F_Port)
Between the two FICON directors via E_Ports (the link between E_Ports on the switches is an interswitch link)
Link from the F_Port to a FICON adapter card in the control unit port (N_Port) of the storage device.
The physical paths are the actual FC links connected by the FICON switches providing the physical
transmission path between a channel and a control unit. Note that the links between the cascaded FICON
switches may be multiple ISLs, both for redundancy and to ensure adequate I/O bandwidth.
Domain
Area
AL Port
In a cascaded FICON environment, 16 bits of the 24-bit address must be defined for the zSeries server to
access a FICON CU. The FICON switches provide the remaining byte used to make up the full 3-byte FC port
address of the CU being accessed. The AL_Port (arbitrated loop) value is not used in FICON and is set to a
constant value. The zSeries domain and area fields are referred to as the F_Ports port address field.
It is a 2-byte value, and when defining access to a CU attached to this port using the zSeries Hardware
Configuration Definition (HCD) or IOCP, the port address is referred to as the link address. Figure 5 further
illustrates this, and Figure 6 is an example of a cascaded FICON IOCP gen.
16 of 40
MAINFRAME
Technical Brief
17 of 40
MAINFRAME
Technical Brief
The connections between the two directors are established through the Exchange of Link Parameters (ELP).
The switches pause for a FLOGI, and assuming that the device is another switch, they initiate an ELP
exchange. This results in the formation of the ISL connection(s).
In a cascaded FICON configuration, three additional steps occur beyond the normal FICON switched pointto-point communication initialization. A much more detailed discussion of the entire FICON initialization
procedure can be found in Chapter 3 of the IBM Redbook, FICON Native Implementation and Reference
Guide, pp 23-43.
The three basic steps are:
1.
If a 2-byte link address is found in the CU macro in IOCDS, a Query Security Attribute (QSA) command
is sent by the host to check with the fabric controller on the directors if the directors have the high
integrity fabric features installed.
2.
3.
If it is an affirmative response, indicating that a high integrity fabric is present (fabric binding and
insistent domain ID), the login continues. If not, login stops and the ISLs are treated as invalid (not a
good thing).
18 of 40
MAINFRAME
Technical Brief
Support of Insistent Domain IDs. This means that a FICON switch will not be allowed to automatically
change its address when a duplicate switch address is added to the enterprise fabric. Intentional
manual operator action is required to change a FICON directors address. Insistent Domain IDs prohibit
the use of dynamic Domain IDs, ensuring that predictable Domain IDs are being enforced in the fabric.
For example, suppose a FICON director has this feature enabled, and a new FICON director is
connected to it via an ISL in an effort to build a cascaded FICON fabric. If this new FICON director
attempts to join the fabric with a domain ID that is already in use, the new director is segmented into a
separate fabric. It also makes certain that duplicate Domain IDs are not used in the same fabric.
Fabric Binding. Fabric binding enables companies to allow only FICON switches that are configured to
support high-integrity fabrics to be added to the FICON SAN. For example, a Brocade M-Series FICON
director without an activated SANtegrity feature key cannot connect to an M-Series FICON
fabric/director with an activated SANtegrity feature key. The FICON directors that you wish to connect
to the fabric must be added to the fabric membership list of the directors already in the fabric. This
membership list is composed of the acceptable FICON directors World Wide Name (WWN) and
Domain ID. Using the Domain ID ensures that there will be no address conflicts, that is, duplicate
domain IDs when the fabrics are merged. The two connected FICON directors then exchange their
membership list. This membership list is a Switch Fabric Internal Link Service (SW_ILS) function, which
ensures a consistent and unified behavior across all potential fabric access points.
Managing Cascaded FICON Environments and ISLs: Link Balancing and Aggregation
Even in over-provisioned storage networks, there may be hot spots of congestion, with some paths
running at their limit while others go relatively unused. In other words, the storage network may be a
performance bottleneck even if it has sufficient capacity to deliver all I/O without constraint. This typically
happens when a network does not have the intelligence to load balance across all available paths. The
unused paths may still be of some value for redundancy, but not for performance. Brocade has several
options for supporting more evenly balanced cascaded FICON networks.
NOTE: The FICON and SAN FC protocol (the FC-SW standard) utilizes path routing services that are based on
the industry-standard Fabric Shortest Path First (FSPF) algorithm of that FC protocol. This is not the CHPID
path; it is the connections between FICON switching devices (which cause a network to be created) that will
utilize FSPF.
FSPF allows a fabric (created when CHPIDs and storage ports are connected through one or more FICON
switching devices) composed of more than one switching device (also called a storage network) to
automatically determine the shortest route from each switch to any other switch. FSPF selects what it
considers to be the most efficient path to follow when moving frames through a FICON fabric. FSPF
identifies all the possible routes (multiple path connections) through the fabric and then manages initial
route selection as well as sub-second path rerouting in the event of a link or node failure.
19 of 40
MAINFRAME
Technical Brief
The Brocade 5000 (FICON), Brocade 24000 and 48000 (FICON) Directors, and the Brocade DCX (FICON)
Backbone support source-port route balancing via FSPF. This is known as Dynamic Load Sharing (DLS) and
is part of the base FOS as long as fabric and E_Port functions are present. FSPF makes calculations based
on the topology of a FICON network and determines the cost between end points. In many cascaded FICON
topologies, there is more than one equal-cost path across ISLs. Which path to use can be controlled on a
per-port basis from the source switch. By default, FSPF attempts to spread connections from different ports
across available paths at the source-post level. FSPF can re-allocate routes whenever in-order delivery can
still be assured (DLS). This may happen when a fabric rebuild occurs, when device cables are moved, or
when ports are brought online after being disabled. DLS does a best effort job of distributing I/O by
balancing source port routes.
However, some ports may still carry more traffic than others, and DLS cannot predict which ISLs will be
hot when it sets up routes since they must be allocated before I/O begins. Also, since traffic patterns tend
to change over time, no matter how routes were distributed initially, it would still be possible for hot spots to
appear later. Changing the route allocation randomly at runtime could cause out-of-order delivery, which is
undesirable in mainframe environments. Balancing the number of routes allocated to a given path is not
the same as balancing I/O, and so DLS does not do a perfect job of balancing traffic. DLS is useful, and
since it is free and works automatically, it is frequently used. However, DLS does not solve or prevent most
performance problems, so there is a need for more evenly balanced methods, such as trunking.
On Brocade M-Series FICON switches, FSPF works automatically by maintaining a link state database that
keeps track of the links on all switches in the FICON fabric and also associates a cost with each link in the
fabric. Although the link state database is kept on all FICON switches in the fabric, it is maintained and
synchronized on a fabric-wide basis. Therefore, every switch knows what every other switch knows about
connections of host, storage, and switch ports in the fabric. Then FSPF associates a cost with each ISL
between switching devices in the FICON fabric and ultimately chooses the lowest-cost path from a host
source port, between switches, to a destination storage port. And it does this in both directions, so it would
also choose the lowest-cost path from a storage source port, between switches, to a destination host port.
The process works as follows. FSPF is invoked at PLOGI. At initial power on of the mainframe complex and
after the fabric build and security processes have been fulfilled, individual ports supported by the fabric
begin their initial PLOGI process. As each port (CHPID and storage port) logs into a cascaded FICON fabric,
FSPF assigns that port (whether it will ever need to use a cascaded link or not) to route I/O over a specific
cascaded link. Once all ports have logged in to the fabric, I/O processing can begin. If any port is taken
offline and then put back online, it will go through PLOGI again and the same or a different cascaded link
might be assigned to it.
There is one problem with FSPF routingit is static. FSPF decisions are made in the absence of data
workflow that may prove to be inappropriate for the real-world patterns of data access between mainframe
and storage ports. Since FSPF cannot know what I/O activity will occur across any specific link, it is only
concerned about providing network connectivity. It has only a very shallow concern about performance
number of hops (which for FICON is 1 so that metric is always equal) and speed of each cascaded link
(which can be different and can result in costing each cascaded link as a lower-to-higher cost link).
FSPF static routing can result in some cascaded links being over-congested (due to a shortage of buffer
credits and/or high utilization of bandwidth) and other cascaded links being under-utilized. FSPF does not
take this into account as its only real function is to ensure that connectivity has been established. Although
mainframe end users have long exploited the MVS and z/OS ability to provide automatic CHPID I/O loadbalancing mechanisms, there is not an automatic load-balancing mechanism built into the FC-SW-2 or FCSW-3 protocol when cascaded links are used.
20 of 40
MAINFRAME
Technical Brief
So on one hand a FICON cascaded fabric allows you to have tremendous flexibility and ultra high availability
in the I/O architecture. You can typically enjoy decreased storage and infrastructure costs, expanded
infrastructure consolidation options, ease of total infrastructure management, thousands more device
addresses, access to additional storage control units per path, optimized use of all of your storage
resources, and higher data availability. Also, higher data availability in a cascaded FICON zSeries or z9
environment implies better, more robust DR/BC solutions for your enterprise. So from that point of view,
FICON cascading has many positive benefits.
But on the other hand a plain, FSPF-governed, unmanaged FICON cascaded environment injects
unpredictability into enterprise mainframe operations where predictability has always ruled. So you must
take back control of your FICON cascaded environment to restore predictability to mainframe operations
and stable, reliable, and predictable I/O performance to applications.
All vendors provide the following:
A means of influencing FSPF by configuring a preferred path (cascaded link) between the FICON
switches on a port-by-port basis
A means to prevent a frame in a FICON switching device from transferring from a source port to a
blocked destination portincluding cascaded link port.
But what do these mechanisms mean to you and how do you decide what to use to control your
environment to obtain the results you want? First you have to know what you want to accomplish. Often you
want to have the system automatically take care of itself and to adjust to changing conditions for
management simplicity and elasticity of the I/O system in general to respond to situational workloads and
unusual events. For other enterprises, it might be rigid control over the environment even if it means more
work in managing the environment and less elasticity in meeting shifts in I/O workload hour to hour, day to
day. So choosing the correct management strategy means that you must have a general understanding of
each of the cascaded link control mechanisms, so that you can wisely plan your environment.
The next section presents best practices in FICON cascaded link management.
21 of 40
MAINFRAME
Technical Brief
Backpressure. A condition in which a frame is ready to be sent out of a port but there is no transmit BB
credit available for it to be sent as a result of flow control from the receiving device.
Bandwidth. The maximum transfer bit-rate that a link is capable of sustaining; also referenced in this
document as capacity.
Domain. A unique FC identifier assigned to each switch in a fabric; a common part of the FC addresses
assigned to devices attached to a given switch.
Fabric Shortest Path First (FSPF). A standard protocol executed by each switch in a fabric, by which the
shortest paths to every destination domain are computed output to a table that gives the transmit ISLs
allowed when sending to each domain. Each such transmit ISL is on a shortest path to the domain, and
FSPF allows any one of them to be used.
Flow. FC frame traffic arriving in a switch on a specific receive port that is destined for a device in a
specific destination FC domain elsewhere in the fabric. All frames for the same domain arriving on the
receive port are said to be in the same flow.
Oversubscription. A condition that occurs when an attempt is made to use more resources than are
available, for example when two devices could source data at 1 Gbit/sec and their traffic is routed
through one 1 Gbit/sec ISL, the ISL is oversubscribed.
22 of 40
MAINFRAME
Technical Brief
At the hardware level, the switches on both sides of the trunk must be able to handle the division and
reassembly of several multi-gigabit I/O streams at wire speed, without dropping a single frame or delivering
even one frame out of order. To add to the challenge, there are often differences in cable length between
different ISLs. Within a trunk group, this creates a skew between the amounts of time each link takes to
deliver frames. This means that the receiving ASIC will almost always receive frames out of order and must
be able to calculate and compensate for the skew to re-order the stream properly.
There are limitations to the amount of skew that an ASIC can tolerate, but these limits are high enough that
they do not generally apply. The real-world applicability of the limitation is that it is not possible to configure
one link in a trunk to go clockwise around a large dark-fiber ring, while another link goes counterclockwise.
As long as the differences in cable length are measured in a few tens of meters or less, there will not be an
issue. If the differences are larger than this, a trunk group cannot form. Instead, the switch creates two
separate ISLs and uses either DLS or DPS to balance them.
23 of 40
MAINFRAME
Technical Brief
the other methods, as illustrated in Figure 8, which shows frame-level trunking operating within port groups,
and DLS operating between trunks. On the Brocade 48000 and DCX, trunking port groups are built on
contiguous 8-port groups called octets. There are four octets: ports 0 7, 8 15, 16 23, and 24 31.
The Brocade 5000, 48000, and DCX have flexible support for trunking over distance. Buffers are shared
across 16-port groups, not limited by octets. For example, it is possible to configure up to 8-port 4 Gbit/sec
trunks at 40 km (32 Gbit/sec trunk group) or 4-port 4 Gbit/sec trunks at 80 km (16 Gbit/sec trunk group).
In some cases it may even be more desirable to configure trunks using 2 Gbit/sec links. For example, the
trunk group may cross a DWDM that does not have 4 Gbit/sec support. In this case, an 8-port 2 Gbit/sec
trunk can span up to 80 km. The above example is per 16-port blade or per 16 ports on the 32-port blade in
the Brocade 48000.
McDATA feature key support. A unique feature key is required for each switch that will have Open
Trunking enabled
Open Trunking enable/disable. A user configurable parameter that allows Open Trunking to be
supported on all ISLs for a switch; the default is disabled.
Per-port offloading thresholds. When the bandwidth consumption of outbound traffic on an ISL
exceeds the configured threshold, an attempt may be made to move flows to other equal-cost, but less
heavily loaded ISLs.
Per-switch low BB credit threshold. When the percentage of time that a port spends with 0 (zero) BB
credit exceeds this threshold, an attempt may be made to move flows to other equal-cost, but less
heavily loaded ISLs
Event generation enable/disable for Low BB Credit Threshold Exceeded and Bandwidth
Consumption Threshold Exceeded. If enabled, these events appear in the Event Log, as well as
events that indicate when the condition has ended.
Open Trunking Reroute Log. This log contains entries that indicate a flow reroute.
The objective of the Open Trunking feature is to make the most efficient use of redundant ISLs. Consider
the fabric configuration in Figure 10 with five HBA N_Ports (on the right), six storage N_Ports (on the left),
and four ISLsand assume that all N_Ports and ISLs are 2 Gbit/sec. SW1 and SW2 are two Brocade MSeries FICON directors that support Open Trunking.
24 of 40
MAINFRAME
Technical Brief
25 of 40
MAINFRAME
Technical Brief
The long-term (about a minute or so) statistical rates of data transmission between each ingress port
(ISL or N_Port) and each destination domain
The long-term statistical loading of each ISL, measured in the same time span as the above
The long-term average percentage of time spent with zero transmit BB credits for each ISL.
In the initial release of Open Trunking, a combination of ingress port and destination domain is called a
flow. So the first item in the list above simply states that the statistical data rate of each flow is measured.
Open Trunking uses these statistics to reroute flows as needed so as to minimize overall perceived
overloading. For example, in Figure 10, if ISL1 is 99 percent loaded and has traffic from HBA1 and HBA2,
while ISL2 is 10 percent loaded with traffic from HBA3, it might reroute either the flow from HBA1 or HBA2
onto ISL2. The choice is determined by flow statistics: If the flow from HBA1 to SW1 is 1.9 Gbit/sec, it does
not reroute that flow, because doing so would overload ISL2. In that case only the flow from HBA2 to SW1 is
rerouted.
Unfortunately, Open Trunking cannot help ISLs that spend a lot of time unable to transmit due to lack of BB
credits. This is a condition that is normally caused by overloaded ISLs or poor-performing N_Ports
elsewhere in the fabric, not at the local switch. The 0 (zero) BB credit statistic is primarily used to ensure
that Open Trunking does not make things worse by rerouting traffic onto ISLs that are lightly used but have
little or no excess bandwidth due to credit starvation.
It should be noted that the 0 (zero) BB credit statistic is not just the portion of time spent unable to transmit
due to credit starvation. It also includes the portion of time spent transmitting with no more transmit
credits. Since a credit is consumed at the start of a frame and not at the end of a frame, an ISL that is
transmitting may have no transmit BB credits. It is common for an ISL to be 100 percent loaded and still
have a 0 (zero) transmit BB credit statistic of close to 100 percent.
26 of 40
MAINFRAME
Technical Brief
Statistics, even when measured accurately, may be in a state of flux when measured.
The cost function can be improved by offloading traffic from a lightly loaded ISL onto an even more
lightly loaded ISL, but the minimal improvement in latency would be imperceptible to the user.
To put it simply, too many flows would be rerouted too often if flows were rerouted every time the cost
function could be improved. Therefore multiple checks have been implemented on the rerouting selection
process. These all prevent flows from being rerouted, even in cases in which the cost function would be
improved. Some of these can be adjusted by Brocade EFCM, CLI, or SANpilot configuration as follows:
Two versions of the ISL statistical data rate are kept, one designed to underestimate the actual data
rate and the other designed to overestimate it. When making a rerouting decision, the statistics are
used in such a way as to result in the most conservative (least likely to reroute) decision.
No flow is rerouted from an ISL unless the ISL utilization is above a minimum threshold, called the
offloading bandwidth consumption threshold, or unless it spends more than low BB credit threshold
portion of its time unable to transmit due to lack of BB credits. If one of these conditions is not present,
there is no condition that justifies the cost of rerouting. Both of these parameters are user
configurable.
No flow is rerouted to an ISL unless the ISL expected utilization, computed by adding the flow data rate
to the ISL current data rate, is less than an onloading bandwidth consumption threshold. There is an
onloading bandwidth consumption threshold for each ISL capacity. This threshold is not user
configurable.
No flow can be rerouted if it has been rerouted recently. A period of flow reroute latency must expire
between successive reroutes of the same flow. This latency is not user configurable.
Periodic Rerouting
Periodically, every load-balancing period, a rerouting task runs that scans all flows and decides which
ones to reroute using the criteria discussed above. The load-balancing period is not user configurable.
27 of 40
MAINFRAME
Technical Brief
28 of 40
MAINFRAME
Technical Brief
User
Set
Default
What it Affects
Comments
Basic averaging
time
No
5575 ms
Flow reroute
latency
No
60 sec
Sample tree
sample time
No
100 ms
Load-balancing
period
No
45 sec
Failover disable
time
No
60 sec
Offloading
bandwidth
consumption
threshold
Yes
Default
offloading
bandwidth
consumption
threshold for
ISL capacity
Default offloading
bandwidth
consumption
threshold
(1 G and 2 G)
No
66% (1G)
Onloading
bandwidth
consumption
threshold
(1 G and 2 G)
No
Frame size
estimation
weighting
No
64
Low BB credit
threshold
Yes
50%
Threshold on percentage of
sample time during which the
ISL has experienced a 0 (zero)
BB credit condition
75% (2G)
75% (4G)
75% (4G)
66% (1G)
75% (2G)
75% (4G)
75% (4G)
29 of 40
MAINFRAME
Technical Brief
30 of 40
MAINFRAME
Technical Brief
In order to reduce the likelihood of these types of host or array problems, significant resources were
invested to reducing the occurrence of OOOFs when Open Trunking determines a reroute is necessary.
M-EOS 6.0 and later includes optimizations to allow the original path to drain remaining queued frames
prior to starting transmission on the new rerouted path. In addition to a small delay to allow frames to drain
from the current egress port, a small additional delay is included to allow the downstream switch some time
to deliver the frames it has already received. This prevents OOOFs resulting from unknown downstream
congestion when the re-route occurs.
Even with the improvements to reduce OOOF delivery by temporarily delaying transmission of frames on
the new path, it is strongly recommended that Open Trunking be used in single-hop configurations to reduce
incremental increase in the possibility of an OOOF. Fabric configurations with more than one hop are
acceptable as long as the hop count between data paths (N_Port to N_Port) is limited to one.
Through extensive testing, it has been determined that the delay imposed by allowing the original path to
drain does not significantly impede performance. In fact, the net delay introduced with these enhancements
is typically less than 10 ms. In most situations, the congestion resolved by the reroute typically would have
caused much longer frame delivery delays than the new Open Trunking behavior introduces. Product testing
also shows that the new enhancements have virtually eliminated the occurrence of OOOFs, even in
extremely congested and dynamic fabric conditions.
31 of 40
MAINFRAME
Technical Brief
32 of 40
MAINFRAME
Technical Brief
If a Preferred Path fails, FSPF assigns a functional cascaded link to the F_Port flows on the failed
connection. When the Preferred Path is returned to service, its original F_Port flows are re-instated. Any
F_Ports that are not assigned to a cascaded link via Preferred Path are assigned to a cascaded link using
the FC SPF process.
NOTE: If FSPF becomes involved in allocating cascaded links, then these flows will co-exist with the flows
created using Preferred Path on the same cascaded links. This could create congestion on one or more
cascaded links if it is not taken into consideration.
The limitations of Preferred Path are as follows:
Open Trunking does not manage Preferred Path data flows, so Open Trunking cannot do any automatic
decongestion of Preferred Path links.
Preferred Path cannot be used to ensure a FICON Cascaded 1-hop environment. This is due to the fact
that Preferred Path will fail over to another random cascaded link path if its primary Preferred Path
fails, which could lead to a multi-hop FICON fabric. Use port blocking to ensure that FICON pathing
contains only a single hop.
The Brocade 48000 and DCX do not currently support "preferred pathing" or "blocked E_ports" for
cascaded FICON directors. Brocade has a command called uRouteConfig, which allows you to set up
static routes for specific ports on the switch. But uRouteConfig requires aptpolicy=1 (port-based
routing). (Note that IBM requires port-based routing for Brocade 48000 and DCX FICON Cascade
Mode.) However, uRouteConfig is not supported when the chassis configuration is chassisconfig 5,
which is the chassis configuration for the Brocade 48000 and DCX.
If you are going to use Preferred Path, then assign every F_Port in the fabric to a cascaded link using
Preferred Path and do not let FSPF do any cascaded link assignments.
Disk and tape should not share the same cascaded links. FICON and FCP traffic should also be
separated across the cascaded links. Use Preferred Path in these cases to direct these frame traffic
flows to different cascaded links.
Prohibit Paths
A FICON frame and an FCP frame are essentially the same with the exception of the data payload the frame
is carrying. So, technically speaking, a cascaded link (ISL) can carry both FICON and FCP frames
consecutively over their links with no problems. But many customers want to separate FICON network traffic
from SAN network traffic (and maybe even from disk replication traffic), and then make sure that these
different flows of network traffic remain independent but uncongested as well. Keep in mind that creating
these separate network data flows across a fabric is a business decision and not a technical decision. But if
it is a business requirement, there are technical means to do it.
If you run systems automation and have implemented the function known as I/O operations (IO-Ops), then
from the MVS or zOS console you can use in-band FICON commands that allow you to block or unblock
ports on any FICON switching devices. If you do not use IO-Ops, then by utilizing EFCM you can configure the
PDCMs by using the address configuration matrix to block or unblock ports. In that case, you use EFCM
management routines rather than MVS or zOS procedures to do this blocking and unblocking.
First, block all SAN ports on a switching device in the fabric from transferring frames to all of the FICON
ports on the switching device. Then, block all FICON ports on a switching device in the fabric from
transferring frames to all of the SAN ports on the switching device. This completely stops the flow of frame
traffic from the blocked ports as both source and destination ports. It is done at the hardware level, so
regardless of how you zone a port or prefer a path, a frame will NEVER pass to the blocked port from the
33 of 40
MAINFRAME
Technical Brief
source port unless and until you revise the PDCM blocking configuration. And you can do exactly this same
blocking for the cascaded link ports.
For example, for 10 cascaded links, you can use 4 links for SAN FC traffic only and the other 6 for FICON
traffic only. Choose 6 of the cascaded links for FICON only and block all SAN ports from using those cascaded
links. Block those 6 cascaded link ports from connecting to all of the SAN ports. Then perform the same
procedure, but this time block the 4 remaining cascaded links away from the FICON ports. This creates two
separate flows of network traffic that will never intermingle. At this point, you should consider implementing
some form of trunking to manage both of these now physically separate network data flows independently;
decongesting cascaded links but only in the set of cascaded links assigned to that specific flow.
PDCM port blocking is the strongest method you can use to control frame flow through a switching device.
For that reason you should be very careful when you use it, since it affects FSPF, Preferred Path, and
Trunking algorithms. FSPF cannot assign a blocked cascaded link as a route across the network for ports
that are blocked from using it. You can configure a blocked cascaded link as a Preferred Path, but no
frames will ever be sent to it from the ports that are blocked.
Open Trunking cannot move work from a congested cascaded link to an uncongested cascaded link if that
uncongested link is blocked from connecting to the port with the work flow that will be moved.
NOTE: If you are experiencing a problem getting a switching port to connect to another switching port, it
might be a PDCM, hardware-blocked port. A blocked port can be very difficult to diagnose and quickly
troubleshoot. The only way you will know is to check the PDCM addressing matrix using FICON CUP or EFCM.
34 of 40
MAINFRAME
Technical Brief
TI zones are supported on Condor and Condor 2 (ASIC) Brocade FICON switches running in Brocade
native mode. TI zones cannot be used in FICON environments running in interop or McDATA fabric
mode.
TI zones are not defined in an FC standard and are unique to Brocade. However, their design conforms
to all underlying FC standards, in the same way as base Fabric OS.
TI zones are not backward compatible, so traffic isolation is not supported in FICON environments
with switches running firmware versions earlier than FOS 6.0.0. However, TI zones in such a fabric do
not disrupt fabric operation in switches running older firmware versions. You must create a TI zone with
members belonging to FICON switches that run firmware version 6.0.0 or later.
When a zone is marked as a TI zone, the fabric attempts to isolate all inter-switch traffic entering a switch
from a member of that zone to only those E_Ports that have been included in the zone. In other words, the
domain routes for any of the members (N_Port or E_Port) to the domains of other N_Port members of the
zone are set to use an E_Port included in the zone, if it exists. Such domain routes are used only if they are
on a lowest-cost path to the target domain (that is, the FSPF routing rules will continue to be obeyed). The
fabric will also attempt to exclude traffic from other TI zones from using E_Ports in a different TI zone. This
traffic shaping is a best effort facility that will do its work only as long as doing so does not violate the
FSPF lowest cost route rules. This means that traffic from one TI zone may have to share E_Ports with
other TI zones and devices when no equal-cost routes can be found using a preferred E_Port. And if a
preferred E_Port fails, traffic fails over to a non-preferred E_Port if no preferred E_Ports offer a lowestcost route to the target domain. Similarly, a non-TI devices traffic uses an E_Port from a TI zone if no equal
cost alternatives exist.
As mentioned earlier, TI zones do not appear in the effective zone set for a number of reasons. First, the
members are defined using D,I notation. Doing so allows the routing controls to be determined at the time
the zones are put into effect, eliminating the significant overhead that would be required if WWNs were
used and the routing controls were discovered incrementally, as devices come online. But the use of D,I in
TI zones would cause issues on switches running versions of FOS earlier than 6.0.0 if included in the
effective set, setting all zones referencing devices included in a TI zone to Session mode based on mixed
mode zoning. Additionally, the intent of a TI zone is to control routing of frames among the members and
35 of 40
MAINFRAME
Technical Brief
is not intended to zone them all together. The Zone daemon (zoned) extracts all TI zones from the defined
zone database whenever a change is made to the defined database and pushes them to the nsd for
application.
When a TI zone is being activated, the nsd in each switch determines if any of the routing preferences
for that zone apply to the local switch. This determination must include the appropriate screening for
Administrative Domain (AD) membership if ADs are being used. If any valid TI zones are found that apply
to members on this switch, the nsd in turn pushes the TI zone to fspfd. The FSPF daemon (fspfd) is
responsible for applying the routing controls specified by TI zones.
The fspfd applies those preferences using a new set of APIs (that is, ioctls) provided by the kernel routing
functions.
36 of 40
MAINFRAME
Technical Brief
An N_Port can be a member of only a single TI zone, because a port can have only one route to any
specific domain. This non-duplication rule is enforced during zone creation and modification. If ADs
are configured, this checking is done only against the current ADs Zone database. The zone --validate
command checks against the defined database of all ADs.
An E_Port can be a member of only a single TI zone. Since an E_Port can be a source port (that is, for
incoming frames) as well as a destination, the same one route to a specific domain rule applies to
E_Ports and forces this limitation. The same checking is done as described for N_Ports.
If multiple E_Ports are configured that are on the lowest-cost route to a domain, the various source
ports for that zone are load balanced across the specified E_Ports.
A TI zone provides exclusive access to E_Ports (for outbound traffic) as long as other equal-cost, nondedicated E_Ports exist. Only source ports included in the zone are routed to zone E_Ports as long as
other paths exist. If no other paths exist, the dedicated E_Ports are used for other traffic. Note that
when this occurs, all traffic routed to the dedicated E_Port uses the dedicated path through switches,
regardless of which ports are the source.
If used within an AD, the E_Ports specified in a TI zone must be in that ADs device list, enforced during
zone creation and modification.
Since TI zones must use D,I notation, the ADs device list must be declared using D,I for ports that are
to be used in such zones, enforced during zone creation and modification.
Take care if you are using TI zones for shared ports (E_Ports or N_Ports) because of the limitation that
a given port can appear in only one TI zone. Conflicting members across ADs can be detected by the
use of zone validate, and best practice dictates that such situations not be allowed to persist. (It
might be best not to allow ISL Bind or TI zones to reference a shared N_Port or E_Port, since one AD
administrator can then interfere with actions of another AD administrator. But this may be hard to do.)
Following is an example of implementing FICON and FCP (SAN) intermix on the same fabric(s) to more rigidly
control FICON and cascaded links in this type of environment.
The challenge for mixing FCP and FICON comes from the management differences between the two
protocols, primarily the mechanism for controlling device communication. Because FICON and FCP are FC4
protocols, they do not affect the actual switching of frames, therefore the differences are not relevant until
the user wants to control the scope of the switching through zoning or connectivity control. Name Server
zoning used by FCP devices, for example, provides fabric-wide connection control. By contrast, PDCM
connectivity control typically used by FICON devices provides switch-wide connection control.
Mainframe and storage vendors strongly recommend that if you are implementing intermix you should block
the transfer of any and all frames from a FICON switch port to all SAN connected ports. And then you will
need to do the reverse as well, blocking the transfer of any and all frames from a SAN switch port to all
FICON connected ports. But what about the cascaded links (called ISLs in the SAN world)? Can they be
shared by both FICON and FCP?
37 of 40
MAINFRAME
Technical Brief
SUMMARY
For the mainframe customer, FICON cascading offers new capabilities to help meet the requirements of
todays data center. Your challenge is to ensure performance across the FICON fabrics cascaded links
to insure the highest possible level of data availability and application performance at the lowest
possible cost.
38 of 40
MAINFRAME
Technical Brief
Class 1. A class of service providing a dedicated connection between two ports with confirmed delivery
or notification of non-delivery.
Class 2. A class of service providing a frame switching service between two ports with confirmed
delivery or notification of non-deliverability.
Class 3. A class of service providing a frame switching datagram service between two ports or a
multicast service between a multicast originator and one or more multicast recipients.
Class 4. A class of service providing a fractional bandwidth virtual circuit between two ports with
confirmed delivery or notification of non-deliverability.
Class 4 is frequently referred to as a virtual circuit class of service. It works to provide better quality of
service guarantees for bandwidth and latency than Class 2 or Class 3 allow, while providing more flexibility
than Class 1 allows. Similar to Class 1, it is a type of dedicated connection service. Class 4 is a connectionoriented class of service with confirmation of delivery (acknowledgement) or notification that a frame could
not be processed (reject). Class 4 provides for the allocation of a fraction of the bandwidth on a path
between two node ports and guarantees latency within negotiated QoS bounds. It provides a virtual circuit
between a pair of node ports with guaranteed bandwidth and latency in addition to the confirmation of
delivery or notification of non-deliverability of frames. For the duration of the Class 4 virtual circuit, all
resources necessary to provide that bandwidth are reserved for that virtual circuit, so it is frequently
referred to as a virtual circuit class of service.
Unlike Class 1, which reserves the entire bandwidth of the path, Class-4 supports the allocation of a
requested amount of bandwidth. The bandwidth in each direction is divided up among up to 254 Virtual
Circuit (VC) connections to other N_Ports on the fabric. When the virtual circuits are established, resources
are reserved for the subsequent delivery of Class 4 frames. Like Class 1, Class 4 provides in-order delivery
of frames. A Class 4 circuit includes at least one VC in each direction with a set of QoS parameters for each
VC. These QoS parameters include guaranteed transmission and reception bandwidths and/or guaranteed
maximum latencies in each direction across the fabric. When the request is made to establish the virtual
circuit, the request specifies the bandwidth requested, as well as the amount of latency or frame jitter
acceptable.
Bandwidth and latency guarantees for Class 4 virtual circuits are managed by the QoS Facilitator (QoSF), a
server within the fabric. The QoSF is at the well-known address xFF FFF9 and is used to negotiate,
manage, and maintain the QoS for each VC and assure consistency among all the VCs set up across the full
fabric to all ports. The QoSF is an optional service defined by the Fibre Channel Standards to specifically
support Class 4 service. Because the QoSF manages bandwidth through the fabric, it must be provided by a
Class 4-capable switch.
At the time the virtual circuit is established, the route is chosen and a circuit created. All frames associated
with the Class 4 virtual circuit are routed via that circuit insuring in-order frame delivery within a Class 4
virtual circuit. In addition, because the route is fixed for the duration of the circuit, the delivery latency is
39 of 40
MAINFRAME
Technical Brief
deterministic. Class 4 has the concept that the VCs can be in a dormant state, with the VC set up at the
N_Ports and through the fabric but with no data flowing or a live state, where data is actively flowing.
To set up a Class 4 virtual circuit, the CircuiT Initiator (CTI) sends a QoS Request (QoSR) extended link
service command to the QoSF. The QoSF verifies that the fabric has the available transmission resources to
satisfy the requested QoS parameters, and then forwards the request to the CircuiT Recipient (CTR). If the
fabric and the recipient can both provide the requested QoS, the request is accepted and the transmission
can start in both directions. If the requested QoS parameters cannot be met, the request is rejected.
In Class 4, the fabric manages the flow of frames between node ports and the fabric by using the virtualcircuit flow control mechanism. This is a buffer-to-buffer flow control mechanism similar to the R_RDY FC
flow control mechanism. Virtual-circuit flow control uses the VC ready (VC_RDY) ordered set. VC_RDY
resembles FC R_RDY, but it contains a virtual circuit identifier byte in the primitive signal, indicating which
VC is being given the buffer-to-buffer- credit. Managing the flow of frames on ISLs must also support the
virtual-circuit flow control to manage the flow of Class 4 frames between switches.
Each VC_RDY indicates to the N_Port that a single Class 4 frame is needed from the N_Port if it wishes to
maintain the requested bandwidth. Each VC_RDY also identifies which virtual circuit is given credit to send
another frame. The fabric controls the bandwidth available to each virtual circuit via the frequency of
VC_RDY transmission for that circuit. One VC_RDY per second is permission to send 1 frame per second
(2 kilobytes per second if 2 K frame payloads are used). One thousand VC_RDYs per second is permission
to send 1,000 frames per second (2 megabytes per second if 2 K frame payloads are used). The fabric is
expected to make any unused bandwidth available for other live Class 4 circuits and for Class 2 or 3
frames, so the VC_RDY does allow other frames to be sent from the N_Port.
There are potential scalability difficulties associated with Class 4 service, since the fabric must negotiate
resource allocation across each of the 254 possible VCs on each N_Port. Also, Fabric Busy (F_BSY) is not
allowed in Class 4. Resources for delivery of Class 4 frames are reserved when the VC is established, and
therefore the fabric must be able to deliver the frames.
Class 4 is a very complex issue. For more detailed information, refer to Kembels Fibre Channel Consultant
series of textbooks. In addition, because of the complexity, Class 4 was never fully adopted as a standard.
Further work on it was stopped, and much of the language has been removed from the FC standard.
FC-FS-2 letter ballot comment Editor-Late-002 reflected the results of surveying the community for interest
in using and maintaining the specification for Class 4 service. Almost no interest was discovered. It was
agreed to resolve the comment by obsoleting all specifications for Class 4 service except the VC_RDY
primitive, which is used by the FC-SW-x standard in a way that is unrelated to Class 4.
Therefore, other mechanisms/models for QoS in FICON (FC) were considered, such as the method used by
InfiiniBand.
2008 Brocade Communications Systems, Inc. All Rights Reserved. 07/08 GA-TB-017-01
Brocade, Fabric OS, File Lifecycle Manager, MyView, and StorageX are registered trademarks and the Brocade B-wing symbol,
DCX, and SAN Health are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries.
All other brands, products, or service names are or may be trademarks or service marks of, and are used to identify, products or
services of their respective owners.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning
any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes
to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes
features that may not be currently available. Contact a Brocade sales office for information on feature and product availability.
Export of technical data contained in this document may require an export license from the United States government.
40 of 40