Data Center

Practices the Good,

the Bad, & the Ugly

Jonathan Jew - J&M Consultants, Inc.

Co-chair BICSI Data Center Subcommittee
Chair TIA TR-42.6 Telecom Administration
Vice-chair TIA TR-42.3 Pathways & Spaces
Vice-chair USTAG to ISO/IEC JTC 1 SC 25
Image by Thiago Rodrigues Santos WG 3
Lessons Learned from 30+
years of data center design
30+ years
90+ data center projects
Co-Chair BICSI Data Center Design &
Implementation Subcommittee
US Project Lead for ISO/IEC 24764 and
ISO/IEC 11801-5 international data center
cabling standards
Primary contributor ANSI/TIA-942,
ANSI/TIA-942-A, and editor of ANSI/TIA-
942-B revision
 Learn something from
every project
Best practice what to do
Poor practice what not to do
New ideas different way of doing
something (e.g., using foam inserts instead of
brushes around cable openings)

New standards and practices

Changes in technology and in the
industry mean that common
practices may become bad practices
(e.g., facing cabinets all the same way or not sealing cable
openings)
 Use Standards-Based
Structured Cabling
Single-use cabling, removing and installing new cables is disruptive
Difficult to troubleshoot and manage

Example of point-to-point cabling (mid-1990s mainframe data center)

 Use Standards-Based
Structured Cabling
- Can
Organized, Reusable, Flexible
install new connections quickly
Changes are less disruptive
Easier to manage and troubleshoot = higher availability

Example of structured cabling (data center redesigned in 2000s)

ADMINISTRATION &
LABELING
 Label Patch Panels &
Patch Cords

Easy to troubleshoot Difficult to troubleshoot and make changes

and make changes
Only one label (ne pas disconnect )
All ports and cords labeled
(with near and far end port IDs)
 ISO/IEC & TIA Labeling
ISO/IEC TR 14763-2-1 and TIA-
606-B have compatible labeling
schemes
Good labeling and documentation
reduces troubleshooting time and
errors, and thus improves
availability
Cabinet Naming
Scheme

Label cabinets
and racks with
ID on front and
back
Grid coordinates on walls
606-A-1
Patch#s
RU

on
Panelrails
ID
Scheme
 Labeling Patch Panels &
Cables

At minimum, label patch panel with ID of patch panel and

1st or last port of every subpanel
Even high density patch panels must be labeled (maybe
on the edge, on a cover, or a horizontal cable manager
above or below)
 Labeling of Patch Cords
Place labels where they can be read without disconnecting a cable
Labels should be permanent, machine printed labels
Labels are the same on both ends of the cable, however, the order
should be reversed on each end

Local Equip Optional Label with Local Equip

- Rack/cabinet ID Additional Information - Rack/cabinet ID
(e.g. circuit id, server id, business
- Equip/panel ID partner, or name of equipment - Equip/panel ID
- Slot/Port or Port andends of channel - Slot/Port or Port

To Cabinet AD04
Switch sfcoresw01 To Cabinet AD07
Slot 3 port 2 Panel 42 port 24
AD04 sfcoresw01 sfcoresw01 AD07-42:24
sfcoresw01:03/02 To To To AD04
To AD07-42:24 sfhda03sw02 sfhda03sw02 sfcoresw01:03/02

Remote Equip AD07-42:24 Remote Equip

- Rack/cabinet ID - Rack/cabinet ID
- Equip/panel ID
- Slot/Port or Port
to AD04 - Equip/panel ID
- Slot/Port or Port
sfcoresw01:03/02
 Life Cycle Refresh
Rule of thumb is to replace data
center server and network
equipment ~ 3 to 5 years for
maximum availability.
5 to 7 years is more common
Old equipment has higher failure rate
Replacement often required to
deploy new technologies
Replace equipment before end of
support by manufacturer
 Impact of Documentation,
Labeling, & Equipment Refresh

Major reduction
in outages at a
top ten global
technology
company
 Impact of Documentation,
Labeling, & Equipment Refresh

Prioritize work
to maximize impact

1st year was to

improve the data
centers and
network core

2nd+ years were

to improve the edge
(office areas and
labs)
Color-code patch cords

Color-code patch cords to

simplify identification. In
this example for a data
center in the USA:

-Yellow = Production A
-Red = Production B
-White = Management
Color-coded, labeled,
locking power cords
Label power cords at
both ends to avoid
disconnecting the
wrong power cord
Consider color coding
power cords and power
strips
Consider locking power
cords, receptacles, or
retention clips
ELECTRICAL & BONDING
Labeled, color-coded, &
locking electrical
receptacles
Label electrical
receptacles with PDU,
RPP, breaker
Consider color coding
electrical circuits
Electrical receptacles
should be locking to
prevent accidental
disconnection (twist-
lock or IP67
watertight)
Non-locking, unlabeled, not
color-coded electrical
receptacles
This data center in
Australia has
electrical receptacles
that:
-Do not lock (can
easily be pulled out)
-Are not labeled
-Are not color-coded
 Maximum
and Track Load on Every
Power Strip
Load power strips to 40% maximum to
avoid over-loading circuits if a circuit or
power strip fails
Maximum load by US code is 80% (to
ensure that circuit breaker is not
tripped)
Best practice is Data Center
Infrastructure Management (DCIM)
system centralized monitoring
Alternatively, use power strips with
local display that shows current load
Check load on power strips before
adding equipment
 Attaching Mesh-BN to
Pedestals
Do not use cable ties to bond computer Use bonding hardware designed for
room mesh bonding network to pedestals the purpose and listed by a testing lab
 Cable trays properly bonded to
computer room mesh bonding network
 Network equipment not bonded to
rack grounding bar as recommended
by manufacturer bonding cable
comes with equipment

Racks & cabinets bonded serially and not

individually to computer room mesh-BN as
specified in ISO/IEC 30129, TIA-607-C, BICSI-607
Power & UTP
cables are too
close
 Separation of UTP/STP from
shielded power cables run in
parallel
Qty of Electrical Circuit Type Separation
Circuit Distance
s (inches)
Double distances if
1 20A 110/240V 1-phase 0
shielded power cables not
shielded.
2 20A 110/240V 1-phase 0.1 in
shielded No separation
3 20A 110/240V 1-phase 0.2 in required if there is a
shielded properly earthed 1.5
mm thick galvanized
4 20A 110/240V 1-phase 0.25 in steel containment
shielded around either the
5 -15 20A 110/240V 1-phase 1 in power or twisted-pair
shielded cables

16 - 30 20A 110/240V 1-phase 2 in

shielded
31 - 60 20A 110/240V 1-phase 4 in
shielded
61-90 20A 110/240V 1-phase 6 in
shielded
TELECOMMUNICATIONS
PATHWAYS
Small diameter
fiber patch
cords and
trunks should
not be mixed
with heavier
cables that can
damage them
 Install Patch Cords &
Trunks in Solid Bottom
Trays or Duct

Solid bottom in duct prevents microbends that increase attenuation

 Install Patch Cords &
Trunks in Solid Bottom
Trays or Duct
Solid bottom can be installed in cable ladder or wire basket tray
 Dont install fiber cords or
trunks in ladder or wire basket
tray without bottom
 Improper use of cable ties as
supports
Do not use non-continuous supports
(e.g., cable ties, hooks) to support fiber
patch cords or trunks
Use Velcro instead of cable ties for
UTP/STP
Keep patch cords off the
floor
Dress patch cords properly to keep them off the floor
or the base of racks, where they can be stepped on
Leave room for growth in cable trays and
carefully plan cable tray intersections

Tray is full with area of cable at 50% of area of tray

Intersection is too full when two full trays meet
 Cable Trays
Minimum clearance above cable
trays 200 mm and preferably at least
300 mm
Confirm with raised floor designer or
ceiling structural engineer that they
can support cable tray load
No sharp edges or exposed threaded
rods that can damage cables
Maximum depth of cables is 150 mm
Max depth of fiber patch cords 100
 Overhead Cabling
Best practice where ceiling heights permit
as under floor cabling can create airflow
disruption & turbulence
Must be higher than tallest anticipated
cabinet or rack
Suspend cable tray from ceiling to allow
cabinets to be removed
Separation from fluorescent lights 130
mm minimum
 Overhead Cable Tray
Cable tray and duct suspended
from ceiling using threaded rod

Best practice to cover

threaded rod to avoid
cable abrasion
 Under Floor Cabling
Full cable trays could potentially
block airflow if not properly planned
& coordinated (place in hot aisles
avoid front of air conditioning)
Upper cable trays should be narrower
than lower trays to allow access
Recommend at least 50 mm from top
of cable tray to bottom of floor tiles
to allow cables to exit
Example of Wire Basket Cable Trays For
Cabling Under 900 mm Raised Floor
Top tray at least 50 mm from bottom of tile

24" 24"
RAISED FLOOR TILE 1.375" 1.375" RAISED FLOOR TILE
8" 2" 2" 8"
6" 6" 4"
FIBER 4" FIBER
1" SUPPORT STRUT SUPPORT STRUT 1"
Main aisle fiber tray 4"

300 mm between
12" cable
12" trays
12" min clearance
between trays
36"
18" 18"

COPPER CABLING 6" 6" COPPER CABLING

1" SUPPORT STRUT SUPPORT STRUT 1"

Main aisle copper tray 6"

HOT AISLE 10.625" 4.625" HOT AISLE

Server Row Server Row

Main aisle and server cabinet aisle trays at

different levels to avoid congestion at intersections
Spacing between trays not 200 mm
in this data center in Tokyo
These commonly used electrical trays have
sharp edges that damage cables
 No cables on concrete in
contact with earth
Cable should not be installed Use cable matting or other floor
directly on concrete in contact cover to avoid placing cables on
with earth to avoid moisture bare concrete
Remove unused cables
CABINETS AND RACKS
 Equipment Cabinets
Front rails of
cabinets must be set
back to provide
adequate room for
patch cables and
wire managers
Adequate space for
cable management
at front, sides, and
rear
Arrange switches
and patch panels to
minimize patching
between cabinets &
racks
These switches
dont have
adequate
clearance to
cool properly

- 600 mm with no
cabinet side panel
- 300 mm with
cabinet side panel
 Plan for growth & consider
airflow

Leave space
for new equipment
& patch panels

Equipment needs
adequate space for
side-to-side airflow
 Horizontal & vertical cable
management
Provide adequate vertical and
horizontal cable management for
fully populated cabinet
BICSI COMMUNITITY &
BICSI-002 ARE GOOD
SOURCES FOR DATA
CENTER BEST PRACTICES
 BICSI-002 Data Center
Standard
BICSI-002-2014 Data Center Design
& Implementation Best Practices is
an update to the 2011 version
Supplements applicable local data
center standard. It references
ANSI/TIA-942-A (US & Canada),
CENELEC EN 50173-5 (Europe),
ISO/IEC 24764 (International)
Basis for BICSI Data Center Design
Consultant (DCDC) certification
Available in Spanish
 DC Operations &
Maintenance
New BICSI Data Center Operations &
Maintenance standard being
developed
Includes participants from a wide
variety of organizations
Include participants from several
countries on every continent except
Antarctica
 Questions?
Jonathan Jew
President J&M Consultants, Inc.
Website: www.j-and-m.com
Email: jew@j-and-m.com
Co-chair BICSI data center subcommittee
Editor TDMM 13th Edition Telecom
Administration
Chair TIA TR-42.6 telecom administration
subcommittee
Vice-Chair TIA TR-42.3 pathways & spaces
Vice-chair USTAG to ISO/IEC JTC 1 SC 25 WG 3
Editor ISO/IEC TR 14763-2-1 telecom admin
identifiers
SME BICSI TI&M Telecommunications
ADDITIONAL PRACTICES
 Energy Efficiency
Use blanking panels
in open spaces in
cabinets & racks
Use brushes &
grommets for cable
openings
Use baffles for
equipment cooled
side-to-side
 Seismic cabinets and
racks
In seismically active locations consider
seismic cabinets or racks
They must be supported from the
structural floor or on a steel stand
attached to the structural floor
Steel Seismic Rack

Aluminum is brittle so
seismic racks are
typically made of steel

Rack on the left is bolted

to a steel stand attached
to the concrete floor
below (it is not on the
raised floor)
 Light color cabinets enhance
lighting

Front edge of cabinets should

be on edge of floor tile to permit
two full tiles for ventilated tiles
PLANNING AHEAD
Preparing for Category 8

Plan rows, pods, distributor (IDF,

HDA, ZD) locations to permit
migration to Cat 8
24 m max permanent link (panel-
to-panel)
30-32 m channels (3-4 m patch
12 & 32-fiber MPOs

12-fiber MPO has 4 of 12

unused fibers for 40G and
100G

400G Ethernet will use 32-

fiber MPO and waste no
fibers

Expect 3 x 12-fiber MPO

to 32-fiber MPO adapter
cables (that waste only 4
fibers)