Engineering IT Infrastructure

Introduction to Data Centre Design

Barry Elliott
BSc RCDD MBA CEng

Spaces and Places

November 09
www.Capitoline.eu

Engineering IT Infrastructure

Spaces and Places

Suitability of location Defining and sizing the Spaces Basic facilities requirements

www.Capitoline.eu

Engineering IT Infrastructure

Physical location and architectural issues

Does the data centre structure already exist, or are building works required? Are there any known seismic problems in the area? Any known ground subsidence problems in this area, e.g. old mining works? Any known security/ criminal problems likely with this area? Is connection to mains electrical, gas, water, sewage and telecoms services available? Is there a very close proximity to main roads, railway lines, airports, oil or chemical storage or works or military or nuclear installations?
www.Capitoline.eu

Engineering IT Infrastructure

Physical location and architectural issues

Is there easy access to the site by road with parking/loading facilities available? Are there lifts/goods lifts available if the data center is not on the ground floor? Are lifts big enough for the likely sizes of loads, e.g. UPS units etc. Are there any excessive external sources of noise or vibration that will impact upon the data center site? Will the proposed data center be a cause of noise or disturbance to adjoining offices? Are there any potential EMC problems, e.g. mobile phone masts, lift motors on the other side of a wall, proximity to radar transmitters etc? Is there access to a suitable external site for the air conditioner heat exchangers? Is the building or room susceptible to lightning strikes?
www.Capitoline.eu

Engineering IT Infrastructure

TIA 942 location recommendations are:

The building should not be in the flight path of any nearby airports. The building should be no closer than 0.8 km ( mile) from a railroad or major interstate highway to minimize risk of chemical spills. The building should not be within 0.4 km ( mile) of an airport, research lab, chemical plant, landfill, river, coastline, or dam. The building should not be within 0.8 km ( mile) of a military base. The building should not be within 1.6 km (1 mile) of nuclear, munitions, or defence plant. The building should not be located adjacent to a foreign embassy. The building should not be located in high crime areas. EMC risks must be minimised by locating away from radar transmitters and mobile phone masts

www.Capitoline.eu

Engineering IT Infrastructure

BICSI draft standard on data centre design

503 pages To be published in 2009 Complementary to ANSI TIA 942

www.Capitoline.eu

Engineering IT Infrastructure

Very useful American information

www.Capitoline.eu

TIA 942
Railways Highways Military bases Airport Major met area Height above flood level Waterway or coastline 100 year flood plain Fire/police hospital Canals/dams/reservoirs High voltage power lines, petrol stations Radar, TV Fossil fuel power plant Nuclear power plant 1.6 km 0.4 km 0.4 km 0.8 km 0.8 km 1.6 km 0.4 km

BICSI

Engineering IT Infrastructure BICSI

0.8

1.6 km 1.6 km 13 km >8 km <48 km <16 km >3m 1.6 km 91 m < 8km 3.2 km 1.6 km 5 km 8 km 80 km

www.Capitoline.eu

Engineering IT Infrastructure

Offices next to Buncefield oil depot, Dec 2005

Building on the flight path of Tehran airport

www.Capitoline.eu

Engineering IT Infrastructure

www.Capitoline.eu

Engineering IT Infrastructure

Vodafone data center in Istanbul flooded Sept 2009

www.Capitoline.eu

Engineering IT Infrastructure

www.Capitoline.eu

Engineering IT Infrastructure

The Spaces
Work Area Telecommunications Rooms
Or Telecommunications enclosures for small projects

Equipment Rooms Entrance Rooms and Entrance Facilities Computer Rooms Data Centres
www.Capitoline.eu

Horizontal cabling

Engineering IT Infrastructure Telecom

Outlet, TO Work Area Patch Cable

Telecommunications rooms

Backbone/Riser cabling

Equipment or Telecommunications Room Campus cabling External telecoms cabling Entrance facility

Main Distribution Area

Computer Room

www.Capitoline.eu

Engineering IT Infrastructure

Telecommunications Room (TR) and Telecommunications Enclosure (TE)

A TR is an enclosed architectural space for housing telecommunications equipment, cable terminations, and cross-connect cabling. A TE is a case or housing for telecommunications equipment, cable terminations, and cross connect cabling.

www.Capitoline.eu

Engineering IT Infrastructure

Equipment Rooms
An ER is an environmentally controlled centralized space for telecommunications equipment that usually houses a main or intermediate cross-connect. ERs differ from TRs in that ERs are generally considered to serve a building, a campus, a tenant, or a SP, whereas TRs serve a floor area of a building.

www.Capitoline.eu

Engineering IT Infrastructure

Entrance Facility
An EF is expected to provide the following: Point of demarcation between the SPs and customer premises cabling (if required). Primary (electrical) protection devices governed by the applicable electrical codes. Space to house the transition between cabling used in the OSP to cabling approved for intrabuilding use. This usually involves transition to fire-rated cable

www.Capitoline.eu

Engineering IT Infrastructure

www.Capitoline.eu

Engineering IT Infrastructure

Sizing the Spaces

Engineering IT Infrastructure

Telecommunications Room (TR) Guidelines Floor Space Served There must be at least one TR or ER per floor. Multiple rooms are required if the cable length between the HC (FD) and the telecommunications outlet location, including slack, exceeds 90 m (295 ft) or if the usable floorspace to be served exceeds:

www.Capitoline.eu

Engineering IT Infrastructure

TR sizing at least 9 sq m

www.Capitoline.eu

Engineering IT Infrastructure

Size of ER
Allow 9 sq m per work area Multiply No of work areas by 0.07 sq m Minimum size 3 x 5 m e.g. 5000 sq m of office space 5000/9 = 555 555 x 0.07 = 39 sq m
www.Capitoline.eu

Engineering IT Infrastructure

Size of Entrance Facility

Based on frames, ANSI/TIA/EIA-569-A
569A also states a min of 9 space served 10 000 20 000 40 000 50 000 60 000 80 000 100 000

sq m for multi-tenanted buildings

Ent Room area 7 sq m 10 sq m 14.5 sq m 17.5 sq m 20.5 sq m 25 sq m 31 sq m
www.Capitoline.eu

Floor space dimensions 3660 x 1 930 3660 x 2 750 3660 x 3 970 3660 x 4 775 3660 x 5 600 3660 x 6 810 3660 x 8 440

Engineering IT Infrastructure

Serving a useable floor area of. <335 m2 <929 m2 <1860 m2

Data centre Support areas

Tier 1 20% of CR

Computer Room

Telecommunications Enclosure

Telecommunications Room

Equipment Room

Tier 2 30% Tier 3 80% Tier 4 100%+

ER TR
Entrance Facility EF (2 EF for Tiers 3&4)

TE

EF

TR

CR

<7.2 m2

7.2 15 m2

15-99 m2

100 m2 + (per CR)

Summary of Spaces
www.Capitoline.eu

Engineering IT Infrastructure

Sizing the Computer Room

www.Capitoline.eu

Engineering IT Infrastructure

With typical 8-tile pitch 1 rack takes a footprint of 0.6 m x 2.4 m = 1.44 m (min pitch 7 tiles, max 9 tiles)

With realistic overheads, 1 working rack per 3.16 sq m

What size computer room? By the rack

www.Capitoline.eu

Engineering IT Infrastructure

What size of computer room? By the server..

Average rack mounted server, 370 W
29 models, 50 to 900 W, 1 to 4 U Average fitting into a 4 kW rack, 11 units For every eight server racks, one comms and one storage rack 11 servers per rack =0.34 sq m per server
www.Capitoline.eu

300

Sq m of floor space

Engineering IT Infrastructure

11 servers per rack, 4 kW per rack, 0.34 sq m per server. 8-tile pitch

250

200

150

100

50

servers
0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750
www.Capitoline.eu

Engineering IT Infrastructure

IT Yield: How much floor space does useful ICT work?

Theoretical maximum, ASHRAE 7-tile pitch model, 48% Realistic average, 20% Equivalent to one rack per 3.3 sq m

www.Capitoline.eu

Engineering IT Infrastructure

Floor space utilisation

Existing sites

average rack footprint=0.64 sq m (weighted average of 600 and 800 racks)

New build projects

floor space No of racks racks/sq m efficiency 369 144 2.6 25.0% 400 140 2.9 22.4% 195 68 2.9 22.3% 170 57 3.0 21.5% 55 18 3.1 20.9% 43 14 3.1 20.8% 235 74 3.2 20.2% 199 60 3.3 19.3% 210 60 3.5 18.3% 189 44 4.3 14.9% 100 20 5.0 12.8%

floor space No of racks 76 31 41 15 47 15 26 8 47 14 160 45 44 9 47 9 77 14 78 14 95 15 210 27 77 5 average median

racks/sq m 2.5 2.7 3.1 3.3 3.4 3.6 4.9 5.2 5.5 5.6 6.3 7.8 15.4 5.3 4.9

efficiency 26.1% 23.4% 20.4% 19.7% 19.1% 18.0% 13.1% 12.3% 11.6% 11.5% 10.1% 8.2% 4.2% 15.2% 13.1%

average median
Source: Capitoline survey

3.3 3.1

19.9% 20.8%

www.Capitoline.eu

Engineering IT Infrastructure

Ideal Computer Room size

If max throw distance of aircon unit is 12 m then max width is 24 m
24 m max
CRAC unit Rows of racks

6 m min 24 m max

Ideal size is between 144 and 600 m2

www.Capitoline.eu

Engineering IT Infrastructure

Ideal Computer Room size

Rooms larger than 600 square metres tend to be difficult to manage They require large amounts of air conditioning CRAC units probably need to be in the middle of the floor, which makes calculations difficult Gas fire suppression quantities will be huge. A 900 sq m room will be >3600 m3
www.Capitoline.eu

Engineering IT Infrastructure

Sizing the Data Centre building and site

www.Capitoline.eu

Engineering IT Infrastructure

Separate UPS/battery room recommended if capacity exceeds 100 kVA

The relationship of the Spaces within a Data centre (TIA 942)

www.Capitoline.eu

Engineering IT Infrastructure

Space
Computer room, server room, machine room etc Control room General office area Telecommunications Entrance Facility

Function
To house the computer racks and communications equipment. A generic space with sufficient air conditioning, power supplies and communications cabling to allow a non-application specific IT environment with best use of space An area, adjoining the computer room, where all control and monitoring functions relevant to the site are concentrated An office area where the IT staff can work A room or area where all external communications cabling enters the building. It serves as a point of demarcation between different owners of cabling, provides a point for over-voltage protection and allows a transition from external (flammable) cables to internal cables. (Required BS 6701) Two required for T3/T4 If inert gas is used as the main fire suppression system then it requires a large volume for storage. Alternatively the gas bottles may be placed against a wall adjacent to the Computer Room or if a fluorocarbon gas is used (placed within the Computer Room) this area may be dispensed with

Fire gas suppression store

Electrical switch room A room where the external power cables enter the building and forms a point of demarcation between different cable owners plus all main switching and metering UPS and battery room Generator room For loads in excess of 100 kVA (TIA 942) it is recommended to have a separate UPS and battery room to save space and heat load in the main Computer Room To house the standby diesel generators. This may be in or adjacent to the main building

www.Capitoline.eu

Engineering IT Infrastructure Oil store To house the diesel fuel to run the standby generators for between 8 and 96 hours. This may be in or adjacent to the main building. The electrical switch room, UPS and generator must all be close to each other to minimise electrical losses in long power cables An area to store and unpack equipment and to build items like racks without making dust and causing disruption in the main Computer Room An area adjacent to the main doors to allow heavy equipment to be shipped into the building A secure entrance with anti-piggybacking- airlock controls A room to hold meeting and provide additional office space

Storage and build area Delivery and loading area Main entrance Planning and meeting room Internal staff facilities Electrical substation

Male/female/disabled toilets. Shower room. Basic dining area and kitchen facilities Due to the power load it is likely that separate electrical substation would be needed by the utility company. This should be away from the main building to reduce EMC issues If split DX units are to be used then a condenser unit is needed for each Computer Room DX unit. These must be in a secure area either adjacent to the main building or even on top of it, but preferably not over the computer room itself A secure main gate leading onto hard standing area of sufficient space and strength for HGVs to unload heavy equipment and manoeuvre

Air conditioning condensers Main gate and hard standing

External staff facilities Parking space for cars, bicycle storage and smoking shelter

www.Capitoline.eu

Engineering IT Infrastructure

Ratio of support areas to Computer Room Uptime Institute

Tier 1 Tier 2 Tier 3 Tier 4 20% 30%+ 80-90% 100%+

www.Capitoline.eu

Engineering IT Infrastructure

CR 240 m2 Total 620 m2 Support to CR, 160%

www.Capitoline.eu

Engineering IT Infrastructure

CR 240 m2 Building 620 m2 (2.6 x CR), 160% support area Total site footprint 1674 m2 (2.7 x building) www.Capitoline.eu

Engineering IT Infrastructure

CR 340 m2 Building 744 m2 (2.2 x CR), 120% support area Total site footprint 1674 m2 (2.25 x building)

www.Capitoline.eu

Engineering IT Infrastructure

CR 480 m2 (2 x 240) Building 990 m2 (2 x CR), 106% support area Total site footprint 1998 m2 (2 x building) www.Capitoline.eu

Engineering IT Infrastructure

Facilities requirements
Room dimensions and height Floor strength Connection of services External services Access, loading, DDA etc Dcor Lighting Fire Regulations

www.Capitoline.eu

Engineering IT Infrastructure

Facilities requirements, in separate chapters

Raised floor design Power supplies, UPS and earthing HVAC Cable containment and management Fire detection and suppression Generic cabling and interconnection

www.Capitoline.eu

Engineering IT Infrastructure

Data centres are places of work and are subject to the same rules as any other place of work
EU Construction Products Directive 89/106/EEC EU Electromagnetic Compatibility Directive 89/336/EEC EU Low Voltage Directive 2006/95/EC EU Measures to improve safety & health of workers at work 89/391/EEC EU Health & safety requirements of the workplace directive 89/654/EEC Minimum requirements for the provision of H&S signs at work 92/58/EEC Noise at work directive 2003/10/EC Energy services directive 2006/32/EC EU Energy performance of buildings directive 2002/91/EC

www.Capitoline.eu

Engineering IT Infrastructure

Room height (TIA 942)

Minimum ceiling height: The minimum height in the computer room shall be 2.6 m (8.5 ft) from the finished floor to any obstruction such as sprinklers, lighting fixtures, or cameras.. A minimum of 460 mm (18 in) clearance shall be maintained from water sprinkler heads.
Absolute min slab to slab height is 2.9 m, i.e. 400 mm under floor, 2.1 m racks, 400 mm air return path

www.Capitoline.eu

Engineering IT Infrastructure

Floor strength (TIA 942)

Minimum floor loading capacity The minimum distributed floor loading capacity shall be 7.2 kPA. The recommended distributed floor loading capacity is 12 kPA. The floor shall also have a minimum of 1.2 kPA hanging capacity for supporting loads that are suspended from the bottom of the floor (for example, cable ladders suspended from the ceiling of the floor below). The recommended hanging capacity of the floor is 2.4 kPA.

www.Capitoline.eu

Engineering IT Infrastructure

Lighting (TIA 942)

Lighting shall be a minimum of 500 lux in the horizontal plane and 200 lux in the vertical plane, measured 1 m above the finished floor in the middle of 4 all aisles between cabinets.

www.Capitoline.eu

Engineering IT Infrastructure

Access
Doors shall be a minimum of 1 m wide and 2.13 m high, without doorsills, hinged to open outward (code permitting) or slide side-toside, or be removable. Doors shall be fitted with locks and have either no center posts or removable center posts to facilitate access for large equipment. Exit requirements for the computer room shall meet the requirements of any other local requirements. DDA. Ramps to be not less than 1:12 (see Raised Floor section). All pathways to be >900 wide

www.Capitoline.eu

Engineering IT Infrastructure

Decor (TIA 942)

Dcor. Floors, walls, and ceiling shall be sealed, painted, or constructed of a material to minimize dust. Finishes should be light in color to enhance room lighting. Floors shall have anti-static properties as per IEC 61000-4-2.

www.Capitoline.eu

Engineering IT Infrastructure

Connection of services
Electricity supply
Who from, capacity, where does it enter the building, more than one?

Telecommunications
Who from, capacity/type, where does it enter the building, more than one?

Gas, water, drainage, sewage No other service should cross the Computer Room space
www.Capitoline.eu

Engineering IT Infrastructure

External services
Suitable location for standby generators
Noise, weight, security, fuel storage, proximity

Suitable location for external air conditioning components

DX Condenser units, central chiller system etc, weight, security, power supplies, proximity
www.Capitoline.eu

Engineering IT Infrastructure

Fire Regulations

Fire Plan and Risk assessment Emergency lighting and signage Doors and Emergency exits

Engineering IT Infrastructure

Fire safety plan

The fire safety plan is a multilayered approach that requires a coordinated plan for Designing for low flammability and fire risk Operating with low risk Emergency exits Emergency lighting Emergency exit signage Fire detection, appropriate to the area covered Fire alarm Multi-level automatic fire suppression Manual fire alarm and portable fire extinguishers Staff training and fire drills in place Maintenance plan for all equipment involved

www.Capitoline.eu

Engineering IT Infrastructure

What Is Emergency Lighting?

The lighting required for use when the supply to normal lighting fails, enabling safe exit from a building

www.Capitoline.eu

Engineering IT Infrastructure

Directives
The Construction Products Directive (89/106) The Workplace Directive (89/654) The Signs Directive (90/664) The Regulatory Reform (Fire Safety) Order

www.Capitoline.eu

Engineering IT Infrastructure

EXIT SIGNS
There are, and have been 3 types of exit signs, with an appropriate level of legislation enforcing each type :
.

This sign is now obsolete & was replaced with a pictogram sign from 24th December 1998

The Interim Format Sign, This sign is deemed to comply with Signs Directive and can be used within premises that already use this format.

The European Signs Directive Format as designated by the Health & Safety (Signs & Signals) Regulations. This is now the preferred format and must be applied to all new applications.
www.Capitoline.eu

Engineering IT Infrastructure

A 7 point plan can now be considered for the detailed design.

Stage 1
Points of emphasis Mandatory locations to cover specific hazards and highlight safety equipment and signs :

a) NEAR STAIRS each tread should receive light

b) NEAR CHANGES OF LEVEL

www.Capitoline.eu

Engineering IT Infrastructure

Stage 1

c) NEAR CHANGES IN DIRECTION & INTERSECTIONS ALONG CORRIDORS

d) TO ILLUMINATE EXIT DOORS & SAFETY SIGNS

www.Capitoline.eu

Engineering IT Infrastructure

Stage 1

e) NEAR FIRE FIGHTING & FIRST AID EQUIPMENT

f) TO ILLUMINATE FIRE CALL POINTS

www.Capitoline.eu

Engineering IT Infrastructure

Stage 1

g) OUTSIDE AND NEAR TO EACH FINAL EXIT

h) TO ILLUMINATE EXIT DOORS

www.Capitoline.eu

Engineering IT Infrastructure

Fire exits
Normal risk, furthest distance to the fire exit must be less than 18 m if the room has only one exit For rooms with more than one exit the max distance must be > 45 m Essentially this means any room more than 80 sq m needs more than one exit
www.Capitoline.eu

Engineering IT Infrastructure

PAS 79:2005 Fire risk assessment. Guidance and a recommended methodology 750 mm pathways for escape 900 mm if wheelchairs access required

www.Capitoline.eu