Cundall Critical Systems

Breakfast Discussion – Comparison of

Cooling Techniques
September 2012
08:00 Coffee

08:15 Welcome and Introductions

08:20 Cooling Techniques

08:50 Group Discussion

10:00 End
Critical Systems Team

Malcolm Howe
Partner

Paul Gilvey Matthew Winter

Director-Critical Director-Critical
Systems Systems

Peter Liddle
Associate
What is Driving Change?
Rising
Environmental
Operating
Considerations?
Costs?

Shifts in Client /
User
Legislation?
Perceptions and
Expectations?

Developments
in IT
Equipment?
What are the Opportunities?

• Improved PUE
Increased Cooling • Reduced Operating Cost
Efficiency / • Reduced Carbon Emissions
Opportunity for • Smaller Utility & Back-Up Power
Supplies Needed
Free Cooling • More Attractive to Tenants

• Spatial Fit on Tight Site

• Ability to Operate at Low Load
Modularity / • Phased Build-out to match slow
Flexibility load Growth
• Adaptable to Changes in Hall size
• Improved Resilience
What are the Constraints?

• Local weather conditions may limit

Environmental opportunity for Free Cooling
Constraints • Atmospheric Pollution (DAO)
• Dependency upon water supply

• Configuration of Site/Building
• Resistance to New Technology
• Concerns over unsealed data halls
Other Constraints • Constraints over Cabinet
Arrangement
• Capital Cost
Traditional Chiller + CRAC
Approach
 Traditional Chilled Water System

• Various options for heat

rejection PUE 2.0 to 1.5
• Free-cooling optional
• Pressurised floor void
• Optional cold aisle
containment
• Three mediums of heat
transfer
• Robust and proven 10-17ºC ChW
Temp
technology
28-32ºC Return
Temp
• PUE greatly influenced
by Compressor Power

18ºC Server
Inlet Temp
Traditional Chilled Water System

Where is the Energy Consumed?

Annual Energy Consumption
Typical Site: Chillers + CRACs IT Load
PUE: 1.5 - 2.0 Data Hall Lighting

UPS Losses

Switchgear & Cable Losses

Transformer Losses

VSD Losses

Generator Ancillaries

Tx no load losses

Miscellaneous

Fan Power

Chiller Power

Compressor Energy Pump Power

(kWh) Humidification Power

The largest energy consuming element apart from the IT load is the Compressor
 Optimised / Hybrid
Chiller + CRAC Systems
 Optimised Chilled Water System

• Hybrid heat rejection to

maximise free-cooling
• Elevated chilled water 18-25ºC ChW
Temp
temperature
• Pressurised floor void
• Hot aisle containment
20-23ºC Server
• Variable speed ECDC Inlet Temp

CRAC Fans controlled on

pressure differential
• Three mediums of heat
18-25ºC ChW
transfer Temp

• Limited turn-down
capability
• PUE influenced by 20-23ºC Server

Compressor Power Inlet Temp

PUE 1.5 to 1.25

 Turn-Key Chiller & CRAC Solutions
Airedale ECHO System
PUE up to 1.2
• Turn-key solution - all system
components sourced from
the same manufacturer
• Elevated chilled water
temperatures and reduced
air volumes
• Pressurised floor void
• Active Cabinet Exhaust
discharges warm air to
ceiling void
• Variable speed CRAC Fans
controlled via pressure match
ACE unit duty
• System components inter-
operable via common
controls bus
Turn-Key Chiller & CRAC Solutions

• 50% less air volume

and elevated water
temperatures allow for
free cooling 95% of the
year (in London)
• 5% of year uses
mechanical cooling
(when temp > 21 C)
• No fixed hot/cold aisle
configuration
• Improved performance
compared with Hybrid
chiller + CRAC solution
• Tied-in to single
manufacturer!
Optimised Chilled Water System

Where is the Energy Consumed?

Annual Energy Consumption
Best Practice: Free Cooling chillers and CRACs
PUE: 1.5 – 1.25

Even when Free Cooling

Chillers are used, the largest
energy consuming element
apart from the IT load
remains the Compressor

Compressor
Energy (kWh)
 DAO –
Direct Air Optimisation
Direct Air Optimisation (DAO)
Where is the Energy Consumed?
Annual Energy Consumption
Best Practice: Direct Air Optimisation IT Load

PUE: 1.2 Data Hall Lighting

UPS Losses
Switchgear & Cable Losses
Transformer Losses
VSD Losses
Generator Ancillaries
Tx no load losses
Miscellaneous
Fan Power
Chiller Power
Pump Power
Compressor Humidification Power
Energy (kWh)

Compressor Energy may be significantly reduced by use of DAO

ASHRAE TC9.9 2011
(Guidelines for Data Processing Environments)

Allowable Allowable

Class A1 - Controlled Data Centre Environment (Enterprise Servers, Storage Products)

Recommended 18-27°C at server inlet (5.5dew point - 60% rh)

Allowable 15-32°C at server inlet (20-80% rh)
53
Data Centre World
DAO – Penthouse Solution
DAO – Roof-Mounted AHUs

Path of least Adiabatic cooling

resistance
DX Coil

_ _

Return air duct to AHUs

Cabinets in Hot Aisle Containment

Hourly Scatter Plot of Temperature
Direct Air Optimisation (DAO)
Adiabatic Process

Data Hall Return Air Plenum

The Side-Entry Solution

Side supply/return plenum

Air distribution is asymmetric
AHU Mixing

Rows perpendicular with supply air
AHU
Large plant area required next to external
wall of data hall

The Top-Down Solution

Primary cooling plant located on the roof

A space saving solution but with greater AHU
AHU
structural requirements

General Characteristics of DAO

Hot aisle containment

Data Hall

No requirement for floor void

Use of standard prefabricated ventilation plant

Large filter requirements and external
contamination exposure

Good opportunity for use of Evaporative
Cooling!
 DAO schematic – Free-Cooling mode

Exhaust Air
Exhaust
Fans
Return Air

Damper
Supply Air
100% Free cooling
Mixing
Fresh Air Intake

Supply Air Cooled

DX

Supply
Attenuator Evaporative
Fans
Cooler
Filters

Water
Supply
 DAO Schematic – DAO + Evaporative Cooling

Exhaust Air
Exhaust
Fans
Return Air

Damper
Supply Air
100% Free cooling

Fresh Air Intake

Supply Air Cooled

DX

Supply
Attenuator Evaporative
Fans
Cooler
Filters

Water
Supply
 DAO Schematic – Direct Air + Active Cooling

Exhaust Air
Exhaust
Fans
Return Air

Damper
Supply Air
100% Free cooling

Fresh Air Intake

Supply Air Cooled

DX

Supply
Attenuator Evaporative
Fans
Cooler
Filters

Water
Supply

16
 DAO Schematic – Recirc. + Active Cooling

Exhaust Air
Exhaust
Fans
Return Air
Closed

Damper
Supply Air
100% Free cooling

Fresh Air Intake

Supply Air Cooled

Closed DX

Supply
Attenuator Evaporative
Fans
Cooler
Filters

Water
Supply

16
 Shortfalls of DAO

• Back-up mechanical cooling plant must satisfy the full cooling load.
• Power reserve is required to support mechanical cooling plant:
• This applies to both the Utility Supply and Standby Generator
• Plant space is needed for DAO AHUs and Mechanical Cooling Plant.
Penthouse arrangement difficult to retro-fit into existing buildings
• Intensive air filtration required:
• Risk of contaminants entering the Data Hall
• Air pollutants may cause corrosion issues on IT equipment over
time
• Extensive humidification (water consumption):
• Spray humidifiers may lead to dust problems within Data Halls
• Complex control strategy to address multi-mode operation.
18
 IAO –
In-Direct Air Optimisation
Indirect Air Optimisation (IAO)
Evaporative Heat-Exchange Solution

Evaporative Process
Air distribution is asymmetric

Rows perpendicular with supply air

Units can be placed on roof

Data Hall Return Air

EVAPORATIVE
COOLER

General Characteristics of IAO

Hot aisle containment

No requirement for floor void

Utilises Evaporative Cooling

No direct air movement into data hall

Less reliance on external air conditions

Large water requirement

Can be scaled to match IT load
Benefits of IAO compared with DAO

• Reduced / eliminated requirement for mechanical cooling systems

• Frees-up available site power for IT Equipment
OR
• Smaller Utility / Standby Power supply
• Reduced plant space
• Reduced risk of contamination (sealed Data Hall)
• Potentially reduced fan power consumption (less filtration required)
• Reduced water consumption (less humidification)
• Simplified plant control strategy (fewer modes of operation)
• Greater potential for evaporative cooling (humidification levels not
critical)
• Simplified maintenance
18
IAO – Typical AHU Arrangement

18
IAO - Schematic

Exhaust Supply
Fans Fans

Exhaust Air Evaporative Return Air

Cooler

HX

Supply Air Cooled

Water
Supply Air
Supply

16
 Cundall Studies

Comparison of Cooling Technology – London, LHR

*based on hypothetical test case 1MW data centre (N+1)

 Cundall Studies

Comparison by Region of Cooling Technology

IAO Free-Cooling
Region
pPUE Chiller pPUE

London Heathrow 1.05 1.16

Madrid 1.05 1.18

Abu Dhabi 1.09 1.32

Beijing 1.06 1.2

Shanghai 1.07 1.22

Moscow 1.05 1.14

*based on hypothetical test case 1MW data centre (N+1)

Market leading IAO units

• Dublin Weather File

Munters Oasis™ EXCOOL Kyoto-Cooling MENERGA Adcoolair Eaton Williams**

PUE (partial)** 1.05 1.03 1.16 1.08 1.07

DX - 0h, DX - 0h, DX - 720h, DX - 178h, DX - 4h,

Chiller Operating hours [h] Adiabatic - 8269h Adiabatic - 1324h Adiabatic - h Adiabatic - 5195h Adiabatic - h

Energy Total 490,981 301,968 1,526,218 747,767 632,417

Consumption
[kWh] Energy Savings [%] 76% 85% 26% 64% 69%

Energy 29,458.84 18,118.10 91,573.08 44,866.00 37,945

Annual Costs Water 5,502 2,168 - 3,279 -

[£]*** Total Costs 34,961 20,286 91,573 48,145 37,945
Cost Savings [%] 72% 84% 26% 61% 69%

** The Eaton Williams unit is mixed mode direct and indirect

 IAO Max Ambient – Dublin Weather Data

20 YEARS

ASHRAE Fundamentals 20yr design data
MIN (oC DB) MAX (oC DB)
-7.9 27.8

The majority of hourly ambient temperatures in Dublin

fall below the point at which evaporative or mechanical
cooling is required (Excool)

18
Weather Scatter Plot - Dublin
IAO Overlay
30
30

Moisture Content (g Water/kg Dry Air)

25

25
20

20
15

15
10
10
Adiabatic Spray
5
Humidification
5
0 Mixing with (No Mechanical
Warm Data Cooling)
Hall Air
0
-10 0 10 20 30 40 50 60
Dry Bulb Temperature (°C)
Wrap-up & Conclusion

• Traditional Chiller & CRAC approach not capable of

meeting current ‘Best Practice’ energy performance of
targets.
• Hybrid / Optimised Chiller & CRAC systems can deliver
low PUE and will continue to be applicable on many
projects, due to site constraints.
• Direct Air Optimisation offers significant energy benefits,
but has a number of shortcomings
• Indirect Air Optimisation in gathering pace and offers the
energy benefits of DAO, whilst avoiding many of the
shortcomings
Group Discussion

But what do all of YOU

think?
Group Discussion

Comparison of Cooling Techniques

Organisations present:

Cundall, Telehouse, Telecity, HSBC, Morgan Stanley,

Digital Realty Trust, Virtus

The following pages summarise the discussion which

took place following the presentation of the slides.

The views expressed are the views of the organisation

present.
Group Discussion

Comparison of Cooling Techniques

Note raised by Discussion Material

When looking for a solution, Digital have to consider the long term scenario of say 10 - 15 years as
Digital Realty Trust for example pension funds are among likely investors.
Digital adopt the use of a combination of direct/indirect/chilled water and are an advocate of indirect
air optimisation.

Direct could be considered slightly more energy efficient without heat exchange etc.
There is also a space requirement to consider when looking at these new cooling systems and
limitation of large air handling units on real estate.

New Data Centres will continue with legacy equipment as some data centres cannot fit all of the
Cundall equipment in, new cooling systems have demonstrated that more space is required.

Perception is that there is not a great understanding of cooling trends in some sectors. We have
recently reviewed documentation where PUE has been requested within them. One in particular
had “PUE shall be no greater than 1.25” and then described a conventional chiller and CRAC
solution which can not achieve this figure therefore there is obviously no understanding of what is
required to deliver but there is a realisation that it is important to drive down energy.

We have seen sites designed for 600kw and have been using 130kw and have been in operation
Virtus for 2 years. Plant is therefore massively oversized. The site will only get more efficient as the load
is taken up and the IT is addressed.

The market is restricted to Munters and Excool out there at the moment
Group Discussion

Comparison of Cooling Techniques (continuation)

Note raised by Discussion Material

This can be a problem dealing with perception when trying to sell to tenants.
Cundall
Yes, especially at the corporate level.
Virtus
Tenants generally are not interested in PUE, they are only interested in TCO (total cost of
Virtus ownership), i.e. how much is it going to cost me.

It might be good to see somebody like the Green Grid doing a study of tenant perception for the
Cundall direct and indirect air schemes.

Perception is strange, for example, a raised floor is not necessarily needed with indirect air
Virtus optimised scheme, however, sites have been built with a floor just because there is an expectation
to see one.

The change incorporating 2 steps is possibly too far to go at the moment from conventional to
Virtus having no CRAC units or raised floor at the same time.

In the schemes you have looked at on site how has the return air path been dealt with?
HSBC
Usually by hot aisle containment although the site visited had the solution shoehorned in and the
area was about 6 metres in height with no hot aisle containment and despite concerns the system
Virtus
worked as intended.
Group Discussion
Comparison of Cooling Techniques (continuation)
Note raised by Discussion Material

Most of the data centres operated or run by those present will look very similar in terms of what is
HSBC provided inside and its only with the advent of these two supplier led solutions in response to
reducing efficiency that help to get down energy levels. We would have to see examples of the
systems working for this to be considered as a viable option.

If the colo world is to drive this, it will attract financial institutions we need more players and it is in
its infancy at moment, but the benefits we can see are huge.

Yes, but putting costs aside, the beauty about the solution is that it is really simple in terms of
Virtus technology, basic fans etc. and as a consequence there is also a reduction in BMS points etc.

For a global solution, there is a requirement to be consistent about technology used; therefore it will
HSBC be easier to manage.

Virtus The biggest problematic area for Data Centres at the moment is BMS and failures, etc. i.e.
dampers not closing where required, and pumps or valves not opening.

HSBC Any facility is business driven, and as such design can be dumbed down for example there
probably only is about 15% of the IT system that needs critical support, but has 100% backed up.
The design can be optimised with, high resilience for the critical15% and make further choices to
categorise type of technology that dual infrastructure is needed for.

The regulatory and compliance requirements can be fulfilled quite easily with this approach.
Group Discussion
Comparison of Cooling Techniques(continuation)
Note raised by Discussion Material

Cundall A question for the Financial Institutes – do you think if there are more people who offer this the
more attractive it becomes?

HSBC No one wants to be the first, but have to look at innovative developments going forward. Senior
boards will drive this when they familiarise themselves with what can be achieved by reducing
requirements like PUE.

There may be more attractive solutions out there but with some of the Colo stuff it is easier to go
with the global solutions that exist currently, we therefore may need to revisit again at some stage
in the future.

Cundall Will we ever see a system that goes water cooled instead of air?

Cundall In Dublin we saw a system that had server boards water-cooled to processes but this was due to
high-energy use application like CGI.

Talking to six sigma on CFD side, they are not seeing a threat to their market to drift away from air
cooled to water-cooled.

There are other cooling mediums e.g. CO2 which has been around for some time but has not been
developed further.

HSBC One of the biggest challenges is the manufacturers attitude to raising temperatures within ASHRAE
Standard and their reluctance to accept that the temperature bands can be increased, IBM in
particular are happy continuing with the temperatures they have always operated at.

HSBC There is also a requirement for a strong partnership with IT to look at how it all goes together etc.
without impeding server locations and maintaining airflow.
Group Discussion

Comparison of Cooling Techniques (continuation)

Note raised by Discussion Material

Cundall We have also discovered Server fans are set to run mode from the factory default and could be
changed to Eco mode and quite a lot of energy can be saved by simple dipswitch enabling.
Facebook also deliberately pressurise the room with respect to the hot aisle to try to force air
through the server as a way of reducing server load, as the main plant is more efficient in shifting
air than the localised server fans.

However, the downside of this is that the PUE may increase, as the server fans are not included in
the PUE calculation.

Cundall Monitoring of cooling.

Because this cooling system is conceptually brand new there is no standardised way of testing a
unit. At a recent factory test we asked for the method statement and then set about producing one
in conjunction with the supplier and then implemented it. There are standards for individual
components etc. but not for indirect cooling but applies to cooling towers etc.

Virtus We have witnessed a mock up area and a CFD model was required which subsequently showed
that the system mock-up was working satisfactorily.

Perhaps if the product were not named it would be more successful its nothing new why not call it a
direct air handling unit or indirect air CRAC unit to gain acceptance.

Cundall Why is it only coming in now? The ASHRAE Standard has widened the operating environmental
envelope.

US Congress started to drive this and looked at some desk top PC’s and realised that they were
not taking the expected loads etc. and also wondered why do you need this control?
Group Discussion

Comparison of Cooling Techniques (continuation)

Note raised by Discussion Material

HSBC Perhaps what comes from this is that we need to open up this forum to a wider group, HSBC would
certainly be interested in talking with colleagues throughout global offices perhaps on web in air.
The forum is greatly needed in the sector and it is good that it’s independent.

Cundall We will open up to a wider audience and will look at further topics for discussion for example DCIM
and will circulate to all interested parties.

We will certainly take HSBC up on the offer of a web in air to their colleagues.