Green Computing Power Efficient Management in Data Centers

Using Resource Utilization as a Proxy for Power

THESIS

Presented in Partial Fulfillment of the Requirements for the Degree Master of Science in
the Graduate School of The Ohio State University

By

Ralston Augustine Da Silva

Graduate Program in Computer Science and Engineering

The Ohio State University

2009

Master's Examination Committee:

Rajiv Ramnath, Advisor

Paul Sivilotti

Jay Ramanathan
 Copyright by

Ralston Augustine Da Silva

2009
 Abstract

Many organizations are working towards reducing the carbon footprint of their data

centers; i.e. reducing their power consumption. Server virtualization is used to decrease

power consumption by consolidating multiple servers onto a few physical machines.

Virtualization provides increased flexibility by providing a means to dynamically move

virtual machines from one physical machine to another. Using resource utilization as a

proxy for power, we build models of power consumption for individual server types, and

use this information along with business value and SLA information, to efficiently

allocate virtual machines to physical machines.

ii
 Dedication

This thesis is dedicated to a greener planet earth.

Then the LORD God took the man and put him in
the garden of Eden to tend and keep it.
Genesis 2:15 NKJV

iii
 Acknowledgments

I would like to acknowledge the advice and guidance of my advisor, Dr. Rajiv Ramnath

and Dr. Jay Ramanathan, who helped me to put my thoughts together and produce this

work. It was an honor to work with Dr Ramnath who was encouraging and supportive

through the entire process. He is such a joy to work with.

I would also like to thank Nationwide, specifically JP Byrne for the openness in

collaborating with CETI, our research group, and for opening up their data centers and

exposing us to real-life problems. Special thanks to all my brothers and sisters from

church, who cooked food when I was busy, prayed for me when I was sick, and stood by

me through the tough times in grad school.

Lastly, and most importantly, I would like to thank my parents my mom for all her

sacrifices, and my late dad, for providing all my needs. Through the hard times he was

going through, he always encouraged me to finish what I started, and I am glad I fulfilled

his dream by completing the Masters degree.

iv
 Vita

March 2000 ...................................................Secondary School Certificate Examination,

Goa Board of Secondary and Higher

Secondary Education, Goa, India

March 2002 ..................................................Higher Secondary School Certificate

Examination in Science, Goa Board of

Secondary and Higher Secondary Education,

Goa, India

2006 ...............................................................B. E. Computers, Goa University, Goa,

India

2007 to present ..............................................Graduate Student, Department of Computer

Science and Engineering, The Ohio State

University

Fields of Study

Major Field: Computer Science and Engineering

v
 Table of Contents

Abstract ............................................................................................................................... ii

Dedication .......................................................................................................................... iii

Acknowledgments.............................................................................................................. iv

Vita...................................................................................................................................... v

Table of Contents ............................................................................................................... vi

List of Tables ..................................................................................................................... xi

List of Figures ................................................................................................................... xii

Chapter 1: Business Problem ............................................................................................. 1

Carbon Footprint ............................................................................................................. 1

Capacity Management ..................................................................................................... 2

Increase ROI on power .................................................................................................... 6

Data Center Management Problem ................................................................................. 8

Chapter 2: Problem Analysis ............................................................................................ 11

Data Center Scenario ..................................................................................................... 11

Problems with Existing Approaches ............................................................................. 13

Lack of Power Modeling ........................................................................................... 13

vi
 Inability to trace the use of power ............................................................................. 15

Inability to curtail rapid growth in power demand .................................................... 16

Inability to quantify the tradeoffs when using new green technologies .................... 17

Inability to Match Applications to Hardware ............................................................ 18

Need for a Comprehensive Architecture ....................................................................... 19

Need for a Reference Architecture ................................................................................ 20

Need for a combined approach of IT and cooling ......................................................... 22

Benefits of Traceability ................................................................................................. 24

Chapter 3: Green Practices................................................................................................ 30

Server Consolidation ..................................................................................................... 30

Virtualization................................................................................................................. 30

Dynamic Voltage and Frequency Scaling ..................................................................... 31

Smarter Cooling Solutions ............................................................................................ 32

Continuous Monitoring and Redeployment (Dynamic Migration) ............................... 33

Heterogeneity Awareness .............................................................................................. 34

Strategic Replacement of Hardware .............................................................................. 36

Chapter 4: Solution Approach .......................................................................................... 37

Traceability in the data center from requests to power ................................................. 37

Using traceability and profiling to promote an optimized datacenter ........................... 41

vii
 Machines running at higher optimizations - close to the SLA .................................. 41

Business value Profiling for SLA .............................................................................. 42

Application profiling ................................................................................................. 43

Hardware Power Profiling ......................................................................................... 44

Chapter 5: Research Contributions ................................................................................... 45

Power Management ....................................................................................................... 45

Power profiling of hardware ......................................................................................... 53

Workload Generator .................................................................................................. 55

Generating CPU Load ............................................................................................... 57

Generating Memory Load ......................................................................................... 62

Generating I/O Load .................................................................................................. 64

Generating Network Load ......................................................................................... 65

Chapter 6: Results ............................................................................................................. 69

Regression ..................................................................................................................... 69

Chapter 7: Remaining Research........................................................................................ 81

Business Value Profiling ............................................................................................... 81

Business IT Alignment .............................................................................................. 87

RED Transaction Model ............................................................................................ 88

Defining an SLA curve .............................................................................................. 95

viii
 OLA for resource ....................................................................................................... 97

Application Profiling ................................................................................................... 100

Normalization of resource utilization units ............................................................. 101

Improved Chargeback Model .................................................................................. 101

Resource Utilization into costs .................................................................................... 102

Power along with other ROI stuff............................................................................ 103

Other Costs .............................................................................................................. 104

Side effects and Constraints .................................................................................... 105

Chapter 8: Application of framework after completing the remaining research ............ 107

Integrated Continuous Improvement ........................................................................... 108

Identify Problems .................................................................................................... 109

Identify Solutions .................................................................................................... 109

Apply Solutions ....................................................................................................... 110

Measure Indicators .................................................................................................. 110

Prioritizing Opportunities............................................................................................ 112

Initially - Static Optimization .................................................................................. 112

Demand Management of Services ........................................................................... 119

Ongoing - Dynamic Optimizations.......................................................................... 119

Conclusion................................................................................................................... 120

ix
References ....................................................................................................................... 122

Appendix A: Workload Generator Code ........................................................................ 126

SyntheticWorkloadGenerator ...................................................................................... 126

WindowsOS ................................................................................................................ 128

WindowsStatus ............................................................................................................ 130

WindowsThread .......................................................................................................... 134

CPULoadThread.......................................................................................................... 136

CPUWorkerThread...................................................................................................... 138

IoLoadThread .............................................................................................................. 139

IoWorkerThread .......................................................................................................... 141

MemoryLoadThread.................................................................................................... 143

MemoryWorkerThread................................................................................................ 145

NetworkLoadThread ................................................................................................... 147

NetworkWorkerThread ............................................................................................... 149

Appendix B: [R] Code .................................................................................................... 151

x
 List of Tables

Table 1: Regression Results .............................................................................................. 72

Table 2: Comparing Regressions using different number of variables............................. 74

Table 3: Comparing the errors .......................................................................................... 75

Table 4: Regression results for second machine ............................................................... 77

xi
 List of Figures

Figure 1: Carbon dioxide emissions of data centers ........................................................... 1

Figure 2: Percentage of Data Centers that anticipate the need for additional capacity. ..... 3

Figure 3: Power Consumption of Data Centers .................................................................. 8

Figure 4: The data center scenario .................................................................................... 12

Figure 5: Need for traceability two intermingled systems ............................................. 25

Figure 6: Tracking the application usage of each incoming request ................................ 38

Figure 7: Tracking the resource usage of each application............................................... 39

Figure 8: Converting the resource utilization into power ................................................. 39

Figure 9: The correlation of incoming requests to power consumption ........................... 40

Figure 10: Model of a data center that uses virtualization and consolidation .................. 47

Figure 11: Hierarchy of Power Managers ......................................................................... 52

Figure 12: Workload Generator ........................................................................................ 56

Figure 13: Workload Generator with feedback loop ........................................................ 57

Figure 14: CPU utilization of idle machine ...................................................................... 59

Figure 15 : More granular view of Utilization of an idle CPU ......................................... 60

Figure 16 : Class Diagram of Synthetic Workload Generator .......................................... 61

Figure 17: Memory Hierarchy .......................................................................................... 63

Figure 18: Class diagram- Processor Memory I/O and Network Load Generation ......... 68

Figure 19: Power consumption of two machines with varying cpu utilization ................ 79
xii
Figure 20: Power consumption machines with varying cpu util in cycles per second ..... 79

Figure 21: Power consumption of two machines with varying memory utilization ......... 80

Figure 22: Treating each level as a business unit ............................................................. 85

Figure 23: Interaction ........................................................................................................ 89

Figure 24: Executing an incoming request with two interactions..................................... 90

Figure 25: Agile Interactions ............................................................................................ 92

Figure 26: Red Service Interactions with BioS Values .................................................... 93

Figure 27: SLA and Power relationship using two RED Transactions ............................ 96

Figure 28: Continuous Improvement Methodology ....................................................... 108

Figure 29: Application Business Value v/s SLA ............................................................ 113

Figure 30: Hardware Cost v/s Capacity .......................................................................... 115

Figure 31: Hot and Cold Aisle Approach ....................................................................... 118

xiii
 Chapter 1: Business Problem

Carbon Footprint

Over the years, as data centers consume more and more power, their carbon footprint has

increased. Figure 1 shows a graph from McKinseys report[28] that places data centers carbon

emissions close to the emissions of the airline industry and steel plants. Data centers need to be

more energy efficient and environment friendly.

Figure 1: Carbon dioxide emissions of data centers. Source: 2006 McKinsey Report Uptime
Institute

Based on the above, clearly there is a need for the enterprise to reduce the growth in data center

power demand. This requires a more comprehensive and effective approach to power

management than the piecemeal approach of employing various best practices, with little or no

1
attention to coordination. Such a comprehensive management methodology would benefit the

enterprise by: (1) increasing the life of the data center; (2) increasing profitability for the

enterprise by maximizing ROI for the data center; (3) reducing the power consumed by idle

servers; (4) increasing revenue by allowing the enterprise to better leverage the capacity of its

data center resources; (5) preventing free riding within the enterprise by allowing development

of an improved chargeback model, which more accurately charges business units for the capacity

that they use; and (6) promoting a greener data center, which may have public relations and

perceived corporate environmental responsibility benefits. How this approach offers all these

benefits is described in Chapter 5: Research Contributions. In the following section however,

the factors in data centers which have led to the current crisis in capacity management and power

management are characterized.

Capacity Management

In the past, data centers (DCs) were designed and managed with availability, reliability, and

contribution to business value as the principle goals. Capacity limitations were not generally

significant, at least with respect to power; i.e., power did not constitute a constraint on the

operation of the DC in practice. If power was considered at all in the design or management of

the data center, it was only with respect to achieving a data center lifetime which was considered

reasonable, typically in the range of 15 - 20 years [1]. As long as this expectation was met, and

the enterprise derived sufficient value from the power being used, there was no further

consideration given to power. The enterprise suffered no ill effects from this lack of concern with

2
power even as late as approximately the last five to seven years. More recently, however, several

important developments have resulted in power capacity becoming a significant constraint in

virtually all data centers.

One of these developments being that the steady increase in the use of information technology

(IT) in todays enterprise has led to a rapidly increase in the demand for power in data centers.1

In contrast to the expectation of a 15 - 20 year life cycle of the past, Figure 2 dramatically

illustrates the change. Even data centers which have significantly expanded capacity within the

last five years are still facing the prospect of insufficient capacity in the next four years.

Figure 2: Percentage of Data Centers that anticipate the need for additional capacity.
Source: Hardwiring Green into Infrastructure and Data

Now, these data centers that have seen a huge increase in capacity have another problem. It is

difficult to trace how the power is being used. Once a request enters the data center, we usually

do not have information about which application is servicing the request. In addition we neither

have information about which particular server is being used, nor do we know the effect of

servicing that request on power consumption. In other words we do not have traceability with

1
[1] notes that processing requirements in data centers, and the attendant need for power, grew more
than 15 times between 1990 and 2005.

3
respect to power. Once a request is initiated; it is lost in the system. This type of measurement

has not been done in the past, and now over time it has become increasingly difficult since

todays enterprise is a collection of a complex set of systems. These complex systems address

different operational needs, but work together to address larger business goals. Though these

systems are interdependent, they typically have been developed over time and are very different

from each other. The wide variety of systems, as well as their complex interactions, makes

traceability of power with respect to the operation of these systems more difficult to achieve.

Further, successful enterprises involve constant change due to innovation or attempts to increase

efficiency. Indeed, it is often this continuous change that has helped the organization to survive.

Constant changes to the enterprise mean constant changes to the different systems that form the

enterprise. These changes are reflected in terms of infrastructure changes due to additions to the

existing architecture by extending it in lieu of redesigning it, and other implementation changes.

It is difficult to predict the effects of all these changes on the complex enterprise, and each

change introduces new challenges in traceability. The people who make decisions that impact the

evolution of the enterprise need to have good metrics that help them make the best decisions.

Due to the complexity of the system, these metrics are hidden in a series of complex parameters,

which are dependent on each other. The relationships between these parameters and the business

value they represent are not known - which means the people in charge of infrastructure cannot

justify their investments to the business stakeholders. The different IT systems implementing the

complex enterprise system consist of a huge collection of servers. These servers live in data

centers, which house the various applications required by the enterprise. Let us consider a web

4
application. A web application is usually has a three tiered architecture the presentation, the

business logic and the database layers. Each of these components runs on some servers. In the

current scenario, if we put a lot of instrumentation in place, we can measure the power consumed

by each server. This is an aggregate value, and we do not know how much is spent on each

transaction or how much energy is consumed by a particular application. This is again due to the

lack of traceability. The different stakeholders need to know how the service level agreements

can be affected by changes to the architecture, and they also need to know how changes would

affect the business value associated with the applications. In other words, they need to be able to

make informed decisions and be proactive instead of reactive.

As the enterprise grows, along with becoming more complex, these systems also grow in size

and require the addition of new physical infrastructure. This causes the datacenter to grow in

size. Typically, data centers are built with the possibility of expansion in mind, but it is not only

additional space, but also electrical capacity, that is needed. The ability to expand is always

limited and thus at some point the enterprise must build a new data center once the capacity of its

existing data centers is reached.

5
 Increase ROI on power

The complex nature of todays enterprise is reflected in the data center. As the enterprise grows,

it is constantly adapting to changes in the environment. The acquisition of new companies and

merging of IT systems is one example of change the enterprise faces. Another reason for change

can be the adoption of new technologies. These changes over time, lead to a heterogeneous data

center, which grew in an ad hoc manner, or sometimes in a more systematic manner. Either way,

it leads to a very complex system where different systems interoperate with each other in a very

knotty configuration to produce the desired service.

This complex nature of the enterprise leads to the inability to measure and manage power in a

comprehensive way. The result is that the business can neither track nor optimize ROI (Return

on Investments) or TCO (Total Cost of Ownership) for investments in data center infrastructure

(hardware, software, facilities, human resources, etc.); but the business has no metrics to

determine power costs associated with a particular infrastructure element. This gap in

traceability also means that the business cannot make good decisions with respect to

infrastructure investment or management.

The enterprise is unable to understand the dynamics of the trade-offs between performance and

power consumption, and therefore cannot make sound choices about power expenditures versus

value added for the enterprise. On the capacity management front, this amounts to an inability to

6
make informed choices regarding DC lifetime, because the enterprise cannot quantify the

tradeoff between DC lifetime reduction and the benefit to the enterprise of making a particular

change in DC management or operation which adds some amount of value for the enterprise.

This moves the DC closer to power out, i.e., a state where the power infrastructure capacity of

the data center has been reached.

In order to address the rapidly increasing power requirements, many enterprises have adopted

various best practices in recent years, including facilities improvements, virtualization and

measures to dynamically control power usage. These efforts however, have been done in

isolation, and as best practices, but their effect as a whole is not looked into. Many measures can

work against each other if their effects on each other are not studied. For example, consolidation

using virtualization has been a major area of concern these days. However if the consolidation is

not studied holistically, it can actually lead to an increase in power consumption. The

consolidated servers generate more heat. They also create hot spots in the data center, which are

difficult to cool. This can lead to an increase in cooling power consumption, if the data center

does not have localized cooling, which would result in the entire data center being overcooled to

account for the hotspots. Thus, inspite of all the power saving efforts data center power demands

have continued to grow at a faster rate than expected. Not all the increase in the power demand

can be attributed to the growth in the enterprise. Figure 3 shows that the growth of power

demand in the recent past has continued unabated. Because of this the solution that the

enterprises turn to is to building new DCs. Virtualization, consolidation and using blade servers

has slowed this trend somewhat, but data centers continue to run out of power capacity. Building

new DCs is costly, and must be justified in terms of ROI. Further, if the underlying issues are not

7
effectively addressed, any newly built data center would be expected to suffer a similar fate to

the one(s) it replaces; i.e., it will run out of capacity sooner than expected.

Figure 3: Power Consumption of Data Centers Source: Hardwiring Green into

Infrastructure and Data Center Investments - Data Center Operations Council

Data Center Management Problem

Another problem for data center managers is the difficulty in understanding the dynamics of the

trade offs between performance and power consumption. To do this they have to know the

business value associated with a particular application or server and be able to compare the

business value generated to the power consumed by the application. The information needed

does not stop there. They also need to know how changes to the infrastructure which could save

power can affect business value. If they had this information they could make sound choices

about power expenditures versus value added for the enterprise.

8
Having such detailed information can enable the enterprise to look into the Service Level

Agreements (SLAs) set for the various applications, and would force them to justify the

requirement of a particular SLA. Studying the business value would help the enterprise see the

value generated by a particular SLA, and how much the business value would be impacted if

there was a reduction on the SLA. Reducing the SLA a little bit could result in huge power

savings this is because higher SLAs could mean more availability, more redundancy, etc.

which directly implies more power consumption.

Moreover, as stated previously under the section Increase ROI on power, even if adoption of

best practices had been enough to reduce or reverse the growth in the need for power,

coordination of the various approaches must also be considered, so that they do not conflict,

work at cross-purposes, or otherwise result in significantly sub-optimal data center operation.2

The literature however reveals that few attempts have been made at developing a comprehensive

integrated architecture which can be applied to power management, regardless of the particular

technologies used in the DC.3 Further, there is no research which addresses the modeling of

power, or the implementation of traceability of power in the modern data center in a

comprehensive way. Specifically, the correlation between a capacity increase and the power

consumption increase, as well as the corresponding change in the operating cost for the power

consumed, has not been studied.

2
For a discussion of some of the challenges and benefits of such coordination, see, for
example, [2] and [3].
3
One example of work which attempts to address this gap is [3].

.
9
There is therefore a need to model the relationship between service level agreements (SLAs),

operating level agreements (OLAs), and the power consumed. Such a model would allow the

enterprise to understand the relationship between IT performance and power consumed, and

therefore to make more informed business decisions about how power is used.

10
 Chapter 2: Problem Analysis

This section looks at the problem in more detail and describes the problem and the questions the

solution is expected to answer.

Data Center Scenario

Figure 4 attempts to describe the problem at the data center. The data center consists of a huge

collection of servers, or physical machines, which have to be provided with power and cooling.

The lowest block represents the data center. The four blocks on top of the data center represent

four physical machines.

Now, if we consider the data center environment to be consolidated and virtualized, then every

physical machine will contain one or more virtual machines. If there is no virtualization, then we

can treat the machine as if it is housing a single virtual machine, as can be seen in the figure the

backward right side physical machine does not have a hypervisor installed on it it can thus run

only one operating system instance, but it can still run multiple applications.

11
 Figure 4: The data center scenario

Every virtual machine runs at least one application, and these applications service transactions,

which are incoming requests. The requests are shown as black dots in the figure.

The questions the data center management teams seek to answer are questions like Which

applications should we club together and put on a single virtual machine? and Which virtual

machines should we run on a single physical machine? Which virtual machines should we group

together to run on the same physical machine? .Other advanced questions could include How

to dynamically move virtual machines and applications around to make sure we always have an

optimal mix on the machines, and at the same time ensure that the SLAs are met?

12
The incoming requests, which are represented as dots in Figure 4, have business value associated

with them. We want to be able to trace this business value down to the level of resource

allocation and power consumption and make more informed decisions when allocating virtual

machines to physical ones, or deciding which applications go together on a single physical

server.

Problems with Existing Approaches

As discussed above, the problem in enterprise data center power management is that there is

neither an effective approach to model power consumption, nor is there a method to implement

traceability of power in the data center. Finally, there is no comprehensive architecture for

managing power dynamically in the data center that could make use of power modeling and

power traceability data to allow optimum tradeoffs between value to the enterprise and power

consumption.

Lack of Power Modeling

Power modeling, or the ability to predict power consumption by hardware, and power

requirements of applications, is necessary to evaluate the current operation of the data center, and

also to predict the effects of changes which are being considered. Any approach which cannot

predict power with reasonable accuracy must be based to a large extent on guesswork.

13
In most data centers, there is relatively little information available on the power being used. Even

when data is collected, it is typically not granular enough to be useful in understanding how

power is being used, or to be useful in validating attempts to reduce power consumption.

There are two general approaches to the measurement of power. The first approach is direct

measurement of the power consumed using an instrument such as a wattmeter. While the most

direct, this approach has several limitations: (1) The additional hardware cost; (2) The time delay

in collecting and aggregating all the data collected from the watt meters; (3) The fact that very

few DCs have been built with such direct measurement capability, and therefore it would have to

be retrofit in virtually all data centers. A much more significant problem is that it would be

extremely difficult to achieve power measurement that would be granular enough with this kind

of approach. Metering individual server racks, or perhaps even individual servers, would be

feasible, but metering individual subsystems in each server would be extremely difficult. As

argued below, however, power data on each subsystem is required to enable tracing of power and

dynamic power management in the data center.

Since direct measurement of power is problematic, various approaches to measuring power

indirectly, or we could say, modeling power, have been used. A common approach in the

literature has been to model system power use as the aggregate of the power usage of each

subsystem, including CPU, memory, disk I/O, network, etc. Both [4] and [5] take such an

approach, and demonstrate the general validity and accuracy of modeling power in this way.

A significant advantage of this approach is that it does not need the addition of any

instrumentation. The resource utilization can be collected by using counters provided by the

14
operating system. Also there are many commercially available tools, like Hewlett-Packards

SiteScope and IBMs Best/1 that collect resource utilization information. Most data centers

use these tools to collect metrics to be used in capacity management. If this data can be used to

accurately model power consumption of server systems and software applications, the capability

to predict dynamic power consumption would be within the reach of many, if not all, enterprise

data centers. The proposal is to leverage this data, which is already being collected by most data

centers, to model power use of servers and applications in the data center by building models of

hardware power use, as well as application power profiles, based on the resource utilization data

collected. This work is described below in Chapter 5: Research Contributions under the heading

Power profiling of hardware.

Inability to trace the use of power

Another difficult challenge is tracing the use of power - the ability to identify how power is used

to do IT work, or execute transactions, in the data center. Since a given transaction is typically

executed by various physical, and perhaps virtual, servers, in a distributed environment, accurate

modeling and sufficiently granular measurements are required to enable this type of tracing.

Tracing the use of power is important not so much for the management of power per se, but

rather for the business purpose of being able to determine how much power is being used by a

transaction relative to the value that it generates for the enterprise. This information is useful in

various ways, including in making improvements in business processes, allocating scarce

resources in the way that will maximize value to the enterprise, and in improved chargeback, as

15
discussed further in Chapter 5: Research Contributions under the heading Power profiling of

hardware.

Once the subsystem resource use of a given service component running on a particular piece of

hardware can be modeled, tracing the power consumed in processing service requests - using the

power model described above - also becomes possible. This approach also supports another

necessary objective, namely, the ability to dynamically manage power in the data center. For

example, as the data center is operating, the power use of numerous different alternative

configurations of the data center can be modeled easily, and then the configuration which is best

from a power perspective can be chosen.

Inability to curtail rapid growth in power demand

As new hardware and applications are added to the data center, the management is unable to

determine precisely how power capacity will be affected. Although various best practices have

been followed in most data centers, these are to some extent one size fits all solutions which

may not work well, or at least, may not be optimum, in a given data center. This observation is

borne out by the fact that, as discussed above in Chapter 1: Business Problem, best practices

have not succeeded in significantly reducing the rapidly increasing demand for power. The

inability to measure, model, and trace power prevents the enterprise from determining if a given

best practice will result in a net benefit or detriment with regard to the demand for power in its

data center.

16
 Inability to quantify the tradeoffs when using new green technologies

The data center management teams have to deal with the adoption of new green technologies.

New technologies provide great benefits, but also pose new management problems. Some green

techniques are listed in Chapter 3: Green Practices. Unless there is a means to quantify the

benefit gained by using a particular technique, there is no way for the management team to

quantify the power savings. As an illustration, consider virtualization, which is a technique

which enables a single piece of hardware to run multiple virtual machines. These virtual

machines (VMs) can be dynamically moved from one physical server to another, to provide

dynamic load balancing. Now, this seems to be a good technology to use, but there are many

issues which still need to be taken care of. One of them is the tradeoff between flexibility of

running multiple VMs each housing a single app, and the overhead of running many virtual

machines it might be more efficient to run an aggregation of many applications on a single

virtual machine and thus decrease the number of virtual machines (and the overhead associated

in running each one of them). A key issue is that, for a given number of applications, a larger

number of VMs gives more flexibility, but also involves more overhead in running the VMs. For

example, if we have ten applications, and we put them on ten separate VMs, then we have

maximum flexibility with respect to how these ten virtualized applications can be matched to

physical servers, how they can be moved around, etc. but we also have ten different VMs, which

means we have roughly ten times the overhead incurred by one VM. On the other hand, if we

put two of the applications on a single VM, we have five VMs total, which means we have less

17
flexibility - fewer choices in how to match the applications to physical servers, but we only have

half the overhead. The tradeoff between flexibility and overhead has to be managed well, to

obtain as much benefit as possible from the flexibility without incurring excessive overhead

costs. Without a means to measure the power consumption impact of a particular decision, it

becomes difficult to justify the decisions made when using new power saving measures.

Inability to Match Applications to Hardware

The aim behind application to hardware matching in the data center is to optimally match

applications to hardware with respect to the amount of resources consumed by the application.

This is because the types of hardware running in the data center can be heterogeneous. For

example, some servers could be very efficient in performing CPU intensive computations, but

consume a lot of power for I/O intensive applications. Thus, a lot of power can be saved by using

the most appropriate hardware for a particular application. In this sense, this type of matching is

similar to what Nathuji et al. call platform heterogeneity [12]. Since applications vary in their use

of different system resources, a corresponding variation in power use on different hardware

systems is to be expected. Application to hardware matching attempts to take advantage of this

variation by optimally matching the resource utilization profile of the application to the power

characteristics of the physical server system on which it is deployed. Although this idea appears

simple enough in principle, There is no previous research which provides a methodology for

matching along these lines. This is probably due to the fact that good power models for

applications running in DCs have not been generally available previously. Without a power

18
model that correlates application resource utilization with application power consumption, and

also hardware power consumption, the only way to do application to hardware matching is to

actually run each application on each of the different server systems running in the data center,

and study its impact on power. This appears impractical, and the benefits may not outweigh the

costs. By modeling power in terms of resource utilization, this problem can be overcome,

because we can accurately estimate the power use of the application running on a particular

server system without actually running it.

Need for a Comprehensive Architecture

This thesis seeks to present a comprehensive data center architecture which is general enough to

be implemented in virtually any data center. Further, the methods should be relatively easy to

implement, by using tools which typical data centers already have available. Such an architecture

would serve the role of coordinating the various methods that are used in the data center for

power management, including power modeling and tracing, as described above. The

management architecture would also allow dynamic changes to be made to the configuration of

the data center as IT and power demand changes. Without such a comprehensive architecture,

capable of dynamically responding to changing DC conditions, there can be no assurance that

significant amounts of power are not being wasted at any given point in time. We need a system

that monitors IT load and power conditions, and determines if some alternate configuration of

the DC would use less power while doing the same IT work and meeting all relevant Service

Level Agreements (SLAs). While doing this, however, we also need to quantify the cost of

19
having such an architecture, which includes the cost of monitoring and the cost of actually

implementing the dynamic changes.

Need for a Reference Architecture

This thesis addresses the types of problems which have led to the current challenges which

enterprise DCs face in managing power. In particular, it addresses the gaps in modeling power

and in tracing the use of power, as well as the need for a more comprehensive approach to the

management of power in the modern enterprise data center, by developing a reference

architecture within the Adaptive Complex Enterprise (ACE) framework [27]. The ACE

framework provides a conceptualization of the modern service enterprise, as well as abstractions

for modeling transactions in the enterprise. These abstractions provide mechanisms for realizing

traceability of power, and also for managing power in the data center, both locally and globally.

The model enables the enterprise DC to function in a sense and respond (S/R) manner, by taking

account of performance per unit of value generated for the enterprise, as well as power

requirements. Such S/R operation enables the enterprise DC to manage power while responding

dynamically to ever-changing service requirements, by effectively coordinating the diverse

technologies within it.

We thus need a comprehensive reference architecture for dynamic power management. The only

attempt at something approaching a general data center architecture in the literature is [16]. This

work generally incorporates the advantages of virtualization, heterogeneity awareness, and

20
abstractions for power management in virtualized systems, all of which have been identified

above. This paper also provides for coordinated management of IT power and cooling power.

Therefore, the high-level features of the architecture for data centers that is presented in [16] are

adopted here. But these ideas are extended and generalized to address certain significant

limitations. In particular, the framework in [16] appears to be limited to data centers which have

room-based, as opposed to hot and cold aisle, or some other cooling infrastructure configuration.

We, however, need to develop an architectural model which is general in nature, and which

could be applied to any data center.

The model presented in [16] provides for communication between the cooling power

management module and the IT power management module but does not appear to provide for

information to be fed back to the IT power module to assist it in reaching globally optimum

decisions on IT power management, i.e., decisions which minimize the net power consumed for

both IT and cooling. We propose a feedback mechanism to overcome this limitation.

A fourth limitation of the management architecture in [16] is that it has no clear component for

modeling or predicting the power use of particular applications, or of specific hardware. This

limitation also results in a lack of traceability. Nathuji et als model appears to be aimed

primarily at reducing power consumption, rather than on identifying the correlation between

servicing requests in the data center and the power used. This limitation is addressed by

developing a model of power use that relates to resource utilization, and can therefore capture

this correlation. This model is explained in Chapter 4: Solution Approach and Chapter 5:

Research Contributions.

21
 Need for a combined approach of IT and cooling

The two key factors listed below, make the use of static measures alone insufficient for

managing power in the data center. Accordingly, the thesis proposes that a Sense and Respond

(SaR) architecture is necessary for optimizing power use.

First, power consumption in modern data centers varies widely, even over short periods of time.

Two major consumers of power in data centers are IT and cooling [20]. Without even

considering the power used by the cooling infrastructure, the dynamic variation due to IT power

fluctuations is significant. [21] reports a range of variation between 45% and 106% for typical

enterprise class servers.

Second, although current servers use relatively less power at idle than previous generations of

hardware, every server consumes some power at idle, while at the same time it does no useful IT

work. As can be seen from the range of power variation cited above, even modern hardware

typically consumes at least 50% of its peak power consumption even at idle, as can be seen in

Figure 19. For this reason, significantly reducing server idle time, in order to increase server

utilization as much as possible, has become critical to minimizing power use while still

maintaining required performance levels.

Because the number and nature of incoming requests, and the corresponding power use in the

data center vary significantly over time, no single static configuration of the data center will be

22
optimum at all times. Rather, the power management mechanism for the DC must be able to

respond dynamically to changing, and to some extent unpredictable, conditions. In this sense, the

DC management mechanism cannot simply attempt to service the routine requests which it

receives while minimizing cost per request. Rather, the DC must be able to sense ever-changing

IT and power requirements, in order to dynamically respond to the requests in a way that

optimizes the tradeoff between resource utilization and value generation. Thus, I propose that a

Sense-and-Respond architecture, which is capable of this type of dynamic adaptation, is

required.

23
 Benefits of Traceability

Previously, there was very little concern for traceability in the data center. While it is critically

important for data center managers to know how much power is being used, which is addressed

by power modeling, it is arguably just as important to know how the power is being used; in

short, how much power is too much (or too little) depends on its use. If the use is important

enough, a great deal of power may not be too much. There is no way, however, for the enterprise

to make these judgments without detailed information about how power is being used in the data

center.

To understand what traceability means, and what its actual implications are, lets look at an

example. Consider an online site that sells cell phones. Customers visit the site and order a

phone, and the business has the phone shipped to their home. The system managing these

transactions has many subsystems, for example, an inventory management system, a system

connected to the suppliers of the phones, the courier company, customer support, etc. For

simplicity, well look at only two systems in our example. One system is the one that supports

the website that the customer uses to place the order, and the other is a website the suppliers use

to indicate that they have sent a shipment to the company. Figure 5: Need for traceability

illustrates these two systems at an abstract level.

24
 Figure 5: Need for traceability two intermingled systems

From the figure we can see that the customer requests are load-balanced and split into two web

servers, WebServer001 and WebServer002. The Supplier requests are serviced by

WebServer003. The business logic layer consists of two servers, AppServer001 and

25
AppServer002. The requests from WebServer001 are serviced by AppServer001 and

AppServer002. However AppServer002 services requests from all the three WebServers. Finally,

the data stored in database servers DB_01 and DB_02 are used by both the AppServers.

Such sharing of systems are common in data centers, and in fact, as the systems are more

complex, there are much more dependencies and complex interactions among various

subsystems. Now, an incoming request can be tracked for SLA requirements, since the requests

enter and exit the system from the same points; but there is no trace as to how the request was

serviced. The request gets lost in the complex systems in the enterprise. we cannot distinguish

which requests the database servers are being used for, or we cannot separate out the requests

serviced by AppServer002.

How does this affect power consumption? The various power saving measures applied to the

data center requires changes to the infrastructure. However, when the infrastructure is so

complex, we cannot distinguish and separate out the work done by each server which means

we do not have a means to know the SLA that each individual server has to meet. This leads to a

situation where the servers are over-provisioned, and planned for extra capacity capacity they

sometimes never use leading to systems being idle. Idle systems are very inefficient, and

consume power without doing a proportional amount of work. They also generate heat which

requires additional power to cool. Having traceability would mean that the request would be

tracked from the time it enters the system, till it leaves the system. Information about which

server is servicing the request would be available and this information would help the data center

managers make informed decisions on changes to the data center. They could quantify the

26
impact of their changes, and even justify the costs of implementing new green strategies with

respect to the cost savings they achieve.

Traceability gives the organization many benefits. The unit of execution that can be traced is

called a transaction. Tracing the workload and treating it as transactions allows us to do the

following:

1. Business Value of a Transaction

Traceability allows us to track the business value generated when a transaction is

executed. It helps us to provide the business stakeholders with fine grained information.

For example, we can now keep track of the revenue associated with a particular

transaction. This business value can be aggregated to find the business value of an

application, and any tradeoffs made to reduce power consumption can be made to the

applications which have the lowest business value. Knowing the business value also

helps the business select an appropriate SLA for the application thus reducing the

practice of trying to get as high SLA that the business unit owning the application can

afford.

27
 2. Resource consumption of a transaction

With the fine grained data that we have, we can also calculate the amount of resources

needed to execute a particular transaction. For instance, we could predict the amount of

CPU, Memory and I/O needed for a particular request. This information helps us to

allocate resources efficiently and to increase the utilization of the resources. Higher

resource utilization means lower idle time, which means that the resources are being

utilized properly.

3. Power consumption of a transaction

The resource utilization data collected can be used as a proxy to determine the amount of

power consumed. Quantifying the amount of power consumed by a transaction allows us

to calculate the cost associated with servicing the requests. If the fixed and variable costs

of the resources are taken into consideration, we could have a means of quantifying the

energy spent, and justifying if it is worth the business value it will generate.

4. Carbon footprint of a transaction

Once we quantify the power consumption of a transaction, we can convert this into

carbon equivalent values, and hence we can calculate the carbon footprint of transactions.

This information could be used to provide new services like inform the customer of the

carbon footprint of the transaction, etc, and can be also used to make the data center more

competitive, by having them inform the user of how power-efficient they are.

28
 5. Implementation cost of a transaction

Measuring the power consumption of a transaction could also be used to help developers

develop power efficient coding practices. Lets illustrate this benefit with an example.

Consider the case where a file has to be sent over a network. We can either compress the

file and then send it over the network, or send it without compressing. There are benefits

and drawbacks in either case. If the file is compressed, it will take a certain amount of

time to perform the compression, which affects the SLA, and it will also use some

resources (CPU, Mem, I/O) to perform the compression, which would result in a certain

power consumption. On the other hand, if the file is sent without compressing, it would

take longer time to be transferred, which again could affect the SLA, and on the power

consumption front, it will consume more power using the network resources. Providing

the programmers with a model of the power in the data center will empower them to

make decisions that will produce power efficient code, that will help reduce power

consumption in the data center, and also help in better management by using the metrics.

Having traceability and studying the power consumption can also quantify the savings,

and help justify the extra time the power-aware development would take.

29
 Chapter 3: Green Practices

Existing research in green computing has led to several recommendations for making the data

center green. These green practices have made their way to being best practices for any

organization. This section lists these green practices and describes each of them.

Server Consolidation

Most of the servers running in a data center are running close to idle. The servers are not very

efficient when run at idle, and thus consume a lot of power. However, at higher utilization levels,

the servers are more power efficient. Thus, if we consolidate many applications onto a fewer

number of servers, the servers can be run at higher utilizations and would be more power

efficient.

Virtualization

From a power conservation perspective, by virtualizing and consolidating a number of

applications on one server, all of which were previously being run on multiple physical servers,

two beneficial changes occur [6]. First, all of the other servers can now be shut down, and

therefore, the power they were consuming previously is saved. Second, the physical server on

30
which the applications are consolidated will spend much less time at idle, and therefore, its

otherwise wasted idle power will be significantly reduced. Virtualization and consolidation can

be done statically, i.e., a number of applications being run on multiple servers can be virtualized

and consolidated permanently on a single physical server, but consolidation of multiple virtual

servers on a single physical machine can also be done dynamically, which could provide even

greater flexibility and power savings, discussed below under the heading Heterogeneity

Awareness.

Dynamic Voltage and Frequency Scaling

Dynamic voltage and frequency scaling (DVFS) is another method for reducing power, and can

reduce both unnecessary server idling, which consumes power with no benefit, and also over

performance of server systems, where the server executes the application at a higher performance

level than is required by the relevant SLA, and therefore also uses power unnecessarily. This

method is highly dependent on hardware characteristics, and perhaps on the ways in which those

hardware characteristics can be manipulated. Some server systems allow CPU voltage and

frequency scaling (VFS) to be set in the system BIOS; others allow VFS to be changed

dynamically by the operating system; some systems also allow VFS to be done by applications.

W. Bircher and L. John [13] investigate dynamic frequency scaling, specifically in multi-core

processors, but also illustrates the very significant power savings that can be achieved with little

or no loss of performance. [14] shows a useful approach to control voltage and frequency scaling

31
dynamically using a feedback control loop, and accompanying significant reductions in power

consumption.

Smarter Cooling Solutions

Two papers make differing, but persuasive, arguments that data center power consumption

cannot be optimally controlled without managing both IT power and cooling power in a

coordinated fashion. Nathuji et al. argue that managing IT power and cooling power

independently may lead to less than optimal results [16]. For example, a particular workload

allocation to some set of VMs on certain physical servers may result in minimum power

consumption on the IT side while executing that workload, but if this workload allocation results

in hot spots in the data center, so much additional power might be required on the cooling side to

remove the excess heat from the hot spots, that any power savings on the IT side will be negated,

or even exceeded by, the additional power that is needed for cooling. Further, while a different

allocation of the IT load may be less than optimal with respect to IT power consumed or with

respect cooling power consumed, it may still be the best option from the point of view of

combined IT and cooling power. Clearly, since both IT and cooling consume significant

amounts of power in the data center, an approach must be adopted which minimizes the net

power consumed, rather than minimizing power on either side independently, which will often

result in a less than optimum solution with regard to the net power consumed.

32
Niles makes the somewhat different argument in [17] that, not only must IT power and cooling

power be considered and managed in a coordinated fashion, but beyond this, that cooling

infrastructure in the data center must provide for row-based cooling, i.e., more granular control

of cooling. This is necessary, Niles argues, because virtualization, along with high-density

servers, makes dealing with the presence of hot spots in the data center a constant challenge. We

can observe that data centers that do not have granular control of cooling pay a heavy price for

hot spots, because the entire room-based cooling system must be run at a higher level in order to

remove the heat that is created in the hot spot areas. This presents the real possibility that any

power savings from dynamic migration and consolidation of workloads running on virtual

servers will be more than negated by hot spots that are created by running such loads on high-

density servers. The very disconcerting result is that one of the principal advantages of

virtualization for reducing power usage in data centers is lost. Nathuji et al., however, propose

an approach for addressing this challenge which does not appear to require row-based cooling

[16]. The key idea is to make use of fine-grained intelligence gathered regarding the ability of

the cooling system to dissipate heat, and regarding temperature effects in the data center, to

avoid the occurrence of hot spots in the first place.

Continuous Monitoring and Redeployment (Dynamic Migration)

S. Niles, in an APC white paper, discusses benefits of virtualization, including the fact that it

enables dynamic migration and consolidation of workloads based on IT resource demands [7].

When a physical server is being utilized at a low level, its virtual servers can be migrated to

33
another physical server. This provides the opportunity for physical servers to be dynamically

powered up or shut down in response to changing loads, which reduces total data center power

consumption [8]. Such an ability to migrate and consolidate workloads supports a dynamic

architecture for data center power management, because it enables dynamic allocation of data

center resources based on the current demand, which is subject to significant fluctuation.

Recent research has focused on models and mechanisms for managing virtual machines (VMs)

in the data center, in order to manage dynamic migration, and to take maximum advantage of the

power savings that virtualization and consolidation offer [2, 9, 10, 11].

Heterogeneity Awareness

R. Nathuji et al. discuss yet another advantage of virtualization [12]; namely, it provides a looser

coupling between the IT load and the underlying physical platform. This loose coupling can be

leveraged by seeking optimal matches between workload characteristics and the hardware on

which it is run. In typical data centers, the heterogeneous nature of the physical platforms present

in the data center presents the opportunity to save significant power by matching load

characteristics and hardware. Nathuji et al. report an average reduction in power use of 20% for

one such approach [12]. Our approach to power modeling also supports such matching, by

allowing characterization of the load in terms of resource requirements, and by facilitating the

identification of hardware, through examination of its power consumption in terms of resource

utilization, which is optimal for the load.

34
 Cloud Computing

Cloud computing is a style of computing in which dynamically scalable and often virtualized

resources are provided as a service over the Internet. A number of advantages of cloud

computing for the enterprise have been recognized. Among these are improved TCO, reduced

infrastructure costs, improved business agility, converting fixed costs to variable costs, and of

course, with respect to power, acquiring IT services which can be scaled as needed, even on a

temporary basis to deal with peaks in IT requirements, and which do not impose power capacity

constraints [18]. Although enterprises have been hesitant thus far to adopt cloud computing

broadly as a source of IT services, there are cases where cloud computing might be

advantageous, without raising the typical concerns about security, reliability, meeting SLAs, and

other issues raised by cloud computing which have not yet been fully resolved. [18] suggests that

there are four cases where the cloud should be considered by CIOs, namely, for: (1) new

initiatives where budgets are very constricted; (2) business processes which have widely varying

or unpredictable load patterns; (3) services provided by non-core systems which are

commoditized; and (4) systems where infrastructure management and operations costs are high. 5

Our own view is that, although cloud computing can be part of an overall capacity management

strategy at the present time, it will probably continue to play a somewhat limited role for most

enterprises, until the issues which currently limit its use can be satisfactorily resolved. There can

be micro consolidation, where we consolidate virtual machines onto physical servers within the

data center, and also macro consolidation, where we consolidate virtual machines over

35
geographically separated areas allowing us to reduce cooling costs by taking advantage of

climatic and weather differences of these geographically separated areas. For instance, we can

run a high compute load in America during the night, and then during the day, we can run it in a

data center in China, thus saving on cooling costs by always having the application run at night,

when the atmosphere has significantly lower temperatures.

Strategic Replacement of Hardware

Strategic hardware replacement attempts to replace older, less efficient hardware with newer

systems which have been designed with features which support power conservation. Once power

usage can be accurately modeled, we can determine if replacement of certain hardware would

result in a large enough power reduction, and if it will offset the total cost of the hardware and

the person hours required to replace it. In particular, it is anticipated that certain server systems

in typical DCs may be more heavily utilized for certain commonly used applications, and thus

may be using a significant fraction of the power in the DC. If these systems were replaced with

more modern energy efficient hardware, the total power savings could be considerable. Such a

strategy can only be undertaken, though, once power usage can be reliably modeled at the level

of server system resource utilization. Several major hardware manufacturers are offering server

systems that are significantly more energy efficient than in the past [15]; again, though, the more

information DC managers have about the resource utilization of their applications, the better the

choices they can make with respect to the optimum characteristics of hardware they choose to

replace less efficient systems.

36
 Chapter 4: Solution Approach

The approach taken is to have a model of the datacenter, where we can trace the use of power

down to the level of individual transactions performed on the data center. Doing this helps us to

quantify the value generated by the power spent, and hence provide the business stakeholders

with this information specifically the power consumed by applications. They can use this data

to make decisions concerning the data center.

This approach uses resource utilization as a proxy for power. The incoming requests need some

resources for their execution. The incoming requests can be converted into their corresponding

resource utilization, and then a model for power can be used to convert the resource utilization

into power. This section gives an overall view of how this process is carried out.

Traceability in the data center from requests to power

As requests for service are made, these incoming requests are monitored and a trace is

maintained of all the applications that are used to execute the request. A request usually interacts

with more than one system. Figure 6 describes how the execution of every incoming request can

be split up into executions of transactions serviced by various applications.

37
 Figure 6: Tracking the application usage of each incoming request

We can therefore see that every incoming request can be split up into multiple transactions which

act as requests for service for various applications. Now, each application ideally runs on at least

one server, and in turn requires resources to execute the transaction. Figure 7 describes how

every application requires multiple resources to service a transaction. The usage of these

resources can be programmatically collected from the operating system, or can be approximated

by knowing the type of transaction.

38
 Figure 7: Tracking the resource usage of each application

Every resource on a server is a hardware device and these pieces of hardware consume power.

(Except in a virtual machine, where there is another level of abstraction, but each virtual resource

can be mapped to actual resource utilization and power is consumed by the actual resources). If

we study the correlation of resource utilization and power consumption we can have an accurate

model of the power consumption of a particular server in terms of resource utilization. Once we

learn this correlation, we can predict the power consumption of a resource based on how much

the resource is used. This is illustrated in Figure 8.

Figure 8: Converting the resource utilization into power

39
Now that we have studied how the incoming requests can be traced to their usage of different

applications, how the different applications utilize different resources and how the different

resources consume power while servicing the requests, we are at a point where we can connect

the dots and be able to predict the power consumption of an individual request. Figure 9

combines Figure 6, Figure 7 and Figure 8 and provides the big picture of how the power

consumption can be traced to each incoming request.

Figure 9: The correlation of incoming requests to power consumption

40
 Using traceability and profiling to promote an optimized datacenter

What we now have is a method to convert incoming requests into power. This profiling provides

the business with information on how the power is used in an organization. This traceability can

be used to track request types and find out which requests consume the most power. It also

enables the organization to have fine grained information on their power consumption and use

this information to detect and plug leaks in the system.

This section describes how we can use the profiling of applications and machines to understand

the patterns of application usage and power consumption. We can use this traceability to

optimize data center operations.

Machines running at higher optimizations - close to the SLA

By studying the SLA of applications, we can determine the minimum resources needed to

maintain this SLA. Usually, in todays enterprise the applications are given more resources than

required to maintain the SLA. When a single application resides on a machine that is designed

for peak load, we are in a situation where most of the time, we are actually doing better than the

SLA. This may or may not add any business value.

41
With consolidation and virtualization, we have multiple applications on the same physical

machine. We are now in a position to run applications closer to the SLA. One of the ways to

allocate applications/VMs to physical machines is to allocate high business value apps first, and

then proceed to add the lower business value apps, while trying to minimize the total number of

servers used while maintaining the SLA of each application. In other words, we are using the

business value to allocate resources to the transactions that are executed by the apps.

We could make the enterprise even more adaptive if we allocated resources based on the

incoming transaction request. We can collect all the resources in a priority queue, and execute

the higher business value transactions first, while making sure the SLAs for all the applications

are met. We can thus provide higher reliability, and a better service, for the transactions which

have higher business value.

Business value Profiling for SLA

Another area for improvement is aligning the SLA to better reflect business value. Many times,

an enterprise assigns a random SLA to the application. The SLA must be dependent on the

business value and the organization must be able to quantify and justify the SLA requirements of

an application. We call this process Business Value Profiling for SLA. It involves studying the

effects of resource availability on the SLA and trying to meet the SLA with the least resources

possible.

42
 Application profiling

Our method for resource profiling of applications involves collecting a large sample of

subsystem resource utilization data for the application (typically one week or more), and then

calculating several key values for the application. These values include the average utilization for

each type of resource (CPU, memory reads/writes, disk I/O, network), and also the standard

deviation for utilization of each resource. Temporal patterns or cycles can be obtained from the

resource utilization data; for example, some applications may be more heavily used on certain

days of the week, or at certain times of the day. We could also study resource usage patterns with

respect to other applications if we can find two applications that have complementary usage

patterns, we can consolidate these applications on to the same virtual machine. For example, if

application1 has a high CPU utilization and low memory utilization when application2 has a low

CPU utilization and high memory utilization, and vice versa, then Application1 and Application2

could be put together on the same virtual machine. The total effect would be that the virtual

machine has lesser fluctuation in resource usage which makes the behavior of the virtual

machine easier to predict. All of this information is useful for matching the application to the

hardware on which it runs (described in the following section), and if the application is

virtualized, deciding which applications to combine on a single VM.

43
 Hardware Power Profiling

The final step in this profiling is to convert resource utilization into power consumption. The

power consumption values associated with a particular resource utilization vary from server to

server. For example, 30% CPU Load could correspond to 100W power on one machine, and

200W power on another machine. We therefore need a means to profile a particular piece of

hardware and learn the correlation between resource utilization and power for that particular

machine.

A methodology for characterizing the power profile of a computer system was developed. A

more detailed description of the tools used can be found in Chapter 5: Research Contributions. A

high-level description of the methodology is presented here. The first tool is a synthetic

workload generator, which is designed to subject the subsystems of the machine with a wide

range of loads in various combinations. For example, the CPU is run at a range of values from

0% to 100%. Along with the workload generator that is run on the hardware, concurrent system

power measurements are taken using a wattmeter. Subsystem resource utilization can be fetched

from the operating system, and these values are collected at regular time intervals, as the

workload generator is running. Finally, a regression model (a hardware power profile) is

constructed which correlates system power usage with resource utilization. The model can be

constructed using a neural network, or using regression.

44
 Chapter 5: Research Contributions

The previous chapter summarized our approach to solving the data center power management

problem. It described how we introduce traceability in the data center, and introduced the terms

business value profiling, application profiling, and hardware profiling. This chapter goes

one step further and describes how hardware profiling is implemented. Hardware Profiling is a

major contribution of this thesis. Chapter 6: Results, introduces the power predictor, and enlists

the results of this hardware profiling. The business value and application profiling were not

implemented. However, they are described in detail in Chapter 7: Remaining Research. The idea

behind using these techniques is presented and the method to put all these techniques together is

provided as part of the remaining research.

Power Management

This sub section describes, at a high level, how the proposed management mechanism which

monitors the conditions in the data center will be capable of making dynamic changes to reduce

power consumption or meet service requirements.

The data center consists of a large number of physical servers. In a non-virtualized environment,

each physical server may have one or more applications running on it. This is also true for a

virtualized environment, where a hypervisor runs on the physical hardware, and there may be

45
one or more virtual servers / virtual machines (VMs) running on the hypervisor. In turn, there

may be one or more applications running on each virtual server. The process of running multiple

applications/virtual machines on a single server is called consolidation, as previously described

in Chapter 3: Green Practices. Consolidation increases resource utilization and hence increases

efficiency by consolidating the load on a few machines and allowing for the other machines to be

turned off. For intelligent and efficient management of this system, power usage, load, and

business value information will have to be transmitted across the various levels in this hierarchy:

physical machine hypervisor VM(s) application(s); and this information has to be used for

efficient management at each level.

The following diagram is a model of a data center which uses Virtualization and Consolidation.

46
Figure 10: Model of a data center that uses virtualization and consolidation

47
Figure 10 illustrates how incoming requests can be broken down into transactions and

sub-transactions and how these sub-transactions are ultimately serviced by different

applications residing on different machines. The diagram tries to generalize the following

example: A user initiates a web request. The request enters the data center through a web

server. This request is then serviced by the application servers, the database servers, etc,

and while doing so, it triggers changes to the data stored in some reporting servers too.

Traditionally each application ran on a single dedicated server. This guaranteed

availability and reliability, but proved to be very inefficient. The inefficiency was due to

underutilized servers that were employed with over-provisioning to guarantee their

availability.

In the model in Figure 10, the physical servers are virtualized. (i.e. They run multiple

virtual machines at once and the applications run on these virtual machines). A single

physical machine can be used to run multiple virtual machines. Each physical machine

has a hypervisor installed on it which enables all these virtual machines run. Running

many virtual machines on a single physical machine ensures that the physical machine is

fully utilized, or at least that the utilization is at a power efficient level. Running servers

at low utilization levels is inefficient [23]. This is because most servers are designed to

run at a high utilization level.

48
Running multiple virtual machines on a single physical server, however, presents new

architectural challenges. We need to decide which application/virtual machine

combination is good and which is not. For example, running two computation intensive

applications on the same virtual machine probably would not provide the benefits that

could be gained by running a combination of computation intensive and I/O intensive

applications on a single virtual machine. We need to decide which applications to

consolidate on a particular virtual machine.

We can divide the decisions we need to make into two broad categories: Static decisions

and dynamic decisions.

Static Decisions: The architectural decisions that comprise deciding which applications

run on which virtual machines are called static decisions. This requires a study of the

application profile, and understanding what kinds of resources the application requires.

They are called static decisions because these decisions have to be made before the

virtual machines are deployed once they are deployed it is difficult to change the

application mix on the VMs which implies the application to VM mapping is static.

Applications can be moved around dynamically only if they are designed to do so, but

regardless of the application, the operating system and application mix the virtual

machine, can be moved from one physical system to another.

49
Dynamic Decisions: consist of deciding which physical machine a particular virtual

machine should run on at a particular instant of time. This is a decision that has to be

made in real-time, after studying the current workloads of the virtual machines. Most

commercially available hypervisors are able to save the state of a virtual machine, pause

it, and then move it over to another physical machine and then resume the virtual

machine from the state it left off. This is an exciting feature and provides a lot of

flexibility to the data center managers.

To make these two sets of decisions, the infrastructure business unit needs to have

granular data on the current load, the current power consumption etc, and we hence need

a system that provides this traceability information. We need to quantify business value

of applications, so that when we need to make a tradeoff between two applications, we

give preference to the application with the higher business value. This information can be

gained by tracking the incoming requests and quantifying their value and then keeping

track of them as they go through the different levels of the data center. Having such

traceability will enable us to make better decisions when allocating virtual machines to

physical machines. For example, when a physical machine is overloaded, we can decide

to move out a virtual machine to another physical machine, while reducing the impact on

other virtual machines, especially the ones running the high business value applications.

The traceability information will help us make these decisions.

50
All this points out the need to monitor the business value and power costs at different

levels. We need to make some decisions locally and some decisions globally. The

proposal is to have a global power manager to decide which virtual machine runs on

which physical machine. In addition to the global power manager, a local power

managers runs on each physical machine, virtual power managers run on each virtual

machine, and some form of application power manager runs along with each application.

This allows for autonomic management along with global management. Figure 11:

Hierarchy of Power Managers, shows the power managers at different levels of the

system. Business value information is propagated downwards to the global power

manager. The upward arrows indicate the power/resource utilization data moving to the

layers above, and the downward arrows indicate the business value and SLA information

that is sent to the global power manager from the incoming request. The global power

manager uses this business value and SLA information to make its decisions.

In addition to the above, while making these decisions, the global power manager and

cooling power manager need to communicate, so that IT power and cooling power can be

managed in a coordinated manner, to prevent conflicts between the global power

manager and the cooling manager. For instance, if the global power manager consolidates

the virtual machines onto a group of servers situated close to each other, it could result in

a hot spot which is difficult to cool efficiently; specially in a data center that does not

have localized cooling. The entire data center temperature would have to be lowered to

51
try to cool the hot spot. A more efficient allocation would therefore make sure that

the physical servers running the load are evenly distributed across the entire data center.

Figure 11: Hierarchy of Power Managers

The solution described here takes a control systems approach. The global power manager

tries to optimize the allocation of virtual machines to the physical machines by taking

into account the business value and power consumption. This approach has power

managers at different levels, and they work together to come up with the optimal

solution. Such a complex system might not seem to be a good idea, and putting such a

system in place would require implementations at different levels the hypervisor, the
52
operating system on the virtual machine, and also be part of the application

implementation. The reason for taking this approach is that this solution can be applied to

most of the current data centers. A data center is a complex collection of various

applications, operating systems, servers, network devices, power distribution devices and

cooling infrastructure. Each of these subsystems is produced by different providers, and

hence the system is very loosely coupled. The Hierarchy of power managers allows for

this diversity by having different levels of managers which can be implemented

individually by the different producers of the various sub-systems. Also, since the effect

of the power savings can be traced down to each device, we can study the costs

associated with each device and perform a cost-benefit analysis which will help the

organization decide if a particular device is economically feasible for their datacenter or

not.

Power profiling of hardware

The previous section suggested a control systems approach to managing the data center.

This control system needs a feedback loop where current power consumption information

is collected and sent to the global power manager. This feedback loop can be

implemented using watt meters connected to the various devices. However, implementing

such a system in hardware has some drawbacks:

53
 It is expensive and requires a complex set of power information collection

devices. E.g. Watt meters spread across the entire data center.

It cannot provide the data quickly as we need to have real time

information to make the decisions there is a time lag introduced when

external systems have to interface with the global power manager.

It cannot predict the power consumption- this system can only report

historical data it does not allow us to solve what-if kinds of questions

which are needed to predict the impact of architectural decisions on power

consumption.

All this creates a need for a system where we use a proxy for power that can be measured

programmatically using the current IT systems which are already in place. We now

describe how resource utilization can be used as a proxy for power. The rationale behind

this approach is that the power consumed by a system is actually the sum of the power

consumed by the subcomponents (resources) of the system. Thus, if we study the

resource utilization, we should be able to predict the power consumption of the machine.

i.e, the power consumption is proportional to the resource utilization.

54
 Workload Generator

We justified the need for hardware power profiling in the previous section. The detailed

method is described in the next sub-section , which describes how we learn the profile of

a machine. To ensure that the whole profile of the machine is learned, the physical

machine must be loaded at different levels and we must collect data across the entire

range. For example, if we only run the hardware at 20% load, and then run it at 80% load,

we will only learn the machine characteristics at 20% and 80%. The whole profile of the

machine will not be learned. In the current scenario for a large percentage of the time,

servers run at 0-5% CPU Utilization [28]. Thus, if we try to learn the profile of a

machine, and just run the machine for some time, we will only learn the profile for 5-10%

of the maximum resource utilization. If the power consumption curve is not linear, and if

we use the same characteristics for high utilization levels, we might end up with a wrong

prediction of the power consumed.

Thus, we need to generate various loads at different levels that can be used to learn the

profile of the hardware. We use a workload generator to generate this load. The workload

generator generates load for the physical machine. Figure 12: Workload Generator,

shows how the workload generator generates the amount of resource utilization specified

55
by the Resource Utilization Specification. The resource utilization could be varied to

make the workload generator generate various amounts of load. In particular, we generate

CPU, memory, I/O and network loads, because these resources are the ones that consume

the most power [29]

Figure 12: Workload Generator

This method of generating load would suffice, if there were no external load acting on the

system. However, in reality, there is an operating system running on the physical

machine, and this operating system generates some load on the machine. If there are other

programs installed on the machine, they also generate some load. Therefore producing a

specified amount of load is not a straightforward task, and needs a system that increases

or decreases the load produced so that the net effect is that of the desired load. Figure 13

depicts the workload generator implemented with a feedback loop. In this system, the

feedback loop constantly measures the load on the system and the either increases or

decreases the amount of load generated in an attempt to generate the specified resource

utilization.

56
 Figure 13: Workload Generator with feedback loop

The workload generator was implemented to generate CPU, Memory, I/O and Network

Load.

Generating CPU Load

The CPU is the main consumer of power [29], and so we start with generating CPU load.

When we say that a CPU is running at 30% utilization, what we mean is that for a given

interval, the CPU was used for 30% of the time, and was idle for 70% of the time. The

CPU utilization is thus best understood if it is measured as a percentage utilization over

an interval, and not an instantaneous measurement.

57
If the time interval of measurement is small, there will be a lot of variation in the

measured values, and it will be difficult for the workload generator to match the load

needed by accounting for the load specified by the feedback loop. The workload

generator produced very bad results when small intervals (eg. 1 second) were used for the

feedback loop the workload generator kept generating fluctuating amounts of load and

could never settle down. To understand the utilization patterns and select an appropriate

interval to measure the information for the feedback loop, CPU resource utilization data

of a machine was collected without generating any artificial load. The profile can be seen

in the Figure 14.

Here are the specifications of the machine that was used to decide on the measurement

interval:

Operating System: Windows XP Professional (5.1, Build 2600) Service Pack 3

System Manufacturer: Dell Computer Corporation

System Model: OptiPlex GX270

BIOS: Phoenix ROM BIOS PLUS Version 1.10 A06

Processor: Intel(R) Pentium(R) 4 CPU 2.60GHz (2 CPUs)

Memory: 1022MB RAM

Network: Intel(R) PRO/1000 MT Network Connection

58
 120

100

80
1 second
2 Seconds
60
4 Seconds
8 seconds
40
16 seconds

20

0
1 6 1116212631364146

Figure 14: CPU utilization of idle machine

As we can see from Figure 14, the CPU Utilization of an idle machine is not constant, but

varies a lot. The figure shows the consumption when power was sampled at 1, 2, 4, 8 and

16 seconds. It was noticed that as we increase the interval beyond 4 seconds, the

utilization values stabilized. It is easier for the workload generator to match such a stable

utilization. Figure 15 is a magnified view of the idle CPU. From this graph it is observed

that at 8 seconds, the CPU utilization is relatively stable. The feedback loop of the

workload generator, therefore, measures CPU utilization at 8 second intervals, and tries

to generate the remaining load to match the specified load.

59
 90

80

70

60

50 1 Second
2 Seconds
40
4 Seconds
30 8 Seconds
16 Seconds
20

10

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Figure 15 : More granular view of Utilization of an idle CPU

The workload generator was implemented in C++ and was written in a way where a few

classes can be rewritten to port the code to any operating system. Figure 16 describes the

class diagram of the synthetic workload generator with only the sections that generate the

CPU load. WindowsOS, WindowsStatus and WindowsThread are classes that are specific

to the operating system Windows. The CPULoadThread class generates CPU load. It uses

different worker threads to generate load for each processor on the system. The Synthetic

workload generator class is the main class and it accepts the required amount of load and

then sends that number to the CPULoadThread.

60
The WindowsStatus class reads the status of resource utilization from the Operating

System and presents it to the SyntheticWorkloadGenerator class. The WindowsOS class

is used to do things like write an output to the console, and can accept inputs from the

users.

Figure 16 : Class Diagram of Synthetic Workload Generator

61
This program can also generate memory, I/O and network load by replacing the classes

CPULoadThread and CPUWorkerThread with classes like MemoryLoadThread and

MemoryWorkerThread. To extend this program, classes like MemoryLoadThread,

MemoryWorkerThread, IOLoadThread, IOWorkerThread NetworkLoadThread and

NetworkWorkerThread are also derived from WindowsThread. They are not included in

the class diagram in figure 16 for the sake of simplicity. The combined class diagram is

depicted in Figure 18: Class diagram- Processor Memory I/O and Network Load

Generation. The workload generator corresponding to this figure would then generate

loads for different resources on the system.

Generating Memory Load

Memory consists of RAM modules installed on the computer being profiled. Memory

consumes a certain amount of power when the computer is powered on. However, the

most power is consumed when the memory is written to or read from. Thus, if

information on the number of memory reads or writes is collected, we can correlate this

information to the power consumption.

Generating memory reads is not a straightforward task. This is because there is a faster

memory, called cache, which is between the processor and the RAM. Now, when a

62
program reads a certain memory location, the contents of the memory are brought into

cache. Further reads to the same location will not result in memory reads, as the data is

already in cache, and can be quickly fetched. Figure 17: Memory Hierarchy depicts how

the processor accesses the memory through cache.

Figure 17: Memory Hierarchy

Since cache is in between the memory and the processor, we have to generate cache

misses in order to generate memory reads. (When data is not found in the cache, it has to

be read from the main memory. This is called a cache miss). The workload generator

described in the previous section was implemented in C++ and worked on the Windows

platform. Windows provides counters for processor utilization, but it does not provide

counters for memory reads and writes it only provides counters to find out the

percentage of memory currently being used. However, since memory consumes power

63
based on reads and writes, and not on the total memory being used, these counters cannot

be used. Instead, we used the counters for cache misses as an indication of memory

accesses, since every cache miss results in an access to memory.

Cache misses were generated by allocating a huge array usually bigger than the cache

size, and then sequentially reading / writing to the elements of this array through a loop.

Since the array is bigger than the cache, all the elements of the array cannot fit into cache

at the same time, and so some elements must be fetched from memory. By continuously

looping through all the elements of the array, we can generate cache misses.

The MemoryWorkerThreads generate cache misses by using the method described above.

The MemoryLoadThread spins off MemoryWorkerThreads, sleeps for some time and

then wakes up and stops the MemoryWorkerThreads. It then proceeds to sleep for some

more time and then repeats the process. By varying the sleep time, and the size of the

array, we can vary the amount of cache misses (memory accesses) generated

Generating I/O Load

I/O Load in this case corresponds to the number of disk accesses in a given time interval.

When not being used, the disk slows down the spinning and some power efficient disks

64
can also stop spinning. When the disk is accessed, the disk needs to spin and the

read/write head is activated. Windows provides counters to measure the number of bytes

written to/read from per second. Using this information, we can predict the disk activity

and correlate this information to the power consumption of the system. To generate I/O

load, we programmatically read from, or write to a file. By varying the amount of bytes

we read/write, we can vary the disk activity (I/O Load). The IoLoadThread and

IoWorkerThreads work together to accomplish this task. The IoLoadThread starts and

stops IoWorkerThreads based on the amount of load being generated. The

IoWorkerThreads open a file and read data from the file into a buffer, and then they open

another file and write to that file. In reality, the amount of energy used to perform reads

and writes differ from each other. In our case, we just generate/measure I/O bytes/second,

and do not differentiate between reads and writes, because the wattmeter we used does

not measure power at a high level of granularity, and so, both reads and writes appear to

consume the same amount power.

Generating Network Load

The network is the backbone of the data center. It brings in the requests that need to be

serviced, and returns the results back to the users when the request is serviced. The

network load on a system can be studied by observing the network traffic on the network

65
interface cards installed on the system. Windows provides counters to programmatically

read the number of bytes transmitted per second. By correlating this information with the

power consumed, we can account for network load in our process of predicting power

consumption.

The network load is generated by opening up a socket and programmatically sending data

to a remote server. By varying the amount of data sent in an interval, we can vary the

amount of network load generated. Implementing incoming network traffic would require

an external program to connect to the computer through a socket and send data. This was

not done because it would make the profiling dependent on an external component. Also,

the wattmeters used in this case had low granularity and would not see the difference in

incoming and outgoing network load. The network load data was collected as total

network traffic in bytes/second.

All the parts of the workload generator work together to produce load. In the combined

program, a random number generator generates random values for processor, memory,

I/O and network loads. The workload generator then tries to generate the corresponding

load for a certain time interval (Can be specified by the user). Once the time period is up,

the random number generator generates new values for the processor, memory, I/O and

network. This process continues as long as the workload generator is running.

66
The following figure is the class diagram of the combined workload generator, generating

processor, memory, I/O and network loads simultaneously.

67
Figure 18: Class diagram- Processor Memory I/O and Network Load Generation

68
 Chapter 6: Results

The previous chapter described the workload generator that generated varying amounts of

load on the machine being profiled. This chapter describes how the profile is learned, and

presents the results. The information about resource utilization is collected from the

machine being profiled using performance logs. The power cord of the computer being

profiled goes through wattmeter. This wattmeter is connected to another computer that

collects the wattage information in real time. We used a commercially available

wattmeter called Watts Up? Pro ES. Once the experiment is over, the data is collected

and the relationship between the resource utilization and power consumption is learned

using regression. In this case, we used the [R] statistical tool (R version 2.10.0).

Regression

We collected resource utilization data for the processor, memory, I/O and network. The

specific windows performance monitor logs that were used are:

69
 Processor \Processor(_Total)\% Processor Time

Memory \Memory\Page Faults/sec

I/O \System\File Data Operations/sec

Network \Network Interface(*)\Bytes Total/sec

The values for these variables were measured every second, and the wattmeter was also

instrumented to collect wattage information every second. The independent variables for

the regression were Processor, Memory, I/O and Network utilization, and the dependent

variable was power measured in watts.

Intuitively, using more of any of the independent variables should increase the power

consumed. For instance, if we increase the processor utilization from 10% to 30% we

should be consuming more power. Also, if we increase the network utilization, we should

incur an increase in power consumption. However, we do not know how the power

increases is it directly proportional to the resource utilization, or is it proportional to the

square of the utilization? The regression helps us find out the correlation between the

independent and dependent variables.

70
The specifications of the computer used for the regression is given below:

Operating System: Windows XP Professional (5.1, Build 2600) Service Pack 3

System Manufacturer: Dell Computer Corporation

System Model: OptiPlex GX270

BIOS: Phoenix ROM BIOS PLUS Version 1.10 A06

Processor: Intel(R) Pentium(R) 4 CPU 2.60GHz (2 CPUs)

Memory: 1022MB RAM

Network: Intel(R) PRO/1000 MT Network Connection

71
 Without Adding Adding Adding Adding Adding Adding
Variable Adding I/O2 Adding I/O3
Polynomials Processor2 Processor3 Memory2 Memory3 Network2 Network3
2.03e-1*** -1.32e-2 -7.86e-2. -7.61e-2* -8.17e-2** -1.72e-1*** -1.7e-1*** -1.05e-1*** -1.03e-1***
Processor
(2.51e-2) (2.38e-2) (5.69e-2) (6.29e-2) (6.32e-2) (3.38e-2) (3.38e-2) (3.47e-2) (3.53e-2)
3.78e-3*** 6.52e-3*** 6.51e-3*** 6.51e-3*** 1.12e-2*** 1.12e-2*** 9.52e-3*** 9.45e-3***
Processor2
(3.26e-4) (3.07e-3) (3.05e-3) (3.05e-3) (1.34e-3) (1.35e-3) (1.34e-3) (1.35e-3)
-2.43e-5 -2.46e-5* -2.39e-5* -6.16e-5*** -6.18e-5*** -5.22e-5*** -5.17e-5***
Processor3
(2.59e-5) (2.49e-5) (2.49e-5) (1.19e-5) (1.19e-5) (1.17e-5) (1.18e-5)
-7.56e-6 1.03e-4. 9.91e-5 8.1e-6 2.76e-4 1.33e-3*** 2.04e-3*** 1.95e-3*** 1.94e-3***
Memory
(1.18e-4) (1.33e-4) (1.39e-4) (4.52e-4) (5.31e-4) (3.55e-4) (8e-4) (7.95e-4) (7.95e-4)
1.09e-8 -6.07e-8 -1.94e-7*** -3.35e-7** -3.2e-7** -3.2e-7**
Memory2
(4.18e-8) (8.51e-8) (7.74e-8) (1.63e-7) (1.62e-7) (1.62e-7)
4.66e-12 8.41e-12* 1.52e-11* 1.45e-11* 1.45e-11*
Memroy3
(4.95e-12) (4.82e-12) (8.4e-12) (8.31e-12) (8.32e-12)
I/O -6.28e-4*** -7.76e-4*** -8.43e-4*** -8.34e-4*** -8.34e-4*** -1.57e-3*** -1.93e-3*** -1.96e-3*** -1.95e-3***
72

(1.98e-4) (2.28e-4) (3.13e-4) (3.37e-4) (3.37e-4) (1.06e-4) (3.73e-4) (3.7e-4) (3.7e-4)

1.3e-8*** 4.24e-8* 5.03e-8* 5.04e-8*
I/O2
(8.73e-10) (2.88e-8) (2.86e-8) (2.86e-8)
-2.38e-13 -3.06e-13. -3.08e-13.
I/O3
(2.33e-13) (2.31e-13) (2.31e-13)
1.39e-8 5.74e-8*** 6.20e-8*** 6.13e-8*** 6.33e-8*** 6.44e-8*** 6.38e-8*** -2.96e-7*** -3.37e-7***
Network
(1.20e-8) (1.12e-8) (9.92e-9) (9.97e-9) (1.02e-8) (1.01e-8) (1.01e-8) (3.93e-8) (9.29e-8)
1.05e-14*** 1.35e-14***
Network2
(1.07e-15) (6.37e-15)
-6.38e-23
Network3
(1.29e-22)
7.25e+1*** 7.35e+1*** 7.38e+1*** 7.38e+1*** 7.38e+1*** 7.41e+1*** 7.41e+1*** 7.6e+1*** 7.61e+1***
Constant
(2.07e-1) (2.06e-1) (2.78e-1) (2.87e-1) (2.87e-1) (2.19e-1) (2.19e-1) (2.92e-1) (3.7e-1)
Adjusted
0.0416 0.05675 0.05718 0.05714 0.05716 0.06703 0.06709 0.07333 0.07328
R2
Dependent Variable: Power (Watts)
Coefficients of variables (Robust Standard Errors in Parentheses)
With Robust Standard Errors , Number of observations : 14262. Significance levels: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Table 1: Regression Results

72
Table 1 presents the results of the regression. As we can see, just using processor,

memory and I/O, we come up with a regression where the memory and network are not

significant. However, after adding square terms, all the independent variables are

significant. However there isnt a big change in the adjusted r2 value.

The following relationships were noted (Using absolute values for the coefficients):

Power Consumption(Processor) = 1 Processor + 2 Processor2 3 Processor3

Power Consumption(Memory) = 4 Memory 5 Memory 2 + 6 Memory 3

Power Consumption(I/O) = 7 I/O + 8 I/O 2 9 I/O 3

Power Consumption(Network) = 10 Network + 11 Network 2 12 Network 3

If we keep adding more powers, the regression gets better and better as seen in Table 2.

We also add log(variable) and e-1/variable in the table.

73
 Variable Without Adding Adding Adding
Polynomials Polynomials log(variable) e-1/variable
2.03e-1*** -1.03e-1*** 1.19e-1* 2.65e-1***
Processor
(2.51e-2) (3.53e-2) (4.83e-2) (5.52e-2)
9.45e-3*** 3.65e-3* -1.76e-3***
Processor2
(1.35e-3) (1.59e-3) (1.82e-3)
-5.17e-5*** -1.19e-5 2.87e-5
Processor3
(1.18e-5) (1.3e-5) (1.4e-5)
-5.14e-1*** 3.27e-1*
Log(Processor)
(7.65e-2) (1.52e-1)
-8.09*
e-1/Processor
(1.36)
-7.56e-6 1.94e-3*** 1.38e-3. -1.6e-3***
Memory
(1.18e-4) (7.95e-4) (8.95e-4) (9.7e-4)
-3.2e-7** -2.5e-7 2.13e-7*
Memory2
(1.62e-7) (1.76e-7) (1.84e-7)
1.45e-11* 1.16e-11 -8.562e-12
Memroy3
(8.31e-12) (8.94e-12) (9.215e-12)
3.52e-1*** 1.52
Log(Memory)
(7.64e-2) (2.04e-1)
-7.78***
e-1/Memory
(1.17)
-6.27e-4*** -1.95e-3*** -1.82e-3*** 5.72e-4***
I/O
(1.98e-4) (3.7e-4) (3.84e-4) (4.92e-4)
5.04e-8* 5.71e-8* -6.42e-8
I/O2
(2.86e-8) (2.9e-8) (3.31e-8)
-3.07e-13. -3.77e-13 5.42e-13*
I/O3
(2.31e-13) (2.33e-13) (2.63e-13)
-3.17e-1* -3.27*
Log(I/O)
(1.56e-1) (4.65e-1)
1.37e+2***
e-1/I/O
(2.56e+1)
1.39e-8 -3.37e-7*** -1.33e-6. -2.26e-6***
Network
(1.2e-8) (9.29e-8) (1.69e-7) (3.38e-7)
1.35e-14*** 6.12e-14*** 1.03e-13***
Network2
(6.37e-15) (9.16e-15) (1.63e-14)
-6.38e-23 -8.55e-22*** -1.52e-21***
Network3
(1.29e-22) (1.67e-22) (2.74e-22)
2.06*** 4.52***
Log(Network)
(2.86e-1) (7.09e-1)
-2.04e+3***
e-1/Network
(4.29e+2)
7.25e+1*** 7.61e+1*** 4.84e+1*** 1.95e+3***
Constant
(2.07e-1) (3.7e-1) (3.8) (4.21e+2)
Adjusted R2 0.04165 0.07328 0.08481 0.09508

Table 2: Comparing Regressions using different number of variables

74
Next, we use the coefficients we obtained from the regression to predict the power

consumption of some data that was not used in the regression (test data). The predicted

values were then compared with the actual values and the mean square error was

calculated. This is described in Table 3.

Mean Square Error Error

+
Without polynomials 23.54 /- 4.85
+
Adding Polynomials 21.78 /- 4.67
+
Adding Log(variable) 22.54 /- 4.75

Adding e-1/variable 23.24 +

/- 4.82

Table 3: Comparing the errors

For our purpose, it is enough to know that there is a relationship between the variables,

and we just need to be able to predict how much power a particular workload would

consume on a given machine with respect to another machine (i.e. we need to compare

the power efficiencies, and we dont really need absolutely accurate power models).

From Table 3, we can see that there is only a slight increase in accuracy as we add more

variables. We would only need this accuracy if we can use this information to do VM to

Physical machine allocations at this level of granularity. Also adding more variables like

log(variable) and e-1/variable increase the R2 value of the regression, but also increase the

75
error when used to predict data other than the data used in the learning. So, in our case,

we can just use the regression without raising the independent variables to any powers, or

without adding the logarithmic and exponent components.

Using the coefficients we calculated using regression, we can build a power predictor,

which takes the resource utilization as input, and predicts the amount of power that

particular combination of resource utilization would produce.

76
 Without Adding Adding Adding Adding
Adding I/O2 Adding I/O3 Adding Network2 Adding Network3
Variable Polynomials Processor2 Processor3 Memory2 Memory3
6.05e-1*** 1*** 9.2e-1*** 9.2e-1*** 9.8e-1*** 9.9e-1*** 9.41e-1*** 7.18e-1*** 6.33e-1***
Constant
(8.7e-3) (1.54e-2) (2.98e-2) (3.81e-2) (3.39e-2) (3.37e-2) (3.54e-2) (3.37e-2) (3.09e-2)
-5.88e-3*** -2.69e-3*** -2.69e-3*** -4.43e-3*** -4.84e-3*** -3.27e-3*** 1.25e-3 4.12e-3***
Processor
(2.19e-4) (9.22e-4) (1.13e-3) (1.01e-3) (1e-3) (1.06e-3) (1.01e-3) (9.43e-4)
-2.72e-5*** -2.72e-5*** -1.5e-5* -1.19e-5* -2.39e-5*** -4.99e-5** -7.03e-5***
Processor3
(7.36e-6) (8.62e-6) (7.84e-6) (7.83e-6) (8.30e-6) (7.74e-6) (7.35e-6)
9.85e-4*** 6.94e-4*** 6.81e-4*** 6.83e-4*** 3.82e-4 2.16e-3*** 1.72e-3*** 2.09e-3*** 2.23e-3***
Memory
(3.27e-4) (3.03e-4) (2.93e-4) (2.53e-4) (2.85e-4) (4.35e-4) (3.85e-4) (3.98e-4) (3.9e-4)
-1.22e-10*** 1.29e-7*** -2.17e-7*** -6.35e-8 -8.24e-8 -1e-7*
Memory
(2.46e-8) (4.07e-8) (8.26e-8) (6.29e-8) (6.42e-8) (6.5e-8)
-5.89e-12*** 5.76e-12** -4.08e-13 6.25e-14 5.92e-13
Memory2
(1.61e-12) (2.93e-12) (2.12e-12) (2.16e-12) (2.22e-12)
-7.59e-4*** -1.23e-3*** -1.22e-3*** -1.22e-3*** -1.56e-3*** -2.96e-3*** -1.46e-3*** -1.71e-3*** -1.52e-3***
Memroy3
(3.14e-4) (2.9e-4) (2.81e-4) (1.96e-4) (1.84e-4) (3.15e-4) (3.84e-4) (3.9e-4) (3.8e-4)
1.98e-7*** -2.69e-7*** -3e-7*** -3.77e-7***
I/O
(4.11e-8) (9.61e-8) (9.48e-8) (9.32e-8)
77

2.74e-11*** 3.02e-11*** 3.37e-11***

I/O3
(6.57e-12) (6.37e-12) (6.19e-12)
3.76e-7*** 3.6e-7*** 3.63e-7*** 3.63e-7** 3.61e-7*** 3.61e-7*** 3.62e-7*** 8.04e-7*** 1.36e-6***
I/O3
(1.56e-9) (1.55e-9) (1.7e-9) (1.79e-9) (1.76e-9) (1.75e-9) (1.78e-9) (5.42e-9) (1.22e-8)
-9.11e-15*** -3.93e-14***
Network
(1.1e-16) (6.07e-16)
3.74e-22***
Network3
(7.22e-24)
1.67e+2*** 1.67e+2*** 1.67e+2*** 1.67e+2*** 1.67e+2*** 1.67e+2*** 1.67e+2*** 1.67e+2*** 1.66e+2***
Constant
(2.53e-2) (2.43e-2) (2.54e-2) (2.57e-2) (2.53e-2) (2.53e-2) (2.57e-2) (1.83e-2) (1.5e-2)
Adjusted
0.8964 0.9077 0.9079 0.9079 0.9081 0.9084 0.9088 0.9414 0.9544
R2
Dependent Variable: Power (Watts)
Coefficients of variables (Robust Standard Errors in Parentheses)
With Robust Standard Errors , Number of observations : 17952 Significance levels: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Table 4: Regression results for second machine

77
Table 4 describes another machine that was profiled. The following are the specifications

of that machine:

Operating System: Windows Server 2003 Standard Edition

System Manufacturer: Dell Computer Corporation

System Model: Dell Server PE2650

BIOS: Dell Computer Corporation A17 10/9/2003

Processor: Intel Xeon CPU 2.80GHz, 2.79GHz

Memory: 1.5 GB of RAM

Network: Broadcom NetXtreme Gigabit Ethernet

We will use this information to see how we can compare the profiles of the two

machines. The next few figures compare the power consumption curves of the two

machines. Figure 19 shows the differences in power consumptions of the two machines

for varying levels of CPU consumption. Figure 20 compares the Power consumption for

varying MIPS.

78
 Figure 19: Comparison of power consumption of two machines with varying cpu
utilization

Power Consumption v/s Cycles per sec

250

200
Power (Watts)

150
Machine1
100
Machine2
50

0
0
250
500
750

2750
1000
1250
1500
1750
2000
2250
2500

3000
3250
3500
3750
4000
4250
4500
4750
5000
5250
5500
5750
6000

Figure 20: Comparison of power consumption of two machines with varying cpu
utilization in cycles per second

The next figure compares the CPU consumption of the two computers for varying

amounts of memory load. Instead of plotting the Power v/s memory utilization, the figure

contains the slope which is the differential change in Power consumption with increasing
79
memory utilization. We can see that as the memory usage increases, the slope tends to

zero, which means the utilization stabilizes at higher memory utilizations.

Figure 21: Comparision of power consumption of two machines with varying memory
utilization

Using the CPU and memory power consumption curves, we can make efficient

allocations of virtual machines to physical machines. We can combine CPU intensive and

memory intensive virtual machines to increase the utilization of the physical machines,

and hence reduce their power consumption.

80
 Chapter 7: Remaining Research

This chapter describes the remaining two kinds of profiling Business Value profiling

and Application Profiling. Once this work is done, it can be combined with the hardware

profiling described in the previous two chapters. Together, all this information can be

used to implement the hierarchy of power managers.

Business Value Profiling

Data centers house many servers which generate revenue by servicing incoming requests.

These requests can be accesses to a website, a request for a quote, or any other type of

computing service. There is value generated when these requests are serviced, and this

value generates revenue for the company. Every incoming request requires some

resources for its execution. Business value profiling of applications consists of studying

all the requests for servic4e from an application and then quantifying both the business

value generated by servicing the request, and the resources needed.

By using RED (Request-Execution-Delivery) transactions, as posited by the ACE model

[19], we have a way to correlate business value generated by a transaction with the SLAs

for the applications that perform the transaction, as well as correlate the transaction with
81
the power needed to execute it. Executing a transaction may involve various other

transactions as sub-transactions. This view of how requests are serviced in the enterprise

allows a much more precise characterization of all the component services that work

together to generate value for the enterprise. Since our hardware and application power

profiles allow resource utilization to be correlated with power, we can also determine

how much power was used to service a request enabling the enterprise to examine how

much value was generated per unit of power expended. Since applications that service

requests also have associated SLAs, the tradeoff between value generation, SLA and

power can be assessed, and modified if necessary. For example, managers may decide

that a particular applications SLAs should be lowered, because the transactions it

performs are not generating enough value to justify the cost of higher performance, and

correspondingly, higher power consumption.

It is not enough to study the performance of individual applications - the data center has

to be modeled as a whole if we want to be able to optimize it. This section describes a

data center where the business value of the incoming requests is traced to the applications

servicing these requests. The business value is traced further down the chain through the

sub-requests/sub-transactions to the physical machines that actually execute the requests,

and even traced down to their consumption of electricity and cooling power. This tracing

thus allows the enterprise to determine how the business value generated by servicing of

the request is correlated with the power resources needed to service the request.

82
The enterprise architecture model described in this section aims to trace the value of these

transactions down to their power consumption. This is an attempt to have better

traceability of the costs associated with the application, and find out the relationship

between business value, the service level agreements, the operating level agreements and

the actual costs for the physical machines, the power consumption, and other utilities

such as cooling.

The Adaptive Complex Enterprise (ACE) model [19] was used here, where every

transaction is broken down into transactions. Every transaction can be viewed as having

Requests, Execution and Delivery (RED) portions. The organization is interested in

servicing customer requests. Business value is derived by servicing requests from the

customer. The global power manager needs to have information on which requests are

more important than others, and when it makes a tradeoff between performance and

efficiency, it needs to have this information available in a form that it can use. The

business value must be quantified and converted into something that can be compared

with power. We can begin this process by quantifying the business value for different

kinds of transactions.

Todays dynamic enterprise operates in a competitive environment, and provides

customized solutions to its customers. To stay competitive, the enterprise must adopt

practices like mass customization. This involves servicing various non-routine requests.

The non-routine requests cannot have a fixed cost associated with them, due to their non-

83
routine nature. To understand the cost of a non-routine request, we need to have granular

measurements of the resources needed to fulfill such requests.

Non-Routine requests require a better model that tracks the actual cost of servicing the

request. In addition to having continuous improvement, we need to locate areas for

improvement and work towards increasing operational efficiency and decreasing costs.

Currently businesses do not treat IT as a separate business unit, and even if they do, a

fixed price is allocated for IT services. Dividing the business into different levels can

help increase efficiency, and provide a better model that reflects the real costs associated

with IT services. If tracked, the cost for non-routine requests can be accurately

determined, and the customer can be charged based on this information.

We need to treat each level as a business and each level has to track its costs

independently. This will help us to have a more granular view of the associated costs, and

will also create an atmosphere where each unit can optimize its operations and decrease

cost. Figure 22: Treating each level as a business unit, divides the organization into four

levels. The incoming request passes through each level, and at each level it requests

services from other levels. The incoming request to each level is treated as a RED

transaction (which is explained in the section RED Transaction Model), and is broken

down into sub-transactions which are serviced by the lower layers. This allows us to keep

track of the cost of each sub-transaction and also quantify the business value generated by

each transaction.

84
Figure 22: Treating each level as a business unit

85
Treating each level as a business unit provides many benefits. We can allocate capacity

according to the business value and justify the cost for adding new capacity. The cost for

providing the service can be accurately tracked and we can identify areas for

improvement, and then do a cost-benefit analysis before proceeding to make any changes

to the process or organization. Each business unit will have to manage its own assets and

costs, and is thus forced to create profits. This encourages continuous improvement and

makes the business units more efficient. Each level can divide the capacity into Request-

servicing capacity, Infrastructure service capacity, Operational interaction, and

Operational capacity.

The chargeback model has other benefits that include:

Matching of services to business need

Forcing the organization to control expenses and decrease costs

Highlight areas of service provision that are not cost-effective

Disciplining business units by providing a fair consumption-based chargeback

model - nothing is free, or has a fixed charge anymore

Ensuring better alignment with enterprise goals

Pinpointing areas of innovation by finding areas that are inefficient

This process is described in detail in this section. The description covers how Business

value is converted into SLAs and how SLAs are converted into OLAs. OLAs in turn are

converted into resource requests. Resource requests can then provide an accurate estimate

86
of the costs. Examples of services that can be quantified are power, cooling, hardware,

software and maintenance.

Business IT Alignment

Now that we have divided the organization into various businesses, have a better

chargeback model, and we can quantify the cost of providing services. Quantifying costs

and having a traceable workflow is the first step to having a process that incorporates

continuous improvement.

At each stage of improvement, we identify areas of opportunity, where we can improve

the process and reduce operational costs. The Cost-Benefit Analysis Method (CBAM)

specifies how to do this [22]. The different stakeholders in sub-businesses consider the

performance and availability of the system to identify inefficient points, and increase the

efficiency. While doing this, the security of the system needs to be taken into

consideration. Also, the system should be built in a way that is easy to modify. CBAM

helps stakeholders reflect upon and choose among the potential architectural alternatives.

Organizations need to know how to invest their resources to maximize their gains, meet

their schedules and minimize their risks. The Architecture Tradeoff Analysis Method [30]

provides a method for understanding the technical tradeoffs that must be made as design

and maintenance decisions are made.

87
Another important component is the Return on Investment (ROI). Every architectural

decision must take long-term costs into account. Quantifying the ROI can help businesses

make informed choices that would have an impact on hardware buying decisions, and

other decisions that can influence data center operations. These architectural decisions

need to be linked to the business goals and quality attributes.

RED Transaction Model

Another area for research is building a QoS (Quality of Service) predictor which uses

complex parameters to predict the QoS. The aim is to be able to reduce operating costs

while maintaining the minimum QoS. It is another way to make sure the SLA is

maintained.

The Adaptive Complex Enterprise (ACE) Framework [19] is used to model the system,

and describe the correlation between the operating level metrics and the BioS Goals.

BioS stands for Business Infrastructure Operations Strategy, and represent the different

stakeholders in the enterprise. BioS goals are the combined view of the goals of the

different stakeholders. The customer requests are serviced while fulfilling these BioS

Goals.

88
Figure 23 describes an interaction in ACE, and shows how the incoming request is

executed and the service delivered. The BioS Goals have to be met as the RED

Transaction takes place. BioS Goals are:

B Lower power consumption per transaction, i.e., decrease operating cost

I Choose the right servers or resources

O Migrate virtual servers for load balancing, or in the case of physical servers,

Find the right mix of applications to run on them such that the total cost is

minimized

S Promote customer satisfaction by meeting SLAs, promoting a greener tomorrow.

Figure 23: Interaction (source [19])

The overall goal is to transform the input request into the desired output while

maintaining the required QoS. The QoS is a metric to account for fulfillment of the

business goals.

89
We can think of the interaction as a user visiting a website and performing a transaction,

such as buying a book from an online store.

The next diagram, Figure 24, depicts how a customer request event moves from one

interaction to another. Modeling the system in this way allows us to delay the assignment

of roles until the interaction is itself ready to execute. This is called delayed binding. This

allows interactions to execute while allowing the virtual machines to be dynamically

moved across physical servers.

By delaying allocation of roles, we can make the system respond to nonroutine requests

more efficiently, by allocating the most efficient combination at the instant of time when

the request is being serviced.

Figure 24: Executing an incoming request with two interactions (source [19])

90
Another benefit of modeling the system in this manner is that it enables us to identify the

Value-Add and Non-Value Add time from the customer or request originators

perspective. The customer simply wants the outcome of the transaction (the service

requested), and is generally not concerned with the Non-Value Add tasks performed, like

the website keeping track of its inventory, its budget, the shipping options, the price

fluctuation of its suppliers, etc. The user also does not want to see delays due to the sub

transactions such as the database query, perhaps a configurator running in the

background, or a system that is taking time to calculate the power consumption and

deciding on the most power efficient way of servicing the request.

91
 Figure 25: Agile Interactions (source [19])

Figure 25 shows how we can overlap the execution of interactions to reduce the Non-

Value Add time. Agility is achieved here through improved communication and

coordination between the different roles.

92
Customers initiate an Interaction by a Request event as illustrated by the black dot in

Figure 23, Figure 24 and Figure 25. The request then moves through the Requirements,

Execution and Delivery stages reaching the Agreement, Customer Business, and

Provider Business milestones respectively.

Figure 26: Red Service Interactions with BioS Values (source [19])
The objective of the RED Interaction structure is to make explicit important transitions

from all perspectives and points of metrics collection. The perspectives are:

Customer perspective

Conceptual RED Interaction and performance perspective

Provider perspective

93
RED Transactions can be broken down into smaller sub-transactions. Each sub-

transaction may be considered as a RED transaction of its own. Coming back to our

earlier example of a user buying a book at an online store, this transaction can be broken

down into creating a session, buying the book, and having it shipped to an address. These

RED interactions can be broken down further into RED transactions on the different

applications performing these actions. We can take it a step further by breaking these

transactions down into http requests, database queries and other external transactions with

external systems such as credit card companies and shipping companies. At every step,

we collect metrics that correspond to the amount of work done, and relate this to the BioS

goals. We thus have a system where the business value can be traced down to individual

agents working in the enterprise.

Lets look at this process in detail with the help of an example. Consider a database

server. The server has to meet a particular SLA; This could be a certain amount of time

each database transaction has to be completed within. As long as we complete the

transaction in no more than the maximum time allowed, we meet the SLA. We can

execute the transaction faster, but it does not, strictly speaking, add any business value

we just need to meet the SLA, not exceed it.

We have now described the problem in terms of a single server and have to provide the

infrastructure and operational managers with a system that helps them make decisions. At

the same time, the overall process of collecting and merging this information, can be used

94
to consolidate the information gained and provide the business and other stakeholders

with better metrics to make strategic decisions.

Defining an SLA curve

To meet this transaction processing time, we require a certain amount of CPU, memory

and I/O. The power consumption of the server can be thought of as the power

consumption of these individual components. Servers are designed to be efficient at

certain combinations of these parameters.

As seen in the diagram, we have the software/application profiling where we are

learning how the SLAs of that particular application can be met by various combinations

of utilization of the resources. We also need hardware profiling which is learning how

the resource utilization affects the power consumption of that particular hardware.

The question that remains is how we relate the information learned from the hardware

and software profiling, and use it to decrease power consumption. The following diagram

describes that methodology at a high level.

95
 Figure 27: SLA and Power relationship using two RED Transactions

In Figure 27, the work is divided into two RED transactions one from the customers

perspective and the other from the providers perspective. The customer is concerned

with obtaining his requested service in a minimum amount of time. As long as the SLA is

met, the customer is happy. A faster response, though, would improve the customer

experience.

The provider has the task of providing the infrastructure for the customer. He has various

ways of satisfying the requirement by designing the system to provide different resource

utilizations each having a different impact on the power consumption. Lower power

96
consumption means increased savings in power and cooling costs; however, increased

resource utilization would mean increased usage of power. In addition to increased cost,

however, there is also a physical limit to the amount of resources that can be provided,

because of hardware and power capacity constraints.

With these two RED transactions in place, the problem now boils down to selecting the

right combination of resources to meet the SLAs while minimizing power consumption.

This can be done by the infrastructure and operational managers.

The business and strategic managers can look at this system of metrics and play what if

games such as reducing the power consumption to a point where the SLAs are not met,

and study how much it costs them when they do not meet the SLAs and quantify the

savings achieved through a reduction in power consumption. If the SLA is very high, or

overly stringent, then a slight decrease in the SLA might not cost much, but could result

in huge energy and cooling savings. This system provides the stakeholders with tools to

make intelligent, well-informed choices and would result in significant savings.

OLA for resource

To perform optimization what we need is not a single SLA value, but a curve that

displays the relationship between SLA and business value. Having such a curve will help

us understand the tradeoffs associated with lowering the SLA and its effect on business
97
value. It will also help us justify the choice of the selected SLA. Examples of some useful

curves would be Business Value vs. SLA and Business Value vs. Performance.

If the required SLA is over-estimated, and the cost of providing that high SLA is not

justified, we would need to reduce the SLA and cut costs, especially when the cost

involved could be detrimental to the environment, for example, the case of having a

larger carbon footprint.

Conversion of SLA into OLA Units

The SLA provides a metric which represents the value delivered to the customer. It could

be for example, the response time of the company website. This is useful to quantify the

satisfaction of the customer, and to achieve business goals. However, to optimize

operations, we need to have the SLA translated into a different value called the Operating

Level Agreement (OLA). The OLA corresponding to the previous example would be the

availability of the system, or some performance metric. Translating the SLA into the

OLA is an important step in quantifying and justifying the OLAs in the organization too,

since we can now have a mapping of the business value to an OLA, and argue whether

the operational costs are justified.

98
Conversion of OLA into Role Sets

The infrastructure departments of the organization are involved in allocating resources to

the other business units, and charging those units for the services. Resources include

pieces of hardware like servers, and other utilities like power and cooling. The

infrastructure business unit can measure power consumption and cooling costs associated

with providing the service. Inefficient servers can be replaced by efficient ones, and

operational costs can be reduced. Sometimes saving power involves an associated

performance loss. High business value applications could be impacted by such losses. To

make the most cost-effective decisions, we need a mapping of the OLAs to role sets. Role

sets enumerate the resources needed to fulfill the OLA. The resources could be the

amount of computing power needed, the amount of bandwidth needed, or other such

metrics. Having the OLARole Set mapping allows us to quantify the business value

associated with the roles, and push for continuous improvement in the infrastructure

business unit.

99
 Application Profiling

Application profiling consists of studying the resource utilization of the application over

a period of time, and understanding how the application uses the different resources. , By

having this information we can figure out things like when the application is used the

most, which resources it uses, and use this information to optimize the resource

utilization.

There are various tools that can be used to collect resource utilization data. Windows has

performance monitor counters that can be registered with the OS, and logs could be

collected. HP has a tool called SiteScope, which collects resource utilization data from

the machines in the data center.

This information can be studied to find patterns in the usage of power. Once the patterns

are learned, we can use the information to make better allocation of resources.

100
 Normalization of resource utilization units

Although our approach offers many advantages, as explained above, there is also a

complication which must be overcome for it to work well. Specifically, the resource

utilization units on different hardware will not be uniform, and therefore must be

normalized, so that the management system will have a consistent unit for decision

making. For example, if an applications power profile gives its average CPU utilization

as 8% on one piece of hardware, it may be a significantly different value on different

hardware with a different clock speed, instruction pipelining design, or instruction set

architecture. This normalization can be done using statistical techniques.

Improved Chargeback Model

Since our approach allows tracing of power use in the data center, and tracing of request

servicing through RED transactions, it also offers the benefit of an improved chargeback

model. A general difficulty with chargeback is that it is difficult to measure how much

capacity or how much of a resource a certain business unit is using, and therefore, it is

difficult to charge the business unit in a way that accurately reflects its use of resources.

The problems created for the enterprise include the occurrence of free riding by some

business units, which use a lot of resources but are only charged a fixed cost. By
101
improving chargeback, free riding can be minimized or eliminated, and this is an

incentive for all business units to consume only the resources they need enabling

greener habits in the organization.

Resource Utilization into costs

We have now gone through the process of converting incoming requests to SLA

requirements, SLA requirements to OLA requirements, Converted OLA into resource

utilization, and we are now at a point where we have to convert resource utilization into

costs.

The major cost associated with resources is power consumption. This is addressed in The

subsection Power along with other ROI stuff. The next subsection Other Costs

describes the other costs associated with servicing requests. Finally subsection Side

effects and Constraints describes the side effects and constraints that contribute to the

costs.

102
 Power along with other ROI stuff

The resources in the system have costs associated with them. These costs can be

procurement costs, operational costs, and maintenance costs. The enterprise has many

shared resources and this makes it very difficult to quantify the business value of these

resources.

Role sets are groups of resources needed to perform a task. Using role sets allows us to

model the shared resources phenomenon. A single resource can be a part of more than

one role set. By quantifying the time, or percentage consumption of a particular resource,

we can divide the costs incurred in supporting that resource. The costs can then be

compared with the business value provided by the role set, and thus the business value

associated with a resource can be quantified.

103
 Other Costs

The chargeback model provides a means for capacity management while maintaining

flexibility. Resources are not tightly coupled with the applications, but are provided using

various role sets. The chargeback model also takes power consumption into

consideration. Quantifying costs of resources allows us to improve throughput and reduce

costs. There are also savings associated with licensing of applications. When we

consolidate applications, there is an opportunity to use fewer licenses as a side effect,

specifically for applications that are licensed per server.

Other costs to be included in the chargeback model include the fixed costs of the facility,

and costs of the server hardware. Having the business value traced to the lower

operational levels allows us to quantify the value generated by each resource and

prevents the practice of charging the same cost to all the users of the infrastructure. The

overhead costs should not be charged evenly, but should be imposed according to

utilization of the resource. This allows for a more accurate calculation of ROI.

An additional benefit which impacts customer relationships too, is making the enterprise

environmentally responsible, measuring its carbon footprint, and quantifying the decrease

in power consumption and the reduction in the organizations carbon footprint. To

improve energy efficiency and promote competition, it would be of great value if the
104
organization could track power consumption and move this information up the value

chain and deliver this information to customers along with the response to the incoming

request. One way of doing this is using a metric for energy use. For example, we might

convert the carbon footprint associated with servicing the request to a value like number

of leaves the equivalent carbon footprint of a leaf. This idea could be extended to larger

groups of transactions also, (100,000 leaves = 1 Tree, 100,000 Trees = 1 Forest, and so

on). The customer could be provided with these values, and they can compare the number

of leaves they use for a transaction performed on different competitors websites. Tracing

metrics like this can increase customer loyalty and promote better customer relationships.

Side effects and Constraints

There are some costs that cannot be related to any particular business application. We call

such costs side effects. Examples of side effects include the heat generated by systems

and the cost involved with cooling. Side effects are unwanted outputs of the system,

which cannot be prevented. One way to trace this cost is to correlate it with the resource

utilization and include it as a resource utilization cost. Another way of modeling this cost

is to consider side effects as requests generated by the system, and then there is a cost

associated with fulfilling this request.

There are also some constraints that prevent the application of this optimization approach

to the entire organization. An example of such a constraint is an application that cannot

105
run on a virtual server simply because the vendor of the application does not support the

application running on virtual machines. Another example could be an operating system

needed for legacy reasons that cannot be virtualized due to technical reasons. There are

other such constraints that need to be modeled. One way of doing this is to make the

constraints part of the role specification, and treat these roles as separate resources that

cannot be shared and so every business unit is assigned a fixed cost to reflect the use of

these shared resources.

106
 Chapter 8: Application of framework after completing the remaining research

The adoption of the complex framework described in the previous sections requires a

number of changes to the existing organization. Building the framework from scratch is

easy, but todays enterprise poses a challenge where we need to add traceability to

existing systems, and monitor power of the existing physical infrastructure. We therefore

need a methodology with a step by step approach, which helps us to incrementally add

features to the current systems. The section Measure Indicators describes the different

metrics that are accepted industry standards. At each stage of the continuous

improvement methodology, we come back to these standard metrics to check if the

overall efficiency of the data center is increasing. The section Prioritizing

Opportunities specifies areas of opportunity, where quick changes to the infrastructure

can result in sizable increases in power efficiency. The next section discusses a

continuous improvement methodology.

107
 Integrated Continuous Improvement

This section presents a continuous improvement methodology which outlines a process of

continuously making changes to the existing infrastructure. Every step works towards

creating a greener infrastructure in the data center. We now describe the various steps in

the Integrated Continuous Improvement Methodology.

Figure 28: Continuous Improvement Methodology

108
 Identify Problems

Problems are opportunities to increase efficiency and decrease costs. The first stage to

improvement is therefore, identifying these problems. We start with the problems that

give us the greatest gain when solved.

Identify Solutions

If the change involves adding new hardware, we must learn the profiles of the new

hardware and compare it with the hardware profiles for existing resources. Solutions

consist of fixing inefficiencies in the system. We can use the RED analysis explained in

the previous chapter to help solve the problems by finding the inefficient points in the

system.

After completing the RED analysis, we build an application profile using the resource

utilization logs. The methodology for performing this application profiling is described in

The section titled RED Transaction Model. Now that we have the application profiles,

and know the resource usage patterns for each application, we can use different

algorithms to find solutions, for example using a bin-packing or heuristic algorithm for

efficient allocation of virtual machines to physical machines.

109
 Apply Solutions

After identifying the solution, allocate people and time to apply the solution. The solution

could be a simple measure like enabling DVFS, or it could be something as complex as

implementing traceability in a business unit.

Measure Indicators

We need to make sure we are making progress at every iteration of the continuous

improvement cycle. If we do not, it could mean that the power saving measures that were

just introduced could be interfering with existing measures.

At every stage in the continuous improvement model, we should use industry standard

metrics to verify if the data center operation is increasing in efficiency. We use the

following Green Grid metrics to help us validate the changes we make at each level. PUE

stands for Power Usage Effectiveness and DCIE stands for Data Center Infrastructure

Efficiency.

110
 PUE = Total Facility Power
Total IT Power

DCIE = 1 = IT Equipment Power x 100%

PUE Total Facility Power

Power profiling of machines eliminates the need to have physical granular measurements

of power. The power consumption can be predicted based on the hardware power profile

and resource utilization. However, periodically we need to measure power physically to

make sure that our predictions are right and that there is an actual increase in efficiency in

the data center. The PUE and DCIE calculations need these physical power

measurements.

The continuous optimization process also consists of continually increasing the

granularity at which the improvement cycle is done. Initially, the judgments and

evaluation required can be undertaken by subject matter experts in the relevant areas.

Next the hardware profiling, application profiling, and power predictor components can

be done only for hardware and applications which are believed to consume a lot of power

in the data center. As more and more hardware and applications are profiled, the dynamic

management capabilities could be implemented partially in the data center, again

beginning with large power consumers, and on each subsequent improvement cycle,

more and more hardware and applications can be added. This approach spreads the

111
implementation work out over time, but allows the enterprise to reap significant benefits

on each optimization cycle.

Prioritizing Opportunities

This section provides guidelines on how to prioritize the changes needed in the enterprise

and also, how to get the maximum benefit out of each iteration.

Initially - Static Optimization

To begin with, we look at areas of opportunity where static measures can lead to

significant energy savings. These changes are called static changes because they remain

in place till the next measure is implemented, i.e. they do not change between the time

they were first implemented to the time the next change is put in place. Some of the

things that can be done are described below.

Software Considerations - Business Value v/s SLA

Figure 29 is a graph that plots the Business Value versus the Service Level Agreement

associated with an application. The graph helps us to identify which applications to

112
consolidate/virtualize first, and gives us the order to consider the various applications for

optimization

Figure 29: Application Business Value v/s SLA

We first consider applications which have a low SLA and low business value. These

applications are considered first because they have the least impact on the system they

have low SLAs - they do not demand fast response times, and secondly, they have low

business value, so any impact on performance that might arise as a side effect to

consolidation will be unlikely to be detrimental. Next, we tackle the applications that

have a low business value, but have a high SLA. The high SLA means that we cannot

consolidate too many servers together, but we can attempt to consolidate some of them.

113
These applications might sometimes have an SLA that is too high. We can reduce the

SLA and thereby reduce the cost associated with maintaining the high SLA. This can be

justified because of the low business value of the application. The next set of applications

to consider are the ones that have a high business value but low SLA. These applications

can be consolidated/virtualized because of their low SLAs, which allow for

consolidation. Finally, we can consider the applications which have high business value,

and demand high SLAs. These applications are very crucial to the organization as they

generate a lot of revenue. Care should be taken when consolidating/virtualizing these

applications because a drop in the SLA can have huge financial implications.

Hardware Considerations- Hw cost vs performance

We need to consider the hardware, and replace less efficient pieces of hardware with

more efficient ones. Figure 30 shows the relationship between cost and capacity for

different servers in the data center. The costs are the fixed costs for the physical hardware

and the utility cost of running the servers.

114
 Figure 30: Hardware Cost v/s Capacity

We first identify the servers that have a high cost, but low capacity. These servers

consume a lot of energy, but have a very low computing, memory or I/O capacity. These

servers must be replaced with more efficient servers that provide a higher capacity for the

power they consume. Next, we consider the machines that have a low cost and low

capacity. These machines have low costs associated with them, and they also have a low

capacity, so ideally they would be running applications that do not require high capacity.

These servers could be consolidated and put into a single efficient physical server that has

a higher capacity. After this we take a look at the servers that have a high cost associated

with them, and demand a high capacity. These servers are usually running high business
115
value applications that require a huge capacity. However, they have a huge cost

associated with them. These servers could be running critical applications and so, the

effects of changing the hardware must be carefully studied before replacing the hardware.

Finally the servers which have a high capacity and low cost are considered for

replacement. These servers are efficient, but they have to be refreshed periodically with

more efficient machines, as part of the periodic hardware refreshing cycle.

Allocation of Resources

Allocate resources to increase utilization and minimize consumption of power. This is a

problem where we minimize the total number of physical servers running, but maximize

the resource utilization of each physical machine by moving the virtual machines around.

Consolidation of Servers

Each server runs a different application. For instance, if there are multiple web

applications, each one is run on a different physical server. Most of these servers run well

below their capacity and if we consolidate multiple applications onto a single server, we

can save a lot of power. For example, consider a machine that consumes 60W at idle, and

100W at peak load. If two machines are running, each will consume at least 60W,

totaling 120W. The actual consumption can be anywhere between 120W 200W. But if

we consolidate the servers their consumption will be something between 60W 100W.

This clearly is a huge savings.

116
The first step therefore, would be to identify servers that are relatively idle, and

consolidate them into single machines. We could use the application profile to study the

time at which applications consume resources and consolidate servers that use different

sets of resources at different intervals in a way that the usage profiles complement each

other. We could also use expert opinion to decide which servers have to be consolidated.

Replace servers with more efficient ones

Many vendors have started producing power efficient servers. Old servers consume a lot

of power, and replacing these servers can save a lot of energy. The company must have a

server refresh methodology that promotes the use of more power efficient servers.

Servers that have dynamic voltage and frequency scaling (DVFS) must be bought, and

the DVFS capabilities must be turned on. Other power saving features must be activated

and the latest BIOS updates installed on the systems.

Encourage Virtualization as a means for better server consolidation

Virtualization is a technology which allows multiple virtual machines to run on a single

physical server. A hypervisor runs on the hardware, and it provides an environment to run

many virtual machines. Virtualization must be encouraged in the organization as virtual

machines are easier to move around and hence allow for dynamic changes in the system

117
organization. Consolidation, or running many virtual machines on a single server,

increases the utilization of the server and reduces running costs.

Cooling Optimizations

Cooling consumes a lot of power. Possible cooling optimizations include having hot and

cold aisles, where machines are positioned in a manner where there are alternating hot

and cold aisles. In a cold aisle, all the fronts of the servers face each other. Cold air

emerges from perforated floor tiles in the cool aisle. Air from the cold aisle is sucked into

the server. Hot aisles are aisles that have backs of all the servers facing each other. The

aim is to get the hot aisles as hot as possible, and the cold aisle must be as cold as

possible. This increases the circulation of air, and allows for better heat transfer from the

machines.

Figure 31: Hot and Cold Aisle Approach [25]

118
 Demand Management of Services

Demand management consists of conducting a study on the applications that consume the

most power and check to see if the SLA is too high. In other words, match the SLA to the

business value.

Ongoing - Dynamic Optimizations

After going through all the above changes, the organization has to make sure that it is

continuously improving its efficiency. It can make these decisions based on continuous

measurements- as the measurements get more granular, the system will be better

understood. We can now use the hierarchy of power managers, where each power

manager makes local decisions and the global power manager makes dynamic decisions

about which physical machine should a particular VM run on.

Having a joint optimization of power and cooling could make the data center very power

efficient. The aim is to put the organization in a Continuous Improvement frame of

mind

119
 Conclusion

Green computing is constantly becoming more relevant, and many organizations are

working towards reducing the carbon footprint of their data centers. This reduction in

carbon footprint is achieved by reducing the data centers power consumption, which in

turn results in savings for the organization. Many new techniques have been used to

achieve this reduction in power. One of them is virtualization.

Virtualization helps us consolidate multiple servers onto a few physical machines, which

increases their utilization, and decreases their power consumption. However,

virtualization also provides us new possibilities it allows us to increase flexibility in the

data center by allowing us to dynamically move virtual machines from one physical

machine to another. However, such a complex system needs a good management system

to be effective. We need a system that takes power consumption and cooling power into

account, in addition to server load, while allocating virtual machines to servers. There is

thus a need to measure power consumption, and an even greater need to be able to predict

the power consumption effects of changes in the data center.

We saw how resource utilization can be used as a proxy for power how it could be used

to measure current power, and also how it could be used to predict future power

consumption. Putting this resource utilization information together with cooling

120
information, business value and SLA information and of course the traditional server

workload information results in a mine of information that could be used to make

intelligent and efficient allocations in the data center. Future implementations can take

this information and implement the comprehensive data center management architecture

specified in this thesis.

121
 References

[1] MacKinnon, C. The changing data center life cycle: one day here & the next day

gone. Processor, December 22, 2006.

[2] S. Kumar, V. Talwar, P. Ranganathan R. Nathuji, and K. Schwan. M-channels and m-

brokers: new abstractions for co-ordinated management in virtualized systems. In

Proceedings of the Workshop on Managed Many-Core Systems (MMCS), June 2008.

[3] R. Nathuji, Mechanisms for coordinated power management with application to

cooperative distributed systems. PhD thesis, School of Electrical and Computer

Engineering, Georgia Institute of Technology, 2008.

[4] D. Narayanan. Power by proxy: a new aprpoach to measurement. August 2002.

research.cs.ncl.ac.uk/cabernet/www.laas.research.ec.org/cabernet/workshops/radicals/200

3/papers/paper.pdf_5.pdf.

[5] C. Isci and M. Martonosi. Runtime power monitoring in high-end processors:

methodology and empirical data. In Proceedings of the 36th International Symposium on

Microarchitecture (MICRO-3603), December 2003.

[7] S. Niles. Virtualization: optimized power and cooling to maximize benefits.

American Power Conversion White Paper, 2008.

[8] J. Chase, D. Anderson, P. Thakar, A. Vahdat, and R. Doyle. Managing energy and

server resources in hosting centers. In Proceedings of the 18th Symposium on Operating

Systems Principles (SOSP), 2001.

122
[9] R. Nathuji and K. Schwan. VirtualPower: coordinated power management in

virtualized enterprise systems. In Proceedings of Twenty-First Symposium on Operating

Systems Principles (SIGOPS), October, 2007.

[10] S. Kumar, V. Talwar, P. Ranganathan, and K. Schwan. vManage: coordinated

management in virtualized systems.

www.hpl.hp.com/personal/Partha_Ranganathan/papers/2008/2008_hotac_vmanage.pdf

[11] J. Stoess, C. Lang, and F. Bellosa. Energy management for hypervisor-based virtual

machines. In Proceedings of the USENIX Annual Technical Conference, June 2007.

[12] R. Nathuji, C. Isci, and E. Gorbatov. Exploiting platform heterogeneity for power

efcient data centers. In Proceedings of the IEEE International Conference on Autonomic

Computing (ICAC), June 2007.

[13] W. Bircher and L. John. Analysis of dynamic power management on multi-core

Processors. In Proceedings of the 22nd Annual International Conference on

Supercomputing (ICS), June 2008.

[14] D. Suleiman, M. Ibrahim, and I. Hamarash. Dynamic voltage frequency scaling

(DVFS) for microprocessors power and energy reduction. In 4th International

Conference on Electrical and Electronics Engineering, December 2005.

[15] P. Kurp. Green computing: are you ready for a personal energy meter?

Communications of the ACM. October 2008.

[16] R. Nathuji, A. Somani, K. Schwan and Y. Joshi. CoolIT: coordinating facility and

IT management for efficient datacenters. In Proceedings of HotPower 08. December,

2008. Retrieved March 3, 2008 at http://www.usenix.org/events/hotpower08/.

123
[17] S. Niles. Virtualization: optimized power and cooling to maximize benefits.

American Power Conversion White Paper, 2008.

[18] J. Thethi. Realizing the value proposition of cloud computing: CIOs enterprise IT strategy

for cloud. Infosys White Paper, 2009.

[19] J. Ramanathan and R. Ramnath. Co-Engineering Applications and Adaptive Business

Technologies in Practice Enterprise Service Ontologies, Models, and Frameworks. Igi Global,

2009.

[20] Report to congress on server and data center energy efciency. Technical report,

U.S. Environmental Protection Agency, Energy Star Program, August 2007.

[21] J. Spitaels. Dynamic power variations in data centers and network rooms.

American Power Conversion White Paper, 2005.

[22] Carnegie Mellon website, http://www.sei.cmu.edu/architecture/cbam.html

[23] L. A. Barroso and U. Hlzle. The Case for Energy-Proportional Computing

Google.

[24] N. Rasmussen. An Improved Architecture for High-Efficiency, High-Density data

centers. The Green Grid White Paper #126.

[25] Jack Pouchet, Creating Energy Efficient Data Centers, Energy & Power

Management, January 2007;

[26] R Development Core Team (2009). R: A language and environment for statistical

computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-

0, URL http://www.R-project.org.

[27] Ramanathan, J. (2005 May). Fractal Architecture for the Adaptive Complex

Enterprise. Communications of the ACM, 48(5), 51-57

124
[28] James M. Kaplan, William Forrest and Noah Kindler, Revolutionizing Data Center

Efficiency, McKinsey & Company 2008

[29] Rong Ge, Xizhou Feng, Kirk W. Cameron, Performance-constrained Distributed

DVS Scheduling for Scientific Applications on Power-aware Clusters University of

South Carolina, Columbia, SC

[30] Kazman, R. Klein, M. Barbacci, M. Longstaff, T. Lipson, H. Carriere, J.

The architecture tradeoff analysis method, Software Eng. Inst, Carnegie Mellon Univ,

Pittsburgh, PA;

[31] Hardwiring Green into Infrastructure and Data Center Investments - Data Center

Operations Council

125
 Appendix A: Workload Generator Code

This appendix contains the source code used for the workload generator. The workload

generator was implemented in C++. The class structure is described in Figure 16 and

Figure 18.

SyntheticWorkloadGenerator

The SyntheticWorkloadGenerator class is the class which contains the main() function,

and hence is the entry point for the console application. It asks the user to input the

interval of change, and then generates random values between 0 and 100 which are used

to generate loads for CPU, Memory, I/O and Network for each time interval.

SyntheticWorkloadGenerator.cpp
#include "stdafx.h"
#include "CPULoadThread.h"
#include "MemoryLoadThread.h"
#include "IoLoadThread.h"
#include "NetworkLoadThread.h"

int _tmain(int argc, _TCHAR* argv[])

{
OPERATING_SYSTEM os;

//Generate CPU Load

CPULoadThread cpuThread;
cpuThread.Create();
cpuThread.Start();

126
 //Genrate Memory Load
MemoryLoadThread memoryThread;
memoryThread.Create();
memoryThread.Start();

//Genrate I/O Load

IoLoadThread ioThread;
ioThread.Create();
ioThread.Start();

//Genrate Network Load

NetworkLoadThread networkThread;
networkThread.Create();
networkThread.Start();

int cpuLoad = 0;
int memoryLoad = 0;
int ioLoad = 0;
int networkLoad = 0;
srand( (unsigned)time( NULL ) );

os.print("\nEnter the interval of change (minutes) : ");

int minutes = os.getInt();
int times = 120 ;//will stop after interval * 120 minutes

do
{
//Generate Load Automatically

cpuLoad = abs(rand())%101;
memoryLoad = abs(rand())%101;
ioLoad = abs(rand())%101;
networkLoad = abs(rand())%101;

os.print("\nGenerating ");
os.printInt(cpuLoad);
os.print("% CPU, ");
os.printInt(memoryLoad);
os.print("% Memory, ");
os.printInt(ioLoad);
os.print("% I/O and ");
os.printInt(networkLoad);
os.print("% Network Load.");

//Set Desired CPU, Memory I/O and Network Load

cpuThread.SetParameter(cpuLoad);
memoryThread.SetParameter(memoryLoad);
ioThread.SetParameter(ioLoad);
networkThread.SetParameter(networkLoad);

//Change Load every 5 minutes

os.sleep(minutes*60*1000);

times--;
}
while(times>0);

cpuThread.Stop();
memoryThread.Stop();
ioThread.Stop();
networkThread.Stop();
os.getInt();
return 0;
}

127
 WindowsOS

The WindowsOS class abstracts out all the windows dependent code, so that in the future,

the code written can be extended to other platforms by just replacing this class with a

corresponding class for the other platform.

WindowsOS.h

#pragma once
#define OPERATING_SYSTEM windowsOS

//This class implements all the windows specific commands needed by the program
class windowsOS
{
public:
windowsOS(void);
~windowsOS(void);
void print(char *);
void printInt(int);
void printLF();
void sleep(int);
int getInt();
int getTickSinceMidnight();
int getNumberOfCPUs();
};

128
 WindowsOS.cpp

#include "StdAfx.h"
#include "windowsOS.h"

windowsOS::windowsOS(void)
{
}

windowsOS::~windowsOS(void)
{
}

//Print a string of text

void windowsOS::print(char * text)
{
std::cout<<text;
}

//Print an integer value

void windowsOS::printInt(int value)
{
std::cout<<value;
}

//Print a new line

void windowsOS::printLF()
{
std::cout<<"\n";
}

//Sleep for required number of milliseconds

void windowsOS::sleep(int milliSeconds)
{
Sleep(milliSeconds);
}

//Perform a cin>>integer
int windowsOS::getInt()
{
int value;
std::cin>>value;
return value;
}

//Get milliseconds passed since midnight

int windowsOS::getTickSinceMidnight()
{
return GetTickCount();
}

//Get total number of CPU cores on the system

int windowsOS::getNumberOfCPUs()
{
SYSTEM_INFO info;
GetSystemInfo(&info);
int n=info.dwNumberOfProcessors;
return n;
}

129
 WindowsStatus

The WindowsStatus class abstracts out all the code that collects resource utilization

information from the operating system. It provides get() functions to get the current

resource utilization information.

WindowsStatus.h
#pragma once
#include <pdh.h>
#include <pdhmsg.h>
#define STATUS_OBJ WindowsStatus

#define SLEEP_INTERVAL 1000

#define IO_COUNTER _T("\\System\\File Write Bytes/sec")
#define MEMORY_COUNTER _T("\\Memory\\Page Faults/sec")
#define CPU_COUNTER _T("\\Processor(_Total)\\% Processor Time")
#define NETWORK_COUNTER _T("\\Network Interface(Intel[R] PRO_1000 MT Network Connection -
Packet Scheduler Miniport)\\Bytes Total/sec")

//This class gets the CPU, Memory and I/O Status

class WindowsStatus
{
private:
HQUERY queryCpu;
HQUERY queryMem;
HQUERY queryIo;
HQUERY queryNetwork;
HCOUNTER cpuCounter;
HCOUNTER memCounter;
HCOUNTER ioCounter;
HCOUNTER networkCounter;

public:
WindowsStatus(bool &);
~WindowsStatus(void);
double getCpu(void);
double getMem(void);
double getIo(void);
double getNetwork(void);
int getCacheSize(void);
};

130
 WindowsStatus.cpp

#include "StdAfx.h"
#include "WindowsStatus.h"

WindowsStatus::WindowsStatus(bool &success)
{
success = true;

// Open a CPU query object.

if( PdhOpenQuery (NULL, 1, &queryCpu) != ERROR_SUCCESS )
{
success = false;
}

// Add one counter that will provide the cpu data.

if( PdhAddCounter(queryCpu, CPU_COUNTER, 0, &cpuCounter)!= ERROR_SUCCESS )
{
success = false;
PdhCloseQuery (queryCpu);
}

// Open a Memory query object.

if( PdhOpenQuery (NULL, 1, &queryMem) != ERROR_SUCCESS )
{
success = false;
}

// Add one counter that will provide the memory data.

if( PdhAddCounter(queryMem, MEMORY_COUNTER, 0, &memCounter)!= ERROR_SUCCESS )
{
success = false;
PdhCloseQuery (queryMem);
}

// Open a I/O query object.

if( PdhOpenQuery (NULL, 1, &queryIo) != ERROR_SUCCESS )
{
success = false;
}

// Add one counter that will provide the I/O data.

if( PdhAddCounter(queryIo, IO_COUNTER, 0, &ioCounter)!= ERROR_SUCCESS )
{
success = false;
PdhCloseQuery (queryIo);
}

// Open a Network query object.

if( PdhOpenQuery (NULL, 1, &queryNetwork) != ERROR_SUCCESS )
{
success = false;
}

131
// Add one counter that will provide the I/O data.
if(PdhAddCounter(queryNetwork, NETWORK_COUNTER,0,&networkCounter)!= ERROR_SUCCESS)
{
success = false;
PdhCloseQuery (queryNetwork);
}

//To make sure the CPU query is started(Prevent it returning zero on first access)
if( PdhCollectQueryData (queryCpu) != ERROR_SUCCESS )
{
success = false;
}

WindowsStatus::~WindowsStatus(void)
{
PdhRemoveCounter(cpuCounter);
PdhCloseQuery (queryCpu);
PdhCloseQuery (queryMem);
PdhCloseQuery (queryIo);
}

//Check the CPU consumption since the last time the query was run.
double WindowsStatus::getCpu(void)
{
//Collect Query Data
if( PdhCollectQueryData (queryCpu) != ERROR_SUCCESS )
{
return -1;
}

PDH_FMT_COUNTERVALUE ItemBuffer;

// Format the performance data record and return the value.

if (PdhGetFormattedCounterValue (cpuCounter, PDH_FMT_DOUBLE, (LPDWORD)NULL,
&ItemBuffer)!= ERROR_SUCCESS)
{
return -1;
}

return ItemBuffer.doubleValue;

//Get memory consumption

double WindowsStatus::getMem(void)
{
//Collect Query Data
if( PdhCollectQueryData (queryMem) != ERROR_SUCCESS )
{
return -1;
}

// Format the performance data record and return the value.

PDH_FMT_COUNTERVALUE ItemBuffer;
if (PdhGetFormattedCounterValue (memCounter, PDH_FMT_DOUBLE, (LPDWORD)NULL,
&ItemBuffer)!= ERROR_SUCCESS)
{
return -1;
}
return ItemBuffer.doubleValue;
}

//Get I/O Consumption

double WindowsStatus::getIo(void)
{
//Collect Query Data
if( PdhCollectQueryData (queryIo) != ERROR_SUCCESS )
132
 {
return -1;
}

// Format the performance data record and return the value.

PDH_FMT_COUNTERVALUE ItemBuffer;
if (PdhGetFormattedCounterValue (ioCounter, PDH_FMT_DOUBLE, (LPDWORD)NULL,
&ItemBuffer)!= ERROR_SUCCESS)
{
return -1;
}
return ItemBuffer.doubleValue;
}

//Get Network Consumption

double WindowsStatus::getNetwork(void)
{
//Collect Query Data
if( PdhCollectQueryData (queryNetwork) != ERROR_SUCCESS )
{
return -1;
}

// Format the performance data record and return the value.

PDH_FMT_COUNTERVALUE ItemBuffer;
if (PdhGetFormattedCounterValue (networkCounter, PDH_FMT_DOUBLE, (LPDWORD)NULL,
&ItemBuffer)!= ERROR_SUCCESS)
{
return -1;
}
return ItemBuffer.doubleValue;
}
//Returns the cache page size in bytes
int WindowsStatus::getCacheSize(void)
{
SYSTEM_INFO si;
GetSystemInfo(&si);
return si.dwPageSize;
}

133
 WindowsThread

The WindowsThread class abstracts out code that implements threading (again, this is to

make the code platform independent). It provides a function called ThreadFunction that is

overwritten in the derived classes, and in this way, the thread functionality is

implemented.

WindowsThread.h
#pragma once
#include "WindowsOS.h"
#include "afxwin.h"

//This class implements the Thread functionality for windows

class WindowsThread
{
private:
CWinThread *workerThread;

public:
OPERATING_SYSTEM os;
int parameterInt;
WindowsThread(void);
~WindowsThread(void);
void SetParameter(int);
void Create();
void Start();
void Stop();
void Suspend();
void Resume();
virtual void ThreadFunction(); //virtual so that child class functions have a
preceedence even if base class pointer is used
static UINT MyThreadFunc(LPVOID);
};

134
 WindowsThread.cpp
#include "StdAfx.h"
#include "WindowsThread.h"

UINT WindowsThread::MyThreadFunc(LPVOID pParam)

{
WindowsThread *t=(WindowsThread*)pParam;
t->ThreadFunction();
return 0;
}

WindowsThread::WindowsThread(void)
{
}

WindowsThread::~WindowsThread(void)
{
}

//Sets an integer value that can be accessed by the thread

void WindowsThread::SetParameter(int param)
{
parameterInt=param;
}

//Create the thread - creates the thread as a suspended thread

void WindowsThread::Create()
{
workerThread =
AfxBeginThread(MyThreadFunc,(LPVOID)this,THREAD_PRIORITY_NORMAL,0,CREATE_SUSPENDED,0);
}

//Start the thread that was created in a suspended state

void WindowsThread::Start()
{
workerThread->ResumeThread();
}

//Stop a running thread

void WindowsThread::Stop()
{
AfxEndThread(0);
}

//Suspend a thread
void WindowsThread::Suspend()
{
workerThread->SuspendThread();
}

//Resume a suspended thread

void WindowsThread::Resume()
{
workerThread->ResumeThread();
}

//Thread function - Called when the thread is run. Implement thread functionality here
void WindowsThread::ThreadFunction()
{}

135
 CPULoadThread

This class generates the CPU Load required. It creates a thread for each core of the

processor, and then either sleeps, or does some work to create the desired load.

CPULoadThread.h

#pragma once
#include "windowsStatus.h"

//This Class represents the thread that generates CPU Load

class CPULoadThread : public WindowsThread
{
private:
int load;
int generate; //Amount of load generated which is varied to match
the load)
int threads; //Number of threads(Depends on number of cores)
STATUS_OBJ *status;
public:
CPULoadThread();
~CPULoadThread();
void ThreadFunction(); //Function overriden from the base class
};

136
 CPULoadThread.cpp
#include "StdAfx.h"
#include "CPULoadThread.h"
#include "WindowsStatus.h"
#include "CPUWorkerThread.h"

CPULoadThread::CPULoadThread()
{
threads = 0;
load = 0;
generate = 0;
bool success = false;
status = new STATUS_OBJ(success);
if(!success)os.print("\nERROR: Failure in setting up Status Object");
}

CPULoadThread::~CPULoadThread()
{delete status;}

//Run threads = number of CPU cores, each thread generates required percentage of load
void CPULoadThread::ThreadFunction()
{
//Wait for user input
while(parameterInt > 100 || parameterInt < 0)
os.sleep(1000);

int count;//count for loop running the workload for the generate value selected
double cpu;

while(true)
{
cpu = status->getCpu();
load = parameterInt;
if(load > cpu) //If less load than req is generated, generate more load
if(load - cpu > 10 )
generate += 5;//to reach desired goal quickly make big jump
else
generate++;
else if(load < cpu) //If more load than req is generated, gen less load
if(cpu -load >10)
generate -= 5;//to reach desired goal quickly make big jump
else
generate--;
count=8*10; //8 Seconds (CPU load settles down if measured at 8 seconds)
while(count >0)
{
//Start two threads - one for each core, run them for Generate sec
//Sleep for 100-generate number of seconds (Total time 100ms)
threads = os.getNumberOfCPUs();
for(int i=0;i<threads;i++)
{
CPUWorkerThread t;
t.SetParameter(generate);
t.Create();
t.Start();
}
os.sleep(100-generate);
count--;
}
}
}

137
 CPUWorkerThread

The CPUWorkerThread is the thread that runs on each individual processor core. The

CPULoadThread creates the worker threads which run for the specified time, and then

die.

CPUWorkerThread.h

#pragma once
#include "WindowsOS.h"

//This Class represents the worker threads that actually implement the loading of the CPU
//One thread per CPU Core
class CPUWorkerThread : public WindowsThread
{
public:

void ThreadFunction();//Function override from the Base Class

};

CPUWorkerThread.cpp

#include "StdAfx.h"
#include "CPUWorkerThread.h"

void CPUWorkerThread::ThreadFunction()
{
//If no user input, convert default into value bet 0 and 100
if(parameterInt > 100 || parameterInt < 0)
parameterInt = abs(parameterInt)%100;//os.sleep(1000);

//Run for given number of milliseconds and then die

//Generate Load for req no of milliseconds
int milliseconds = parameterInt;
int start = os.getTickSinceMidnight();//Get no of milliseconds since midnight
int end;
do
{
234+345;//Random work
end = os.getTickSinceMidnight();
}
while(end-start < milliseconds);
Stop();
}

138
 IoLoadThread

The IoLoadThread is the thread that generates the I/O Load.

IoLoadThread.h

#pragma once
#include "windowsStatus.h"

//This Class represents the thread that generates CPU Load

class IoLoadThread : public WindowsThread
{
private:
STATUS_OBJ *status;
public:
IoLoadThread();
~IoLoadThread();
void ThreadFunction();//Function overriden from the base class
};

139
 IoLoadThread.cpp

#include "StdAfx.h"
#include "WindowsStatus.h"
#include "IoLoadThread.h"
#include "IoWorkerThread.h"

IoLoadThread::IoLoadThread(void)
{
bool success = false;
status = new STATUS_OBJ(success);
if(!success)
{
os.print("\nERROR: Failure in setting up Status Object");
}
}

IoLoadThread::~IoLoadThread(void)
{
delete status;
}

void IoLoadThread::ThreadFunction()
{
//Wait for user input
while(parameterInt > 100 || parameterInt < 0)
os.sleep(1000);

//Single thread to generate I/O load

IoWorkerThread t;
t.Create();
t.Start();

while (true)
{
t.SetParameter(parameterInt);
os.sleep(SAMPLEINTERVAL);
}
t.Stop();
}

140
 IoWorkerThread

The IoWorkerThread is the thread that generates the I/O Load. It is started by the

IOLoadThread.

IoWorkerThread.h

#pragma once
#include "WindowsOS.h"
#include <fstream>
#include <iostream>
using namespace std;

#define READBLOCK 1000000

//this is the size of block that is read NO_OF_BLOCKS times every interval
//vary this if the block is too big that it causes the prog to crash

#define WRITEBLOCK 1000000

//this is the size of block that is read NO_OF_BLOCKS times every interval
//vary this if the block is too big that it causes the prog to crash

#define SAMPLEINTERVAL 10000 //The interval at which we check the load generated

#define READFILE _T("C:\\LoadGenerationRead.txt")

//File that is read to generate I/O Reads

#define WRITEFILE _T("C:\\LoadGenerationWrite.txt")

//File that is read to generate I/O Reads

#define NO_OF_BLOCKS 50 //Number of Blocks read every sample interval

class IoWorkerThread : public WindowsThread

{
char* filename; //Filename of the file being read
public:
~IoWorkerThread(void);
void ThreadFunction(); //Function to override from the Base Class
int length; //Length of file
//(Remove if we dont want to read the whole file
//and are just reading one line again and again
};

141
 IoWorkerThread.cpp

#include "StdAfx.h"
#include "IoWorkerThread.h"

IoWorkerThread::~IoWorkerThread(void)
{
Stop();
}

void IoWorkerThread::ThreadFunction()
{
//If no user input, convert default into value bet 0 and 100
if(parameterInt > 100 || parameterInt < 0)
parameterInt = abs(parameterInt)%100;

int amountofload; //Amount of load to generate (Ranges 0 - 100)

int sleepTime; //Amount of sleep time ranges from 0ms - SAMPLEINTERVALms
int readAmount; //Amount of data read each time (Ranges 0 READBLOCK)
char buffer[READBLOCK];//Buffer to store read data

while(true)
{
amountofload = parameterInt;//Keep updating amount of load to be generated
sleepTime = SAMPLEINTERVAL;
readAmount = amountofload * READBLOCK/100;

for(int i=0;i< NO_OF_BLOCKS; i++)

{
//Generating read load and reading stuff into the buffer
ifstream infile (READFILE,ifstream::binary);
infile.read(buffer,readAmount);
infile.close();

//Generating write load using the stuff read into the buffer
ofstream outfile (WRITEFILE,ofstream::binary);
outfile.write(buffer,readAmount);
outfile.close();
}
os.sleep(sleepTime);
}
}

142
 MemoryLoadThread

The MemoryLoadThread is the thread that generates the memory load. It creates a worker

thread to do the task.

MemoryLoadThread.h

#pragma once
#include "windowsStatus.h"

//This Class represents the thread that generates Memory Load

class MemoryLoadThread : public WindowsThread
{
private:
STATUS_OBJ *status;
char* data;

public:
MemoryLoadThread(void);
~MemoryLoadThread(void);
void ThreadFunction();//Function overriden from the base class
};

143
 MemoryLoadThread.cpp

#include "StdAfx.h"
#include "WindowsStatus.h"
#include "MemoryLoadThread.h"
#include "MemoryWorkerThread.h"

MemoryLoadThread::MemoryLoadThread(void)
{
bool success = false;
status = new STATUS_OBJ(success);
if(!success)
{
os.print("\nERROR: Failure in setting up Status Object");
}
}

MemoryLoadThread::~MemoryLoadThread(void)
{
delete status;
}

void MemoryLoadThread::ThreadFunction()
{
//Wait for user input
while(parameterInt > 100 || parameterInt < 0)
os.sleep(1000);

int generate = parameterInt;

//Single thread to generate cache misses

MemoryWorkerThread t;
t.SetParameter(generate);
t.Create();
t.Start();

//100 threads to generate cache misses

/*MemoryWorkerThread t[100];
for(int j=0;j<100;j++)
{
t[j].SetParameter(generate);
t[j].Create();
t[j].Start();
}*/

while (true)
{
t.SetParameter(parameterInt);
os.sleep(1000);
}

t.Stop();
/*for(int j=0;j<100;j++)
t[j].Stop();*/

144
 MemoryWorkerThread

The MemoryWorkerThread is the thread that generates the memory load. It is created by

the MemoryLoadThread.

MemoryWorkerThread.h

#pragma once
#include "WindowsOS.h"

#define PAGE_SIZE 4096

#define NO_OF_PAGES 60

class MemoryWorkerThread : public WindowsThread

{
public:
~MemoryWorkerThread(void);
void ThreadFunction();//Function override from the Base Class

};

145
 MemoryWorkerThread.cpp

#include "StdAfx.h"
#include "MemoryWorkerThread.h"

MemoryWorkerThread::~MemoryWorkerThread(void)
{
Stop();
}

void MemoryWorkerThread::ThreadFunction()
{
//If no user input, convert default into value bet 0 and 100
if(parameterInt > 100 || parameterInt < 0)
parameterInt = abs(parameterInt)%100;

//One Cache Page Size 4KB

//int cachePageSize = status->getCacheSize();
//int cachePageSize = 4096;

//Cache of 512 KB -> 128 pages

//Allocate block one size bigger than cache

char data[NO_OF_PAGES+1][PAGE_SIZE];//[129][4096]
int count=0;
while(true)
{
int amountofload = parameterInt;
int i=0;
//read blocks of data so that they are brought into cache
//for(int i=0;i<PAGE_SIZE;i++)
// data[count][i]++;
data[count][0]++;

// Going to one more than the cache size guarantees a miss

count = (count + 1)%(NO_OF_PAGES + 1);
if(count == NO_OF_PAGES)
os.sleep(100-abs(amountofload)%100);
}
}

146
 NetworkLoadThread

The NetworkLoadThread is the thread that generates the required amount of network

load.

NetworkLoadThread.h

#pragma once
#include "windowsStatus.h"

//This Class represents the thread that generates Network Load

class NetworkLoadThread : public WindowsThread
{
private:
STATUS_OBJ *status;
public:
NetworkLoadThread(void);
~NetworkLoadThread(void);
void ThreadFunction();//Function overriden from the base class
};

147
 NetworkLoadThread.cpp

#include "StdAfx.h"
#include "WindowsStatus.h"
#include "NetworkLoadThread.h"
#include "NetworkWorkerThread.h"

NetworkLoadThread::NetworkLoadThread(void)
{
bool success = false;
status = new STATUS_OBJ(success);
if(!success)
{
os.print("\nERROR: Failure in setting up Status Object");
}
}

NetworkLoadThread::~NetworkLoadThread(void)
{
delete status;
}

void NetworkLoadThread::ThreadFunction()
{
//Wait for user input
while(parameterInt > 100 || parameterInt < 0)
os.sleep(1000);

//Single thread to generate I/O load

NetworkWorkerThread t;
t.Create();
t.Start();

while (true)
{
t.SetParameter(parameterInt);
//os.print("Generating following amount of Network = ");
//os.printInt(status->getNetwork());
//os.printLF();
os.sleep(SAMPLEINTERVAL);
}
t.Stop();
}

148
 NetworkWorkerThread

The NetworkWorkerThread is the thread that generates network load by opening a port

and sending data to some remote computer.

NetworkWorkerThread.h

#pragma once
#include "WindowsOS.h"

using namespace std;

#define WRITEBLOCK 10000

//this is the size of block that is read (0 - 100) times every interval
//vary this if the block is too big that it causes the prog to crash

#define SAMPLEINTERVAL 1 //The interval at which we check the load generated

#define SEND_NO_OF_TIMES 100

//The number of times we send Writeblock bytes for every sampleinterval ms

#define SEND_TO "74.125.53.100" //Computer we are sending the data to

#define SEND_PORT 27015 //Port to which the data is sent

class NetworkWorkerThread : public WindowsThread

{
public:
~NetworkWorkerThread(void);
void ThreadFunction(); //Function to override from the Base Class
};

149
 NetworkWorkerThread.cpp

#include "StdAfx.h"
#include "NetworkWorkerThread.h"
#include <windows.h>
#include <stdio.h>
#include "winsock2.h"

NetworkWorkerThread::~NetworkWorkerThread(void)
{
Stop();
}

void NetworkWorkerThread::ThreadFunction()
{
//If no user input, convert default into value bet 0 and 100
if(parameterInt > 100 || parameterInt < 0)
parameterInt = abs(parameterInt)%100;
int amountofload; //Amount of load to generate (Ranges from 0 - 100)
int sleepTime; //Ranges from 0ms to the SAMPLEINTERVAL
int writeAmount; //Amt of data that is read each interval (0 to READBLOCK)

WSADATA wsaData;
SOCKET SendSocket;
sockaddr_in RecvAddr;
int Port = SEND_PORT;
char SendBuf[WRITEBLOCK];

// Initialize Winsock
WSAStartup(MAKEWORD(2,2), &wsaData);

// Create a socket for sending data

SendSocket = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);

// Set up the RecvAddr structure with the IP address of

RecvAddr.sin_family = AF_INET;
RecvAddr.sin_port = htons(Port);
RecvAddr.sin_addr.s_addr = inet_addr(SEND_TO);

while(true)
{
amountofload = parameterInt; //Keep updating amt of load to be generated
sleepTime = SAMPLEINTERVAL;
writeAmount = amountofload * WRITEBLOCK/100;

// Send a datagram to the receiver

for(int i=0;i<SEND_NO_OF_TIMES;i++)
sendto(SendSocket, SendBuf, writeAmount, 0, (SOCKADDR *) &RecvAddr,
sizeof(RecvAddr));
//Sleep for Sampleinterval ms
os.sleep(sleepTime);
}

// When the application is finished sending, close the socket.

closesocket(SendSocket);

// Clean up and quit.

WSACleanup();
}

150
 Appendix B: [R] Code

This appendix contains the script used in the R statistical tool to perform the regression

described in Chapter 6: Results.

#=============Using package with Robust standard error=================

# (Needed to throw out outliners)

install.packages('Hmisc'); install.packages('Design')
library(Design)

#=================Preparing Input file for regression =================

#Input File(with resource util and power)

file.42<-"input.csv"

#read the data into a table named 'inputdata'

inputdata<-read.table(file.42,header=T,sep=",")

#tells you the class...should be a data frame

class(inputdata)

#makes variables easier to work with

attach(inputdata)

#======================Performing Regression===========================

#Compute values for independent variables

Processor2 = Processor^2
Processor3 = Processor^3
Memory2 = Memory^2
Memory3 = Memory^3
Io2 = Io^2
Io3 = Io^3
Network2 = Network^2
Network3 = Network^3

#linear model named after combination of independent variables

regressionRobust<-ols(Watts ~ Processor + Processor2 + Processor3 +
Memory + Memory2 + Memory3 + Io + Io2 + Io3 + Network + Network2 +
Network3, x=T, y=T)

regression<-lm(Watts ~ Processor + Processor2 + Processor3 + Memory +

Memory2 + Memory3 + Io + Io2 + Io3 + Network + Network2 + Network3)

151
#=========================Display the results==========================

sink(file="Summary.txt")
print("+++++++++++++++++++Robust Standard Error++++++++++++++++++++")
print(robcov(regressionRobust))
print("+++++++++++++++Without Robust Standard Error++++++++++++++++")
print(summary(regression))
sink()

#Display the coefficients

sink(file="coefficients.txt")
print("+++++++++++++++++++Robust Standard Error++++++++++++++++++++")
print(coef(regressionRobust))
print("+++++++++++++++Without Robust Standard Error++++++++++++++++")
print(coef(regression))
sink()

152