2017 3rd International Conference on Information Management
Container Oriented Job Scheduling Using Linear Programming Model
Dong Zhang, Bing-Heng Yan, Zhen Feng*
State Key Laboratory of High-end Server & Storage
Technology
Beijing, China
Inspur Electronic Information Industry Co., Ltd.
Jinan, China
e-mail: {zhangdong, yanbh, fengzh}@inspur.com
Abstract—With Docker gaining widespread popularity in the
recent years, the container scheduler becomes a crucial role for
the exploding containerized applications and services. In this
work, the container host energy conservation, the container
image pulling costs from the image registry to the container
hosts and the workload network transition costs from the
clients to the container hosts are evaluated in combination. By
modeling the scheduling problem as an integer linear
programming, an effective and adaptive scheduler is proposed.
Impressive cost savings were achieved compared to Docker
Swarm scheduler. Moreover, it can be easily integrated into
the open-source container orchestration frameworks.
Keywords—container; docker; scheduling; integer linear
programming; energy conservation; container image registry
I. INTRODUCTION
Since the rise of Docker, container, the technology that
has evolved for decades, has gained widespread popularity in
recent years. The industry has seen the superior advantages
inherent in Docker, which remodels the “build, ship and run”
procedure in the software engineering. Remarkably, the
lightweight and high-performance features pave the way to
more opportunities for Docker in novel applications and
cloud-native services.
Docker has established a de facto standard for the
container engine, however, the job scheduling and
orchestration in the containerized environments is still an
open issue. The open-source community proposed container-
native orchestration frameworks such as Kubernetes, Mesos
Marathon and Docker Swarm; the academic circles put
forward creative scheduling algorithms to pursue load-
balance or energy-efficient optimal solutions. The joint
efforts from both the communities make the container
technology steps further to the practical and commercial
usages.
In this paper, the authors consider a typical scenario
where the concurrent jobs are scheduled in the context of
containerized workload dispatch. Requests are initiated from
the client points for computing capacities, the computing
servers pull the related container images from the image
registry and launch containers to handle the received requests.
Mainly two kinds of costs are observed in the scenario, the
container image transition costs from the image registry to
978-1-5090-6306-2/17/$31.00 ©2017 IEEE
Chi Zhang, Yu-Xin Wang
School of Computer Science and Technology
Dalian University of Technology
Dalian, China
e-mail: zhangchi12@mail.dlut.edu.cn, wyx@dlut.edu.cn
the container hosts, and the energy costs of the container
hosts. The illustrated scenario settings and the potential cost
considerations lead towards our proposed methodology, that
minimizes the total costs on the premise that computing
capacity requirements are matched for both the clients and
servers.
This paper reveals itself in three points, firstly, it presents
a novel algorithm for the typical container-based applications;
secondly, the container image pulling costs, the workload
network transition costs from the clients to the container
hosts and the host energy costs are considered before making
the image pulling and container-launching decisions; and
thirdly, to our best knowledge, this is the first work to
employ the integer linear programming model to solve the
native container scheduling problem.
The remainder of this paper is organized as follows.
Section 2 reviews the related works, which inspired our
proposal; section 3 describes the typical container oriented
application scenario and details the linear programming
based solution; section 4 presents the experimental results
and gives a further analysis; section 5 concludes the paper
and introduces our future work.
II. RELATED WORKS
The scheduling methodologies in the cloud environment
have been studies for years since the coming of cloud
computing era. In the early period, the researches on
scheduling primarily focused on the collaboration between
the virtual machines and the physical hosts, for the purpose
of performance-oriented load balance or energy-oriented
workload consolidation. Some works investigated the on-
demand resource (processor cores, main memory, I/O
devices, etc.) provisions to virtual machines, while others
moved to a higher level of virtual machine migrations among
physical servers. To list some notable literatures, Srikantaiah
et al. 2008 studied the inter-relationships between energy
consumption, resource utilization, and performance of
consolidated workloads [1]. They observed that the optimal
operating points for the energy performance trade-offs for
consolidation exist, and modeled the consolidation problem
as a modified bin packing problem. Zhao et al. 2009
introduced the MEmory Balancer (MEB), which
dynamically monitors the memory usage of each virtual
174
machine, predicts its memory needs and periodically
reallocates host memory [2]. The memory predictor is based
on the building up of the LRU histogram, and monitors the
swap space usage of each guest OS. Padala et al. 2009
presented a feedback control system which consists of an
online locally linear model estimator to capture the dynamic
relationship between application-level performance and
resource allocations (i.e. CPU, disk) and a MIMO resource
controller to determine appropriate allocations of multiple
resources [3]. Urgaonkar et al. 2010 employed the queueing
information available in the system instead of the predictors
to make online control decisions, which is quite adaptive to
unpredictable changes in the workloads and does not require
the estimation and prediction of its statistics [4]. Morin et al.
2011 also modeled the workload (VM) placement problem
as an instance of the multi-dimensional bin-packing problem
and designed an ant colony optimization inspired algorism to
compute the placement dynamically [5]. Beloglazov et al.
2011 proposed an energy-aware resource allocation heuristic
approach in two steps, firstly, VMs are allocated using
modified best fit decreasing (MBFD) algorithms; secondly,
optimization of the current VM allocation is conducted by
the minimization of migrations (MM) policy and the highest
potential growth (HPG) policy [6]. Sun et al. 2013 put
forward a two-tiered resource (CPU, memory and disk)
allocation mechanism to provide on-demand capacities to the
concurrent applications [7]. Application performance as well
as CPU utilization improvement was observed with a set of
proposed algorithms. Yuan et al. 2014 presents a particle
swarm optimization based virtual machines resources
scheduling algorithm, which takes processing elements
computing capacity and the computational complexity into
considerations [8]. Zhang et al. 2015 proposed an inverse
clustering-based job placement method to balance the
workloads among the nodes and meet the resource
requirements for various jobs [9].
Virtual machine has been the dominant virtualization
form for years, however, with the birth of Docker, container
promptly gains its power as a competitive cloud service form.
Actually, before Docker seized imaginations and opened the
doors to larger container usage, the container technology has
walked through a long evolutionary process (Osnat, 2016
[10]). The concept of containers first emerged in the year
1979, in the Unix V7 system, the chroot system call was
introduced to segregate different processes with independent
root directories. Two decades later to 2000, the Jails was
introduced to FreeBSD, the “jails” can be seen as smaller
systems of FreeBSD, with the ability to assign an IP address
for each system and configuration. Then in 2001 a Jails-like
VServer patch for the Linux kernel was raised, which could
partition resources (file systems, network addresses, memory)
on a computer system. From then on, the container
technology accelerated the pace and a set of novel models
and features were released, such as the Oracle Solaris
Containers in 2004, Open VZ in 2005, Google Control
Groups (cgroups) in 2006, LXC (LinuX Containers) in 2008,
CloudFoundry Warden in 2011 and the Google LMCTFY
(Let Me Contain That for You) in 2013. Eventually in the
year 2013, Docker exploded in popularity. Docker is not a
175
brand-new technology, it has taken advantages of all the
previous accumulations, especially the technologies of
chroot, cgroups and union filesystem. Just as Ben Golub, the
CEO of Docker.Inc said in the Docker Con 2015: “Thank
you to the giants. We know we are standing on your
shoulders.”
Although Docker is still facing challenges from
comparable or emerging technologies such as CoreOS rkt,
unikernel or Intel clear container, Docker has established a
de facto standard for the container engine. Industrial and
academic communities begin to take more efforts to the
higher-level container scheduling and orchestration systems.
The Docker Swarm scheduler features three strategies:
spread, binpack and random. The spread strategy spreads
containers thinly over many machines, the binpack strategy
packs containers into machines as few as possible, the
random strategy works as its name suggests. Kubernetes
divides scheduling into two stages, the predicates stage and
the priorities stage. The predicates stage finds all the
available nodes for the request pods by judging the ports,
resource fits, disk conflicts and hostnames. The priorities
stage chooses the best fit node from the available set based
on the strategies such as the least resource utilization, service
spreading or complete equality. Generally speaking, the
strategies from the industrial and the open-source
communities are efficient, but too simple to handle
complicated application scenarios. Meanwhile, academic
researchers are investigating more adaptive scheduling
strategies. Monsalve et al. 2015 introduced the concept of
time slicing for the dynamic CPU resource allocation in the
containerized cloud environments [11]. Only a single
container with all its threads and processes is scheduled at
any time on shared resources.
In this paper, an efficient and adaptive container
scheduling algorism is put forward. The authors have
borrowed merits from the VM scheduling methodologies,
and the container image pulling cost, workload network
transition costs and the energy conservation are included into
consideration. Moreover, the proposed algorism is designed
for the typical deployment scenario of containers, and it can
be easily integrated into existing container orchestration
frameworks.
III. METHODOLOGY
A typical containerized application scenario is illustrated
in Fig. 1. Here the resources (i.e., processor cores, memories)
requested by the client points, and the resources available in
the container host, are measured by units. In the data center,
we have n container hosts, for the k-th host, it has the
computing capacities of Capacity(k) units. In the upside of
the figure, we have m client points. The client points can be
desktops, where the requests are initiated via the web service,
the clients can also be sensors, in the context of internet of
things (IoT). For the t-th client, suppose it requests
workloads of Workload(t) units. There exists network
transition cost (e.g. latency, bandwidth, maximum
connections) between the clients and hosts, we denote the
cost between the k-th host and t-th client is cost(k, t) per unit.
Besides, to launch the containers on the hosts, the hosts must
pull the responding containers from the image registry. The
image is composed of multiple layer files, if all the required
layer files are already stored in the host, the cost is zero;
however, if only parts of the layer files are ready, then the
missing layers should be downloaded from the image
registry, which leads to extra costs (illustrated in Fig. 2).
Figure 1. Containerized application scenario
Figure 2. Multiple-layer image pulling from the registry
In the production environment, the t-th client initiates a
task request with Workload(t). The workloads are divided
into multiple batches and dispatched to one or more
container hosts. The container hosts pull the container
images from the image registry and launch containers to
execute the workloads. The key issue in the whole workflow
is the designing of a scheduler to dispatch the workloads to
container hosts, to meet the following criterions:
(1) All the workloads from clients are containerized
and processed in the hosts.
(2) The workloads scheduled to a specific host cannot
exceed the host’s computing capacity.
For the purpose of global optimization, the electric power
consumed by all the hosts, the image pulling costs and the
costs during the workload network transition should be
minimized.
To meet the criterion (1), denote x(k, t) as the workloads
from the t-th client, which are assigned to the k-th host. Then
the constraint (1) is set as:
σ௡௞ୀଵ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൌ ‘”Ž‘ƒ†ሺ‫ݐ‬ሻ
where x(k, t) is a natural number .
The constraint (1) ensures that all the workloads from the
client k should be processed on the hosts.
To meet the criterion (2), the constraint (2) is set as:
σ௠
௧ୀଵ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൑ ƒ’ƒ…‹–›ሺሻ (2)
The constraint (2) ensures that all the workloads running
on the k-th host should not exceed its capacity.
The costs happened in the workload network transition is
defined as:
௡
݂ଵ ሺ‫ݔ‬ሻ ൌ σ௠
௧ୀଵ σ௞ୀଵ ‫ݐݏ݋̴ܿݏ݊ܽݎݐ‬ሺ݇ǡ ‫ݐ‬ሻ ή ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ (3)
The correlations between the server utilization and the
electric power consumption have been widely investigated,
in this work, the authorss employ Morin et al. 2011’s model
to estimate the power consumption of the container hosts [5]:
ܲ௞ ሺ—ሻ ൌ ൫ܲ௞ǡ௠௔௫ െ ܲ௞ǡ௜ௗ௟௘ ൯ ൈ — ൅ ܲ௞ǡ௜ௗ௟௘ (4)
With ܲ௞ǡ௜ௗ௟௘ and ܲ௞ǡ௠௔௫ being the average power values
for the k-th host, when the system is idle and fully utilized,
respectively. In our scenario, the idle state means that no
container is running in the host, while the fully-utilized state
means the containers running on the hosts have reached the
host’s capacity.
The character u in function (4) means the utilization rate
and it can be generated by:
σ೘
೟సభ ௫ሺ௞ǡ௧ሻ
—ൌ (5)
஼௔௣௔௖௜௧௬ሺ௞ሻ
Then the energy cost is summarized as:
σ೘
೟సభ ௫ሺ௞ǡ௧ሻ
݂ଶ ሺ‫ݔ‬ሻ ൌ σ௡௞ୀଵ ቆ൫ܲ௞ǡ௠௔௫ െ ܲ௞ǡ௜ௗ௟௘ ൯ ൈ ൅ ܲ௞ǡ௜ௗ௟௘ ቇ
஼௔௣௔௖௜௧௬ሺ௞ሻ
(6)
Without loss of generality, we suppose the task from
client t requires the image t to run on the container host, then
the image pulling cost is defined as:
݂ଷ ሺ‫ݕ‬ሻ ൌ σ௠ ௡
௧ୀଵ σ௞ୀଵ ‫ݐݏ݋̴݈݈ܿݑ݌‬ሺ݇ǡ ‫ݐ‬ሻ ή ‫ݕ‬ሺ݇ǡ ‫ݐ‬ሻ (7)
where y(k, t) = 1 if the node k pulls the image t from the
image registry, y(k, t) = 0 if not. In another word, y(k, t) = 1
if x(k, t) > 0, y(k, t) = 0 if x(k, t) = 0. To meet this constraint,
we set
‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൑ ‫ݕ‬ሺ݇ǡ ‫ݐ‬ሻ ή ƒ’ƒ…‹–›ሺ݇ሻ (8)
In function (8), if y(k, t) = 0, then x(k, t) is strictly
confined to 0.
(1) Take functions (3), (6) and (7) into consideration, the
resulting cost function is introduced as:
176
݂ሺ‫ݔ‬ǡ ‫ݕ‬ሻ ൌ ݂ଵ ሺ‫ݔ‬ሻ ൅ ߙ݂ଶ ሺ‫ݔ‬ሻ ൅  ߚ݂ଷ ሺ‫ݕ‬ሻ ൌ
௡ ௡
σ௠
௧ୀଵ σ௞ୀଵ ‫ݐݏ݋̴ܿݏ݊ܽݎݐ‬ሺ݇ǡ ‫ݐ‬ሻ ή ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൅ ߙ σ௞ୀଵ ቆ൫ܲ௞ǡ௠௔௫ െ

σ೘
೟సభ ௫ሺ௞ǡ௧ሻ
ܲ௞ǡ௜ௗ௟௘ ൯ ൈ ൅
஼௔௣௔௖௜௧௬ሺ௞ሻ

௡
ܲ௞ǡ௜ௗ௟௘ ቇ ൅ ߚ σ௠
௧ୀଵ σ௞ୀଵ ‫ݐݏ݋̴݈݈ܿݑ݌‬ሺ݇ǡ ‫ݐ‬ሻ ή ‫ݕ‬ሺ݇ǡ ‫ݐ‬ሻ

(9)
where the weighting factors ߙ and ߚ are introduced to
balance the three costs.
This work aims to dispatch the workloads from all the
client points to the container hosts to achieve the minimal
costs of the energy consumption and the network transition.
This is reflected in the objective function (10):
‹  ݂ሺ‫ݔ‬ǡ ‫ݕ‬ሻ ൌ ݂ଵ ሺ‫ݔ‬ሻ ൅ ߙ݂ଶ ሺ‫ݔ‬ሻ ൅  ߚ݂ଷ ሺ‫ݔ‬ሻ
Subject to the following constraints:
σ௡௞ୀଵ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൌ ‘”Ž‘ƒ†ሺ‫ݐ‬ሻ
σ௠
௧ୀଵ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൑ ƒ’ƒ…‹–›ሺሻ
‫ ݔ‬൒ Ͳ and ‫ݔ‬is integer
‫ ݕ‬is binary
Thereby, an integer linear programming representation is
introduced by the objective function (10) and the constrains
(11), (12) and (13). Then the optimal solutions to the
‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻǡ ݇ ൌ ͳǡʹǡ ǥ ݊ǡ ‫ ݐ‬ൌ ͳǡʹǡ ǥ ǡ ݉ can be reached.
The complete procedure is presented in Algorithm 1.
IV. EXPERIMENTAL RESULTS AND ANALYSIS
In this section, the authors conducted series of
experiments to compare the total costs of the proposed
method and the binpack and random methods employed in
the Docker Swarm orchestration framework. For the sake of
simplification and to expel other factors, the three scheduling
algorithms (LP in our proposal, Binpack and Random in
Docker Swarm) are implemented via MATLAB in a
controlled experimental environment.
Figure 4. Image pulling cost from the registry
To gain a first sight of the three scheduling methods, a
(10) small-scale test was performed initially, with two clients and
four container servers. The client workloads and host
capacities were generated in random, but were limited
between 10 and 20. The network transition costs between
clients and hosts were also random numbers between 1 and 4.
(11) The image pulling costs were set between 0 and 3, based on
the layers required to be downloaded from the registry.
(12) Without loss of generality, the weighting factors α and β in
the function (7) were set to 1. Following Morin et al. 2011’s
(13) practice, the idle and fully utilized power values were fixed
to 171 and 218 Watt, on a Dell PowerEdge 1950 server with
(14) 4 GB of RAM and two Intel Xeon 5148 2.33 GHz CPUs [5].
The randomly generated host capacities, client workloads
and workload network transition costs are shown in Fig. 3.
The image pulling costs from the registry are shown in Fig. 4.
Algorithm 1: Liner Programming based Scheduling
Algorithm for Containers
Inputs:
Container host capacities: ƒ’ƒ…‹–›ሺሻǡ ݇ ൌ ͳǡʹǡ ǥ ǡ ݊
Client requested workloads: ‘”Ž‘ƒ†ሺ‫ݐ‬ሻǡ ‫ ݐ‬ൌ ͳǡʹǡ ǥ ݉
Network transition costs per workload unit:
–”ƒ•̴ܿ‫ݐݏ݋‬ሺ݇ǡ ‫ݐ‬ሻ
Container image pulling costs between image registry and
hosts: ’—ŽŽ̴ܿ‫ݐݏ݋‬ሺ݇ǡ ‫ݐ‬ሻ
Outputs:
Workloads placed on the host k for the client t: ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ
Procedure:
# All the workloads from the client k should be
processed on the hosts:
௡

෍ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൌ ‘”Ž‘ƒ†ሺ‫ݐ‬ሻ

௞ୀଵ
# All the workloads performed on the host k should not
Figure 3. A small-scale containerized scenario demo exceed its capacity:

177
 ௠

෍ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൑ ƒ’ƒ…‹–›ሺ݇ሻ

௧ୀଵ
# The network transition costs between all the clients
and hosts:
௠ ௡

݂ଵ ሺ‫ݔ‬ሻ ൌ ෍ ෍ ܿ‫ݐݏ݋‬ሺ݇ǡ ‫ݐ‬ሻ ή ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ

௧ୀଵ ௞ୀଵ
# The energy consumption costs for all the container
hosts:
Figure 7. Random scheduler with total cost 167.9
௡
σ௠௧ୀଵ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ
݂ଶ ሺ‫ݔ‬ሻ ൌ ෍ ൭൫ܲ௞ǡ௠௔௫ െ ܲ௞ǡ௜ௗ௟௘ ൯ ൈ To simulate the large-scale concurrent scenarios, we
‫ݕݐ݅ܿܽ݌ܽܥ‬ሺ݇ሻ scaled out the quantities of hosts and clients, from 20 hosts
௞ୀଵ
with 30 clients, to 50 hosts with 100 clients, and 100 hosts
൅ ܲ௞ǡ௜ௗ௟௘ ൱ with 200 clients. The client workloads, host capacities,
network transition costs and the image pulling costs were
# The pulling cost for image k to host t: randomly generated, between 10 and 20, 50 and 100, 1 and 4,
0 and 3 respectively. The tests were performed for five
݂ଷ ሺ‫ݔ‬ǡ ‫ݕ‬ሻ ൌ rounds under the same scales, to observe the stability of the
σ௠ ௡ ௠ ௡
௧ୀଵ σ௞ୀଵ ‫ݐݏ݋̴݈݈ܿݑ݌‬ሺ݇ǡ ‫ݐ‬ሻ σ௧ୀଵ σ௞ୀଵ ‫ݐݏ݋̴݈݈ܿݑ݌‬ሺ݇ǡ ‫ݐ‬ሻ ή schedulers. The results are recorded in Tables 1 to 3 and an
‫ݕ‬ሺ݇ǡ ‫ݐ‬ሻand y is binary intuitive comparison is shown in Fig. 8.
It can be seen that the Binpack and Random schedulers
# To minimize the total costs: got comparable total costs, while the LP scheduler achieved
‹ ݂ሺ‫ݔ‬ǡ ‫ݕ‬ሻ ൌ ݂ଵ ሺ‫ݔ‬ሻ ൅ ߙ݂ଶ ሺ‫ݔ‬ሻ ൅  ߚ݂ଷ ሺ‫ݕ‬ሻ impressively results and the total costs were reduced by more
# Employ the integer linear programming method to than 50% in almost all the settings.
pursue the optimal solution to ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ and ‫ݕ‬ሺ݇ǡ ‫ݐ‬ሻ To observe the growth curves of the total costs with the
growing scales, we performed a group of experiments that
Fig. 5 to Fig. 7 demonstrate the outcomes with three the client number increased from 100 to 200, and the
scheduling algorithms. It can be seen that all the schedulers container host number increased from 50 to 200. The curves
behaved as expected. For the total costs, LP scheduler in our are illustrated in Fig. 9. There are heavy overlaps between
proposal achieved the lowest cost, the Random scheduler got the curves: the curves of the LP method coincide with each
the total cost of 167.8, for our surprise, the Binpack other although they used different quantities of hosts, the
scheduler resulted in the highest cost. curves of Binpack and Random wound together and it is hard
to distinguish them. However, we can still tell the clear story
that the costs by the LP method are far lower than other
methods. Moreover, with the increase of the client number,
the costs of the LP method increase by about 2000 from
2300 to 4300, while the costs of the other two increase by
about 5000 from 5000 to 10000. Therefore, the LP method
enjoys a far lower growth rates.

TABLE I. TOTAL COSTS UNDER 20 HOSTS AND 30 CLIENTS

LP Binpack Random
Figure 5. LP scheduler with total cost 137.01
Round 1 705.3 1630.2 1560.7
Round 2 698.1 1447.9 1624.3
Round 3 692.7 1543.7 1831.0
Round 4 631.7 1237.5 1387.3
Round 5 718.5 1516.5 1671.4

TABLE II. TOTAL COSTS UNDER 50 HOSTS AND 100 CLIENTS

LP Binpack Random
Round 1 2285.3 4904.5 5536.4
Round 2 2214.7 4844.7 4993.4
Round 3 2231.1 5047.5 4820.2
Round 4 2203.8 4640.2 4965.8
Figure 6. Binpack scheduler with total cost 185.3 Round 5 2268.5 4648.7 5097.0

178
 TABLE III. TOTAL COSTS UNDER 100 HOSTS AND 200 CLIENTS TABLE IV. ENERGY CONSUMPTIONS UNDER 100 HOSTS AND 200
CLIENTS
LP Binpack Random
Round 1 4398.3 9488.3 9844.8 LP Binpack Random
Round 2 4421.8 9828.4 9811.3 Round 1 1587.4255 1994.5625 2053.9633
Round 3 4592.4 9775.1 10509.9 Round 2 1544.9884 1833.5165 1991.5988
Round 4 4366.9 9743.8 9849.2 Round 3 1524.3855 1918.8469 1964.6325
Round 5 4334.9 9680.2 9796.3 Round 4 1542.1529 1884.4235 1952.5552
Round 5 1500.0833 1890.6129 1851.3717

TABLE V. TRANSITION COSTS UNDER 100 HOSTS AND 200 CLIENTS

LP Binpack Random
Round 1 3011 7671 7441
Round 2 3009 7615 7815
Round 3 2921 7374 7316
Round 4 2946 7222 7419
Round 5 2948 6899 7403

From Table 4, it can be observed that the LP scheduler

Figure 8. Comparison on the total costs under varied scales
In practice, the users may only consider the energy
consumptions or network transition costs alone. Therefore,
we further extended the experiments by modifying the
objective function as:
still got the best performance. The Binpack scheduler
outperformed the Random scheduler in most cases, it is
reasonable since that Binpack is targeted for the energy
conservation. In Table 5, where only the transition costs
were covered, we can observe remarkably large performance
margins between the proposed LP scheduler and other
schedulers.

σ೘
೟సభ ௫ሺ௞ǡ௧ሻ
‹  ݂ሺ‫ݔ‬ሻ ൌ σ௡௞ୀଵ ቆ൫ܲ௞ǡ௠௔௫ െ ܲ௞ǡ௜ௗ௟௘ ൯ ൈ
஼௔௣௔௖௜௧௬ሺ௞ሻ
ܲ௞ǡ௜ௗ௟௘ ቇ
In cases where only the energy consumption matters, or:
‹  ݂ሺ‫ݔ‬ሻ ൌ σ௠ ௡
௧ୀଵ σ௞ୀଵ ܿ‫ݐݏ݋‬ሺ݇ǡ ‫ݐ‬ሻ ή ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ
In cases where only the network transition cost counts.
The experimental results for the objective functions (15)
and (16) are shown in Tables 4 and 5 respectively, under the
100 hosts and 200 clients setting.
Figure 9. Cost growth curves with growing scales
V. CONCLUSION
൅
In this paper, we put forward an Integer Linear
Programming-based solution to the containerized application
(15) scheduling in the cloud environment. The proposed method
has achieved impressive performance in the energy
conservation, registry image pulling costs and the network
transition costs savings. It makes a step further to an adaptive
(16) and effective scheduler for the native container as a service
scenarios.
We have planned our future work in two fields. Firstly, in
the current scheme, the workload is represented by units, we
intend to extend the workload into multiple dimensions, to fit
for the multiple resource requests, leading to that the
workload will become a vector with CPU and memory
dimensions. Secondly, the scheduler prototype is currently
written in MATLAB, we will reconstruct it into the GO or
Python language, to make the proposed scheduler as a plugin
for the open-source container orchestration frameworks. The
scheduler prototype source code will be available at GitHub.
ACKNOWLEDGEMENT
This work is supported by the National High Technology
Research and Development Program (863 Program) of China
(NO. 2015AA015301).
REFERENCES
[1] S. Srikantaiah, A. Kansal, F. Zhao, “Energy aware consolidation for
cloud computing,” Proceedings of the 2008 conference on Power
aware computing and systems, San Diego, California, 2008, pp: 10-
10
[2] W. M. Zhao, Z. L. Wang, “Dynamic memory balancing for virtual
machines,” Proceedings of the 2009 ACM SIGPLAN/SIGOPS
international conference on Virtual execution environments,
Washington, DC, USA, 2009.

179
[3] P. Padala , K. Y. Hou, K. G. Shin, X. Z., M. Uysal , Z. K. Wang , S.
Singhal , A. Merchant, “Automated Control of Multiple Virtualized
Resources,” Proceeding of the 4th ACM European Conference on
Computer Systems, Nuremberg, Germany, 2009, pp: 13-26.
[4] R. Urgaonkar, U. C. Kozat, K. Igarashi and M. J. Neely, "Dynamic
resource allocation and power management in virtualized data
centers," 2010 IEEE Network Operations and Management
Symposium - NOMS 2010, Osaka, 2010, pp. 479-486.
[5] E. Feller, L. Rilling and C. Morin, "Energy-Aware Ant Colony Based
Workload Placement in Clouds," 2011 IEEE/ACM 12th International
Conference on Grid Computing, Lyon, 2011, pp. 26-33.
[6] A. Beloglazov, J. Abawajy and R. Buyya, “Energy-aware resource
allocation heuristics for efficient management of data centers for
Cloud computing,” Future Generation Computer Systems, 2012, vol.
28(5), pp: 755-768
[7] Y. Song, Y. Sun and W. Shi, "A Two-Tiered On-Demand Resource
Allocation Mechanism for VM-Based Data Centers," in IEEE
Transactions on Services Computing, vol. 6, no. 1, pp. 116-129, First
Quarter 2013.
180
[8] H. Yuan, C. B. Li , M. K. Du, “Virtual Machine Resources
Scheduling Based on Improved Particle Swarm Optimization in
Cloud Computing,” Journal of Software, 2014, 9(3).
[9] D. Zhang, B. H. Yan, Z. Feng, K. Y. Qi, Z. Y. Su, “Inverse
Clustering-based Job Placement Method for Efficient Big Data
Analysis,” Proceedings of the 2015 IEEE 17th International
Conference on High Performance Computing and Communications,
2015 IEEE 7th International Symposium on Cyberspace Safety and
Security, and 2015 IEEE 12th International Conf on Embedded
Software and Systems, Washington, DC, USA, 2015, pp. 1796-1799.
[10] R. Osnat, “A Brief History of Containers: From 1970s chroot to
Docker 2016,” http://blog.aquasec.com/a-brief-history-of-containers-
from-1970s-chroot-to-docker-2016
[11] J. Monsalve, A. Landwehr and M. Taufer, "Dynamic CPU Resource
Allocation in Containerized Cloud Environments," 2015 IEEE
International Conference on Cluster Computing, Chicago, IL, 2015,
pp. 535-536.