LOAD BALANCING USING PROCESS

MIGRATION FOR LINUX BASED DISTRIBUTED

SYSTEM
1

Ravindra A. Vyas, 2Hardik H. Maheta,3Vipul K. Dabhi, 4H. B. Prajapati

Department of Information Technology

Dharmsinh Desai University
Nadiad, India
1
ravindravyas.it@gmail.com, 2hardikmaheta@gmail.com, 3vipul.k.dabhi@gmail.com, 4harshad.b.prajapati@gmail.com

Abstract This paper presents implementation of load balancing

mechanism using master-slave model and Berkeley Lab
Checkpoint/Restart (BLCR) toolkit. The overall goal is to create
a Master-Slave model through which we can migrate processes
from highly loaded nodes to some dedicated lightly loaded nodes.
The agent running on master node divides total work into equal
sub tasks and delegates these sub-tasks to several independent
slave nodes. The master node computes final result by
aggregating the partial results returned by slaves.
Keywords Process migration;
balancing; Distributed systems

I.

Fault

tolerance;

Load

INTRODUCTION

In Distributed System some nodes are highly loaded and

some nodes are lighly loaded. This situation creates poor
performance in Distributed System. Overall approach in load
balancing is to balance load of every node. Using load
balancing we can efficiently utilize the resources.
Load balancing in a distributed environment can be done
through transferring a process form heavily loaded node to
lightly loaded node. Process migration is the transfer of
process from one node to another node in network of nodes. It
is very useful mechanism for balancing the load on distributed
environment. In proposed approach master slave model is
created to equally distribute load from heavily loaded master
node to dedicated identical slave nodes.
There are two types of process migration.
(1)Pre-emptive Process Migration
(2) Non-pre-emptive Process migration.
Pre-emptive process migration involves the transfer of a
process whose execution is partially completed on host node
whereas Non pre-emptive process migration involves transfer
of process whose execution has not yet begun. In both the
cases we need to transfer a process related information
consisting a virtual memory image, a process control block,
unread I/O buffers and messages, file pointers, timers that
have been set, etc to receiving node.

Processes on the master node are check pointed by BLCR

toolkit (using command CR_CHECKPOINT). These check
pointed processes are then distributed, using multi-threaded
socket programming in Java, among available slave nodes at
run time. The check pointed processes are restarted (invoked)
remotely on remote slave nodes by BLCR toolkit (BLCR
command CR_RESTART). Thus, we can distribute tasks from
heavily loaded Master node to several slave nodes.
Checkpoint/Restart mechanism (provided by BLCR) allows us
to save current status of one or more processes into a file and
later restart the execution of these processes from saved point
onward. This mechanism provides fault-tolerant process
execution in distributed environment. Checkpoint/Restart
(BLCR) allows pre-emptive process migration.
This paper presents implementation of fault tolerance process
migration mechanism using Master-slave model and Berkeley
Lab Checkpoint Restart Toolkit. The remainder of the paper
is organised as follows. Section II describes mechanism of
process migration. Section III describes implementation and
tools & technology description used in proposed approach.
And Section IV & V describes results and conclusion
respectively and section VI describes the possible future work.
II.

MECHANISM OF PROCESS MIGRATION

For implementing pre-emptive as well as non-pre-emptive

Process migration there are two types of algorithms available:
(1) Sender-initiated algorithm
(2) Receiver-initiated algorithm
In Sender-initiated algorithm overloaded node wants to send
(transfer) its process to another node. While in receiverinitiated algorithm under-loaded node wants to receive a
process from another node which is overloaded [4].
Process migration consists migration of process state and
address space of process. Process state (a few kilobytes)
contains the contents of registers, program counter, I/O
buffers, interrupt signals, etc. And address space (several
megabytes) contains program code, data and stack [1].
Address space can be transferred using several approaches.

In Total freezing, no execution is done while address space is

transferred. This is the simplest and slowest among all. In Pretransferring, address space is transferred while the process is
still running on the source node, and after the transfer of
address space, the modified pages are picked up. In Transfer
by reference approach, the process is restarted before the
address space of process is migrated and the pages are fetched
from the source node as the process needs them. Process
migration is used to balance load of any node in network by
reducing average response time of process to complete. It is
used to speedup individuals job and gains higher throughput.
By moving process closer to resource we can utilizes resource
effectively and it helps us to reduce network traffic. Process
migration improves systems reliability by moving replica of
process to another node. It also helps to improve system
security by moving sensitive data to secure node.
In our approach, processes are submitted by the clients to
Master node (central Server) without taking concern about
sufficient memory or processing power are available on
Master node which are require to execute these processes.
Master node does not have sufficient memory or CPU power
to execute the submitted processes then Master node transfers
it to slave nodes in round robin fashion. Here load (available
memory and CPU power) of the Slave node is known in
advance to the Master node. Master node accepts the process
and starts its execution. If Master node does not have
sufficient memory to execute this process then Master node
perform checkpoint using BLCR mechanism. Once checkpoint
file is created, it is transferred to the available node using file
transfer mechanism. Slave node receives a process from
Master node and re-starts execution using BLCR mechanism.
This whole model is implemented using JAVA multithreaded
programming.
III.

Master Node (Central Server): The master node

locates all slave nodes when an overload signal is
received. Master node finds IP address of all ready
slave nodes to transfer processes using java
multithreading model.
Check pointer: Master node checks point process
using CR_CHECKPOINT and creates its checkpoint
file.
File Transfer: The File Transfer establishes a
connection with the main server and transfers the
checkpoint files created by Check Pointer to slave
node when master node is overloaded.

IMPLEMENTATION OF LOAD BALACING USING

MASTER -SLAVE MODEL

This section describes BLCR toolkit and also describes

commands provided by this toolkit for process migration. This
section also describes the proposed architecture for Master
slave model for load balancing using process migration for
Linux based distributed system.
A. Tools & Technology
We have used Berkeley Lab Checkpoint/Restart (BLCR)
toolkit and integrate it with java multithreaded programming
mechanism to migrate processes for load balancing on UNIX
operating system.
Berkeley lab Checkpoint/Restart (BLCR) toolkit is used to
checkpoint process at source node and create checkpoint file
of process, socket programming in java is used to migrate the
process to specific slave node based on its IP address. We use
multithreading in java to migrate equal number of process to
all slave nodes concurrently.
B. Architecture
Components of Proposed Model and its application

Figure 1: Architecture of Master Slave model for the Process Migration

Restart The re starter uses CR_RESTART command

to re-start every process on slave nodes.
Load calculator: At user level we can measure
system load by reading the information residing in
the /proc file system or using sysinfo system call
(/usr/include/sysinfo.h). We can use sysinfo system
call (include/sysinfo.h) to get the Current load of the
CPU. Master Node obtains current load of the entire
available slave nodes using this load calculator
module.

C. Steps for Process Migration in UNIX environment using

Berkeley BLCR toolkit.
For process migration using this model, BLCR application
must be installed on Master node as well as on all the slave
nodes. In this section we show how to install BLCR and which
modules must be enable to perform process migration using
this model.
1) Loading Kernel modules
Kernel modules required for the BLCR application must be
residing within /lib/blcr and those modules must be loaded
before we start the BLCR application.
During installation process we must know the platform
whether it is 32 bit or 64-bit based on that installation will be
take place , installation prefix /usr/local is recommended for
centos 5.5.
After installation process, it is required to verify that
following two modules must exist into /blcr/2.6.12-.234
directory and this directory path must be prefixed with your
installation directory [5].
1) blcr_imports.ko
2) blcr.ko
If these modules are not loaded during start-up of Master and
all slave nodes then they can be loaded manually by using
following commands:
/sbin/insmod [Prefix] /blcr/2.6.12-.234/blcr.ko
/sbin/insmod [Prefix] /blcr/2.6.12-.234/blcr_imports.ko

running the executable you can load this using following

command.

2) Making an Applicable checkpoint

Following steps are required to create a checkpoint of a
process [5], [6].

$ cr_checkpoint --term PID checkpoint_filename

a) Create an executable for the program

Create the application using C program and make it
executable. For this research work we have written a program
for counting factorial of a given number in C programming
language and factorial calculated is written into file.
Compilation and execution of the above program is shown
below
$ gcc o factorial factorial.c
$. /factorial
b) Run executable within BLCR environment
It is required to set Environment before we execute the C
program using BLCR toolkit, certain libraries must be loaded
into environment, and it can be easily loaded using cr_run
command. Executable created in previous step is to be pass as
an argument of this command.
$ cr_run Executable [ arg1 arg2 ...]
There is no need to load required libraries explicitly as this
method of loading libraries only works with dynamically
executable applications. LD_LIBRARY_PATH environment
variable must set to the directory path where required
executable is residing, so the command for process migration
gets executed easily. libcr_run.so file must be loaded before

$ env LD_PRELOAD=
executable

BLCR_LIBDIR/libcr_run.so.0

3) Process Checkpoint at Master node and Restart at Slave

Stop execution of process at master node and its checkpoint
file can be created using command named cr_checkpoint. This
command has following arguments.
$ cr_checkpoint PID checkpoint_filename
PID is the process id of executing process.
Checkpoint-filename is the name of file created after
checkpoint take place and which contains the image of process
when checkpoint is taken using this command.
Default behaviour of this command cr_checkpoint is process
continue running after the command is executed but normally
in our master-slave model we need to stop the process and
then migrate it to other slave node where the process is
restarted. To fulfil this requirement BLCR gives
cr_checkpoint command with --term option, once you execute
the command in following manner it stop execution of
currently running process and checkpoint file is created.

If migrate the check-pointed file to slave node, and when we

restart that file, it needs the executable at slave node.If we
create checkpoint file using following command
$ cr_checkpoint term -- save all PID checkpoint_filename
Then executable is combined within check-pointed file, so no
need to put executable at every slave node. The newly created
checkpoint file is called context file, migrated From Master
Node to Slave Node. It is possible to execute any of the above
command by writing code snippet given below into Master
node file which is written using JAVA programming
language.
String Mcheck= ;
Mcheck="cr_checkpoint term --save all 3269 Context.01"
Process p1 = Runtime.getRuntime().exec(Mcheck);
D. Restarting the process
Once checkpoint file is created at Master node, it needs to
migrate checkpoint file from Master Node to Slave Node and
restart it for further execution. Checkpoint file can be re-start
using following command on Slave node.
$ cr_restart context-filename
Restart of process on slave node is performed according to
options used during checkpoint file creation at master node.

During this restore process slave will not use the same process
id specified into the context file but it will create new process
when restarting the process. If it is require that process id on
slave node is same as the process id on master node then
checkpoint needs to create on Master Node with -- no restorepid option.
Restart of the checkpoint process at slave node is done using
following command.
$ cr_restart context.15005
With above command slave can run the context file as many
times as slave wants to execute it but requires that once it
starts execution using cr_restart, it must completed and only
then after slave can restart the same context file It is possible
to execute any of the above command by writing code snippet
given below into Slave node file which is written using JAVA
programming language.
String SlaveCmd=;
SlaveCmd = " cr_restart Context.01"
Process p1 = Runtime.getRuntime().exec(SlaveCmd);
E. Steps of proposed approach
1) Start master node first and then calculate load of master
node using Load calculator module.
2) If load of master node exceeds some predefined
threshold limit then user selects some processes which
are makes master node highly loaded.
3) Master node check points all those process which are
selected by user using check pointer module.
4) Master node start SLAVE IDENTFYING module to
find total number of slave on LAN to accept migrating
processes.
5) Once total number of slave nodes are found then Master
starts its PROCESS DISTRIBUTION module to
equally distribute all checkpointed processes.
6) Repeat above procedure (Step 1 to 5) periodically.
7) Each of slave node accept process checkpoint file and
restart migrated process using Restarter module and
then send partial computed result to master node.

Figure 2: Topology for Experiment

We have taken two slave nodes for experimental purpose. It

is also possible to incorporate more number of slave nodes.
BLCR tool must be installed on Master node as well as on
every Slave node. Master Node must contain master execution
file and slave node must contains slave execution file which
are written in java programming language.
As shown in figure 3 processes are submitted to master node
and master node checkpoints them using cr_checkpoint. Once
process get check- pointed then master node get the status of
all slave nodes whether the node is working or not if the node
is working then it is added into slave node list. Load of each
slave node is obtained by reading /proc file of slave node and
then after master node migrates the process to slave node for
further execution.

Figure 3: Process migration on Master Node.

IV.

RESULTS OF EXPERIMENT

In this section we describe the topology used to perform the

experiment and obtained results. To implement this model
discussed into section III we use following topology contains
one Master Node and two slave nodes.

As shown in figure 4 once checkpoint file is migrated from

Master node to slave node then slave node starts execution of
the check-pointed process using cr_restart command and it
completes the execution. Here slave node does not have name
of the application. Output of the process is written into file and
by processing that file we can obtain the final output.

Duell J, Hargrove P and Roman, The Design and Implementation of

Berkeley Lab's Linux Checkpoint/Restart, Lawrence Berkeley National
Laboratory Technical Report LBNL-54941.
[7] Bruce Eckel, Thinking in Java, 3rd edition, 2002 by Prentice-Hall of
India.
[8] Brian Goetz, Tim Peierls, Joshua Bloch, Joseph Bowbeer, David
Holmes, Doug Lea Java Concurrency in Practice, 1st edition, 2006 by
Addison Wesley.
[9] M. L. Powell and B. P. Miller, Process migration in DEMOS/MP,
Proc. of the Ninth ACM Symp. on Operating Systems Principles, pp.
110-118 ACM SIGOPS, (October 1983). In Operating Systems Review
17:5
[10] A. S. Tanenbaum and R. van Renesse. Distributed Operating Systems.
ACM Computing Surveys, 17(4):419-470, December 1985
[11] Gerald J. Popek and Bruce J. Walker, editors. The LOCUS Distributed
System Architecture. MIT Press, Cambridge, Mass., 1985
[12] Fred Douglis and John K. Ousterhout. Process Migration in the Sprite
Operating System. In Proceedings of the 7th International Conference
on Distributed Computer Systems, pages 18-25, September 1987.
[6]

Figure 4: Process migration on Slave node.

V.

CONCLUSION

We implement a master-slave model using JAVA and

Berkeley Lab Checkpoint/Restart (BLCR) toolkit. Berkeley
Lab Checkpoint/Restart (BLCR) toolkit is used to migrate
process from one node to another node and we implemented
process migration in cluster environment using java
multithreading and socket programming.
Here we have been able to collect the information of all
nodes in Cluster environment using Load Calculator
component which uses /proc file system. We have been able to
perform load balancing by equally distributing load to all slave
nodes and restarting them on slave node. This increases the
performance of the Cluster by minimizing the execution time
for the processes sent by client on master node.
Using this architecture one can distribute load of master
node slave nodes efficiently and quickly. We wish the
suggested solution will be helpful to the load balancing with
process migration in distributed systems.
VI.

FUTURE WORK

We proposed process migration using BLCR toolkit and

java socket programming. For the future work we propose to
maintain log activity of process migration. We have
implemented this process migration model for small cluster
but for future work can overcome the scalability issue also it is
possible to perform process migration after taking network
traffic into consideration.
REFERENCES
[1]
[2]

[3]

[4]

[5]

Pradeep K. Sinha, Distributed Operating Systems: Concepts and

Design, 1997 by IEEE, Prentce-Hall of India.
Narayan Joshi, Dr. D. B. Choksi, Checkpointing Process Virtual
Memory Area for Process Migration; International journal of Emerging
Technologies and Applications in Engineering Technology and
Sciences; June-2010; pp-42-44.
Jonathan M.Smith and John Ioannidis. Implementingremote fork() with
checkpoint/restart. Technical Report CUCS-365-88, Columbia
University,1988.
Available
at
ftp://ftp.cs.columbia.edu/pub
/reports/reports-1988/cucs-365-88.ps.gz.
Vatsal Shah, Viral Kapadia, Load Balancing by Process Migration in
Distributed Operating System; International Journal of Soft Computing
and Engineering (IJSCE)ISSN: 2231-2307, Volume-2, Issue-1, March
2012
Berkeley
Lab
Checkpoint/Restart
Homepage.
http://ftg.lbl.gov/checkpoint