OSDI 2002

Optimizing the Migration of Virtual Computers

C.P. Sapuntzakis, R. Chandra, B. Pfaff, J. Chow, M. Lam, and M. Rosenblum

Main Idea

To quickly move the state (state in disks, memory, CPU registers, and I/O devices) of a running computer across a network. They employ techniques to reduce the data over the network. 20 minutes over 384 Kbps network without installing the application or even the underlying operating system
2

Benefits

Ease System Administration: Distribute a capsule with new changes Ease User Mobility: Instantaneous access to active environments everywhere Backups: Save snapshots of capsules as backups Other possible benefits: Recovery, Fail-over, Balance loads

Platform used

x86 computers as they are common, cheap, run most of the commonly used software and have tools for migration. VMware GSX server 2.0.1 running on Red Hat Linux 7.3 (kernel 2.4.18-10).

Related Work

Disco, Denali, VMware GSX server Rauch et al. Distribute disk images from central repository Emulab, Cluster-on-demand also use disk images Work by Schmidt introduces the term capsule Process level migration work - Condor, libckpt, CoCheck Object-level systems - Legion, Emerald, Rover LBFS (helped in HCP and hash cache design)

Virtual Machine Monitors

Layer of software on top of hardware that exports a virtual machine abstraction Encapsulates all of the necessary machine state Requires no cooperation from the software running on the monitor Provides the same device interface to the virtual machine regardless of the underlying hardware

Collective

Capsule-based system architecture - Collective Capsule: Is the hardware state including entire operating system and applications, and running processes and data. Optimization techniques are invisible to the user and require no modifications to the operating system.

Technical Mechanisms (brief)

Ballooning :- Zeroes unused memory Copy-on-write (COW) disks: Track just the updates to capsule disks. Demand Paging : fetches only needed blocks. Hash-Based Compression: Hashed Copy (HCP) avoids resending of existing data

Ballooning

Similar to VMWare ESX Server (Waldspurger) run a balloon program

asks the OS for a large number of pages then zeroes those pages

more compression smaller capsule, faster capsule start-up but may release pages that will be used more demand paging, slower initial execution
9

The Collective: COW

A capsule hierarchy
root - root disk (a complete capsule disk) nodes - COW disks

Capsule disk = a path from root to COW disk Latest disk = COW disk at end of the path
cant write to disks in interior of hierarchy can write only to disks at leaf nodes

10

Capsule Hierarchy

11

Capsule Disks

COW Implementation
bitmap file sequence of extent files

Disk Write
write to latest disk update bitmap

Disk Read

search bitmaps of COW disks from the bottom up

12

COW Disk Chains

Every time an active capsule is suspended and resumed, a new COW disk is created This can create long COW disk chains Solution: Promotion
Add to the disk all of its parents blocks that are not present in its own Then can move the disk up one level in the hierarchy (the child becomes a sibling of its parent)

13

shim library

A layer of virtualization that sits between the VM and the C library Intercepts the VMs I/O requests to plain disk Redirects them to a local disk server Translates them to COW disk requests Executes I/O operations on COW disks

COW disks can be locally or remotely stored

14

Demand Paging

Fetch disk data only when needed Keep a local shadow COW disk for each remote COW disk

contains all the locally cached blocks of the remote COW disk

15

Capsule Execution

Transfer the memory image and COW disk bitmaps (if do not already exist locally) Create a new latest COW disk locally For each remote COW disk, create the corresponding shadow COW disk (if does not already exist locally) Invoke the VM on the capsule

16

HCP (Hashed Copy)

Inspired by LBFS and rsync Dont send any data blocks that the receiver already has Hashed Copy algorithm:
Send a hash of the data blocks Receiver has data block that hashes to same value copy local version of block Else, get the data block from the sender

17

HCP Hash Cache

Persistent cache Primed cache:

The cache reads ahead in the remote file This is used to avoid multiple disk accesses when two files contain runs of similar blocks

18

Evaluation Methods

Migration to propagate SW updates Effectiveness and Interaction - Ballooning - Demand Paging - Hash Compression Superbowl XXXIX: Active vs. Inactive Capsules Simulation of user migration

19

Migration to propagate SW updates

Measuring: HCP

Operating Systems Upgrade

Software Installation Software Upgrade
20

Operating Systems Upgrade

Installation of new Red Hat 7.3 capsule: = 1.6 GB Savings incurred from Hashing: = 30% redundancy Savings with gzip: = 25% of the 1.6 GB capsule

21

Operating Systems Upgrade 2

Installation of new Red Hat 7.2 capsule: = 1.6 GB Savings incurred from Hashing: = 30% redundancy = 70% remaining = 1.15 GB remaining Savings with gzip: = 25% of the 1.6 GB capsule = 400 MB remaining = good job gzip!!! - (gzip + HCP)???
22

SW installation

Difference of HCP transfer of COW disk and size(installed packages)

1 MB/ .9 MB 1x 2500 KB/100 KB = 25x 600 KB/10 KB = 60 times larger

SW updates

HCP transfer of COW disk size(upgraded packages)

SW installation

Not much of a difference

HCP transfer of COW disk size(upgraded packages)

Migration of Active Capsules: User Mobility

Measuring effectiveness and interaction of ballooning, demand paging and HCP. Systems: - Windows: measure interactive processes - Linux: measure CPU + I/O bound processes Speeds: - DSL: 384 KB/sec - LAN: 100 MB/sec
27

Results
Windows: interactive processes Local - interactive: user 5 minutes (but cant compare) - CPU (compilation): 45 seconds LAN - interactive part: 1.3 minutes vs. ??? - CPU: 1.8 minutes DSL - interactive part: 4.4 minutes vs. ??? - CPU: 7 minutes

28

Results
Linux: CPU and I/O bound VMware, ran locally - smttls, processor simulation: 3 minutes - gcc,: kernel compile: 4 minutes - Apache: 1 minute, but what task did they do? vs. DSL and LAN - no slower than local execution

29

 Execution: disk data: Demand paging Migration: memory image: Ballooning + HCP

Active vs. Inactive Capsules

Measure: best use of capsules
Inactive Active Mount over NFS Unballooned COW: Empty Cache Ballooned COW: Primed Cache Unballooned, Primed COW: Locally Cached Ballooned, Primed

31

manual

Simulation of user migration

Measure: Ballooning, HCP, and gzip Winstone user interaction 12 snapshots during replay on Active capsule |data in COW| amount of disk traffic

33

Conclusions

Over DSL, capsules can be migrated in 20 minutes or less The Collective facilitates administration Amount of data transferred is reduced use gzip

35

Remaining Questions

Promotion = User Data Loss? Upgrades to immutable capsules = User Data Loss? an abstraction that provided a private namespace to a group of processes that can be migrated as a unit

36