Professional Documents
Culture Documents
Zpools
Vdevs
ZFS datasets
Snapshots / Clones
ZFS properties
ZFS updates
These exercises are meant to provide a primer into the value and flexibility of Oracle
Solaris 11 ZFS for the enterprise. Upon completion of this lab, the learner will
understand the simplicity and power of the ZFS file system and how it can help
address business requirements with Oracle Solaris 11 storage technology and will
be well on their way to mastering this powerful technology.
2 Overview
ZFS is the default file system in Oracle Solaris 11. This lab will follow the basic
system administration duties revolving around storage in a basic system. As in any
installation or implementation well follow a basic path for building our storage
infrastructure
1) Hardware setup and initial storage connection and assignment. (VirtualBox,
virtual disks, and files)
2) Creating pools. Storage devices in ZFS are grouped into pools. A pool
provides all of the storage allocations that are used by the file systems and
volumes that an installation will require.
3) Creating file systems which can be assigned to users and applications and
For Oracle employees and authorized partners only. Do
not distribute to third parties.
2011 Oracle Corporation
Page 1 of 41
3 Pre-requisites
This lab requires the use of the following elements:
A current laptop with at least 3GB memory and 100GB free disk space
Oracle VirtualBox Software (4.0.16 with Extension Pack installed)
Oracle Solaris 11 11/11 PreBuilt VM for Oracle VM VirtualBox
The following assumptions have been made regarding the environment where this
lab is being performed:
1. Network connectivity to the Internet is not necessary
2. We will only work with a single Solaris 11 Virtual Instance
4 Lab Setup
4.1 Oracle VirtualBox Hypervisor Software basics
Your system should already have Oracle VirtualBox hypervisor software installed
and ready to use. For this lab we will require a GUI interface and will be using the
pre-built Oracle Solaris 11 VM image. We only need to acquire it and import it to
get running quickly.
Download Virtual Box
VirtualBox Installation Notes:
Page 2 of 41
Make sure that you have the Oracle Solaris 11 VM Image copied to your
laptop hard disk.
Unzip the Solaris 11 image to your hard disk
In the VirtualBox Manager Screen click New
Click Next on the welcome screen
Click on the File Menu and
choose Import Appliance
Page 3 of 41
Page 4 of 41
Page 5 of 41
Page 6 of 41
Page 7 of 41
Page 8 of 41
Page 9 of 41
Page 10 of 41
Page 11 of 41
Page 12 of 41
Note from this output that the pool names datapool has a single ZFS virtual device
(vdev) called raidz1-0. That vdev is comprised of our four disk files that we created
in the previous step.
The RAIDZ1-0 type vdev provides single device parity protection, meaning that if
one device develops an error, no data is lost because it can be reconstructed using
the remaining disk devices. This organization is commonly called a 3+1, 3 data disks
plus one parity.
ZFS provides additional types of availability: raidz2 (2 device protection), raidz3 (3
device protection), mirroring and none. We will look at some of these in later
exercises.
Before continuing, let's take a look at the currently mounted file systems.
# zfs list
Page 13 of 41
One thing to notice in the ZFS makes things easier category is that when we created
the ZFS pool with one simple command, ZFS also created the first file system and
also mounted it. The default mountpoint is derived from the name of the pool but
can be changed easily.
Note:
Things we no longer have to do with ZFS are
Create a filesystem
Make a directory to mount the filesystem
Add entries to /etc/vfstab
Weve decided that we need a different type of vdev for our datapool example. Lets
destroy this pool and create another.
# zpool destroy datapool
# zfs list
Page 14 of 41
All file systems in the pool have been unmounted and the pool has been destroyed.
The devices in the vdev have also been marked as free so they can be used again.
Notice how easy it is to destroy and theres no destroy? Are you sure? warning.
Next, lets create a simple pool using a 2 way mirror instead of raidz.
# zpool create datapool mirror disk1 disk2
Now the vdev name has changed to mirror-0 to indicate that data redundancy is
provided by mirroring (redundant copies of the data).
What happens if you try to use a disk device that is already being used by another
pool?
# zpool create datapool2 mirror disk1 disk2
Page 15 of 41
The usage error indicates that /dev/dsk/disk1 has been identified as being part of an
existing pool called datapool. The -f flag to the zpool create command can override
the failsafe in case datapool is no longer being used, but use that option with
caution.
Note that a second vdev (mirror-1) has been added to the pool.
To see if your pool has actually grown, do another # zfs list command.
# zfs list datapool
Page 16 of 41
As before, we have created a simple mirrored pool of two disks. In this case, the disk
devices are real disks, not files. We've told ZFS to use the entire disk (no slice
number was included) and if the disk was not labeled, ZFS will write a default label.
ZFS Storage pools can be exported in order to migrate them easily to other system.
Storage pools should be explicitly exported to indicate that they are ready to be
migrated. This operation flushes any unwritten data to disk, writes data to the disk
indicating that the export was done, and removes all knowledge of the pool from the
system.
Lets export pool2 so that another system can use it.
Page 17 of 41
Note that our pool, pool2 is no longer in our list of available pools.
The next step will be to import the pool, again showing how easy ZFS is to use.
# zpool list
Notice that we didn't have to tell ZFS where the disks were located. All we told ZFS
was the name of the pool. ZFS looked through all of the available disk devices and
reassembled the pool, even if the device names had been changed.
If you dont know the name of the pool ZFS will provide the names of available
For Oracle employees and authorized partners only. Do
not distribute to third parties.
2011 Oracle Corporation
Page 18 of 41
Page 19 of 41
Pool properties are described in the zpool(1M) man page. Pool properties provide
information about the pool, effect performance, security, and availability. To set a
pool property, use zpool set. Note that not all properties can be changed (ex.
version, free, allocated).
Set the listsnapshot property to on. The listsnapshot (also listsnaps) controls
whether information about snapshots is displayed when the zfs list command is
run without the t option. The default value is off.
# zpool set listsnapshots=on pool2
# zpool get listsnapshots pool2
Page 20 of 41
When you patch or upgrade Oracle Solaris, a new version of zpool may be available.
It is simple to upgrade or downgrade an existing pool. Well create a pool using an
older version number, and then upgrade the pool.
# zpool destroy pool2
# zpool create -o version=17 pool2 mirror c4t0d0 c4t1d0
# zpool get version pool2
Page 21 of 41
Its that simple. Now you can use features provided in the newer zpool version, like
log device removal (19), snapshot user holds (18), etc.
This concludes the section on pools. There is a wealth of features that we havent
explored yet. Check out the man page for many other features that you can take
advantage of.
Lets clean up before proceeding to the next lab.
# zpool destroy pool2
# zpool destroy datapool
Page 22 of 41
Remember that our pool is automatically mounted so lets go ahead and create some
data and store it in the resulting file system.
# ps fe > /pool3/psfile.txt
# ls l /pool3
First lets check the status of the file system for size and then lets split the pool and
create our instant backup copy. We will provide a name for the resulting second
pool and call it pool4.
#
#
#
#
Page 23 of 41
Note that our pool now only contains a single disk but the size is still the same. And
running the ls command shows that our file is still there and has not come to any
harm.
Our new pool doesnt show up in the list because it still needs to be imported. Lets
do that now.
# zpool import pool4
# zpool status pool3 pool4
Page 24 of 41
That confirms our split pools. Now lets verify that our file has been duplicated in
the filesystem.
# ls l /pool3
# ls l /pool4
Now just for the heck of it, lets put the mirror back together. If this were a
production system you would ensure that complete and proper backups were done
before playing with splits and joins like this in a filesystem no matter how
trustworthy the software may be.
First well need to destroy pool4 because it has the disk we want to put back into the
mirror. Then well use the attach subcommand to bring a new disk into our
nonredundant single disk pool as a mirror. With the attach command you need to
list the existing device first and then the device you wish to join into the mirror.
# zpool destroy pool4
Page 25 of 41
The mirrored pool is now back to normal and the file it contained is still intact.
# ls l /pool3
Page 26 of 41
By using zfs list -r datapool, we are listing all of the datasets in the pool
named datapool. As in the earlier exercise, all of these datasets (file systems) have
been automatically mounted.
If this was a traditional file system, you might think there was 19.55 GB (3.81 GB x
5) available for datapool and its 4 datasets, but the 4GB in the pool is shared across
all of the datasets. To see an example of this behavior, type the following
commands:
# mkfile 1024m /datapool/bob/bigfile
# zfs list -r datapool
Page 27 of 41
Notice that in the USED column, datapool/bob shows 1GB in use. The other datasets
show just the metadata overhead (31k), but their available space has been reduced
to 2.95GB, the amount of free space available to them after the consumption of 1GB
by the datapool/bob dataset.
zfs
zfs
zfs
zfs
create
create
create
create
datapool/fred/documents
datapool/fred/documents/pictures
datapool/fred/documents/video
datapool/fred/documents/audio
Page 28 of 41
Page 29 of 41
The first thing to notice is that the available space for datapool/fred and all of its
children is now 2GB, which was the quota we set with the command above. Also
notice that the quota is inherited by all of the children.
The reservation is a bit harder to see.
Original pool size 3.91GB
In use by datapool/bob 1.0GB
Reservation by datapool/fred 1.5GB
So, datapool/joe should see 3.91GB - 1.0GB - 1.5GB = 1.4GB available.
Page 30 of 41
With ZFS it can be done with a single command. In the next example, let's move
datapool/fred to a directory just called /fred. First lets look at the current
mountpoint.
# zfs list -r datapool
Notice that not only did the command change datapool/fred, but also all of its
Page 31 of 41
In this example, rpool/dump is the dump device for Solaris and it 516MB.
rpool/swap is the swap device and it is 1GB. As you can see, you can mix files and
devices within the same pool.
Unlike a file system dataset, you must specifically designate the size of the device
when you create it, but you can change it later if needed. It's just another dataset
property. Create a volume.
# zfs create -V 500m datapool/vol1
This creates two device nodes: /dev/zvol/dsk/datapool/vol1 (cooked) and
/dev/zvol/rdsk/datapool/vol1 (raw). These can be used like any other raw or
cooked device. We can even put a UFS file system on it.
# newfs /dev/zvol/rdsk/datapool/vol1
Page 32 of 41
Expanding a volume is just a matter of setting the dataset property volsize to a new
value. Be careful when lowering the value as this will truncate the volume and you
could lose data. In this next example, let's grow our volume from 500MB to 1GB.
Since there is a UFS file system on it, we'll use growfs to make the file system use the
new space.
# zfs set volsize=1g datapool/vol1
# growfs /dev/zvol/rdsk/datapool/vol1
Page 33 of 41
We can use our point in time snapshots to create new datasets called clones Clones
are datasets, just like any other, but start off with the contents from the snapshot.
Clones and snapshots make efficient use of storage. Clones only require space for
the data that's different than the snapshot. That means that if 5 clones are created
from a single snapshot, only 1 copy of the common data is required.
Remember that datapool/bob has a 1GB file in it? Let's take a snapshot of the
datapool and then create some clones.
#
#
#
#
#
zfs
zfs
zfs
zfs
zfs
snapshot datapool/bob@original
clone datapool/bob@original datapool/newbob
clone datapool/bob@original datapool/newfred
clone datapool/bob@original datapool/newpat
clone datapool/bob@original datapool/newjoe
Page 34 of 41
We can see that there's a 1GB file in datapool/bob. Right now, that's the dataset
being charged with the copy, although all of the clones can use it.
Now let's delete it in the original file system, and all of the clones, and see what
happens.
# rm /datapool/*/bigfile
# zfs list r o space datapool
Notice that the 1GB has not been freed (avail space is still 382M), but the USEDSNAP
value for datapool/bob has gone from 0 to 1GB, indicating that the snapshot is
Page 35 of 41
The 1GB file that we deleted has been freed because the last snapshot holding it has
been deleted.
You can also take a snapshot of a dataset and all of its children. A recursive
snapshot is atomic, meaning that it is a consistent point in time picture of the
contents of all of the datasets. Use -r for a recursive snapshot.
# zfs snapshot -r datapool/fred@now
# zfs list -r -t all datapool/fred
Page 36 of 41
The last item well cover is a new command in Oracle Solaris 11, the ZFS diff
command. The diff command enables a system administrator to determine the
differences between different ZFS snapshots.
Lets start by creating some snapshots and adding files to a users home directory.
Assuming you have the labuser in your Solaris instance lets use that home
directory for our example.
# zfs list r t all rpool/export
Create a text file in the users home directory and take a snapshot call the snapshot
snap1
# ps fe >> /export/home/labuser/psfile.txt
# zfs snapshot rpool/export/home/labuser@snap1
Page 37 of 41
Now lets run the diff command and see what happens.
# zfs diff rpool/export/home/labuser@snap1 \
rpool/export/home/labuser@snap2
Note: I planned this lab to just incorporate the files we created but since were on a
laptop the profile-5000-charging.csv file kind of popped in there, you may or may not
have that file in your output.
Page 38 of 41
6.8 Compression
Compression is a useful feature integrated with the ZFS file system. ZFS allows both
compressed and noncompressed data to coexist. By turning on the compression
property, all new blocks written will be compressed while the existing blocks will
remain in their original state.
Lets create a 500MB file we can do some compression on. Type the following
commands:
# zfs list datapool/bob
# mkfile 500m /datapool/bob/bigfile
# zfs list datapool/bob
Now let's turn on compression for datapool/bob and copy the original 500MB file.
Verify that you now have 2 separate 500MB files when this is done.
Type the following commands:
# zfs set compression=on datapool/bob
# cp /datapool/bob/bigfile /datapool/bob/bigcompressedfile
# ls -la /datapool/bob
Page 39 of 41
7 Summary
In this lab you learned about the power of the ZFS File System in Oracle Solaris 11.
We discussed and performed exercises to familiarize you with zpools and virtual
devices (vdevs). We learned about ZFS datasets like snapshots and clones. You
were also exposed to the myriad of ZFS properties and ways that ZFS can easily be
updated.
The exercises were meant to provide initial exposure to these features and
hopefully a basis for continued learning and eventual expertise in this powerful
storage technology thats an integral part of Oracle Solaris 11.
8 Resources
For more information and next steps, please consult additional resources: Click the
hyperlinks to access the resource.
Oracle Solaris 11 ZFS Technology Page articles whitepapers, and more on ZFS
Oracle Solaris Admin: ZFS File Systems documentation on ZFS
ZFS Best Practices Guide excellent real world reference for all things ZFS
ZFS Evil Tuning Guide great resource for detailed tuning and hard to find
For Oracle employees and authorized partners only. Do
not distribute to third parties.
2011 Oracle Corporation
Page 40 of 41
Page 41 of 41