Professional Documents
Culture Documents
+35% 450
Capacity (GB)
Max Sustained
DR (MB/s)
+15%171 Read Seek (ms)
73
75
3.6
-1% 3.4
2002 2008
Interface type
ATA
SATA
SCSI
IO service times are predominately FC
seek + rotational latency + queueing time SAS
2010 IBM Corporation
Disk performance
When do you have a disk bottleneck?
Random workloads
Reads average > 15 ms
With write cache, writes average > 2.5 ms
Sequential workloads
Two sequential IO streams on one disk
You need more thruput
IOPS vs IO service time - 15,000 RPM disk
500
IO service time (ms)
400
300
200
100
0
25 50 75 100 125 150 175 200 225 250 275 300 325
IOPS
40X
33X
5X
1X 1X
HDD SSD HDD SSD
Access time is drive-to-drive, ignoring any caching by SAS controller
# mkramdisk 16M
/dev/rramdisk0
# mkfs -V jfs2 /dev/ramdisk0
mkfs: destroy /dev/ramdisk0 (yes)? y
File system created successfully.
16176 kilobytes total disk space.
Device /dev/ramdisk0:
Standard empty filesystem
Size: 32352 512-byte (DEVBLKSIZE) blocks
# mkdir /ramdiskfs
# mount -V jfs2 -o log=NULL /dev/ramdisk0 /ramdiskfs
# df -m /ramdiskfs
Filesystem MB blocks Free %Used Iused %Iused Mounted on
/dev/ramdisk0 16.00 15.67 3% 4 1% /ramdiskfs
2 1 2 3 4 5
3
datavg
4
# mklv lv1 –e x hdisk1 hdisk2 … hdisk5
# mklv lv2 –e x hdisk3 hdisk1 …. hdisk4
5 …..
Use a random order for the hdisks for each LV
RAID array
PV
Max PPs per VG and max LPs per LV restrict your PP size
Use a PP size that allows for growth of the VG
Use a PP size that allows your LVs to be spread across all PVs
Unless your disk subsystem ensures your LVs are spread across all
physical disks
Valid LV strip sizes range from 4 KB to 128 MB in powers of 2 for striped LVs
Sequential IO
Typically large (32KB and up)
Measure and size with MB/s
Usually limited on the interconnect to the disk actuators
Check for trace buffer wraparounds which may invalidate the data, run
filemon with a larger –T value or shorter sleep
Useful flags:
-T Puts a time stamp on the data
-a Adapter report (IOs for an adapter) for both physical and virtual
-m Disk path report (IOs down each disk path)
-s System report (overall IO)
-A or –P For standard AIO or POSIX AIO
-D for hdisk queues and IO service times
-R to reset min and max values for each interval
-l puts data on one line (better for scripts)
-p for tape statistics
-f/-F for file system statistics (AIX 6.1 TL1)
2010 IBM Corporation
Using iostat
# iostat <interval> <count> For individual disk and system statistics
tty: tin tout avg-cpu: % user % sys % idle % iowait
24.7 71.3 8.3 2.4 85.6 3.6
Disks: % tm_act Kbps tps Kb_read Kb_wrtn
hdisk0 2.2 19.4 2.6 268 894
hdisk1 5.0 231.8 28.1 1944 11964
hdisk2 5.4 227.8 26.9 2144 11524
hdisk3 4.0 215.9 24.8 2040 10916
...
# iostat –ts <interval> <count> For total system statistics
System configuration: lcpu=4 drives=2 ent=1.50 paths=2 vdisks=2
tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc
0.0 8062.0 0.0 0.4 99.6 0.0 0.0 0.7
Kbps tps Kb_read Kb_wrtn
82.7 20.7 248 0
0.0 13086.5 0.0 0.4 99.5 0.0 0.0 0.7
Kbps tps Kb_read Kb_wrtn
80.7 20.2 244 0
0.0 16526.0 0.0 0.5 99.5 0.0 0.0 0.8
Shows average IO service times for reads and writes, IO rates, IOPS (tps) and time in the hdisk driver
queue
One can calculate R/W ratio and average IO size
Time spent in the queue indicates increasing queue_depth may improve performance
# lvmstat -l lv00 1
Log_part mirror# iocnt Kb_read Kb_wrtn Kbps
1 1 65536 32768 0 0.02
2 1 53718 26859 0 0.01
Log_part mirror# iocnt Kb_read Kb_wrtn Kbps
2 1 5420 2710 0 14263.16
Log_part mirror# iocnt Kb_read Kb_wrtn Kbps
3 1 4449 2224 0 13903.12
2 1 979 489 0 3059.38
2010 IBM Corporation
Using NMON
# nmon - then press “a” for all adapters or “^” for FC adapters
VMM
First field is a count of IOs delayed since boot due to lack of the specified buffer
For pbufs, use lvmo to increase pv_pbuf_count (see the next slide)
For psbufs, stop paging or add paging spaces
For filesystem fsbufs, increase numfsbufs with ioo
For external pager fsbufs, increase j2_dynamicBufferPreallocation with ioo
For client filesystem fsbufs, increase nfso's nfs_v3_pdts and nfs_v3_vm_bufs
(or the NFS4 equivalents)
Run # ioo –FL to see defaults, current settings and what’s required to make the
changes go into effect
The max_xfer_size attribute also controls a DMA memory area used to hold
data for transfer, and at the default is 16 MB
Changing to other allowable values increases it to 128 MB and increases the
adapter’s bandwidth
Often changed to 0x200000
This can result in a problem if there isn’t enough memory on PHB chips in
the IO drawer with too many adapters/devices on the PHB
Make the change and reboot – check for Defined devices or errors in the
error log, and change back if necessary
For NPIV and virtual FC adapters the DMA memory area is 128 MB at 6.1
TL2 or later
2010 IBM Corporation
Queue depth tuning
How to determine if queues are being filled
With SDDPCM: # pcmpath query devstats <interval> <count>
# pcmpath query adaptstats <interval> <count>
With SDD: # datapath query devstats <interval> <count>
# datapath query adaptstats <interval> <count>
With iostat: # iostat –D for data since system boot and
# iostat –D <interval> <count> for interval data
With fcstat: # fcstat fcs0
Transfer Size: <= 512 <= 4k <= 16K <= 64K > 64K
1032 1825589 40591 54946 35583
Indicates that we queued 128 IOs to the hdisk driver
If queue_depth is 128, we filled the queue
# iostat -D hdisk10
VMM
avgc - Average global non-fastpath AIO request count per second for the specified
interval
avfc - Average AIO fastpath request count per second for the specified interval for
IOs to raw LVs (doesn’t include CIO fast path IOs)
maxg - Maximum non-fastpath AIO request count since the last time this value was
fetched
maxf - Maximum fastpath request count since the last time this value was fetched
maxr - Maximum AIO requests allowed - the AIO device maxreqs attribute
If maxg or maxf gets close to maxr or maxservers then increase maxreqs or
maxservers