WarpDrive Oracle Best Practices

Best Practices for Optimizing Oracle Database Performance with the LSI WarpDrive Acceleration Card
Version 2.1 June 2011
DB15-000740-02
Best Practices for Optimizing Oracle Database Performance with the LSI WarpDrive Acceleration Card June 2011
Revision History
Version and Date Version 2.1, June 2011 Version 2.0, February 2011 Version 1.0, January 2011 Description of Changes Minor updates to the instructions in Section 4.3 and Section 4.4. General editing improvements. Second release with updates from Oracle. Initial release of this document.
LSI, the LSI & Design logo, and WarpDrive are trademarks or registered trademarks of LSI Corporation or its subsidiaries. Oracle is a registered trademark of Oracle Corporation. All other brand and product names may be trademarks of their respective companies. LSI Corporation reserves the right to make changes to the product(s) or information disclosed herein at any time without notice. LSI Corporation does not assume any responsibility or liability arising out of the application or use of any product or service described herein, except as expressly agreed to in writing by LSI Corporation; nor does the purchase, lease, or use of a product or service from LSI Corporation convey a license under any patent rights, copyrights, trademark rights, or any other of the intellectual property rights of LSI Corporation or of third parties. LSI products are not intended for use in life-support appliances, devices, or systems. Use of any LSI product in such applications without written consent of the appropriate LSI officer is prohibited. This document contains proprietary information of LSI Corporation. The information contained herein is not to be used by or disclosed to third parties without the express written permission of LSI Corporation. Corporate Headquarters Milpitas, CA 800-372-2447 Document Number: DB15-000740-02 Copyright 2011 LSI Corporation All Rights Reserved Email globalsupport@lsi.com Website www.lsi.com
Table of Contents
Table of Contents
1 Advantages of Using Solid-State Storage with a Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2 LSI WarpDrive Solid-State Storage Acceleration Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 3 Configuring an Oracle Database to Use LSI WarpDrive Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.1 Oracle 11g Release 2 Database Smart Flash Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.2 Using the WarpDrive Card as Primary Storage of the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.3 Implementation Trade-Off Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4 Tuning the Server Operating System and the Oracle Database for Better I/O Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.1 Monitoring and Tuning the Operating System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.2 Tuning the WarpDrive Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.3 Creating Storage Redundancy with Oracle ASM Using WarpDrive Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 4.4 Creating Storage Redundancy with Software RAID Using Multiple WarpDrive Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 4.5 Monitoring and Tuning the Oracle Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 5 System Test Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 5.1 Test System Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 5.2 Test Database Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.3 Tuning the Operating System and the Database. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.4 Enabling Flash Cache on One WarpDrive Card Using a File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.5 Creating an Oracle Tablespace Using Two WarpDrive Cards and ASM with Normal Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 6 Test Observations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
LSI Corporation Confidential -3-
Advantages of Using Solid-State Storage with a Database
Best Practices for Optimizing Oracle Database Performance with the LSI WarpDrive Acceleration Card
This document explains how to increase Oracle database performance and reduce user response time by using the LSI WarpDrive 300-GB PCIe solid-state storage acceleration card (hereinafter called the WarpDrive card). Offering high performance with low latency and a low CPU burden, the solid-state storage PCIe small form-factor WarpDrive card maximizes transactional I/O performance for applications like online transaction processing, data warehousing, and data mining. By delivering the I/O performance of hundreds of traditional drives, the WarpDrive card improves user satisfaction by enabling applications that use the Oracle database to meet your service-level agreements for response times and throughput.
Advantages of Using Solid-State Storage with a Database

Online transaction processing (OLTP) and data warehousing (DW) are typical types of applications that use an Oracle database. OLTP and DW applications have demanding requirements for short response times and high throughput, which makes it difficult for database administrators (DBAs) to maintain these systems as the number of users grows and the amount of data increases. During the application life cycle, performance bottlenecks might originate in one or more areas, such as the network, the processor, and the storage devices. Correcting a bottleneck in one area might cause another bottleneck to appear in another area. Solid-state storage provides a new tool for DBAs to solve their performance issues. For example, 5 ms is the typical response time for a small data read from a hard disk drive. Flash-based devices like the WarpDrive card can complete the same read, on average, in 50 s to 300 s, an improvement of several orders-of-magnitude in response time. Based on flash memory technology, solid-state storage provides performance levels that fall between the performance levels of hard disk drives and DDR3 memory. Initial implementations of solid-state drives (SSDs) were intended to replace a hard disk drive in direct-attach storage or RAID subsystems. Mounting SSDs on a PCIe card is a recent innovation that alleviates throughput constraints that are caused by the storage interface. The WarpDrive card supports up to 1.2 Gb/s and over 200K input/output operations per second (IOPs). You can achieve higher levels of performance by using multiple WarpDrive cards.
LSI WarpDrive Solid-State Storage Acceleration Card
LSI WarpDrive Solid-State Storage Acceleration Card

This section describes the LSI WarpDrive 300-GB PCIe solid-state storage acceleration card. The following figure shows the WarpDrive card.
Figure 1 WarpDrive Card
The WarpDrive card offers high performance with low latency and a low CPU burden. The WarpDrive card maximizes transactional I/O performance for Oracle databases and for other applications that require high-performance computing. The WarpDrive card performs consistently across reads and writes regardless of workload by using industry-standard and widely deployed LSI SAS software for easier system integration and management and a faster time to market. The WarpDrive card has the following features:

PCIe 2.0 host interface and PCIe small form-factor Storage capacity of 300 GB (uncompressed) Error-correcting code (ECC) protection up to 24 bits per 512 bytes LSISAS2008 controller Power supplied entirely by the PCIe slot, with no external power requirements; consumes less than 25 W Low latency, with minimal CPU burden and host memory footprint Optimized performance, best-in-class for reads and writes Simple integration using the SAS infrastructure Enterprise reliability Support for plug-and-play Bootable Support for both the Windows operating system and the Linux operating system A maximum of 240K 4-KB read input/output operations per second (IOPs) Up to 200K 4-KB write IOPs Average latency of less than 50 s Low host burden; no static CPU and memory overhead Software is optimized for SSD performance
Configuring an Oracle Database to Use LSI WarpDrive Cards
Configuring an Oracle Database to Use LSI WarpDrive Cards

This document describes two methods to configure an Oracle database using LSI WarpDrive cards. The first method uses the Oracle Database 11g Release 2 Database Smart Flash Cache feature, and the second method moves specific parts of the database to the WarpDrive card. Each implementation has its own cost and performance trade-offs.
3.1
Oracle 11g Release 2 Database Smart Flash Cache

With Oracle Database 11g Release 2 Enterprise Edition, Oracle introduced a feature called Database Smart Flash Cache. This feature allows customers with SSD hardware to increase the effective size of the Oracle database buffer cache without adding more main memory to the system. For transaction-based workloads, Oracle database blocks are normally loaded into a dedicated shared memory area in main memory called the system global area (SGA). The Database Smart Flash Cache feature allows you to expand the database buffer cache beyond the SGA in main memory to a second-level cache on flash memory. This document shows that you can use WarpDrive cards for the Oracle Database Smart Flash Cache feature to increase performance of your Oracle database. Both OLTP and DW environments can benefit from using Database Smart Flash Cache with WarpDrive cards. It is appropriate to use WarpDrive cards to achieve maximum performance in new Oracle database deployments with I/O-intensive workloads, as well as in existing database environments with memory constraints. Benefits include both improved transaction throughput and improved application response times. The following types of Oracle database environments can potentially make effective use of WarpDrive cards:

Workloads with repeated short transactions in which many users access the same data Storage systems that exhibit intensive disk read activity Systems under heavy main memory pressure that prevents more memory being allocated to the SGA buffer cache
3.2
Using the WarpDrive Card as Primary Storage of the Database

On Oracle systems that do not use Enterprise Edition or systems that use database releases before Oracle Database 11g Release 2, you can achieve performance benefits by storing portions of the database on WarpDrive cards. If the database is smaller than the capacity of the installed solid-state storage, you can move it all to the WarpDrive card. For large databases that exceed the WarpDrive cards capacity, you can achieve significant performance improvement by moving parts of the database to the WarpDrive card while leaving the rest on hard disk drives. Because WarpDrive cards have no spinning disk latency, they excel with random I/O transactions. Sequential writes or reads are not appreciably faster than hard disk drives. You should move randomly accessed data like an index file or data file to the WarpDrive card. Leave the online redo log files on the hard disk drives, because they are sequentially written. You can use pairs of WarpDrive cards with data mirrored across them to protect the data in the event of a board failure. In our tests we created a mirror over multiple WarpDrive cards. Tests used Oracle Automated Storage Management (ASM) with normal redundancy and software RAID to create a file system mirror, as described in Section 4.3, Creating Storage Redundancy with Oracle ASM Using WarpDrive Cards.
Tuning the Server Operating System and the Oracle Database for Better I/O Performance
3.3
Implementation Trade-Off Summary

The following table summarizes the implementation trade-offs.
Implementation Database Smart Flash Cache Advantages Considerations
Because this implementation uses a cache, The database version 11g Release 2 mirroring is not required, so less hardware is Enterprise Edition is required to use this required. feature.
Using the WarpDrive card for primary Older versions of the Oracle database can be WarpDrive cards must be installed in pairs storage used. The Enterprise version is not required. and mirrored for data integrity.
Currently, the Oracle Database Smart Flash Cache feature is available only on Linux and Solaris environments.
You can achieve significant database I/O improvement by tuning the operating system and the Oracle database. This section lists monitoring and tuning tips that you can implement before you introduce the WarpDrive cards to the configuration. Oracle Enterprise Manager (OEM) is an excellent tool to use for tuning the operating system as well as the database. OEM scripts and tools allow you to drill down in each environment to see what is going on, and they offer advice on how to tune the area in question. Tools such as Automatic Workload Repository (AWR) and Active Session History (ASH) also add great value to this tuning effort.
4.1
Monitoring and Tuning the Operating System

Use the iostat command and the vmstat command to monitor system resources on a server based on the Linux operating system or the UNIX operating system.
The iostat command monitors input and output device loading by observing the time that the devices are active in relation to their average transfer rates. The iostat command generates reports that you can use to monitor system configuration changes to better balance the input/output load between physical disks. The vmstat command reports information about processes, memory, paging, and CPU activity. The output from this utility shows server utilization of the various resources: disks, memory, and CPU.
To monitor system resources on a Windows system, run the Windows Performance Monitor using the Oracle options. Use these utilities during testing to see any improvements made after each change to the system. OEM presents valuable server information and storage performance information you can use to tune the server.
4.2
Tuning the WarpDrive Card

This section explains some actions you can take to tune the WarpDrive card for maximum performance using the Linux operating system.
Align the WarpDrive card on a 4-KB boundary in the Linux operating system. The following example shows how to calculate the boundary alignment, using the WarpDrive specifications. User input appears in bold. fdisk -l /dev/sdy Disk /dev/sdy: 300.1 GB, 300122898432 bytes 255 heads, 63 sectors/track, 36487 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes The following calculation shows how to use cylinder 1 as the starting cylinder when running the fdisk command: (1 cylinder * 16065 blocks/cylinder * 512 bytes per block) / 4096 = 2008.125 Thus, the 1-cylinder configuration is not divisible by 4 KB. Instead, you must start at cylinder 8: (8 cylinders * 16065 blocks/cylinder * 512 bytes per block) / 4096 = 16065 Now the result is evenly divisible by 4 KB. Running fdisk produces the following results on the WarpDrive card: fdisk -l /dev/sdy Disk /dev/sdy: 300.1 GB, 300122898432 bytes 255 heads, 63 sectors/track, 36487 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks ID System /dev/sdy1 8 36487 293025600 83 Linux Bypass journaling when you create a file system. Instead of using EXT-3 for the file system, use EXT-2, which eliminates double writes to WarpDrive cards in certain cases. The reduction in writes increases performance and prolongs the life of the WarpDrive cards. Use the following command to select EXT-2. mkfs t ext2 /dev/sdy1
Modify the kernel I/O scheduler. The I/O schedulers in the latest Linux releases have new I/O capabilities, including options to modify these settings at boot time. In the tests described in this section, we used the NOOP I/O scheduler and the noatime file system mount option. To invoke the NOOP scheduler for the system, add this line to the /etc/grub.conf file, and bounce the system. kernel /vmlinuz-2.6.18-194.el5 ro root=/dev/VolGroup00/LogVol00 rhgb quiet elevator=noop To verify that the NOOP scheduler is enabled, issue this statement as root. (Replace sdy with your block device.) [root] cat /sys/block/sdy/queue/scheduler [noop] anticipatory deadline cfq In addition to changing the I/O scheduler, use the noatime file system mount option, which was added to the /etc/fstab file. This option eliminates the need for the system to create writes to the file system when objects are only being read. This option also enables faster access to the files, plus the benefit of less wear on the WarpDrive card. This example shows how the /etc/fstab entry invokes the noatime option: /dev/sdy1 /u05 ext2 defaults,noatime 1 2
4.3
Creating Storage Redundancy with Oracle ASM Using WarpDrive Cards

Currently, the WarpDrive card does not provide data redundancy using a single card. To provide redundancy for production data when using the WarpDrive card, our tests verified that you can mirror data over two installed WarpDrive cards to eliminate a single point of failure. In our tests we used two WarpDrive cards along with Oracle ASM to create the storage for the Oracle database. Follow these steps to configure ASM over two WarpDrive cards: 1. Issue an fdisk command to create a primary partition on each WarpDrive card. When you create the primary partition, align it to a 4-KB boundary. Our test case started with cylinder 8. See Section 4.2, Tuning the WarpDrive Card, for the actual steps required to accomplish this. The commands that you use are as follows: fdisk /dev/sdy fdisk /dev/sdz
Our tests started at cylinder 8 and used the default value 36487 for the ending cylinder. The Linux operating system provides the Oracle ASM tool to create the ASM disks and the diskgroup. 2. Use these Oracle ASM commands to create one ASM disk for each WarpDrive card: ./oracleasm createdisk DG1 /dev/sdy1 ./oracleasm createdisk DG2 /dev/sdz1 3. Use these commands to create the diskgroup, specifying normal redundancy and using two failgroups: SQL> create diskgroup datawh normal redundancy failgroup flgrp1 disk 'ORCL:DG1' failgroup flgrp2 disk 'ORCL:DG2'; After creating the DATAWH diskgroup over both WarpDrive cards, our tests issued this SQLPLUS command to create an Oracle tablespace using this diskgroup: Create bigfile tablespace soe datafile +DATAWH size 200g;
4.4
Creating Storage Redundancy with Software RAID Using Multiple WarpDrive Cards
Currently, the WarpDrive card does not provide data redundancy using a single card. To provide redundancy for production data when using the WarpDrive card, our tests verified that you can mirror data over two WarpDrive cards to eliminate a single point of failure. In our tests we used two WarpDrive cards along with software RAID to create the storage for the Oracle database. Follow these steps to configure software RAID over two WarpDrive cards: 1. Issue an fdisk command to create a primary partition on each WarpDrive card. When you create the primary partition, align it to a 4-KB boundary. For our test case, we started with cylinder 8 and used the default value 36487 for the ending cylinder. See Section 4.2, Tuning the WarpDrive Card, for the actual steps that are required. Use the following fdisk commands: fdisk /dev/sdy fdisk /dev/sdz 2. 3. Create the mirror (/dev/md0) with two WarpDrive cards (/dev/sdy and /dev/sdz): mdadm --create /dev/md0 --level=mirror --raid-devices=2 /dev/sdy1 /dev/sdz1 To verify the RAID1 configuration, enter this command as root: mdadm --detail /dev/md0 CAUTION We successfully completed all of these system and storage modifications in the LSI lab to perform benchmark tests. Before implementing any of these modifications in your environment, be sure to test them thoroughly.
4.5
Monitoring and Tuning the Oracle Database

There are many areas that you should monitor when tuning a database, especially in the I/O area. The main items to monitor are hot disks. Hot disks are the disks in the system that are more frequently used, whereas other disks in the system are seldom used. When you identify these hot disks, you can redistribute database objects to the disks that are less frequently used, thereby spreading the I/O more evenly over all available disks instead of just a few selected disks. Or, in the case of using WarpDrive cards, you can distribute the I/O activity to devices that can handle the increased load much more effectively. In other words, you want the WarpDrive cards to be hot disks because they can handle the load.
Recent Oracle releases include the following useful tools to monitor the database.
Oracle Enterprise Manager (OEM). OEM is a comprehensive management and monitoring system that provides information that helps with tuning and administrative tasks. When you take advantage of the OEM management and tuning packs, you expand what OEM can provide. Automatic workload repository (AWR), which requires a license. You can use the AWR to collect the following database performance statistics: Wait events that identify performance problems Time model statistics that indicate the amount of database time associated with a process Active session history (ASH) statistics Some system and session statistics from the v$sysstat and v$sesstat views Object usage statistics Resource-intensive SQL statements You can query these Oracle database views to look at more specific areas: v$filestat v$sysstat v$system_event v$session_wait Turn on trace events The following tools allow you to test the performance of the storage subsystem: The Oracle I/O calibration stored procedure uses the actual running database, including real application clusters (RACs), to get a more realistic view of the I/O capabilities of the database. (This tool is new in Oracle 11g.) The Oracle ORION tool, which enables you to stress the storage array using certain access patterns without installing an Oracle database. The ORION tool destroys any data on the logical unit numbers (LUNs) being tested.
4.5.1
Areas of the Database to Tune First This section lists some of the critical areas and issues that affect how efficiently the database handles data I/O transactions.
The efficiency of the application code. If the application is not tuned correctly, it is very difficult to tune the database. Implementation of Stripe And Mirror Everything (SAME). Hardware resources. The amount of memory allocated to the buffer cache can have a dramatic effect on how quickly physical I/O is performed.
The amount of available CPU. The disk service times. Are the disk service times within reason? Is the storage system saturated from outside applications even before it starts this database?
The configuration of the LUNs. Are the LUNs created correctly for the database being built? Determine which RAID level to use with the correct configuration needed to satisfy the IOPs or MB/s requirements of the application. For example, if the application is an OLTP type of system, the RAID 5 write penalty lowers the performance.
The number of disk spindles. Are there enough disk spindles to meet the IOPs or MB/s requirements?
LSI Corporation Confidential - 10 -
System Test Results
If the storage system has reached the limits of the physical drives, I/O tuning on the database level does not improve the performance.
Do the storage parameters and the operating system I/O parameters match what was set in the database? For example, the value of the block size multiplied by db_file_multiblock_read_count should match what the operating system or volume manager has set for the maximum amount of I/O that it (the operating system or volume manager) can perform. (These adjustments to storage parameters have a lesser impact in the later releases of the database.)
4.5.2
Increasing I/O Efficiency at the Database Level This section contains some tips on improving I/O efficiency at the database level.

Use ASM to optimize your storage environment. Use multiple block sizes for different tablespaces. The database might indicate the uses of a smaller block size for the data tablespaces, but the index tablespaces might benefit from using a larger block size. Implement partitioning to separate high-access data from low-access data in the same table (license required). Implementing partitioning for tables and indexes allows the data to be tiered over different types of storage devices depending on the access. For example, in a data warehouse environment, some tables have current data that is accessed frequently alongside data that has been aged out or accessed infrequently. In this situation, you can partition the table and place the current partition, which is frequently accessed, on the WarpDrive card for better performance. As the data is aged out, migrate this data to a partition on the HDD. Migrate heavily accessed database objects to the WarpDrive card. We migrated such objects in our test scenario. Review Statspack or AWR reports. Pay particular attention to these sections within the reporting tools: Tablespace I/O Statistics section. The information in this section helps you to determine which tablespaces have the highest I/O activity. Instance CPU section. If the system is CPU bound, migrating database objects to the WarpDrive card might not improve performance and might instead have the reverse effect. This outcome might occur because when the WarpDrive cards eliminate I/O bottlenecks, your system can become more CPU bound. Segments by Physical Reads section. This section lists, in descending order, the most active database objects, based on physical reads and the percentage amount of the total read I/O activity. The section also lists the tablespace in which the object is located. Segments by Physical Writes section. This section lists, in descending order, the most active database objects, based on physical writes and the percentage amount of total write I/O activity. The section also lists the tablespace in which the object is located.
System Test Results

This section describes the system we used to test the Oracle database performance improvements achieved with WarpDrive cards. This section also describes the results of the multi-stage test procedures to improve the database performance.
5.1
Test System Description

In our tests, we ran OLTP-simulated benchmarks and measured significant gains using one or two WarpDrive cards. We used the following test system configuration:

HP ProLiant DL370 G6 server Dual Intel Xeon processor X5570

System Test Results
48 GB of RAM @ 1333 MHz LSI 9210-8i SAS host bus adapter LSI SAS 2x36 expander HP DG0146FAMWL disk drives 146-GB 2.5-in. small form-factor 6G SAS 10K RPM drives The benchmark was performed using the Oracle Swingbench tool
We used a 100-user load for the tests, with no latency for the transactions. This user load resulted in a heavily accessed system both for CPU and for disk I/O.
5.2
Test Database Settings

We used the following database settings for all tests:

Software RAID 0 over six LUNs for the UNDO tablespace Software RAID 0 over six LUNs for the online REDO logs All other tablespaces were striped over 10 individual LUNs (Swingbench tablespace was moved around between different LUNs for the various tests) A 250-GB Oracle Database Smart Flash Cache (when used) on the WarpDrive card SGA=16g filesystemio_options=async disk_async_io=TRUE 4-GB redo logs To implement Oracle Smart Flash Cache using a single WarpDrive card, we configured the WarpDrive card as a file system, as described in Section 4.2, Tuning the WarpDrive Card. We applied a patch for Oracle Database 11g Release 2 to enable Oracle Database Smart Flash Cache. The patch is 8974084:META BUG FOR FLASH CACHE 11.2PL BUGS TO BACKPORT TO 11.2.0.1 OEL. After installing the patch and bouncing the database, we set the DB_FLASH_CACHE_FILE parameter and the DB_FLASH_CACHE_SIZE parameter in the database to enable this feature. We used these settings in the tests when invoking Oracle Database Smart Flash Cache: SQL> alter system set db_flash_cache_file='/u05/flash.dbf' scope=spfile; SQL> alter system set db_flash_cache_size=250g scope=spfile; SQL> show parameter flash NAME TYPE VALUE -------------------------------------------------------db_flash_cache_file string /u05/flash.dbf db_flash_cache_size big integer 250G
These commands enable Oracle Database Smart Flash Cache inside the database using file system /u05 and allocating 250 GB to this cache. The database must be bounced before it uses this cache.
To implement Oracle Database Smart Flash Cache using multiple WarpDrive cards, we installed and implemented ASM to enable redundancy over multiple WarpDrive cards. See Section 4.3, Creating Storage Redundancy with Oracle ASM Using WarpDrive Cards for details about implementing ASM over multiple WarpDrive cards. We used these settings in the tests when invoking Oracle Database Smart Flash Cache: SQL> alter system set db_flash_cache_file= '+DATAWH/flash.dbf' scope=spfile; SQL> alter system set db_flash_cache_size=250g scope=spfile; SQL> show parameter flash NAME TYPE VALUE -------------------------------------------------------db_flash_cache_file string +DATAWH/flash.dbf db_flash_cache_size big integer 250G
System Test Results
5.3
Tuning the Operating System and the Database

Before adding the WarpDrive cards to the test configuration, we tuned the Linux operating system and the Oracle database using techniques described in Section 4, Tuning the Server Operating System and the Oracle Database for Better I/O Performance. We ran tests of transactions per minute (TPM), transactions per second (TPS), and response time before and after the tuning. At this point, the test system used all hard disk drives, with no WarpDrive cards. The results that follow emphasize how important it is to tune operating system and database performance before changing the physical configuration.

Average TPM before tuning: Average TPM after tuning: Average TPS before tuning: Average TPS after tuning: Average response time before tuning: Average response time after tuning:
63259 85212 1073 1442 99 ms 73 ms
5.4
Enabling Flash Cache on One WarpDrive Card Using a File System

For the next test, we installed one WarpDrive card and enabled Oracle Smart Flash Cache (250 GB). For information about the database setup, see Section 5.2, Test Database Settings. The test produced these database performance results:

Maximum TPM: Average TPM: Maximum TPS: Average TPS: Maximum response time (ms): Average response time (ms):
343% faster 375% faster 385% faster 373% faster 483% faster 330% faster
5.5
Creating an Oracle Tablespace Using Two WarpDrive Cards and ASM with Normal Redundancy
For this test, we installed two WarpDrive cards using ASM with the Normal Redundancy option. Thus, the configuration had no single point of failure and was suitable to persist production data. We placed all of the tables and indexes for the Swingbench test kit (160-GB total storage usage) in this tablespace. Section 4.3, Creating Storage Redundancy with Oracle ASM Using WarpDrive Cards, describes this configuration. The database performance results, compared to the original baseline test, were as follows:

Maximum TPM: Average TPM: Maximum TPS: Average TPS: Maximum response time (ms): Average response time (ms):
478% faster 595% faster 479% faster 655% faster 307% faster 660% faster
Test Observations
Test Observations
The tests described in the previous section included all of the tuning steps to the hardware and software, as described in Section 4.1, Monitoring and Tuning the Operating System, and Section 4.2, Tuning the WarpDrive Card. We also performed tests to determine the performance improvements just from implementing the 4-KB alignment on the WarpDrive cards, the change to the I/O scheduler, and adding noatime to the mount command for the WarpDrive card. We completed the original baseline tests before applying the tuning steps. During baseline tests, we noticed that performance dropped with a 500-user load and system contention (that is, CPU, wait times, service times) increased dramatically. Comparing test results using a 500-user load with just these three tuning steps applied, the sustained throughput gains were as follows:

Average TPM increased by over 20,000. Average CPU utilization went down from 91 percent to 75 percent. Average response time decreased by 12 percent.
These system changes not only increased the performance when using a 100-user load, but they also improved the performance of the higher user loads. The system did not drop off dramatically when using a 500-user load because the operating system and storage were operating more efficiently.

WarpDrive Oracle Best Practices

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

WarpDrive Oracle Best Practices

Uploaded by

Copyright:

Available Formats

Best Practices for Optimizing Oracle Database Performance with the LSI WarpDrive Acceleration Card

Version 2.1 June 2011

LSI Corporation Confidential -3-

Advantages of Using Solid-State Storage with a Database

Advantages of Using Solid-State Storage with a Database

LSI Corporation Confidential -4-

LSI WarpDrive Solid-State Storage Acceleration Card

LSI WarpDrive Solid-State Storage Acceleration Card

Configuring an Oracle Database to Use LSI WarpDrive Cards

Configuring an Oracle Database to Use LSI WarpDrive Cards

Oracle 11g Release 2 Database Smart Flash Cache

Using the WarpDrive Card as Primary Storage of the Database

LSI Corporation Confidential -6-

Implementation Trade-Off Summary

Monitoring and Tuning the Operating System

Tuning the WarpDrive Card

Creating Storage Redundancy with Oracle ASM Using WarpDrive Cards

LSI Corporation Confidential -8-

Monitoring and Tuning the Oracle Database

LSI Corporation Confidential -9-

System Test Results

System Test Results

Test System Description

HP ProLiant DL370 G6 server Dual Intel Xeon processor X5570

System Test Results

Test Database Settings

LSI Corporation Confidential - 12 -

System Test Results

Tuning the Operating System and the Database

63259 85212 1073 1442 99 ms 73 ms

Enabling Flash Cache on One WarpDrive Card Using a File System

LSI Corporation Confidential - 13 -

LSI Corporation Confidential - 14 -

You might also like