Microsoft SQL Server Customer Solution Case Study

First American Title Insurance Runs MissionCritical Application on SQL Server 2008

Overview
Country or Region: United States Industry: Financial Services Customer Profile Based in Santa Ana, California, First American Title Insurance Company is the second largest title company in the United States. Business Situation First American Title Insurance Company needed an enterprise-grade solution to support its mission-critical First American Software Technology (FAST) application and its 10-terabytes of data. Solution The company upgraded its FAST application database to Microsoft SQL Server 2008 Enterprise (64-bit) to take advantage of enhanced Online Indexing, Table Partitioning, and other features. Benefits 96 percent reduction in scheduled downtime for re-indexing 400 percent improvement for table partitioning Easier troubleshooting Enterprise-grade scalability

With SQL Server 2008, weve been able to implement Online Indexing on 99 percent of our tables, gaining a 96 percent reduction in the time required for offline reindexing.
Lianne Kwock, Senior Database Administrator, First American Title Insurance Company

One of the largest title insurance companies in the nation, First American Title Insurance Company, a subsidiary of The First American Corporation, traces its history back to 1889. The company offers title services through its direct operations and an extensive network of agents throughout the United States and abroad. Over the years, First American acquired several companies, leading to a complex IT environment that at one point had more than 50 different title and escrow systems. The company united these disparate systems with its First American Software Technology (FAST) solution, deployed on the Microsoft Application Platform, recently upgrading its FAST solution to Microsoft SQL Server 2008 and making use of features such as Online Indexing and Table Partitioning. The company stores 10 terabytes of information on SQL Server 2008, with 900 million rows in the largest table.

Fast Facts Total FAST data on SQL Server 2008 Largest SQL Server 2008 instance Largest table Number of FAST users Number of concurrent users Peak loads Reduction in scheduled downtime for indexing Improvement in Table Partitioning performance 10 terabytes 8.5 terabytes 900 million rows 15,000 7,000 15,000 batch requests per second Reduced from 13 hours every two weeks to 1 hour per month 400 percent

inefficient reporting across disparate systems, and slower new product development because of the complexity and cost of application integration. To address these obstacles, the company developed First American Software Technology (FAST), an integrated title and escrow system and the largest IT initiative in the companys history. Built from the ground up over a period of three years, the new solution replaced all 50 title and escrow systems across the company with a state-of-the-art, Web-based, centrally managed solution. FAST proved highly efficient for the company, as did its use of the Microsoft Application Platform, including Microsoft SQL Server 2005 Enterprise Edition (64-bit) and Internet Information Services (IIS) 6 running on the Microsoft Windows Server 2003 Datacenter Edition for Itanium-based Systems operating system. The solution is hosted on an HP Integrity Superdome with 32 Intel Itanium 64-bit processors. Always searching for ways to enhance the services it provides, First American IT wanted to take advantage of enhancements to the Online Indexing and Table Partitioning features of SQL Server 2008, as well as new features including compression technology.

Situation
For more than 120 years, First American Title Insurance Company has been providing title and settlement services for the residential and commercial real estate industry. The company offers its services through its direct operations and an extensive network of agents throughout the United States and abroad, and is the largest subsidiary of The First American Corporation, a FORTUNE 500 company. First American is one of the largest title insurance companies in the United States and has enjoyed rapid growth. Its growth has included a number of acquisitions, which have contributed to a highly heterogeneous IT environment. At one point, the company had more than 50 different title and escrow systems, which ran on a number of platforms including the Novell, UNIX, and Microsoft Windows operating systems. Some of these systems were stand-alone applications running on a single desktop computer, while others were server-based and supported dozens or hundreds of users. Having such a distributed and heterogeneous IT environment presented several disadvantages for the company including higher IT management costs,

Solution
First American upgraded its database software to SQL Server 2008 R2 Enterprise (64-bit), a process that went smoothly. Upgrading to SQL Server 2008 was easy for us, says Lianne Kwock, Senior Database Administrator at First American Title Insurance Company. Prior to upgrading, we ran the Upgrade Advisor utility to identify any compatibility issues. All we found was that we needed to change a handful of stored procedures. We also

Headquarters - The First American Corporation in Santa Ana, California (Photo copyright, The First American Corporation)

converted our eight SQL Server Integration Services packageswhich we mainly use for importing data from other sources for reportingto SQL Server 2008. The company, after testing SQL Server 2008 in its labs, was eager to upgrade to take advantage of a number of enhancements and new features of the database software, including: Online Indexing. The company is using the Online Indexing feature, enhanced for SQL Server 2008, to decrease the need for scheduled downtime. First American values Online Indexing because it enables database administrators to create or rebuild indexes without taking the database offline. Table Partitioning. Table Partitioning, enhanced for SQL Server 2008, enables faster data loads and simplified maintenance for very large tables by giving First American database

administrators the ability to treat multiple tables as a single entity. Resource Governor. SQL Server 2008 enables organizations to provide a consistent and predictable response to end users with the introduction of Resource Governor. First American can use Resource Governor to define resource limits and priorities for different workloads, and to help ensure resources cannot be unduly impacted by poorly constructed queries or other unusual workloads. Data Collector and Management Data Warehouse. This new feature in SQL Server 2008 provides First American the ability to capture performance metrics of SQL Server instances in a centralized way. The company now is able to capture performance counters, profiler traces, server activities and resource usage to a central data repository and to view all performance reports in one place. The company uses Resource Governor and Data Collector within its test environment. Dynamic Management Views. The Dynamic Management Views (DMVs) feature of SQL Server 2008 provides a wide range of reports that give database administrators greater transparency across multiple aspects of database health and performance. SQL Server includes over 70 DMV reports for comprehensive views of performance data and recommendations for improving database performance. Of special interest to First American was the Systems Views Map, which shows the key system views included in SQL Server, and the relationships between them. Replication. First American has enjoyed easier and more robust replication of data and schema since upgrading the database software. Designed to increase data availability by distributing the data across multiple databases, the companys

Architecture First American used a multi-tier architecture to create its integrated title and escrow system, which includes an 8.5-terabyte SQL Server 2008 database.

FAST solution also uses replication to provide a real-time copy of the database for reporting, to reduce loads on the database supporting online transaction processing. Data Compression. First American plans to use SQL Server 2008 Data Compression to help the company store more data on its hard drives, reducing the need for purchasing additional storage systems. The same feature also reduces the number of disk reads that databases must perform by incorporating more data into system memory. Backup Compression. With SQL Server 2008 Backup Compression, the compression is performed in memory before the data is transferred to disk. Backups run significantly faster since less disk I/O is required. First American plans to use Backup Compression to reduce the storage required to keep backups online, reducing the overall cost of keeping disk-based backups. The company worked with Microsoft Services, the consulting and enterprise support division of Microsoft, to update the architecture of the FAST application to gain efficiencies including use of the Open XML

Formats features of Microsoft Office Word 2007 to programmatically generate documents. Using Open XML Formats enhances the document generation functions of FAST, including integration with data stored on SQL Server. First American generates tens of thousands of documents per day. The new solution improves the speed at which documents can be generated and provides flexibility for modifying existing and creating new kinds of documents. An enterprise-grade database is required because the FAST title document management system supports a number of internal applications, including missioncritical line-of-business operations. FAST also supports a number of third-party external applications as a service provider to others in the title insurance and related industries. First Americans FAST application supports 15,000 users, with up to 7,000 concurrent users during peak usage. SQL Server 2008 supports some 15,000 batch requests per second for FAST. FAST Architecture FAST is a multi-tiered intranet Web-based application that services users across a wide area network infrastructure. The multi-tier structure includes: Client Tier. Accessed using Microsoft Internet Explorer, FAST screens are rendered using custom components that are automatically downloaded to manage client-side processing logic. The client tier also includes Adobe Acrobat Reader so users can preview FAST documents. Web Application Tier. The Web application tier uses IIS to serve content to the Internet Explorer client browser. The Web application tier also serves as a

middle-tier application server, created using Microsoft development tools, to encapsulate business logic and data. Message Queuing, a component of the Windows Server 2003 operating system, supports asynchronous interfacing with the document delivery tier. FAST Web services support delivery of FAST data to non-FAST systems. Document Delivery Tier. When a user requests a document, the request is dispatched to a document delivery server to bind user data with a FAST userdefined document template. The binding of the data and the user-defined template is done using Microsoft Office Word, including Words XML Formats feature. Interfaces Tier. The interfaces tier enables FAST to interoperate with a number of non-FAST First American systems. Database Tier. FAST is supported by SQL Server 2008, hosted on a shared HP Integrity Superdome computer with 64 Intel 64-bit processors and 128 gigabytes (GB) of RAM. The largest SQL Server instance holds 8.5 terabytes of FAST data, with 900 million rows in the largest table, and 750 million rows in the next largest. A second instance holds 1.5 terabytes of document workflow data. (One fourth of the processors and memory are dedicated to the second instance.) All tiers are supported by the 64-way Superdome server. Message Queuing is used to connect the solutions subsystems in a highly reliable, fully integrated manner by providing guaranteed message delivery, efficient routing, security, and prioritybased messaging. An EMC Symmetrix storage area network provides external storage. To maximize availability for its FAST solution, First American maintains a second, identical infrastructure at another First American data center in Dallas, Texas.

Mission-Critical Support Model Windows Server 2003 Datacenter Edition running on the HP Superdome provides First American with more than just a scalable and reliable platform. The Datacenter High Availability Program, a partnership between Microsoft and server manufacturers like HP, complements the server and operating system with a mainframe-style support model that helps First American keep its mission-critical solution up and running.

Benefits
First American enjoyed a 96 percent reduction in the need to schedule downtime for re-indexing, and saw a 400 percent improvement in the performance of Table Partitioning after upgrading FAST to SQL Server 2008. The company also benefits from easier troubleshooting using SQL Server 2008 DMVs, Resource Governor, and Data Collector, and enjoys enterprisegrade stability from the Microsoft Application Platform. 96 Percent Reduction in Scheduled Downtime for Re-Indexing With 10 terabytes of data in its FAST system, First American values the Online Indexing feature of SQL Server, which enables concurrent modifications (updates, deletes, and inserts) to the underlying table or clustered index data. This means the index can be defragmented while the database continues to work, serving First American users. While Online Indexing was introduced as part of SQL Server 2005, First American only deployed the feature for a subset of its tables because of limitations on available disk space on the storage area network (SAN) it had at the time. After upgrading to SQL Server 2008 and adding SAN capacity, the FAST team was able to use online

Our testing found a 400 percent improvement in performance when comparing SQL Server 2008 Table Partitioning with that of SQL Server 2005.
Israel Negrete, Director of Software Development, The First American Title Insurance Company

indexing across nearly all of its tables. The company currently isnt using Online Indexing on a table used for binary large objects (BLOBs). Our application has a lot of very large tables, Kwock says. In the past, we had to re-index every weekend or performance would degrade. With SQL Server 2005 we implemented Online Indexing across just some of our tables and achieved a 25 percent reduction in scheduled downtime for re-indexing. With SQL Server 2008, weve been able to implement Online Indexing on 99 percent of our tables, gaining a 96 percent reduction in the time required for offline re-indexing. Weve reduced our need for scheduled downtime for re-indexing from 13 hours every two weeks to just a single hour each month. FAST is so central to First Americans operations that the reduction in the need for scheduled downtime is a big deal for all concerned. The greatly reduced need for re-indexing downtime is significant for our operations, as it means that agents don't have to stop processing title and escrow files during the weekends because we have maintenance to do, says Israel Negrete, Director of Software Development at First American Title Insurance Company. From a user perspective, our people now have an application that's up and running almost 24/7. The business really likes that weve upgraded. They see the difference. 400 Percent Improvement in Table Partitioning First Americans FAST solution has some very large tables, the largest of which has more than 900 million rows. Even with Online Indexing, tables that approach a billion rows can require a lot of time for reindexing. This is why the company has long

been interested in the Table Partitioning feature introduced in SQL Server 2005. Table Partitioning simplifies maintenance for very large tables by giving database administrators the ability to divide tables into multiple parts, while continuing to manage them as a single entity. First American performed lab tests comparing SQL Server 2008 Table Partitioning with the earlier version and was impressed by the findings. Our testing found a 400 percent improvement in performance when comparing SQL Server 2008 Table Partitioning with that of SQL Server 2005, Negrete says. We now plan on implementing Table Partitioning for our largest tables, the ones that were so large we didnt implement Online Indexing. The company plans to use Table Partitioning to divide a table with nearly a billion entries covering a span of several years, into monthly partitions. This will work especially well for First American because nearly all activity involves the most recent entries. There is very little activity for entries more than a year or two old, which means that there is little need for re-indexing the older entries, Kwock says. Table Partitioning will greatly reduce our re-indexing needs. And weve seen a huge performance improvement with Table Partitioning in SQL Server 2008. Easier Troubleshooting The FAST team at First American became big fans of SQL Server Dynamic Management Views after they were introduced as part of SQL Server 2005, and were happy to find that SQL Server 2008 includes new DMVs and extensions to existing views. DMVs are designed to give

If a box slows down in our test environment, Resource Governor provides a nice view into how resources are being used.
Israel Negrete, Director of Software Development, First American Title Insurance Company

database administrators and other personnel a clear view into what's going on inside SQL Server, by providing server state information that can be used to monitor the health of a server instance, diagnose problems, and tune performance. The group is also using the Resource Governor and Data Collector features of SQL Server 2008 as diagnostic and troubleshooting tools. DMVs help us to look at different aspects of a server, Kwock says. We use DMVs to look at everything from CPU performance, memory usage, disks, tempdb, blocking issues, to query performance. Everything is in there. SQL Server 2008 provides dozens of DMVs and they enable us to find out the information that we need. The team values the low processing overhead of DMVs. We can run DMVs directly against our production boxes, Kwock says. They dont require a lot of resources. SQL Server 2008 Resource Governor was designed to enable organizations to provide a consistent and predictable response to end users by defining resource limits and priorities for different workloads, and to help ensure resources cant be unduly impacted by poorly constructed queries or other unusual workloads. The FAST team is using Resource Governor in their test environment to limit the resources used by ad hoc queries to avoid surprises. And the team is using Data Collector as a troubleshooting tool, taking advantage of its ability to monitor resource usage for each workload in a group, and its ability to provide a graphical display of Performance Monitor (Perfmon) and related information.

If a box slows down in our test environment, Resource Governor provides a nice view into how resources are being used, Negrete says. It also displays the relevant Perfmon information. Kwock agrees, Data Collector is a great place to begin troubleshooting because it provides a nice graphical and exacting view into the allocation of CPU, memory, query statistics, and other resources. When we need more information we can go to our DMVs. Enterprise-Grade Stability The FAST solution has proven so mission critical throughout First Americans operations that it requires enterprise-grade performance and stability from its database software, which is what the company has enjoyed with SQL Server. With more than 10 terabytes of data stored on SQL Server 2008, and with 900 million rows in our largest table, we require absolute enterprise-grade stability, which is what weve found with our solution, says Negrete. Weve enjoyed faultless performance with SQL Server 2008, and all the scalability we need. Our primary database can support 7,000 concurrent users, processing up to 15,000 batch requests per second, and these are fairly complex batch requests. Summary In summary, First American Title Insurance Company has found the enterprise-grade performance it requires by deploying its mission-critical FAST solution on the Microsoft Application Platform, including SQL Server 2008 Enterprise (64-bit). The company is taking advantage of SQL Server 2008 features including enhanced Online Indexing, Table Partitioning, and DMVs.

For More Information

For more information about Microsoft products and services, call the Microsoft Sales Information Center at (800) 4269400. In Canada, call the Microsoft Canada Information Centre at (877) 5682495. Customers in the United States and Canada who are deaf or hard-of-hearing can reach Microsoft text telephone (TTY/TDD) services at (800) 892-5234. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information using the World Wide Web, go to: www.microsoft.com For more information about HP products and services, visit the Web site at: www.hp.com For more information about First American Title Insurance Company products and services, call (800) 8543643 or visit the Web site at: www.firstam.com

Microsoft Server Product Portfolio

For more information about the Microsoft server product portfolio, go to: www.microsoft.com/servers/default.mspx

Software and Services

Microsoft Server Product Portfolio Windows Server 2003 Datacenter Edition for Itanium-based Systems Microsoft SQL Server 2008 R2 Enterprise (64-bit) Microsoft Office Microsoft Office Open XML Technologies Microsoft Internet Information Services 6

This case study is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Document published June 2010

Hardware

HP Integrity Superdome with 64 Intel 64bit processors and 128 GB of RAM

Partners

HP