How PG&E implemented SAP BIA and did performance tuning

Darin Lemos PG&E Bjarne Berg Director SAP BI Comerit Inc.

[ ANUP MAHESHWARI
ASUG INSTALLATION MEMBER MEMBER SINCE: 2008

[ SHERRYANNE MEYER [ AJAY VONKARERY

ASUG INSTALLATION MEMBER MEMBER SINCE: 1999 ASUG INSTALLATION MEMBER MEMBER SINCE: 2000

[ Learning Points
 See a practical example from PG&E and understand what you can expect on your implementation  Learn how BI-A works  Explore benefits and limitations of the tool

Real Experience. Real Advantage.

[ Agenda
 Introduction  Pacific Gas & Electric Project Overview  What is SAP BI-A
 Overview  Building aggregates  Benefits

 How we implemented BI-A  Sizing the environment

 Team organization and project plan  Testing

 BI-A Limitations  PG&E - Our top-10 Lessons Learned  Wrap-up

Real Experience. Real Advantage.

In this session.

 We will look what SAP NetWeaver BI Accelerator can and cannotdo to optimize dashboard performance.  Explore how we PG&E planned, and implemented SAP NetWeaver BI Accelerator.  Examine performance benchmarks for SAP NetWeaver BI Accelerator  Explore the top 10 lessons learned from the project team and see a sample work plan and an organization chart from this project as a reference for your own SAP NetWeaver BI Accelerator initiative
Real Experience. Real Advantage.
4

[ Agenda
 Introduction  Pacific Gas & Electric Project Overview  What is SAP BI-A
 Overview  Building aggregates  Benefits

 How we implemented BI-A  Sizing the environment

 Team organization and project plan  Testing

   

BI-A Limitations PG&E - Our top-10 Lessons Learned Other Query Performance tricks Wrap-up

Real Experience. Real Advantage.

PG&E Company Overview

Pacific Gas and Electric Company, incorporated in California in 1905, is one of the largest combination natural gas and electric utilities in the United States.

 There are approximately 20,000 employees who carry provide natural gas and electric service to 15 million people throughout a 70,000-square-mile service area in northern and central California.
Real Experience. Real Advantage.
6

PG&E Company Overview

 SAP since 1996  Currently on SAP ERP 4.72, upgrade to ECC 6.0 in 2009  8,500 users for all back office and field operations Finance/Controlling HR Org/Benefits/Personnel/Payroll Employee Self Service: 20,000 employees; 18,000 retirees Manager Self Service Supply Chain Management --Materials Management & Supplier Relationship Management Equipment/Work Order Management in Plant Maintenance Business Warehouse  First major go live was a 50,000 person day project in October 2007
Real Experience. Real Advantage.
7

PG&E SAP Project Overview

Detailed Design Phase
Design the foundation for the deployment of new processes and tools Design SAP customizations Design training and performance support

Roadmap Phase
Vision and Strategy To-be business capabilities Build business case for change

Aug 04 - Nov 04

Feb 05 Oct 05

Nov 05 Sep 06

Oct 06

Design Phase
Design Level 1 and Level 2 processes Application architecture Design To be organization to enable future business processes and technology
Real Experience. Real Advantage.

Foundational Release: Build, Test, and Deployment Phases

Build Customizations Build Procedures and Training Convert master and transactional data Assembly Test Product Test Deployment

PG&E Project Scope - EAM WRM

Real Experience. Real Advantage.

PG&E Project - Scope and Modules

SAP  PG&E has used SAP for almost 10 years in various Modules & forms Versions  R/3 4.7 utilized- (PM, PS, MM, SD,SRM)  Modifications to existing FI and HR-CATS functionality  Integration with third party applications: ClickSoftware (ClickPlan, & ClickSchedule), ITRON LD-Pro and ESRI GIS  SAP BI 7.0

Impacted Audience

8,500 (1,500 core EAM, rest for supply chain)

Real Experience. Real Advantage.

10

PG&E Project Benefits

Real Experience. Real Advantage.

11

PG&E Project BW and History - Performance Needs

Delivering High Performance Analytics to a potential user community of 8,500 users requires special considerations for design, technical architecture and hardware. I the spring of 2008, PG&E therefore started an examination of BI-A as potential tool for increasing performance of the financial dashboards in SAP BI. The scope of this effort was primarily finance, but the benefits were also intended to benefit a larger user community.
Real Experience. Real Advantage.
12

[ Agenda
 Introduction  Pacific Gas & Electric Project Overview  What is SAP BI-A
 Overview  Building aggregates  Benefits

 How we implemented BI-A  Sizing the environment

 Team organization and project plan  Testing

 BI-A Limitations  PG&E - Our top-10 Lessons Learned  Wrap-up

Real Experience. Real Advantage.

13

Why BI-A? Disk speed is growing slower than other HW components

Technology Drivers
1990 CPU 2009 Improvement

Architectural Drivers
1990 Disk based data storage Simple consumption of apps (Fat client UI, EDI) Generalpurpose, applicationagnostic database 2009 In-memory data stores Multi-channel UI, high event volume, cross industry value chains Applicationaware and intelligent data management
Source: 1990 numbers SAP AG 2009 numbers Dr. Berg

0.05
MIPS/$

8.30
MIPS/$

166x 425x

Memory

0.02
MB/$

8.51
MB/$

Addressable Memory

216
100
Mbps

264
10
Gbps

248x
100 x

Network Speed

Disk Data Transfer

5
MBPS

400
MBPS

80x

Physical hard drive speeds only grew by 80 times since 1990. All other hardware components grew faster. Real Experience. Real Advantage.

14

[BI Accelerator Works by Indexing InfoCubes in Hardware memory

BI Accelerator has been simple to install and operate for established SAP BI teams You use process chains to maintain the BI-A engine after each data load
SAP BW Any tool

Real Experience. Real Advantage.

IBM, Sun, HP & Fujitsu have standard solutions ranging from $32K to $300K+ that can be installed and tested in as little as 2-4 weeks (+ SAP license fees)

32 GB Blades are certified by SAP

15

[ BI Analytical Engines Query executing priorities

Information Broadcasting / Precalculation Information Broadcasting / Precalculation

Query Cache

Query Cache

Aggregates

SAP NetWeaver BI Accelerator

InfoProvider

Query Execution Without BI Accelerator

Query Execution With BI Accelerator

Aggregates can be replaced with BI-A, while the memory cache is still useful.
Real Experience. Real Advantage.
16

[ Performance is Near-Linear: Benchmarks

BIAs strength resides in its near-linear scalability Performance is measured in terms of:
1. BI-A index creation time 2. Multi-user throughput per hour 3. Average report response time 4. Average number of records touched by each report

BIA Currently reads data from InfoCubes. DSOs & InfoObjects are still read from base/physical tables (even when the InfoObject is indexed as part of master data).
Real Experience. Real Advantage.
17

[ Most customers have found BIA Admin to be minimal

The training & admin to run BI Accelerator has proven to be minimal. Training is included in the SAP courses: BW310, BW360 and in the online course BW361

Many companies, have spent 2-5 days in BIA training (total for 1-2 administrators) and a work through a single interface:
RSDDBIAMON (Checks available at: RSRV)

In the past, SAP has also offered a 2 day workshop called BI Upgrade & Accelerator (WNABIA)
Real Experience. Real Advantage.
18

[ Agenda
 Introduction  Pacific Gas & Electric Project Overview  What is SAP BI-A
 Overview  Building aggregates  Benefits

 How we implemented BI-A  Sizing the environment

 Team organization and project plan  Testing

 BI-A Limitations  PG&E - Our top-10 Lessons Learned  Wrap-up

Real Experience. Real Advantage.

19

[ Sizing was critical, but good tools were available to help

SAP has integrated key sizing parameters for BIA in the new QuickSizer. You can give it a try at: http://service.sap.com/quicksizer
Customer with existing BI systems and data loaded can get good sizing estimates by running the SAP program available in SAP Note: 917803
Real Experience. Real Advantage.
20

[ Some Hardware options we looked at

Operating Systems and Storage SUSE Linux 64-bit, Intel based Memory based and file storage (no database for Accelerator) Available Hardware Platforms HP (HP ProLiant and HP StorageWorks) IBM (IBM Blade Center and IBM Total Storage) FujitsuSiemens (FSC Blade Server and EMC Storage) SUN Microsystems (SUN Blade Server and StorageTek)
The hardware implementation is done by the vendors. Be prepared for lead times on delivery of large systems (Made-to-Order is common).
Real Experience. Real Advantage.
21

[PG&E - Our Initial Hardware sizing (July 2008)

Environment Production Production Production Production Production Production Production Production QA QA QA QA QA QA QA QA Development Development Development Development Development Development Development Development Area Blade servers Memory Processors Processor speed Network cards External storage File system Chassis Blade servers Memory Processors Processor speed Network cards External storage File system Chassis Blade servers Memory Processors Processor speed Network cards External storage File system Chassis Recommended size 14 Blades 2x8 GB (2x4) DDR2 total 16 GB 2 x Quad Core Intel Xeon Processor 3.00 GHz+ 2 x Gigabit Cisco cards Dedicated disks (500 GB+) General Parallel file system (GPFS) 14 blades capacity 14 Blades 2x8 GB (2x4) DDR2 total 16 GB 2 x Quad Core Intel Xeon Processor 3.00 GHz+ 2 x Gigabit Cisco cards Dedicated disks (500 GB+) General Parallel file system (GPFS) 14 blades capacity 4 Blades 2x8 GB (2x4) DDR2 total 16 GB 2 x Quad Core Intel Xeon Processor 3.00 GHz+ 2 x Gigabit Cisco cards Dedicated disks (300 GB+) General Parallel file system (GPFS) 14 blades capacity IBM example* BladeCenter HS21 -8853G6U 39M5797 2 x Quad Core Intel Xeon Processor 3.00 GHz 32R1760 DS-4800 GPFS H-series (rack-mount/9U) 88524XU BladeCenter HS21 -8853G6U 39M5797 2 x Quad Core Intel Xeon Processor 3.00 GHz 32R1760 DS-4800 GPFS H-series (rack-mount/9U) 88524XU BladeCenter HS21 -8853G6U 39M5797 2 x Quad Core Intel Xeon Processor 3.00 GHz 32R1760 DS-4800 GPFS H-series (rack-mount/9U) 88524XU

We found that the BIA should be sized for critical applications. Most companies use BIA only for Production, while others have a complete landscape
Real Experience. Real Advantage.
22

PG&E - Our Query Performance Improvments

Queries Current performance (seconds)
169.25 243.40 171.39 136.84 179.09 201.90 166.37 84.66 173.60 122.91 171.31 91.01 45.06 83.37 96.28 69.32 107.86 163.16 100.94 98.82 62.01 52.59 32.99 76.94 24.48 62.43 108.49 46.38 25.87 27.39 42.83 36.18 24.60 26.55 21.98 20.51

With BIA
14.54 21.37 18.28 15.72 21.25 25.23 24.59 14.39 29.82 21.40 29.86 18.12 9.43 17.55 23.52 17.73 28.58 45.07 28.18 29.56 19.35 17.52 11.44 28.46 9.55 24.49 42.84 18.86 11.09 11.83 18.98 16.51 11.34 12.54 10.60 10.09

Improvement
1064% 1039% 838% 770% 743% 700% 577% 488% 482% 474% 474% 402% 378% 375% 309% 291% 277% 262% 258% 234% 220% 200% 188% 170% 156% 155% 153% 146% 133% 132% 126% 119% 117% 112% 107% 103%

CS Expense - 5 Year History Charges to Partners via Orders - Units Charges to Partners via Orders $ Std Cst Var - PCC Count ActPlan vs Target by MWC - Summary BU Expense - 5 Year History CR - Expense Order Cost In Labor Cost In Hours 24 Month Rolling FC - Amounts ActPlan vs Target by MWC - Expense SR_RCC_Order Costs by Planning and Actual Orders Labor Prod ST Hours CWIP - LOB Paid OT Hours ActPlan vs Target by MWC - Non Earnings PCC Labor $/Qty by CE, Plan, Act - Mid-Year ActPlan vs Target by MWC - OBS CR - PCC Costs Billed Out Annual Amt by MWC Monthly Run Rate vs Annual EOY Monthly Run Rate by Primary Cost Elem Activity Price Report (KSBT) PCC Non-Labor $ by CE, Plan, Act - Mid-Year Month/YTD/Annual Amt by MWC Actual Cost Line Items for Cost Centers (Material & Service) CR - PCC Costs ActPlan vs Target by MWC - Capital CR - PCC Costs Billed Out to CC Order: Actuals since Inception Paid Labor Hours (excludes Overtime) SR_PCC Costs_In_Out - Graph Actual Cost Line Items for Orders (Other CO Postings) Actual Cost Line Items for Cost Centers (Non-PO Invoices) Billable Hours PCC Std CstExperience. Real Advantage. Real Var Plan - Mid-Year PCC Billable Hrs, $ by Actv, Plan, Act - Mid-Year

The slowest queries that spent most of their time on the database side benefited the most Overall 36 of 70 queries were at least twice as fast.

23

PG&E - Our Query Performance Improvments

Queries Current performance (seconds)
21.17 38.13 11.36 63.62 34.33 34.33 17.00 16.66 14.79 8.23 32.81 30.41 49.62 32.81 21.39 17.66 27.19 17.18 24.89 24.48 18.34 25.43 6.68 24.52 6.01 15.43 6.15 39.63 17.80 93.04 4.57 4.10 3.63 10.37

With BIA
10.74 19.44 5.90 33.55 18.28 18.34 9.19 9.18 8.23 4.58 18.35 17.12 28.09 18.86 12.34 10.51 16.25 10.43 15.33 15.59 11.90 16.55 4.50 16.68 4.23 11.58 4.77 35.42 15.91 85.98 4.29 4.02 3.94 11.99

Improvement
97% 96% 93% 90% 88% 87% 85% 81% 80% 80% 79% 78% 77% 74% 73% 68% 67% 65% 62% 57% 54% 54% 48% 47% 42% 33% 29% 12% 12% 8% 7% 2% -8% -14%

Actual Cost Line Items for Cost Centers (CATS & Order Confm) Actual Cost Line Items for Cost Centers (Other FI Postings) Summary Removal Costs Cost Report by Order/Cost Element (PORR/PORY) SR_PCC Costs by Cost In and Cost Out SR_PCC Costs by Cost In and Cost Out SR_PCC Costs_In_Out - Graph PCC Activity Prices Spend - Tracked/Not Tracked Savings % Actual Cost Line Items for Orders (NEBS) Actual Cost Line Items for Orders (Material & Service) PCC Detail Planning Report (KSBL) Actual Cost Line Items for Orders (Other FI Postings) PCC Headcount by Month - Classification SR_PCC Costs by Cost In and Cost Out - 12 Months Actual Cost Line Items for Orders (Non-PO Invoices) Actual Cost Line Items for Cost Centers (Other CO Postings) Actual Cost Line Items for Cost Centers (NEBS) Actual Cost Line Items for Orders (CATS & Order Confm) Actual Cost Line Items for Cost Centers (CO Reposting) Actual Cost Line Items for Orders (CO Reposting) Conforming % Activity Type Planning Conforming % PCC Headcount by Month - Classification (Graph) Removal Costs Resource Report by PCC/Activity Type (PORV) PCC FTE Count by Planning Version - Mid-Year CWIP - LOB & Orders Savings $ Savings $ Savings Real Experience. Real Advantage. % Conforming % and Savings %

58 of 70 queries were at least 50% faster

Before BIA the average query execution took 58.8 seconds, after BIA the average query took 17.9 seconds (295% faster).

24

[ PG&E - Query Performance Improvments OVERALL

Query execution time before BI-A
10 9 8

Number of Queries

7 6 5 4 3 2 1 0 10 15 20 25 30 35 40 45 50 55 60 70 80 90 100 110 120 140 160 180 200 220 240 260

Seconds

Query execution time after BI-A

25 20
Number of Queries

15 10 5 0 10 15 20 25 30 35 40 45 50 55 60 70 80 90 100 110 120 140 160 180 200 220 240 260 Real Experience. Real Advantage. Seconds

The major improvement was to make query execution more predictable and overall faster

25

[ PG&E - Workbook Performance Improvments with BIA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Current With BIA Top 20 Workbooks Improvement performance (seconds) (seconds) ORD205 Actual+Plan by MWC W/Plng Orders 614.63 94.75 649% SR_RCC_202 -Order Costs by planning and Actual Orders 171.31 29.86 574% ORD225 - Actuals since Inception 25.87 11.09 233% Forecast Model (Summary Reports) 438.34 113.01 388% ORD418 - Compliance & Savings 58.38 45.78 128% ASM001 - CWIP (LOB, MWC, Order) 155.61 106.08 147% ORD201 Reports 76.94 28.46 270% ORD207 - 5 year historical & FCST 100.94 28.18 358% PMG008_24Mo PRJ Fin by Area/Div/RCC/MWC/Ord/PrjMngr/Mo 173.60 29.82 582% PMG005 Reports 63.62 33.55 190% ORDPMG001-PORV 39.63 35.42 112% ORD230 - Line Items for Order 209.31 119.23 176% PCC130 - Line Items for Cost Center 168.79 88.73 190% PCC016 - PCC Detail Planning Report (KSBL) 126.73 62.29 203% PCC018 - Mid-Year PCC Planning Assumptions for SLSR 179.26 74.95 239% PCC116 - Standard Cost & Labor Trend for SLSR Groups 136.84 15.72 870% SR_PCC_102 - PCC Costs by In-Out Costs 94.82 47.77 198% RCC209A - BU/CS Expense: 5 year history 371.15 39.77 933% SR_PCC_401 - Charge-Out Cost Center Type A or E 815.92 165.53 493% ORD209 - Plan vs Actual By MWC and primary CE 160.83 48.91 329% Average run times (same selection criteria) 209.13 60.95 363%

Average performance improvement for workbooks was 363% faster

Real Experience. Real Advantage.
26

[Performance Info PG&E BI-A Indexes build times

Infocubes No of Records ZMIM_C04 19,867,026 ZMIM_C03 19,200,412 ZMIM_C01 2,287,483 ZKCC_C26 125,570 ZKCC_C13 16,287,318 ZKCC_C12 32,808,858 ZKCC_C09 1,509,754 ZKCC_C06 25,346,239 ZKCC_C04 2,936,394 ZKCC_C03 35,154,491 ZKCC_C02 33,985,298 ZKCC_C01 27,747 ZFAA_C02 6,611,084 ZFAA_C01 765,838 KOD_C208 28 KOD_C207 3,059,811 KCC_C508 1,036,593 KCC_C408 3440 KCC_C407 1,350,169 KCC_C23 3,058,715 KCC_C22 20,745,219 KCC_C208 3,346,213 KCC_C207 2,784,692 Total 232,298,392 RealaverageReal Advantage. Experience. 2,565,132 Time to build Indexes ( sec) 327 196 95 15 138 208 68 341 71 615 225 26 1520 113 10 148 63 11 84 677 149 168 166 90.56 minutes records indexed per minute

232,298,392 records in 23 infocubes were indexed in 90.56 minutes. On average we were able to build indexes on 2.56 million records per minute

27

[ Team organization at PG&E

1. We had a small team of 3 people dedicated to the BIA project supported by one basis person, and the rest of the BW team 2. Hardware team supported by the vendor was also involved in a part-time capacity. 3. Overall the effort was combined with a performance tuning project that also included building of aggregates, caching and query redesigns.
Real Experience. Real Advantage.
28

[ PG&E - Testing
1. We had testers and requirements gathered from the finance community.The goal was at least 50% faster on the 38 web dashboards. 2. We achieved:
a) 363% faster for workbooks on average b) 70% faster for queries on average c) 58 of 70 queries were at least 50% faster d) Index builds took about 1.5 hrs

Real Experience. Real Advantage.

29

[ Agenda
 Introduction  Pacific Gas & Electric Project Overview  What is SAP BI-A
 Overview  Building aggregates  Benefits

 How we implemented BI-A  Sizing the environment

 Team organization and project plan  Testing

 BI-A Limitations  PG&E - Our top-10 Lessons Learned  Wrap-up

Real Experience. Real Advantage.

30

[ PG&E - Top 10 Lessons Learned

1. Plan to order hardware early - lead times can be long 2. Make sure you have a committed project sponsor 3. Create benchmarks for workbooks, queries and cockpits to document success 4. Update to latest service pack 5. Involve your basis team and environment management team early
Real Experience. Real Advantage.
31

[ PG&E - Top 10 Lessons Learned

6. Be realistic, not all queries will benefit equally from SAP BIA 7. Plan for testing and gradual rollout of BIA to key users 8. Make sure failovers and spare blades are part of your hardware sizing 9. Create a small dedicated team with access to external expertise as needed 10. Perform query tuning and potential redesign as part of a larger system performance tuning effort.
Real Experience. Real Advantage.
32

[ Agenda
 Introduction  Pacific Gas & Electric Project Overview  What is SAP BI-A
 Overview  Building aggregates  Benefits

 How we implemented BI-A  Sizing the environment

 Team organization and project plan  Testing

 BI-A Limitations  PG&E - Our top-10 Lessons Learned  Wrap-up

Real Experience. Real Advantage.

33

[ How does it work in reality??

Currently, the BIA performs aggregation and data selection for the query, all other processing is done by the OLAP analytical engine. Not all query processing will be dramatically faster In the next SAP NetWeaver release the BIA will handle more of the analytics processing such as top-5 products sales which is currently done in the OLAP analytical engine. SAP BIA is not used when the result set exceeds 3 million records (max. default). When the result set is less, the data is sent as one large data package to the application server (need fast network).

Real Experience. Real Advantage.

34

[ BIA is becoming mainstream

Some of SAP reference clients

BIA is no longer something exotic. Many of the large BI systems have already implemented BIA and many more projects are under way in Europe and in the Americas.

Nike

Once you exceed a few hundred critical users and/or 34 TB of data you should seriously consider SAP BIA
Real Experience. Real Advantage.
35

The Future SAP BI-A RoadMap

Prior to 2008
SAP NetWeaver BI Accelerator
Accelerator for InfoCube Read Times Low Cost of Operations High Availability / Backup & Recovery

2008
Optimized BusinessObjects Webl integration on top of SAP NetWeaver BI/BIA BusinessObjects Polestar integration: SelfService BI based on search paradigm

2009/2010
Analytic Capabilities (e.g. top n) DataStore Object Acceleration Standalone BIA for source-independent acceleration via Business Objects Agile BI: Local and source-independent innovation space for departments

Real Experience. Real Advantage.

36

[ Agenda
 Introduction  Pacific Gas & Electric Project Overview  What is SAP BI-A
 Overview  Building aggregates  Benefits

 How we implemented BI-A  Sizing the environment

 Team organization and project plan  Testing

   

BI-A Limitations PG&E - Our top-10 Lessons Learned Wrap-up Question and Answers

Real Experience. Real Advantage.

37

Key Learnings

 BI-A is a useful tool for performance tuning, but it is not a solve-all solution for poorly made queries  It was easy to implement (very quick)  Look at how your queries are executing (i.e. database or application server usage) when estimating benefits of implementing BI-A  Once you exceed a few hundred critical users and/or 3-4 TB of data you should seriously consider SAP BIA  The administration of BI-A is simple and anyone can learn it in a few days  Plan on waiting for hardware, we found that most was made-to-order  Make sure you have redundancies and check that what you order is what got shipped.
Real Experience. Real Advantage.

38

[ Resources
Dr. Bjarne Berg's web page -- 85+ presentations, tutorials & articles http://csc-studentweb.lrc.edu/swp/Berg/BB_index_main.htm

Large scale testing of SAP BI Accelerator on a NetWeaver Platform https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/b00e7bb5-3add-2a103890-e8582df5c70f

Real Experience. Real Advantage.

39

[ Questions and Answers

How to contact us: Dr. Bjarne Berg Darin Lemos bberg@comerit.net D4JLY@pge.com

Real Experience. Real Advantage.

40

 Thank you for participating.

Please remember to complete and return your evaluation form following this session. For ongoing education on this area of focus, visit the Year-Round Community page at www.asug.com/yrc

]
41

SESSION CODE: INSERT SESSION CODE

Real Experience. Real Advantage.