Professional Documents
Culture Documents
September 2012
Test Detail .......................................................................................................................... 19 Results ............................................................................................................................... 19 Other Metrics....................................................................................................................... 22 Benchmark Results for CM - Multi-user Testing ................................................................. 24 Test Detail .......................................................................................................................... 24 Results ............................................................................................................................... 25 Other Metrics....................................................................................................................... 31 Account Reconciliation Manager Performance ..................................................................... 32 An Example of a Heavy Load ARM Configuration ............................................................... 32 Tuning Parameters Oracle DB ............................................................................................. 32 Tuning Parameters JVMs .................................................................................................... 33 Tuning ................................................................................................................................ 34 Results ............................................................................................................................... 34 Benchmark Results for ARM - Manual Testing ................................................................... 35 Test Detail .......................................................................................................................... 35 Results ............................................................................................................................... 35 Other Metrics....................................................................................................................... 36 Benchmark Results for ARM - Multi-user Testing .............................................................. 37 Test Detail .......................................................................................................................... 37 Results ............................................................................................................................... 39 Other Metrics....................................................................................................................... 49 Summary ............................................................................................................................. 50 Using Other EPM Products ................................................................................................. 50 Separating Out Products ................................................................................................... 50 Finally ............................................................................................................................... 50
Introduction
This document is written for people who install and configure the components in an EPM/BI environment, and people who monitor and tune the performance of the components in an EPM/BI environment. This document focuses on Oracle Hyperion Financial Close Management (FCM). It is assumed that readers understand the administration and performance tuning fundamentals; for example, server hardware, web servers, java application servers, databases. Readers should also be familiar with the how the Financial Close Management application is used in their environment. It is also assumed that the reader understands the roles of the different components within an EPM environment needed for each EPM Products being used. Components you often find within an FCM based EPM environment include Oracle HTTP Server (OHS), WebLogic, Shared Services, ERPi and SOA. Financial Close Management as of 11.1.2.2 contains two distinct feature-sets: Close Manager (CM) and Account Reconciliation Manager (ARM). From a performance and tuning perspective, these feature-sets are regarded as integrated entities, but have different performance tuning requirements. This document is divided into three main sections: 1. Financial Close Management Performance - Covers performance issues common to both feature-sets of FCM 2. Close Manager Performance - Covers issues mostly unique to CM 3. Account Reconciliation Manager Performance - Covers issues mostly unique to ARM At the end of each section, benchmarking details and results are included to give the reader an idea of typical end-user performance and system resource usage. The Account Reconciliation Manager Performance section also contains a set of results for ARM used to generate the tuning recommendations. As a result of the different tuning requirements for CM versus ARM, there are some tuning recommendations which vary slightly between the CM and ARM sections in this document. The information presented in this guide is a clear indicative on tuning a system to support large loads. For example, to look for a tuning recommendation to run a FCM schedule with 2000 tasks, you would look for the examples given in this guide along with the hardware information. From this information you should arrive at a close estimate on the things that you need for your environment. Start with making those changes to the system. Also see the Oracle Enterprise Performance System Installation and Configuration Troubleshooting Guide. Performance tuning is an exercise that is influenced by a large number of external factors and the limitations and constraints of different tech-stacks in use. At the end of this document there is a summary section which applies to both CM and ARM.
Performance Terminology
This guide uses the following performance terms: Scalability - The system's ability to perform within specification under increasing user load, data load and hardware expansion. Latency - The time between the issuing of a request and the time when the work actually begins on the request. Think time - The time a real user pauses to think between actions. Resource utilization - A consumption metric, for example, the percent of CPU usage. Response time - A time metric, for example round-trip time it takes the server to deliver a Web page. Throughput - A rate metric (requests per unit of time), for example, requests per second, bits per second. For example, if an application can handle 20 customer requests simultaneously and each request takes one second to process, this site has a potential throughput of 20 requests per second. Understanding Key Performance Drivers - A factor that dictates performance is called a key performance driver. Knowing how the drivers behave in combination further enhances your ability to
Oracles Hyperion Financial Close Management Performance Tuning Guide 4
deploy Oracle Hyperion EPM System optimally, based on the unique requirements of each deployment. Hardware Capacity - Factors such as number of servers, quantity and speed of processors, available RAM, network speed. Technical Platforms Tuning - Fine tuning other third party software required for installing and running Oracle Hyperion EPM products; for example: relational databases, Java application servers, Web servers, Server / Client Operating System and browsers. Business Application Design - Application design is an important factor in system performance, (for example, structure, size, and use of product features in designing application databases, reports, schedules, profiles, templates, copy-to-period, data-load.) Business Process Usage - Activities carried out by users in the normal flow of your business cycle. Business process usage has three components: User Activity - Activities available to users for data load or data entry, database processing (for example, consolidations, copy, clear), and reporting and analysis Rate of User Activity - A number of transactions executed by one user per one hour User Concurrency - Number of users for each activity being carried out simultaneously
Workload Types
FCM has several types of workloads to monitor: There is the pure interactive workload of a user performing something in the FCM Web user interface (for example, within EPM Workspace) that results in activity on the servers until the point the users gets a confirmation and/or result in Workspace. During this time the user normally waits for the reply before continuing and no background processing continues after the user gets the reply. Background tasks are run (automatically or trigged by the user via Workspace or from an external event in another product). Background tasks may not directly interact with users via Workspace, although they may check progress from within Workspace indirectly. We can also regard some long running Workspace foreground tasks as background tasks, such as the Import & Export functions. Background tasks often involve large sets of data. Some tasks are a mix of the previous two. Changing the status of some objects within FCM will just ask SOA to schedule the work; therefore this is an interactive task that triggers a background task. From a performance and capacity planning point of view, these different types of workloads have different tuning objectives. Background tasks tend to be batch, similar to the way all available resource are used until the work is complete their driver being the volume and complexity of the data being processed. Performance delays with interactive or foreground tasks are more noticeable to users, and tend to have a lower resource impact per user on the system.
Database Considerations
During installation the installation engineer will typically configure different databases (SQL Server) or schemas (Oracle) for different EPM products and components. For example (XXX_SOAINFRA, XXX_MDS & XXX_ORASDPM) are normally used for SOA in an Oracle DB. In case the database is also used for hosting schemas of other products, you must take these schemas into considerations when changing parameters so that the change will not affect the functioning of other products. Most DBAs configure the DB to use a dedicated instance for all EPM
Oracles Hyperion Financial Close Management Performance Tuning Guide 5
products in the one environment, and use separate databases or schemas within the instance for each of the EPM products and components. For good performance, including the ability to cope with large or complex datasets, the Database Administrator (DBA) engineer must optimize the database instance for CM and ARM. Optimization includes: Placing the database files on suitable disks Setting database parameters (init.ora / spfile in the case of Oracle) Running routine maintenance tasks Oracle and SQL Server Database Optimization The administrator performs routine database optimization to preserve optimal performance after operations in which large amounts of data are entered into the system or copied within the system. In the case of an Oracle DB, this optimization normally equates to running dbstats against the relevant schema. In Close Manager, optimize after: A template with a large number of tasks has been deployed as a schedule A large number of tasks have been imported into a template or a schedule In Account Reconciliation Manager, optimize after: A data load from ERPI has been performed A large number of transactions have been imported A large number of reconciliation profiles have been copied to periods To optimize Financial Close Management tables, use an available SQL connection tool and connect using the Financial Close Management schema user and execute: Gathering schema statistics using Oracle:
where 'FCC' is replaced with the schema name appropriate for the ARM schema Gathering schema statistics using SQL Server:
where
Oracle Enterprise Manager (for monitoring the Oracle Database) The Oracle Enterprise Manager database console can be used to analyze the performance of the Oracle database. It provides alerts automatically when poor performance is detected via the Automatic Database Diagnostic Monitor. The database console is launched via the URL:
The ADM Performance Analysis section displays current alerts and the Performance tab contains charts illustrating database performance.
For best performance, it is recommended that both the initial memory allocation value and maximum memory allocation value are set to the same amount. This reduces memory fragmentation and it bypasses part of the VM memory manager processing which dynamically requests memory from the operating system as usage increases. The Sun Java virtual machine has an additional memory parameter, , which needs to be set to a sufficient value to accommodate memory-intensive operations. The HIT installer provides an appropriate default value, but if out-of-memory errors are experienced, this value can be increased in the and (or ) files. Monitoring the Java Virtual Machine (JVM) Memory Several tools exist which allow the administrator to monitor the performance of a Java virtual machine and to observe how much CPU time is being spent reclaiming memory. All of the following tools have memory usage charts as well as counters which indicate how often memory reclamation occurred and how much time was spent reclaiming memory. Java virtual machine memory is subdivided into two main pools: a pool for short-lived and young objects known as the nursery, and a pool for longlived, permanent, or old objects known as old space. Each pool has a garbage collector housekeeping process which reclaims memory periodically and this process consumes some CPU power. Ideally garbage collections should be infrequent and fast and the following tools allow the administrator to determine if the garbage collection is running too often or requiring too much CPU. JVM Monitoring - Sun JConsole The utility Windows executable or Linux script is located in:
JVM Monitoring - Oracle Enterprise Manager The Oracle Enterprise Manager (OEM) console requires that the WebLogic Admin server is running. The EM console is accessed via the URL Navigate to click on FinancialClose0 and select Performance Summary
Oracles Hyperion Financial Close Management Performance Tuning Guide
, right
8
The default display includes a memory usage chart. Click Show Metric Palette to add additional metrics. Recommended metrics to observe JVM performance are:
Typically large chunks of the nursery memory will be consumed and freed very often while the old space memory, if the JVM is tuned properly, will remain fairly consistent, unless a large number of users are logging in or logging out of the system and sessions are being created or eliminated. JVM Monitoring - JRocket Mission Control Far more in-depth JVM monitoring can be performed if you are using the default JRocket JVM. You can use the Oracle JRocket Mission Control. This can also be used if you are using the Sun JVM, but the flight recorder option will not be available. To see more on this refer to the Using Oracle JRockit Mission Control Monitoring information at the end of this section.
Tuning SOA
How each feature set uses SOA is different, so the intended use of Financial Close Management (CM or ARM) will influence the tuning process. High load with CM tends to have a bigger impact on SOA than the Financial Close Management JVM, whereas higher impact on ARM tends to only impact the Financial Close Management JVM. This impact is in the form of memory use and CPU use. The main item to tune for SOA is the maximum number of connections each data-source can have open in its connection pool. Monitoring and tuning of these connections is via the WebLogic Console. Although ARM does not use SOA as much as CM, ARM still uses many JDBC / data-source connections. Also, make sure to monitor the total number of sessions that your database can accept so that none of the data-sources run short of connections / sessions to connect to the database. Refer to the link below for the standard database tuning recommendations for the SOA server:
http://docs.oracle.com/cd/E23943_01/doc.1111/e17364/tuning.htm
To adjust the operating system: 1. Open the Control Panel, then open the System item, and then select the Advanced tab. 2. In the Performance section, click Setting to invoke the Performance Options dialog. 3. Select the Advanced tab and in the Memory Usage section, select Programs.
A side effect of using the 3GB switch is a reduction in the operating systems Free System Page table. This number of free system page table entries should not go below 7,000 and an ideal number is 24,000. The number of free page table entries can be viewed using the Performance Monitor tool. Add the Memory counter Free System Page Table Entries and make note of the values for minimum and average. You can then use the switch to achieve a desirable amount.
Enabling Mission Control Monitoring Support By default, the server nodes are installed without support for Mission Control monitoring enabled. Server startup scripts must be modified in order for the Mission Control tool discover and connect to JRockit virtual machine instances. An extra initialization parameter is added to the arguments which are passed to the virtual machine executable. To add the extra initialization parameter: 1. Locate the startup script servers) The file is typically in a location such as: for non-Windows
where [Weblogic Domain] is the directory of the domain which you want to monitor. 2. Locate the line with the following text: 3. Add the following argument to the end of the line, separated by a space: Enabling Flight Recorder Monitoring Support To enable Flight Recorder Monitoring support: By default, the server nodes are installed with support for JRockit flight recorder monitoring disabled. To enable the support: 1. Modify Server startup scripts for the Mission Control tool to activate flight recorder monitoring. 2. Remove the initialization parameter from server startup scripts. 3. Locate the startup script ( for non-Windows servers). 4. The file is typically in a location; for example: where [Weblogic Domain] is the directory of the monitored domain. 5. Make a backup copy of the file.
Oracles Hyperion Financial Close Management Performance Tuning Guide 11
7. Locate the startup script setCustomParamsFinancialClose.bat (setCustomParamsFinancialClose.sh for non-Windows servers) The file is in the location: 8. Make a backup copy of the file and locate and remove the text: .
12
Connecting Mission Control to a WebLogic Server Instance In the Mission Control application, a list of discovered WebLogic server nodes is displayed:
If it is unclear which server node is the desired node, select a node, choose Start Console, and chose the Runtime category. In the System Properties panel, locate the property.
13
Recording Virtual Machine Metrics Select the pertinent WebLogic server node, right-click and select Start Flight Recording.
You can select from a predefined profiling template and enable or disable specific metrics based on the type of analysis being performed. Once recording is enabled, the recording will be displayed in the Flight Recorder Control panel of the Mission Control interface. Recordings can also be opened thru the File menu.
14
Developers can benefit from the recordings code profiling that identifies heavily utilized classes or from the event profiling which quantifies time spent; for example, by garbage collection, blocked threads, code compilation.
Benchmarking Configuration
During product development tests are run to check the performance of Financial Close Management. Although these tests are designed to provide feedback to the Development Team, this information can be used as a rough benchmark guide by customers and partners implementing FCM. Later in this document there are two Benchmark Results sections showing the results of these tests for CM and ARM respectively. The tests were performed on the same test environment. Details of this environment are covered here. Environment Detail - Hardware Two servers were used for the testing, excluding the server used to drive the multi-user test (workload injector) for example, this is a single-node FCM environment. Server environments: pegbvps06: o o o o o o Dell PowerEdge 1950 v3 2 x quad-core Intel Xeon L5420 2.5GHz 16GB of memory 250GB local SATA disk Gbit NIC Windows 2008 R2 64bit Enterprise
pegbvps12: o o o o o o o o HP DL180 2 x quad-core Intel Xeon E5430 2.66GHz 8GB of memory Hardware RAID controller with 12 x 450GB SAS disks Gbit NIC Windows 2003 R2 64bit Enterprise Oracle 11gR2 database Database files all stored on RAID0+1 LUN
15
Environment Detail - Software All software installed on pegbvps06; for example, a single-node configuration. EPM PS2 (11.1.2.2) GA release ARM component upgraded to PSU300 pre-GA release (build#1613) Essential Financial Close Management components installed: o Financial Close Management (ARM and CM feature-sets) o Foundation (Hyperion Shared Services + Workspace) o SOA o WebLogic o The embedded Web server was used (not IIS or OHS) o ERPi/ODI was not installed. Data entry to Financial Close Management was entered directly to load the Financial Close Management database tables. Environment Detail - Tuning The following tuning was performed: Oracle database:
In some earlier testing of ARM for PSU300, it was necessary to increase the number of OPEN_CURSORS in Oracle from the default of 300 to 6000 for large concurrent user volumes. However, this is less of an issue with the later (and so GA release) versions of PSU300. WebLogic data source connection pool maximums (set higher than likely to be needed in a real production environment): accountreconciliation_datasource = 300 EDNDataSource = 200 EDNLocalTxDataSource = 200 EPMSystemRegistry = 300 mds-owsm = 200 mds-soa = 500 OraSDPMDataSource = 500 SOADataSource = 500 SOALocalTxDataSource = 500 In Financial Close Management In setSOADomainEnv.cmd:
16
Multiple tests have been made with different numbers of Financial Close Management tasks that start at the same time (300, 500 and 1000) and it has been observed that the default setting that comes along with configuration of the SOA server for the first time would handle the load. The only changes needed are changes to the data sources connection pool sizes.
17
Tuning Data Sources The recommended number of connections for the various data sources that are seen after completely configuring Financial Close Management are listed in the table below. Use the below numbers in the Maximum Capacity field for each data source. With the following numbers, Financial Close Management could successfully start 300 Financial Close Management tasks at the same time. Make sure to increase the total number of sessions that your database can accept so that none of the data sources run short of sessions to connect to the database. The number of sessions that a database should make available should be greater than the Sum of (Number of connections specified for each data source times the number of managed servers that a data source is targeted to). Data Source Default Numbers Initial Capacity EDNDataSource EDNLocalTxDataSource EPMSystemRegistry financialclose_datasource mds-owsm mds-soa OraSDPMDataSource SOADataSource SOALocalTxDataSource 0 0 0 1 0 0 0 0 0 Maximum Capacity 20 20 150 30 15 50 200 50 50 Suggested Numbers Initial Capacity 0 0 0 1 0 0 0 0 0 Maximum Capacity 20 20 150 30 15 50 20 100 50
18
Open schedule" ends when the UI "opening schedule" dialogue closes. processing1-The initial single-threaded background processing by Financial Close Management and SOA. processing2-The multi-threaded background processing. After processing2, the tasks are all marked as open, before this they will have a status of pending.
19
Key: Black = total CPU, max 97.802 over all 8 cores Green = CPU for Financial Close Management JVM (10th scale), max 281.810 (i.e. almost 3 cores) Blue = CPU for SOA JVM (10th scale), max 572.872 Red = CPU for Foundation JVM (full scale), max 17.880 Yellow = CPU for WL JVM (full scale), max 45.057 Dotted = system-wide run queue, max 35 The above listing clearly shows the high CPU use of this task. This starts at 13:58:20 (the "processing1" stage). A little over a minute later the work becomes multi-threaded (using all the available CPU resource available on the server - the "processing2" stage). This work for processing2 is performed by SOA (in blue) and Financial Close Management (in green) and even some WL (in yellow). Note: Be aware of the different scales when reading this graph.
Oracles Hyperion Financial Close Management Performance Tuning Guide 20
The graph below shows the CPU and Heap use for the SOA JVM:
21
The graph below shows the CPU and Heap use for the Financial Close Management JVM:
Other Metrics After a number of multi-user tests and the above test was run a number of times, the servers key memory metrics were as follows: Free memory = 3961 MB (physical memory is 16GB). Committed bytes = 12589 MB (this is the server's overall VM use)
22
The following screenshot shows the state of the WebLogic data sources after the test finished. Some of the values are the result of earlier system use, but it gives an idea of how high values do not get even when doing a heavy background task:
23
The following screenshot shows the composites and instances in SOA that resulted from a 1000 task schedule being created and opened:
2. Log on to Workspace. 3. Open CM application, bringing up task list. 4. Open a task (each vuser uses a different task). 5. Click Add Comment. 6. Enter comment text. 7. Click OK to save the comment. 8. Close task dialogue, returning to task list. 9. Log off, returning to Workspace home page. 10. Close browser (but do not clear browser cache). 11. Wait for between 60 and 90 seconds randomly - this is called pacing. 12. Repeat the above task sequence from the start. Each vuser pauses between 1 and 3 seconds randomly to simulate the pauses real users would have - this is called think-time. Vusers were introduced to the system in two batches of 50 with a 10 minute delay between the two batches. Both batches start a new vuser at the rate of one per 20 seconds. Each vuser starts the above sequence of tasks as soon as it is started. Once all 100 vusers are on and running, the test continues for one hour, with all 100 vusers cycling through the task list repeatedly. After the hour the ramp-down starts, stopping each vuser at a rate of 5 per 30 seconds.
Results Scenario Name: Duration: Maximum Running Vusers: Total Throughput (bytes): Average Throughput (bytes/s): Total Hits: Average Hits per Second: Transactions: Total Passed: Total Failed: Total Stopped:
Transaction Name a01_001_home a01_002_logon a01_003_cm a01_004_opentask a01_005_addcom a01_006_com_ok a01_007_close a01_008_logoff Minimum 0.015 0 0.453 0.246 0.078 0.031 0.063 0.016
cm_scale100.lrs 1 hour, 53 minutes and 57 seconds. 100 2,988,414,356 437,030 645,059 94.334 42,152 0 0
Pass Fail Stop 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Average Maximum Std. Dev. 90% 0.04 0.033 0.534 0.313 0.136 0.081 0.215 0.047 0.484 0.738 1.266 1.188 0.828 0.859 1.219 0.747 0.027 0.024 0.095 0.072 0.062 0.063 0.228 0.037
0.043 5,269 0.043 5,269 0.693 5,269 0.334 5,269 0.203 5,269 0.16 5,269
25
The following graph shows how the users were introduced during the test run:
The following graph shows how the transaction response times changed over time. Note: There is no significant increase in response times as more users were added.
26
The following graph shows how the transaction response times changed over time: Note: There is no significant increase in response times as more users were added.
27
The following graph shows the impact of the test on selected server metrics. It is seen that the only process making any real use of the CPU is the Financial Close Management JVM. It peeks at well over 300%, which indicates it is running multi-threaded:
This graph also shows that CPU increases broadly inline with the number of concurrent users on the system. Key: red blue green yellow black orange = CPU for Foundation JVM = CPU for SOA JVM = CPU for Financial Close Management JVM (at half scale when presented on graph area) = CPU for WL JVM = Total CPU over all 8 cores = Total CPU over all 8 cores on DB server
Be aware of scaling and the impact of granularity when reading the above graph; for example, many of the points are averages, and the true maximums (even for a millisecond) are as shown in the table under the graph. You can see the server's memory usage from the metric list at the bottom of the above graph: Free memory ~= 3845 MB (physical memory is 16GB). Committed bytes ~= 13160 MB (this is the server's overall VM use).
28
The following graph shows the CPU and Heap use for the SOA JVM:
29
The following graph shows the CPU and Heap use for the Financial Close Management JVM:
30
Other Metrics The test created 1514 comment records. The following graph shows the WebLogic data source use after the test finished. Note: The Active Connections High Count of 35 for financialclose_datasource / FinancialClose0 JVM.
31
* Foundation includes Shared Services and Workspace # Using embedded web server Tuning Parameters Oracle DB The following are the formulas that Oracle recommends for setting the initialization parameters. Using an available SQL connection tool, connect using an account with administrative privileges and execute:
For example, to support 500 simultaneous sessions, set the initialization parameters:
32
where is the maximum number of concurrent users multiplied by 50 plus 200 more cursors to handle background processing and SOA. For example, for 200 concurrent users, the number would be 10200 Restart the database service. It might be simpler to connect to the database and run the commands by hand rather than updating a script file, copying it to the server, running SQL Plus, and executing the script. To get the current value of these settings, run the following query:
In
33
Tuning The above tuning settings are set high to make good use of the available resources on the servers. For most tasks many could be set lower with no impact on performance. The above WebLogic settings were maximum values used for testing ARM to 125 concurrent users. We recommend you start with these values for a heavy-load 125 user system, then lower them having looked at the WebLogic Console's Services / Data Sources (Monitoring tab) screen's peak values after a period of mixed and varied heavy use. The most important tuning parameter is to run the following (via SQL+/Dev) on the Oracle DB:
where system is the name of the user/schema owner of the Financial Close Management ARM database objects. Run this parameter initially after you load the bulk data, just before running the ARM copy to period task. Note: If you do not run the parameter initially, the reconciliations can take days to become open especially when you copy 5,000+ profiles to a period for automatic reconciliations. You must also run this stats process at regular intervals as part of ongoing system maintenance. Results This configuration was able to support the following workload: 50 users repeatedly accessing the BI Dashboard. 50 users repeatedly adding and deleting transactions against reconciliations. 25 users repeatedly submitting and rejecting reconciliations. 1 to 3 second pause between mouse clicks. 60 to 90 second delay between each repeated cycle per user. CPU on Host2 ran up to 90%, but was hardly used on the other servers. Note: The provision of CPU capacity on Host3 is to allow for Financial Close Management CM use, its not expected to be used for this type of ARM workload. About 3.5GB of the allocated 4GB heap was used by the Financial Close Management JVM on Host2.
34
35
The graph below shows the impact of the test on selected server metrics. No other process had any significant CPU use, but together they add up to take the overall CPU from just over one of 8 cores (12.5%) used by Financial Close Management to 23% at peak for all processes on the server. You can see from this that the copy-to-period task is single-threaded:
Key: Black = Total CPU, max 23.440 over all 8 cores Green = CPU for Financial Close Management JVM (full scale), max 100.937 Other Metrics After the above test is run, the servers key memory metrics were as follows: Free memory = 9896 MB (physical memory is 16GB). Committed bytes = 7094 MB (this is the server's overall VM use).
36
The arm01 script executes the following tasks for each allocated vuser: Open browser and access EPM Workspace home page. Log on to Workspace. Open ARM, bringing up existing BI dashboard. The dashboard contains 4 widgets, showing a range of information for the scale05 period. Log off, returning to Workspace home page. Close browser (but do not clear browser cache). Wait for between 60 and 90 seconds randomly - this is called pacing. Repeat above task sequence from start.
The arm02 script executes the following tasks for each allocated vuser: 1. Open browser and access EPM Workspace home page. 2. Log on to Workspace. 3. Open ARM, bringing up existing BI blank dashboard. 4. Add status widget. 5. Add aging widget. 6. Add reconciliation list widget. 7. Add transaction list widget. 8. Delete each widget, in reverse order to how they were added. 9. Log off, returning to Workspace home page. 10. Close browser (but do not clear browser cache). 11. Wait for between 60 and 90 seconds randomly - this is called pacing. 12. Repeat above task sequence from start. The arm03 script executes the following tasks for each allocated vuser: 1. Open browser and access EPM Workspace home page. 2. Log on to Workspace. 3. Open ARM, bringing up existing transaction list. (lists transactions for scale05 period) 4. 5. 6. 7. 8. Refresh transaction list. Log off, returning to Workspace home page. Close browser (but do not clear browser cache). Wait for between 60 and 90 seconds randomly - this is called pacing. Repeat above task sequence from start.
The arm04 script executes the following tasks for each allocated vuser:
Oracles Hyperion Financial Close Management Performance Tuning Guide 37
1. Open browser and access EPM Workspace home page 2. Log on to Workspace 3. Open ARM, bringing up existing reconciliation list (lists reconciliations for scale04 period) 4. Open a reconciliation (each vuser uses a different reconciliation). 5. Click on the Explained Balance tab. 6. Click on Add Transaction. 7. Fill in comment and value fields (date is left as default). 8. Click Save. 9. Close the reconciliation dialogue, returning to the reconciliation list. 10. Open the same reconciliation. 11. Click on the Explained Balance tab. 12. Select the transaction previously added (should be the 1st and only one). 13. Click Delete Transaction. 14. Click Yes on the dialogue to confirm deletion. 15. Close the reconciliation dialogue, returning to the reconciliation list. 16. Log off, returning to Workspace home page. 17. Close browser (but do not clear browser cache). 18. Wait for between 60 and 90 seconds randomly - this is called pacing. 19. Repeat above task sequence from start. The arm05 script executes the following tasks for each allocated vuser: 1. Open browser and access EPM Workspace home page. 2. Log on to Workspace. 3. Open ARM, bringing up existing reconciliation list. (lists reconciliations for scale05 period) 4. Open a reconciliation (each vuser uses a different reconciliation). 5. Click Add Comment. 6. Fill in comment field. 7. Click OK to save the comment. 8. Close the reconciliation dialogue, returning to the reconciliation list. 9. Open the same reconciliation. 10. Select the comment previously added (the first and only one). 11. Click Delete Comment. 12. Close the reconciliation dialogue, returning to the reconciliation list. 13. Log off, returning to Workspace home page. 14. Close the browser (but do not clear the browser cache). 15. Wait for between 60 and 90 seconds randomly - this is called pacing. 16. Repeat above task sequence from start The arm06 script executes the following tasks for each allocated vuser: 1. Open browser and access EPM Workspace home page. 2. Log on to Workspace. 3. Open ARM, bringing up existing reconciliation list. (lists reconciliations for scale06 period) 4. 5. 6. 7. 8. Open a reconciliation (each vuser uses a different reconciliation). Click Submit. Click Yes to submit, returning to the reconciliation list. Log off, returning to Workspace home page. Log on to Workspace.
38
9. Open ARM, bringing up existing reconciliation list. (lists reconciliations for scale06 period) 10. Click Reject, returning to the reconciliation list. 11. Log off, returning to Workspace home page. 12. Close the browser (but do not clear browser cache). 13. Wait for between 60 and 90 seconds randomly - this is called pacing. 14. Repeat above task sequence from start. Each vuser pauses between 1 and 3 seconds randomly to simulate the pauses real users would have - this is called think-time. The 175 vusers were introduced to the system using the following ramp-up and ramp-down schedule: 1. Start 50 vusers - 1 per 20 seconds 2. Run for 10 minutes at 50 users 3. Start 50 vusers - 1 per 20 seconds 4. Run for 10 minutes at 100 users 5. Start 25 vusers - 1 per 20 seconds 6. Run for one hour at 125 users 7. Start 25 vusers - 1 per 20 seconds 8. Run for 10 minutes at 150 users 9. Start 25 vusers - 1 per 20 seconds 10. Run for one hour at 175 users 11. Ramp-down - stop all users - 5 per 30 seconds Each vuser starts to execute its sequence of tasks (depending on the script it has been assigned) as soon as it has been started, continuing to cycle around its list of tasks as dictated by its LR script until the ramp-down phase. Results Scenario Name: Duration: Maximum Running Vusers: Total Throughput (bytes): Average Throughput (bytes/s): Total Hits: Average Hits per Second: Transactions: Total Passed: Total Failed: Total Stopped: scale175.lrs 3 hours and 45 minutes. 175 14,444,780,207 1,069,904 1,996,024 147.843 133,979 100 0
39
Transaction Name a01_001_home a01_002_logon a01_003_armdb a01_004_logoff a02_001_home a02_002_logon a02_003_arm a02_004_addstatus a02_005_addaging a02_006_addreconlist a02_007_addtxlist a02_008_deltxlist a02_009_delreconlist a02_010_delaging a02_011_delstatus a02_012_logoff a03_001_home a03_002_logon a03_003_arm a03_004_refresh a03_005_logoff a04_001_home a04_002_logon a04_003_arm a04_004_recon a04_005_expbal a04_006_addtx a04_007_save a04_008_close a04_009_recon a04_010_expbal a04_011_selecttx a04_012_delete a04_013_deleteyes a04_014_close a04_015_logoff a05_001_home a05_002_logon
Minimum 0.016 0 0.078 0.016 0.016 0 0.063 0.125 0.094 0.094 0.109 0.094 0.094 0.047 0 0.016 0.016 0 0.203 0.188 0.063 0.016 0 0.485 0.641 0.125 0.219 0.063 0.344 0.328 0.063 0.063 0.063 0.109 0.344 0.047 0.016 0
Average Maximum 0.035 0.023 0.15 0.041 0.036 0.026 0.103 0.157 0.124 0.132 0.179 0.18 0.115 0.071 0.015 0.038 0.036 0.024 0.251 0.584 0.082 0.037 0.026 0.592 0.704 0.151 0.295 0.178 0.744 0.381 0.146 0.141 0.123 0.172 0.731 0.072 0.037 0.024 0.656 0.938 22.266 0.719 0.875 0.5 0.891 1.594 1.204 0.672 0.828 5.174 1.015 0.563 0.375 1.157 0.688 0.594 2.673 1.579 1.125 0.656 0.391 1.751 1.329 4.15 1.063 1.079 5.768 5.205 0.668 0.797 2.285 1.188 1.391 0.828 0.719 0.563
Std. Dev. 90% 0.022 0.032 0.303 0.042 0.028 0.025 0.048 0.063 0.051 0.042 0.047 0.156 0.049 0.032 0.022 0.043 0.027 0.025 0.081 0.08 0.044 0.031 0.018 0.096 0.068 0.11 0.064 0.049 0.153 0.131 0.042 0.043 0.068 0.05 0.065 0.042 0.032 0.024 0.031 0.033 0.219 0.051 0.031 0.032 0.11 0.172 0.141 0.141 0.207 0.211 0.13 0.082 0.022 0.041 0.031 0.033 0.297 0.672 0.1 0.047 0.032 0.672 0.782 0.172 0.36 0.208 0.828 0.438 0.16 0.163 0.141 0.188 0.813 0.079 0.047 0.031
Pass
Fail Stop 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
40
5,900 0 5,900 0 5,900 0 5,900 0 2,280 0 2,280 0 2,278 2 2,278 0 2,278 0 2,278 0 2,278 0 2,278 0 2,278 0 2,278 0 2,278 0 2,278 0 2,937 0 2,937 0 2,935 2 2,935 0 2,935 0 1,626 0 1,626 0 1,600 26 1,600 0 1,600 0 1,600 0 1,600 0 1,600 0 1,600 0 1,600 0 1,600 0 1,600 0 1,600 0 1,600 0 1,600 0 2,336 0 2,336 0
a05_003_arm a05_004_recon a05_005_addcom a05_006_ok a05_007_close a05_008_recon a05_009_selcom a05_010_delcom a05_011_close a05_012_logoff a06_001_home a06_002_logon a06_003_arm a06_004_recon a06_005_submit a06_006_submityes a06_007_logoff a06_008_home a06_009_logon a06_010_arm a06_011_recon a06_012_reject a06_013_logoff
0.485 0.625 0.156 0.063 0.391 0.344 0.047 0.063 0.391 0.047 0.016 0 0.485 0.625 0.094 0.078 0.047 0 0 0.485 0.625 0.125 0.047
0.582 0.723 0.207 0.099 0.447 0.387 0.08 0.089 0.445 0.07 0.036 0.024 0.581 0.705 0.11 0.462 0.071 0.001 0.018 0.575 0.71 0.485 0.07
2.126 17.485 4.866 4.642 1.094 1.313 0.578 0.911 1.125 0.563 0.938 0.578 1.913 1.579 0.704 1.454 0.594 0.109 0.453 1.282 2.048 0.938 1.125
0.1 0.358 0.114 0.136 0.06 0.06 0.035 0.039 0.057 0.032 0.036 0.024 0.105 0.078 0.044 0.097 0.037 0.005 0.017 0.08 0.079 0.068 0.044
0.656 0.797 0.234 0.101 0.516 0.453 0.083 0.096 0.517 0.079 0.041 0.032 0.672 0.782 0.125 0.552 0.08 0 0.022 0.656 0.797 0.563 0.082
2,307 29 2,307 0 2,307 0 2,307 0 2,307 0 2,307 0 2,307 0 2,307 0 2,307 0 2,307 0 1,295 0 1,295 0 1,280 15 1,280 0 1,280 0 1,280 0 1,280 0 1,280 0 1,280 0 1,254 26 1,254 0 1,254 0 1,254 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
The following graph confirms how the users were introduced during the test run:
41
The following graph shows how the transaction response times changed over time for the vusers running the arm01 script. Note there's no significant increase in response times as more users were added. Initial very high response times are due to the impact of caching (or lack of for the first iterations) - especially at the browser level:
42
The following graphs show the same for the other 5 scripts:
43
44
45
46
47
The following graph shows the impact of the test on selected server metrics. It can be seen that the only process making any real use of the CPU is the FINANCIAL CLOSE MANAGEMENT JVM. It peeks at 587%, which confirms it is running multi-threaded.
This graph also shows that CPU increases broadly inline with the number of concurrent users on the system. Key: red = CPU for Foundation JVM blue = CPU for SOA JVM green = CPU for FINANCIAL CLOSE MANAGEMENT JVM (at 1/5th scale when presented on graph area) yellow = CPU for WL JVM black = total CPU over all 8 cores orange = total CPU over all 8 cores on DB server Be aware of scaling and the impact of granularity when reading the above graph; for example, many of the points are averages, and the true maximums (even for a millisecond) are as shown in the table under the graph. Also, from the metric list at the bottom of the above graph can be seen the server's memory usage: Free memory ~= 6085 MB (physical memory is 16GB). Committed bytes ~= 10887 MB (this is the server's overall VM use).
48
Other Metrics The following graph shows the distribution of failed transactions over the duration of the test run. The total number of failed transactions is within acceptable limits at 100 out of 133,979. These failures are where the LR script sees a response from the server it was not expecting:
These failed transactions are currently being investigated. They appear to be related to an internal locking issue. In all cases a real user would just need to log out and in again before continuing, with it being very unlikely they would see the same error again for a long while. These failures only occur under repetitive concurrent multi-user load.
49
Summary
Like many products, real-world FCM implementations need performance tuning and ongoing monitoring. Most situations also require some form of capacity planning before the initial implementation and when any major change in workload or configuration is planned. Financial Close Management, with the hardware configuration detailed in Benchmarking Configuration earlier, comfortably copes with 100 concurrent Close Manager users and 175 concurrent ARM users. At 35% concurrency 175 users would be a user base of 500. Looking at the CPU and memory, both CM and ARM users could not use the system at the same time (this scenario will be tested shortly). The only change may be the need to increase the 4GB heap allocated to the Financial Close Management JVM.
Finally
Tuning is key for larger data or user volumes. Its especially important for dbstats to be run on the Oracle DB after large data loads. The system is not good at letting you know that it is running slow because of poor database response times; therefore it can be hard to know if the DB instance has been allocated enough resource to meet your workloads requirements. Please be aware that any testing is limited to the exact test scenario sometimes small changes in use or configuration can have a big impact. Take these results as a guide, and always do testing of your own system and workload pattern and volumes to confirm it will be fit for purpose.
Oracles Hyperion Financial Close Management Performance Tuning Guide 50
Copyright 2012, Oracle and / or its affiliates. All rights reserved. http://www.oracle.com
51