IBM Software Group

Essentials of IBM Rational Performance Tester V8.0

Module 9: Ramping-Up Tests

Copyright IBM Corp. 2005, 2009

Objectives: Ramping-Up Tests
To learn how to use schedules to execute
target workloads
To monitor test execution and evaluate test
results
To describe typical ramp-up problems and
solutions
To understand the impact of JVM heap size

2
Where are we in the performance testing framework?

Test Planning Test Development

and Design and Validation
Record
RecordTransactions
Transactions
Workload
WorkloadAnalysis
Analysis Modify
ModifyTests
Tests
Performance
PerformanceTest
TestPlan
Plan Design
DesignWorkloads
Workloads
Validate
Validateand andDebug
DebugTests
Tests

Test
TestLab
LabHardware
Hardware Reset
ResetSystem
SystemandandData
DataState
State
System
SystemSoftware
Software Run Monitoring Tools
Run Monitoring Tools
Data
DataSet
Set Run
RunTests
Tests
Test
TestandandMonitoring
MonitoringTools
Tools Evaluate
EvaluateResults
Results
Set up Test Execute Tests and
Environment Gather Measurements

3
Topics: Ramping-Up Tests
Execute schedules
Output > test results
Monitor test execution
Ensure execution is realistic, not impacted by
environment, and achieving targets
Evaluate test results
Ensure validity test execution and statistical data

4
Moving from test and validation phase to execution

Reset system under test

Monitor target components Application
Enable verbose logging Server
Cluster

Rational Web Server

Performance Tester
and Rational Agent
Controller MQ Series
Server
EJB Server WebSphere
Server

Load Balancing
Firewalls
Secure Gateways
Database
Server
Virtual Testers
(Rational Agent Controller
and agent)
5
Using schedules to execute target workloads
After you have validated that your Performance
Schedule mimics your target workload, you can
use it in your performance test runs
Valid schedules can be run
With various user load levels (100 users, 250
users, and so on)
With various data sets
On various configurations
With different user pacing (fast users, slow users,
and so on)

6
Agents
Install Rational Agent
Controller on agent
machines
Run Rational Agent
Controller prior to test start
RAService.exe (on
Windows)
It runs as a service
Create locations to
represent remote
machines on the
workbench
Add locations to schedules
7
Installing and configuring
Rational Agent Controller
The agent software for installation on remote
machines to be used for distributed load
generation
Needs to be installed separately.
No license required for agents.
IBM Rational License Client
License manager required for load generation
beyond five users
Runs with five or less virtual testers without a
license and does not require the license manager
IBM Rational License Server
Must have enough licenses to run target user load
8
Install launchpad
On the Rational
Performance Tester
workbench
machine, the agent
(Rational Agent
Controller) is
installed by default
On agents or
remote machines,
the agent (Rational
Agent Controller) is
installed stand-
alone
9
Preparing for large user runs
Install and configure all necessary
components of the test environment
Rational Performance Tester
Agents
System Under Test
Run and configure monitoring tools (Microsoft
Windows Performance Monitor, IBM Tivoli
tools, and so on)
Modify schedule settings for target workload
User load, user pacing
Locations (remote agents)
Logging
10
Pinpointing errors at runtime
Use Rational Performance Tester to monitor
test activity with real-time reports
Validate load
Confirm page hits or responses
Look for a string of VP or page failures

11
Pinpointing errors at runtime using other tools
Monitor all test hardware
Test driver
Test agents
System Under Test components
Web server
Application server
Database server
Network components
Remember to set the resource monitoring
option in the schedule

12
Agentless system monitoring
During a performance test run, Rational Performance Tester can capture
resource monitoring information from
IBM Tivoli Monitoring
UNIX (and Linux) rstatd monitor
Windows Performance Monitor
Does not require any installation on any tier of the application
Subset of default counters selected to monitor key resources.

13
Typical ramp-up glitches
Test or schedules not appropriate for load
Steady state targets not achieved
System activity not generated for the test duration
Test settings incorrect
Not enough virtual users (agents)
Logging too much data
Test data set insufficient
Not enough licenses to perform test
Rational Performance Tester
Back-end servers (databases)

14
Typical ramp-up glitches (cont.)
Insufficient pipe, concurrent connections,
handles, or bandwidth
Overloaded network components
NICs, hubs, routers, gateways
Overloading or corrupting system
Response failures or timeouts
Maxing cache limits or queue lengths
Invalid data causing instability or errors

15
Ramp-up techniques and solutions
Schedule settings
Stagger test ramp
Start users in small groups
Delay between groups
Distribute virtual users across agents that can
operate efficiently under the target load
Monitor agents
Compensate for the ramp delay and ensure
activity is being generated for the target duration
Add loops or increase iterations
Monitor page hits during test run

16
Ramp-up techniques and solutions (cont.)
Schedule settings (cont.)
Reduce logging levels
Use sampling
Increase logging dynamically if there are inexplicable test
errors
Set a run duration or run stop time

17
Ramp-up techniques and solutions (cont.)
Impact of Java Virtual Machine (JVM)
heap size
Test hardware configuration
Test logging sampling suggestions
Using paced loops to distribute load and set
transaction rates

18
Impact of JVM heap size
High volume load testing requires significant
amounts of driver memory.
Java implementation means available memory
is limited to the JVM maximum heap size
regardless of the amount of memory on the
system.

19
Impact of JVM heap size (cont.)
Two JVMs in Rational Performance Tester
Rational Performance Tester Eclipse Workbench
(javaw.exe)
Memory usage relates mostly to size of projects assets
TPTP EMF Statistics model for reports
Mostly proportional to the number of samples (length of run) and
number of total unique page elements
TPTP EMF Execution History model
Mostly proportional to the number of sampled users and content level
TPTP EMF Test Behavioral model for Rational Performance Tester
Performance Tests
Mostly proportional to the amount of communication data
Driver [Rational Performance Tester execution engine]
(java.exe)
Mostly proportional to number of users and amount of I/O to buffer or
store (for example, large GIFs, large posts)
20
Impact of JVM heap size (cont.)
Workbench JVM (javaw.exe) is overhead,
independent of the number of users in a
running test
Size starts at 50MB 100MB and can grow
significantly when viewing and editing large assets
or while monitoring long runs
However, javaw.exes process memory
consumption then will typically stabilize at
maximum heap size (plus overhead)
Editing very large assets can exceed maximum
heap size and lead to Eclipse exiting (sometimes
by a JVM crash)

21
Impact of JVM heap size (cont.)
How much memory is really being used per user ?
Determined by size of the driver execution engine (java.exe)
This is affected by JVM heap-size (size will grow until JVM
garbage collection kicks in)
Do not assume that memory usage is proportional (you cannot
take the size at 100 users and extrapolate to 1000 users)
Auto-setting of drivers JVM maximum heap-size (how it
works)
Default is 256MB the first time a test is launched on a driver
First run on remote driver checks available system memory size
Subsequent runs on that same driver set JVM heap-size to
[available memory: 256MB] (capped at 1900 MB), unless set by
user on that drivers location (see next slide)
Auto-detection of driver out-of-memory condition
22
Setting drivers JVM maximum heap size
You can increase heap by adding the RPT_VMARGS
property as an attribute of the location where the test
is run
Open the location
Click Attributes
Click Add to create a Property Name of
RPT_VMARGS
Choose a Property Value of -Xmx1024m (for example)
The -Xmx is the JVM max heap argument
The 1024m specifies 1024 megabytes (1GB) of max heap
This only works properly if the system has at least that much, or
preferably more, real RAM
Do not use more than the maximum supported value of
the JVM
Windows: ~ 2000MB (recommend 1900 for good measure)
UNIX (Linux): 1900MB
23
Setting drivers JVM maximum heap size (cont.)
2. Edit
Attributes 3. Add new
property 4. Fill in New Property
1. Open
Name and Value
Location

24
Setting drivers JVM maximum heap size (cont.)
What about setting JVM Maximum heap size
for local users?
By default, users run on the local machine and the
location is hidden (a temporary location is created
dynamically at runtime)
You can override this by creating an explicit
location for running local users
Then set the JVM heap size for that location
Important: Consider both the workstation and local
driver JVM heap sizes
Total must leave ample remaining physical memory for
O/S and any other active host programs

25
Setting drivers JVM maximum heap size (cont.)
Creating a new location

1. Click File >

New > Other

2. Click Test > Test

Assets > Location

3. Click Next and follow

Wizard steps

26
Workbench maximum JVM heap size & heap status
Workbench JVM garbage collection is dynamically managed
Heap status monitor is helpful to force a garbage collection
JVM option can be set
in the Eclipse.ini
-vm
C:\Program
Files\IBM\RPT70\jdk\j
re\bin\javaw.exe
-vmargs
-Xquickstart
-Xms40m
-Xgcpolicy:gencon
-Xscmx96m
-Xshareclasses:
singleJVM,keep

Click to force a
Heap Monitor garbage collection

27
Test hardware configuration suggestions
Dedicate Workbench
Do not run any users locally
Fewer faster or bigger driver systems are better
than many smaller ones
This reduces stats model aggregation overhead on
Workbench
Processing and stats model size grows proportional to
(1 + number of Drivers)
Use dual or quad processor systems with at least
2GB RAM

28
Test hardware configuration suggestions (cont.)
Use 1GB NIC cards
Consider two NIC cards per driver system to segregate
traffic
One for traffic to and from servers under test (network
under test)
One for traffic back to the Workbench
Keeps stats counter traffic from adding monitoring traffic to the
network under test

29
Test hardware configuration suggestions (cont.)
To fully utilize systems with > 2GB memory
Run multiple engines by defining separate
locations that map to the same system
You must use network aliases to supply unique
hostnames to accomplish this
This will allow utilization of 1900 MB memory per engine
Dedicate systems to performing the load test
and allocate at least 80% or more of the
systems available memory to the JVM heap

30
Restrict logging for statistics
Use sampling options on the schedule to
reduce CPU and memory overhead
Statistics
If you do not need response times or server health for
page elements (that is, page statistics are sufficient), then
set the Statistics log level to Pages
Consider using less frequent sampling by increasing the
Statistics sampling interval beyond the default of five
seconds:
10 - 60 second range is reasonable, especially for longer running
tests

31
Restrict logging: Test log
Test Log
Be very careful with the Test Log settings
Volume of test log can easily become excessive without very
low sampling rates
Default for statistics log level is Page
Default sampling is for five fixed number of users per user
group
Consider reducing if you have many user groups, or more importantly if
you set statistics log level to All (for example, to do schedule and test
debugging)
Note that Test Log events are cached on the Driver
and not transferred to the Workbench until post-run
This avoids excessive network traffic and Workbench model
loading overhead during the run while measurements are
being taken
32
Using paced loops to distribute load
When running multi-user runs, perhaps for as few as
50 users, avoid lock-step (where all users hit the
server with the same activity at the same time)
Sustain a steady state transaction rate for some
period of time
Both of these can be accomplished by using loops and
controlling the rate of iterations
Check Randomly to vary the delay between iterations
to distribute load with typical user variation
Uses Poisson arrival rate = negative exponential waiting time
distribution
Check the Delay before the first iteration of the loop to
avoid initial lock-step
Uses random delay uniformly distributed over loops duty cycle
This can be used even with one iteration to distribute (or stagger)
user startup over a particular interval of time
33
Using paced loops to distribute load (cont.)
1. Add Loop
3. Set Iteration Rate
2. Select Control the (transaction rate),
rate of Iterations and select both delay
boxes

34
Review: Ramping-Up Tests
1. When preparing for large user runs, what are
three things to accomplish ahead of time?
2. What are some typical ramp-up problems?
3. What is the impact of JVM heap size?

35
Exercise 9: Ramping-Up Tests
Configure a performance schedule to
accommodate for running a large user load
Set user load and modify user pacing
Change log levels
Confirm that test data will accommodate test
execution (datapools)

36