You are on page 1of 49

Planning for performance testing

Test today or there is no tomorrow!

11_PerformanceTestPlan.ppt Page 1 of 49
Presentation outline
ƒ Review Performance Targets
¾ Business team, development team, test team need to agree
ƒ Scenarios
¾ Representative scripted use cases
¾ User mix
¾ Single point failure scenarios
ƒ Production configuration test plan
¾ Start simple, work toward complex
¾ Provide mechanisms to bypass sub-systems
ƒ Four phases of performance testing
ƒ Staffing and scheduling
ƒ Risk analysis

11_PerformanceTestPlan.ppt Page 2 of 49
Context
ƒ Application has trace built in
¾ Particularly response time trace for critical functions
ƒ Code is in SCM
ƒ Unit test process is in place
ƒ System integration test process is in place
ƒ Application installation and configuration scripts are available
ƒ Where are we in the project life cycle?
ƒ Where are we on the project time line?

11_PerformanceTestPlan.ppt Page 3 of 49
Items in the performance testing plan
ƒ Test scenarios development
ƒ Test data generation development
ƒ Simulated alternatives to substitute for scarce or unavailable
systems
ƒ Acquisition and build of the staging environment
ƒ Staging environment is under control
¾ Isolated from the rest of the world
¾ Usage is controlled
¾ Configuration change is controlled
ƒ Record keeping process

11_PerformanceTestPlan.ppt Page 4 of 49
Review Performance Targets

Establish agreement on the goals and know what they are

11_PerformanceTestPlan.ppt Page 5 of 49
Performance Targets
ƒ Establish terminology and use it consistently
¾ Terms like “hit”, “request”, “user”, “active user”, “day”, “visit”, etc, all need to be
understood.
¾ The meanings of terms vary widely and it is important to established shared definitions
among the teams involved.
ƒ Peak concurrent users
¾ Perhaps more precisely, the “active sessions” as in HTTP sessions
ƒ Peak hits/sec
¾ pages/sec makes more sense in the context of the application
¾ Translate page requests to “hits” which is easier to measure.
¾ In our example we’ll use 800 hits/sec
¾ Rule of thumb for peak: 3 to 5 times average

ƒ Average response times


¾ Typical: 2 to 10 seconds depending on type of request.
¾ Variance?
¾ Max?

11_PerformanceTestPlan.ppt Page 6 of 49
Performance targets
ƒ Throughput targets (KB/sec)
¾ Typical page size (KBytes)
¾ pages/sec
¾ Derive KB/sec
¾ Make sure the network can handle the traffic.

11_PerformanceTestPlan.ppt Page 7 of 49
Performance Targets
ƒ User arrival rates
¾ Login rates from historical data
¾ Size of HTTP session state (from developers and testing)
¾ HTTP session timeout parameter
¾ Estimate login rate from concurrent load and length of stay.

11_PerformanceTestPlan.ppt Page 8 of 49
Develop Scenarios, Simulators, Data sets

11_PerformanceTestPlan.ppt Page 9 of 49
Scenarios
ƒ Browse, search, register, order, request, buy, sell, cancel
¾ Types of queries
¾ Size of result sets
¾ Types of create, update, delete operations
¾ Systems and paths to them that need to be exercised
ƒ User mix
¾ Devise a representative profile
ƒ Resources involved
¾ Are any running under existing load, e.g., enterprise systems
¾ Paths to those resources, e.g., messaging, direct, transactional
ƒ Resource contention
¾ Tables in databases
¾ Connections to messaging systems
¾ Connections to specialized servers (LDAP, auth servers, mainframe, etc.)

11_PerformanceTestPlan.ppt Page 10 of 49
Simple scenarios, then complex
ƒ Start very simple and work toward complex
¾ Home page, login, user home page, simple browse, simple search, single
update, etc.
¾ Some test scripts are implemented just to get baselines
– “Ping” tests for various paths through the system
– Establish “best case” performance of some sub-system, e.g., login
¾ Build scripts from simple pieces that can be rearranged into realistic
sequences with minimal “programming”

11_PerformanceTestPlan.ppt Page 11 of 49
Develop sample data sets
ƒ Scripts for generating users
ƒ Scripts for generating representative dummy data sets
ƒ Scripts for cleaning up and repopulating
ƒ Dummy data sets are preferred
¾ More controlled data sample
¾ Don’t need to worry about privacy issues
¾ Easier to vary quantity and nature of the data set

11_PerformanceTestPlan.ppt Page 12 of 49
Develop simulators
ƒ Often needed for enterprise servers.
ƒ Allow testing to proceed when sub-systems aren’t available.
ƒ Provides consistently repeatable behavior

11_PerformanceTestPlan.ppt Page 13 of 49
Provide hooks to bypass components, subsystems
ƒ On/off or level set switches for things built into the application
¾ Data caches
¾ Audit trail systems
¾ Trace logging
ƒ Bypass capability for things built into the system architecture
¾ Security servers
¾ SSL accelerators
¾ Caching proxies

11_PerformanceTestPlan.ppt Page 14 of 49
Fault Scenarios
ƒ Failure mode testing under load
¾ Drop a security server
¾ Drop an HTTP server
¾ Drop and app server
¾ etc

11_PerformanceTestPlan.ppt Page 15 of 49
Establish record keeping system
ƒ Test results table
¾ Test ID, Software configuration ID, System configuration ID
¾ Version of the application code
¾ Date, time
¾ Key results and reference to results file
ƒ Software configuration table
¾ Database tuning changes
¾ App server tuning changes
¾ LDAP, security server
¾ Application version and tuning/configuration parameters
ƒ System configuration table
¾ What subsystems were used
¾ Sub-system configuration
ƒ Record who was involved in running the test and monitoring each sub-
system.
ƒ Place to archive the test records and collected data.
¾ File system with consistent directory and file naming scheme

11_PerformanceTestPlan.ppt Page 16 of 49
Production and staging configurations

11_PerformanceTestPlan.ppt Page 17 of 49
Production configuration
RPSS Database Enterprise
Internet (WebSeal) HTTP Server Server
user Servers
requests IP Sprayer WebSphere
Servers

FW1 FW2

misc. servers
FW3

misc. servers

LDAP Master
LDAP Replicas

11_PerformanceTestPlan.ppt Page 18 of 49
Identify who is responsible for each subsystem
ƒ Database server and application database tuning
ƒ Enterprise systems, specialized servers
ƒ Messaging systems
ƒ Application servers
ƒ Directory servers (LDAP or whatever)
ƒ HTTP servers
ƒ Firewalls
ƒ Proxy servers
¾ Reverse proxy security servers
¾ Caching proxies
¾ SSL accelerators
ƒ Load balancers
ƒ Network infrastructure
ƒ Hardware platforms

11_PerformanceTestPlan.ppt Page 19 of 49
Staging configuration
Load test Enterprise
generators Database Server
Server (simulator)
HTTP WebSphere
Servers Servers

RPSS
(WebSeal)

FW1 IP Sprayer
misc. servers
FW3

FW2 misc. servers

LDAP Replicas
LDAP Master
Load test
controller

11_PerformanceTestPlan.ppt Page 20 of 49
Four phases of performance testing

11_PerformanceTestPlan.ppt Page 21 of 49
Four phases of testing
ƒ Phase 1: baseline tests, simple configurations
¾ Bottleneck hunting
¾ Application design shakedown
¾ Throughput saturation curve
ƒ Phase 2: initial scale-up
¾ More realistic system configurations
¾ More realistic scenarios with more realistic data sets and user mixes
¾ Failure mode testing under load
ƒ Phase 3: high volume scale-up
¾ Realistic configurations
¾ Realistic test scenarios with realistic loads
ƒ Phase 4: Incremental roll-out
¾ Limited operation in production

11_PerformanceTestPlan.ppt Page 22 of 49
Phase 1 performance testing

Simple configuration, bottleneck hunting, throughput saturation curve

11_PerformanceTestPlan.ppt Page 23 of 49
Objectives of Phase 1 testing (1/3)
ƒ Shake out problems with the basic configuration of the systems
used to run the infrastructure components including the network.
ƒ Shake out problems with the load generation tools and the scripts
and scenarios used to perform the tests.

11_PerformanceTestPlan.ppt Page 24 of 49
Objectives of Phase 1 testing (2/3)
ƒ Shake out problems with the application design and
implementation.
ƒ Get a baseline for component performance and simple
configuration performance.
¾ Get “best case” numbers for response time for basic operations
– Single user tests
– Simple scripts that exercise a specific aspect of the system
• Login, user home page, simple query,
• Simple create, update operations (buy, sell, etc.)

11_PerformanceTestPlan.ppt Page 25 of 49
Objectives of Phase 1 testing (3/3)
ƒ Find the throughput saturation point for single-system configuration
¾ Single unit of hardware for each sub-system
¾ Bottleneck hunting

Saturation Point
50
Transactions/Sec

40
Throughput

30

20
Heavy Load Zone
10
Light Load Zone Buckle Zone

Concurrent Users

11_PerformanceTestPlan.ppt Page 26 of 49
Start Phase 1 testing early!
ƒ The overall project plan should begin phase 1 performance testing
within weeks of the start of application development
ƒ Use “vertical slice” of the application for critical use-case paths
¾ Monitor supporting systems, for example:
– DB performance profile
– LDAP performance profile

11_PerformanceTestPlan.ppt Page 27 of 49
Phase 1 test configuration
Enterprise
Database Server
Server (simulator)

Load test
generators
WebSphere
HTTP Servers
Servers
RPSS
FW1 IP Sprayer (WebSeal)
misc. servers
FW3

FW2 misc. servers

LDAP Replica

LDAP Master
Load test
controller

11_PerformanceTestPlan.ppt Page 28 of 49
Be able to simplify further

Load test
generators
Database
HTTP Server
Server
WebSphere
Server Enterprise
Server
(simulator)

Misc.
simulation
servers

LDAP Master
Load test
controller

11_PerformanceTestPlan.ppt Page 29 of 49
Phase 1 exit criteria
ƒ Throughput curve for single system
¾ Transactions/sec reaches target
¾ Response times are acceptable
¾ Known bottlenecks removed
– Ready to go to the next level of load expected in phase 2
ƒ Baseline results for simple use-cases
¾ Handy to have for sanity checks in later phases
ƒ Performance effects of various components are well documented
¾ Impact of firewalls
¾ Impact of security server
¾ Impact of caching proxy
¾ Impact of SSL accelerator
¾ Etc

11_PerformanceTestPlan.ppt Page 30 of 49
Phase 2 performance testing

Initial scale-up

11_PerformanceTestPlan.ppt Page 31 of 49
Objectives of Phase 2 testing
ƒ Introduce more realistic test scenarios and user profile mix.
ƒ Use real enterprise servers for at least some test runs
¾ If necessary, off hours test runs
¾ Confirm simulated behavior
ƒ Determine scalability as more hardware is added.
ƒ Continue to uncover bottlenecks that will need to be addressed.
ƒ Determine system response under stress conditions:
¾ Load surges on top of various levels of existing load.
¾ Sustained running under maximum load.
– 12 to 24 hour endurance test
ƒ Confirm network traffic projections
ƒ Failover testing under load.
¾ Failover testing tends to be time consuming.

11_PerformanceTestPlan.ppt Page 32 of 49
Phase 2 hardware configuration
Load test
generators Database Enterprise
Server Server
HTTP WebSphere
Servers Servers

RPSS
(WebSeal)

FW1 IP Sprayer
misc. servers
FW3

FW2 misc. servers

LDAP Replicas
LDAP Master
Load test
controller

11_PerformanceTestPlan.ppt Page 33 of 49
Phase 2 exit criteria
ƒ No problems with the configuration
ƒ Throughput curve with 2 app servers, 2 HTTP servers, etc
¾ Does horizontal scaling result in doubling of throughput with acceptable
response times?
ƒ Any newly found bottlenecks are removed
¾ Projections are that the full site load can be handled.
ƒ Network load projections are acceptable.
ƒ Failure scenario tests pass

11_PerformanceTestPlan.ppt Page 34 of 49
Phase 3 performance testing

Full scale-up testing

11_PerformanceTestPlan.ppt Page 35 of 49
Objectives of Phase 3 testing
ƒ High number of simulated users
¾ Get additional scaling data points to gain confidence
¾ Looking to see if supporting systems will handle high volume
ƒ Increase available resources for handling more load
¾ If possible, run tests on a full complement of hardware
¾ More typically, some subset of the production configuration
ƒ Ensure no bottlenecks in supporting systems
ƒ Confirm network traffic projections
ƒ Longer endurance runs
¾ 72 hours or more
ƒ More realistic scenarios with more coverage
¾ Looking for odd combinations that may break the system

11_PerformanceTestPlan.ppt Page 36 of 49
Phase 3 exit criteria
ƒ No problems with the configuration
ƒ Throughput curve with more app servers, more HTTP servers, etc
¾ Does horizontal scaling result in linear increase in throughput with
acceptable response times?
ƒ Any newly found bottlenecks are removed
¾ Projections are that the full site load can be handled.

11_PerformanceTestPlan.ppt Page 37 of 49
Phase 4 performance testing

Incremental production roll-out: there’s nothing like the real thing

11_PerformanceTestPlan.ppt Page 38 of 49
Objectives of Phase 4 testing
ƒ Testing the application in the production setting
¾ Use a test context root or test virtual IP.
¾ Exercise the application from outside the site
ƒ Final Q/A before full roll-out.
¾ Monitor performance carefully with live load

11_PerformanceTestPlan.ppt Page 39 of 49
Phase 4 testing in production setting
ƒ Depending on the application specifics it is likely that special
design and implementation considerations will be needed.
¾ Stateless applications have an advantage
¾ Affinity routing is important
¾ For existing application, may require that UI is identical
¾ Users get changed over once and subsequent visits return to new app
ƒ Limited user population
¾ For intranet applications, control the user population
¾ For Internet applications this is more problematic
– Throttle the traffic at the front end
– Make routing part of login
ƒ Achieving an incremental rollout to live traffic may not be feasible

11_PerformanceTestPlan.ppt Page 40 of 49
Staffing and scheduling

People and time estimates

11_PerformanceTestPlan.ppt Page 41 of 49
Performance testing staffing estimates
ƒ Core performance test team: 2 to 5 people
ƒ 1 each for the following (check all that apply)
¾ Database
¾ Enterprise server systems
¾ Any specialized enterprise server systems
– Document management, HR, supply chain management, etc
¾ Messaging infrastructure (JMS provider)
¾ Application server
¾ HTTP server
¾ LDAP Directory Server
¾ Security – authentication server
¾ Security – authorization server
¾ Network
¾ Operating system

11_PerformanceTestPlan.ppt Page 42 of 49
Factors that affect schedule
ƒ Complexity and scope of the application
ƒ Quality of the application design and implementation
¾ See best practices presentations and white papers
ƒ Performance targets
¾ Low volume applications are easier
¾ High volume applications take a lot more work
ƒ A show stopper in the design initiates a development cycle
ƒ Product issues
¾ Defects in the products
¾ Misunderstanding of product capabilities or bad practices
¾ Interoperability issues
ƒ Availability of dedicated systems for testing
¾ Without dedicated systems, the process drags out dramatically
¾ Availability of sufficient hardware to drive the test load
ƒ Availability of expertise
¾ Waiting for skilled people, who know the app, and the systems

11_PerformanceTestPlan.ppt Page 43 of 49
Performance Testing Time Line Estimates
ƒ Phase 1 – base system testing
¾ 4 to 8 weeks
ƒ Phase 2 – initial scale up
¾ 4 to 8 weeks
ƒ Phase 3 – high volume scale up
¾ 2 to 4 weeks
¾ Resource constraints may be uncovered in supporting sub-systems
¾ For example, existing systems not sized to handle new high volume
ƒ Phase 4 – incremental production rollout
¾ 1 week

11_PerformanceTestPlan.ppt Page 44 of 49
Risk Analysis

What happens if we don’t do performance testing?

11_PerformanceTestPlan.ppt Page 45 of 49
Test coverage versus project risk reduction
ƒ No performance testing
¾ Disaster
ƒ Only complete phase 1, jump immediately into production
¾ Very high risk
ƒ Only complete phase 2 (initial scale-up)
¾ Somewhat high risk
ƒ Complete phase 3 (high volume testing)
¾ Least risky short of incremental production roll-out
¾ How thoroughly do your tests match production reality?
¾ Will the application and supporting systems stand up to real load patterns?
ƒ Able to implement phase 4 (incremental production roll-out)
¾ Requires the most work, time, and careful planning.
¾ Maximum risk reduction

11_PerformanceTestPlan.ppt Page 46 of 49
Performance testing coverage trade-offs

Phase 1

Phase 2
Increasing Risk

Phase 3

Phase 4

Increasing Effort

11_PerformanceTestPlan.ppt Page 47 of 49
Shameless Plug

ISBN: 0201844540
Available at fine eTailers,
Brick and Mortar stores,
and PubOrder (IBM)!

11_PerformanceTestPlan.ppt Page 48 of 49
References
ƒ Joines, Willenborg, Hygh: Performance Analysis for Java Web
Sites
¾ Developing a Performance Test Plan (chapter 6)
¾ Test Scripts (chapter 7)
¾ Test Environment Construction and Tuning (chapter 9)

11_PerformanceTestPlan.ppt Page 49 of 49

You might also like