Professional Documents
Culture Documents
11_PerformanceTestPlan.ppt Page 1 of 49
Presentation outline
Review Performance Targets
¾ Business team, development team, test team need to agree
Scenarios
¾ Representative scripted use cases
¾ User mix
¾ Single point failure scenarios
Production configuration test plan
¾ Start simple, work toward complex
¾ Provide mechanisms to bypass sub-systems
Four phases of performance testing
Staffing and scheduling
Risk analysis
11_PerformanceTestPlan.ppt Page 2 of 49
Context
Application has trace built in
¾ Particularly response time trace for critical functions
Code is in SCM
Unit test process is in place
System integration test process is in place
Application installation and configuration scripts are available
Where are we in the project life cycle?
Where are we on the project time line?
11_PerformanceTestPlan.ppt Page 3 of 49
Items in the performance testing plan
Test scenarios development
Test data generation development
Simulated alternatives to substitute for scarce or unavailable
systems
Acquisition and build of the staging environment
Staging environment is under control
¾ Isolated from the rest of the world
¾ Usage is controlled
¾ Configuration change is controlled
Record keeping process
11_PerformanceTestPlan.ppt Page 4 of 49
Review Performance Targets
11_PerformanceTestPlan.ppt Page 5 of 49
Performance Targets
Establish terminology and use it consistently
¾ Terms like “hit”, “request”, “user”, “active user”, “day”, “visit”, etc, all need to be
understood.
¾ The meanings of terms vary widely and it is important to established shared definitions
among the teams involved.
Peak concurrent users
¾ Perhaps more precisely, the “active sessions” as in HTTP sessions
Peak hits/sec
¾ pages/sec makes more sense in the context of the application
¾ Translate page requests to “hits” which is easier to measure.
¾ In our example we’ll use 800 hits/sec
¾ Rule of thumb for peak: 3 to 5 times average
11_PerformanceTestPlan.ppt Page 6 of 49
Performance targets
Throughput targets (KB/sec)
¾ Typical page size (KBytes)
¾ pages/sec
¾ Derive KB/sec
¾ Make sure the network can handle the traffic.
11_PerformanceTestPlan.ppt Page 7 of 49
Performance Targets
User arrival rates
¾ Login rates from historical data
¾ Size of HTTP session state (from developers and testing)
¾ HTTP session timeout parameter
¾ Estimate login rate from concurrent load and length of stay.
11_PerformanceTestPlan.ppt Page 8 of 49
Develop Scenarios, Simulators, Data sets
11_PerformanceTestPlan.ppt Page 9 of 49
Scenarios
Browse, search, register, order, request, buy, sell, cancel
¾ Types of queries
¾ Size of result sets
¾ Types of create, update, delete operations
¾ Systems and paths to them that need to be exercised
User mix
¾ Devise a representative profile
Resources involved
¾ Are any running under existing load, e.g., enterprise systems
¾ Paths to those resources, e.g., messaging, direct, transactional
Resource contention
¾ Tables in databases
¾ Connections to messaging systems
¾ Connections to specialized servers (LDAP, auth servers, mainframe, etc.)
11_PerformanceTestPlan.ppt Page 10 of 49
Simple scenarios, then complex
Start very simple and work toward complex
¾ Home page, login, user home page, simple browse, simple search, single
update, etc.
¾ Some test scripts are implemented just to get baselines
– “Ping” tests for various paths through the system
– Establish “best case” performance of some sub-system, e.g., login
¾ Build scripts from simple pieces that can be rearranged into realistic
sequences with minimal “programming”
11_PerformanceTestPlan.ppt Page 11 of 49
Develop sample data sets
Scripts for generating users
Scripts for generating representative dummy data sets
Scripts for cleaning up and repopulating
Dummy data sets are preferred
¾ More controlled data sample
¾ Don’t need to worry about privacy issues
¾ Easier to vary quantity and nature of the data set
11_PerformanceTestPlan.ppt Page 12 of 49
Develop simulators
Often needed for enterprise servers.
Allow testing to proceed when sub-systems aren’t available.
Provides consistently repeatable behavior
11_PerformanceTestPlan.ppt Page 13 of 49
Provide hooks to bypass components, subsystems
On/off or level set switches for things built into the application
¾ Data caches
¾ Audit trail systems
¾ Trace logging
Bypass capability for things built into the system architecture
¾ Security servers
¾ SSL accelerators
¾ Caching proxies
11_PerformanceTestPlan.ppt Page 14 of 49
Fault Scenarios
Failure mode testing under load
¾ Drop a security server
¾ Drop an HTTP server
¾ Drop and app server
¾ etc
11_PerformanceTestPlan.ppt Page 15 of 49
Establish record keeping system
Test results table
¾ Test ID, Software configuration ID, System configuration ID
¾ Version of the application code
¾ Date, time
¾ Key results and reference to results file
Software configuration table
¾ Database tuning changes
¾ App server tuning changes
¾ LDAP, security server
¾ Application version and tuning/configuration parameters
System configuration table
¾ What subsystems were used
¾ Sub-system configuration
Record who was involved in running the test and monitoring each sub-
system.
Place to archive the test records and collected data.
¾ File system with consistent directory and file naming scheme
11_PerformanceTestPlan.ppt Page 16 of 49
Production and staging configurations
11_PerformanceTestPlan.ppt Page 17 of 49
Production configuration
RPSS Database Enterprise
Internet (WebSeal) HTTP Server Server
user Servers
requests IP Sprayer WebSphere
Servers
FW1 FW2
misc. servers
FW3
misc. servers
LDAP Master
LDAP Replicas
11_PerformanceTestPlan.ppt Page 18 of 49
Identify who is responsible for each subsystem
Database server and application database tuning
Enterprise systems, specialized servers
Messaging systems
Application servers
Directory servers (LDAP or whatever)
HTTP servers
Firewalls
Proxy servers
¾ Reverse proxy security servers
¾ Caching proxies
¾ SSL accelerators
Load balancers
Network infrastructure
Hardware platforms
11_PerformanceTestPlan.ppt Page 19 of 49
Staging configuration
Load test Enterprise
generators Database Server
Server (simulator)
HTTP WebSphere
Servers Servers
RPSS
(WebSeal)
FW1 IP Sprayer
misc. servers
FW3
LDAP Replicas
LDAP Master
Load test
controller
11_PerformanceTestPlan.ppt Page 20 of 49
Four phases of performance testing
11_PerformanceTestPlan.ppt Page 21 of 49
Four phases of testing
Phase 1: baseline tests, simple configurations
¾ Bottleneck hunting
¾ Application design shakedown
¾ Throughput saturation curve
Phase 2: initial scale-up
¾ More realistic system configurations
¾ More realistic scenarios with more realistic data sets and user mixes
¾ Failure mode testing under load
Phase 3: high volume scale-up
¾ Realistic configurations
¾ Realistic test scenarios with realistic loads
Phase 4: Incremental roll-out
¾ Limited operation in production
11_PerformanceTestPlan.ppt Page 22 of 49
Phase 1 performance testing
11_PerformanceTestPlan.ppt Page 23 of 49
Objectives of Phase 1 testing (1/3)
Shake out problems with the basic configuration of the systems
used to run the infrastructure components including the network.
Shake out problems with the load generation tools and the scripts
and scenarios used to perform the tests.
11_PerformanceTestPlan.ppt Page 24 of 49
Objectives of Phase 1 testing (2/3)
Shake out problems with the application design and
implementation.
Get a baseline for component performance and simple
configuration performance.
¾ Get “best case” numbers for response time for basic operations
– Single user tests
– Simple scripts that exercise a specific aspect of the system
• Login, user home page, simple query,
• Simple create, update operations (buy, sell, etc.)
11_PerformanceTestPlan.ppt Page 25 of 49
Objectives of Phase 1 testing (3/3)
Find the throughput saturation point for single-system configuration
¾ Single unit of hardware for each sub-system
¾ Bottleneck hunting
Saturation Point
50
Transactions/Sec
40
Throughput
30
20
Heavy Load Zone
10
Light Load Zone Buckle Zone
Concurrent Users
11_PerformanceTestPlan.ppt Page 26 of 49
Start Phase 1 testing early!
The overall project plan should begin phase 1 performance testing
within weeks of the start of application development
Use “vertical slice” of the application for critical use-case paths
¾ Monitor supporting systems, for example:
– DB performance profile
– LDAP performance profile
11_PerformanceTestPlan.ppt Page 27 of 49
Phase 1 test configuration
Enterprise
Database Server
Server (simulator)
Load test
generators
WebSphere
HTTP Servers
Servers
RPSS
FW1 IP Sprayer (WebSeal)
misc. servers
FW3
LDAP Replica
LDAP Master
Load test
controller
11_PerformanceTestPlan.ppt Page 28 of 49
Be able to simplify further
Load test
generators
Database
HTTP Server
Server
WebSphere
Server Enterprise
Server
(simulator)
Misc.
simulation
servers
LDAP Master
Load test
controller
11_PerformanceTestPlan.ppt Page 29 of 49
Phase 1 exit criteria
Throughput curve for single system
¾ Transactions/sec reaches target
¾ Response times are acceptable
¾ Known bottlenecks removed
– Ready to go to the next level of load expected in phase 2
Baseline results for simple use-cases
¾ Handy to have for sanity checks in later phases
Performance effects of various components are well documented
¾ Impact of firewalls
¾ Impact of security server
¾ Impact of caching proxy
¾ Impact of SSL accelerator
¾ Etc
11_PerformanceTestPlan.ppt Page 30 of 49
Phase 2 performance testing
Initial scale-up
11_PerformanceTestPlan.ppt Page 31 of 49
Objectives of Phase 2 testing
Introduce more realistic test scenarios and user profile mix.
Use real enterprise servers for at least some test runs
¾ If necessary, off hours test runs
¾ Confirm simulated behavior
Determine scalability as more hardware is added.
Continue to uncover bottlenecks that will need to be addressed.
Determine system response under stress conditions:
¾ Load surges on top of various levels of existing load.
¾ Sustained running under maximum load.
– 12 to 24 hour endurance test
Confirm network traffic projections
Failover testing under load.
¾ Failover testing tends to be time consuming.
11_PerformanceTestPlan.ppt Page 32 of 49
Phase 2 hardware configuration
Load test
generators Database Enterprise
Server Server
HTTP WebSphere
Servers Servers
RPSS
(WebSeal)
FW1 IP Sprayer
misc. servers
FW3
LDAP Replicas
LDAP Master
Load test
controller
11_PerformanceTestPlan.ppt Page 33 of 49
Phase 2 exit criteria
No problems with the configuration
Throughput curve with 2 app servers, 2 HTTP servers, etc
¾ Does horizontal scaling result in doubling of throughput with acceptable
response times?
Any newly found bottlenecks are removed
¾ Projections are that the full site load can be handled.
Network load projections are acceptable.
Failure scenario tests pass
11_PerformanceTestPlan.ppt Page 34 of 49
Phase 3 performance testing
11_PerformanceTestPlan.ppt Page 35 of 49
Objectives of Phase 3 testing
High number of simulated users
¾ Get additional scaling data points to gain confidence
¾ Looking to see if supporting systems will handle high volume
Increase available resources for handling more load
¾ If possible, run tests on a full complement of hardware
¾ More typically, some subset of the production configuration
Ensure no bottlenecks in supporting systems
Confirm network traffic projections
Longer endurance runs
¾ 72 hours or more
More realistic scenarios with more coverage
¾ Looking for odd combinations that may break the system
11_PerformanceTestPlan.ppt Page 36 of 49
Phase 3 exit criteria
No problems with the configuration
Throughput curve with more app servers, more HTTP servers, etc
¾ Does horizontal scaling result in linear increase in throughput with
acceptable response times?
Any newly found bottlenecks are removed
¾ Projections are that the full site load can be handled.
11_PerformanceTestPlan.ppt Page 37 of 49
Phase 4 performance testing
11_PerformanceTestPlan.ppt Page 38 of 49
Objectives of Phase 4 testing
Testing the application in the production setting
¾ Use a test context root or test virtual IP.
¾ Exercise the application from outside the site
Final Q/A before full roll-out.
¾ Monitor performance carefully with live load
11_PerformanceTestPlan.ppt Page 39 of 49
Phase 4 testing in production setting
Depending on the application specifics it is likely that special
design and implementation considerations will be needed.
¾ Stateless applications have an advantage
¾ Affinity routing is important
¾ For existing application, may require that UI is identical
¾ Users get changed over once and subsequent visits return to new app
Limited user population
¾ For intranet applications, control the user population
¾ For Internet applications this is more problematic
– Throttle the traffic at the front end
– Make routing part of login
Achieving an incremental rollout to live traffic may not be feasible
11_PerformanceTestPlan.ppt Page 40 of 49
Staffing and scheduling
11_PerformanceTestPlan.ppt Page 41 of 49
Performance testing staffing estimates
Core performance test team: 2 to 5 people
1 each for the following (check all that apply)
¾ Database
¾ Enterprise server systems
¾ Any specialized enterprise server systems
– Document management, HR, supply chain management, etc
¾ Messaging infrastructure (JMS provider)
¾ Application server
¾ HTTP server
¾ LDAP Directory Server
¾ Security – authentication server
¾ Security – authorization server
¾ Network
¾ Operating system
11_PerformanceTestPlan.ppt Page 42 of 49
Factors that affect schedule
Complexity and scope of the application
Quality of the application design and implementation
¾ See best practices presentations and white papers
Performance targets
¾ Low volume applications are easier
¾ High volume applications take a lot more work
A show stopper in the design initiates a development cycle
Product issues
¾ Defects in the products
¾ Misunderstanding of product capabilities or bad practices
¾ Interoperability issues
Availability of dedicated systems for testing
¾ Without dedicated systems, the process drags out dramatically
¾ Availability of sufficient hardware to drive the test load
Availability of expertise
¾ Waiting for skilled people, who know the app, and the systems
11_PerformanceTestPlan.ppt Page 43 of 49
Performance Testing Time Line Estimates
Phase 1 – base system testing
¾ 4 to 8 weeks
Phase 2 – initial scale up
¾ 4 to 8 weeks
Phase 3 – high volume scale up
¾ 2 to 4 weeks
¾ Resource constraints may be uncovered in supporting sub-systems
¾ For example, existing systems not sized to handle new high volume
Phase 4 – incremental production rollout
¾ 1 week
11_PerformanceTestPlan.ppt Page 44 of 49
Risk Analysis
11_PerformanceTestPlan.ppt Page 45 of 49
Test coverage versus project risk reduction
No performance testing
¾ Disaster
Only complete phase 1, jump immediately into production
¾ Very high risk
Only complete phase 2 (initial scale-up)
¾ Somewhat high risk
Complete phase 3 (high volume testing)
¾ Least risky short of incremental production roll-out
¾ How thoroughly do your tests match production reality?
¾ Will the application and supporting systems stand up to real load patterns?
Able to implement phase 4 (incremental production roll-out)
¾ Requires the most work, time, and careful planning.
¾ Maximum risk reduction
11_PerformanceTestPlan.ppt Page 46 of 49
Performance testing coverage trade-offs
Phase 1
Phase 2
Increasing Risk
Phase 3
Phase 4
Increasing Effort
11_PerformanceTestPlan.ppt Page 47 of 49
Shameless Plug
ISBN: 0201844540
Available at fine eTailers,
Brick and Mortar stores,
and PubOrder (IBM)!
11_PerformanceTestPlan.ppt Page 48 of 49
References
Joines, Willenborg, Hygh: Performance Analysis for Java Web
Sites
¾ Developing a Performance Test Plan (chapter 6)
¾ Test Scripts (chapter 7)
¾ Test Environment Construction and Tuning (chapter 9)
11_PerformanceTestPlan.ppt Page 49 of 49