Professional Documents
Culture Documents
Mukesh Jain
Principal Quality Manager
Global Online Services
Microsoft Corporation
1
We will talk about…
z What is Quality?
z Types of Performance Measurement
z How to Measure Web App Performance?
z Traps we fall into and how Six Sigma
showed us the right path
z Overview of Six Sigma
z Using Six sigma to Improve Performance
z 5 Steps to Improve Web App Performance
z Ensuring problems do not come back
z Questions
2
What’s the biggest problem today
Schedule
3
Defects are inevitable
z Reliable
z Responsive / Performance
z Security / Privacy
z Think Global
5
Why is Performance important?
6
Solving Performance Problems
Your app is having performance issues
z Traditional way
z Put more people
z Do more performance testing
z But Performance Testing tools
z Outcome/Results
z We often march on a journey to solve a problem without
understanding the problem
z If we don’t know what we are improving, how would we
know if we have improved?
z Structured way of solving the problem
z Use Six Sigma, we can define the problem, measure the
right thing, analyze root cause and then solve the right
problem – the right way…
7
What is Six Sigma?
z Structured Problem solving methodology
z Solving the right problem – the right way
z Focus on finding and fixing the root cause
z Ensuring problem does not come back
z Drive Continuous Improvement
z DMAIC
8
Identify major user
Monitor to scenarios & Perf
ensure perf do 1.DEFINE Goals
not degrade with
future changes to
the code
Use right set of tools
to measure & baseline
5.CONTROL current Performance
2.MEASURE Measure the following
(SUSTAIN) WebPage Page Load Time (1 & 2)
Time to First Byte
Performance # of round trips
Avg Bytes Download
Improvement
DMAIC cycle
¾ Project selection ¾ Key output variables ¾ Key causes (vital few) ¾ Improvement strategy ¾ Lock in the results
¾ Project charter (metrics or Y’s) of defects (X’s) ¾ Prioritize solutions (control plan)
¾ Critical to Customer ¾ Possible causes of ¾ Tested & measured •Mistake proofing
(CTQ) needs defects (X’s) solutions •Control points
¾ High level process map ¾ Data collection and ¾ Final solutions •Monitoring Plan
presentation plan •Positive hand off
¾ Current Performance of control plan
¾ Internal/ external ¾ Financial impact of the
benchmarking project
10
The Six Sigma Project - challenges
z Bottom-up approach
z Difficult to get buy-in
z Myth: Six Sigma cannot be used for software
z We don’t need a Six Sigma project to
understand this problem and to fix it
z Its common sense – we can fix it
z Do it if you have time – on your personal time
z Measuring the right things
z Avoiding temptation of jumping to solution
11
Six Sigma – Define Phase
z Project Selection
z The right project (Strategic, ROI, etc.)
z Project Charter
z Why are we doing this? Business Case
z Goal, Scope, Time-line
z Exec Sponsors, Team & Resources
z Web App XYZ (which drives 40% of revenue to the company) have
performance problems (based on survey). This is impacting significant
% of users and we are seeing decline in users to the site.
z Customer Focus
z Voice of Customer (VOC) – what do customer need
z Critical to quality (CTQ)
z Customers have repeatedly complained about the performance of the
XYZ app. They are not able to complete transactions in timely manner.
z Current Situation
z Process Map
z Current Performance
12
The Define Phase: Problem Statement
13
The Define Phase: Goal
15
The Define Phase: Voice of Customer
z Performance at-least as good as last release
z UI Responsiveness
16
The Define Phase: Critical To Quality (CTQ)
Examples:
17
International Internet Routes
Notes
•Minimum bandwidth to be seen on this map is 14 Gigs
•Does not report bandwidth within a country
Latency and the impact to page load
times based on number of round trips
RT = Round Trips
Red line is 34 RT
Green line is 21 RT
Dotted Green is 10
RT
Blue line is 3 RT
Source data for timings is 75th percentile for country in question from: http://msncore/performance/netsmart/Netstats.asp
Microsoft Confidential 19
Network Round trip delays…
Microsoft Confidential 20
Performance/Load/Stress Testing
z Performance testing
z User-Scenarios testing (typical case & best case)
z Establish Baseline & perform trend analysis
z Detect performance issues
z Tools: WebRunner, WANSim, HttpWatch, etc.
22
Types of Performance Measurements
z Client UI Response Time
z Server Response Time
z Load/Stress
z Bytes over wire/Throughput
z Availability
z Latency (anywhere in the world)
z Browser Processing & Rendering time
z User Machine - Resource utilization
z Perceived Performance
Microsoft Confidential 24
Measure
z Business Measurements
z # of unique users
z # of page views
z Click-Thru-Rate (CTR)
z Revenue
z Market Share
z Net Promoter
z Other Measurements
z % of people on PLT1 / PLT2
z Errors
z Abandon rate (Incomplete/Closed/Click-Away)
25
Measure: Key variables for Performance
z Factors attributing to Web Page Performance
z # of Files, Static/Dynamic content
z Page Load Time
z Bytes download
z DNS Lookup time
z Peak hours / Load & Stress
z User spread / Global?
z Data Centers / CDN / Redirects
z Multiple versions of the app
z Web Page Architecture (parallel/sequential download)
z Compression
z Expiry dates
z Keep-Alive
z PLT1 / PLT2 (Caching?)
26
Performance – Cause-Effect Diagram
Application / Web Page End-User
# of Files
Machine config
JS/CSS processing
Page Architecture
# of Servers
App versioning
Server/Infrastructure
28
PLT1/PLT2 User Distribution
Analysis
•From the data it looks like 17% of the users are in PLT1 and 83% in PLT2
•There are several users in a middle category (16K to 21K)
29
Impact of slow performance on user
31
Bytes & Page Load Time distribution
32
Hourly Users & PLT distribution
33
Six Sigma - Improve
z Improvement strategy & Plan
z Improvement solution selection
z Generate ideas – Involve diverse team
z Identify and rank solution alternatives
z Pilot solutions and select final solution
z FMEA
z Design of experiments
z Test and implement final solution
z Communication plan
z ToBe Process Map
z Track improvement, monitor trend
z Share success/failure stories Æ Best Practices 34
Six Sigma – Control (Sustain improvements)
Lessons Learned:
z Six Sigma Process Measurements helped us uncover
problems in a structured way and come up with
solutions to eliminate them or minimize its impact.
z Implementing improvements requires involvement
from everyone at all levels and all disciplines
Suggestions:
z By getting all disciplines (dev/pm/test) involved we
can focus on preventing the problems rather than
relying on finding and fixing them.
z Integrate new practices into the development cycle
once the improvements have been validated
37
Results
Note: Specific % & # are purposely removed from this presentation
z Product benefits:
z Page size drastically reduced
z Site performance improved
z Increased click-thru-rate
z Overall impact:
z Increased focus on Performance
z More investment in performance
(expanding the performance group)
38
Visual Round Trip Analyzer
39
Visual Round Trip Analyzer
40
Visual Round-Trip Analyzer (VRTA)
Client Port – Browser to
Server Connection
Servers
Bandwidth
Utilization
Color coded
By File Type
Time in Seconds
http://msncore/performance/netsmart/VRTA_animation/sample1/main.html
Use more parallel TCP ports
Bad Good
42
Unblock Java Script
43
Unblock JS
Solution I
This method has been
successfully used by the
Windows Live Hotmail Team
==================
function AsyncLoad()
{
var l = arguments.length;
for (var i=0;i<l;i++)
{
document.write("<script src='" + arguments[i] + "'></" +
"script>");
}
}
AsyncLoad(
"file1.js",
"file2.js",
"file3.js");
=====================
45
Use Compression
46
Keep-Alive TCP ports
47
Questions?
Mukesh Jain
Mukesh.Jain@microsoft.com
48