Generating Business

Value through TCoE

AppLabs Confidential
1
Testing Organization Models
Evolutionary approach to building a TCOE is pragmatic. Slowly growing a Test COE yields
opportunities to introduce mechanisms to overcome the pains of centralization.
AppLabs Confidential
2
TCoE services summary
Characteristics:
Common approach
Ability to support frequent releases
Common methodology and deliverables
Manage key set of few metrics
Centralisation of tasks / skills / functions
Benefits:
Consistent quality
High performance
Faster time to market
Flexibility to support business needs
Lower costs
AppLabs Confidential
3
Introduction TCOE
TCOE
Tools & Assets
Process &
Test
Methodology
People &
Organisation
Leverage a dedicated group of skilled test
resources
Utilise offshore resources
Ensures appropriate skills are available
People & Organisation
Standardised approach and deliverables
with greater delivery consistency
Rapid adoption of best practice
Clear process accountability / responsibility
Disciplined test management
Process & Test Methodology
Effective leveraging of tools and automation
Assets made available to projects
Maintenance of multiple environments
Tools & Assets
Lower cost
Consistent quality
Flexibility to support
business needs
Control
Benefits
Higher quality
High performance
Speed to market
Benefits
Lower cost
Consistent quality
Speed
Benefits
Characteristics:
Common Approach
Ability to support frequent releases
Common methodology and deliverables
Management to key set of Few metrics
Centralisation of key testing tasks / skills /
functions
AppLabs Confidential
4
AppLabs SCORE methodology
AppLabs proven proprietary methodology
Maps processes and metrics to organization business goals
Focuses on building organization wide test platform
Core + Flexi resource model
Cost goes down exponentially with each addition of product into TCoE
AppLabs Confidential
5
Objectives of Assessment
The objective of the assessment is to Understand, Analyze and
Recommend on Clients global test organization (including vendors):
Category Test Areas
Process
Test processes, methodology and their integration into life cycle model
Test spend, budgets and allocations
Test strategy, estimation and planning, coverage and testware management
Test metrics, quality of evaluation (across life cycle and production)
Test Teams
Experience and expertise
Teams organization, roles & responsibilities and utilization
Training and knowledge management
Communication and reporting
Infrastructure
Test hardware and software asset inventory
Test environments for various test events
Infrastructure utilization
Usage of tools and techniques in optimizing infrastructure
Tools
List of Test tools used (COTS and OpenSource)
Frameworks (Custom and Industry standard) and its reuse across client organization
Tool utilization (license centralization)
ROI analysis of tools implemented
AppLabs Confidential
6
TCoE Framework
AppLabs Framework for Assessment
Clients Vision & Goals
AppLabs SCORE
Methodology
Test Process Standardization Index
Process Artifacts Creation / Modification
Plan
Test Metrics & Measurement Roadmap
Test Process Deployment Plan
TCoE Organization Model
Resource Skill Matrix
Capacity Planning and Utilization Report
Proposed Knowledge Management
Framework
Infrastructure Inventory Report
Test Environments and Utilization Report
Test Infrastructure Optimization Plan
Test Tool Inventory and Utilization Report
Recommended Test Frameworks
ROI Analysis of Tools Implemented
Proposed Tool Optimization Plan
Understand Information gathered
360 degree Analysis
Actionable Recommendations
Establish baseline for Measuring
Improvements
Process Review
Infrastructure Review
Tools Review
Test Teams Review
Understand Analyze Recommend
Test Maturity Rating
Test Spend Analysis Report
AppLabs Confidential
7
Centralization of Testing Services
Business / Product / Technology based frameworks
Resource management Types, skills and knowledge management
Process methodologies
Reviews
Review expertise
Review techniques
Review process
Review database
Test management
Test case
Defect
Test data
Roles & Responsibilities
Risk management
Plan
Budget
Reviews
Prioritization
Measurement & Reporting
Metrics framework
MIS
Escalation
SLA
Testing infrastructure Inventory, scheduling and operational processes
Test engineering tools
Core testing
Functional and
regression
Test strategy &
approach
Methodology
Templates
authoring & review
Test suite selection
Certification
(Compatibility)
Compatibility
matrix
Acceptance
criteria
Performance
Test types
Load, stress,
etc
Tools
Capacity
planning
Security
Test
approach
Analysis
Tools
Vulnerability
assessment
Scorecard
Consulting
Test process
review
Baseline
Audit
Process
Improvement
Tools and
Automation
Test strategy
Approach &
methodology
Framework
Tool feasibility
Coverage
AppLabs Confidential
8
Benefits of a TCoE
Case studies
AppLabs Confidential
10
Business value of testing
AppLabs has successfully demonstrated Return on Investment (ROI) for
several customers in categories mentioned below. Metrics mentioned under
each category are just a sample to showcase cost advantage, quality
improvement and time-to-market
Test infrastructure
Reduction in test tool licensing
costs
Hardware infrastructure utilization
Reduction in hardware servers
Test data reusability
Test processes
Test processes standardization
Process compliance
Test maturity level
Test Engineering
Defects in production
Defect containment
Reduction in test cycle duration
Test case authoring productivity
Test case execution productivity
Test coverage
Effort to find a defect
Test Automation
Test script authoring productivity
Automation ROI
AppLabs Confidential
11
Quality Analysis Tools & Techniques
Activity Technique / Tool
Defect Analysis
Fish - bone / Ishikawa Diagram
Orthogonal defect classification
Pareto Charts
Exploratory Factor Analysis with Varimax rotation
Identifying Defect Prone Areas
State Transition Diagrams
Traceability matrix
Pareto charts
Failure Modes and Effects Analysis
Identifying Vulnerable test data
Decision tables
Stratified random sampling ( risk based or proportional)
Data mining and Data modeling
Test Case Design
Requirements ambiguity testing
Business flow matrix
Equivalence Partitioning and Boundary value analysis
Decision tables and State transition diagrams
Test execution
Prioritized testing
Rhino Hunting
Exploratory Testing
Leading Insurance
Provider in US
AppLabs Confidential
13
ROI / Efficiency Gains
Test Authoring - Cost Savings
Key Initiatives:
Used All-Pairs techniques
Participated in business requirements/review
meetings
Reverse traceability to avoid redundant test
cases
2-Level formal review before Sign-off from
Business team (reduction in rework)
Test Execution - Cost Saves
Key Initiatives::
Mapping defects to test objectives
Participation of test teams in early SDLC
cycles to gain application knowledge
Automating 40% of regression suite
Early Defect detection AFT/SIT/UAT
Releases
Releases
AppLabs Confidential
14
Post Production Root Cause Distribution
Analyzed the root causes for all the warranties and built awareness
across all the groups to drive reduction in warranties
Root Cause for each production defect has been applied as
lessons learnt in the respective teams
AppLabs Confidential
15
Defect detection percentage has decreased for Functional, Regression and Warranty phase
Early defect detection ensured adherence to release schedules
Early Defect Detection
Integration System Regression Warranty
2007 2008 2009
AppLabs Confidential
16
Cost Saving
$1.34m
50%
Cost Saving
$1.99m
75%
Baseline
Onsite/Offshore ratio
Initial onsite: offshore = 80%
Reduced onsite: offshore to 38%
Currently it is 15%
Key Initiatives:
Constant review and override of
onsite/offshore ratio
Maximized offshore capability using
effective governance
Onsite/Offshore rotation model
ROI / Efficiency Gains - Summary
AppLabs Confidential
17
Executive dashboard
AppLabs Confidential
18
Oct 07 Agent dashboard (functional, automation, regression)
Aug 08 Agent commissioning system (functional)
Sep 08 Sales mgmt (functional, regression, maintenance, compatibility, integration)
Sep 08 Contact management system (functional, regression, automation)
Nov 08 Policy processing system (automation QTP/Certify )
Dec 08 Helpdesk application (functional, regression, automation)
Jan 09 Fast Quote 2 (rate validation)
Apr 09 Fast Quote 3 (rate validation)
Mar 09 Service management system (vulnerability)
Phase I Discovery
(Oct 07 Mar 08)
Phase II - Steady State & Transformation (Apr 08
Mar 10)
Phase III - Transformation
(Apr10 )
Acquired knowledge on
business processes
Understood technology and
architecture of applications
Reviewed current testing
processes
Devised testing strategies
Defined qualitative and
quantitative goals
Institutionalize Insurance
certification program
Common practice and methodology for testing
Revamp testing and implementation life cycle for current
applications
Enforce discipline in SDLC by adherence to defined milestones
Increase the rigor in usage of automation tools
Focused groups for various test events
Rollout and implementation of QA practices for other applications
Rollout of QA practices and delivery of QA services for other
Lines of Business and other divisions
Develop and roll out infrastructure optimization and non
functional testing services
Continuous improvement
Annual review of core team
Quarterly review of flexi team
Multi Lines
Personal Lines
Dedicated Team size 240 (15 onsite +
225 offshore)
Dec 09 FAB Regression
Aug 09 Claims (functional, automation, performance)
Journey
Feb 09 Service management system (integration)
Largest e-Learning
Player
AppLabs Confidential
20
Largest e-Learning Player
Set up
(Nov03-Oct04)
Transform
(Nov04- Dec06)
Deliver
(Jan07- onwards)
Benchmarking
Test scope
Testing effort
Test approach
Standardization of QA processes
Basic reporting functions for
Defect life-cycle
Resolution management
Test case authoring / execution
Established basic communication model
Methodology
Adopted agile practices for QA
Quarterly audits and reviews
Introduced quality metrics
Service delivery performance
Monitored effort and schedule
Resource utilization
Scope change
Monitored defect detection and defect removal efficiency
Built local test lab Isolation server and QA environment
Trusted partner for QA
Manage the entire quality function of
customer with minimal supervision
Integrated AppLabs into client
business process and release
management
Nov 2003 Manual (Functional, Compatibility, Multi Lingual, Integration)
Nov 2004 Test Automation (Functional, Integration)
Oct 2005 Performance
Dec 2006 Certification
Jan 2008 Engineering (L2 & L3)
Team size 150
Oct 2009 Professional Services
AppLabs Confidential
21
Defect Detection Cost Savings
Effort to find defect reduced by 4 times
for core release
Effort to find defect reduced by 30% for
maintenance releases
Cost savings of $2.58 Million for core
releases
Cost savings of $191 K for maintenance
releases
Cost to find a defect
Drivers
Enhanced product knowledge through AppLabs Knowledge Management Framework
Better test coverage by using Bi-directional requirements traceability matrix
Formalized Statistical analysis and impact analysis to find vulnerable test areas
Defect Taxonomies used to drive testing effort
AppLabs Confidential
22
Test Engineering Efficiencies
Better product quality & decrease in test cycle
hours due to early defect detection
Volume of defects found during sprints is
50%higher
Reduction in overall testing efforts by 20%
(appr. 10,000 hrs)
Effort variance reduced YOY due to better Test
Planning
Defects acceptance ratio improved by 7%
because of defect triage and SME knowledge
Cost saving due to test engineering efficiencies is
$152 K
Defects in Sprints
Effort Variance
D
e
f
e
c
t
s
Defects Acceptance Ratio