Chief Skeptical Officer A New C-Level Role

Alan McSweeney

Why Does This Happen?

Too Late

The Big Idea

Reduced Functionality Requiring Manual Workarounds

Not What Is Wanted Or What Was Required/ Envisaged

Lack Of Integration

And Continue To Happen Time and Again?

Too Expensive

The Big Failure

Requires Rework Or Replacement

And What Can Be Done to Stop It?

October 18, 2011

High Cost Of Operation

Not Scalable And/Or Poor Performance

2

Spectrum Of Failures
Performance and/or Operational Problems More Expensive to Operate Than Planned Project Late and/or Over Budget Specified Business Benefits and Savings Not Delivered Functionality Delivered Does not Meet Business Requirements Significant Rework Required Solution Largely Unused And/Or Unusable Complete Project Success: On-time, On-budget and Delivering Specified Benefits

Increasing Frequency/ Probability of Occurrence

Complete Project Failure: Cancelled, Unused, Rejected

October 18, 2011

What Is The Role Of The Chief Skeptical Officer?

Role with end-to-end view and responsibility from idea definition to delivery Provide positive skepticism around idea definition and solution specification Be an independent voice Be an advocate for and champion of standards Maintain a repository of learning and standards Embed learning into the organisation

October 18, 2011

The Key Functions Of The Chief Skeptical Officer

Remove Arbitrariness, Relativity And Subjectivity

Provide Independent Oversight of Decisions

Require Assertions To Be Well Supported By Evidence

Ensure Rational Decision-Making Identify and Eliminate Behavioural And Other Biases On DecisionMaking Avoid Systematic Distortion and Misrepresentation Ensure Evidence-Based Judgements Question Assumptions, Attitudes Of Knowledge, Opinions and Beliefs Stated As Facts

Track End-To-End View From Concept To Delivery

October 18, 2011

The Journey From Idea To Successful Operation

Idea, Business Need, Business Benefits Process Definition and Solution Design Costing Implementation and Operational Solution Implementation and Delivery Solution Operation

Benefits Realised

October 18, 2011

The Journey From Idea To Successful Operation

Idea, Business Need, Business Benefits Process Definition and Solution Design Costing Implementation and Operational

But Promised Benefits All Too Frequently Not Delivered

Solution Implementation and Delivery Solution Operation

Benefits Realised

October 18, 2011

Frequently Too Many Handoffs On The Journey From Idea To Successful Operation
Idea, Business Need, Business Benefits

Handoff and Information Loss

Process Definition and Solution Design

Handoff and Information Loss

Costing Implementation and Operational

Handoff and Information Loss

Solution Implementation and Delivery

Chinese Whisper Effect as Initial Concept Moves to Implementation

Handoff and Information Loss

Solution Operation

Handoff and Information Loss

Benefits Realised

October 18, 2011

What Causes the Disconnect From Idea To Delivery

Mi sco mm Too Idea, Business u n Ma Need, Business ica ny Benefits tio Ha n, nd Iss o To ues ffs B Process -En Le etw Definition and d V ft U ee Solution Design iew nre n S or solv tage En ed s, d-T , A To Costing o-E ssu o O Implementation nd mp ften and Operational Re tio Se sp ns pa on M ra sib ad te Solution ilit e, N Te Implementation y o O ams and Delivery ne , W ith
Solution Operation

En d-

Benefits Realised

October 18, 2011

What Can Go Wrong Along The Journey From Idea To Successful Operation
Idea, Business Need, Business Benefits Process Definition and Solution Design Costing Implementation and Operational Poor Requirements Analysis, Poor Design, Poor Management Solution Implementation and Delivery Solution Operation Poorly Defined Need, Benefits Exaggerated Inadequate Analysis, Poor Or Incomplete Design Overly Optimistic Initial and Operational Cost Estimates

Promised/Expected Benefits Not Delivered

Slow, Unreliable, Expensive To Operate, Not Integrated, Requires Manual Workarounds

Benefits Realised

October 18, 2011

10

Cost of Fixing Errors During Project Lifecycle

Errors/ gaps/ omissions become significantly more expensive to fix at later stages of the solution
Solution Implementation and Delivery Costing Implementation and Operational Process Definition and Solution Design Idea, Business Need, Business Benefits
October 18, 2011

Benefits Realised

Solution Operation

Relative cost to remediate errors at the end can be 50100 (or more) times more expensive than at the start
11

Role Of The Chief Skeptical Officer Is To Take End-To-End View, Enforce Discipline and Perform Reviews
Idea, Business Need, Business Benefits Process Definition and Solution Design

Getting It Right Here Reduces/ Avoids Problems Here

Ch ief En Ske Costing su pt re ica Implementation Th l O and Operational at Th ffice e S rs Solution tar Ro t D le Implementation eli Is T and Delivery ve o Ta rs An ke d C En an d-T De o-E liv nd er Th View eE T nd o
October 18, 2011

Solution Operation

Benefits Realised

12

What Is The Chief Skeptical Officer Trying To Protect You Against?

Cognitive Bias Poor or inaccurate judgements, illogical interpretations and decisions, characterised by patterns of behaviour Strategic Misrepresentation Deliberate misrepresentation in budgeting caused by distorted incentives Planning Fallacy Systematic tendency to underestimate how long it will take to complete a task even when there is past experience of similar tasks over-running Optimism Bias Systematic tendency to be overly optimistic about the outcome of actions Focalism Systematic tendency to become inwardly focussed and to lose situational awareness and appreciation of wider context during times of stress
October 18, 2011 13

Cognitive Bias - Types

Many classifications and types of cognitive bias Can be very difficult to avoid because of their embedded nature and emotional/irrational basis
Decision-Making and Behavioural Biases - affecting belief formation and business decisions Probability and Belief Biases - affecting way in which information is gathered and assessed Attributional Biases - affecting the determination what was responsible for an event or action

Cognitive biases are very real and can have damaging effects
October 18, 2011 14

Decision-Making And Behavioural Biases

Relying too heavily on one piece of information when making a decision Anchoring Believing things because many others believe the same Bandwagon Assigning greater weight to apparently dominant factors Attention Bias Interpreting information so as to that confirms preconceptions Confirmation Seeing oneself as less biased than others Blind Spot Strong preference for immediate payoffs relative to later ones Hyperbolic Discounting Greater preferences just because of familiarity - Exposure Effect Paying more attention and giving more weight to the negative rather than the positive Negativity Bias Looking for information even when it cannot affect action Information Bias
October 18, 2011

Placing too much importance on one aspect - Focusing Effect Looking to reduce a small risk to zero rather than a greater reduction of a larger risk Zero-Risk Bias Rejecting new evidence that contradicts an established paradigm Semmelweis Effect Making decisions based to what is pleasing to imagine instead basing decisions on evidence and rationality Wishful Thinking Assigning a higher value to disposal/loss compared with cost of acquisition Sunk Cost Effect Viewing a harmful action as worse than an equally harmful omission or inaction Omission Bias Justifying increased investment based on the cumulative prior investment despite new evidence suggesting that the decision was wrong - Irrational Escalation
15

Probability And Belief Biases

Excessive or inflated belief one's performance, ability Overconfidence Effect Belief gaining plausibility through increasing repetition - Availability Cascade Assigning greater weight to initial or recent events more than subsequent or later events Serial Position Effect Assigning a lower probability to the whole than the probabilities of the parts Subadditivity Effect Avoidance of risk or the negative by pretending they do not exist Ostrich Effect Judging future events in a more positive light than is warranted by actual experience Optimism Bias

Perceiving patterns where none exist Clustering Selecting an options for which the probability of a favorable outcome is known over an option for which the probability of a favorable outcome is unknown - Ambiguity Effect Considering information to be correct if it has any personal meaning or significance Subjective Validation Overestimating the likelihood of positive rather than negative outcomes Valence Effect Failure to examine all possible outcomes when making a judgment Attentional Bias

October 18, 2011

16

Attributional Biases

Where skilled underrate their abilities and unskilled overrate their abilities DunningKruger Effect Defending the status quo System Justification Overestimation of agreement False Consensus Effect

October 18, 2011

17

Strategic Misrepresentation

Deliberate misrepresentation in planning and budgeting caused by distorted incentives Response to how organisations structure rewards and give rise to motivations Systematic (and predictable) misrepresentation
Deliberately understand costs to gain acceptance with understanding that costs will increase Not willing to face reality of high costs Overstatement or understatement of requirements Competition for scarce funds or jockeying for position Inclusion of ideology into planning

Underlying system and processes need redesign to eliminate

October 18, 2011

18

What Are The Key Disciplines The Chief Skeptical Needs To Promote

Disciplines and Skills

Benefits Management and Realisation Capital Planning and Investment Management Business and Process Analysis Solution and Process Analysis and Architecture Design

Practices
Reference Class Forecasting Audits and Checklists

October 18, 2011

19

Reference Class Forecasting

Technique to improve accuracy in plans and projections by basing them on actual performance in a reference class of comparable actions Based on knowledge about actual performance
Analyse distributions and probability of cost overruns Compare the proposed project with the reference class distribution to establish the most likely outcome

Used to avoid Planning Fallacy, Strategic Misrepresentation and Optimism Bias

October 18, 2011

20

Benefits Management And Realisation

To increase the likelihood of success from IT and other investments, organisations must identify the different causes of benefits before developing any project implementation plan Types of IT projects with very different approaches to benefits management
Fixing or improving something that exists
Resolve problem Improve integration

Implementing a new initiative

New system New processes

Need to focus on the changes needed to achieve results and take full advantage of new facilities offered rather than on just IT features Effective change management is crucial to achieving benefits
October 18, 2011 21

Achieving Benefits Some Questions To Ask

Why is there a need to improve? What improvements are needed? What improvements are possible or achievable? Have the improvements been agreed by all stakeholders? What benefits will be realised by each stakeholder if the business objectives are achieved? How can each benefit be measured? Who owns each of the benefits and will be accountable for its delivery? What business changes are needed to achieve each benefit? Have the explicit links between each benefits and required business changes been identified? Who will be responsible for ensuring the business changes are made successfully? How and when can the changes be made? Who will make the changes? Does the business have the ability and capacity to make the changes?

October 18, 2011

22

Benefits Management And Benefits Dependency Network

Benefits Dependency Network (BDN) is an approach to linking:

Information Technology Enablers, Changes and New Capabilities enabling technologies and functions and facilities needed to support the realisation of the identified benefits and to allow the necessary changes to be undertaken Business Changes business activities and new ways of working that are required to ensure that the desired benefits are realised Enabling Changes - prerequisites for achieving the business changes or that are essential to bring the new system into effective operation Business Objectives - high level priorities in relation to the drivers, outcomes and improvements to be delivered on completion of the project Business Benefits - outcomes of a change that are deemed to be positive by a stakeholder and that are valuable to the organisation and are measureable Stakeholders - individuals or groups who will benefit from the project and are either affected by or directly involved in making the changes needed to realise the benefits

October 18, 2011

23

Types Of Changes

Business Changes - Permanent changes to working practices, processes, procedures, interactions and relationships that will cause the benefits to be delivered
Generally need new IT system to be in place May require enabling changes to be implemented

Enabling Changes - Typically one-time changes that are pre-requisites for making the business changes or are necessary to bring the new system into effective operation
Can be made in advance of system implementation Training Processes redesign New work practices Changes to job roles and responsibilities
October 18, 2011 24

Benefits Dependency Network

Require Information Technology Enablers, Changes and New Capabilities Allow Enable Provide for Delivery of

Enabling Changes

Business Changes

Business Benefits

Business Objectives

BDN provides a framework for explicitly linking the overall investment objectives and the requisite benefits with the business changes which are necessary to deliver those benefits and the essential IT functionality to both drive and enable these changes to be made. BDN forms part of the benefits realisation plan Helps keep the focus on benefits realisation during the program execution Allows variations of the project or program to be assessed for their impact on benefits realisation

October 18, 2011

25

Benefits Dependency Network

Means to Achieve Changes

Information Technology Enablers, Changes and New Capabilities

Ways to Achieve Changes

Results of Changes

Enabling Changes

Business Changes

Business Benefits

Business Objectives

October 18, 2011

26

Benefits Dependency Network

Understand Why the Business Will Use the Information Technology System
What Information Technology Capabilities and Features Offer What Users Do With Information Technology Capabilities What Information Technology Capabilities Achieve What the Business Gains From Information Technology Capabilities Why the Business Uses The Information Technology Capabilities

Understand What Information Technology System is Needed to Deliver on Requirements

What Information Technology Capabilities Are Required What the Business Needs to Provide

What the Business Has to Do

What the Business Needs To See

What the Business Wants

October 18, 2011

27

Benefits Dependency Network

Shows the link from the solution, through business activities, outcomes and benefits to the organisation's overall drivers Used to confirm that the solution being introduced will actually provide the results you are seeking Any function within the proposed solution that is not linked to a benefit is potentially of doubtful value Functions with many link or links to key benefits can be identified and given extra attention BDN is a complex approach that requires benefit identification, assessment, validation and realisation maturity within the organisation Imposes a rigour on the organisation in analysing benefits from projects
October 18, 2011 28

Capital Planning And IT And Other Investment Core Requirements

Determine the scale, scope, and sources of funding for IT and other areas Assign financial resources to competing activities within the IT portfolio Establish a balance between capital expenditure (new projects) and operating expenditure (running systems delivered by past projects) Optimise the total cost of ownership Manage IT and other portfolios for value and not just cost IT and other areas need to implement a process for justifying its costs and be seen to be taking these steps

October 18, 2011 29

Capital Planning And IT and Other Investment Management

Aligns IT and other investments to organisation strategy (scoring) Prioritises investments (ranking) Provides strategic criteria for investment analysis Conduct annual IT and other portfolio management reviews Provides recommendation to stop, slow, maintain or accelerate program funding Identifies redundant/inefficient systems Integrates IT and other architectures within investments Ensures compliance with funding standards

October 18, 2011

30

Characteristics Of Credible Cost Estimates

Clear identification of requirements of the ultimate deliverable Broad participation in preparing estimates Availability of valid data for performing estimates historical, experience, benchmarks Standardised and comprehensive estimate structure that includes all possible sources of cost Provision for uncertainties include known costs explicitly and allow for unknown costs Recognition of inflation Recognition of excluded costs Independent review of estimates for completeness and realism Revision of estimates for significant changes in requirements
October 18, 2011 31

Challenges Of Developing Good Cost Estimates

Requires detailed, stable, agreed requirements Agreed assumptions Access to detailed documentation and historical data for comparison Trained and experienced analysts Risk and uncertainty analysis Identification of a range of confidence levels Adequate contingency and management reserves

October 18, 2011

32

Reasons for Good And Bad Cost Estimates

Effect i ve Unce Risk and r ta Ident Analy inty ificat s is Rang ion of a e of Confi De dence Docu tailed Level m s and H entation Adeq Train istor ua Conti e Data ical ngen te Exper d and cy an ien Mana Detai Analy ced geme d led, S nt sts Reser ves Agree table, d Requ ireme Agr n ts Assum eed ption s Unfa Techn miliar First- ology or Time Use Unre Unre alistic or liable Data No o Comp r Limited ariso Avail n Data able Proje ct Ins t abilit y Ineffe and U ctive Risk ncer Analy tainty s is Probl em Acces s Getting s to D ata Unre Assumalistic ption s New Pr ocess e

Unre Proje asonable ct Bas elin e Overo ptimi

sm

Comp le or Te x Project chnol ogy

Untra Inexp ined and er ie Analy nced sts Unre alistic Savin Project gs

Lost of reasons for and causes of inaccurate cost estimates

October 18, 2011 33

Sources of Risk and Uncertainty In Estimating Costs

Lack of understanding of the project requirements Shortcomings of human language and differing interpretations of meaning of project Behaviour of parties involved in the cost estimation process Haste Deception Poor cost estimating and pricing practices

October 18, 2011

34

IT Investment

IT is commonly seen as failing to deliver value for money Benefits and value must be actively managed for Realising and assessing business benefits from IT-enabled investments involves more than simply assessing Total Cost of Ownership for IT-related projects and managing the IT budget Key requirements
Ability to get lifetime costs right Ability to define benefits correctly and effectively Ability to manage the benefits management process Ability to increase and sustain benefits management maturity

Use existing methodologies and frameworks to implement key requirements quickly

ITIM Benefits Dependency Network ValIT Organisational change and commitment

Effective benefits management enables organisations to clearly and consistently articulate ITs contribution to achievement of business objectives
October 18, 2011 35

Business and Process Analysis And Design Builds Bridge From Business To Solution
Problem Requirement Current State Business Analysis and Solution Design Solution Design: Translate Requirements into Solution Solution Desired Future State Business Analysis: Elicit Requirements Analyse Communicate Validate

Business analysis is a key driver of business value Solution delivery start with business analysis

October 18, 2011 36

Weaknesses In Business Analysis Capabilities And Competencies At The Root of Many Project Failures
Poor Requirements
Poor Analysis Practices

Size/Capacity/ Complexity

Poor Strategic Alignment

Large Project, Complex, Difficult Changes and Processes Large Project Team and Multiple Stakeholders

Inadequate Business Case, Undefined Problem/Need Business Benefits Not Measured

Business Requirements Not Captured

Business Needs Not Met Opportunities Lost

Inadequate Resource Allocation and Prioritisation Inadequate Business Involvement

Unproven Technology

Inadequately Explored Solution Options

Investment Wasted

Dynamic, Changing Environment

Solution Design Not Aligned to Business Needs

Poor Focus on Business Needs

October 18, 2011

Uncertainly/ Ambiguity

Poor Solution Design

37

Analysis-Related Causes Of Failures

Poor Requirements
Poor Analysis Practices

Size/Capacity/ Complexity

Poor Strategic Alignment

Large Project, Complex, Difficult Changes and Processes Large Project Team and Multiple Stakeholders

Inadequate Business Case, Undefined Problem/Need Business Benefits Not Measured

Business Requirements Not Captured

Business Needs Not Met Opportunities Lost

Inadequate Resource Allocation and Prioritisation Inadequate Business Involvement

Unproven Technology

Inadequately Explored Solution Options

Investment Wasted

Dynamic, Changing Environment

Solution Design Not Aligned to Business Needs

Poor Focus on Business Needs

October 18, 2011

Uncertainly/ Ambiguity

Poor Solution Design

38

Taking A Complete View of Systems And Processes Is Essential to Solution Success

Overall solution operates with a mix of automated and manual processes in a structured or ad-hoc manner o deliver the required results Understanding the overall set of processes and their operation is crucial to successful results Need to see the entire picture to understand how a solution should operate
Systems/applications are just one part of this universe

Unambiguous definition of processes is required Processes that are to be automated define the scope of the development and implementation work
October 18, 2011 39

Complete View of Systems and Processes Is Essential to Solution Success

External Manual Interaction External Component
Automated Process

External Manual Interaction External Component

Manual Process

Manual Process

System Component

System Component

Manual Process

Automated Process

System Component

Automated Process

Manual Process

External Manual Interaction

October 18, 2011

External Component

External Manual Interaction

40

Solution Design and Implementation Sequence

Defines where the business wants to go
Business Plan

Business Need

Business need identifies solutions that will allow delivery of plan Defines the benefits to be achieved by the solution Defines the detailed requirements of the solution

Business Benefits

Requirements Definition

Defines the processes that will be implemented by the solution Defines the solution design to implement the processes Creates a detailed technical design for implementation

Process Design

Solution Architecture and Design Technical and Detailed Design

Implementation

Implements the detailed design

October 18, 2011 41

Consistent Approach to Business And Process Analysis

Adopt a consistent and robust framework in business analysis

Enables effective benefits realisation through a solution which meets the business need

Key elements
Establish enterprise standards, procedures and governance Standardise on infrastructure, analysis methods and operational procedures Develop competence and skills

Key benefits
Implement solutions that meet business needs Increase the ability of the business to adapt quickly to changes Reduce risk, complexity, redundancy and support complexity Align business and IT Enable re-use and faster time-to-market Present one face to the business (customer) Increase business value
42

October 18, 2011

More Information
Alan McSweeney alan@alanmcsweeney.com

October 18, 2011

43