Risk Monitoring

Week #4 CRISC Exam Prep ~ Domain #3

Bill Pankey
Tunitas Group

CRISC
Risk
Monitoring
Domain

Job Practice

Collect and validate data that measure key risk indicators

(KRIs) to monitor and communicate their status to relevant
stakeholders.
Monitor and communicate key risk indicators (KRIs) and
management activities to assist relevant stakeholders in their
decision-making process.
Facilitate independent risk assessments and risk management
process reviews to ensure they are performed efficiently and
effectively.
Identify and report on risk, includingcompliance, to initiate
corrective action and meet business and regulatory
requirements.
2

Agenda
Key Risk Indicators

What are they

How to construct
How are they used
How are they improved
How are they reported

Data Aggregation
Benchmarking
No distinct RiskIT monitoring process
3

Expansive View of Risk

Monitoring
Risk Governance Objective
Monitor the overall
performance / effectiveness
of the risk management
program and recommend
improvement (COSO)

Risk Management Objective Ensure

the current and emerging levels of risk
are within tolerance levels.
4

2nd Order Uncertainty

Risk is a statement about the potential for
[future] loss events
Manage by control; avoidance; sharing

Risk monitoring identifies and evaluates changes

in risk
Whether potential for loss is increasing / decreasing
Added opportunity to manage risk
Surprisingly, statements about changes is risk are less
ambiguous, more objective than the statement of risk

Key Risk Indicator (KRI)

Metric / observation used to
track risk level at specific time point
where likely unacceptable loss or trouble ahead

Indicator becomes key when

Tracks an important risk
Is reliable, cost effective,

# of unpatched systems is a risk indicator,

but may not be key
What is the risk that is being tracked?
How important is that risk?
6

KRI are not KPI

KPI could be KRI
KRI are leading indicators
Intended to be predictive of future loss / outcome

Key Performance Indicators are lagging

indicators
Report on accomplishment of activity / process

A given KPI could be used as a KRI or

component of KRI
% of [expected] function points delivered on time
Measure of project efficiency / could be used as
indicator of project delivery risk
7

KRI Uses

KRI proxies Risk

Measures

Operational
risk
management
Strate
gy
Normalizati
on
Risk
Communication
Compliance ??
?

Source: Risk Management Association 2005 survey of

Why KRI are Important

RISK FACTOR ~ condition influencing
frequency, magnitude business impact of the
loss event / scenario

Change in risk factor [some] risk indicator

Logically and [typically] temporally prior to risk
event
9

KRI Selection
Unlimited # of risk indicators in logs, alarms,
reports

What to select for regular monitoring as KPI

Reflects management priorities
Stakeholder concern
Strategic and / or operational business impact
Management utility / basis of management report
Basis for risk communication
What gets measured, gets done, Drucker, The Practice
of Management
10

Effectivene
ss

KPI Goodness Criteria

1. Associated with one or more specific risks

2. Measureable at specific points in time
3. Objective finding rather than a subjective
assessment
4. Track at least one risk factor
5. Actionable
11

Comparabil
ity

KPI Goodness Criteria

1. Quantified (#, %, ratio, rate)

2. Well defined / reproducible
3. Time independence
4. [Business] Process independence
5. Auditable
6. Comparable across organizations (?)
12

Efficiency

KPI Goodness Criteria

1. Timely, readily available in reasonable

time frame
2. Cost effective to collect as a
production of automated system, byproduct of process or service
3. Obvious easily understood and
communicated
13

ISACA
Best
Practice
1. Data access

KPI Process Steps

Ensure timely, reliable data delivery

2. Data validation

Match definition; complete; within range; missing

data(?); duplicates (?); reliability of derived values;
referential integrity
reasonableness checks!

3. Data analysis

Statistical computations
Conclusions / inference

4. Reporting

Right people, right format

14

ISACA
Best
Practice
5. Optimization criteria

KPI Optimization

Sensitivity

Appropriate level of alert; # of red flags; Critical conditions, etc

Timing

How much lead time

Frequency

How rapidly may new risk condition develop; express themselves

Corrective action

Utility for tracking remediation effort; assigning priority (MBO)

KPI autocorrelations, correlations with each other,

loss and performance data provide an empirical
basis for optimization

Regression analysis
15

KPI Validity: Causal

Factors
Utilize expertise of subject matter experts,
process and service owners
1.Identify the risk scenarios of greatest concern
2.Decompose scenario into leading risk factors
3.Identify indicators for those factors
Observable measures; critical thresholds or
change

4.Develop KPI reporting scheme

16

KPI Timing
Early indication greater
opportunity for correction

Failed IT project
Loss of key IT personnel
KRI: Mid-level Staff retention
rate
Clinger-Cohen Act of 1996 new demand for certified IT
professional
KRI: Industry Salary Index (annual)
17

Pro-Active Risk
Management

18

Aggregation
Address management concern regarding
overall risk :
Specific business objectives

Strategic | Operational

Customer
Product
Regulation
Business Unit
arbitrary level of specificity

Management interest is greatest when all

variety of risk is reported
19

Aggregation
Diversity Problem:
For different risk domains (say IT, Legal,
Finance ..)
KRI
KRI
KRI
KRI
KRI

are specialized
have different time periods
have different granularity
have varying sensitivity
have varying relevance, reliability and validity

This is a problem that is not solved so much

as overcome
20

Aggregation Heuristic
Report risk as dimensionless quantity (kounts,
%)
Report risk as % red, yellow, green

Code each KPI relative to threshold (red or green)

For each risk, count the number of associated KPI above
of below threshold

Report risk as increasing of decreasing

# KPI indicating lesser risk / # KPI indicating greater risk

Loss data

Cumulative impact of loss events

Possible impact of managed events(intervention)

21

Management Report
Visual display of risk based on risk
indicators
With thanks to Excel, easy to produce
Dashboards .Red light / green light
gauges
Heat Maps
Spider Diagrams

22

Industry Risk
Benchmarks
Collection & summarization of risk data
Loss data
KRI

Source for:

Validation of enterprise results

Trend analysis
Risk analysis data
Comparative benchmarking (company)

23

e.g. Standard KPI

Specification
www. KRIeX.org Repository

~ 2500 KPI specified and monitored

24

e.g., Loss Data

25

Bottom Line
Express organizational 'risk appetite
in terms of a KPI threshold value
Alert management to trends that may
affect achievement of objectives
Use KPI to initiate mitigation activity
Provides measurable data conducive
to aggregation
Assists in demonstrating compliance
26

Next Week
CRISC Domain #4
Control Design & Implementation

27