Enterprise Risk
Structuring an Efficient
Program for
Model Governance
by Niall Lynas and Elizabeth Mays
Model governance refers to the practices undertaken to
promote the proper development, use, and ongoing vali-
dation of models. The purpose of model governance is to
minimize the risks associated with relying on models to
make or inform decisions within an institution.
Financial institutions use models for a wide range of pur-
poses, including forecasting credit losses, valuing complex
derivatives, and evaluating exposures to changing interest
rates. Banks are relying more and more on models to make
automated decisions and to provide guidance to risk manag-
ers on other complex and important decisions.
This heightened reliance, in combination with regulatory
issuances related to modeling and a generally increasing fo-
cus on risk mitigation over the past decade, has led banks to
invest in improving their model governance infrastructures.
Furthermore, models have been assigned a large share of
the blame in the recent financial crisis for failing to capture
important risks embedded in transactions, lending prod-
ucts, and securities. These issues have quickened adoption
44 March 2010 The RMA Journal
of model governance initiatives across the industry.
There has been a tendency for model governance initia-
tives to gravitate in one of two directions. When portfolios
are stable and experiencing few losses, or if model risk is not
deemed to be a priority, there is a desire to implement what
could be considered a “tick the box” approach (a structure
designed to satisfy minimum regulatory requirements, but
with as little expenditure as possible). But when portfolios
have experienced significant losses that models failed to
predict, or when management has experienced regulator or
auditor pressure, there can be a tendency to implement an
extensive program that applies the most rigorous governance
requirements to a comprehensive set of models.
Neither approach is usually appropriate. Clearly, a limited
governance infrastructure that 1) lacks senior management
support, 2) is executed by personnel who lack the requi-
site skills, or 3) misses key governance elements can lead
to significant model risk. A history of stable, predictable
losses and satisfactory model performance should not lull
Olly/Shutterstock
institutions into believing model governance is not needed.
Historical stability is no guarantee that the future will be
similar. In fact, the very reason models are used is to lever-
age predictive information to identify movements in key
performance metrics before those movements occur.
An overly extensive program also can be harmful. First,
there is the cost of staffing governance and validation func-
tions. Given the widespread use of models, this cost can be
very significant. Second, making governance too challenging
can hinder the ability of management to adapt model usage
as the business environment changes. Worse, requirements
that are viewed as too onerous can lead business leaders to
avoid using models altogether.
This article discusses factors that should be considered
when planning a model governance framework, including
understanding why governance is important and the role that
key elements of governance play in mitigating model risks.
Finally, we sketch out an example framework that categorizes
models according to their degree of risk, and the complexity
of the risk being managed, in order to ensure that appropriate
practices are applied in all circumstances.
Why Do We Need Model Governance?
Before designing a governance framework, organizations
should take time to consider why model governance is im-
portant. Models are representations of relationships between
causal variables and outcome variables; as such, they are
subject to being wrong. All models carry risk.
The development and ongoing management of models
can involve complex processes requiring specialized knowl-
edge and consideration of many types of information. If not
performed correctly, there can be sizable negative impacts
on the business. These include rapid deterioration of model
performance (leading to financial loss), models that perform
significantly worse than those used by peer organizations,
and materially inaccurate risk measurement. Model errors
and poor model controls also subject financial institutions
to considerable compliance risk and reputation risk.
Model risk can arise from many sources. First, there could
be mistakes made in model building, including using inap-
propriate assumptions and statistical techniques or flawed
data. It’s not uncommon to find errors in data used to develop
models, especially when data is being combined from several
sources. Second, a model may fail to incorporate all risk
factors that influence the outcome being modeled. Also, the
relationship between risk factors that are included and the
outcome may change relative to what it was at the time the
model was developed. This can happen when market par-
ticipants, economic conditions, or processes inside the bank
change. Another primary source of model risk arises from
inappropriate application of models to populations beyond
those for which the model was originally designed.
Although bank regulators have put forth model gover-
Before designing a
governance framework,
organizations should
take time to consider
why model governance
is important.
nance guidelines, governance practices should be followed
not just to comply with regulatory guidelines, but to help
limit the risk of improper and imprudent decisions being
made as a result of relying on models. Governance practices
are intended to ensure that models are properly built, that
they address the needs they are intended to address, that
users are fully informed of model assumptions and limita-
tions (and all models have limitations), and that models
continue to perform as intended over time.
Elements of a Sound Program for Model Governance
Model Policy
Perhaps the most important ingredient to a sound gover-
nance program is a model policy. A formal policy is crucial
to successful implementation of a governance program,
as it should provide interpretation of regulatory require-
ments that are applicable to the organization, as well as
the organization’s own approach to mitigating model risk.
Given the considerable costs associated with governance,
the policy should reflect the “tone at the top” and convey
senior management’s commitment to model governance,
both through statements within the policy and by the level of
senior management engagement that the policy requires.
Model Inventory
A model inventory is a catalogue of models that allows
management to have a holistic view of the model risk faced
by the organization and to understand where (organiza-
tionally) that model risk resides. The inventory also helps
promote consistent application of governance practices
across business units.
In Bulletin 2000–16, the OCC recommends that a cata-
logue of models be maintained at the corporate-wide level
and, via the Comptroller’s Handbook on Retail Lending, it
provides a useful list of what information should be included
The RMA Journal March 2010 45
 for each model: name, description, type, date developed,
source (in-house versus vendor), purpose, last validation
date, next validation date, and the names of management
contacts.1 The model
In drafting independent inventory should also
include a log of all
review requirements, an outstanding issues that
important consideration is pertain to the models,
whether a unit dedicated including validation
status and due dates,
solely to validation and management con-
should be established. tacts for remediation.
Some organizations
also use their model inventory as a repository for storing
all model-related documents, including model development
documentation and validation reports.
Model Documentation
Model documentation serves two purposes: It facilitates
independent review of the model development and serves
as a corporate memory of the methods, data, and assump-
tions embedded in the model. Model users and reviewers
may refer to the documentation over the life of the model
to recall details that will help inform decisions related to
model performance and usage. Auditors and regulators also
will want to see this documentation to evaluate model de-
velopment practices and methods.
Institutions may wish to adopt varying documentation
standards for different categories of models, but there is a
core set of topics that should be addressed for all model
developments:
• What is the purpose of the model and how will it be
used?
• What is the outcome being predicted or estimated?
• Which data was used to develop the model (sources,
variables, historical time frames, counts, and so forth)?
• What are the assumptions or judgmental components
of the model?
• What are the model limitations? For example, if the model
was built for a direct-to-consumer home equity portfolio,
it should probably not be used for a broker portfolio. If a
model was built using data from a prime auto portfolio,
it should not be used for a subprime portfolio.
• What are the methodologies and techniques used? For
example, is the model statistical, is it a set of cash flow
equations, is it based on simple moving averages, or a
combination of all three? Does it make use of a Monte
Carlo simulation or scenario analysis?
• Who are the model developers and approvers?
• How and how often will the model be benchmarked and
validated going forward?
• How well did the model perform on a “benchmark” data
set (defined below)? This information provides a standard
46 March 2010 The RMA Journal
for making the decision to replace the model in the event
of its deterioration.
It may be useful for the bank to develop templates that lay
out exactly what information is required as part of the docu-
mentation. Having consistent documentation standards for
similar model development projects facilitates independent
review and allows the model developers to know what is
expected at the start of any development project.
Independent Peer Review
Modeling processes, especially model development pro-
cesses, can be complex and comprise many different
procedures. Often, these will include the application of
statistical analyses that may not be within the expertise of
the managers overseeing the development project or of the
auditors who perform regular reviews. In these circum-
stances, an independent review by a “peer” individual or
team (which has experience in the type of modeling ap-
proach that was used and at identifying risks in modeling
processes) can help reduce model risk. It also can facilitate
the identification of assumptions that are inappropriate
or not fully understood. This latter risk is particularly
important, as assumptions can come in many different
forms ranging from data assumptions (for example, that
the specific sample chosen is the most relevant to the one
on which the model will be used) to statistical assumptions
(for example, losses will follow a specific distribution, or
there is a specific relationship between dependent and
independent variables).
In drafting independent review requirements, an im-
portant consideration is whether a unit dedicated solely to
validation should be established. A dedicated unit allows
validation personnel to focus more exclusively on valida-
tion work and on improving their validation skills (which
can differ from the skills needed in developing models).
Furthermore, a structure involving a dedicated unit is also
commonly viewed as enhancing independence, which, as
noted in RMA’s Model Validation and Governance Survey,2
may now be considered a “leading practice.”
However, larger organizations may also have the option of
pulling resources from other development teams to provide
independent reviews. This can enable validation person-
nel to keep their development skills sharp, and it often
allows for a larger pool of resources from which validation
personnel can be selected (thus making it easier to identify
reviewers with appropriate experience).
Whichever structure is used for independent reviews,
the organization should be careful to adhere to the spirit of
independence. This usually goes beyond simply requiring
that the reviewers had no involvement in the modeling
process; it requires that they be free of influence that would
prevent them from providing an honest critique of issues
related to the model.3
Back Testing
Perhaps the most important principle of model governance
is that models be back tested on an ongoing basis to ensure
they continue to perform in line with expectations. Back
testing compares model forecasts or estimates generated in
the past to actual outcomes. Using these results, analysts can
calculate model accuracy measures, which are then compared
to benchmark accuracy measures to determine if the model
remains accurate enough to use in its current form.
In developing benchmark accuracy measures, model
performance should be measured on the development data
sample as well as on a separate “validation sample.”4 One of
these samples (usually the validation sample) is used to estab-
lish a benchmark set of performance metrics to which future
model back-testing results will be compared. The relevant
metrics will depend on the type of model, but may include
the Kolmogorov–Smirnov (K–S) statistic for scorecards, or
perhaps root mean squared error for forecasting models.
By comparing the back test’s metrics to the benchmark
metrics, an analyst can see if deterioration in model accuracy
has taken place. Accuracy thresholds should be identified,
although they should not be viewed as a “pass/fail” mark.
Instead, they indicate a point at which additional actions
should be taken. This may simply involve running additional
reports or making an ad hoc analysis so that reasons for shifts
can be understood and the need for model alterations can
be evaluated. If a model is found to be underperforming, a
decision must be made to modify or replace it. Sometimes
models can be improved by “calibrating” model parameters
using recent data. Other times, models will need to be rebuilt,
or an entirely new model with a different structure will need
to be implemented. In either case, a timeline should be set out
for calibrating or replacing the model, and the documentation
and independent review process will start once again.
Benchmarking
Benchmarking compares a model’s results to those of a second
model that predicts the same (or a similar) outcome. Large
differences between the two models’ results should be evalu-
ated, and the reasons for the differences should be used to
detect inappropriate assumptions or errors in calculations.
Benchmarking is best performed using independently
constructed alternative models or “well validated” existing
models. If an organization determines that such benchmarks
don’t exist, and if the model risk is not considered high, it
may also determine that developing a benchmark model
is not warranted. In such cases, benchmarking can still be
performed by either selecting models that predict similar
outcomes or identifying industry metrics associated with
similar portfolios.5
Computational Testing
Model code should be tested on a regular basis to make
sure there are no calculation errors. This testing can include
selecting a sample of records for manual review or develop-
ing separate code to implement the same algorithm.
Oversight and Change Management
A key component of a governance program is the need
for adequate oversight and change management controls.
Oversight should involve active participation by managers
in the management of the model. While managers may
not need deep knowledge of all the nuances of the devel-
opment process, they must have sufficient familiarity to
enable them to understand the key model risks related to
the model’s intended use. This will entail understanding
the data that was (and wasn’t) used, the general process
that was used to specify the model, the nature of the risks,
and the key assumptions. Management approval for the
use of new models or changes to existing models should
be formalized.
Having access to model details or permission to change
a model may be an important issue, depending on the
model purpose and construct. Access to models used for
decision making typically will be restricted to reduce the
chance that knowledge of decision criteria will influence
data inputs to the model. Restricting access to models used
to quantify significant financial statement entries may be
a key control under a company’s Sarbanes–Oxley Section
404 compliance program.
Stress Testing
Although stress tests have received much attention in recent
years, formalized approaches can still be quite rare and are
often limited to Basel II reporting requirements. Two types
of stress tests that should be applied to most models are
scenario analysis and sensitivity analysis.
Scenario analysis involves computing model output under
a specific set of assumptions, which are usually associated
with macroeconomic conditions. This type of test is of par-
ticular importance for any model that estimates the future
value of a metric that fluctuates significantly throughout the
various stages of the economic cycle.6 Ideally, a series of
scenarios should be de-
signed at the enterprise A key component of a
level so that results can
be consolidated across governance program is
portfolios. the need for adequate
Sensitivity analysis
shows how model out-
oversight and change
put is impacted by fluc- management controls.
tuations in one or more
variables or parameters. It provides insight into whether the
model is too reliant on one or more variables or parameters
and indicates the level of model risk due to the potential
volatility of model output.
The RMA Journal March 2010 47
 Figure 1
Model Risk Categorization
High
Complexity
B D
Modeling Process Complexity
A C
Out of Scope
Low
Complexity
Low Risk
Purpose
Example of a Model Governance Framework
Given that the definitions of a model and model risk are
open to interpretation, and in light of the considerable
cost of model governance, an essential step in developing
governance infrastructure is to define the scope of tools
and analyses that will be considered “models” and subject
to the requirements of model policies. This step will typi-
cally exclude ad hoc analyses, simple historical averages,
and other tools where model risk is deemed immaterial.
Once the scope has been defined, the cost of governance
can be reduced further by categorizing models according
to their levels of risk. To do so properly, however, one may
need to take multiple considerations into account when
determining the treatment of each model. Consideration
should be given to a model’s complexity of design and
its importance within the bank in determining risk man-
agement and business decisions (referred to as “model
purpose”).
For example, consider two models, both deemed to
have high model risk. The purpose of the first model is to
forecast losses, and it influences a range of processes such
as loss reserves, budgeting, and collections staff manage-
ment. Even if the modeling approach is straightforward and
not considered complex, the model may be deemed high
risk because of its widespread influence on management
decisions. It may be appropriate to require an independent
review carried out by a team outside the risk and loss fore-
casting function so that it is as far removed organizationally
from the development team as possible.7
Now consider a model that is used to value an asset or
liability and is considered high risk owing to the complexity
of the modeling process. In this case, it may be much easier
to identify individuals who don’t have a vested interest in
the model output, but are close organizationally to the team
48 March 2010 The RMA Journal
that developed the model. In fact, selecting such individu-
als may be the most appropriate choice if the modeling
approach involves highly specialized skills typically found
only within one area of the organization.
Consideration of “model purpose” and “model complexity”
allows us to categorize an inventory of models into quad-
rants as illustrated in Figure 1. By categorizing models in this
way, the quadrants represent homogeneous groups of models
relative to the importance of each of the model governance
elements that were listed earlier. Therefore, we can describe
how each element should be applied to the models in each
of the four quadrants, as illustrated in Table 1.
The size and culture of an organization will affect the
manner in which such a framework should be imple-
mented, including the definition of a model, the number
of categories and how they are defined, and how each
High Risk governance element is applied to each category. However,
this example shows how the potentially complex risks
associated with models can be addressed with a frame-
work that facilitates ease of implementation while allowing
flexibility to avoid some of the more burdensome costs
associated with governance.
Conclusion
Banks that ignore model risk or implement frameworks
that are too light to really control model risk do so at their
own peril. There are many organizations that can attest to
the significant costs, financial and otherwise, of assuming
that their models will continue to work well, only to find
that they don’t. Nevertheless, many institutions may view
the potentially high costs associated with implementing a
sound governance program as a barrier.
If an organization takes the time to carefully plan its
approach to model governance—taking into account the
reasons why governance is important, the various aspects
that make up model risk, and the elements of governance
and how they can be used to mitigate those risks—and then
structures a program that targets the elements of governance
to the models where each element will be most effective, a
structure can be implemented that mitigates key risks while
not becoming unnecessarily burdensome. v
••
Elizabeth Mays is senior vice president and head of consumer risk modeling and ana-
lytics for JPMorgan Chase. Contact her at elizabeth.mays@chase.com. Niall Lynas is
a senior risk manager at JPMorgan Chase. Contact him at niall.lynas@chase.com.
The views expressed in this paper do not necessarily represent the views of JPMorgn
Chase & Co.
Notes
1. Although written for retail lending, and the inventory pertains to
credit-scoring models, the items listed apply to most types of models.
 Table 1

Approach for Applying Governance Elements to Model Categories

Quadrant
Governance Requirement
A B C D

Inventory tracking Required.

Documentation
Independent peer review
Back testing/empirical validation
Benchmarking
Model monitoring
Computational testing
Oversight and change management
Stress testing
Reduced level of documentation required. High level of documentation required.
Not required. Required. The review team needs Required. The review team must Required. The review team needs
to have experience in the specific have experience relevant to the to have experience in the specific
modeling approach and purpose, model purpose, but does not need modeling approach and purpose.
but can be selected from any area experience in the specific approach A high degree of organizational
of the organization (as long as its being used. A high degree of separation is required to ensure
members were not involved in the organizational separation is required independence.
model development or ongoing to ensure independence (e.g.,
management). validation personnel are selected
from a different group or line of
business).
Simple metrics (actual versus Back testing of all model segments, with tolerance thresholds and action Full back testing, including analysis
predicted at the portfolio level). plans. of individual model variables,
tolerance thresholds, and action
plans.
Appropriate benchmarks are required to be identified and reviewed. Appropriate benchmarks are required Benchmarks must be independently
Benchmarks can be industry metrics or models that predict similar outcomes, to be identified and reviewed. constructed alternative models or
and trends are analyzed instead of absolute values. Benchmarks can be industry metrics “well validated” existing models.
or models that predict similar Further analysis is required if
outcomes, and trends are analyzed significant unexplainable differences
instead of absolute values. A are identified.
summary of reasons for differing
trends in results is required to be
compiled on a regular basis.
Required on a quarterly basis. Required on an annual basis. Required on a quarterly basis. Required on a quarterly basis. Must
include analysis of individual model
characteristics.
Required on an annual basis. Required on a quarterly basis.
Protocol for these items does not Protocol must be formalized, with designated oversight personnel, clearly defined approval authorities, and security
need to be formalized. mechanisms to prevent unintended or unapproved alterations to the model.
Required for forecasting models, but not for models used to rank order. Required for all model types.

2. See The RMA Journal, November 2009, pp. 60–65.
3. True independence may also call for physical separation of review-
ers from development personnel. In the Securities and Exchange
Commission’s 2008 assessment of oversight at Bear Stearns (SEC’s
Oversight of Bear Stearns and Related Entities), the Office of Audits
noted that “model validation personnel, modelers, and traders all sat
together at the same desk” and that this has “the potential disadvan-
tage of reducing the independence of the risk management function
… in both fact and appearance.”
4. The validation sample should not be used in the actual model
development. Ideally, it should be an “out of time” validation sample
from a time period different from the development sample. Evaluat-
ing the model on an out-of-time validation sample will help ensure
that the model performs well in environments different from the one
represented by the model development sample.
5. While the absolute values of these benchmarks may not be of in-
terest given the different purposes or portfolio profiles, they can be
useful for identifying trends that may be relevant to the model be-
ing benchmarked. For example, if a benchmark displays a change
that is greater than the model output change by more than a cer-
tain threshold, this could trigger additional analysis to determine
the reasons for the change (and, specifically, whether the model is
capturing key predictive information that the benchmarks may be
accessing).
6. While scenario analysis is mostly applicable to forecasting models,
it also can be used to show how well models rank-order risk under
economic conditions that are more extreme than those observed dur-
ing development or “out of time” sample windows. For these “rank
ordering” models, scenario analysis also can be used to show how
cutoffs might need to be altered as conditions change.
7. The nature and size of a portfolio for which a model will be used
may also influence the “purpose” categorization of models. For ex-
ample, models used for very small portfolios may be classified as
having a purpose that poses lower risk to the bank than those used
for larger portfolios.

The RMA Journal March 2010 49