An Integrated Approach to Performing

Pre-implementation Reviews

Securities Industry and Financial Markets

Association February 29, 2012
Andy Ellsweig, Director
Technology Risk Advisory
Services

Discussion Approach

1)Introductions / Course Objectives

2)Integrated Audit Overview
3)Timing and Scope of Integrated
Pre-imp Audits
4)Wrap-up / QA

Introduction
In todays business environment where companies are increasingly
turning to technology solutions (e.g., Mobile Apps, web apps, Cloud
Computing) to gain a competitive edge, major system implementations
can have a huge impact (negative or positive) on organizations.
The scope of the audit and staffing mix on pre-implementation reviews
will vary depending on the type of system under review, the platform the
system is processed on, whether it is processed in house or outsourced,
the timing of audit involvement and the business processes supported
by the application.
Because of the complexity of some of these applications and the impact
new systems can have on business operations and financial reporting,
an integrated approach to auditing these implementations can help add
more value to our organizations.

Course Objectives
To provide an approach and framework for
planning and executing integrated audits of new
system implementations.
To evaluate the potential audits that could be
performed for new system implementations and
determine the optimal team composition &
timing.
Facilitate an Interactive Discussion
Share Thoughts & War Stories
4

Integrated Auditing
Overview

Integrated Auditing Objectives & Goals

To provide an approach to performing risk-based

reviews (that builds upon the traditional IT General
Controls Analysis) in order to assess & communicate
Information Technology risks from a business or
regulatory perspective.

Integrated Business Audit Approach

Todays audit approach now requires the merging of both
Traditional and Information Technology audit approaches.
Not all controls to be evaluated will be system-specific, but in
todays environment, many of the controls will be platform
and application related.
The integrated business approach
focuses on business risk. Access
Controls provide the primary
preventative controls within the
integrated business audit approach
and will be implemented across all
information technology layers.

Integrated Business Audit Approach

Today we will focus on performing
integrated auditing as part of a pre- or postimplementation application review.
This Layer includes the application
programs, libraries and resources
independent of the platforms in which
they reside.
Application Programs cover major
functional processes, security (menu
options and transaction level security)
is evaluated within this layer.
We will touch upon relevant areas within the platform and networking layers
but only enough to link to specific application risks.

Changes In Approaches to IT Auditing

Where is Integrated Auditing headed?

We now have new platform environments (e.g., Web apps, Mobile

apps), new techno-jargon.

We frequently have to explain WHY we audit Windows, UNIX or

SAP Basis, IOS, ANDROID

Todays integrated reviews require the auditor to review and

assimilate volumes of data about, what resources are accessible,
what applications do, how the users Use or Abuse the system and
how the computer system functions.

Audit tools must be selected and an approach developed that will

require auditors to build upon the traditional logical
control/application analysis.
9

Changes In Approaches to IT Auditing

We now must assess controls in light of business risks

Assess IT controls at different layers

Need to determine if controls are really functioning as designed

Identify new areas where exposure/failure may occurAs well as

identifying what additional manual controls/surprises are out there

Must be done from a Business perspective

Auditors must adopt a holistic approach that is equally adaptive to

Traditional and newer technology environments

10

Business Risk The Driving Force to Integrated

Auditing

The common thread throughout the design and implementation of the

Companys Information Technology Architecture and Development of the
Audit Plan is the significance of identifying the Business Risk.

Business risk is the threat of an event or action that will adversely affect an
organizations ability to achieve its business objectives and execute its
strategy successfully.

Effective business risk management begins with assessment. Management

must constantly review the risk of catastrophic economic loss, business
interruption and loss of business reputation.
Business risks arise as much from the likelihood that
something required or planned wont happen as they
do from the threat that something bad will happen.
11

Business Risk The Driving Force to Integrated Auditing

Must first identify the business risks
The source(s) of risk should be clarified and measured
(e.g., High/Medium/Low)
We can then develop the comprehensive integrated
audit plan that includes a balance between risks and
controls.
One of the major challenges facing the auditor today is
understanding what must be evaluated as part of the
review.

12

Business Risk The Driving Force to Integrated Auditing

Application Layers
Potential control elements that can be examined during
integrated audits of new systems.

PROCESS Business functions and processes that use IT (which

generally includes most core business processes)

APPLICATION Application software and functions

DATA MANAGEMENT File structure and DBMS software controls

PLATFORM Hardware platform including OS and system software

NETWORK LAN, WAN, Internet, Intranet and support systems

PHYSICAL Components that house, support and process IT

13

Business Risk The Driving Force to Integrated Auditing

The integrated audit approach attempts to strike a

balance between business risks and controls within
the various layers
The trade off between the Cost of the risk and the
Cost of the controls being a deciding factor
This balance should be considered in determining
reportable findings

14

Business Risk The Driving Force to Integrated Auditing

What are the next steps?
We acknowledge that business risks should drive our
scope and involvement
Information technology controls can be segregated into
multiple layers
There is a need for an integrated audit approach that
address each area based upon business risk
So how de we apply these concepts to large-scale
implementations?

15

Major System Implementations

Integrated Scope Considerations

Stages of Audit Involvement

The timing and scope of pre- and post-implementation
audits is based on a number of factors:
Size of implementation (revenue, expenses, financial
transactions, etc) and materiality
Regulatory Requirements (SOX, privacy, trade
monitoring)
Timing of implementation (if timeline too compressed,
can they support an audit?)
Resources - major cause of failures (i.e., dedicated
implementation team and business resources)
Type of methodology used
17

Obstacles to Implementing Controls

Lack of Awareness of Security/Control Guidelines

Application developers do not have adequate visibility or

knowledge of IT policies

Ambiguous Policy Statements

Vague policies are difficult to translate into actionable code

Security & Controls Are Viewed as a Disabler

General perception at all levels of the organization that

security and controls delay projects

Unclear Ownership and Accountability

Ownership of security and control processes and data are

not well defined or understood

18

Stages of Audit Involvement - Potential Audit Types

Project Risk Review
System Infrastructure Assessments
High level Design Review
Data Conversion reviews
Integration Testing Reviews
Detailed Business Process Assessment
Go-Live Readiness
Post-implementation Audit
Post Mortem review
19

Potential Scope Areas - High-Level Project Risk

Review
Can be performed at various stages of the implementation or proactively
throughout the project and should include both IT and financial auditors
Auditors should be granted a seat at the table and participate in ongoing
project status meetings and proactively vet control concerns
Some areas to monitor:

Critical success factors (i.e., Project Steering Committee, Dedicated

Project Mgr.)

Risk Management (i.e., risks quantified, contingencies, frequent

management reporting)

Requirements Management
Project Management
Quality (i.e., peer review inspections, Quality Assurance Plan)
Configuration Management
Organization/Staffing
Supplier/Sub-Contractor Management
20

Potential Scope Areas - System Infrastructure

Assessment
Can be performed as soon as the system Infrastructure is
built
Typically performed by IT Auditors
Will cover many of the key IT SOX controls
Review can focus on key controls within the components
of the system architecture
Some Potential Scope areas include:

Application Controls Modules (i.e., PS PeopleTools,

SAP Basis)

The operating systems and DB platforms (UNIX, LINUX,

Windows, z/OS, IOS, ANDROID, SQL, Oracle, DB2)
21

Potential Scope Areas - System Infrastructure

Assessment (Cont.)
Potential Scope areas (Cont.)

Administrative transaction access and control

Operating system and database level security testing (using
scanning tools and scripts, where possible)
Middleware Components (i.e., MQ, CORBA, DCE, Encina, BEA,
Weblogic)
System General controls including,
Backup/Recovery
Security Administration
Security Policy
Capacity Planning
Support and Escalation Processes and
System/Performance Monitoring
Change Controls
22

Potential Scope Areas - High-level Design

Review
Can be performed as soon as the design documents
are completed (up to a year prior to the
implementation)
Team composition should include financial/operational
and IT auditors
Most of the work will be conceptual and will be based
on the documentation in place
Can cover managements plans for implementing
security and segregation of duties controls
Can cover planned implementation of key configurable
controls supporting the critical business processes
23

Potential Scope Areas - High-level Design

Review (Cont.)
Review of to be business process flows
Although not typically part of a business process review,
can include a checkpoint of the key project life cycle
activities, including testing, data conversion, training, and
contingency planning
Systems documentation
Planned interfaces (include plans for assessing data
integrity controls)

Must include caveat that due to the timing of the review, several aspects of the
project that were still in progress or under development could be not be
assessed in their entirety. Should give SPECIFIC examples of items not covered.

24

Potential Scope Areas - High-level Design

Review (Cont.)
Information Gathering Analysis/Review

Review business impact documentation and

proposals used to sell Sr. Management
Interview project management personnel
Review project charter
Review project plans
Interview systems development personnel
Interview infrastructure groups
Interview end-users
Obtain vendor system administration, configuration,
programming and user manuals
25

Potential Scope Areas - Data Conversion

Reviews
Timing Can be 3-4 months prior to go live or as soon as data is
converted
Can include IT and Financial auditors
Covers procedures and controls surrounding the data conversion
Confirm whether documentation has been retained on the
conversion process, including:
The file conversion plan (as outlined in the previous section);
Actual results (both before and after) and the reconciliation
involved
Problem logs and resulting actions
Test scripts and supporting files (e.g., Extract files, load files)
Evidence of user sign-off
26

Potential Scope Areas - Data Conversion,

Reviews (Cont.)
Were procedures outlined in the file conversion
plan followed?
In order to determine whether conversion was
satisfactory:
Evaluate the reconciliation between data files held on
the old and new systems
Ensure that all differences were investigated

Can re-perform data conversion items for accuracy

and completeness

27

Potential Scope Areas - Integration Testing

Review
Should be performed 2-3 months before go live
Review documentation of test results for completeness and
level of detail, including expected and actual results, for a
sample of test scripts
Review management of testing problems and related
processes
Review level of business user involvement in scope, execution
and approval of integration testing
Good time to evaluate users knowledge of the system
Can cover system performance including response time (by
trans type)
Usability Testing Final check to determine how well the
system meets the users needs
28

Potential Scope Areas - Detailed Business

Process Assessment
Can be performed 1-2 months before go live and include
financial and IT auditors
Should cover key financial SOX controls
Timing is highly dependent on how well the implementation
is going
Review will assess the current as-is and the future to-be
business process flows and will evaluate the manual and
automated controls within these processes
The processes under review should be based on RISK
Should document the critical business processes and
interfaces chosen for review and the rationale for choosing
them.
29

Potential Scope Areas - Detailed Business

Process Assessment (Cont.)
Should evaluate the critical interfaces for the following:
Completeness controls (automated file-level or manual
reconciliation reporting)
Accuracy controls (data validations)
Security (including temporary file locations)
Error-handling procedures
Ownership
Completeness of interface testing documentation
acceptance testing and integration testing
The presence of negative control testing
Handling of problem tickets that resulted from testing.
30

Potential Scope Areas - Detailed Business

Process Assessment (Cont.)

Depending on nature, timing and extent of testing, a

specific control or report could be tested by:
Inspection of system configurations
Inspection or re-performance of reconciliations with
supporting details
Re-performance of the control activity using system data.
Inspection of user access listings
Execution of SOD tools
Re-performance of control activity in a test environment.

31

Potential Scope Areas - Detailed Business

Process Assessment (Cont.)
Common Application Controls
Input and access controls
Data checks and validations
Automated authorization, approval, and override
Automated SOD
Pended items

File and data transmission controls

Checks for completeness and validity of the content including data size,

date and time, volume of records and authentication of the source

Possible Tests include:

Observe transmission reports and error reports
Observe validity and completeness parameters and settings
Review the access to set and amend the configurable parameters

32

Potential Scope Areas - Go Live Readiness

Can occur up to 1 month prior to go live and include IT & Financial
Auditors
Should include an assessment of:
Contingency Planning
High-level and detailed plans for each supported business group
or entity
Process during planned downtime (if any)
Business sustainability should extended downtime occur
Communication of plans to end users
Training
Completeness of training
User satisfaction with training
Management should track who goes to training
Good tool for audit group to learn about the system

33

Potential Scope Areas - Go Live Readiness

(cont.)
Integration Testing
Documentation of test results for completeness and level of detail,
including expected and actual results, for a sample of test scripts
Management of reported problems and the related processes
Are problem fixes appropriately retested
Volume Testing
Scope, method, and summary results of volume testing performed
for transactional and batch processing
Post-Production Support
Organizational structure
Procedures
Coverage times
Communication to end users
Always include caveat of what was not covered
34

Potential Scope Areas - Post Implementation

Audit
These should cover the most critical business processes
and interfaces reviewed as part of the detailed business
process pre-implementation review
Team composition should include financial & IT auditors
Can confirm that the controls that were reviewed in concept,
were implemented as planned
Includes testing of key controls
Includes transactional testing
Follow-up on findings from the pre-implementation audits
and confirm that there were no changes in the areas
previously tested
35

Potential Scope Areas - Post Mortem Reviews

Can include financial & IT auditors, although it is typically performed by
PMO type function
Post-mortem reviews focus on:
Cost/ROI
Schedule, and
Quality metrics
Examining specifications and expectations of project deliverables
The review process enhances:
IT support
Goals
Communication for future projects
Benefits include less project failures, increased
project quality, reduced costs, an accelerated
learning process and improved project management
36

Questions???

37

Thank You !!!

Andrew Ellsweig, CPA, CGEIT

Director
RSM McGladrey, Inc.
212.372.1810
andy.ellsweig@mcgladrey.com

38