Copyright 2005 Information Systems Audit and Control Association. All rights reserved. www.isaca.org.

IT Performance Improvement With COBIT and the SEI CMM

By Debra Mallette, CISA, CSSBB, SEI CMM and CMMI Assessor, and Managed Change Master, and Monica Jain, CSQA
he promise of business performance improvement is millions of US dollars in annual cost savings and product quality improvements. The risk is that even a company that dramatically improves its efficiency and product quality can fail miserably. The challenge is that many executives realize that radical changes are needed, but they have to increase todays level of performance while making those changes. Improving the performance of the IT organization can accelerate the business to greater performance. That is the promise of improving IT performance. The risk when the IT organization does not deliver is that the whole organization suffers, losing assets, sales, customers and momentum, or even goes under. Improving and sustaining business performance requires resources. These are indirect, overhead or risk mitigation costs and show up on the businesss income statement as overhead. The IT organization represents a significant part of this overhead. With constant pressure to decrease overhead, the IT organization represents an attractive target for cost reduction. At the same time, almost all of ITs operational and maintenance costs sustain performance and mitigate risks. Sustaining current performance, continuously reducing costs, decreasing exposure to risk and safely improving performance are the IT challenges. standard for IT. Both incorporate continuous improvement principles, so that by putting the processes and controls in place, the foundation is built for continuously improving performance, reducing costs and decreasing risk over time. When most business people think performance improvement, they generally think Six Sigma. General Electric Corporations highly publicized Six Sigma implementation program has been attributed with myriad performance improvements reported in the companys annual report. Six Sigma is not a maturity model; rather, Six Sigma refers to a numeric description of variation, statistical methods that bring processes under measurable control applied in business process improvement projects, and a strategic program to drive businesswide performance improvement to the bottom line. Perhaps less publicized is that Six Sigma implementation experts recommend not attempting a Six Sigma improvement program until the foundation maturity level 3 is laid (i.e., processes are defined and used repeatedly). COBIT and the SEI CMM can be used to bring the business and IT to a level of maturity and performance that can be accelerated toward more improvement using Six Sigma.

The Decision
Deciding which model to use is a classic cost-benefit tradeoff. Each maturity model has a particular focus of control from which the improvement benefit is derived. The SEI CMM focuses on practices that bring software engineering, such as IT application software engineering, under control while COBIT processes are aimed at a broader range of IT practices. A decision matrix mapping benefits, costs and alternatives clearly communicates the trade-offs and can be used in making the decision and in implementation communications. Figure 1 is a decision matrix showing evaluation criteria and maturity model options for SEI CMM, COBIT or the two combined. The SEI CMM is comprised of five levels of maturity (figure 2). Each level is a conceptual step or stage of process definition resulting in control, effectiveness and efficiency in producing software. The starting point is initial (maturity level 1), an ad hoc approach. Progression is expected through the repeatable (maturity level 2), defined (maturity level 3) and managed (maturity level 4), culminating in the optimizing (maturity level 5) level. The key practice areas (KPAs) are grouped by level. To be assessed as repeatable, the organization must implement all of the level 2 KPAs and show evidence of having met the goals and objectives for those practices. The level 2 KPAs shown in figure 2 are: requirements management, software project planning, software project tracking and oversight, software subcontract management, software quality assurance and software configuration

Maturity Models and Performance Improvement

Maturity models are guidelines for process performance improvement and can be applied by business and IT. Adopters of maturity models claim results that include significant reduction in defects, reduced project cycle times, increased productivity, improved employee satisfaction, increased customer satisfaction, reduced costs and reduced exposure to risks. These results are reproducible using the model to assess the current state and compare it to that of the industry (best-inclass) to identify opportunities. Implementation costs are controllable and the costs, risks and optimum methods for implementation can be learned from the experiences of others. The Software Engineering Institutes Capability Maturity Model (SEI CMM) for software engineering has been widely publicized and adopted, particularly by large engineering organizations serving the US government. The IT Governance Institutes Control Objectives for Information and related Technology (COBIT) is generally accepted as the de facto

INFORMATION SYSTEMS CONTROL JOURNAL, VOLUME 3, 2005

Figure 1Maturity Model Options

Decision Criteria Organizational fit Size of target population and model complexityimpact on communication, learning and use SEI CMM Excellent for software engineering, including IT software engineering Dependent on size of software engineering population within IT. Assessment and gap analysis take weeks, and implementation takes months. Quality focus may be difficult for software engineers who are focused on art rather than discipline. Cost of quality and quality language may be less understandable for IT management. Key process areas are grouped for implementation by maturity level. The integral processes foundation is synergistic for later implementation. Quality audits are synergistic with internal audit needs. SEI CMM and COBIT Depends on software engineering population size Dependent on size of IT Synergy: COBIT can direct organization. Documentation is SEI CMM investment for concise and readable by IT most benefit. SEI CMM professionals and IT management. target can be limited to Assessment and gap analysis take software engineering in days, and implementation takes context of COBIT for IT weeks or months. It may be organization. SEI CMM less understandable for those experience reduces the unfamiliar with language investment required of controls. to use COBIT. Internal audits operational Similarity of practices metrics can be leveraged enables leverage of to identify and target implementation guidance improvement opportunities. across both models. Key goal indicators and performance indicators supplement SEI CMM, and SEI CMM integral practices supplement COBIT implementation guidance. COBIT Excellent for IT

Synergies across practices and within organization

Decision Criteria: Organizational Fit

Figure 2SEI CMM MAturity Levels
Optimizing (5)
Process change managementPCM Technology change managementTCM Defect preventionDP Software quality managementPR Quantitative process managementQPM Peer reviewsPR Intergroup coordinationIC Software product engineeringSPE Integrated software managementISM Training programTP Organization process definitionOPD Organization process focusOPF Software configuration managementSCM Software quality assuranceSQA Software subcontract managementSSM Software project tracking and oversightPTO Software project planningSPP Requirements managementRM

Managed (4)

Defined (3)

Repeatable (2)

Initial (1)

The value of any model in driving performance improvement depends on whether the model will identify opportunities for improvement appropriate to the organization. COBIT can be used to identify weaknesses and opportunities for improvement in efficiency, effectiveness, confidentiality, integrity, compliance and reliability. COBIT can also be used to optimize management of people, applications, technology, facilities and data. These are IT opportunities. Implementing SEI CMM KPAs delivers improvements in effectiveness and efficiency of people, applications and technology. There are few SEI CMM references to processes or goals to assure confidentiality or data integrity. Security, business continuity and disaster recovery risk mitigation practices are largely missing from the SEI CMM.

management. To be assessed as defined, all of the KPAs for level 3 must be implemented with evidence of having met the goals and objectives for those areas, as well as continuing to meet the goals and objectives for level 2 KPAs.

Decision Criteria: Size of Target Population and Model Complexity

Performance improvement is fundamentally organizational change to orchestrate people using processes and technology. The constraint in organizational change is on the ability of every person to know, understand, believe and do/manage the work using the improved processes. These are the targets of the change. Management must sponsor the change, change agents lead the change and targets make the change. Target, historical and cultural risks must be mitigated. The larger the target population, the more costly and risky the change. The target population for the SEI CMM in IT is the application software engineers, their managers and IT managementa subset of the IT organization. It has fewer targets than COBIT, and fewer people are required to understand the model. This is good, because the size and intricacy of the SEI CMM are intimidating. The focus is on precision and thoroughness of the assessment process; assessments can take a week or more. Assessors are expected to be trained and certified.

COBIT
COBIT comprises 34 IT processes organized into four domains: Plan and Organize, Acquire and Implement, Deliver and Support, and Monitor and Evaluate. The COBIT management guidelines contain the maturity model, process description, information criteria and IT resources, which indicate the improvement potential, critical success factors, key goal indicators and key performance indicators for each process. The COBIT framework, detailed control objectives and audit guidelines improve the IT organizations level of control, mitigating risks and sustaining performance. The management guidelines can be used with this knowledge base and COBIT Online benchmarking to prioritize and guide improvement.

INFORMATION SYSTEMS CONTROL JOURNAL, VOLUME 3, 2005

Gap analysis takes weeks, implementation takes months, and there may be a lag after implementation before seeing results. The quality focus may be difficult for software engineers concerned with creativity rather than discipline, and the cost of quality language requires management interpretation to derive financial performance expectations. The target population to understand COBIT is the entire IT organization and management, including business management. Fortunately, the COBIT documentation has been designed to address the needs of each of the target populations: Executive Summary for business management, Management Guidelines for IT management, Control Objectives for process implementers and Audit Guidelines for auditors. The entire COBIT 3rd Edition package consists of fewer than 500 pages. It is clear and concise and takes a relatively short period of time to read and comprehend. Self-assessments for prioritization and gap analysis take days, implementation takes weeks and results are almost immediately apparent. A drawback for management is the emphasis on control and risk mitigation rather than performance opportunity. COBITs language of control is also not generally well understood by engineering organizations without previous exposure to audit or financial controls. Creating awareness of the need for controls and risk mitigation in those not familiar with the concept can get in the way of understanding the value.

project tracking and oversight, quality management, audit, training, process documentation, configuration and change. These resulted in high-level correlation (see figure 3). The second pass was for more inclusive correlation based on similarities in the activities intent and goals, and the third pass examined the potential for fulfilling COBIT detailed control objectives using SEI CMM practices. Figure 4 shows the information graphically. The KPAs are sorted by their SEI CMM level (shown in figure 2). The count totals are shown in the bars, with a total bar for each capability maturity model level 2, level 3, and levels 4 and 5 (left X-axis) correlated to each COBIT process (Y-axis) with the percent coverage (right Y-axis) of COBIT detailed control objectives superimposed.

SEI CMM Best Practice Guidance for COBIT Processes

COBIT is best used to decide what and how much to improve the IT processes, while best practice models such as the SEI CMM provide better guidance for how to implement the improvements. There is a greater depth and precision of guidance available from the SEI CMM for control of software engineering. Figure 5 shows the COBIT processes that will receive the most benefit from SEI CMM guidance. For example, PO11 manage quality benefits from KPAs, including level 2 KPA software quality assurance and level 4 KPA software quality management and quantitative process management.

Decision Criteria: Synergies

Synergies are opportunities to leverage processes to reduce costs and risks. The SEI CMM key practice areas identified as integral practice areas are designed to be enablers for the functional practice areas with which they are staged and higher levels of maturity. Functional practice areas in the set of maturity level 2 KPAs include requirements management, software project planning, project planning tracking and oversight, and software subcontract management. Integral practice areas are software configuration management and software quality assurance. Integral practice areas are enablers and risk mitigators for the functional practice areas and the higher levels of maturity. For example, software configuration management, an integral practice area, is an enabler for requirements management, enabling requirements traceability and mitigating risks to software quality by controlling changes. SCM also enables software product engineering, a level 3 KPA. Software quality assurance, which includes software audits, is critical to assuring that all the level 2 KPAs remain under control, and is an enabler for bringing new KPAs under control. Control for the new KPAs is assured by expanding the scope of the audits. COBIT leverages internal audit to sustain the process controls and give guidance on what to improve to identify opportunities. In addition, the COBIT processes are organized into planning, doing and monitoring based on the plan-docheck-act continuous improvement cycle. A correlation analysis of SEI CMM and COBIT overlap helps in understanding the possible synergies between the models. This analysis was performed using a three-pass approach. The first pass established a high-level mapping of COBIT processes to SEI CMM KPAs to the practice and goal level. The COBIT process descriptions were compared with SEI CMM KPAs grouped by maturity level, looking for significant matching process words, such as technology, project plan,

Using COBIT and SEI CMM to Lead Process Improvement

Recommended steps include: 1. Identify opportunities for improvement. The opportunities could be identified by looking at internal audit findings mapped to COBIT control objectives and a COBIT assessment and/or benchmark. 2. Evaluate the expected benefit from the improvement. The COBIT key goal indicators and the why do it statement from COBIT can be used if process measures are not already available. 3. Use correlations and mapping of SEI CMM key practices to COBIT control objectives to identify control objectives that are met and strengthened using SEI CMM practices. COBIT control objective correlation to SEI CMM practices indicate where the SEI CMM KPAs have a higher probability of giving more accurate and precise guidance than using COBIT alone. 4. Decide whether the expected benefit justifies a full SEI CMM assessment if not already available. 5. Base the implementation strategic and tactical plans on assessed opportunities linked to best practices. Set goals and milestones to reach an IT-wide balanced maturity level. Establish priorities for processes to improve based on the desired improvement and the planning and monitoring processes to create the feedback loops foundational to sustaining performance and generating additional opportunities. As an example of step 5, use maturity level 2 as the target. The IT organization goal is to sustain performance and realize improvement with the minimum investment beginning with the SEI CMM staged guidance. The IT organization implementation plan should address AI1 identify automated solutions, AI2

INFORMATION SYSTEMS CONTROL JOURNAL, VOLUME 3, 2005

Figure 3Correlation Matrix: COBIT With Correlation to SEI CMM KPAs

COBIT Processes SEI CMM KPAs Correlated to COBIT Through Activity and Intent Plan and Organize IC TCM TCM IC TCM OPF,OPD, IC, TCM,SSM TCM PCM SEI CMM KPAs High-level Correlation COBIT Detailed Control Objectives Fulfilled 5 of 8 0 of 4 4 of 5 6 of 15 1 of 3 6 of 11 0 of 8 1 of 6 6 of 8 14 of 14 16 of 19 Percent of Percent of KPAs to COBIT Fulfilled With COBIT SEI CMM 63% 0% 80% 40% 33% 55% 0% 17% 75% 100% 84% 11% 0% 6% 28% 6% 6% 0% 6% 17% 28% 33%

PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 PO10

Define a strategic plan Define the information architecture Determine technological direction Define the IT organization and relationships Manage the IT investment Communicate management aims and direction Manage human resources Ensure compliance with external requirements Assess risks Manage projects

PO11 Manage quality

AI1 AI2 AI3 AI4 AI5 AI6

Identify automated solutions Acquire and maintain application software Acquire and maintain technology infrastructure Develop and maintain procedures Install and accredit systems Manage changes

RM SPP, ISM SPP, PTO, ISM SPP, PTO, SPP, PTO, ISM, ISM SQA, SPE SQA, OPF, SQA, OPF, SQM, SQM, TP, ISM TP, ISM, QPM Acquire and Implement RM, TCM RM, SPE, TCM SPE, SSM, SCM SPE, SSM, SCM, RM SCM, TCM, PCM SCM, TCM, SSM ISM, OPF, OPD OPF, OPD, PCM, SPE, ISM SPE SPE, ISM SCM, PCM, SCM TCM Deliver and Support SSM SSM SPP,ISM SPP, PTO OPD, TP SCM DP SPP, PTO, ISM SPP, PTO, ISM OPD, TP,SPE SQA SCM DP SPP, PTO, ISM

4 of 18 6 of 17 3 of 6 3 of 4 6 of 14 5 of 8

22% 35% 50% 75% 43% 63%

17% 22% 17% 28% 11% 6%

DS1 DS2 DS3 DS4 DS5 DS6 DS7 DS8 DS9 DS10 DS11 DS12 DS13 M1 M2 M3 M4

Define and manage service levels Manage third-party services Manage performance and capacity Ensure continuous service Ensure systems security Identify and allocate costs Educate and train users Assist and advise customers Manage the configuration Manage problems and incidents Manage data Manage facilities Manage operations Monitor the processes Assess internal control adequacy Obtain independent assurance Provide for independent audit Defect prevention Intergroup coordination Integrated software management Organization process definition Organization process focus Process change management PTO: QPM: RM: SCM: SPE: SPP:

IC Monitor and Evaluate QPM, PCM QPM, PCM SQA SQA SQA, PR SQA, PR,SSM SQA Legend: SEI CMM KPAs Used in Correlation Matrix Project tracking and oversight Quantitative process management Requirements management Software configuration management Software product engineering Software project planning SQA: SQM: SSM: TCM: TP:

0 of 7 6 of 8 0 of 9 3 of 13 0 of 21 3 of 3 2 of 3 2 of 3 6 of 8 3 of 5 3 of 30 0 of 6 0 of 8 4 of 4 3 of 4 6 of 8 4 of 8

0% 75% 0% 23% 0% 67% 67% 67% 75% 60% 10% 0% 0% 100% 75% 75% 50%

0% 6% 0% 11% 0% 17% 17% 6% 6% 6% 17% 0% 0% 11% 6% 17% 6%

DP: IC: ISM: OPD: OPF: PCM:

Software quality assurance Software quality management Software subcontract management Technology change management Training program

INFORMATION SYSTEMS CONTROL JOURNAL, VOLUME 3, 2005

Figure 4COBIT and SEI CMM Correlation

SEI CMM KPA Count
4.5 35%

Figure 5COBIT and SEI CMM KPA By Level

7 6 5 4 3 2 1 0 AI1 AI2 AI3 AI4 AI5 AI6 M1 M2 M3 PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 P10 PO11 DS1 DS2 DS3 DS4 DS5 DS6 DS7 DS8 DS9 DS10 DS11 DS12 DS13 M4

4 30% 3.5 25% 3 20%

COBIT Process

KPA Count

2.5

SEI CMM Level 2 KPAs

2 15%

SEI CMM Level 3 KPAs

SEI CMM Level 4 and 5 KPAs

1.5 10% 1 5% .5

Summary
Sustaining current performance while continuously reducing costs, decreasing exposure to risk and carving out resources to safely improve performance from a budget constantly targeted for cost reduction is the IT challenge. Maturity models can tell where there are opportunities to improve the organizations performance. By using a maturity model, the organization can safely and predictably reproduce the performance improvement results of others with confidence in the approach and the expected expenditure of resources and benefits to be derived. Using any model requires an investment in learning, assessment and implementation. Best practice maturity models tell how to attain the improvements with the most precision and accuracy and may require more investment because of their detailed and specialized guidance. Model synergies, including continuous improvement practices, leverage sustaining costs for higher returns. Using COBIT with SEI CMM combines the best of both worlds to improve IT performance and drive the results to the business bottom line.

SEI CMM Level 2 KPAs

SEI CMM Level 3 KPAs

SEI CMM Level 4 and 5 KPAs

acquire and maintain application software, PO11 manage projects, PO10 manage quality and DS2 manage third-party services. The integral processes to sustain performance are AI6 manage change and DS9 manage the configuration. Additional COBIT planning and monitoring processes to sustain performance and generate additional opportunities are PO1 define the strategic plan, emphasizing capability improvement planning, and M1 monitor the processes, so that the organization recognizes the expected process performance improvement results from the capability maturity improvement projects. M2 assess internal control adequacy also sustains the performance and generates information that can be leveraged to identify additional opportunities.

References Case Study

An assessed SEI CMM level 3 engineering organization was supported by an engineering IT organization self-assessed with COBIT maturity levels averaging at level 2 and ranging to level 4. The engineering IT organization found that less than 6 percent of its resources were applied in those COBIT processes most highly correlated with the SEI CMM. The project list and allocation of resources to projects accounted for approximately 50 percent of the resources. The organization was familiar with capability maturity models and adapted readily to using COBIT to guide its capability improvement strategy. The management team performed the self-assessment and put an improvement program in place with goals to improve PO1, M1 and asset management (not a COBIT process). While continuing to drive for capability improvement, investing 2.5 full-time-equivalent employees (FTEs) with approximately US $5,000 in travelrelated expenses, the IT staff was cut in half, capital expenditures were eliminated, and the expense budget was cut by 60 percent. Through these reductions, the IT organization was able to maintain ISO 17999 compliance, a satisfactory level of internal control compliance, and performance within service level agreement targets for 3.5 to 4.5 out of five 9s availability, service request cycle time average closure of less than 24 hours and customer satisfaction of very satisfied. Harry, Mikel, Ph.D.; Richard D. Schroeder; Six Sigma: The Breakthrough Strategy Revolutionizing the Worlds Top Corporations, Random House, 1999 IT Governance Institute, COBIT Management Guidelines, COBIT Framework and COBIT Control Objectives, 2000 IT Governance Institute, COBIT Online, www.isaca.org/cobit Keen, Peter G. W.; The Process Edge: Creating Value Where It Counts, Harvard Business School Press, 1997 Kimpton, Clarence; Denys Martin; Overview of Principal IT Evaluation Models: Tools for IT Auditors, Information Systems Control Journal, vol. 5, 2001 LaMarsh, Jeanenne; Changing the Way with Change: Gaining Control over Major Organizational Change, Addison-Wesley Publishing, 1995 Martin, James; The Great Transition: Using the Seven Disciplines of Enterprise Engineering to Align People, Technology, and Strategy, American Management Association, 1995 Paulk, M.C., et al; Capability Maturity ModelSM for Software, CMU/SEI-93-TR-24, Carnegie Mellon University, Software Engineering Institute, USA, 1993

INFORMATION SYSTEMS CONTROL JOURNAL, VOLUME 3, 2005

Debra Mallette, CISA, CSSBB, SEI CMM and CMMI Assessor, and Managed Change Master is a process program manager for a large healthcare IT organization. Her experience ranges across industries and organizations. She has been published and has presented at the Motorola Software Engineering Symposium and SEI CMMs SEPG. Her specialty is strategic capability improvement for enterprises making the transition to the information age. She can be contacted at debra.mallette@kp.org.

Monica Jain, CSQA is a process consultant at Covansys Corporation, USA, specializing in technology and business consulting. Her areas of interest include implementation of CMM, CMMI, ITIL, and conducting audits and assessments. She has also cleared the ITIL Foundation Certification examination conducted by EXIM UK. She can be contacted at mjain@covansys.com or monica_j18@yahoo.com.

Information Systems Control Journal, formerly the IS Audit & Control Journal, is published by the Information Systems Audit and Control Association, Inc.. Membership in the association, a voluntary organization of persons interested in information systems (IS) auditing, control and security, entitles one to receive an annual subscription to the Information Systems Control Journal. Opinions expressed in the Information Systems Control Journal represent the views of the authors and advertisers. They may differ from policies and official statements of the Information Systems Audit and Control Association and/or the IT Governance Institute and their committees, and from opinions endorsed by authors' employers, or the editors of this Journal. Information Systems Control Journal does not attest to the originality of authors' content. Copyright 2005 by Information Systems Audit and Control Association Inc., formerly the EDP Auditors Association. All rights reserved. ISCATM Information Systems Control AssociationTM Instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. For other copying, reprint or republication, permission must be obtained in writing from the association. Where necessary, permission is granted by the copyright owners for those registered with the Copyright Clearance Center (CCC), 27 Congress St., Salem, Mass. 01970, to photocopy articles owned by the Information Systems Audit and Control Association Inc., for a flat fee of US $2.50 per article plus 25 per page. Send payment to the CCC stating the ISSN (1526-7407), date, volume, and first and last page number of each article. Copying for other than personal use or internal reference, or of articles or columns not owned by the association without express permission of the association or the copyright owner is expressly prohibited. www.isaca.org

INFORMATION SYSTEMS CONTROL JOURNAL, VOLUME 3, 2005