An integrated evaluation system based on the continuous improvement model of IS performance

Choon Seong Leem and Injoo Kim

1. Introduction
Recently, the importance of Information Systems (IS) is being rapidly increased as a key strategic mean promoting the efficiency of enterprise activity. According to dramatic progresses of info-technology, the typical users of IS are expected to use various applications in dynamic enterprise environments. Furthermore, most enterprises pursue the renovation of business process and strategies through IS. In order to adequately response these trends, enterprises have to establish comprehensive concepts and goals based on evolutionary characteristics of IS and to identify their objectives from the continuous evaluation of current IS conditions by a scientific and systemic methodology. In spite of these needs, previous researches have not been very successful for identifying accurate problems based on the evaluation of performance maturity of IS. Additionally, as they dealt with conceptual evaluation frameworks and focused on fragmentary evaluation areas, it is difficult to figure out the detailed evaluation results of IS for the entire scope of enterprise. This paper examines the evaluation issues of enterprise IS performance dealing with: (1) suggestions for the performance improvement model based on the evolutionary characteristics of IS; (2) development of an integrated evaluation system based on the improvement model; and (3) verification of efficiency and applicability of the evaluation system through a large-scale case study.

The authors Choon Seong Leem is a Professor at the School of Computer and Industrial Engineering, Yonsei University, Seoul, Korea. Injoo Kim is a Research Fellow at the Naval Forces Development Command, Kyeunggi, Korea Keywords Continuous improvement, Performance management, Modelling, Information systems Democratic People's Republic of Korea Abstract In order to achieve competitive business goals, every enterprise needs to evaluate the current level of information systems performance and their utilization. The evaluation measures the technical capacity and operational capability of enterprise information systems and diagnoses their effectiveness in business goals and efficiency in resources. An integrated evaluation system is developed based on the continuous improvement model of information systems performance. The system has been applied to performance measurement of information systems with a huge set of data from Korean industries, and proven reliable and practical Electronic access The Emerald Research Register for this journal is available at www.emeraldinsight.com/researchregister The current issue and full text archive of this journal is available at www.emeraldinsight.com/0263-5577.htm
Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . pp. 115-128 # Emerald Group Publishing Limited . ISSN 0263-5577 DOI 10.1108/02635570410522080

2. Previous research
This work focuses on improvement of IS performance by systematic evaluation methodology. Previous researches can be classified into two types regarding improvement models and evaluation models of IS performance. Also, the researches related to the evaluation models concern three kinds of topic which are evaluation model, evaluation fields, and evaluation items of IS performance.

115

An integrated evaluation system

Choon Seong Leem and Injoo Kim

Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . 115-128

2.1 Previous researches on the improvement model of IS performance There are two types of researches related to improvement of IS performance. The one is on improvement processes and the other is on improvement stages of IS performance. Plan-Do-Check-Act (PDCA), InitiatingDiagnosing-Establishing-Acting-Learning (IDEAL), and Quality Improvement Paradigm (QIP) are typical researches on improvement process. The PDCA initialized by Shewhart (1931) and generalized by Deming (1986) after the Second World War is the improvement process of product quality based on feedback cycle that can optimize unit production process. The QIP by NASA Software Engineering Laboratory is the improvement process of software quality based on the meta-lifecycle model to improve long term quality. This process has several functions; packing, assessing, and increasing comprehension of development experience for software. The IDEAL by Software Engineering Institute (SEI) in the Carnegie Mellon University is the process improvement model focused on project management. This model is composed of five steps that are continuously and recursively performed (McFeeley, 1996). The kaizen model to improve the process performance has been applied to the ESPRIT project. The basic concept of this model is called ``adoption curve'' to take up new technology which is proposed by Conner and Patterson. Table I briefly summarize these researches (see Renaissance Consortium, 1997)
Table I Researches on important processes of IS performance Title PDCA QIP Improvement processes Plan do check act Characterize the environment set goals choose and tailor a process model execute the process analyze the collected data learn and feedback Focus Product quality improvement S/W quality improvement

Also, there are several researches on improvement stages of IS performance. Nolan and Wetherbe (1980) suggested six maturity stages of IS focused on data, and Venkatraman (1997) also proposed a five stage model focused on structure innovation of organization by IS. Vernadt (1996) presented a three stage model of systems integration according to expansion of the Computer Integrated Manufacturing (CIM) integration range. The Capability Maturity Model (CMM) by SEI is composed of five stages derived from the degree of process maturity (Bate et al.,1995). The Information Systems Management (ISM) model by Tan (1999) is based on balance between organizational structure and IT components. This model is originated from MIT90s framework that is composed of the levels of IT-enabled business reconfiguration by Venkatraman. In this model, IS fields are divided into three parts which are external environments, organization environments, and IS environments. Table II shows the researches related to improvement stages of IS performance. 2.2 Previous researches on the evaluation model of IS performance The evaluation diagnoses the current condition, and utilizes its results for future plans, so that the organization could get the better performance. For instance, the Japanese Deming prize, USAs Malcolm Baldrige Award called ``criteria for performance excellence'', and European's ``Business Excellence model'' are known to significantly contribute to quality improvement of products and process. Also, the
Table II Researches on improvement stages of IS performance Title Nolan CIM ISM Improvement stages Initiation contagion control integration data administration maturity Physical system integration application integration business integration Functional integration cross-functional integration process integration business process redesign business redesign or business scope redefinition Performed informally (initial) planned and tracked well-defined quantitatively controlled continuously improving Focus Data System Business

IDEAL

Process improvement Initiating diagnosing establishing acting leveraging or learning

New technology adoption Kaizen Contact awareness understanding evaluation trial use adoption institutionalization

CMM

Process

116

An integrated evaluation system

Choon Seong Leem and Injoo Kim

Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . 115-128

USA, the UK, Japan and OECD are continuously working out the national IS indices, so as to gradually increase the level of IS performance (Jeong, 1996). In the research related to the evaluation model of IS, the DeLone and McLean's (1992) IS success model based on the works by Shannon and Weaver (1949) and Mason (1978) is well known. This model is examined and improved by Seddon and Kiew (1994) which suggests the measures of six fields and proves their appropriateness. Since the IS model did not cover the appropriate measures coincided with the characteristics of organization, Saunders and Jones (1992) developed the ``IS function performance evaluation model'' which encompasses a selection method of appropriate measures corresponding to organization features. Myers et al. (1997) worked out the ``Comprehensive IS assessment model'' that expanded the six evaluation fields of DeLone and McLean's model into eight fields and combined these fields into organizational and external environments. Also, Goodhue and Thompson (1995) and Goodhue(1998) proposed the Task-to-Performance Chain (TPC) model based on a fitting technique into individual performance. The focus of the model is to apply the technique to individual tasks to calculate their positive impact on individual performance. Additionally, there are several researches related to IS framework. Tan (1999) suggested the ``Consistency model'' composed of seven components that expanded the MIT90s model and SEI also proposed a framework composed of seven evaluation fields (Bergey et al., 1997). As researches related to identification of the evaluation items, the Goal-Question(indicator)-Measures (GQM) methodology was introduced by Basili and Rombach (1988), refined by AMI (1992), Pulford et al. (1996) in ESPRIT project, and was applied to the goal-driven software evaluation by Park (1996) in SEI. Especially, Mendonca (1997, 1998) converted the GQM to another Goal-Question-Metric for improvement of evaluation processes. Sometimes researches related to the evaluation model of IS performance (see Table III).

2.3 Limits of previous works Although previous researches addressed IS framework, performance improvement, evaluation processes and evaluation items, but their theoretical models may not be easily applied to real situations. Their inherent limitations are as followed. First, while the improvement of IS performance has to be continuously pursued from IS strategy planning to construction and maintenance, previous researches have a tendency to workout only for limited parts of the whole. Second, although the evaluation model of IS performance put emphasis on identification of the evaluation items and investigation of their relationships, but they are not successful in embodying detailed and protocol evaluation procedures. Third, usual IS frameworks do not address the relationship between IS evaluations and their business implications. As a result, it is difficult to use the evaluation results directly to any related business decision-makings. In order to overcome these limits of previous works, this paper proposes an improvement model of IS performance and develops corresponding evaluation system which can estimate the current IS status and feedback the results into IS construction process. The improvement model is described in the following section 3 and the integrated evaluation system is explained in section 4. A case study with real data from 219 Korean leading companies in section 5 proves the practical values of the model and system.

3. The improvement model of IS performance

This paper defines the IS as integrated systems that collect data, analyze that, generate the new useful information, transmit it and use information related with business activities in organizations, typically business process in enterprises. ``The IS performance'' that is usually divided into several stages is defined as the degree of effectiveness and efficiency in business goal accomplishment by IS. The ``Improvement of IS performance'' implies that the IS

117

An integrated evaluation system

Choon Seong Leem and Injoo Kim

Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . 115-128

Table III Researches on the evaluation model of IS performance Category Evaluation model Researcher Shannon and Weaver Mason DeLone and McLean Barry L. Myers Key elements Technical level semantic level effectiveness or influence Production product receipt influence on recipient influence on system System quality information quality use user satisfaction individual impact organization impact Service quality system quality information quality use user satisfaction individual impact workgroup impact organization impact Task characteristics technology characteristics individual characteristics task-technology fits utilization performance impact Strategy structure technology people management IS strategy information systems users data and technical infrastructure IS organization IT specialists IS management Organization project legacy systems system engineering software engineering technology target systems Goal question indicator measures Capture data user's goals identify relevant entities identify relevant attributes map attributes to existing metrics

Goodhue and Thompson

IS framework

MIT90S Consistency model SEI framework

Identifying process of evaluation factors

GQM Mondonca's GQM

performance is improved to become flexibly commensurate with changes in internal and external environments and various requirements of users, so that the IS performance can be optimized with activities in organization. ``The improvement model of IS performance'' is a representation of their relationships. This paper presents the improvement model of IS performance, which consists of improvement stages and cycles. 3.1 Improvement stage of IS performance The improvement stage of IS performance plays major role in overall evaluation of IS performance. The improvement stage is suggested to consist of five stages and Figure 1 shows them. As shown in Figure 1, the five improvement stage of IS performance in this research are function integration, process integration, business integration, industry integration, and role-model generation. The level of the stage can be determined by the six comprehensive fields of IS performance which are vision, organization and institution, infrastructure, supporting, application, and usage of IS. The ``function integration'' represents to

computerize the individual tasks within isolated systems The ``process integration'' combines the individual processes and functions into corresponding working group via IS. The ``business integration'' is defined to integrate the working groups into the level of entire organization, and the ``industry integration'' should be cover up to partner companies and, individual customers, outside the organization. In the ``role model generation'' stage, the organization can flexibly accommodate to new external environment by itself and naturally create new business models by accumulated information and updated IS. The improvement stage of IS performance has important meanings that can quantitatively represent the current IS status and target IS status in future. Seeing that the IS environments have many diverse qualitative factors and these factors are tangled with each other, it is very difficult for organization to decide level of the stages for current IS status or target IS status. Therefore, in order to decide the stage correctly, these stages should be characterized and explained by various factors. This paper suggests these decision factors based on the IS framework that are divided into six fields; IS vision, IS organization

118

An integrated evaluation system

Choon Seong Leem and Injoo Kim

Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . 115-128

Figure 1 Five improvement stages of IS performance

and institution, Infrastructure, supporting, application, and usage. Table IV summarizes these factors. 3.2 Improvement cycle of IS performance The improvement model of IS performance in this paper consists of three components:
Table IV Decision factors for improvement stages of IS performance Classification IS vision Function integration No or low level ISP Low level project control Informally performed process Process integration Basic ISP Short-term construction plan Planned and tracked process

improvement stages, integrated evaluation system, and construction process, and should be applied by five continuous and circular cycles; initiation, goal establishment, diagnosis and evaluation of IS performance, construction process, and leveraging and learning. Figure 2 shows the cycle.

Business integration High level ISP Mid/long-term construction plan Fragmentary product management Well-defined process

Industry integration Circulatory ISP management Circulatory project management Enterprise product management Quantitatively controlled process

Role-model generation Circulatory ISP management Circulatory project management Enterprise product management Continuous creation of optimized process

IS organization and Basic computerization institution organization

Basic IS organization Basic IS rules, institutions, and directions Enhancement of IS education and expansion of IS minds Network for enterprise Platform for enterprise Integrated DB

Specialized and detailed IS organization Settlement of IS rules, institutions, and directions

Infrastructure

Network for units Stand-only system Platform for working Platform for each function (individual tasks) groups Standardization Development of simple application Computerization of individual tasks Office automation

Global network for companies and customers Platform for enterprise Integrated DB Maintenance of management supporting systems Automated reflection of user requirements EC, CALS, Knowledge sharing CRM, SCM New knowledge creation New business strategy creation

Supporting

Establishment of supporting systems Usage of CASE tools Enhancement of user services IS of working groups Data sharing Supporting of decision making ERP Information sharing MIS Mostly indispensable usage (enterprise) Controlled/self-sustained participation

Application

Usage

Partially indispensable Intermittent usage usage (group) (individual usage) Uncontrolled participation Controlled participation

Entirely indispensable usage Fully self-sustained participation

119

An integrated evaluation system

Choon Seong Leem and Injoo Kim

Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . 115-128

Figure 2 Improvement cycles of IS performance

As shown in Figure 2, the improvement of IS performance can be achieved by five processes. First, the motive to improve IS performance is triggered by stimulus originated from changes in internal and external environment. Second, the organization should establish the goal (IS vision) that can flexibly cope with the trends of IS environment. Third, the organization should evaluate the current IS status, identify future objectives, and analyze the gap through the comparison between goal states and current states. Fourth, detailed problems in current states should be considered in planning and construction of IS projects. Finally, information and knowledge acquired from previous processes should be utilized with recursive iterations of the cycle, the IS environments can be continuously reconciled with management environments of the organization. Figure 3 shows the detailed processes of improving IS performance in parallel with the IS construction steps; planning (ISP), analysis, design, construction, and maintenance.

decomposed into five steps; preparation, measurement, analysis, interpretation, and feedback. The evaluation fields which are originated from IS framework can be decomposed into three parts; measurement factors, influence factors, and evaluation factors. The measurement factors mean the static standpoint of IS framework, the influence factors mean the dynamic standpoint that represents the relationship between subject and object in IS framework, and the evaluation factors are considered to supply useful information to decision-makers. These factors are measured, analyzed, and interpreted by various evaluation methods. Figure 4 shows a schematic diagram of the integrated evaluation system of IS performance. 4.1 Evaluation procedures The evaluation of IS performance is to measure current IS performance, to analyze the measured data, to interpret the analysis result, and to return these outcomes to the improvement model and construction process, so that the current IS performance can be evolved into target stage. The evaluation procedure is defined as a series of sequential actions that should continuously perform these evaluating activities efficiently. This paper presents five practical steps to evaluate the IS performance; preparation, measurement, analysis, interpretation, and feedback which are originated from human problem solving processes. Also the order of evaluation procedures is made up by the relationship

4. The integrated evaluation system of IS performance

The integrated evaluation system of IS performance is designed to diagnose the current IS status, and identify the deficiencies of current status for target systems by gap analysis. This system consists of three parts; evaluation procedures, evaluation fields, and evaluation methods. The evaluation procedures can be

120

An integrated evaluation system

Choon Seong Leem and Injoo Kim

Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . 115-128

Figure 3 Detailed processes of improving IS performance

Figure 4 Integrated evaluation system of IS performance

between input and output of each steps. Although these steps could be decomposed by deeper, this paper introduces second sub-level of the steps. Figure 5 shows the evaluation procedures for IS performance. 4.2 Evaluation framework and perspectives In order to achieve business goals using IS, IS ought to be considered as a part of entire organization, and the evaluation framework of IS performance should be accurately defined with respect to the organization. This paper presents an integrated evaluation framework, as a part of improvement model of IS performance, that can comprehensively

represents entire IS environment as a part of real world. Figure 6 shows this framework. Figure 6 illustrates the integrated evaluation framework embodying all the measures of IS performance. This framework contains static and dynamic perspectives. The static perspective of IS environments consists of external environment, enterprise environment and IS environment while the dynamic perspective of improvement process implies the relationships between activities being continued from legacy IS (level n) to target IS (level n + 1) by various projects. In addition to the two perspectives, this paper considers the evaluation perspective of manager executing the project. Now the framework is extended with three perspectives that are measurement (static perspective), interpretation (dynamic perspective), and evaluation (evaluation perspective). ``Measurement perspective'' of IS framework views the components of IS framework as static ones points, so that organization can measure the level of IS assets and the degree of IS performance. In order to measure IS assets of organization, the IS framework in this standpoint should be gradually decomposed into measurement items. Figure 7 shows the IS framework with measurement perspective which is composed of three-level subject areas.

121

An integrated evaluation system

Choon Seong Leem and Injoo Kim

Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . 115-128

Figure 5 Evaluation procedures of IS performance

Figure 6 Integrated evaluation framework of IS performance

``Interpretation perspective'' of IS framework means the dynamic perspective regarding how the IS environment efficiently contributes to business activities of organization. The evaluation factors in this standpoint are relationships between subjects and objects in IS framework. Figure 8 shows the interpretation perspective of IS framework that is composed of capacity, capability, and performance. The capacity of IS means physical characteristics of components in the IS area such as speed of CPU and expertise of provider.

The capability means the ability to influence the business area with capacity. The performance denotes the realization of improvement of enterprise competitiveness via the IS environment. As shown in Figure 8, measurement process has to be orderly progressed from capacity to performance, and analysis process of these measurements has to be reversibly progressed. Figure 8 Interpretation perspective of IS framework ``evaluation perspective'' of IS framework means the classification of

122

An integrated evaluation system

Choon Seong Leem and Injoo Kim

Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . 115-128

Figure 7 Measurement perspective of IS framework

Figure 8 Interpretation perspective of IS framework

three hierarchical components; field, factor, and item. In the characterization structure, the evaluation fields are decomposed into evaluation factors with respect to the dimension subjects (who), objects (what), and relationships between three perspectives. Table VI summarizes the derived evaluation factors for each evaluation field. The evaluation items can be identified from the characteristics of evaluation factors, and these factors may be measured by measurable items to represent them. According to the characterization structure, this work identifies 345 measurable items which are consist of 46 items in IS vision, 80 items in IS infrastructure, 88 items in IS organization and institution, 22 items in IS supporting, 61 items in IS application, and 48 items in IS usage. 4.4 Integrated evaluation system An integrated evaluation system should have various functions that are data collection, measurement, analysis, interpretation, and feedback. Table VII represents these functions of the evaluation system. In order to pursue automation of these evaluation functions, the conceptual framework of integrated evaluation system as is designed by Figure 9.

evaluation items and factors to manager's interests, so as to help decision making. This perspective is utilized as a criterion to determine an improvement stage, and is decomposed into six fields; IS vision, IS organization and institution, IS infrastructure, IS supporting, IS application, and IS usage. Consequently, the manager or decision-maker can get the practical information using the evaluation perspective. Table V shows the descriptions of six fields of IS performance in this perspective. 4.3 Evaluation fields, factors, and items For characterizing of IS framework, its construction structure that converts the conceptual IS framework to measurable items should be defined holistically. Although the GQM method can be considered, it is difficult to identify input, output, and relationships between components. Thus, it is not easy for an evaluator to practically apply this method to real world. This paper presents the characterization structure of IS framework with

5. Case study
In order to verify adoptability and practicality of the proposed integrated evaluation system, a case study has been performed with the support from Korean Ministry of Information and Communication. The system has been applied to the 219 leading companies in Korea using the 345 measurable items in six fields of IS performance for two years from 1999 to 2000. According to the study, the enterprises could get many useful evaluation results of IS performance. Also, a company could find its relative status of IS performance and assets comparing with in the same industrial sector. With this information of evaluation, these enterprises have been excitedly motivated to improve their IS performance. The results of the case study in concert with the goals of this paper are briefed as followed.

123

An integrated evaluation system

Choon Seong Leem and Injoo Kim

Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . 115-128

Table V Descriptions of six fields of IS performance in evaluation perspective Fields IS vision IS infrastructure Definition and meaning Degree of IS vision's contents, scope, and fidelity in concert with business goal Proportion schemes in keeping with management objectives of organization Function, performance, and retention status of IS infrastructure (facilities, H/W, S/W, DB, network, and CASE tools for system development, etc.) Optimum level of IS infrastructure in concert with application Optimum level of organization, rules, institutions, and direction for IS usage (IS planning/ development/maintenance team, CIO, training system, information resource management system, maintenance system, and so on) Degree of adoption and usage of them Degree of retention and specialty for personnel supporting IS development, maintenance, management Degree of activities for these organizations and personnel Optimum level of application in concert with business Degree of adoption for new technology Degree of effectively cooperating with management strategy and business process Specialty level of general users Degree of usage for IS application in keeping with their task

IS organization and institution

IS supporting

IS application

IS usage

Table VI Evaluation fields and factors of IS performance Level IS vision Subject (who) ISP IS project plan Object (what) Business goal Business process External environment (legal, technical) Available resources Available time Application User IS provider Application Infrastructure (H/W, S/W, N/W) User IS provider Infrastructure (H/W, S/W, N/W) Supporting tool Application User Business process User Relationship (how) Comprehensibility Suitability Availability

IS infrastructure

H/W Database Network Supporting tools IS organization IS institution/rule/direction IS mind IS supporting team

Sufficiency Suitability Usability Satisfactory Adoptability Comprehensibility Supportability Supportability Usability

IS organization and institution

IS supporting

IS application

Application Information for service

Comprehensibility Degree of integration Suitability Satisfactory Usability Satisfactory

IS usage

User

Application IS institution/rule/direction

124

An integrated evaluation system

Choon Seong Leem and Injoo Kim

Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . 115-128

Table VII Function for integrated evaluation system Function Data collection Sub-function Questionnaire generation, measurable item identification and storage Interview material generation, interview item identification and storage Existing document survey and storage Scale (nominal, ordinal interval, ratio) generation and matching it to questionnaire Yardstick generation for each scale Item measurement, scoring and data storage Weight generation Score accumulation for each field and level and ordering Calculations for various useful information Comparison, gap analysis Flowing back-gap analysis results into continuous improvement process

Measurement

Analysis

Interpretation Feedback

Figure 9 A conceptual design of the integrated evaluation system

As shown in Figure 10, the evaluation results in year 2000 show that the most companies in Korea are staying at the ``business integration'' stages. While 14 percent stays at the ``process integration'' stage, only 4 percent barely enter into the ``role model generation'' stage. The comparison between years 1999 and 2000 shows that not a few companies in 1999 have experienced improvement to stand at higher stages in 2000. That is, the companies of ``function integration'', ``process integration'', or ``business integration'' stage in 1999 have been respectively improved into ``business integration'' or ``industry integration'' stages in 2000. Also, the distribution of IS level in 2000 has been compressed against 1999, which it means that the gap between advanced

companies and underdeveloped companies is gradually reduced. However, only 1 percent in the ``industry integration'' stages in 1999 has been evolved into ``role model generation'' in 2000, it implies that it is more difficult for the higher level companies to improve their performance. As shown in Figure 11, IS performance in most evaluation fields are reached about 60 percent, while the IS application and supporting fields stands about 50 percent in 2000. That is, although the increased interests in IS lead these companies to design their IS vision, to use existing IS, and to review their organization and institutions, they do not practically accomplish applying IS technology as well as developing the business application.

125

An integrated evaluation system

Choon Seong Leem and Injoo Kim

Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . 115-128

Figure 10 Level of IS performance based on the improvement stage

Figure 12 Comparison of IS level between years 1999 and 2000

Figure 11 IS performance in evaluation fields

Also, they do not keep efficient maintenance for their IS resources. In Figure 12, the results of comparison between years 1999 and 2000 show that the increasing trend of IS performance is the highest in manufacturing and construction companies, while the lowest in upper 30 companies in performance. This implies that the companies with higher IS performance levels has rather slower speed of improvement. The above phenomenon appears that the improvement in small and medium-sized enterprises, often called SMEs, are more notable them in large-scale companies. Figure 13 represents a list of focused IS indices and their results in year 2000. Not a few

interesting IS trends in Korea can be found. For example, while internet and intranet are actively used, corresponding business applications are not activated relatively, and especially, IT outsourcing barely activated in 2000, Korea. Figure 14 shows the evaluation results of IS performance level and improvement stage for a logistics service company K. The K company's IS level in 2000 is considerably increased. The performance levels of IS support and IS usage fields are significantly improved, although the levels of IS infrastructure and application approach to a standstill. The company can utilize the evaluation results to improve its IS performance in balance and to reach the role-model generation stage. By this case study involving the 219 Korean companies during two years from 1999 to 2000, the proposed the integrated evaluation system of IS performance has been proven adoptable and practical. Not only company K, every company participated in the case study received its own evaluation results of the 345 measurement items with detailed explanations. As the integrated evaluation system gave the quantitative results of company's IS performance, the managers or decision-makers are expected to use them for right alternatives and correct directions on there is strategy.

126

An integrated evaluation system

Choon Seong Leem and Injoo Kim

Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . 115-128

Figure 13 Focused IS indices in year 2000

Figure 14 Level of IS performance in company K

organization has its own business scope, characteristics, and cultures, it is recommended to apply this system with reasonable customizations. Therefore, the proposed system ought to be continuously revised and updated, so as to fit into various organization types and manager's needs.

References
Basili, V.R. and Rombach, H.D. (1988), ``The TAME project: toward improvement-oriented software environment'', IEEE Transactions on Software Engineering, Vol. 14 No. 6, pp. 758-73. Bate, R., Kuhn, D., Wells, C., Armitage, J., Clark, G. and Cusick, K. (1995), A Systems Engineering Capability Maturity Model, Version 1.1, CMU/SEI-95-MM-003, Carnegie Mellon University, Pittsburgh, PA. Beckett, A.J., Wainwright, E.R. and Bance, D. (2000), ``Implementing an industrial continuous improvement system: a knowledge management case study'', Industrial Management & Data Systems, Vol. 100 No. 7, pp. 330-8. Bergey, J.K., Northrop, L.M. and Smith, D.B. (1997), Enterprise Framework for the Disciplined Evolution of Legacy Systems, CMU/SEI-97-TR-007, Carnegie Mellon University, Pittsburgh, PA. DeLone, W.H. and McLean, E.R. (1992), ``Information systems success: the quest for the dependent variable'', Information Systems Research, Vol. 3 No. 1, pp. 60-95. Deming, W.E. (1986), Out of the Crisis, MIT Center for Advanced Engineering Study, MIT Press, Cambridge MA. Goodhue, D.L. (1998), ``Development and measurement validity of a task-technology fit instrument for user evaluations of information systems'', Decision Sciences, Vol. 29 No. 1, pp. 105-38.

6. Conclusion
In order to improve IS performance, this paper develops an integrated evaluation system that can diagnose the performance level, identify the deficiencies of current status, and return this information into improvement processes with a corresponding model. The improvement model and the integrated evaluation system of IS performance are tested with a large-scale case study to prove their adoptability and efficiency. The work offers several benefits; some of which are as follows. First, it defines the concepts of IS performance improvement. Second, it expands the IS framework comprehensively via systematic translation of the conceptual framework into measurable items. Third, it supports practical evaluation tools. In spite of these benefits, as every

127

An integrated evaluation system

Choon Seong Leem and Injoo Kim

Industrial Management & Data Systems Volume 104 . Number 2 . 2004 . 115-128

Goodhue, D.L. and Thompson, R.L. (1995), ``Task-technology fit and individual performance'', MIS Quarterly, pp. 213-36. Jeong, G.-H. (1996), Weights for Informatization Index, NCA, pp. 8-44. McFeeley, B. (1996), IDEAL: A User's Guide for Software Process Improvement, CMU/SEI-96- HB-001, Carnegie Mellon University, Pittsburgh, PA. Mason, R.O. (1978), ``Measuring information output: a communications system'', Information & Management, Vol. 1 No. 5, pp. 219-34. Mendonca, M.G. (1997), ``An approach to improving existing measurement frameworks in software development organizations'', doctoral dissertation, Department of Computer Science, University of Maryland, College Park, MD, pp. 10-16. Mendonca, M.G., Basili, V.R., Bhandari, I.S. and Dawson, J. (1998), ``An approach to improving existing measurement framework'', IBM Systems Journal, Vol. 37 No. 4, pp. 484-501. Myers, B.L., Kappelman, L.A. and Prybutok, V.R. (1997), ``A comprehensive model for assessing the quality and productivity of the information systems function: toward a theory for information systems assessment'', Information Resources Management Journal, Vol. 10 No. 1, pp. 6-25. Nolan, R.L. and Wetherbe, J.B. (1980), ``Toward a comprehensive framework for MIS research'', MIS Quarterly, pp. 1-19. Park, R.E., Goethert, W.B. and Florac, W.A. (1996), Goal-Driven Software Measurement A Guidebook, CMU/SEI-96-HB -002, Carnegie Mellon University, Pittsburgh, PA. Pulford, K., Kuntzmann-Cambelles, A. and Shrilow, S. (1996), A Quantitative Approach to Software Management: The AMI Handbook, Addison-Wesley, Wokingham. Renaissance Consortium (1997), ``Renaissance Framework method and tools support for the evolution and reengineering of legacy systems'', Renaissance Consortium. Saunders, C.S. and Jones, W. (1992), ``Measuring performance of the information systems function'', Journal of Management Information Systems, Vol. 8 No. 4, pp. 63-82. Seddon, P.B.and Kiew, M.Y. (1994), ``A partial test and development of the DeLone and McLean model of IS success'', Proceedings of the International Conference on Information Systems, Vancouver, Canada, pp. 99-110. Shannon, C.E. and Weaver, W. (1949), The Mathematical Theory of Communication, University of Illinois Press, Urbana, IL.

Shewhart, W.A. (1931), Economic Control of Quality of Manufactured Product, D. Van Nostrand Company, New York, NY. Tan, D.S. (1999), ``Stages in information systems management'', Handbook of IS Management, CRC Press LLC, Boca Raton, FL, pp. 51-75. Venkatraman, N. (1997), ``Beyond outsourcing: managing IT resources as a value center'', Sloan Management Review, Spring, pp. 51-64.

Further reading
AMI Consortium (1992), The AMI Handbook: A Quantitative Approach to Software Management, South Bank Polytechnic, London. Callon, J.D. (1996), Competitive Advantage through Information Technology, McGraw-Hill, Maidenhead, pp. 5-12. Chandra, C. and Kumar, S. (2001), ``Enterprise architectural framework for supply-chain integration'', Industrial Management & Data Systems, Vol. 101 No. 6, pp. 290-303. Kim, I.-J. (1999), ``Development of an integrated evaluation system based on the continuous improvement model of information system performance'', PhD dissertation for Yongsei University, Seoul. Kim, I.-J. and Leem, C.-S. (1999), ``An integrated evaluation framework for continuous capability maturity of enterprise information systems'', Proceedings of APIEMS'99, pp. 69-72. Leem, C.-S. (1999), 1999 Annual Reports for Evaluation of IS Performance, IT Research and Consulting. Leem, C.-S. (2000), 2000 Annual Reports for Evaluation of IS Performance, IT Research and Consulting. Loughman, T.P., Fleck, R.A. and Snipes, R. (2000), ``A cross-disciplinary model for improved information systems analysis'', Industrial Management & Data Systems, Vol. 100 No. 8, pp. 359-69. Saad, G.H. (2001), ``Strategic performance evaluation: descriptive and prescriptive analysis'', Industrial Management & Data Systems, Vol. 101 No. 8, pp. 390-9. Sakaguchi, T. and Dibrell, C.C. (1998), ``Measurement of the intensity of global information technology usage: quantitizing the value of a firm's information technology'', Industrial Management & Data Systems, Vol. 98 No. 8, pp. 380-94. Seddon, P.B. (1997), ``A respecification and extension of the DeLone and McLean model of IS success'', Information Systems Research, Vol. 8 No. 3, pp. 240-53. Vernadat, F.B. (1996), Enterprise Modeling and Integration: Principles and Applications, Chapman & Hall, London, pp. 14-16, 317-34.

128