Proceedings of the 39th Hawaii International Conference on System Sciences - 2006

Towards a Framework for Health Information Systems Evaluation

Maryati Mohd. Yusof, Ray J. Paul, Lampros K. Stergioulas

School of Information Systems, Computing and Mathematics, Brunel University, Uxbridge,
Middlesex, UB8 3PH, UK
Maryati.Yusof@brunel.ac.uk; maryati226@yahoo.com

Abstract
The evaluation of Health Information Systems
(HIS) is crucial to ensure their effective
implementation and positive impact on healthcare
delivery. A review of HIS evaluation studies was
carried out which indicated that the improvement of
current methods is required. In order to satisfy this
requirement, a new framework is introduced for the
evaluation of Information Systems (IS) in healthcare
settings. The proposed HOT-fit framework (Human,
Organization and Technology-fit) was developed after
having critically appraised the existing findings of
HIS evaluation studies. It also builds on previous
models of IS evaluation, in particular, the IS Success
Model and the IT-Organization Fit Model. HOT-fit
incorporates the concept of fit between technology,
human and organization. The paper outlined the
proposed methodology of a Fundus Imaging Systems
in a primary care in the UK. The framework can be
potentially used as a tool to conduct better and
comprehensive HIS evaluation.
1. Introduction
The provision of health care is increasingly shaped
by the adoption of Health Information Systems (HIS).
HIS is a group of processes implemented to aid in
enhancing the efficiency and effectiveness of
healthcare organization in performing its functions
and attaining its objectives. HIS range from simple
systems such as transaction processing systems to
complex systems such as clinical decision support
systems (CDSS). To ensure effective implementation
and positive impact of HIS on healthcare delivery,
evaluation of these systems is crucial. HIS
evaluation has been highlighted as one of the most
important topics of Health Informatics research
[8,18]. HIS evaluation is defined as “the act of
measuring or exploring attributes of a HIS (in
planning, development, implementation, or
operation), the result of which informs a decision to
be made concerning that system in a specific context”
[3]. The decision making in design, development,
purchase or management in HIS requires evaluation;
it is a challenge therefore to undertake evaluation
[32].
Evaluation serves a number of purposes. Given
the unpredictable characteristics of IS in general and
the aim of improving clinical performance and patient
outcomes in particular, evaluation is undertaken to
understand system performance [42]. Essentially,
evaluation of health informatics application has the
potential to improve the quality of care and its costs
and to determine the safety and effectiveness of HIS
[28,42]. Evaluation is also performed to improve HIS
by using past experience to identify more effective
techniques or methods, investigate failure and learn
from previous mistake [15].
HIS evaluation has a number of barriers and
problems that pose challenges to its evaluators
[2,15,46]. Research in Health Informatics evaluation
is still in its infancy and what constitutes ‘good’ HIS
is unclear [2]. It is argued that while HIS are .
increasingly being developed, the number of
published evaluations is limited [3,43]. In addition,
evaluation of HIS is difficult to perform, particularly
in selecting a framework to be applied and methods to
be used [3]. However, it seems that such problems
regarding evaluation methods in Health Informatics
can be overcome [2]. New methods and extensions
are needed to improve the existing methods [32].
The organizational and social issues of HIS should
be analyzed before undertaking evaluation, as they are
the main components of such system [37]. Moreover,
technology, human and organization should fit with
each other to realize the potential of HIS. Therefore, a
comprehensive evaluation framework that addresses
technology, human and organization and the fit
between them is essential to ensure effective
evaluation. More work on these areas is needed as
most existing evaluation studies of HIS focus on

0-7695-2507-5/06/$20.00 (C) 2006 IEEE 1

 Proceedings of the 39th Hawaii International Conference on System Sciences - 2006

technical issues or clinical processes, which do not issues arise that relate to its nature, structure, culture
explained why HIS works well or poorly with a and politics [23]. The organizational aspects of
specific user in a specific setting [8,9,24,25,26,27]. evaluation are classified into organizational structure
The aim of this paper is to propose a new and organizational environment. The organizational
framework for HIS evaluation that incorporates structure constitutes organizational nature, hierarchy
comprehensive dimensions and measures of HIS and functional divisions. External environment
effectiveness and fit between the technological, includes regulations governing hospital, the type of
human and organizational factors. This proposed populations being served, epidemics that may occur,
framework, HOT-fit (Human, Organization and government policy concerning financing of health
Technology-fit) will potentially be useful in care, and the supply of physicians.
conducting a better and thorough evaluation study. It Approaches to HIS evaluation were developed
is also possible that it will assist researchers and based on different domains including technical,
practitioners in unfolding the complexity of HIS sociological, economic, human and organizational.
evaluation. The new framework builds from previous Here, a number of frameworks, which were explicitly
models of IS evaluation, namely IS Success Model designed to be used for HIS evaluation, are reviewed
[10,12] and IT-Organization Fit Model [35]. In to identify the evaluation dimensions and measures
developing this model, problems and methods of HIS used to evaluate systems in a healthcare setting (See
evaluation highlighted in the selected Health Table 1).
Informatics literature are discussed. Further, the
aforementioned models in the field of Information Table 1: Selected HIS evaluation frameworks
Systems are presented to explore their applicability in
improving those of Health Informatics. Framework Evaluation Aspects
This paper is organized into five sections. The Technology Human Organization
following section provides a theoretical background 4Cs (Kaplan, HIS and its Communi- Control
1997) development cation Care
of the proposed framework including a brief impact Context
description of early approaches to HIS evaluation. CHEATS Technical Human Clinical
The proposed evaluation framework for HIS is (Shaw, 2002) Education Organization
presented in section three. Section four presents Social Administration
research methodology for this research in progress.
TEAM IS based on Role Structure
Finally, conclusions and recommendations for further (Grant, et al., management
research are given in section five. 2002) level
ITAM IT adoption Individual
(Dixon, user
1999)
2. Theoretical Background
4Cs is developed from the Social Interactionist
This paper reviews the selected literature of Health
Theory, which stands for Communication (interaction
Informatics and Information Systems. First, early
within department), Care (medical care delivery),
approaches to HIS evaluation are analyzed. This
Control (control in the organization), and Context
analysis is followed by the discussion of the IS
(clinical setting) [23]. The evaluation measures of this
Success Model. The limitations of this model are
framework can be make clearer and the Control
related to human and organizational factors and the
aspect need further explanation. CHEATS is a generic
concept of fit; therefore these factors and concept are
framework for evaluating IT in healthcare that has six
presented next. The proposed model combine the
evaluation aspects: Clinical, Human and
strengths of the IT Organization-fit with the IS
organizational, Educational, Administrative,
Success Model, thus, the latter model is subsequently
Technical and Social [36]. CHEATS attempt to
put forward.
provide a more comprehensive evaluation aspects and
more specific measures, especially in the clinical
2.1 Early Approaches to HIS evaluation aspect. However, the dimensions within some of the
aspects, such as technical and human and
Evaluation can be performed at human or
organizational and dimensions within each aspect are
organizational level. Evaluation of human factors
still inadequate and need further work. Further, a
covers a wide range of issues including training,
global framework known as TEAM (Total Evaluation
personnel, personnel attitudes, ergonomics and
and Acceptance Methodology) is constructed based
regulations affecting employment [23]. Beyond this
on systemic and model theories [19]. It has three
individual unit of analysis, larger organizational

2
 Proceedings of the 39th Hawaii International Conference on System Sciences - 2006

dimensions: Role, Time (evaluation phase) and viewed as a dynamic process instead of a static state.
Structure (strategic, tactical, operational management The multidimensional relationships among the
level). The 3D structure of this model illustrates measures of IS success have been tested extensively
clearly the components of system evaluation. in a number of IS studies [11]. Based on these studies,
However, apart from the Role and Time aspects, the an updated version of this model is presented [12]
Structure aspect can be challenging. The selection of (See Figure 1). These measures are included in these
evaluation measures that match the management level six system dimensions: System Quality (the measures
can be challenging as the same measures can be of the information processing system itself),
categorized into more than one management level. As Information Quality (the measures of IS output),
a whole, this framework is quite broad for a specific Service Quality (the measures of technical support or
type of IS evaluation. Meanwhile, an IT service), Information Use (recipient consumption of
implementation and evaluation framework for the output of IS), User Satisfaction (recipient
individual user known as the IT Adoption Model response to the use of the output of IS) and Net
(ITAM) is constructed to study the individual user Benefits (the overall IS impact).
perspective and their potential IT adoption [14]. From
the individual user perspective, this framework
includes comprehensive evaluation criteria and
relationships among them. This framework is clearly
insufficient for a wider scope of evaluation, which
involves the organizational aspect.
Overall, these frameworks complement each other
in that they each evaluate different aspects of HIS.
Thus, these different aspects can be combined in a
single framework to allow comprehensive evaluation
studies. However, these frameworks do not provide
explicit evaluation categories to the evaluator.
Hence, more specific measures within the dimensions Figure 1: Information Systems Success
of each aspect can be defined to facilitate HIS Model
evaluation. In the search for a comprehensive, specific
(Source: DeLone and McLean, 2004)
measures of evaluation framework, two models have
been identified as being complementary of each other
2.2.1 System Quality. The studies on System
in fulfilling the limitations of existing HIS evaluation
Quality are often associated with system performance.
frameworks, namely IS Success Model and IT
Examples of system quality measures are ease of use,
Organization-Fit Model.
ease of learning, response time, usefulness,
This research will make use of the IS Success
availability, reliability, completeness, system
Model based on its comprehensive, specific
flexibility and easy access to help [4,13,29,38,41].
evaluation categories, extensive validation and its
applicability to HIS evaluation. A review of success
2.2.2 Information Quality. Measures of
determinants of Inpatient Clinical IS indicates that the
Information Quality usually focus on information
categories for success of this model can be used to
produced by the system such as a report. Some
assess a specific type of HIS [43]. As mentioned
criteria used for information system quality are
above, IT Organization-fit Model complements the IS
information accuracy, output timeliness, reliability,
Success Model by featuring the concept of fit and
completeness, relevancy, legibility, availability and
organizational factors. IS Success Model is discussed
consistency [4,15,29,39,41]. Most information
in the next section, followed by the IT Organization-
quality measures are subjective, as they are derived
Fit Model.
from the user perspective.

2.2 IS Success Model
In order to organise numerous researches on IS
success categories, a comprehensive taxonomy is
introduced [10]. A model is constructed which
consists of six success categories or dimensions; they
are linked causally and temporally as success is
2.2.3 Service Quality. The new addition to the
IS Success Model – Service Quality – is concerned
with the overall support delivered by the service
provider, regardless of whether the service is
delivered by the internal department or outsourced to
external vendors. Quick responsiveness, assurance,
empathy and following up service are examples of the
service quality dimension.

3
 Proceedings of the 39th Hawaii International Conference on System Sciences - 2006
2.2.4 System Use. The use of information output
such as reports appears to be one of the most frequent
measures to assess the success of IS. The use of
system in terms of system inquiries and system
functions is another example of the measures in this
success dimension. System use is concerned with the
frequency and breadth of system usage. The actual
use of system as a measure of IS success refers to
voluntary instead of mandatory use. Other issues
pertinent to system use are the person who uses it and
their levels of use and training [4,29].
System use is also related to individual knowledge
and belief [5]. These measures are related to human
acceptance and resistance. Jiang et al. [21] regard
resistance as an important factor of system success.
As different types of systems are usually related with
a particular type of function and user, the reasons for
resistance might differ among system types.
Resistance can be viewed from one of the following
theories: (1) people-oriented, (2) system-oriented and
(3) interaction-oriented. People-oriented theory
describes resistance to system results from user’s
(groups or individuals) internal factors. Personal
characteristics such as age, gender, background, value
and belief have been suggested as influencing
individual’s attitude towards technology. System
oriented theory suggests that resistance results from
system design factors or relevant technology
including user interface and system characteristics.
Interaction theory explains resistance from the
interaction between people and system factors; thus,
assessment of a system varies across settings and
users. Job insecurity and fear are some examples of
interaction resistance.
2.2.5 User Satisfaction. User satisfaction is often
used to measure system success. It is subjective in
nature as it depends on whose satisfaction is
measured. Some studies relate user satisfaction to
perceived usefulness and user’s attitudes towards IS.
User satisfaction is defined as the overall evaluation
of user’s experience in using the system and the
system’s potential impact.
2.2.6 Net Benefits. A system can benefit a single
user, a group of users, an organization or an entire
industry. Individual impact is the effect of
information on the behaviour of the recipient. It is
associated with performance as well as changes in
user task such as change in work activity and
improved productivity. Examples of individual impact
measures include time efficiency, work efficiency and
effectiveness, decision quality and error reduction
[4,9]. Organizational impact is the effect of
information on organizational performance. Cost
effectiveness and organizational performance are
some of the examples of organizational impact
measure.
In comparison with existing HIS frameworks, IS
DeLone and McLean’s model illustrates clear,
specific dimensions of IS success or effectiveness and
the relationships between them. However, it does not
include organizational factors that are pertinent to IS
evaluation. Van der Meidjen et al. [43] discovered
that a number of measures such as user involvement
during system development and organizational culture
do not match any of the dimensions of the framework.
The extension of this framework is recommended by
adding the organizational factors, their dimensions
and clinical measures related to the healthcare
domain.
2.3 Human and Organizational Issues
The importance of human and organizational
factors in the development and implementation of IS
has been advocated in IS and Health Informatics
literature. A study conducted by Brender et al., [8]
indicates that ‘soft issues’ including methods, human
and organisational issues are emphasised more in
health care than ‘technical issues’.
HIS are not useful unless clinicians use them, and
clinicians will not use them if there are barriers that
impede the system use. Barriers to using HIS are also
important to consider in HIS evaluation as they
explain the failure and success of these systems.
Culture and process changes are reported to be the
barriers to the wider use of health care systems [28].
Studies cited in Anderson [5] identified a number of
barriers to direct physician use of HIS including low
level of expertise, lack of acceptance, lack of medical
staff sponsorship and alteration of traditional
workflow patterns. Examples of organisational
challenges include hospital culture, such as being risk
adverse, reluctance to invest much in IT and
resistance to change [7,30].
In short, human and organizational factors are
equally important as technical issues with regards to
system effectiveness [26]. Human, organizational
and technical elements should also have a mutual
alignment or ‘fit’ in order to ensure successful HIS
implementation. It is crucial that HIS fit
organizational aspects as well as align with work
routines, management assumptions, patient care
philosophies and users’ needs as the introduction of a
system affects different dimensions of fit in complex
ways [25]. Further, the alignment of organisation
(objectives, structures and processes), technology and
human is an important starting point in IT
4
 Proceedings of the 39th Hawaii International Conference on System Sciences - 2006
implementation as it is one of the strategies that affect
IT investment [45].
2.4 The Concept of Fit
A number of studies in health informatics have
included the concept of ‘fit’ [1,6]. Southon et al. [39]
found that lack of fit among main organization
elements contributes to a large number of system
failures in public health. The fit between technical,
organizational, and social factors is analysed to
identify gaps between current health care systems and
new system features [20]. Kaplan [25] shows that
poor fit between system developers’ goals and
clinicians’ cultural values contributes to user
reluctance to use CDSS. Executive support,
understanding the business, IT-business relations, and
leadership are identified as both enablers and
inhibitors of the fit of IT and business [31].
2.4.1 IT-Organization Fit Model. Management
in the 90s (MIT90s) is a well-known IT-
organizational fit model, which includes both internal
and external elements of fit [35]. Figure 2 illustrates
the concept of fit between the main organizational
elements. Internal fit is accomplished by a dynamic
equilibrium of organizational components including
business strategy, organizational structure,
management processes, and roles and skills. External
fit is achieved by formulating organizational strategy
based on environmental trends and changes such as
market, industry and technology. Within this internal
and external fit as its enabler, IT is expected to affect
organizational performance and its strategy.
Figure 2: The MIT90s (IT-Organization Fit
Model) (Adapted from Scott Morton, 1991)
In order to realize the benefits of IT, three
prerequisites are required for successful IT
transformation. First, organizational vision and the
reasons behind it have to be clear to organizational
members to get them prepared for organizational
changes and hence reduce the challenges in managing
transformation. Second, organizational corporate
strategy (business and IT), information technology
and organizational dimensions have to be aligned with
each other. Third, a robust IT infrastructure such as
electronic network and understood standards should
be equipped within the organization. These three
prerequisites as well as the internal and external fit
may be used to identify the problems in IT
implementation.
It is discovered that this model was relatively
new and has not been extensively utilized in
healthcare [40]. The model was also identified as
being capable of identifying the main organizational
elements which can affect IS as well as emphasizing
the essential alignment or fit between them.
Moreover, the model is comprehensive as it includes
the following factors: technology (IT), human (Roles
and Skills) and organization (Strategy, Structure and
Management Process). However, these factors can
be categorized into more detailed dimensions to
provide more specific evaluation dimensions. For
instance, IT can be further classified into system
quality and information quality, as proposed by
DeLone and McLean [10]. Roles and skills can be
associated with use and user satisfaction. Based on
the strengths and limitations pointed out in both
models, IT-Organization Fit and the IS Success
Model complement each other in presenting a
comprehensive evaluation framework. Organizational
factors, which are lacking in the IS Success Model,
are featured in the IT Organization-fit. Similarly,
specific evaluation dimensions and measures which
are lacking in the IT Organization-fit, are featured in
the IS Success Model. Based on the two models
explained above, a new evaluation framework is
proposed. This framework is referred to as Human-
Organization-Technology-Fit (HOT-fit) and is
presented in the next section.
3. Proposed Evaluation Framework
The proposed evaluation framework was
developed after having critically appraised the
existing findings of HIS and IS evaluation studies. It
also makes use of the IS Success Model in
categorizing its evaluation factors, dimensions and
measures. In addition, the IT-Organization Fit Model
is also used to incorporate the concept of fit between
the evaluation factors: human, organization and
5
 Proceedings of the 39th Hawaii International Conference on System Sciences - 2006
technology. The IS Success Model was extended by
the addition of the following features which are
explained in the following part of this section. (See
Figure 3):
• Organizational factors and their dimensions:
Structure and Environment.
• Fit between technological, human and
organizational factors.
• Two-way relationships between these
dimensions: Information Quality and System Use,
Information Quality and User Satisfaction,
Structure and Environment, Structure and Net
Benefits, Environment and Net Benefits.
Figure 3: Human-Organization-Technology Fit
(HOT-fit) framework
A model of HIS should fit human (stakeholders)
and organizations based on their needs. Therefore,
HIS should work according to human needs and it
should assist humans in performing their task.
Similarly, humans should possess appropriate
knowledge and attitude in order to be able to use HIS
in performing their task. Likewise, healthcare
organizations should be equipped with appropriate
technology and infrastructure in order to realize the
potential of HIS. Further, healthcare organizations
should have the capacity to prepare their staff
members to adapt to any changes resulting from the
uptake of HIS to reduce the challenges in managing
transformation. This can be achieved through strategy
and management such as leadership support,
teamwork and effective communications that are
formed using the roles and skills of the staff.
Moreover, organizational and IT plan should be
aligned with each other to ensure that IT is supporting
the organizational objectives. The fit between human,
organization and technology is illustrated by the thick
arrows in Figure 3. Fit can be measured and analyzed
using a number of measures define in these three
factors including ease of use, usefulness, relevancy,
attitude, training, user satisfaction, culture, planning,
strategy, management and communication. Human,
organization and technology comprise IS which
impacts are assessed in the net benefits. These factors
correspond to eight interrelated dimensions of HIS
success: System Quality, Information Quality, Service
Quality, System Use, User Satisfaction,
Organizational Structure, Organizational
Environment and Net Benefits. The examples of each
dimension are shown in Table 2 (page 10).
These dimensions influenced each other in a
temporal and causal way. System Quality,
Information Quality and Service Quality singularly
and jointly affect both System Use and User
Satisfaction. Some of these relationships are two
ways; for instance, System Use, which relies on user
knowledge and training, can influence the information
quality, since user’s knowledge in using the system
can affect reports, images and prescriptions produced
by the system. The level of System Use can affect the
degree of User Satisfaction and vice versa. Similarly,
the Environment factors such as government policy
and politics can affect organizational Structure while
factors in organizational Structure will affect the
population served in the Environment. System Use
and User Satisfaction are direct antecedent of Net
Benefits. Net Benefits will then affect System Use and
User Satisfaction. Similarly, organizational Structure
and Environment are direct antecedent of Net
Benefits. Net Benefits will then have impact on
organizational Structure and Environment.
Based on its comprehensive dimensions and
outcome measures, the framework could be used to
evaluate the performance, effectiveness and impact of
HIS or IT in healthcare settings [33]. Effectiveness
refers to the accomplishment of specific goals with
accuracy and completeness as well as the correct
utilisation of appropriate resources [17]. In this
research, effectiveness is defined as the ability of a
healthcare organization to continuously accomplish
goals using optimum resources within a specified
time.
3.1 Technology
System quality in a healthcare setting measures the
inherent features of HIS including system
performance and user interface. Examples of
measures are ease of use, ease of learning, response
time, usefulness, availability, reliability, flexibility,
access to technical support, security. Ease of use
6
 Proceedings of the 39th Hawaii International Conference on System Sciences - 2006
assesses whether healthcare professionals regard HIS
as satisfactory, convenient and pleasant to use.
Availability refers to the up time of HIS while
flexibility is concerned with the ability of HIS to
adapt to a health care setting and integrate with other
systems. Further, it is also important to determine
whether the system meets the needs of the projected
users and fits the work patterns of the professionals
for whom it is intended and the overall health system
[4,36].
Measures of information quality are concern with
information produced by HIS including patient
records, reports, images and prescriptions. Criteria
that can be used for HIS quality are information
completeness, accuracy, legibility, timeliness,
availability, relevancy, consistency, reliability, data
entry methods and quality.
Service quality is concerned with the overall
support delivered by the service provider of HIS or
technology, regardless of whether the service is
delivered by the internal department of healthcare
organisation or outsourced to external providers.
Service quality can be measured through quick
responsiveness, assurance, empathy and follow up
service.
3.2 Human
System Use refers to the frequency and breadth of
HIS inquiries and functions. System Use also relates
to the person who uses it, their levels of use, training,
knowledge, expectation and acceptance or resistance.
Knowledge is concerned with computer literacy and
skills [5,9]. Expectation refers to the anticipation of
improved patient care delivery from the use of HIS
[5]. Resistance can be assessed from the following
perspectives: user internal factors; system
characteristics and technology factors and interaction
between human and system factors.
User satisfaction is the overall evaluation of the
user’s experience in using HIS and potential impact of
HIS. User satisfaction can be related to perceived
usefulness and user’s attitudes towards HIS which are
influenced by user’s personal characteristics.
3.3 Organization
The nature of a healthcare institution can be
examined from its structure and environment.
Organization structure consists of nature including
type and size (number of beds), culture, politic,
hierarchy, autonomy, planning and control systems,
strategy, management and communication.
Leadership, top management support and medical
staff sponsorship are also important measures of HIS
success [5,39]. The environment of a healthcare
organization can be analysed through its financing
source, government, politics, localization,
competition, inter-organisational relationship,
population served, and communication.
3.4 Net Benefits
Net benefits capture the balance of positive and
negative impacts on user, which includes clinicians,
managers and IT, staff, system developers, hospitals
or the entire healthcare sector. Net benefits can be
assessed using direct benefits, job effects, efficiency,
effectiveness, error reduction, communication,
clinical outcomes and cost. In the healthcare context,
clinical outcomes can be used as a means of
measurement. Examples of these measures include
costs reduction, which is due to fewer medication
errors and Adverse Drug Effect (ADE); improved
efficiency in patient care delivery, specifically
pertaining to tests and drug orders and increased use
of generic drug brands and number of consultations
and length of waiting lists [9,44]. Clinical outcomes
are also measured through two criteria: morbidity (the
rate of incidence of a disease) and mortality (death
rate). Apart from these quantitative measures, clinical
impacts can also be assessed qualitatively using these
measures: quality of care, impact on patient care and
communication, such as change in communication
style and facilitation of information access [36].
4. Proposed Research Methodology
The planned approach is to use a subjectivist, case
study strategy with qualitative methods. A formative
evaluation will be undertaken on the adoption of a
Fundus Imaging System (FIS) in a Primary Care of
the National Health Service (NHS) in the UK due to
its relative importance in the diabetic care. A
subjectivist approach will be employed in order to
gain an extensive understanding of the healthcare
context surrounding the FIS through detailed,
insightful explanation of the study [15]. Qualitative
methods will be employed to generate a fuller
description of healthcare settings and their cultural
issues and to understand why systems function well or
poorly in a particular setting. Further, a single, in-
depth case study will be undertaken to obtain a
comprehensive view and understanding of the
development process of FIS. A number of data
collection methods will be employed, including in-
depth interviews, participant observation and
document/artefact analysis.
7
 Proceedings of the 39th Hawaii International Conference on System Sciences - 2006
This research approach to FIS evaluation consists
of six iterative phases, which are problem
identification, development of initial evaluation
framework, selection of research strategy and
methods, system evaluation, outcome disseminations
and refinement of evaluation framework [15,23]. The
first three phases were achieved. Evaluation problems
(issues, questions and concerns) have been identified
through a literature review as well as observations
made during immersion. An immersion was carried
out to set the general context of the research as well to
establish rapport with relevant stakeholders. Initial
data collection was gathered during immersion. An
initial evaluation framework was constructed based
on findings from the first phase. This framework will
be used as a guideline in the later phase: evaluation of
FIS. The research strategy and methods were selected
based on the research problem.
In the evaluation phase, the proposed evaluation
framework will be appraised using the case study of
FIS. A pilot study will be conducted prior the actual
case study for the purpose of improving the planning,
monitoring and evaluation process of the research.
Participant observations of daily clinical routines and
meetings will take place at different units,
departments and clinics. During observations and
interviews, individuals including user, clinicians and
IT staff that are involved with the system will be
requested to document or recall their system use and
patient pathways. Patients will be queried about their
perception towards the system. Data will be
collected on planned occasions as well as
spontaneously in a number of iterative cycles. The
data will be recorded and interpreted based on the
expert knowledge of the researchers. Four techniques
will be used to analyze the results: coding, analytic
memos such as reflection notes, displays such as
matrices, flowcharts and concept maps, and contextual
and narrative analysis [22].
Evaluation plan and preliminary findings will be
presented at a project stakeholders meeting. The
purposes of this presentation are to make further
decisions regarding system and system evaluation, get
feedback and comments, validate initial findings and
shape evaluation plan and system development
efforts. Finally, the evaluation will be improved and
refined based on the ongoing literature review and
case study results.
Potential bias acknowledged in qualitative
research approach will be overcome by conducting a
reliability test and data triangulation. Reliability will
be achieved through detailed documentation of
procedures and appropriate record keeping [34].
Triangulation will be accomplished through the use of
multiple evidences from different sources to confirm
the same fact or finding [47]. Participant validation
(where the interviewees review and approve their
results) will also be used to reduce bias.
5. Conclusions
This paper has identified the problems, reviewed
the existing methods and proposed a new framework
for HIS evaluation. In the search for an appropriate,
comprehensive approach to HIS evaluation, a number
of existing frameworks for IS evaluation in Health
Informatics and IS literature are analysed. The review
findings suggest that there is more need to improve
existing methods in HIS evaluation. The strengths and
limitations of these frameworks are discussed and are
used as a basis for the new proposed framework,
HOT-fit. In addition to the literature review, this
framework builds on the IS Success Model and IT-
Organization Fit Model. HOT-fit addresses the
essential components of IS, namely human,
organization and technology and the fit between them.
In order to validate its usefulness, this framework
needs to be tested in clinical settings. Findings from
the fieldwork could be used for further improvement
and refinement of this framework. The framework
should be applied flexibly, depending on different
context and purposes; emphasize can be given on the
most important dimensions and measures.
The contribution of this paper is that it brings
together disparate evaluation studies in both Health
Informatics and IS literature to provide a
comprehensive picture of the-state-of–the-art as well
as research needs of HIS evaluation. It analyses a
number of existing frameworks and models of
evaluation in HIS and IS. The paper also demonstrates
the applicability of IS models in HIS evaluation.
Moreover, this paper introduces a new framework for
HIS evaluation that combines dimensions and
measures from current evaluation frameworks in
Health Informatics and models in IS and review of
both literature. Although it focused on a specific
setting, the evaluation framework of this study will
potentially be useful for researchers and practitioners
to conduct a better and comprehensive evaluation
study on other HIS or IT applications in healthcare
settings.
6. References
[1] Aarts, J., Peel, V., Wright, G., "Organizational issues
in health informatics: a model approach," International
Journal of Medical Informatics, 52, 1998, 235-242.
8
 Proceedings of the 39th Hawaii International Conference on System Sciences - 2006
[2] Ammenwerth, E., Graber, S., Hermann, G., Burkle, T.,
Konig, J., "Evaluation of health information systems -
problems and challenges," International Journal of
Medical Informatics, 71, 2003, 125-135.
[3] Ammenwerth, E., Brender, J., Nykanen, P, Prokosch,
H.-U., Rigby, M., Talmon, J., HIS-EVAL Workshop
Participants., "Visions and strategies to improve
evaluation of health information systems: Reflections
and lessons based on the HIS-EVAL workshop in
Innsbruck" International Journal of Medical
Informatics, 73, 2004, 479-491.
[4] Anderson, J. G., Aydin, C.E., "Overview: Theoretical
Perspectives and Metholodogies for the Evaluation of
Health Care Information Systems," in Evaluating
Health Care Information Systems; Methods and
Applications, A. C. Anderson JG, Jay SJ, Ed.
Thousand Oaks, CA: Sage, 5-29, 1994.
[5] Anderson, J. G., "Clearing the way for physicians' use
of clinical information systems'," Communications of
the ACM, 40, 1997, 83-90.
[6] Berg, M., "Patient care information systems and health
care work: a sociotechnical approach," International
Journal of Medical Informatics, 55, 1999, 87-101.
[7] Bottles, K., "Critical Choices Face Healthcare in How
to Use Information Technology," Medscape General
Medicine, 1, 1999.
[8] Brender, J., Nohr, C., and McNair, P., "Research
needs and priorities in health informatics,"
International Journal of Medical Informatics, 58-59,
2000, 257-289.
[9] Coiera, E., Guide to Health Informatics, 2nd ed:
Hodder Arnold, 2003.
[10] DeLone, W. H., McLean, E.R., "Information Systems
Success: The Quest for the Dependent Variable,"
Information Systems Research, 3, 1992, 60-95.
[11] Delone, W. H., McLean, E. R., "Information systems
success revisited" Proceedings of the 35th Hawaii
International Conference on System Sciences
(HICCS), Big Island, HI, USA. 2002.
[12] DeLone, W. H., McLean, E.R., "Measuring e-
commerce success: applying the DeLone & McLean
Information Systems Success Model," International
Journal of Electronic Commerce, 9, 2004, 31-47.
[13] Doebbeling, B., "Information Technology and Primary
Care at VA: Interdisciplinary Partnership
Opportunities for Providers, Managers, and
Researchers" Health Service Research & Development
Series. Retrieved 21July, 2004 from
http://www.hsrd.research.va.gov/publications/internal/f
orum10_03.pdf. 2003.
[14] Dixon, D. R., "The behavioral side of information
technology," International Journal of Medical
Informatics, 56, 1999, 117-123.
[15] Friedman, C. P., Wyatt, J.C., Evaluation Methods in
Medical Informatics. New York: Springer-Verlag,
1997.
[16] Gawande, A. A., Bates, D.W., "The Use of Information
Technology in Improving Medical Performance - Part
II. Physician-Support Tools," Medscape General
Medicine, 2, 2000.
[17] George, B., "A Framework for IT Evaluation
Research," Americas Conference on Information
Systems (AMCIS 2001), USA, 2001.
[18] Giuse, D. A. and Kuhn, K. A., "Health information
systems challenges: the Heidelberg conference and the
future," International Journal of Medical Informatics,
69, 2003, 105-114.
[19] Grant, A., Plante, I., Leblanc, F., "The TEAM
methodology for the evaluation of information systems
in biomedicine," Computers in Biology and Medicine.
32, 2002, 195-20.
[20] Heeks, R., Mundy, D., Salazar, A., "Why Health Care
Information Systems Succeed or Fail," Institute for
Development Policy and Management, University of
Manchester, Retrieved November 11, 2003 from
http://www.man.ac.uk/idpm/idpm_dp.htm#isps_wp.
1999.
[21] Jiang, J. J., Muhanna, W.A., Klenin, G., "User
resistance and strategies for promoting acceptance
across system types," Information & Management, 37,
2000, 25-36.
[22] Kaplan, B., Maxwell, J., "Qualitative Research
Methods for Evaluating Computer Information
Systems," in Evaluating Health Care Information
Systems; Methods and Applications., Aydin, C. E.,
Anderson, J. G., Jay, S.J., Ed. Thousand Oaks.: Sage,
45-67, 1994.
[23] Kaplan, B., "Organizational Evaluation of Medical
Information Resources," in Evaluation Methods in
Medical Informatics, Friedman, C. P., Wyatt, J.C., Ed.
New York: Springer-Verlag, 255-280, 1997.
[24] Kaplan, B., "Evaluating informatics applications -
clinical decision support systems literature review"
International Journal of Medical Informatics, vol. 64,
pp. 15-37, 2001a.
[25] Kaplan, B., "Evaluating informatics applications-some
alternative approaches: theory, social interactionism,
and call for methodological pluralism." International
Journal of Medical Informatics, 64, 2001b, 39-56.
[26] Kaplan, B., Shaw, N., "People, organizational and
social issues: Evaluation as an Exemplar." Yearbook of
Medical Infomatics 2002, 2002, 63-76.
[27] Kaplan, B., Shaw, N.T., "Future directions in
evaluation research: people, organizational, and social
issues”. Methods of Information in Medicine, 43, 2004,
215-231.
[28] Kuhn, K. A., Giuse, D.A., "From Hospital Information
Systems to Health Information Systems - Problems,
Challenges, Perspective," Yearbook of Medical
Informatics 2001, 2001, 63-76.
[29] Lippeveld, T., "Routine Health Information Systems:
The Glue of a Unified Health System," Workshop on
Issues and Innovation in Routine Health Information in
Developing Countries, Bolger Center for Leadership
Development. Potomac, Maryland, 2001.
[30] Lorenzi, N. M., & Riley, R. T. (2003). Organizational
ISSUES=change. International Journal of Medical
Informatics, 69(2-3): 197-203.
[31] Luftman, J., "Assessing Business Alignment Maturity,"
Communications of AIS, 4, 2000, 1-51.
9
 Proceedings of the 39th Hawaii International Conference on System Sciences - 2006

[32] Moehr, J. R., "Evaluation: salvation or nemesis of
medical informatics?" Computers in Biology and
Medicine, 32, 2002, 113-125.
[33] Roderer, N., "Outcome measures in clinical
information systems evaluation." Medinfo. 2004, San
Francisco, CA, USA, 2004.
[34] Rowley, J., "Using Case Studies in Research,"
Management Research News, 25, 2002, 16-27.
[35] Scott Morton, M. S., The Corporation of the 1990s.
New York: Oxford University Press, 1991.
[36] Shaw, N. T., "'CHEATS': a generic information
communication technology (ICT) evaluation
framework," Computers in Biology and Medicine, 32,
2002, 209-220.
[37] Sittig, D. F., Hazlehurst, B.L., Palen, T., Hsu, J,
Jimson, H., Hornbrook, M.C., "A Clinical Information
System Research Agenda for Kaiser Permanente," The
Permanente Journal, 6, 2002.
[38] Smith, J., Health Management Information Systems: A
Handbook for Decision Makers. Buckingham: Open
University Press, 2000.
[39] Southon, F. C. G., Sauer, C., Grant, C.G., "Information
Technology in Complex Health Services:
Organizational Impediments to Successful Technology
Transfer and Diffusion," Journal of the American
Medical Informatics Association, 4, 1997, 112-124.
[40] Southon, F. C. G., Sauer, C., Dampney, K., "Lessons
from a failed information systems initiative: issues for
complex organisations," International Journal of
Medical Informatics, 55, 1999, 33-46.
[41] Stoop, A. P., Berg, M., "Integrating Quantitative and
Qualitative Methods in Patient Care Information
Systems Evaluation: Guidance for the Organizational
Decision Maker," Methods of Information in Medicine,
42, 2004, 458-462.
[42] Van Bemmel, J. H. a. M., M.A. (eds.), Handbook of
Medical Informatics. Heidelberg: Springer–Verlag,
1997.
[43] Van Der Meidjen, M. J., Tange, H.J., Troost, J.,
Hasman, A., "Determinants of Success of Inpatient
Clinical Information Systems: A Literature Review,"
Journal of the American Medical Association, 10,
2003, 235-243.
[44] van't Riet, A., Berg, M., Hidemma, F., Sol, K.,
"Meeting patients' needs with patient information
systems: potential benefits of qualitative research
methods," International Journal of Medical
Informatics, 64, 2001, 1-14.
[45] Willcocks, L., "Managing technology evaluation -
techniques and processes," in Strategic Information
Management: Challenges and strategies in managing
information systems, Galliers, R. D., Baker, B.S.H.,
Ed. Oxford: Butterworth Heinemann, 365-381,1994.
[46] Wyatt, J. C. and Wyatt, S. M., "When and how to
evaluate health information systems?," International
Journal of Medical Informatics, 69, 2003, 251-259,.
[47] Yin, R. K., Case Study Research: Design and Methods,
vol. 5, 3 ed. Thousand Oaks: Sage Publications, 2003.

Table 2: Examples of the evaluation measures of the proposed framework

Technology Human Organisation Net

System Information Service System Use User Structure Environment Benefits
Quality Quality Quality Satisfaction
Ease of use Completeness Quick Level of use Perceived Nature Financing Direct benefits
Ease of Accuracy responsive- (frequency, usefulness Culture Source Job effects
learning Legibility ness duration) User Planning Government Efficiency
Response Timeliness Assurance Attitude satisfaction Strategy Politics Effectiveness
time Availability Empathy Expectations/ Management Localization Error reduction
Usefulness Relevancy Follow up belief Autonomy Competition Communication
Availability Consistency service Knowledge/ Communication Inter- Clinical
Reliability Reliability expertise Leadership organisational outcomes
Flexibility Data entry Acceptance Top relationship Cost
Access to methods Resistance/ management Population
technical Quality reluctance support served
support Training Medical Communication
Security sponsorship

Perpustakaan Elektronik Koleksi: Moch. Boedi Soetanto

File tersedia pada: http://csdl2.computer.org/comp/proceedings/hicss/2006/2507/05/250750095a.pdf

diakses pada: 8 Januari 2007, jam 12:14 WIB

10