A multi-level

monitoring & evaluation

standard
for health systems
Exploring the option

Pragmeta Knowledge Clout e-book 2010-01

Jan Goossenaerts
4/9/2010
Copyright © the author 2010

Publisher
Pragmeta Knowledge Clout
Acacialaan 6, B-2390 Malle
Belgium

Fax: +31-84-7375723
Email: info@pragmetaknowledgeclout.be

2
Goossenaerts, Jan. A multi-level monitoring & evaluation standard –
exploring the option. Malle, Belgium 2010. Pragmeta Knowledge Clout –
e-book 2010- 01

Keywords health system strengthening, dashboard, multi-level, actor

map, problem mess, problem map, sub-Saharan Africa, enterprise
architecture

Abstract
The “health-care work system in transition” is considered to consist of
three action realms in which all stakeholders participate. “Healthcare
Operations” includes the operational activities such as treating patients,
educating potential health workers or performing medical research
within the health-care work system. In the “Monitoring and Evaluation
realm” health outcomes and the performances of actors are monitored
and evaluated. Under-performance will trigger the “Change realm” to
design and implement changes within the two other realms. In support
of Global Health Initiatives’ implementation plan components such as
“do more of what works,” “fostering stronger systems” and “collaborate
for impact” this report contributes:
- A map of stakeholders and key public health-focused interactions,
recognizing different levels in the social architecture;
- A change approach that seeks to leverage enterprise architecture
insights in a multi-level socio-technical context,
- A shared problem map as a collective resource for all public health
actors, as a basis for achieving sustained impact from projects.
- Ideas on a monitoring and evaluation standard, as a basis for lean
programs.

3
Pragmeta Knowledge Clout e-book explained
This e-book uses hyperlinks to embed in its storyline open-access
content resources that are already available on the internet.
Doing this has several advantages:
- One can avoid to reword and repackage existing content, and one
can build upon others’ work in a direct manner,
- Content that is in the internet could evolve and improve between
the point in time when this e-book was produced, and the point in
time that one is reading it, making use of the hyperlink,
- Where internet content is in wikis or blogs that support discussion
or comments, readers can give comments, to further improve the
state of knowledge in the area of concern.
The concept of systematized knowledge commons1 makes the use of
such hyperlinks extra valuable. On the one hand it draws the attention of
readers to the possibility to contribute to such knowledge commons, or
to make use of them in other situations where they can bring benefits.
By testing, using and expanding the systematized knowledge commons,
their quality and utility will gradually improve.
The expectation of improved quality gives a reason for later returns to
specific “knowledge commons.” In this way the e-book becomes a guide
for a journey.
If it is all about commons, why did you have to pay for this e-book?
Because all internet content are intentionally offered free of charge,
your payment for this e-book, and related publications, is the only
means the author uses to recover the costs of developing and providing
an initial set of systematized knowledge commons that would be needed
to demonstrate their feasibility and contribution to development.

1
This concept is explained at http://www.pragmetaknowledgeclout.be/systematized

4
 Preface

The editing of this e-book was prompted by the discussions on the

Global Health Initiative (http://www.pepfar.gov/ghi/index.htm) during
the Global Pulse 2010 Jam (http://www.globalpulse2010.gov/ ), in
particular by a comment on the need for more harmonized outcome and
performance indicators for health systems, contributed by Eckhard
Kleinau2

“Yet, at present we do not have international consensus about the

best metrics for health systems performance nor is there an
agreed upon standard instrument for measuring it. Instead we
have a multitude of indicators and tools used by donors and
international organizations. While these provide useful insights,
they do not allow us to compare systems from different countries
at global or even regional levels; and we cannot measure how
performance changes over time.”

This text includes content from the M.Sc. Thesis of Bas Wonders
(Eindhoven Univ. of Technology, Dec. 2008, The Netherlands).

2
https://www.collaborationjam.com/minijam3/jam/discussionThread.do?gt=true&jamId=57&f=15219&c=10666#10666
(requires access to Global Pulse 2010)

5
Contents

1 Healthcare in sub-Saharan Africa ...................................................... 7

1.1 The Health-care Sector: Resources and Activities ........................ 7
1.2 Stakeholders .............................................................................. 10
1.3 System Boundary Model............................................................ 13
2 Theoretical Background on the Change System .............................. 21
2.1 Institutional Design .................................................................... 21
2.2 Innovation kinds ........................................................................ 23
2.3 Development under architecture .............................................. 23
2.4 The role of Architectural Frameworks ....................................... 26
2.5 Conclusion ................................................................................. 27
3 Articulating a shared Problem Map ................................................. 28
3.1 Deriving a Problem Map ............................................................ 28
3.2 Diagnostic uses .......................................................................... 29
3.3 Conclusion ................................................................................. 30
4 Monitoring and Evaluation Standards ............................................. 31
4.1 Adopting a programmatic approach .......................................... 31
4.2 Monitoring and Evaluation Standards ....................................... 32
4.3 Change in a Multilevel Context .................................................. 36
4.4 Infrastructure ............................................................................ 37
5 Conclusions ..................................................................................... 38
References ............................................................................................. 39
Appendix 1: Stakeholders ...................................................................... 42
Appendix 2: Problem Mess .................................................................... 44
Appendix 3: Problem Map ..................................................................... 46
Allocating Problems to Levels ............................................................. 46
Generalized Dashboard ...................................................................... 49
Appendix 4: Ishikawa Diagram ............................................................... 55

6
1 Healthcare in sub-Saharan Africa

This chapter first describes the health-care sector as it currently prevails

in sub-Saharan Africa. Some health-care figures are given to depict the
major differences between the healthcare in sub-Saharan Africa and in
Western Europe. A categorization of the stakeholders in the sector is
proposed. The chapter concludes with presenting a health-care system
boundary model (UML use case diagram of the health system functions
that are acknowledged in the relevant literature).

1.1 The Health-care Sector: Resources and Activities

The health-care sector in some countries of sub-Saharan Africa is
described: Ethiopia3, Nigeria4, Tanzania5, Uganda6 and Mozambique7.
These aspects are taken into account: the authorities concerned with
health and healthcare, the health-care providers (hospitals and health
centers), the medical education and finally the medical and health
research.

Health-care Figures
Some facts and figures of the countries Ethiopia, Mozambique, Uganda,
Tanzania and Nigeria are highlighted and compared with the health-care
work system in Western Europe (Table 1: Health-care related indicators
source: WHO, 2008Table 1). Comparing the number of the health workers
per 1000 people and available hospital beds per 10.000 persons in these
countries with the same figures for the Netherlands8 and Norway9 shows

3
For a country actor map and some relevant links, see: http://www.actor-atlas.info/country-
map:ethiopia
4
For a country actor map and some relevant links, see: http://www.actor-atlas.info/country-
map:nigeria
5
For a country actor map and some relevant links, see: http://www.actor-atlas.info/country-
map:tanzania
6
For a country actor map and some relevant links, see: http://www.actor-atlas.info/country-
map:uganda
7
For a country actor map and some relevant links, see: http://www.actor-atlas.info/country-
map:mozambique
8
For a country actor map and some relevant links, see: http://www.actor-atlas.info/country-
map:netherlands
9
For a country actor map and some relevant links, see: http://www.actor-atlas.info/country-
map:norway

7
that the health-care capabilities in the sub-Saharan countries are very
limited (WHO, 2008).

Table 1: Health-care related indicators source: WHO, 2008

Ethiopia: Mozambique Uganda Tanzania Nigeria NL Nor
Population 79.09 30.93 40.43 147.98 16.38 4.71
21.37 (2007)
total (millions) (2007) (2007) (2007) (2007) (2007) (2007)
Number of 1,936 2209 822 34923 50854
514 (2004) 14200
Physicians (2003) (2004) (2002) (2003) (2003)
Physicians
(density per 1 0.03 0,02 0,28
0.03 (2004) 0.08 3,15 3,13
000 (2003) (2002) (2003)
population)
Number of 14270 10729 127580
3947 (2004) 14805 221783 67274
Nurses (2003) (2002) (2003)
Nurses
(density per 1 0,2 0,30 1,03
0.21 (2004) 0.55 13,73 14,84
000 (2003) (2002) (2003)
population)
Other health 7345 29,722 1220
1659 (2004) 4128 NA NA
workers (2003) (2002) (2004)
Other health
workers
0,1 0,15 0,82 0,01
(density per 0,09 (2004) NA NA
(2003) (2004) (2002) (2004)
1000
population
Hospital beds
2.0 7.0 12 50 42
(per 10 000 NA NA
(2004) (2004) (2000) (2003) (2005)
population)

Ownership and Planning

Each individual has the right of access to healthcare (United Nations,
1948). Therefore in most countries the healthcare is arranged within the
government by a ministry of health. In Nigeria healthcare is a shared
responsibility of the federal, state and local governments (Min. of Health
Nigeria, 2008). The Ethiopian ministry of health consists of 18 different
departments. The Health Service Extension and Education Center has the
goal to create a healthy society through positive behavioral change of
the people in the major public health problems that can play an active
role in poverty reduction (Min. of Health Ethiopia, 2008). The ministry of
health in Uganda established a resource centre with the task to develop
a comprehensive integrated health information and surveillance system
and to ensure the use of information at all levels; assess the impact of

8
programs/projects and interventions on the health of the people;
periodically review indicators for monitoring and evaluation of the entire
health sector (Min. of Health Uganda, 2008). The Ministry is divided into
three line divisions, two staff divisions and five units. There are for
instance Nursing Services Unit; a Health Services Inspectorate Unit; a
Preventive Health Services Division and a Curative Health Service
Division (Min. of Health Tanzania, 2008).

Hospitals
Many health-care services are provided by hospitals. The last row of
Table 1 lists available hospital beds per 10.000 people. In Ethiopia, with a
population of 79.09 million people, there are 126 hospitals, 600 health
centers, 1662 health stations and 4205 health posts (University of
Oxford, 2008). In Uganda with a population of 30.93 million people there
are 104 hospitals (57 government, 44 NGO, 3 private), 250 health
centers (179 government, 68 NGO, 3 private) 2 palliative care (1
government, 1 NGO) and 1382 other health-care related organizations
(989 government, 352 NGO, 41 private) (Min. of Uganda, 2008).
The ministry of health of Tanzania provides some dated figures about
health-care facilities in this country. In Tanzania there are 173 hospitals
varying from 4 consultancy/specialized hospitals and 17 regional
hospitals to 68 district hospitals and 84 other hospitals. Most of these
hospitals are governmental. Furthermore there are 479 health centers
and approximately 3800 dispensaries. From 1977 to 1991, private
hospitals were banned in Tanzania with the government taking
responsibility for the delivery of most health services (University of
Oxford, 2008).

Medical Education
The figures on the availability of health workers demonstrate the need
for more health workers in these African countries. This urgent need for
health workers requires a different medical education model. In many
African countries a western inspired model is applied for health-care
delivery. This means that 93% of the health-care work force are
professional health workers (doctors or nurses) and limited use is made
of para-professional health workers (health workers who have been
trained to provide a limited range of treatments.) The education of
professionals requires a considerable time and money, and this forms a

9
barrier for expanding the health-care workforce. Furthermore doctors
prefer to live in urban areas. This means that the people who live in rural
areas receive less medical attention. In countries such as Equatorial
Guinea, Ghana and Senegal there are four times as many doctors in
urban areas than in rural areas per head of the population (Conway et
al., 2007).
Many new health workers need to be educated to close this gap in the
health-care work force. Educating professional health workers would
imply that the amount of medical schools in sub-Saharan Africa needs to
increase from the current 90 to 390. Even when the training capacity
increases to this level it will take more than 20 years to close this gap
(Conway et al., 2007).

Health Research, Protection and Assessment Institutes

To provide proper healthcare, health workers need access to up-to-date
knowledge. For new medical knowledge health-care research is needed.
Research institutes are responsible for the execution of this research. In
western countries these medical research institutes are extensive
organizations. In African developing countries these research institutes
often do not exist, and if there are any it is often limited research they
can perform.
In many cases the functions that are typical for National Public Health
Institutes (NPHI) are performed by several institutes. The key functions
of a NPHI are: Health of the population assessment (assessing the health
status of the population); Health protection (surveillance and response);
Research (evidence to inform policies and programs). In several sub-
Saharan African countries there are initiatives for the development of a
National Public Health Institute (NPHI) within the coming years. The
International Association of National Public Health Institutes (IANPHI
FOLIO, 2008) helps developing countries with the development of their
National Public Health Institute.

1.2 Stakeholders
Appendix 1 provides a list of stakeholders grouped at four levels of a
multi-level social architecture10 (see also Figure 1). At pico level the

10
For details on these levels, see: http://www.pragmetaknowledgeclout.be/target-groups

10
human stakeholders are identified, at micro level the organizational
stakeholders and at meso level the sector stakeholders. In addition the
landscape stakeholders operate on the macro level. These macro-
economic actors can operate on global, national, regional or municipal
scale. Typically, at each of these scales, these actors decide the
allocation of resources to different, often competing sectors and causes.
Human individuals at pico level fulfill various roles in the health-care
work system.
- Professional health workers like nurses, doctors and midwives.
- Para-professionals perform limited medical work for which they
have been specifically trained.
- Medical trainers provide the education of health workers.
- Patients are people how need healthcare.
- Any person is a stakeholder, he or she can learn healthy living
habits, become a patient, or a care-giver for relatives or
neighbours.

11
 Human

Trainer
Patient
HealthWorker ParaMedWorker Physician MedicalTrainer Researcher

C_educ_est C_risk_pool
C_hospital CM_educ_est C_med_research

Company

Agency

A_Gov A_educ_min A_NPHI

United Nations Unesco WHO

Figure 1: Typical actors in (global) health care scenarios

At micro level the following stakeholders can be identified:

- The family or community is where people interact and where, in
some cases healthcare takes place.
- Hospitals form an important stakeholder while their core business
is aimed at providing healthcare.
- Medical education establishments educate and train people who
are or will be working as professional health worker in the health-
care sector.
- Educational centers provide general educational services which
could include health components (schools, universities, adult
learning centers).
- Other stakeholders at this level include telecom providers,
pharmacies, insurance companies, the media, etc.

12
Meso level stakeholders are often organizations that are operating to
serve a certain sector, discipline or industry.
- The World Health Organization (WHO) is the directing and
coordinating authority for health within the United Nations (WHO,
2008a). The WHO is an organization that operates in the field of
health and health-care sector. Therefore it is a meso level
Stakeholder operating at a global scale.
- UNESCO is an organization within the United Nations that is aimed
at the realization of peace and sustainability by means of
education, science, culture and communication (UNESCO, 2008).
- IANPHI is an organization that aims to improve the health capacity
by strengthening and linking national public health institutes
(IANPHI, 2008).
- Another stakeholder at meso level is the Integrating the
Healthcare Enterprise (IHE) consortium, which is a collaboration of
suppliers and users of ICT in healthcare (IHE, 2008). This
collaboration is aimed at solving integration problems with ICT
and medical equipment.
- In Europe and other western countries the health-care work
system is already improved by ICT. Connecting for Health helps
the National Health Service (NHS, 2008) in the UK to deliver better
healthcare by using ICT. The NHS is an example of a NPHI.
- The Open Handset Alliance is an alliance of more than 30
technology and mobile companies that are developing Android.
Android is the first free open mobile platform, which enables
innovative applications for mobile phones and PDA’s (Open
Handset Alliance, 2008). This provides opportunities to develop
services and other applications that can lead to a more efficient
and effective healthcare.
- A last meso level stakeholder is the International Institute for
Communication and Development (IICD). IICD is a non-profit
foundation that applies ICT as tool in combination with traditional
media (radio and television in developing countries) (IICD, 2008).
Health care in developing countries is a priority sector of IICD.
An important international stakeholder on the macro level is the United
Nations (UN). The Millennium Development Goals (United Nations
Development Programme, 2008) is an initiative of the UN. Several of the
goals target the improvement of health. Governments form an

13
important group of stakeholders on national level. Changes in
infrastructure as targeted by the Open Handset Alliance could have an
impact that is much broader than transforming single sector. This must
be considered when investing in them.

1.3 System Boundary Model

In the System Boundary Model an overview is provided of “Health
System Functions” that are acknowledged in the relevant literature. Each
interaction is expressed as a use case (oval). Involved actors include both
the stakeholders, and technical systems that will support them.
Within the health-care work system, actors and supporting/provided
sub-systems are identified. Moreover, all sub-systems may interoperate
in a peer-to-peer network. In the network each sub-system may provide
services to the other sub-systems (Service Oriented Architecture). Some
important dependencies among the use cases are also depicted.
Within the Global Health Enterprise Architecture system boundary
model (Figure 3) one or more use cases are separated by (sub-) system
boundaries (rectangle). Each health-care sub-system life-cycle (LC)
comprises three recurring activities: Operational (OP), Monitoring &
Evaluation (ME) and Change (CH)11. This cross-cutting subdivision implies
that each (life-cycle) use case in the System Boundary Model will be
composed of three "sub" use cases, one for Operations, one for
Monitoring & Evaluation and one for Change (Figure 2).

ME_
«uses»

«uses»
CH_ LC_
«uses»

OP_

Figure 2: Life-cycle Use Case "LC_" with "OP_", "ME_" and "CH_" use cases.

A next step in the architecture design of a health-care information work

system is to provide an overview of the different sub-system boundaries,
and identifying the involved use- cases that these sub-systems have to
11
See the action realms in http://www.pragmetaknowledgeclout.be/road-map

14
support. The System Boundary Model identifies what the different
service systems are in the "overall" information work system. The
health-care information work system must be robust and distributed. It
consists of a number of separate systems at different "socio-technical"
levels (pico, micro, meso and macro). Table 2 lists the different use
cases, and it includes a short description for each of them.

Pico
At pico level persons act in five different roles: as Person, as Patient, as
Trainer as Health-worker and as Researcher.
For each person, we assume involvement in Pi_HealthyLivingLearning.
Depending on the resources available (library, newspaper, radio, mobile
phone with internet access and rfid-reader) such learning will happen in
a different way. For people in sub-Saharan Africa, radio is an often used
system. The use case is concerned with the learning by people how to
care for their health during daily life (not smoking, balanced food
choices, active life, etc). People should also be aware of the available
healthcare in their environment. Involved actors within this work system
are persons. In the case of individual learner support (and tracing), a
learning path repository could be used.
A second human role for which health-care work system support is
required is as patient. In case of illness, injury, or when a vaccination is
required, a person participates in healthcare. A “Care and Cure Path” in
which the person, now a patient, will participate can be defined for
further treatment. The use case is called Pi_TreatmentParticipation. For
each "pathology", patients have to follow specific steps, including for
instance taking medicines, resting, etc. The "Care and Cure Path" for a
specific patient could be communicated orally, but when advanced ICT-
tools are used, such information about the treatment could also be
stored in and retrieved from a repository, and reminders for taking
medicines could be produced by handheld devices.
Two use cases are defined for actors in the role of health worker. A first
use case represents the care, cure and treatment of patients
(Pi_TreatmentDelivery). Professional, paraprofessional or other health
workers are involved as actors in this use case. Information about
patients will be stored in an information carrier, preferably in a patient
database (<<System>>TreatmentPathRepository). The second use case is
learning about care, cure and delivery (e.g., to learn about new
therapies, to receive information about epidemics, etc.) in

15
Pi_TreatmentDeliveryLearning. Via the Medical Education and Learning
organizations, also medical trainers could be involved in the learning. A
learning path repository can be used to support the medical learning
activities.
Finally, people can have the role of medical researcher. Within the
research sub-system boundary the use case Pi_Research is defined,
which represents the execution of medical and health-care related
research. Not only new studies about more effective treatment and
medication are executed within this sub-system, also studies about the
effectiveness of certain health-care campaigns are executed.

Micro
Medical trainers and most health workers and researchers are usually
connected to organizations such as schools, hospitals and research
institutions. These micro level actors are addressed next.
The organizations at micro level are involved in one or more of the four
different (sub) systems that are identified namely, those of Education,
Medical Education, Medical Research and Hospital (here understood to
cover centers with sizes ranging from hospitals to health posts). These
four kinds of sub-systems and the involved use cases and actors will be
described in the following section.

16
 Pi_life_S Pi_healthwork_S
Pi_TreatmentPartici «uses»
pation
Pi_TreatmentDeliver
Pi_HealthyLivingLea
y
Pi_research_S
rning
Pi_TreatmentDeliver
yLearning Pi_Research

«uses» Mi_MedRese
«uses»
«uses» arch_S
<<System>>LearningPathRepository

<<System>>TreatmentPathRepository <<System>>TDLearningPathRepository Mi_MedicalResearch

Mi_Edu_S
Mi_Hospital_S Mi_MedEdu_S

Mi_TreatmentService Mi_CareCureDelivery
s Education
Mi_HealthyLivingTea *
ching

<<System>>MedicalLearningContentRepository

Me_Nation_Edu_S Me_Nation_Medical_S Me_Globe_Medical_S *

*

Me_NatKnowledgeStre
aming <<System>>NatEduProg

<<System>>MedicalKnowledgeRepository
*
<<System>>RAGuidelineRepository <<System>>GuidelineRecommendations
*

*
Me_Globe_Edu_S *

Me_MakeResAwareGuid «uses»
Me_ProvideTreatment «uses»
elines
Defs Me_MedicalKnowledge
Me_GlobKnowledgeStr Provision
eaming

<<System>>GlobEduProg

«uses»
«uses» «uses»
«uses»

<<System>>MedicalGuidelineRepository
Macro-Nation
Macro-Global
Ma_GlobPublicGoodsP
Ma_NatPublicGoodsPr *
rovision
ovision *

Gov
United Nations

Figure 3 System Boundary Model12

The education sub-system (Mi_Edu_S), as provided, for example by

schools and radio-stations, contains the Mi_HealthyLivingTeaching use
case which is aimed at providing education to people who (have to)
learn (or be reminded) about healthy living (in
PiPe_HealthyLivingLearning at pico level). At micro-level, the sub-
system is owned by Educational centers like universities and medical

12
In order not to overload the diagram, most actors depicted in Figure 1 are not included in this
model.

17
schools. At pico-level, the education use case involves the teacher or
trainer, and it is used by PiPe_HealthyLivingLearning.
The system boundary Medical Education (Mi_MedEdu_S) delineates the
sub-system that is involved with the training and education of health
workers. Medical students are educated to become a doctor or a nurse.
Short medical courses exist to train para-professional health workers,
which enables them to carry out specific health work. The use case that
represents the actions within this system boundary is
Mi_CareCureDeliveryEducation. The teaching and training activities that
are provided in this use case are used at pico-level by the use case
Pi_HealthyLivingLearning.
Research on medical and health-care related topics are provided by
research institutions. These research institutions are operating within
the system boundary Mi_research_S.
Within these boundaries the use case Mi_MedicalResearch represents
the medical research that is executed at Research Institutions. This
involves Researchers to provide new insights in care, cure and
treatment, and its localization. The results of these research activities
will be considered when providing recommendations for new guidelines.
Finally, in the sub-system Mi_hospital_S the use case
Mi_TreatmentServices can be identified. Care cure and treatment
services are provided to patients who need it. These services are
provided by the different health workers. For the different services
several guidelines are available, that have to be taken into account when
providing treatment. Different guidelines should be available depending
on the form of the availability resources.

Meso
The key function of the meso-level actors is to consolidate communal
quality resources for use at the micro-level organizations. There are four
meso level sub systems: education and learning at national level and at
global level (Me_Nation_Edu_S & Me_Globe_Edu_S) and medical
services at national and global level (Me_Nation_Medical_S &
Me_Globe_Medical_S).
Within both national and global education and learning system
boundary, the use case Me_KnowledgeStreaming covers the functions
that are required for for providing effective learning content.

18
At global medical level the care and cure definitions for micro level
actors are provided by the use case Me_ProvideTreatmentDefinitions. At
national level it is important that these definitions are translated in such
a way that the scarcity of specific resources is taken into account (Walley
& Wilson, 2007). The use case Me_MakeResAwareGuidelines provides
resource aware treatment guidelines for organizations at micro level.
The use case Me_MedicalKnowledgeProvision is aimed at providing
medical knowledge and increasing the availability and accessibility of
this knowledge.

Macro
At macro level a national and global system boundary can be
distinguished. National and global public goods are provided at this level
(use cases Ma_NationalPublicGoodsProvision &
Ma_GlobalPublicGoodsProvision). These levels are not elaborated in
detail in this report.
As was described before a use case is composed of three "sub" use
cases, one for operations, one for monitoring & evaluation and one for
change (Figure 2). To give a clear view of what is meant by this an
example of the three "sub" systems of the use case
Pi_HealthyLivingLearning. The use case "Operational" represents the
way people are learning about how to live a healthy life. These learning
activities can be mapped in a system boundary model. The use case
"Monitoring & Evaluation" measures the performance of the learning
activities and determines if these learning activities are underperforming
compared to predefined standard scores. In case of underperformance a
change project can be started that will involve the use case "Change" to
improve the situation.

Table 2: Overview of Use cases in the System Boundary Model

Use Case
Pi_HeathyLivingLearning
Pi_TreatmentParticipation
Pi_TreatmentDelivery
Description
Provide learning-options on how to improve health
with the available livelihood assets; including means
for patients to know what to do and where to go in
case of illness or injury
Provide means for Patients to comply with treatment
prescriptions.
(para) Professional or other health worker providing
care, cure or treatment services.

19
Pi_TreatmentDeliveryLearning
Pi_Research
Mi_HealthyLivingTeaching
Mi_ CareCureDeliveryEducation
Mi_MedicalResearch
Mi_TreatmentServices
Me_MedicalKnowledgeProvision
Me_NatKnowledgeStreaming
Me_GlobKnowledgeStreaming
Me_MakeResAwareGuidelines
Me_ProvideTreatmentDefinitions
Ma_NationalPublicGoodsProvision
Ma_GlobalPublicGoodsProvision
Table 3: System actors in the System Boundary Model
System Actor
<<System>>TreatmentPathRep
ository
<<System>>LearningPathRepos
itory
<<System>>MedLearningPathR
epository
<<System>>MedicalLearningCo
ntentRepository
<<System>>NatEduProg
<<System>> GlobalEduProg
Health workers in general are educated and trained
by Medical trainers, who are working for an
educational center.
The research performed by researchers
Provide education to people about options on options
on how to improve health; including means for
patients to know what to do and where to go in case
of illness or injury
Providing education for health workers
Performing medical research to gain medical
knowledge which can be used to improve care &
cure
The provision of care &cure in hospitals
The provision of medical knowledge and increase the
availability and accessibility of medical knowledge
Provide available knowledge for educational centers
like schools and universities.
Knowledge is shared among different stakeholders
during education and learning at global level
Provide context specific medical guidelines that take
in account the available resources
Provide definitions for medical treatment in general
Governments and other authorities provide national
public goods
The UN, the WHO and other authorities provide
global public goods
Description
A repository that contains the treatment path of
patients which contains a record of treatments a
patient received.
A repository that contains the learning path of
individual human beings.
A repository that contains the Care & Cure Delivery
learning path of health workers, which contains what
they have learned and what the have to learn.
A repository that contains medical learning content
that can be use re-used an updated for medical
education
A national program which determines the learning
progress of each subject in all the stages of formal
education.
An international program which determines the
learning progress of each subject in all the stages of
formal education.
20
<<System>>MedicalKnowledge
Repository
<<System>>MedicalGuidelineR
epository
<<System>>RAMedicalGuidelin
eRepository
<<System>>GuidelineRecomme
ndations
Existing medical knowledge is stored, used, updated
and new medical knowledge is added to this
repository
Guidelines provide by NPHI's are stored in a
repository from which they can be retrieved by the
ones who need to use them
The same as for MedicalGuidelineRepository but with
that difference that these guidelines are context
specific.
Recommendations about the change of existing
guidelines or about the development of new
guidelines
21
2 Theoretical Background on the Change System

A health-care work system consists of a network of different actors,

varying from individuals to organizations to governments. Effective and
efficient interaction (as part of OP, ME, and CH_ phases of any of the use
cases) between those actors is almost impossible if there are no
agreements concerning these interactions. For an efficient interaction it
is important that these agreements are commonly accepted through the
entire network. An institutional design would provide rules and
regulations to improve the outcomes within a network where demand-
and supply interact (Morioka, 2006).
The health-care work system is constantly reshaped to adapt to the
changing environment (changes in the political environment, in the
technology, or in the natural environment due to climate changes). This
constant change of a work system is called “Transition Pathways in
Sociotechnical Landscapes”. The transition of the health-care work
system can be described according to the multilevel perspective on
transition (Geels and Schot, 2007). Transition is here interpreted as an
innovation of a complete work system.

2.1 Institutional Design

In health-care there is, like in other markets, an interaction between
supply and demand. Suppliers serve market needs by utilizing their
knowledge and technologies. In healthcare health workers, hospitals and
pharmaceutical companies are providing services and products to fulfill
the needs of patients. At the demand side there is a demand for health-
care services (knowledge to care and cure) and medical supplies
(technology). The interaction between these two market forces is almost
impossible when there is no institutional design. Both supply and
demand side interact with the institutional design (Morioka, 2006). For
instance when there are no guidelines for the different medical
treatments, health workers would treat a patient based on their own
experience lacking to benefit from the broader evidence available on
what treatment works in a certain situation. To deal with this kind of
problems an institutional framework has to be designed.
The institutional design is determined by the rules and regulations within
a certain sector, and it provides standards that will improve the
outcomes that result from the interaction between the different

22
stakeholders. Figure 4 is a simplified graphical representation of the
interaction between demand, supply and institutional design.

Demand side:
Demand pull

Institutional
design
· Rules & Regulations
· Standards

Supply side:
Technology push

Figure 4: Interaction Demand, Supply and Institutional Design (Source:

Morioka, 2006 Simplified)

North (1990) describes institution as follows:

“Institutions are humanly devised constraints imposed on human
interaction. They consist of formal rules, informal constraints and the
enforcement characteristics of both.”
These institutions provide a stable structure to human interaction by
reducing the uncertainty involved in these interactions. The accumulated
constraints provide a framework in which human interaction takes place
by telling what humans are prohibited from doing and under what
conditions an individual is allowed to undertake certain activities (North,
1990). The institutions are the rules of the game.
The institutions affect the economic performance of a sector or society
by their effect on the transaction and production costs (North,1990)
When both supplier and customer have to negotiate about the currency
of payment, the amount of the product or the warrantee involved in the
purchasing transaction of a carton of milk, it is obvious that the
transaction costs will rise dramatically.
The development of the institutional design on meso level is common in
conventional industries, where the institutional design is present in the
form of standards and architectures of dominant designs that are shared
within the industry.

23
2.2 Innovation kinds
Time and human conservatism cause many technologies, ideas and
methods of working to become strongly embedded in the sociotechnical
regime. According to Henderson and Clark (1990) there are different
sorts of innovation; incremental, modular, architectural and radical
(Figure 5).

Figure 5: Framework for defining innovation (Source: Henderson & Clark,

1990)
This framework of innovation can also be applied on the transition of a
sociotechnical regime. A regime can be seen as being based upon a
dominant design in which several components are linked together. In
the health-care work system components like websites, hospitals,
learning content and computer systems are linked together in an
architectural framework. The health system transition involves
incremental, radical, modular or architectural innovation.

2.3 Development under architecture

In enterprise work systems, the programmatic approach to change is
supported by enterprise architecture frameworks. Recent research on
society wide architectural frameworks indicates the possibility to scale
enterprise architecture findings to society-scale work systems
(Goossenaerts et al., 2009).
Many initiatives exist to improve the health-care delivery sector in the
African developing countries. Single projects are executed which results
in incremental, often local or regional, improvements. Every time a
single project goes through the regulative cycle, adjustments are made.

24
Because many of these single initiatives only focus on improvements
within the scope of the concerned project, the contribution to the total
health-care program might be neglected. To keep an overview or ensure
cumulative impact of all these projects it is important to approach
project execution from a program perspective. In the program approach
accumulated re-use of projects results facilitates transformational and
technical advantages.
Enterprise Architecture promises and assures such technical and
transformational advantages. The benefits for the organization have
been summarized as follows (TOGAF, 2005):
· A more efficient IT operation
– Lower software development, support, and maintenance
costs
– Increased portability of applications
– Improved interoperability and easier system and network
management
– A better ability to address critical enterprise-wide issues like
security
– Easier upgrade and exchange of system components
· Better return on existing investment, reduced risk for future
investment
– Reduced complexity in IT infrastructure
– Maximum return on investment in existing IT infrastructure
– The flexibility to make, buy, or out-source IT solutions
– Reduced risk overall in new investment, and the costs of IT
ownership
· Faster, simpler, and cheaper procurement
– Buying decisions are simpler, because the information
governing procurement is readily available in a coherent
plan.
– The procurement process is faster - maximizing
procurement speed and flexibility without sacrificing
architectural coherence.
The phases from the FAST methodology (Whitten et al, 2004) can be
mapped to three levels of project execution: direct decision, operations
project, and ICT project (Figure 6). Direct Decisions (V1), Operations

25
Projects (V2) and ICT projects (V3) offer solutions with increasing
knowledge intensity13.
In each project the decision maker and solution engineer depend on
models of the work system. After the decision to (re)develop the work
system, the first phase is scope definition. In this phase the borders of
the project are defined. In the first decision analysis it is decided
whether or not the performance improvement can be realized by a
direct decision. If possible, the improvement is than directly
implemented. If this is not possible, the feasibility of a performance
improvement project is determined. If it is feasible an operations project
is started. The problem or opportunity is then further analyzed, including
its causes. Again, a decision is made whether or not the improvement
can be realized by changes in business operations. If so, then the work
system’s operations are redesigned. If this is not sufficient to achieve the
improvement, then the feasibility of an ICT project is determined. A third
decision is made, whether or not performance improvements are to be
expected from an ICT project. If improvements are to be expected, the
ICT reliance of the work system is designed and implemented. Thus, for
an ICT project the phases D1, D2, O2, O3, I3, I4, I5, I6, O4 and D3 are
carried out.
Life Cycle Stage
Lfe Cycle Focus
Model Maintenance &
Model
Work System Repository Increment Project
real site work
Operation
Increment Execution
system
(cumulative models) Project Charter Using a methodology
Using a methodology
- Problem Statement
Value and Risk Model D3-O5-I8
- Vision Statement Installation Life Cycle Stage
- Function Model
- Assumptions D1-O1-I1 & Delivery
- Context Statement Scope direct Monitoring &
- Scoping Statement Definition D2
- Indicator Register Evaluation
- Goal Model direct Decision
- Value & Risk Register Analysis
- Project Risks direct Using a methodology
O4-I7
Operations Model Operations Delta Construction
- Principal Model O2-I2 & Testing
- Principal D Problem operations Life Cycle Stage
- Asset Model - Asset D Analysis D2<O3 Primary Process &
- Resource Model operations Decision
- Resource D Analysis
Asset
Maintenance
- Interaction Refinements
- Interaction D operations
Using the assets & IT
I3 I6
ICT Models ICT Models Delta Requirements Physical
Analysis Design &
- Requirements Model - Requirements D ICTA Integration
- Component Model - Component D I4 ICTA
Logical
- Platform Specific Model - Platform Specific D Design
ICTA D2<O3<I5
Decision
Analysis
ICTA

Figure 6: 3-V system development facilitated by cumulative models and

their increments

The 3-V system development model provides a basic workflow for work
system (re)development projects. Furthermore, the model provides
sufficient detailing to support ICT-reliant work system (re)development
13
For more details, see: http://www.ens-dictionary.info/mr-source-3

26
projects. However, the model does not contain clear deliverables which
can be used as guidelines for project execution.
Berre et al. (2004-2006) have developed a use case driven, model-
focused approach to system development, called COMET (component
and model-based development methodology). Although less suitable for
project management, the COMET approach provides a framework to
stratify the models in a repository that are used to depict a system’s
current or desired architecture. Three modeling layers are useful: Value
and Risk Model, (Work system) Operations Model and ICT-related
models. The Project Charter and its Operations Delta and ICT Models
Delta (deliverables) are realized during the execution of multiple
projects that singly and severally affect the work system (processes, ICT-
solutions deployed).

2.4 The role of Architectural Frameworks

In contrast with the build-to-order mindset that has long dominated
Information System development methods and practices (Kaplan et al.,
2004), the adoption of Enterprise Architecture principles implies a strict
separation of work system repository on one hand, and project charters
and deliverables on the other hand. The work system repository
contains the consolidated or cumulative models which should be
maintained during the work system life time, especially for "change-
intensive" work systems (architectural framework). Under "enterprise
architecture", it is important for effective work system life cycle
management to strictly separate the model repository from the work
system (in the health-care work system) on one hand, and project
charters and deliverables on the other hand. The work system
repository is enacted during a system’s operation and maintenance life
cycle, and contains the consolidated or cumulative models resulting
from all past changes in the work system.
When the situation of healthcare in Sub-Saharan African countries is
described from an architectural point of view this could be done by using
the four levels of the multi level perspective. The landscape (macro
level) can be described at global; national and regional level. The
architectural descriptions at this level are generic and are sourced from
elsewhere (Goossenaerts et al., 2009).

27
Table 4 Architecture overview
Level Program support Reference Projects
Macro Landscape Architecture
Meso Sector Architecture “Transforming Government
using Open Standards” NHS – J.
Grewal (2007)
Micro Enterprise Architecture There are many examples of
enterprise architectures that
have proved to be successful
Pico Personal Architecture. (learning the use of mobile
phone)

2.5 Conclusion
The effectiveness of interventions in the multi-actor health system can
be increased by using standards and regulations for interaction.
Regarding desirable programming standards, here a few suggestions:
- Use the multi-level perspective in classifying actors;
- Recognize the three action realms as pillars for effective change;
and adopt the “collective reflection competitive action (CRCA)”
principle14 in determining the scope and approaches of
interventions
- The implication of the CRCA principle for monitoring and
evaluation approaches is explored in the next chapters. One
recommendation is to articulate a shared problem map as a
collective resource for all stakeholders in country health plans
(Chapter 3). The efficient deployment of such shared problem
maps requires monitoring and evaluation standards (Section 4.2)
- Create a multi-level (model driven) programme environment.15
(Section 4.3)

14
http://www.pragmetaknowledgeclout.be/collective-decision-frames#toc12
15
For details, see: Goossenaerts et al. (2009)

28
3 Articulating a shared Problem Map
One recommendation that was derived from bringing enterprise
architecture methods to health system strengthening is the use of a
shared problem map as a collective resource for all stakeholders in
country health plans.
The derivation and utilization of such a map is briefly explained.

3.1 Deriving a Problem Map

Many studies articulate the health-care vulnerabilities in developing
Africa. By extracting these problems from different studies a problem
mess emerges (Appendix 2).
A problem map can provide further insight in problem mess. There are
two structuring steps to create a problem map: the different problems
are grouped according to the multi level perspective (Goossenaerts,
2007), problems and factors are allocated to the different stakeholders
that have been identified. A distinction is made between stakeholders
acting at pico, micro, meso and macro level.
For each actor, different perspectives are used to distinguish problems
more accurately and group them horizontally. Factors are related to
generalizations of the four Balanced Scorecard perspectives namely P1:
Customer; P2: Internal business processes; P3: Learning & Growth; and
P4: Financial (Goossenaerts, 2006). The first generalization (P1) External
stakeholders, collaborative processes & social capital, reflects the
dependency of an actor on external relations, customers, suppliers and
allies. The second generalization (P2) measures the performance of the
joint manifestation of internal assets and the processes in which they
contribute to the value creation by the actor. The third generalization
(P3) is aimed at sustainability, competitiveness and innovation. Finally
the fourth generalization takes into account the value flows and stock
(money, material, information and knowledge).Appendix 3 gives a
tentative and incomplete problem map. The main purpose of the map is
to illustrate problem mapping using a multi level perspective.
The indicator profiles, measured and target values that would indicate
the severity of a problem are not included.

29
3.2 Diagnostic uses
Huynen et al. (2005) proposed a framework in which complex multi-
causality of a population’s health is conceptualized. The framework
differentiates between four different groups of factors determining the
health of the population namely: social-cultural, economic,
environmental and institutional factors. Furthermore a hierarchy is
proposed to make a distinction between factors that are positioned in
the beginning or at the end of the causal chain. A distinction is made
between proxal, distal and contextual factors. Proxal factors are factors
that directly cause diseases or health gains; distal factors are positioned
further back in the chain and affect health of the population via
intermediary factors; and contextual factors are factors at macro level
that are shaping the proxal and distal factors.
At proximal level three different determinants of health of the
population can be distinguished: health services, social environment
lifestyle and physical environment. Health services are the care and cure
services provided by the health-care work system. If there are problems
with the quality or the availability of health services this directly affects
the population’s health.
The social environment affects health. An environment with much
domestic violence or busy dangerous traffic may cause many injuries.
Lifestyle has a direct effect on the population’s health because this
reflects the way people behave; do they have a healthy way of eating, do
they have safe sex or do they drink and smoke a lot?
The physical environment has a direct effect on the population’s health
as it represents the quality and availability of water and food. The quality
of the environment caused by for instance disease prevalence, natural
hazards and the weather also directly influence health. Because the
physical environment is not within the scope of intervention, to improve
the health situation this determinant will not be taken into consideration
in this research. These proximal factors are caused by the distal factors.
Based on this framework a cause and effect diagram is made in which
proximal, distal and conceptual levels are distinguished (Appendix 4).
Also a distinction is made based on the nature (institutional, economic,
social-cultural, and environmental) of the problem factors. The cause
and effect analysis helps deriving the root causes for the poor
population’s health.

30
Also other diagnostics techniques can be used to identify causal chains
and cycles (vicious cycles).16

3.3 Conclusion
Problem messes can be structured by allocating problems and factors to
various actors at multiple socio-technical levels, by their causal
relationships, and by factor distance from specific actors.

16
See for instance at: http://www.pragmetaknowledgeclout.be/collaborative-diagnostics

31
4 Monitoring and Evaluation Standards

A second recommendation that was derived from bringing enterprise

architecture methods to health system strengthening is the use of
monitoring and evaluation standards. The need for such a standard is
explained in reference to the regulative cycle,17 a pattern that provides a
structure for the sustainable and recurring development of any work
system.

4.1 Adopting a programmatic approach

In enterprise work systems, the programmatic approach to change is
supported by enterprise architecture frameworks. Recent research on
society wide architectural frameworks indicates the possibility to scale
enterprise architecture findings to society-scale work systems
(Goossenaerts et al, 2009).
Many initiatives exist to strengthen the health system in the African
developing countries. Single projects result in incremental, often local or
regional, improvements. Every time a single project goes through the
regulative cycle, local adjustments are made. While many initiatives
focus on improvements within a narrow scope, the contribution to the
overall health-care capacity building and synergies might be neglected.
Often when successful project results are transferred to a new situation
they exhibit unintended effects (Madon et al., 2008).
Using a programmatic approach one can keep an overview of ongoing
initiatives, and ensure their cumulative impact and synergies. The
cumulative consolidation and re-use of projects results facilitates
transformational and technical advantages. We explore some
advantages for the health care operations, the change system, and for
the monitoring and evaluation.

Change System and Operations

The actual implementation of changes in the health care operations
system is out of the scope of this report. Rather the focus is on the
conditions that stakeholders must meet to effectively absorb
innovations.
17
For references see: http://www.pragmetaknowledgeclout.be/crb-methodology

32
To realize the transition from a single project approach towards the
program enabled way of working requires awareness among all
stakeholders including health workers, hospitals, schools, research
institutes governments and other nongovernmental organizations.
Organizations such as IANPHI can play an important role in creating such
awareness. IANPHI aims to improve national public health institutes by
developing a framework and a toolkit for the assessment and guiding of
the development of these institutes (IANPHI, 2007).
For a meaningful transition towards a program approach a clear
overview of the health-care work system is required. The different sub-
systems including the involved actors and their activities need to be
more accurately mapped. The system boundary model in chapter 1
provides an initial overview for a (small) part of the health-care work
system, such that dependencies among the various actors become
visible. Based on such a map specific troublesome operations can be
subjected to more specific monitoring and evaluation, which could then
facilitate targeted initiatives that fully exploit the already existing
capabilities. Positioned within a shared map of the health care system,
change projects can better aligned to contribute to the cumulative
development of the entire work system.

4.2 Monitoring and Evaluation Standards

Monitoring and evaluation is aimed to gather information about the
health outcomes and actor performance. At the level of the landscape
this information can be obtained by measuring figures such as child
mortality rates, life expectancy, traffic casualties, or the number of
hospital beds per head of the population. These indicators display the
performance on certain topics in healthcare. The World Health
Organization (WHO) registers these indicators and makes the figures of
all countries available of the past 18 years. Besides these figures from
the WHO, the health departments of governments and other national
and international nongovernmental organizations have their own, often
more fine-grained measures (per geographic area, per hospital) to
monitor the national health-care work system.
Alva et al., (2009) have summarized current efforts in measuring health
system performance and highlight the indicators and performance
benchmarks most frequently used by the global community. This work

33
has been the basis of the assessment and question that Eckhard Kleinau
has contributed to Global Pulse 201018:
“Yet, at present we do not have international consensus about the
best metrics for health systems performance nor is there an
agreed upon standard instrument for measuring it. Instead we
have a multitude of indicators and tools used by donors and
international organizations. While these provide useful insights,
they do not allow us to compare systems from different countries
at global or even regional levels; and we cannot measure how
performance changes over time.”
At the patient’s end of the health care constellation, the International
Classification of Functioning, Disability and Health19 provides an
instrument that can be used for functional status assessment, goal
setting and treatment of an individual.
Currently the initiatives and goals for monitoring and evaluation are
fragmented: across the levels, from macro to pico, and for different
countries, health institutes and other organizations. This means there is
no comprehensive overview, making it difficult to communicate, use and
re-use the knowledge that others have already acquired. Because there
are no standards, stakeholders may be spending much more than the
optimum for monitoring and evaluation.
Many different health-care related aspects will be monitored for many
different countries and even for specific regions and stakeholders. When
a significant amount of this knowledge is gathered in an agreed
framework, and made available via repositories, the scores for standard
indicators can be compared with those of peers, at any socio-technical
level. At each level, each kind of actors can then receive guidance on
problems and improvement targets. At the proximal level, evaluations
and benchmarking20 can provide information about the most urgent
need for improvement.

18
https://www.collaborationjam.com/minijam3/jam/discussionThread.do?gt=true&jamId=57&f=15219&c=10666#10666
(requires access to Global Pulse 2010)
19
Details and classification browser at:
http://www.who.int/classifications/icf/appareas/en/index.html , the ICF Checklist at:
http://www.who.int/classifications/icf/training/icfchecklist.pdf
20
Benchmarking within peer-communities is one kind of service that can help actors to identify
improvement targets: sharing of externalized performance figures triggers search for knowledge that
could be combined and internalized for improved performance. Factor endowments and actor-
context can co-determine the suitability of specific change initiatives for the actor. Peer experience-
sharing group seem to be advisable in an eco-system with roles that are as diverse as in the medical
profession (see the classification of health occupations in the International Standard Classification of

34
Monitoring
Monitoring is aimed at getting a clear view on the performance of
different actors within the health-care work system. This is done by
gathering scores for certain indicators. The results of these measures
should be combined in a scorecard (or dashboard). Scorecards capture,
structure and communicate the performance and support comparing it
with pre-set goals and objectives.

Factor Coverage
Fragmentation in the monitoring & evaluation realm is the result of
many initiatives adopting “initiative-specific” decision frames, with
insufficient attention being given to consolidating decision frame
elements for shared use or for comparison. The Joint
UNECE/OECD/Eurostat Working Group on Statistics for Sustainable
Development contrasts the use of “policy-based” indicator sets, with
“capital-based” indictor sets (UNECE, 2009). In a broad capital-based
view a society’s capital base is seen to comprise these five individual
stocks: financial capital, produced capital like instruments and
medicines, natural capital in the form of natural resources, land and
ecosystems providing services, human capital in the form of an educated
and healthy workforce, and social capital in the form of functioning
social networks and institutions.
By using the generalized scorecard (Goossenaerts, 2006) these various
forms of capital can be made visible in the evaluation and monitoring.

Overcoming Fragmentation
To overcome the problem of fragmentation of goals and initiatives an
architecture in which these scorecards are developed, stored, used and
re-used can play an important role. Such architecture contains (Figure 7)
repositories for different scorecards, a database of scores by actor
profile, guidelines for the development of new scorecards and guidelines
for the use of scorecards.

Occupations (ISCO) (see: Options for the Classification of Health Occupations in ISCO-08)
http://www.ilo.org/public/english/bureau/stat/isco/docs/healthocc.pdf ; Current draft Group
Definitions: Health) with factor endowments that vastly differ (from the health post in a Sub-Saharan
African country to a university hospital in an OECD member country).

35
Ownership
To improve the quality of the knowledge captured by using the
generalized scorecard, in such a way that it can be communicated within
the network requires a good set of guidelines that reflect the particulars
of the actors and systems at the different levels. Guidelines must be
applicable for many or all actors. Standards and regulations, or codes of
conduct, are needed to improve the comparability and accumulation of
data sourced from different actors in different countries. The
institutional design will play an important role in providing such
standards. The National Public Health Institutions (NPHI) that exists in
many countries should lead the convergence in applying such guidelines.
“A NPHI is a science-based organization or network of
organizations that provides national leadership and expertise to
efforts to achieve substantive, long-term improvements in the
public’s health.” (IANPHI, 2007)

IANPHI, an organization that is contributing to the institutional design,

should have the development of standards and guidelines on its agenda.
The existence of a standard for monitoring and evaluation will enable
the development of a comprehensive monitoring tool, it will support the
use of benchmarking, and it will allow health workers to dedicate more
time (again) to the health care itself. The guidelines should also ensure
the capturing of information on the context of the monitored actor.

36
 Develop Scorecard Guidelines for Scorecard
Development
Use Guidelines for
Development

Store in Database

Healthcare proces

GSC Guidelines for Scorecard

Selection GSC
Measuring Performance
Database

Select Scorecard

Result Measure

Store measure
GSC
Scores
Database
Compare Measure

Performance Diagnosis

Figure 7: Monitoring and evaluation architecture

Evaluation
The scores of the scorecards will be stored in a repository. If there is a
need for the performance evaluation, scores in this repository can be
used to determine what the relative performance of specific actors of a
health-care work system is. Based on such evaluation the goals can be
determined for the incremental projects in the change phase. After (or
even during) the execution of the incremental projects the evaluation
can be used to verify that projects have their intended effect.
Furthermore changes in the context over a period of time should be
considered to prevent unintended effects. The possibility of short-term
feedback on the impact of initiative will allow actors to adjust their
actions, if needed. If something doesn’t work, this can become visible
soon.

4.3 Change in a Multilevel Context

When monitoring and evaluation is instrumented for multi-level socio-
technical systems, the stakeholders close to the factors could conclude
that in the observed situation they are under-performing. To change the

37
situation the stakeholder close to the factors and under-performance
should start a change project to improve the performance.
In most African countries such initiatives and projects exist, but
synergies are often missed (WHO, 2009). Projects and initiatives tend to
be executed singly and severally, by stakeholders with a temporarily
engagement with the work system.
A multi-level program approach should be developed that provides clear
objectives at all levels, and for all stakeholders. When starting a project,
the deliverables should be aligned with the goals stated in the country
health plan, and with the factors proximal for the change planner and
agent.
As projects may exhibit unintended effects when transferred to a
different setting (Madon et al., 2007), it is necessary to capture
information about the context in which an incremental project is
executed.
The scarce financial and human resources make it impossible to execute
all necessary incremental projects at once. Therefore priorities have to
be set. Surveys can be used to classify incremental projects according to
the expected impact and costs. Experts knowledgeable about the scope
of the increment can be asked to give their opinion about the expected
impact. The 3-V system (Figure 6) gives an overview about how to
arrange the knowledge required for the incremental projects at a single
level, with repositories of multiple levels linked as explained by
Goossenaerts (2009) to support a multi-level program.

4.4 Infrastructure
An infrastructure to capture and share monitoring & evaluation data,
alongside medical knowledge should be part of the strengthened health
system. As was mentioned in the introduction of this report the
emergence of internet based and mobile wireless technologies provides
many opportunities to share the health knowledge among the actors.
The available infrastructure in sub-Saharan Africa should be taken into
account when developing a format for the exchange of knowledge.
Aspects like the availability of electricity, bandwidth for data sharing and
size of screens (of a simple personal computer, a mobile phone or PDA)
need attention when developing this infrastructure. Also in case of
failure of components within this system the core functions of the health
system must remain operational.

38
5 Conclusions
For every stakeholder in the overall health system the own share in the
work system can be divided into a “share” Operations, a “share”
Monitoring & Evaluation and a “share” Change. Furthermore the use of
the multi level perspective and a generic system boundary model of the
healthcare sector provide a clear overview of the involved actors, their
dependencies and service needs.
A multi-level program approach for change projects should lead to a
more effective and sustainable development of the health-care system.
Governments and multi-lateral organizations must understand the
relevance of such an approach to their policy implementation.
The subject of monitoring and evaluation can vary from health workers
at pico level to governments at macro level. The actors involved in
developing dashboard-structures (common dimensions and per
dimension a choice of indicators) are mainly research organizations,
NPHI’s, health departments of governments and national and
international NGO’s and specialized organizations such as WHO. All
other stakeholders must become users of customized dashboards
(including checklists), they must be confronted with their own and peers
performance as a trigger for learning and action. The critical actors to
engage are micro and meso level actors.
For an effective use and development of dashboards it is necessary that
the standard covers all relevant factors for the various actors. A cross-
level health-system dashboard standard can support the creation of
customized dashboards for all participants in the work system. While a
problem could be signaled by benchmarking one’s performance with
that of peers, the selection of an intervention must be informed also by
situational factors as visible in the dashboards of other actors involved.
A thorough problem analysis would involve the mapping of cross-level
factor dependencies.
Both the definition of (performance) indicator profiles and dashboards,
data collection, problem analysis and diagnosis are time consuming
activities. Rather than dedicating much time to such activities in each
and every project, and for each situation, tools can be developed to map
and analyze problems more efficiently, collaboratively and increasingly
accurate.

39
References

Literature:
Alva, S., Kleinau, E., Pomeroy, A., Rowan, K., November 2009. Measuring the impact
of health systems strengthening. Tech. rep., USAID. URL
http://www.usaid.gov/our\_work/global\_health/hs/publications/impact\_hss.pdf
Berre, A.-J. et al. (2004-2006) COMET – Component and Model-based development
Methodology; Methodology Handbook, http://modelbased.net/comet/index.html
Conway, M. D., Gupta, S., Khajavi, K., (2007), “Addressing Africa’s health workforce
crisis”, The McKinsey Quarterly, November
Frischmann, B. M., (2005), “An Economic Theory of Infrastructure and Commons
Management”, Loyola University Chicago School, United States
Geels, F. W., Schot, J., (2007), “Typology of sociotechnical transition pathways”,
Research Policy 36, pp. 399-417
Goossenaerts, J.B.M., (2006), “Monitoring, Evaluation & Decision Interoperability in
Networked Operations”, Proceedings of Workshops and Doctoral Consortium at
CAISE' 06. Ed. T. Latour and M. Petit June 2006, 664-668.
http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-200/13.pdf
Goossenaerts, J.B.M., (2007), “Institutions for pro-growth conduct in the knowledge
economy.” Goossenaerts, Jan B. M. "Institutions for Pro-Growth Conduct in the
Knowledge Economy." Social Science Research Network Working Paper Series (2008)
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1121643
Goossenaerts, J.B.M, Iluyemi, A. and Lal, K. (2008) Global Health Enterprise
Architecture and the Pathway to Health Outcomes from Science Data, Abstract
submitted to: The 21st CODATA conference on "Scientific Information for Society -
from Today to the Future" in Kiev, Ukraine, Oct. 2008 (
http://www.codata08.org.ua/) http://www.scribd.com/doc/28498779/Global-
Health-Enterprise-Architecture-and-the-Pathway-to-Health-Outcomes-from-Science-
Data
Jan B. M. Goossenaerts, Alexander T. M. Zegers, and Jan M. Smits. A multi-level
model-driven regime for value-added tax compliance in erp systems. Computers in
Industry, 60(9):709–727, December 2009.
Guy, F. Strategic bundling: Information products, market power, and the future of
globalization," Review of International Political Economy, vol. 14, no. 1, pp. 26-48,
2007. [Online]. Available: http://dx.doi.org/10.1080/09692290601081145
Henderson, R. M., Clark, K. B., (1990), “Architectural Innovation: The Reconfiguration
of Existing Product Technologies and the Failure of Established Firms” Administrative
Science Quarterly, Vol. 35, No. 1, Special Issue: Technology, Organizations, and
Innovation. pp. 9-30
Hess, C., Ostrom, E., (2007), “Understanding Knowledge as a Commons”, Cambridge,
Massachusetts, London, England, the MIT Press

40
Huynen, M. T. E., Martens P., Hilderink, H. B. M, (2005), "The health impacts of
globalization: a conceptual framework”, Globalization and Health, 1:14
IANPHI FOLIO, (2007), “Framework for the Creation and Development of National
Public Health Institutes”, IANPHI FOLIO, No1
Iluyemy, A., Briggs, J., (2007), “Exploiting EU-IST Mobile eHealth Programmes for
Developing Health System and Human Resources in Africa: A Short Compendium and
Policy Implications”, Centre for Healthcare Modelling & Informatics, School of
Computing, University of Portsmouth, Portsmouth, United Kingdom
Kaplan, R. S., Norton, D. P., (2004). “Measuring the strategic readiness of intangible
assets.”, Harvard Business Review, 82(2): 52-63
Madon T., Hofman, K. J., Kupfer, L., Glass, R. I., (2007), “Implementation Science” ,
Science, Vol. 318., no. 5857, pp 1728 – 1729
Morioka, T., Saito, O., Yabar H., (2006), “The pathway to a sustainable industrial
society – initiative of the Research Institute for Sustainability Science (RISS) at Osaka
University”, Sustainability Science, Vol. 1, No. 1 pp 65-82
North, D. C., (1990), “Institutions, institutional change and economic performance”,
Cambridge, Cambridge University Press
Sluijs, M.B., Veeken, H., Overbeke, A.J.P.M., (2006), “Gebrekkige informatie in
ontwikkelingslanden: internet alleen is geen oplossing”, Nederlands Tijdschrift voor
Geneeskunde, 150(24), pp 1351-1354
Strien, P. J. van, (1997), “Towards a Methodology of Psychological Practice, the
Regulative Cycle”, Theory and Psychology (7:5), pp 683-70
UN-ECE, 2009. Measuring sustainable development. Tech. rep., UNECE Statistical
Division. URL
http://www.unece.org/stats/publications/Measuring\_sustainable\_development.p
df
Whitten, J.L, Bentley L.D., Dittman, K.C. Systems Analysis and Design Methods, 6th
edition, McGraw Hill, Irwin, Boston, 2004
WHO (2004) "World report on knowledge for better health - strengthening health
systems," World Health Organization, www.who.int, Tech. Rep., 2004. [Online].
Available: http://www.who.int/rpc/meetings/pub1/en/index.html
WHO (2009) World Health Organization Maximizing Positive Synergies Collaborative
Group, June 2009. An assessment of interactions between global health initiatives
and country health systems. The Lancet 373 (9681), 2137-2169.

Websites:
Grewal, J., (2007), NHS Connecting for Health, presented at the OASIS Open
Standards forum in London, http://events.oasis-open.org/home/sites/events.oasis-
open.org.home/files/Jagdip.v2.ppt, last visited on 05-11-08
E-learning in Global Health, (2008), University of Oxford, Oxford, UK
http://tall.conted.ox.ac.uk/globalhealthprogramme/report/default.asp

41
IANPHI, (2008), http://www.ianphi.org/ , last visited on 05-11-08
IANPHI UVRI, (2008), http://www.ianphi.org/where_we_work/uganda_
_uganda_virus_research_institute__uvri_/ last visited on 05-11-08
IHE, (2008), http://www.ihe.net/ , last visited on 05-11-08
IICD, (2008), http://www.iicd.org/about, last visited on 05-11-08
Min. of Health Ethiopia, 2008, http://www.moh.gov.et/ , last visited on 05-11-08
Ministry of Health Nigeria, 2008, http://www.nigeria.gov.ng/NR/exeres/CC1F04E3-
5DC4-414C-A8B7-2FF98B3CD9D0.htm, last visited on 05-11-08
Ministry of Health Tanzania, (2008), http://www.moh.go.tz/ last visited on 04-11-
2008
Min. of Health Uganda, 2008, http://www.health.go.ug
,http://www.health.go.ug/health_units.htm , last visited on 04-11-2008
NHS, (2008), Connecting for Health, http://www.connectingforhealth.nhs.uk/ last
visited on 03-10-2008
OECD, (2008), http://www.oecd.org/document/20/0,3343,en_2649_35845581_
35023444_1_1_1_1,00.html, last visited on 05-11-2008
Open Handset Alliance, (2008), http://www.openhandsetalliance.com/ last visited on
26-09-2009
SciDev, (2008), http://www.scidev.net/en/features/how-mobile-phones-contained-
kenyan-polio-outbreak.html last visited on 2-10-2008
TOGAF, (2005), "The Open Group Architecture Framework,
http://www.opengroup.org/architecture/togaf8-doc/arch/toc.html, last visited on
26-09-2008
UNESCO, (2008), www.unesco.org, last visited on 05-11-2008
United Nations, (1948), Universal Declaration of Human Rights,
http://www.un.org/Overview/rights.html last visited on 28-11-2007.
United Nations Development Programme (2008), http://www.undp.org/mdg/, last
visited on 05-11-2008
University of Oxford, 2008, E-learning in Global Health for Africa,
http://tall.conted.ox.ac.uk/globalhealthprogramme/report/default.asp, last visited
on 24-10-2008
WHO, (2008), http://www.who.int/whosis/database/core/core_select.cfm last
visited on 05-11-2008
WHO, (2008a), http://www.who.int/en/, last visited on 05-11-2008
Worldbank (2007) http://ddp-
ext.worldbank.org/ext/ddpreports/ViewSharedReport?&CF=&REPORT_ID=9147&RE
QUEST_TYPE=VIEWADVANCED& HF=N/CPProfile.asp&WSP=N, last visited on 17-
11-2008

42
Appendix 1: Stakeholders

Within the Multi-Level Perspective a distinction is made between

stakeholders at four different levels. At pico level the human
stakeholders are identified, at micro level the organizational
stakeholders and at meso level the sector stakeholders. In addition the
landscape stakeholders operate on the macro level. These macro actors
can operate on global, national or regional and local scale. Typically, at
their scale, macro actors decide the allocation of resources to different,
often competing sectors.
An important international stakeholder on macro level is the United
Nations (UN). The Millennium Development Goals (United Nations
Development Programme, 2008) is an initiative of the UN and some of
the development goals are aimed at the improvement of health in the
world. Governments form an important group of stakeholders on
national level.
Table 5: List of stakeholders in the sub-Saharan African Healthcare
Stakeholders Role
Personal level (Pico)
In the context of this research each person lives a life, making
Person/ Human choices among available options (habits, learning,...) . A person
can become a patient, a (para) medical professional, a trainer, etc.
A person for whom the need to participate in health-care activities
(treatment) has been confirmed (e.g., for chronic diseases). A
Patient
distinction can be made between patients living in urban or in rural
areas.
Someone who performs medical/health research and publishing
Researcher
research findings
Para Professional A person who has been educated and trained to provide a limited
Health Worker (Para range of treatment steps following prescriptions by medical
Worker) professionals (including for instance nurses, physical therapists,...)
A person with a medical degree who performs medical work such
Professional Health
as diagnosing patients, prescribing treatments, perform
worker (Medical
interventions, and follow up treatment results. A distinction can be
professional, Physician,
made between health workers working in urban or rural areas or
GP, Specialist,...)
working abroad.
Someone who educates, or provides further education to health
Medical trainer
workers (medical and para medical).
Trainer/Teacher Someone who provides training or education.
Organization / Company level (Micro)
An organization where health workers (medical and para medical)
Hospital
providing care, cure and treatment to patients.

43
 An organization that provides educational services in general,
Educational centre including health components (schools, universities, adult learning
centers,...)
Medical educational An organization that educates and trains people who are or will be
centre working as professional health worker in the health-care sector.
Medical Research (lab) Companies or other organizations that do medical research.
Organization that provides mobile calling and data sharing for
Telecom Provider
mobile phones and PDA's
Risk Pool/ Insurance Providing a financial service to pool risks related to the costs of
company health-care.
Sector / Industry level (Meso)
An organization or network of organizations aimed at improving
healthcare in a country. It provides national leadership and
expertise for a country’s effort to protect and improve health. It
National Public Health authorizes health related learning content for the identified target
Institute (NPHI) groups in the country. Related organizations are:
- IANPHI http://www.ianphi.org/
- IHE http://www.ihe.net/
- NHS http://www.nhs.uk/Pages/homepage.aspx
World Health organization. Providing leadership on global health
matters, shaping the health research agenda, setting norms and
standards, articulating evidence-based policy options, providing
WHO technical support to countries and monitoring and assessing
health trends. Contributions to governance of the discipline
concerned with medical and health research and with translating
this knowledge.
Contributing to peace and security by promoting collaboration
among the nations through education, science and culture in order
to further universal respect for justice, for the rule of law and for
the human rights and fundamental freedoms which are affirmed
UNESCO
for the peoples of the world, without distinction of race, sex,
language or religion, by the Charter of the United Nations.
Contributions to governance of the discipline concerned with
education and learning, particularly so for developing countries.
It has responsibility for education programming and provision
Ministries of Education within a certain country; contributions to governance of the
national education & learning sector.
Responsibility for health-care provision within a certain country.
Contributions to governance of national medical and health
Ministries of Health
research and with translating this knowledge for the national
context.
Sector in which ICT related companies are operating in. Related
organizations are:
Information systems - IICD http://www.iicd.org/
- Open handset alliance http://www.openhandsetalliance.com/

Sector that provides mobile communication and data sharing

Telecom providers
services
Macro-economic level
Governing a country, including the allocation of public budgets to
Government
health-care and education
Facilitate cooperation in international law, international security,
UN economic development, and social progress and human rights
issues.

44
Appendix 2: Problem Mess
Problem
Hunger
Poor Hygiene
Poor Sanitary infrastructure
Poor Traffic infrastructure
Religious pressure
Population deviation
Poor ICT infrastructure
In many African countries many people have to
deal with health problems.
There is no money to invest in these problems
In Africa there is a lack of health workers
Classical training model costs too much time
and money
There are not enough medical schools
Exodus of professional health workers (doctors,
nurses, midwives)
Low wages for health workers in these countries
Low job security
Bad working conditions
Relative few paraprofessionals
Limited productivity of health workers
Poor infrastructure
Scarce medical supplies
Lack of knowledge affects productivity of health
workers.
Lack of productivity measures
Availability of medical knowledge among health
workers
There are difficulties in allocating resource.
Lack of measures in resources utilization
Lack of result measures of health-care work
system
Lack of good morale among African health
workers
No good incentive system
Low involvement of NGO’s in medical training
Lack of private training programs
There is a gap between innovations in health
and the delivery to communities in the
developing countries.
Implementation of innovations is untested,
unsuitable or incomplete
Delivery schemes are less likely to be subject to
rigorous scientific research
45
Reference
Common facts
Common facts
Common facts
Common facts
Common facts
Common facts
Common facts
(Conway, Gupta and Khajavi, 2007)
(Madon, Hofman, Kupfer and
Glass, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Madon, Hofman, Kupfer and
Glass, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Iluyemi & Briggs, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Conway, Gupta and Khajavi, 2007)
(Madon, Hofman, Kupfer and
Glass, 2007)
(Madon, Hofman, Kupfer and
Glass, 2007)
(Madon, Hofman, Kupfer and
Glass, 2007)
People living in poverty often have to deal with (Madon, Hofman, Kupfer and
social constraints Glass, 2007)
Limited knowledge of preventive health (Madon, Hofman, Kupfer and
practices among patients in Africa Glass, 2007)
Insufficient access to quality care (Madon, Hofman, Kupfer and
Glass, 2007)
lack of regulations in health-care work system (Madon, Hofman, Kupfer and
Glass, 2007)
Limited support for studies needed to ensure (Madon, Hofman, Kupfer and
maximum impact Glass, 2007)
Established programs loose effectiveness (Madon, Hofman, Kupfer and
Glass, 2007)
When transferred to a new setting, programs (Madon, Hofman, Kupfer and
exhibit unintended effects Glass, 2007)
Splintering of health-care work system (Madon, Hofman, Kupfer and
Glass, 2007)
African countries have problems in curing MDG (Iluyemi & Briggs, 2007)
related diseases
Availability of medical knowledge among health
workers is not captured enough.
Poor performance NPHI IANPHI FOLIO nr. 1 2007
Lack of long term commitment IANPHI FOLIO nr. 1 2007
Lack of infrastructure IANPHI FOLIO nr. 1 2007
Lack of resources IANPHI FOLIO nr. 1 2007
Fragmentation of Public health activities IANPHI FOLIO nr. 1 2007
Lack of information facilities for health workers
in developing countries (Lack of information (Sluijs et al., 2006)
available or and poor access to information)
(Sluijs et al., 2006)
Traditional medical books are old
(Sluijs et al., 2006)
No money for medical journals
Lack of infrastructure (electricity and telephone)
to provide internet access to medical (Sluijs et al., 2006)
information in rural areas.
Often available medical knowledge doesn’t fit (Sluijs et al., 2006)
the local context.
10/90 gap: 10% of research budget is spend on
diseases that are responsible for 90% of the (Sluijs et al., 2006)
total burden on human health.
No interest for commercially uninteresting (Sluijs et al., 2006)
diseases
No incentive structure for medical research in (Sluijs et al., 2006)
developing countries
Clinics claim almost all time from researchers so (Sluijs et al., 2006)
there is lack of time for research
Good researchers become manager or civil
servant instead of performing necessary (Sluijs et al., 2006)
research.
Researchers in developing countries find it hard
to publish there research in scientific journals
due to lack of scientific (for their purpose (Sluijs et al., 2006)
“irrelevant”) knowledge and language problems.
Only a fraction of the relevant knowledge for
developing countries in scientific articles is (Sluijs et al., 2006)
written by researchers from these countries

46
Appendix 3: Problem Map
To provide a clear overview the problems from the previous appendix
are structured in a problem map. A structure is provided by clustering
problems according to the levels in the multi level perspective and
according to the different perspectives of the generalized scorecard.

Allocating Problems to Levels

Pico
Health related problems in sub-Saharan Africa are amongst the most
severe in the world. In general this is caused by poor nutrition, poor
sanitary facilities, the high prevalence of life threatening diseases,
underperforming health-care work systems, and the inability of people
to pay for healthcare.
At pico level people often have minimal means of survival due to no
income, no savings or shelter. A typical problem that people face is the
limited knowledge of preventive living habits. People often don’t know
how to change certain living habits which help them to live a healthy life.
If people become patients because they are in need of medical help,
they often don’t have enough money to pay for medical care. Another
problem in these countries is that patients don’t know where to go and
what medical care is available (Madon, Hofman, Kupfer and Glass, 2007).
Problems that health workers face include low wages, bad working
conditions and low job security (Conway, Gupta and Khajavi, 2007). The
large amount of people needing medical care in combination with the
low number of health workers available makes it hard to do a job as a
health worker. Because of these working conditions many professional
health workers move to countries with better career opportunities.
Furthermore health workers have poor access to specific, medical
knowledge (Sluijs et al., 2006), guidelines and knowledge resources. This
results in relatively low productivity of health workers. Some health
workers spend only 50 to 60 percent of their time on productive work,
the rest of the time they are busy filling out forms and writing reports
(Conway, Gupta and Khajavi, 2007).
A problem that health researchers face is the limited amount of time
they can spend on medical research. Health researchers are often
employed by a hospital. In practice these researchers can only spend a
small amount of their time on health-care related research, because
their employer claims too much of their time to do other health work.

47
Furthermore good researchers are often promoted to manager or civil
servant, which prevent them from performing medical research (Sluijs et
al., 2006).
Micro
At micro level organizations operate. Hospitals are facing an exodus of
health workers resulting in scarce human resources. The fact that the
education of a health worker is time consuming and costly, contributes
to their shortage. Even when utilizing para professional health workers,
human resources are insufficient (Conway, Gupta and Khajavi, 2007).
Furthermore there is a lack of income for hospitals. Other problems
hospitals are facing are the substandard facilities and scarce medical
supplies like medicine and medical equipment. Hospitals also have
limited knowledge of care and cure processes, which makes it hard to
improve their performance.
The lack of health workers is partially due to a lack of medical
educational centers like medical schools and training centers. Teaching
and training materials are poorly available. Especially knowledge
connected with local vulnerabilities such as local diseases, living
conditions and access to medical supplies, is hardly available. Not
enough indigenous knowledge is captured to increase the amount of
region specific knowledge. Only 10 % of the budget on medical research
is spent on those diseases that are responsible for 90% of the health
problems. Also the supply of external knowledge is insufficient. Access to
scientific medical knowledge often is too costly for these educational
centers (Sluijs et al., 2006). If external knowledge is available, it is often
difficult to translate it to the local situation. Training health workers in
sub-Saharan Africa often becomes a problem, because the subjects of
teaching are not always available in the local language. The fact that
impact and performance measurements of educational centers are not
available makes it hard to determine on what level educational centers
should improve. In medical education there is no systematic use of new
media and multi channel delivery.
As in other sectors the scarce financial, human and technological
resources form a barrier to execute the necessary surveillance and
research.
Telecom providers also play an important role because they have to
provide an infrastructure and services that enables mobile
communication. Closed architectures for mobile communication hinder
innovative developments for sharing medical knowledge via mobile

48
technology. Rent-seeking behavior, both by governments (over taxing)
and by providers (overcharging for value added services, verticalities),
which results in pricing that exclude mobile services for the poor
(Frischmann, 2005).
Meso
The coordination of medical research should be on the agenda of
national public health institutes (NPHI). In most sub-Saharan African
countries research related task such as evaluation and analysis of the
health status; public health surveillance; and public health research is
poorly executed. In most cases there is a fragmentation and wasteful
competition between different organizations that should collaborate
(IANPHI, 2007). This results in a poor performance of health institutes.
Financial constraints and the lack of capacity, governance and insight in
value-focused public-private partnerships are the basic problems for
underperforming institutions. It takes decades after the creation of a
NPHI before it can perform its core functions and address a range of
health problems. The lack of long term (donor) commitments is one of
the reasons that NPHI’s in African developing countries are
underperforming. Another reason for the failing of NPHI is that it is not
embedded in the local context. Every country and every region has its
own specific problems that need attention from the NPHI. The poorly
defined role in the health-care chain also makes it hard for and NPHI to
develop (IANPHI, 2007). Furthermore health-care related innovations are
poorly implemented. In case of transferring an established good
functioning health-care program to a new setting it often loses its
effectiveness (Madon et al, 2008).
The information sector which has to support the medical and health-
care sector has to deal with a fragmented information system
infrastructure.
Also the education and learning sectors have to improve. There is a
shortage of medical schools, a poor infrastructure and a lack of
knowledge resources to improve the current situation.
Nongovernmental organizations (NGO) are not sufficiently involved to
overcome the problems in the education and learning sector.
Macro
At macro level some general problems affect the quality of healthcare
and the health of the population. Poor road infrastructure, sanitation
and ICT, and weak institutions further hinder improvements of the
health-care work system.

49
Generalized Dashboard
To measure progress regarding the various problems listed before it is
convenient to use a dashboard with perspectives that are based on
generalizations (Goossenaerts, 2006) of the four balanced scorecard
perspectives namely P1: Customer; P2: Internal business processes; P3:
Learning & Growth; and P4: Financial The first generalization (P1)
focuses on the collaborative processes with relations, customers,
suppliers and other external stakeholders. P2 focuses on internal
business processes, which takes into account the internal assets, human
capital and processes at certain stakeholders. P3, learning and growth,
involves sustainability, competitiveness, innovation, knowledge and
especially knowledge translation. Finally P4, financial in BSC, is
generalized as the value flows and stock (funds) of money, information,
knowledge and material products, including water and food.
These four different perspectives are applied for each actor to make a
more structured overview of the problems and the (missing) assets. In
this way it will become clear where to intervene to tackle a certain
cluster or chain of problems. The table below gives a tentative problem
map.

Factor code:
P2: Internal business Processes & resources P3: Learning & Growth
P4: Financial and other Resource P1: Customer / External Relations
Funds/Flows
Problem coding: PiHW2_Guidelines:
Pi: Level (Pico, Micro, Meso or Macro)
HW: Abbreviation Actor
2: Factor (P1,P2, P3 or P4)
Guidelines: Keyword
Principal Factors
P2: P3
PiH2_Habits: Health risk due to PiH3_Ignorance: Ignorance (of
"myopic" living habits preventive living habits);
PiH3_Motivation: Motivational
indifference (no explicit targets, no
Pico experience of succeeding within the
Human
(person institutional context)
) P4 P1
PiH4_Means_survival: Minimal PiH1_Particpation: Minimal
means of survival; no income; no participation to socio-economic life
savings; no shelter
PiH4_Nutrition: Poor nutrition
(hunger)

50
 P2 P3
PiP2_sustain: Health problems PiP3_Ignorance: Ignorance of
causing inability to sustain oneself curative practices
or "earn a living"
Patient
P4 P1
PiP4_means_cure&care: PiP1_Social_constraints: Social
Insufficient means to buy cure or constraints restrain patients from
care seeking medical help
P2 P3
PiaHW2_Resources: Limited PiHW3_Job_security: Low job
21
resources; security
PiHW2_Guidelines: no or limited PiHW3_Knowledge: Lack of
access to guidelines; knowledge among health workers;
PiHW2_Productivty: Limited PiHW3_Incentive: poor incentive
21
Profession productivity of health workers system;
al health PiHW2_Quality: low quality of care
worker and cure services
P4 P1
21
PiHW4_wages: Low wages PiHW1_Care_chain: Poorly
PiHW4_conditions: Bad working developed or non-existing care-
21
conditions chain; PiHW1_Access_knowledge:
little or no access to specialist
knowledge
P2 P3
PiPHW2_Resources: Limited PiPHW3_Incentive: poor incentive
resources; system
PiPHW2_comply: limited means to PiPHW3_Knowledge: Lack of
Para pro-
comply to cure or care path; knowledge among para medic
fessional
PiPHW2_Productivty: limited workers;
health 21
productivity of health workers
worker
P4 P1
21Error!
PiHW4_Wages: Low wages PiPHW1_CareChain: Poorly
Bookmark not defined.
developed or non-existing care-
PiHW4_Conditions: Bad working chain;
21
conditions
P2: P3:
MiC2_HumanResources: Scarce MiC3_ParaProfessionals: Lack of
Human resources; para professional health workers;
MiC2_MedicalKnowldegeResource MiC3_Incentive: Poor incentive
s: Basic medical books are old, and system;
no access to medical journals; MiC3_EvidenceChain: Breaking of
MiC2_RedTape: Red tape; the evidence chain
MiC2_KnowledgeCareCure:limited
knowledge of care and cure
Micro
Hospital processes and how to improve
(Firm)
them;
MiC2_Productivity: Limited
21
productivity of health workers
P4 P1
MiC4_MedicalSAupplies: Scarce MiC1_Turnover: Large turnover of
medical supplies; personnel;
MiC4_Income: Uncertain income; MiC1_HealthWorkers: Lack of (para
sub-standard facilities professional) health worker;
MiC1_Private: Private/public issue

21
Conway, Gupta and Khajavi, (2007)

51
 P2
MiHF2_Women: Discrimination of
women
Home/
P4
Family
MiHF4_Shelter: No suitable shelter
P2
MiME2_TeachingPractices: Lack of
teaching/training materials and
practices that fit within local context
(diseases, living conditions, access
to medical supplies,...);
MiMe2_NewMedia: No systematic
use of new media and multi-
channel delivery;
Medical
education
centre
P4
MiME4_School: Lack of medical
schools;
MiMe4_Funding: Lack of funding;
MiMe4_Fragmentation:
Fragmentation and wasteful
22
competition
P2
MiPME2_TeachingPractices: Lack
of teaching/training materials and
practices reflecting local means of
delivery and continuing learning;
Para-
MiPME2_Content: No content in
medical
local languages
education
P4
centre
MiPME2_Capacity:Limited
capacity; MiPME2_Funding:Lack of
funding; MiPME2_Fragmentation:
Fragmentation and wasteful
23
competition
P2
MiNPHI2_LackResearcher: There
is a lack of researchers in
developing countries:
MiNPHI2_Institutes: Poor
performance public health
NPHI institutes;
(research) MiNPHI2_Commitment: Lack of
long term commitment;
MiNPHI2_NewMedia: No
systematic use of new media and
multi-channel delivery;
MiNPHI2_Researcher: Good
researchers become manager or
P3
P1
MiHF1_Constraints: Social
constraints restrain patients from
medical help
P3
MiME3_KnoweledgeTranslation:
No knowledge localization practices
(knowledge translation);
MiME3_LearningContent: unclear
public/proprietary boundaries
regarding learning content;
MiME3_Impact: No impact
measurements nor induced practice
correction;
MiME3_Planning: Poor educational
planning
P1
MiME1_KnowledgeSupply:
Insufficient external "knowledge
supply";
MiME1_IndegenousKnowledge:
insufficient capture of indigenous
knowledge;
MiME1_Life-longLearning: No life-
long learning attitudes with the
target groups (medical doctors)
P3
MiPME3_KnowledgeTranslation:
No knowledge localization practices
(knowledge translation);
MiPME3_Impact: No impact
measurements nor induced activity
correction
P1
MiPME1_Standards: No broadly
adopted generic standards;
MiPME1_Life-longLearning: No life-
long learning attitudes with the
target group
P3
MiNPHI3_Embedding: Lack of
embedding in the local context;
MiNPHI3_Regulations: Lack of
regulations;
MiNPHI3_Partnership: Lack of
insight in value-focused public-
private partnerships;
MiNPHI3_Governance: Lack of
incentive structures and
governance

22
WHO (2004)
23
WHO (2004)

52
 civil servant which disables them
from performing research.
MiNPHI2_ResearcherHospital:
Researchers are often claimed by
hospitals and can not perform
enough medical research.
P4
MiNPHI4_capacity: limited capacity;
lack of funding;
MiMe4_Fragmentation:
Fragmentation and wasteful
24
competition
P2
P4
MiTP4_Investments: Lack of
Telecom
investments;
Provider
MiTP4_BusinessCase No clear
business case for open mobile
platform (android)
P2
MeRKT2_ResourceUtilization: Lack
of measures resource utilization;
MeRKT2_Implementation: Poor
implementation of health-care
related innovations;
MeRKT2_Fragmentation:
Fragmentation of health-care
activities;
MeRKT2_Allocating: Difficulties in
allocating resources;
Medical &
MeRKT2_HealthWorkers: Lack of
Health 26 27
health workers , ;
Meso Research
MeRKT2_Exodus: Exodus of
(Sector and Know- 21
professional health workers
) ledge
MeRKT2_ProductivityMeasures:
Transla-
Lack of productivity measures;
tion
MeRKT2_Infrastructure: Poor
infrastructure (roads);
MeRKT2_IncentiveStructure: There
is no good incentive structure in
developing countries for publishing
medical research.
P4
MeRKT4_10/90Gap: 10% of
research budget is spend on
diseases that are responsible for
90% of the total burden on human
P1
MiNPHI4_RoleEvidenceChain:
Poorly defined role in the care and
evidence chain;
MiNPHI4_HumanResources: Lack
of human resources;
MiNPHI4_Governance structure:
25
No clear governance structures
P3 Closed architectures preventing
innovative services over the
telecom networks
P1
MiTP4_Rent-seekingBehavior:
Rent-seeking behavior, both by
governments (over taxing) and by
providers (over-charging for value
added services, verticalities);
MiTP4_Pricing: Pricing that exclude
services for the poor
P3
MeRKT3_KnowledgeResources:
Lack of knowledge resources and
poor access to resources;
MeRKT3_Regulations: Lack of
regulations;
MeRKT3_EffectivenessProgramms:
Established programs loose
effectiveness when transferred to a
new setting,
MeRKT3_UnintendedEffects:
programs exhibit unintended
effects;
MeRKT3_Implementation: Poor
implementation of innovations
P1
MeRKT1_EssentialFacilities:
Inefficient market because of
control of essential facilities
28
(content) by private sector

24
WHO (2004)
25
For an example, of what could be possible, see global reporting initiative
26
Madon, Hofman, Kupfer and Glass, (2007)
27
Conway, Gupta and Khajavi, (2007)
28
Goossenaerts (2007)

53
 health.
P2 P3
MeIS2_Infrastructure: Poor MeIS3_LockIn: relative "lock in"
29
infrastructure (Information System); (live by the laws of vendors)
MeIS2_KnowledgeTranslation:
Infor- Knowledge translation practices do
mation not reflect ferment in the industry
System P4 P1
MeIS4_PublicPrivate: No public MeIS1_EssentialFacilities:
counter weight for privately Inefficient market because of
controlled investments and control of essential facilities
30
roadmap (architecture) by private sector
P2 P3
MeEL2_TrainingModel: Classical MeEL3_Regulations: Lack of
training model costs too much time regulations that nurture: (i) open
21
and money educational resources and (ii)
MeEL2_Schools: Lack of medical innovative educational materials for
21
schools the poor
MeEL2_Infrastructure: Poor
infrastructure(road, hygiene,
electricity, telephone);
MeEL2_ProblemBasedLearning:
Education Insufficient use of "problem based
& Learning learning";
MeEL2_NewMedia: No systematic
use of new media and multi-
channel delivery;
P4 P1
MeEL4_KnowldgeResources: Lack MeEL1_TrainingPrograms: Lack of
of knowledge resources; private (non-governmental) training
MeEL4_Competition: Wasteful programs;
competition MeEL1_InvolvelmentNGO: Low
involvement of NGO in medical
training
Macro
P2 P3
MaR2_Hygene: Poor hygiene
process;
MaR2__Roads: Poor access via
road infrastructure;
MaR2_Sanitary: Poor sanitary
econom
infrastructure;
y)
MaR2_Infrastucture: lack of
National/ electricity and telephone lines to
Regional enable internet access;
MaR2_MDG: African countries
have problems in curing MDG
related diseases
P4 P1
MaR4_Hunger: Poor nutrition
(hunger);
MaR4_AccessCare: Insufficient
access to quality care
global

29
Guy (2007)
30
Goossenaerts (2007)

54
55
Appendix 4: Ishikawa Diagram

Contextual Determinants

Hospitals have limited means to

improve their processes, including
the productivity of their health
Institutional
workers
Poor access to high
quality content Social-Cultural

Lack of customization
Standards
Lack of regulations that nurture:
open educational resources Hospitals cope with Health workers
and innovative educational scarce human resources have limited or no Distal Determinants
materials for the poor Classical training model access to
takes too much time specialist
Private/Public and money knowledge
Issue Health workers have
Para professionals have limited or no access
minimal means to comply to guidelines
To treatment prescriptions Lack of medical
schools
Lack of para professional
health workers Proxal Dterminants
Social constraints
Poor Quality Health restrain people form
Hospitals must
Services seeking medical help
cope with scarce
medical supplies
People are ignorant of
preventive living habits
Lack of customization
Hospital facillities are Standards
often sub-standard People are ignorant of Poor access to high
Lack of customization available medical help quality content
Standards
Lack of customization
Hospitals have limted Poor access to high Standards
Population’s
knowledge of, or access to quality content
Health
medical guidelines

People have insufficient means

to buy cure and care Lack of clean water
Lack of food
People have minimal
High prevalence of
means of survival
infectious diseases

Environmental

Economical

56