Ecological Indicators 17 (2012) 4657

Contents lists available at ScienceDirect

Ecological Indicators
journal homepage: www.elsevier.com/locate/ecolind

A method to assess the relevance of sustainability indicators: Application to the

indicator set of the Czech Republics Sustainable Development Strategy
Tomas Hak, Jan Kovanda , Jan Weinzettel
Charles University Environment Center, Jose Martiho 2, 162 00 Prague 6, Czech Republic

a r t i c l e

i n f o

Keywords:
Environmental sustainability
Indicators
Assessment criteria
Validation
Relevance
Sustainable development

a b s t r a c t
The goal of this article is to critically review the state-of-the-art in assessing the quality of sustainability
indicators and contribute to the development of a suitable methodology for that. We start with a broad
review of the vast body of work in this eld in both practice and academic research. We show that
both scientists and practitioners have sought developing and using methods for assessing quality of the
indicators. They have usually dened some criteria for that; however, neither science nor practitioners
have provided major support by developing reliable as well as practical and operative methods for indicator assessment. Therefore, we propose an innovative new method for indicator assessment from the
perspective of their relevance. We operationalize this criterion and apply it to the environment-related
indicators from the set used for the evaluation of the Czech Republics Sustainable Development Strategy.
2011 Elsevier Ltd. All rights reserved.

1. Introduction
An important impulse for developing sustainable development
indicators arose from the 1992 World Summit in Rio. Agenda
21, adopted at the conference, expressed the need to formulate
indicators in order to better monitor and foster sustainable development. Another of the conference outcomes was the foundation
of the United Nations Commission on Sustainable Development
(UNCSD) with the goal to assist countries in developing and
using sustainable development indicators (UNCSD, 2001). Interest
and various activities related to sustainable development indicators among many international organizations have increased in
the past years. The Organization for Economic Cooperation and
Development has developed and published indicators for both particular areas (resource use and environmental outlook), sectors
(households or transport) as well as developed a standardized
indicators-based framework for countries environmental performance review (OECD, 2005). The United Nations Environment
Programme has regularly published the Global Environmental Outlook, which has used a set of indicators to underline the choices
available to policymakers across a range of environmental, social
and economic challenges (UNEP, 2007). At the regional level, the
European Environmental Agency (EEA) has intensively developed
and used indicators for assessment of the European environment.

Corresponding author. Tel.: +420 220199476; fax: +420 251620441.

E-mail addresses: tomas.hak@czp.cuni.cz (T. Hak), jan.kovanda@czp.cuni.cz (J.
Kovanda), jan.weinzettel@czp.cuni.cz (J. Weinzettel).
1470-160X/$ see front matter 2011 Elsevier Ltd. All rights reserved.
doi:10.1016/j.ecolind.2011.04.034

It has developed both concrete indicators for many environmentrelated areas as well as provided its member countries with
methodological and technical assistance. The statistical body of
the European Union Eurostat established a Working Group in
2001 to respond to the demand for measuring progress towards
sustainability with a set of agreed indicators (Eurostat, 2009).
A number of intergovernmental organizations and national
governments, but also regional and local authorities, local communities, business organizations and other economic actors, academic
institutions and civil society organizations of many kinds, are currently developing and using sets (sometimes called dashboards) of
sustainability indicators. At present, hundreds of different indicators have been suggested and are used in many varied contexts, by
different users and for diverse purposes: Riley (2001) speaks about
an indicator explosion in this context. No exhaustive account probably exists but we can assume the existence of hundreds of various
indices and sets of indicators or even several thousands of such
metrics if individual indicators are included (OECD, 2002; European
Communities, 2004; UNDP, 2005). While sustainability indicators
are used ever more extensively and intensively by a wide range of
users and in many different contexts, it does not necessarily follow
that they are scientically sound and/or used appropriately.
The goal of this article is to critically review the state of the
art in assessing the quality of sustainability indicators and contribute to the development of a suitable methodology for that.
We conducted a quite comprehensive review of the vast body of
work in this eld in both practice and academic research. We
used the review to highlight the abundance of criteria and various frameworks for the assessment but also a serious lack of

T. Hak et al. / Ecological Indicators 17 (2012) 4657

practical guidelines for both the indicator developers and users.

Therefore, after the introductory section and the review of existing
approaches towards evaluating sustainability indicators, we provide a description of the newly proposed assessment method. In
face of its originality, we decided to test it and demonstrate its
feasibility on selected indicators used for measuring sustainable
development in the Czech Republic. Then, we provide arguments
for the use of this method as well as an explanation of results. At the
end, we conclude by offering our recommendations of needed (useful) directions in development of indicators for the Czech Republics
Sustainable Development Strategy.

2. Indicator assessment efforts a review

2.1. Research into the assessment/evaluation of sustainability
indicators
The Bellagio Principles, trying to harmonize the process of
sustainable development implementation, drafted ten principles
to provide a link between theory and practice in sustainability
measurement (Hardi and Zdan, 1997). Despite the fact that the
follow-up activity the Bellagio STAMP looks at indicator evaluation much more closely, it still stays at a level of general guidelines
not readily applicable in practice (see more in Pintr et al., 2012).
Regardless of the amount and character of criteria, the methodology underlying the elaboration and development of indicators
should t scientic standards, which imply a procedure of validation (Girardin et al., 1999). Bockstaller and Girardin (2003)
proposed a methodological framework to validate environmental
indicators. It is based on experience of developers of simulation
models and on a denition of validity as the adequacy for a specic purpose. The indicator will be validated if it is scientically
designed, if the information it supplies is relevant and if it is useful and used by end users. Cloquell-Ballester et al. (2006) proposed
a methodology for indicator validation that requested that indicators be based not only on thorough scientic foundations but also
on recognized social content. It means that the methodology also
incorporates public participation to support consensus building.
The methodology veries suitability of indicators in three stages:
self-validation (done by the developers themselves), scientic validation (independent experts judgment) and social validation
(public participation). Validation is viewed as a multicriteria multiexpert decision problem. The core of the validation is to assess
the correct performance of new indicators from three fundamental
views: conceptual coherence, operational coherence and utility. An
index is calculated for each, and then, the three indices are aggregated into a nal score. As a result, a new indicator is classied into
several (at least two) categories: credible indicators (validated);
non-credible indicators (non-validated). Since this methodology
does not depend on the nature of the indicators, the authors suggest using it for validation of indicators also in other areas such as
development studies, health or international cooperation. In addition, Niemeijer and De Groot (2008) call for a transparent selection
of the best available indicators. They suggest that the selection be
based on (i) a conceptual framework, (ii) individual and set level
criteria, and (iii) selection methodology.
Parris and Kates (2003) proposed a theoretical analytical framework that distinguishes among goals, indicators, targets, trends,
etc. They suggested that various measurement methods might be
characterized by three attributes: salience, credibility and legitimacy. A salient indicator provides relevant information responding
to peoples concerns; it measures progress against policy goals by
comparing indicator values to targets; it answers pertinent questions; it is simple to interpret, accessible and publicly appealing;
and it clearly informs about the extent of the issue(s) represented.

47

This review resulted in a quite poor outcome, which indicates

underdevelopment and underestimation of the importance of this
issue by the scientic community. The lack of research and development of a methodology and guidelines for assessment of indicator
quality has been stressed by several authors (e.g., Bockstaller and
Girardin, 2003; Bauler, 2012). More evidence is gained when talking to many organizations that suffer from oating in a vacuum in
this respect instead of having a solid base for justication of indicator quality. The International Council of Scientic Unions (ICSU)
realized this problem and together with other organizations, it
emphasized the need to focus scientic attention on this issue by
organizing a large project called the Assessment of Sustainability
Indicators (Hak et al., 2007).
2.2. Organizations attempts to evaluate/assess quality of their
indicators
Many organizations have recently developed and used indicators for various reporting tasks. In fact, reporting requirements
requirements to provide information agreed between countries and international bodies such as the European Environmental
Agency, OECD or international conventions have increased to such
an extent that they also generate a great demand on the quality of
exchanged data and information (often in the form of indicators)
(EEA, 2010).
The United Nations Public Administration Network (UNPAN)
has information at the core of its mission (UNPAN, 2000). In order
for its information to be of any use to the interested public, it needs
to meet high quality standards. In the UNPANs view, this quality
has several dimensions, including relevance of the content, credibility of the source, originality, timeliness and neutrality. Although
well-dened theoretically, the validation criteria are far from being
ready to use.
Eurostat may serve as an example of an organization that has
proposed and applied a set of criteria for indicator selection and/or
assessment (Ledoux et al., 2005). Indicators should: capture the
essence of the problem and have a clear and accepted normative
interpretation; be robust and statistically validated; be responsive
to policy interventions but not subject to manipulation; be measurable in a sufciently comparable way across Member States, and
comparable as far as practicable with the standards applied internationally by the UN and the OECD; be timely and susceptible to
revision; and not impose on Member States, on enterprises, nor on
the Unions citizens a burden disproportionate to their benets.
Due to the policy needs and the need to have at least some
metainformation on indicator quality, Eurostat has taken a pragmatic approach and the criteria have been applied with some
exibility. It has been preparing indicator quality proles, useroriented summaries of the main quality features of indicators.
Quality is dened along several dimensions; therefore, the quality
prole aims at a quick overview on how far an indicator is deemed
t for use with regard to its key objectives. The scope of the quality
proles, e.g., for structural indicators, is as follows: Feasibility (by
looking at timeliness and coverage); Technical soundness (comprising overall accuracy and comparability over time and across
countries) and Reliable sources meeting high standards and involving statistical expertise. The quality prole also discusses relevance,
which is considered to comprise the content and suitability of the
indicator to measure appropriately the phenomenon considered
(Eurostat, 2010). Although the evaluation is qualitative in nature
(and thus loaded with subjectivity), it tries to give an unambiguous
formalized picture of the quality of the indicator.
Similarly, the European Environmental Agency has developed a
set of criteria to be used for evaluating its indicators (EEA, 2005).
Unlike many other organizations, it has operationalized them to
some extent. The quality of each indicator has been evaluated using

48

T. Hak et al. / Ecological Indicators 17 (2012) 4657

a subset of criteria. A scoring system has been dened for each

criterion. The cumulative result of the evaluation for the indicator has been calculated as well as presented as a radar diagram.
The criteria have concerned the seven following issues: policy relevance, existence of targets, methodological advancement, data
availability, possibility of assessment trends, spatial coverage, and
comparability between countries. The criteria have been scored on
a two-to-ve-point scale, for example the criterion targets (Does
the indicator monitor progress towards the quantied targets?)
could reach these values: 0 points no targets; 1 point targets
but the indicator does not fully reect these; 2 points qualitative targets (generic); 3 points qualitative targets (specic) or
quantied targets not time-bound; 4 points quantied targets
time-bound. Other criteria have only been scored on a two-point
scale, for example policy relevance (Is the indicator policy relevant,
i.e., supporting EU policies priority issues?) could only reach two
values: 0 points neither EU nor EEA priority policy issue; 4 points
EU and EEA priority policy issue.
The US Environmental Protection Agency (Kurtz et al., 2001)
has prepared technical guidelines to evaluate the suitability of
ecological indicators. The guidelines were used as a consistent
framework for indicator development, review and selection. They
were organized within four evaluation phases: conceptual relevance, feasibility of implementation, response variability and
interpretation, and utility. The framework consists of 15 guidelines (questions) starting with two testing a conceptual relevance:
Is the indicator relevant to (i) the assessment question? and (ii)
the ecological resource or function? The guidelines are not criteria and do not, by themselves, determine indicator applicability or
effectiveness. Rather, they provide a framework for asking relevant
questions about an indicator.
2.3. A review at a glance
When searching for methods of assessment of sustainability
indicators, we have made a detailed review of approaches used
in more than twenty recognized organizations (European Environmental Agency, Organization for Economic Cooperation, United
Nations Environment Programme, US National Research Council,
Eurostat, US Environmental Protection Agency, World Bank, World
Resource Institute, etc.). The organizations have developed and
used the criteria with the aim, for example, to select the appropriate or the best indicators for the measurement system. In total,
we have examined about 260 criteria used for indicator assessment,
i.e., about 12 criteria applied to one indicator set on average (the
numbers of criteria dened by one organization or indicator developer ranged from 3 to 29).1 Despite their varied terminology, the
criteria used by different organizations overlapped in content and
meaning to a great extent. In other words, various organizations
formulate similar requirements for the indicators they use.
Both scientists and practitioners agree that indicators have little chance of being accepted and used for decision making unless
they may prove certain qualities dened and measured against
operationalized criteria. Thus, the indicator developers postulate
that the indicators should be relevant, scientically (conceptually, methodologically, etc.) sound, feasible, effective, pragmatic,
accessible, understandable, etc. However, they do not propose a
procedure or guidelines for validation of indicators (Bockstaller
and Girardin, 2003; Kral and Zemlicka, 2006). They usually dene
some criteria and try to apply them intuitively. At the same time,
science has not provided strong support by developing reliable as

1
Due to the excessive extent, the reviewed criteria and principles used for
assessment and selection of sustainability indicators can be found on the web at
http://www.czp.cuni.cz/indikatory/Annex3.pdf.

well as practical and operative methods for indicator assessment.

Having our own long experience with development and using of
environmental sustainability indicators, we propose a new method
for indicator assessment from the perspective of their relevance.
3. A proposed method
All indicator developers seek success in using their indicators.
In this context, by success we mean massive spread, frequent use
and high popularity. It is known that for some purposes, users may
be willing to accept weaknesses in an indicator if it provides really
important information (Kurtz et al., 2001). Therefore, we delimited
the domain of our interest to providing a method (tool) for indicator developers and users so that they can readily identify what is
pertinent and what is not. We narrowed the wide choice and potential application of the assessment criteria down to one of them:
relevance.
In accordance with the ndings of the literature survey shown
in the previous sections, we start with a notion that relevance
is a term used to describe how pertinent, connected, or applicable something is to a given matter (Wikipedia, 2010). In formal
reasoning, relevance is an important but elusive concept; therefore, we try to make this criterion simple while still not simplistic:
A relevant (salient) indicator provides information responding to
peoples concerns, answers pertinent questions and is of use for
policy making. To make this denition operational and usable for
practical purposes (indicator assessment), we distinguish between
the relevance of the theme of an indicator and the relevance of the
indicator itself for the theme. Also, we distinguish between the relevance of these two matters for the broad public and experts. We
dene these aspects of relevance as follows:
The theme relevance denotes the importance of sustainability
themes represented by particular indicators (the themes include,
e.g., climate change, transport, use of energy, etc.). Having in mind
various user groups, this criterion is to verify that the theme is
meaningful to a broad audience. Since policies and policy making have social effects, we may assume that the citizens wish
to express their opinions and that the particular sustainability themes are publicly debated (we call this public relevance).
Experts, both scientists and practitioners, have their research
agendas which are shaped by many drivers. Even without analyzing them, the appearance of the themes in the outcomes provided
by science reects the interest of the theme to experts (we call
this scientic relevance).
The indicator relevance for the theme is conceived as representativeness of a given indicator for the sustainability theme (it
answers the question how pertinent the particular indicator is for
the theme). Ideally, we might test whether the indicator used for a
certain thematic area (e.g., biodiversity) really provides information on this phenomenon. In a pragmatic way, the relevance of an
indicator can be measured and expressed by how often the particular theme is described by or connected with a given indicator.
Also, this criterion is similarly designed to be applied to two user
groups: expert community (scientic relevance) and the public
(public relevance).
In other words, the theme relevance criterion assesses whether
a theme to which the indicator relates is of concern (and to what
extent), while the indicator relevance for the theme criterion tells
us how closely the indicator is related to the theme. In any case, the
indicator relevance does not imply that the indicator is calculated
or constructed properly in terms of methodological soundness.
We have sought to carry out the assessment in as quantitative and objective a way as possible, which would ensure

T. Hak et al. / Ecological Indicators 17 (2012) 4657

reproducibility of the results. As our experience and the review

of relevant resources have shown, besides lack of operationalization of the validation criteria, another weakness of the indicator
assessment methods proposed so far has been the fact that they
have often been qualitatively based on expert evaluation, which
hampered their reproducibility. Although we might design our
criteria for the qualitative testing (e.g., using the Delphi method,
conferences, interviews, etc.), we wished to base the assessment
on simply applicable and widely available tools so that it can be
done by any expert or interested layman with a connection to the
World Wide Web. Last but not least, we focused on an assessment
resulting in ranking the indicators in relation to each other and only
touched upon the stand-alone assessment of particular indicators.
The latter would require a very different approach, including the
setting of pre-dened and therefore subjective scales.
As a starting point, we take the commonly agreed assumption
that the presence of any topic or subject on the World Wide Web
indicates its relevance for a given audience (Palmer, 2002; Jung
et al., 2007). We used a content analysis a type of secondary
data analysis used to analyze text, interview transcripts, newspapers, books, and Web sites to determine the frequency of specic
words or ideas. In general, the results of content analysis allow
the researchers to identify, as well as quantify, specic ideas, concepts, and their associated patterns, and trends of ideas that occur
within a specic group or over time. We applied a semantic analysis and in particular, a subject-matter analysis in order we get a
frequency with which certain objects (sustainability themes and
the indicators) are referred to (Krippendorff, 1980).
For our test we used the world-wide-web as database, and
Google as search engine. It means, in order to identify and specify the occurrence of sustainability themes on the World Wide
Web, we used Google Search and Google Scholar Search, which
belong to the most widespread internet search engines nowadays
(the same search engines were later also used for the indicators)
(McMillan, 2000). Pavlik (2001) also argues that the Internet is
radically opening up the media, allowing more people and organizations to interact with it and to disseminate materials widely. The
presence of a particular theme on the World Wide Web is quantied
by the number of hits after inserting the key words. We conducted
this analysis fully aware that the search engine may produce even
major inaccuracies that may distort or bias the results (Cilibrasi and
Vitanyi, 2007), however, we accepted this risk without any further
elaboration. Should this analysis is done for real decision making,
the uncertainties and inaccuracies are to be assessed. We conceived
this number of hits as an appropriate proxy for the public relevance
of the theme when using Google Search as well as a proxy for the
scientic relevance of the theme when using Google Scholar Search.
After pre-testing, we found that the result number of records
is strongly inuenced by:
(1) the level of concreteness of the key words;
(2) the number of key words simultaneously inserted into search
engines (the lower the number of key words; the higher the
number of hits); and
(3) using single words or phrases2 (phrases resulted in lower numbers of hits).

2
When a phrase in quotation marks is inserted, the search result page will display all the sites that contain exactly this phrase. When a few single words/phrases
are inserted, the search result page will display all the sites that contain all these
words/phrases regardless of how far they are from each other. It is further possible to
use operators like AND and OR to deliver more specic results. For instance, the
OR operator allows for selection of sites with either of the words inserted (Google
Tutors Google Search Manual, 2010). These operators were not used within our
study.

49

Taking into account the observations from pre-testing, we have

developed four sets of key words (see Table 2 in the case study),
in order to be able to propose a way for identication of the
key words with the greatest internal coherence: (A) sustainability theme denoted by a single key word; (B) sustainability theme
denoted by a single phrase; (C) sustainability theme denoted by
three single words; and (D) sustainability theme denoted by a single word and a word related to all the indicators, e.g., environment
for a set of environmental sustainability indicators. Option B proved
to be the most problematic one as we were not able to devise a
phrase with the same number of words and at the same level of concreteness for all the sustainability themes. The purpose of option D
was to secure that a prevailing number of hits originates from the
domain that all the indicators have in common. The results for the
sustainability themes in absolute values were transformed into a
ranking of themes by their relevance.
To put the criteria of scientic and public relevance of an
indicator in practice, we have developed a set of key words
for particular indicators (see Table 2 in the case study). Subsequently, we inserted them into Google Search and Google Scholar
Search engines simultaneously with the key words for sustainability themes. We received a number of records, which included those
relevant both for the particular theme and for the particular indicator. This number of records was rst divided by the results in
absolute values for corresponding sustainability themes to show
for how large a proportion of hits for these themes the indicators
are relevant. The resulting shares were then transformed into a
ranking of the indicators by their relevance for the theme. With
respect to the relevance of an indicator for the selected theme, we
stuck to only one set of key words; they aimed to be quite specic
for each indicator so the possible variability was much lower than
in the case of themes.
The counting of hits was carried out in a single day for all the key
word sets as the numbers of hits are date-specic due to the continual swelling of the World Wide Web. We ended up with 4 sets
of rankings of sustainability themes for both public and scientic
relevance and 4 sets of rankings of indicators for their relevance for
the theme, both public and scientic. As a nal step, we selected
the key word set which delivered the most average ranking. We
used the least square method:
ei =

(rmj rij )2

(1)

where ei is the sum of squares of the differences between the mean

ranking rmj and the actual ranking of the indicators rij for each key
word set i and each indicator j. It can be hypothesized that the most
average ranking (lowest ei ) would be delivered by the key word set
with the average number of key words, as the number of hits is
proportional to number of key words simultaneously inserted into
search engines. However, the number of hits is also inuenced by
the level of concreteness of the key words and the usage of phrases.
We therefore considered the selection of the most average ranking
to be meaningful. We further considered this average ranking as
best proxy for assessment of the theme/indicator relevance.
4. Testing the method: the case of the indicators in the
Czech Republics Sustainable Development Strategy
4.1. A case study: indicators for testing the proposed criteria
We have applied the criteria of relevance to the indicator set
used for the evaluation of the Czech Republics Sustainable Development Strategy (Ofce of the Government of the Czech Republic,
2005) and published regularly in the Progress Reports (Ofce of the
Government of the Czech Republic, 2006; Government Council for

50

T. Hak et al. / Ecological Indicators 17 (2012) 4657

Table 1
Indicators used for the assessment of the Czech Republics Sustainable Development Strategy (Government Council for Sustainable Development, 2009).
Sustainability theme number and title

Indicator number and title

Brief denition

Public transport

1a

Transport intensity

Use of energy

2a
2b

Energy intensity
Primary energy supply

Climate change

2c
3a

Ratio of transport performance (in person-kilometers and in

tonne-kilometers) to GDP.
Ratio of total primary energy supply to GDP.
Domestic extraction of all energy raw materials plus total
imported energy minus total exported energy.
Share of renewable energy sources in total primary energy supply.
Aggregated emissions of all greenhouse gases expressed in CO2eq
per capita.
Aggregated emissions of all greenhouse gases expressed in CO2eq
per GDP.
Domestic extraction of raw materials and biomass plus total
import of materials minus total export of materials.
Proportion of the total waste production that is used as raw
material.
Amount of fertilizers used (in kilograms of pure nutrients) per
hectare of farmland.
Amount of pesticides used in kilograms per hectare of farmland.
Proportion of the area of farmland categorized as organic farmland
to the total area of farmlands.
Defoliation percentage of an entire treetop converted to the degree
of defoliation (scale 04).
Relative changes in the population of individual bird species in
locations which are the subject of census.
Total amount of expenditures devoted to the maintenance and
improvement of the environment.

Use of resources

4a

Share of renewable energy

Greenhouse gas emissions per
capita
Greenhouse gas emissions per
GDP
Material consumption

Waste treatment

5a

Material use of waste

6a
6

Agriculture
6b
6c

Consumption of mineral
fertilizers
Consumption of pesticides
Organic farming

3b

Forestry

7a

Defoliation

Biodiversity

8a

Bird index

Environmental
protection

9a

Expenditures on
environmental protection

Sustainable Development, 2007, 2009). The indicator set includes

34 indicators organized along the Strategys three pillars (economic, social and environmental) and complemented with three
other areas prioritized in the Strategy (Research, development and
education; European and international context; and Good governance). In order to be in compliance with the focus of this Special
Issue, which is environmental sustainability, we carried out the
indicator assessment for a subset of 14 indicators related to this
domain. The basic information on these indicators is summarized
in Table 1.
Table 2 shows key word sets developed for the sustainability
themes and particular indicators from Table 1. These key word sets
were further processed as described in the Method section. The
selected word related to all the indicators for sustainability theme
key word D was environment to denote that all the indicators
focus on environmental sustainability.
4.2. Results and discussion
Table 3 shows absolute values for the public relevance of sustainability themes (number of hits) and the indicators (per mille
showing for how large a proportion of hits the indicators are relevant for these themes). Table 4 shows the absolute values for the
scientic relevance of sustainability themes and the indicators. The
key word sets, which delivered the most average results, are in bold.
Considering the most average key word set C, Table 3 suggests
that the most publicly relevant theme is Use of Resources with 106
million hits followed by Use of Energy with 31.5 million hits and
Agriculture with 20.6 million hits. On the other hand, the lowest
public relevance was calculated for the themes Public Transport
(2.06 million hits), Waste Treatment (1.71 million hits) and Forestry
(1.36 million hits). With respect to the indicator public relevance
for the theme (key word set B), the best indicators include Material Use of Waste (40.82 per mille), Greenhouse Gas Emissions Per
Capita (23.947 per mille) and Greenhouse Gas Emissions Per GDP
(5.787 per mille) while Primary Energy Supply (0.863 per mille),
Transport Intensity (0.111 per mille) and Organic Farming (0.032
per mille) show the worst results. This summary indicates that the
indicator relevance for the most relevant themes such as Use of

Resources and Use of Energy is not very high, which is not true for
the low relevant theme Waste Treatment. This is an odd outcome,
as we were supposed to focus on the most relevant themes in the
indicator sets and try to have the most relevant indicators for them.
The absolute values for public relevance of sustainability themes
and the indicators can be used to assess their mutual relevance.
It is shown, for instance, that the most relevant theme of Use of
Resources is 3.4 times more relevant then the second one, Use
of Energy, and 77.9 times more relevant than the least relevant
theme of Forestry. Even more pronounced differences are found
for the indicators. Although the most relevant indicator, Material
Use of Waste, is only 1.7 times more relevant than the second one
of Greenhouse Gas Emissions Per Capita, it is 1,276 more relevant
than the worst performing indicator of Organic Farming.
Another perspective is obtained by comparing indicator relevance for particular themes which are represented by more than
one indicator. For instance, the most publicly relevant indicator in
the Use of Energy theme is Energy Intensity, followed by Share of
Renewable Resources and Primary Energy Supply, but the differences in the relevance are not so high (Energy Intensity relevance
is 3 times higher than the relevance of Primary Energy Supply). On
the other hand, in the Agriculture theme, the most relevant indicator of Consumption of Mineral Fertilizers is 99 times more relevant
than the least relevant indicator of Organic Farming, while the relevance of the second indicator of Consumption of Pesticides is quite
close to the relevance of the rst indicator.
Table 4, key word set D, shows that the most scientically
relevant themes include Use of Resources (4.9 million hits), Use
of Energy (4.56 million hits) and Public Transport (3.16 million
hits), while the least scientically relevant themes comprise Waste
Treatment (2.01 million hits), Forestry (1.34 million hits) and Biodiversity (0.653 million hits). Comparison with public relevance
suggests that both the wider public and scientists consider Use of
Resources and Use of Energy very important themes, but while Public Transport is not perceived as important by the public, it is very
important for scientists. Regarding indicator scientic relevance for
theme (key word D), it is highest for Defoliation (14.328 per mille),
Greenhouse Gas Emissions Per Capita (11.587 per mille) and Consumption of Pesticide (9.567), and lowest for Share of Renewable

T. Hak et al. / Ecological Indicators 17 (2012) 4657

51

Table 2
Sets of the tested key words for sustainability themes and the indicators.
Sustainability
theme number and
title

Sustainability
theme key
words A

Sustainability
theme key words B

Sustainability
theme key words C

Sustainability
theme key
words D

Indicator number and title

Indicator key word

Public transport

transport

public transport

transport public
cargo

transport
environment

1a

Transport intensity

energy

use of energy

energy
environment

Energy intensity

Use of energy

energy supply
security

2a

2b

6c

Primary energy
supply
Share of renewable
energy
Greenhouse gas
emissions per
capita
Greenhouse gas
emissions per GDP
Material
consumption
Material use of
waste
Consumption of
mineral fertilizers
Consumption of
pesticides
Organic farming

transport
intensity GDP
energy intensity
GDP
primary energy
supply
share of
renewable energy
greenhouse gas
emissions capita

7a

Defoliation

share of organic
farming
defoliation

8a

Bird index

number of birds

9a

Expenditures on
environmental
protection

environmental
expenditures

2c
3

Climate change

climate

climate change

climate change
temperature

climate
environment

3a

3b
4

Use of resources

resources

use of resources

Waste treatment

waste

waste treatment

Agriculture

agriculture

agriculture

Forestry

forestry

forestry

Biodiversity

biodiversity

biodiversity

Environmental
protection

protection

environmental
protection

resources supply
security
waste production
treatment

resources
environment
waste
environment

agriculture food
supply

agriculture
environment

forestry wood
supply
biodiversity loss
protection
environment
expenditures

Energy (0.496 per mille), Transport Intensity (0.123 per mille) and
Organic Farming (0.039 per mille). Comparing these results with
the public relevance of the indicators shows that apart from some
exceptions (e.g. Greenhouse Gas Emissions Per Capita and Organic
Farming) the views of the wider public and scientists on indicator
relevance are quite different. As it can be assumed that the scientic community is likely to produce more competent and qualied
feedback than the wider public, it is rather the scientic relevance
of the indicators should be taken into account when assessing and
shaping indicator sets.
As scientic relevance produced lower and more evenly distributed results for both the themes and indicators, the differences
in their mutual relevance are also not so large. The most scientically relevant theme of Use of Resources is thus only 1.1 times
more relevant than the second theme of Use of Energy and 7.5 more
relevant than the last theme of Biodiversity. Similarly, the best performing indicator of Defoliation is 1.2 times more relevant than the
second, Greenhouse Gas Emissions Per Capita, and 367 times more
relevant than the worst indicator of Organic Farming.
An interesting research question is a stand-alone assessment of
the theme of relevance using the World Wide Web. This requires
setting a pre-dened maximum number of hits for the themes.
Theoretically, this maximum number should be equal to the total
quantity of all Google Search web pages for public relevance and all
Google Scholar Search web pages for scientic relevance. When the
theme is present on all of these web pages, its absolute relevance
(public or scientic) is uttermost; when it is not present on any of
these web pages, its absolute relevance is equal to zero. We cannot
venture into this topic further for now, as it is related to a different
set of research questions and problems than our approach, but it a
possible path for future research in this eld.
Table 5 shows the ranking of sustainability themes and the indicators for public relevance, while Table 6 shows the ranking of
sustainability themes and the indicators for scientic relevance.

forestry
environment
biodiversity
environment
protection
environment

4a
5a
6a
6b

greenhouse gas
emissions GDP
material
consumption
waste recycling
fertilizer use
use of pesticides

The key word sets, which delivered the most average results,
are again in bold. Tables 5 and 6 transform the results from
Tables 3 and 4 into a more easy-to-grasp format. It shows the order
of the themes/indicators by their public/scientic relevance at rst
glance; however, it does not allow for deeper analysis of the mutual
relevance of the themes and indicators. This expression could be
suitable for higher-level policy-makers who need to get quick and
clear information on the relevance of the indicators they use for
decision-making, rather than for scientists and experts.
Although the key word set D contains, in general, average number of key words, it did not always deliver the most average results:
it performed best ve times out of the eight assessments while the
three other sets performed best once each of them. This proved that
apart from the number of key words the number of hits was inuenced by the level of concreteness of the key words and the usage
of phrases as well. It also shows that it made sense to calculate the
most average key word set.
The selection of the assessment method was primarily driven
by our goal to carry out the assessment in as quantitative and
objective a way as much as possible. In spite of this intention,
some room for subjectivity remained, most of all in the selection of
the key words. To minimize this subjectivity, we developed four
sets of key words and determined the one which delivered the
most average results. With respect to the indicators, we proposed
only one set of quite specic key words. Where there were multiple possible forms of these key words for one indicator (such
as material consumption, consumption of materials or use
of materials for the material consumption indicator), we tested
all the options and selected the one which delivered the highest number of hits. It can be argued that when someone uses the
same set of the key words for this set of sustainability themes
and indicators, they receive results the variability of which will be
given only by time and thus allow for time comparison. We consider this feature an important advantage of the suggested method.

52

Table 3
Absolute values for public relevance of sustainability themes and the indicators.
Indicator public relevance for the theme/per mille/

Theme public relevance/thousands of hits/

Sustainability theme
number and title

Keywords B

Keywords C

Keywords D

Indicator number and title

1a
2a
2b

Public transport

270,000

17,000

2060

119,000

Use of energy

380,000

5780

31,500

104,000

2c
3

Climate change

113,000

45,100

12,600

65,000

3a

3b
4

Use of resources

727,000

6910

106,000

868,000

4a

Waste treatment

134,000

2440

1710

32,700

5a

Agriculture

142,000

143,000

20,600

65,200

6a
6b

7
8
9

Forestry
Biodiversity
Environmental
protection

32,000
16,600
316,000

33,000
16,600
25,800

1360
3510
8640

2620
9610
56,800

ei

240E+09

210E+09

141E+09

199E+09

6c
7a
8a
9a

Transport intensity
Energy intensity
Primary energy
supply
Share of renewable
energy
Greenhouse gas
emissions per
capita
Greenhouse gas
emissions per GDP
Material
consumption
Material use of
waste
Consumption of
mineral fertilizers
Consumption of
pesticides
Organic farming
Defoliation
Bird index
Expenditures on
environmental
protection
ei

Keywords C

Keywords D

0.111
2.612
0.863

0.134
0.237
0.181

0.032
0.971
0.282

0.713

1.118

0.784

0.130

11.062

23.947

19.762

6.600

2.398

5.787

5.317

8.338

0.073

0.918

0.034

0.017

14.328

40.820

175.439

7.920

1.472

3.168

1.316

1.963

3.049

2.748

10.291

1.199

0.037
7.031
1.831
0.111

0.032
2.509
1.910
1.097

5.097
7.051
6.895
1.481

0.003
135.878
0.704
0.211

Keywords A

Keywords B

0.012
0.208
0.114

3052

1706

14,465

12,332

T. Hak et al. / Ecological Indicators 17 (2012) 4657

Keywords A

Table 4
Absolute values for scientic relevance of sustainability themes and the indicators.
Theme scientic relevance/thousands of hits/

Indicator scientic relevance for the theme/per mille/

Keywords A

Keywords B

Keywords C

Keywords D

Indicator number and title

Public transport

11,400

182

73

3160

Use of energy

33,300

69

964

4560

1a
2a
2b
2c

Climate
change

4650

867

1270

2080

3a

3b
4

Use of resources

17,000

224

1950

4900

4a

Waste treatment

5180

165

1210

2010

5a

Agriculture

5670

5670

1870

2540

6a
6b

7
8
9

Forestry
Biodiversity
Environmental
protection

1760
1010
1300

1760
1010
1430

267
348
232

1340
653
2100

ei

754E+06

148E+06

115E+06

31E+06

6c
7a
8a
9a

Transport intensity
Energy intensity
Primary energy
supply
Share of renewable
energy
Greenhouse gas
emissions per
capita
Greenhouse gas
emissions per GDP
Material
consumption
Material use of
waste
Consumption of
mineral fertilizers
Consumption of
pesticides
Organic farming
Defoliation
Bird index
Expenditures on
environmental
protection
ei

Keywords A

Keywords B

Keywords C

Keywords D

0.038
0.251
0.131

1.027
34.498
15.721

1.093
4.066
2.427

0.123
1.697
0.864

0.072

9.840

1.452

0.496

5.312

26.298

11.496

11.587

3.140

15.456

5.094

6.923

0.528

3.946

1.308

1.686

3.398

19.455

6.322

7.463

4.480

4.021

7.380

7.402

5.309

4.586

8.075

9.567

0.019
13.750
3.455
1.354

0.019
8.977
3.446
1.091

0.033
25.468
6.724
7.586

0.039
14.328
4.747
0.838

264

1148

193

T. Hak et al. / Ecological Indicators 17 (2012) 4657

Sustainability
theme number and
title

116

53

54

Table 5
Ranking of sustainability themes and the indicators for public relevance.
Indicator public relevance for the theme

Theme public relevance

Sustainability theme
number and title

Keywords B

Keywords C

Keywords D

Indicator number and title

1a
2a
2b

Public transport

Use of energy

2c
3

Climate change

3a

3b
4

Use of resources

4a

Waste treatment

5a

Agriculture

6a
6b

7
8
9

Forestry
Biodiversity
Environmental
protection

8
9
3

3
6
4

9
6
5

9
8
6

ei

23

72

17

16

6c
7a
8a
9a

Transport intensity
Energy intensity
Primary energy
supply
Share of renewable
energy
Greenhouse gas
emissions per
capita
Greenhouse gas
emissions per GDP
Material
consumption
Material use of
waste
Consumption of
mineral fertilizers
Consumption of
pesticides
Organic farming
Defoliation
Bird index
Expenditures on
environmental
protection
ei

Keywords B

Keywords C

Keywords D

14
9
10

13
6
12

13
11
12

12
7
9

10

11

12

11

14

13

13
3
6
11

14
7
8
10

7
4
5
8

14
1
8
10

11

32

57

34

Keywords A

T. Hak et al. / Ecological Indicators 17 (2012) 4657

Keywords A

Table 6
Ranking of sustainability themes and the indicators for scientic relevance.
Theme scientic relevance

Indicator scientic relevance for the theme

Sustainability theme
number and title

Keywords A

Keywords B

Keywords C

Keywords D

Indicator number and title

Keywords A

1
2

3
1

7
9

9
5

3
2

1a
2a
2b

13
10
11

13
1
4

13
9
10

13
8
10

12

11

12

10

12

14
1
5
8

14
7
11
12

14
1
6
4

14
1
7
11

32

152

45

17

2c
3

Climate change

3a

3b
4

Use of resources

4a

Waste treatment

5a

Agriculture

6a
6b

7
8
9

Forestry
Biodiversity
Environmental
protection

7
9
8

2
4
3

7
6
8

8
9
5

ei

30

78

30

26

6c
7a
8a
9a

Transport intensity
Energy intensity
Primary energy
supply
Share of renewable
energy
Greenhouse gas
emissions per
capita
Greenhouse gas
emissions per GDP
Material
consumption
Material use of
waste
Consumption of
mineral fertilizers
Consumption of
pesticides
Organic farming
Defoliation
Bird index
Expenditures on
environmental
protection
ei

Keywords C

Keywords D

T. Hak et al. / Ecological Indicators 17 (2012) 4657

Public transport
Use of energy

Keywords B

55

56

T. Hak et al. / Ecological Indicators 17 (2012) 4657

We assume that if further developed, the subjectivity might be

diminished to a very acceptable extent. This might be a task for
information science and designers of World Wide Web searching
engines.
This experimental assessment of relevance of the environmental sustainability indicator subset has delivered some useful
suggestions for further development of the Progress Report on
the Czech Republics Strategy for Sustainable Development. This
document underwent a revision in 2009 (Ofce of the Government
of the Czech Republic, 2010) and also the next edition is expected
to go through some restructuring with respect to the indicator
selection and presentation. Given the most average set of key
words, the assessment proved that the sustainability themes of
Use of resources and Use of energy are of the greatest public as
well as scientic relevance, while the theme of Forestry is not very
relevant. The practical recommendation says that relevance of
the themes should be reected by the numbers of corresponding
indicators included in the Progress Report (which is not the case
in the existing Reports; see the sustainability theme of Use of
resources, for example). The recommendation is therefore to
enlarge the numbers of indicators for the highly relevant themes.
With respect to indicator both public and scientic relevance for
the theme, it is lowest for the indicators Transport intensity and
Organic farming. For that reason, it would make sense to try complementing the corresponding sustainability themes with some
other more relevant indicators or directly replace the indicators
in place with some more relevant ones. In any case, the outcomes
suggested by the proposed method are to be debated within the
participatory processes undertaken when drafting indicators for
monitoring national sustainability strategies.
5. Conclusions
There has been no agreement or consensus on a common set of
scientic and management criteria for evaluating or assessing indicators from several points of view. We agree with Parris and Kates
(2003) that therefore, to date, there are not many (if any?) indicators and/or indicator sets that are universally accepted, backed
by compelling theory, rigorous data collection and analysis, and
inuential in policy. Therefore, the scientic community now faces
an urgent task: besides developing better indicators truly informing on societal progress towards/away from sustainability, it is
probably equally challenging to analyze and evaluate the range
of indicators presently used by a great number of institutions and
decision makers at all levels.
The proposed method for the indicator assessment based on the
criteria of relevance was carried out in a quantitative way using
the search engines Google Search and Google Scholar Search. The
assessment was done in four consecutive steps:
1. Selection of several key word sets for particular themes and indicators;
2. Inserting these key words into Google Search and Google Scholar
Search and retrieving numbers of records for these key words;
3. Calculating the ranking of sustainability themes and indicators
by their public/scientic relevance based on the numbers of
records;
4. Selection of the key word set which delivered the most average
ranking.
The results of the assessment were statistically processed and
discussed. The discussion covered the appropriateness of the
proposed procedure and the recommendations stemming from
the experimental assessment conducted for further work on the
Progress Reports. In order to further verify our approach, the next

step should be to apply it to other sets of sustainability indicators

or to a broader group of indicators for one particular sustainability
theme.
Acknowledgments
The work was supported by the 6 FP EU research grant INDILINK (Contract No. 044273) and by the grant SP/4i2/210/07 by the
Ministry of the Environment of the Czech Republic.
References
Bauler, T., 2012. An analytical framework to discuss the usability of (environmental)
indicators for policy. Ecological Indicators 17 (C), 3845.
Bockstaller, C., Girardin, P., 2003. How to validate environmental indicators. Agricultural Systems 76, 639653.
Cilibrasi, R., Vitanyi, P.B., 2007. The Google Similarity Distance. IEEE Transactions on
Knowledge and Data Engineering 19 (March (3)), 370383.
Cloquell-Ballester, V.A., Cloquell-Ballester, V.A., Monterde-Diaz, R., SantamarinaSiurana, M.C., 2006. Indicators validation for the improvement of environmental
and social impact quantitative assessment. Environmental Impact Assessment
Review 26, 79105.
EEA, 2005. EEA Core Set of IndicatorsGuide. European Environmental Agency
(EEA), Copenhagen.
EEA, 2010. EIONET: Reporting Obligations Database (ROD), Available at:
http://rod.eionet.europa.eu/.
European Communities, 2004. EU Member State experiences with sustainable development indicators. Working paper. European Communities, Brussels.
Eurostat, 2009. Sustainable Development in the European Union2009 Monitoring
Report on the EU Sustainable Development Strategy. Eurostat, Luxembourg.
Eurostat, 2010. Structural Indicators: Quality Proles, Available at: http://
epp.eurostat.ec.europa.eu/portal/page/portal/structural indicators/quality.
Girardin, P., Bockstaller, C., van der Werf, H.M.G., 1999. Indicators: tools to evaluate
the environmental impacts of farming systems. Journal of Sustainable Agriculture 13, 521.
Google Tutors Google Search Manual, 2010. Available at: http://www.
googletutor.com/.
Government Council for Sustainable Development, 2007, 2009. Progress Report on
the Czech Republic Sustainable Development Strategy. Prague: Government
Council for Sustainable Development, Prague.
Hak, T., Moldan, B., Dahl, A.L. (Eds.), 2007. Sustainability Indicators: A Scientic
Assessment. SCOPE, vol. 67. Island Press, Washington, Covelo, London.
Hardi, P., Zdan, T.J., 1997. Assessing Sustainable Development: Principles in Practice.
International Institute for Sustainable Development, Winnipeg.
Jung, S., Herlocker, J.L., Webster, J., 2007. Click data as implicit relevance feedback
in web search. Information Processing & Management 43 (3), 791807.
Kral, J., Zemlicka, M., 2006. Data and information qualityfundamental IT restrictions in public administration (in Czech). In: Pour, J., Vorisek, J. (Eds.), Systems
Integration 2006. Prague University of Economics, Prague, pp. 215222.
Krippendorff, K., 1980. Content Analysis: An Introduction to its Methodology. Sage,
Beverly Hills, CA.
Kurtz, J.C., Jackson, L.E., Fisher, W.S., 2001. Strategies for evaluating indicators based
on guidelines from the Environmental Protection Agencys Ofce of Research
and Development. Ecological Indicators 1, 4960.
Ledoux, L., Mertens, R., Wolff, P., 2005. EU sustainable development indicators: an
overview. Natural Resources Forum 29, 392403.
McMillan, S.J., 2000. The microscope and the moving target: the challenge of
applying content analysis to the World Wide Web. Journalism and Mass Communication Quarterly 77, 8098, Spr 2000.
Niemeijer, D., De Groot, R.S., 2008. A conceptual framework for selecting environmental indicator sets. Ecological Indicators 8, 1425.
OECD, 2002. Sustainable Development. Overview of Sustainable Development Indicators Used by National and International Agencies. SG/SD(2002)5. OECD, Paris.
OECD, 2005. Environmental Performance ReviewsCzech Republic. OECD, Paris.
Ofce of the Government of the Czech Republic, 2005. The Czech Republic Strategy
for Sustainable Development. Ofce of the Government of the Czech Republic,
Prague.
Ofce of the Government of the Czech Republic, 2006. Interim Report on the Czech
Republic Strategy for Sustainable Development. Ofce of the Government of the
Czech Republic, Prague.
Ofce of the Government of the Czech Republic, 2010. The Czech Republic Strategic
Framework for Sustainable Development. Unpublished.
Palmer, J.W., 2002. Web site usability, design and performance metrics. Information
System Research 13 (2), 151167.
Parris, T.M., Kates, R.W., 2003. Characterizing and measuring sustainable development. Annual Review of Environment and Resources 28, 13.113.28.
Pavlik, J., 2001. The impact of technology on journalism. Journalism Studies 1 (2),
229237.
Pintr, L., et al., 2012. Bellagio STAMP: principles for sustainability assessment and
measurement. Ecological Indicators 17 (C), 2028.
Riley, J., 2001. The indicator explosion: local needs and international challenges.
Agriculture Ecosystems and Environment 87, 119120.

T. Hak et al. / Ecological Indicators 17 (2012) 4657

UNCSD, 2001. Indicators of Sustainable Development: Guidelines and Methodologies. UN Commission on Sustainable Development, New York.
UNDP, 2005. Measuring Country Performance and State Behavior: A Survey of Composite Indices. UNDP, New York.

57

UNEP, 2007. Global Environmental Outlook. UNEP, Geneva.

UNPAN, 2000. Guidelines on Classication and Validation of UNPAN Information.
UNPAN, New York.
Wikipedia, 2010. Available at: http://en.wikipedia.org/wiki/Relevance.