ENVIRONMENTAL POLICY INTEGRATION AND MULTI-LEVEL GOVERNANCE

Co-ordinating and synthesising research on environmental policy integration and multi-level governance, 2006-2008 A Co-ordinated Action under the European Union's 6th Research Framework Programme
CITIZENS-2004-4.2.2 - Governance for Sustainable Development DG Research

EPIGOV Paper No. 7

Evaluating Integrated Impact Assessments a conceptual framework

Klaus Jacob and Julia Hertin
(Environmental Policy Research Centre, FU Berlin)
Ecologic Institute for International and European Environmental Policy Berlin, 2007 January 2007

EPIGOV is a research project on the modes of governance employed at global, EU, national and regional/local levels to support the integration of environmental concerns into other policy areas. Relevant policy areas are, for example, transport, agricultural, and energy policy. Running over three years (2006-2008), EPIGOV brings together nineteen research institutions from ten European countries.

EPIGOV aims to co-ordinate and synthesise existing research on environmental policy integration and multi-level governance and to generate new research questions and initiatives. To obtain feedback and disseminate results, EPIGOV will also involve policymakers and non-state stakeholders. http://www.ecologic.de/projekte/epigov/

Citing this EPIGOV Paper: Jacob, Klaus and Julia Hertin (2007): Evaluating Integrated Impact Assessments a conceptual framework, EPIGOV Paper No. 7, Ecologic Institute for International and European Environmental Policy: Berlin.

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

Abstract
Impact Assessment (IA) has become a standard procedure of policy making in the European Commission as well as on the national level. It aims to rationalise the decision making process by systematically collecting information about the likely impacts of a planned policy, by evaluating these impacts and thereby providing the basis for choosing the best policy option. IA is seen by many as a core instrument in the context of the better regulation agenda which aims to reduce regulatory burden on business. Recently, additional functions have been assigned to this tool, e.g. increasing the transparency of decision making, improving interdepartmental cooperation, and - crucially - addressing crosscutting issues such as environmental policy integration. With these different functions, the institutional design of IA systems has diversified. IA systems can be distinguished regarding a number of features of their institutional framework and practice, for example their timing in decision making process, the degree of transparency and openness for participation, the presence of quality control mechanisms, and the degree to which impacts are quantified. This paper - which draws on the ongoing research in the EVIA project - aims to set out a framework for exploring the contribution of IA to policy integration. This is done by firstly developing a typology of IA systems based on their institutionalisation, orientation and practice and secondly by developing a framework for the analysis of integrated IA, including explicit hypotheses about how different factors are expected to impact on the ability of IA to improve the evidence-base of policy and to promote (environmental) policy integration.

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

Table of Contents
ABSTRACT................................................................................................3 DEFINITION OF INTEGRATED IMPACT ASSESSMENTS ......................5 CHARACTERISTICS OF IMPACT ASSESSMENTS ................................8 INTEGRATED IMPACT ASSESSMENT..................................................12 APPLICATION OF INTEGRATED IMPACT ASSESSMENTS ................12 QUALITY OF IA .......................................................................................14 IMPACTS OF IA.......................................................................................16 DETERMINANTS OF QUALITY OF INTEGRATED IMPACT ASSESSMENTS ......................................................................................16 VARIABLES AND HYPOTHESES ..........................................................17 CONCLUSIONS.......................................................................................24 BIBLIOGRAPHY......................................................................................24

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

1. THE GOVERNANCE OF POLICY MAKING: BETTER REGULATION

AND ENVIRONMENTAL POLICY INTEGRATION New forms and processes of rulemaking as well as instruments to regulate and to provide public goods are not limited to the intersection of governments and enterprises or citizens, but entail the internal procedures of policy making as well. Governments themselves and their procedure to provide services, goods and to develop regulations are subject of (self)regulation. These procedures and regulations are subject of reform, and some elements of new forms of governance can be observed in this domain as well as in the regulation of private actors. In many countries as well as in European institutions, rule making is subject of Better Regulation Programs. The aim is to decrease the burden imposed by regulation, to increase the knowledge basis and hence the overall quality of regulation and to increase coherence and transparency of policies. Better Regulation Programs having several elements that vary across the countries. They entail programs to review existing laws regarding their appropriateness, effectiveness and efficiency. A recent trend in many countries are programs to review laws (both existing as well as proposals) regarding the administrative burden in terms of information requirements based on the Standard Cost Model (SCM). Some countries adopted quantitative goals to reduce the costs imposed by information requirements. The Netherlands and Germany set up high level advisory councils that review legislative proposals before their enactment regarding administrative burden. More comprehensive Impact Assessments are another central part of Better Regulation Programs. These systems will be discussed in greater detail in this paper. From an EPI perspective, it is a question open to investigation, in how far Better Regulation Programs promote or rather inhibit the consideration of environmental aspects in policy making. On the one hand, Better Regulation Programs are perceived as instruments of deregulation: Regulatory standards that impose costs on business run the risk of being lowered if no immediate and obvious benefits can be proved that outweigh the costs of regulation. For environmental standards, however, is often difficult to calculate the benefits of preserving environmental goods and accordingly, environmental policies run the risk of being watered down. On the other hand, the requirements of linking the policy proposals with broad political strategies, e.g. for sustainable development or for competitiveness and the requirements to take into consideration different kinds of possible impacts, 5

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

including side effects, trade offs and synergies on the environment, has the potential of improving EPI in the early phases of policy making. We therefore analyse procedures of Impact Assessment to explore their potential for improving EPI. We are interested, what features of IA systems and what other factors are likely to determine their contribution for an improved EPI. The paper is organised as follows: in the next section we develop a definition of Impact Assessment to distinguish such procedures from other means to develop expectations about the likely impacts of planned policies. In the next step we elaborate about common features of IA systems as well as properties that vary among the different jurisdictions. A brief review of studies about the implementation however reveals that the expectations regarding the

consideration of concerns of Sustainable Development (SD) in the Impact Assessments are not fulfilled. However, this may be justified by the nature of the policy proposal and it is not necessarily a result of political bargaining or a lack of knowledge or resources. We therefore develop a more elaborated concept of quality in regards of SD-consideration that also takes into account process variables. Finally, we explore a set of variables beyond the IA system that we expect to influence the quality of Impact Assessments.

2. DEFINITION OF INTEGRATED IMPACT ASSESSMENTS

Impact Assessment (IA) has become a standard procedure of policy making in the European Commission as well as on the national level. It aims to rationalise the decision making process by systematically collecting information about the likely impacts of a planned policy, by valuating these impacts and thereby providing the basis for a decision in which the policy is chosen with the best net benefit. Impact Assessment has been defined as an information-based analytical approach to assess probable costs, consequences, and side effects of planned policy instruments (laws, regulations, etc.) (OECD 2001, p. 10). It is distinct from assessment procedures of projects, plans and programmes such as

Environmental Impact Assessment and Strategic Environmental Assessment. Typically, IA is conceived as an assessment of distinct alternatives to achieve a specified policy objective. The specification of the objectives of a policy is usually not part of the assessment, and IA is meant to inform, but not determine the final decision. In the following, we concentrate on integrated Impact Assessments, i.e. those IA that assess several different issue areas. In order to identify such Assessments and to operationalise them for analytical research, it is necessary to develop a

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

clear definition of IA. This paper will contribute to such a definition and it aims to define different types of IA. Whenever decisions on policies are taken, this is done on the basis of certain explicit or implicit expectations about their likely effects. Such expectations are not necessarily gathered in Impact Assessment procedures. Instead, they might be derived from other forms of reasoning among the decision makers. Bureaucrats might discuss their proposals with colleagues, stakeholders, academics or politicians. Researchers attempt to forecast impacts of proposed policies either because they are commissioned to carry out a study or on own accord. A legislative proposal will be debated in parliament or it might be subject to judicial review where the court may collect evidence on impacts. Every decision taken within these arenas is justified by expectations on the future and certain effects of the concerned proposal. To develop a concept of IA that is fruitful for empirical research, we have to adopt a definition which is more specific than the OECD definition cited above which would cover all activities that aim to create expectations on possible impacts of a planned policy. We are particularly interested in recent efforts to encourage integrated Impact Assessment. Since the 1970s, a wide range of different sectoral IA have been developed: Proposals should be assessed regarding potential impacts on the economy, on health and other social aspects, on the environment, on gender, on small and medium enterprises, etc. In several countries as well as the European Union these different sectoral procedures were merged into a single procedure, in order to reduce the burden of having the obligations for many different assessment procedures and to allow a systematic assessment of interlinkages between the different issue areas. Thereby, integrated systems of IA have different historical roots. One precursor is in Regulatory Impact Assessment (RIA) that focuses on the economic or business impacts of policies and to which other aspects such as environmental or social aspects were added later. A different root is Strategic Environmental Assessment (SEA). In some countries, SEA is developed towards a sustainability appraisal by an integration of social and economic aspects (A. Chaker, K. El-Fadl et al. 2006). Within the different functions and objectives tensions may arise. In addition to the merging of different procedures, requirements are needed to deal with trade-offs, interlinkages, synergies but also conflicts between different impacts of a proposal in order to maintain the overall legitimacy of IA. A lack of legitimacy might either

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

degrade an IA into a meaningless exercise or lead to highly contested assessments that delay the process of rule-making.

3. CHARACTERISTICS OF IMPACT ASSESSMENTS

To distinguish IA from other means to develop expectations on the impacts of policies, some characteristics can be observed in practice. We suggest to use the following criteria to provide a narrower definition of IA. 1. IA is formalised: There are written regulations that define the objectives of IA, the individual steps, responsibilities, provide advice on the selection procedure of proposals and issues to be assessed. Their bindingness may, however, vary: The rules may only be formulated as a suggestion or as binding prescriptions for the rule making departments. Requirements for IA can be issued by law, presidential or prime ministerial decree, cabinet directive, cabinet decision, resolution, etc. (OECD 2004) 2. IA has defined purposes: IA is an instrument to achieve governmental objectives beyond the purposes of the individual proposal. It is related to generic objectives to which a government is committed: Improving economic performance, achieving gender equality, avoiding risks to health or to the environment, etc. IA aims to reveal the contributions of an

individual proposal on these generic objectives and the side effects and interlinkages between the main purposes of the proposal and these other objectives. However, some IA systems do not provide substantive criteria, instead they only require the assessment of the effectiveness and efficiency of the planned regulation. These systems of IA do not challenge the objectives of the departments. 3. IA is fact-based: IA is a systematic, scientifically grounded process to identify the likely impacts on a defined list of issue areas: It should be robust, transparent and politically neutral. If value-based judgements cannot be avoided, they should be made explicit. Again, this might be different in practice. While these three characteristics are common to IA, we can identify a number of other significant qualities on which there is considerable variation between countries: 1. Timing in decision making: IA is often described as a separate step of decision making which takes place after the definition of objectives and before the final decision on a policy proposal is taken. IA does not

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

question the objectives of a proposal nor is it supposed to pre-judge the political decisions. Often, the guidelines stipulate a specific timing for the IA, i.e. before the interdepartmental consultation, before the parliamentary decision, etc. However, the distinction of problem definition and setting of objectives on the one side and assessment of impacts on the other side is often not very clear cut in reality: Firstly, objectives are most often vaguely defined, they are changed during the decision making process (and accordingly during the IA), sometimes even through the IA.

Acknowledging this, in some IA systems, the definition of problems and objectives is seen as an integral step of the IA. Secondly, it is not uncommon that IA takes place after the key decisions are taken, it then largely fulfils formal obligations and contributes to the justification of a proposal. 2. Degree of transparency and disclosure: Countries vary regarding the degree of transparency and the disclosure of IA to the public. In some countries, IA is inevitably open to the general public because it is perceived as a tool to increase transparency of rulemaking. Examples are UK, USA, Canada, and the European Union. In other countries, a dominant purpose of IA is the fostering of interdepartmental negotiations and transparency of rulemaking is not perceived as a goal in itself. In these jurisdictions, IA are not published to the general public. In some countries, IA are attached to the legislative proposal and published by the parliament. 3. Degree of involvement of external expertise: IA is typically a duty of the rulemaking department. Frequently, other departments will be involved in delivering data, contributing to the assessment or reviewing the results. IA can be commissioned from other agencies, consultants or academics. Sometimes, IA is commissioned or performed by non state actors that expect to be particularly affected by the planned policy. Likewise, other levels of governance, i.e. the national or even the regional level for European policies or the regional level for national policies may review the IA or perform own IA with their specific needs. The commissioning of IA studies took place extensively during the decision making process on REACH. Up to now, there are no specialised IA departments established, although this could be thought of, in particular for sectoral IA such as Strategic Environmental Assessments.

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

4. Participation: In few countries as well as in the European Union, IA is a frame for the participation of non governmental actors in the development of a proposal. The inclusion of interest groups may have different functions: Either to discuss or to negotiate the proposal with the groups affected or to obtain data . However, in other countries as well as in practice of those countries where participation is foreseen, the inclusion of societal interests is often separated from IA. 5. Main purpose of the IA: In some systems of IA, the function of reducing costs of regulation is the prevailing function of IA. The main focus is on reducing costs of regulation and maximising benefits. Other IA aim to inform the decision maker on specific issue areas (environment, social, health, gender aspects, SMEs, etc). A third category that has gained recently some importance are systems of IA that should contribute to the implementation of Sustainable Development. Typically, these IA covering social, economic and environmental issues. 6. Role of quantification: There are different degrees of formalisation and rigidity regarding the use of quantitative approaches and economic modelling. In some systems of IA, few generic questions are provided and there is no indication that specific methods are proposed or favoured. The questions may be answered qualitatively. Other systems putting great emphasis on quantification and monetisation wherever this is possible. 7. Scope of IA: Most often, IA is limited to new legislation. Countries vary in the inclusion of subordinate regulations. Sometimes governmental strategies (white papers) and international treaties are also subject of assessment. In few countries, existing regulation is subject of IA. 8. Degree of detail: Reports on IA vary considerably regarding their volume (from few pages to hundreds of pages) and accordingly the level of detail. IA procedures may be limited to checkboxing a few generic questions or by an extensive and detailed evaluation for every issue. In some systems for IA, IA is separated in a preliminary scoping phase and an extended IA. 9. Quality Control: Countries vary regarding the existence and strength of institutions to control the quality of IA. Three groups may be distinguished: In some countries, there are no institutions at all, the compliance is left to the departments responsible for the IA. In other countries, there are institutions that should provide assistance to the departments, but they have no rights to sanction an IA. Finally, there are countries with institutions that have the power to impose obligations to the departments

10

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

based on quality. These institutions can be either allocated at the head of government, or independent from government (court of auditors, advisory committees, parliamentary committee). Theoretically, a quality review

could be achieved also by enabling interest groups to object against an IA; in case of disputes someone could think even of a judicial review. Based on these considerations, the following types of IA can be distinguished: Issue Timing Possible types Before decision is taken Justification of decision (Ex post IA) Transparency and disclosure Open: IA published as separate

documents Open: IA attached to the proposal Closed External involvement Performed by the rulemaking unit Performed by specialists (consultants, academics, other units of government) Commissioned by interest groups Participation None, source of data, negotiation of the proposal Main purpose reducing costs of regulation sectoral/specialised IA Implementation Development Quantification Quantitative approach favoured Mainly qualitative approach Scope New legislation Subordinate regulation General strategies International treaties Existing legislation Degree of detail Short IA Detailed IA Separation of scoping and analysis (preliminary/extended IA) of Sustainable

11

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

Issue Quality Control

Possible types None Weak: primarily assistance Strong: external review and sanctioning

4. INTEGRATED IMPACT ASSESSMENT

As rightfully noted by Scrase and Sheate 2002, the term 'integration' can refer to many different issues in IA: to policy integration, integration of knowledge and information sources, stakeholders, issue areas, different environmental media, tools, etc. For the purposes of this paper we are primarily interested in the integration of different impact areas in a single assessment procedure. We are interested in how far such integrated IA are an instrument to integrate cross cutting concerns such as environmental protection into the decision making procedures of specialised departments. The integration might be achieved by procedural and/or by substantive standards for the IA. For example the consideration of environmental concerns could be ensured that the Ministry for the Environment is involved in the assessment procedure. Another means of integration is the definition of specified requirements and the linking of the assessment procedure to other strategy processes. Such substantive

requirements could be a list of questions in the guidelines, or a reference to the national strategy for sustainable development. It is therefore not possible to decide from the outset if an IA system and its practice is a fully integrated IA. Even more complicated, most often principles of proportionality are defined so that in early stages of the assessment procedures issue areas are excluded from IA may also have undesirable effects on the regulatory process, for example by enabling early lobbying activities, delaying decisions, or legitimising decisions that were not open to a public debate. This may lead to a sidelining of weaker interests, such as those representing social or environmental concerns. On this background of different issues, actors and functions of IA, it remains a question open to investigation whether the balance between different objectives can be secured through certain institutional provisions. In the following, we analyse in how far integrated IA systems are actually applied, drawing on our own research as well as on a number of evaluation studies.

5. APPLICATION OF INTEGRATED IMPACT ASSESSMENTS

Although RIA is a very widespread procedure in the OECD (Radaelli 2004), it is not always clear to what extent assessments go beyond a narrow assessment of

12

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

direct economic and administrative costs. The OECDs Environmental Policy Reviews mention the USA, Canada, the UK and the Netherlands with explicit requirements to assess environmental impacts of planned policies (Jacob and Volkery 2007). USA and Canada introduced a requirement to assess policy proposals already back in the late 1960s and early 1970s. These procedures, however, were hardly applied (Bartolomeo, Giugni et al. 2005). A recent pioneer is the UK that developed a system for policy appraisal already back in 1990 (Jordan and Lenschow 2000). The Netherlands developed tools for policy appraisal of proposed legislation in 1994 (the E- und B-Test, see Marsden 1999). Currently, Switzerland is also experimenting with Sustainability Assessment (Federal Office for Spatial Development ARE (CH) 2004). The Commission has introduced an integrated assessment procedure for all legislative proposals (European Commission 2002; European Commission 2005; Hertin, Jacob et al. 2007). Since then, the European approach has been adopted by several Member States (Ecologic, Vrije Universiteit Amsterdam et al. 2007). There are few evaluations of the practice of integrated IA, and most of them focus on the European Commission's system. The first three years of experiences reveal some several shortcomings of the process (see Wilkinson, Fergusson et al. 2004, Renda 2006, Bartolomeo, Giugni et al. 2005, Institute for Miljovurdering / Environmental Assessment Institute 2006, EEAC Working Group on Governance 2006): IAs are not always performed with sufficient quality regarding potential environmental impacts: In many cases, IA is focused on the intended impacts of a policy, while unintended side effects are not covered sufficiently as for example in the IA on the Services Directive. IAs are sometimes used to overemphasise costs compared to benefits of policies: There is a lack of profound methodologies, data but also incentives to assess the benefits of planned policies, in particular regarding non quantifiable and long term effects. The consultation of stakeholders varies considerably, and factual differences need to be taken into account: Environmental organisations often lack the resources for an adequate involvement in the process, which leads to an overrepresentation of industry. IA may also open up opportunities for early lobbying if not carefully designed. The principle of proportionality is not sufficiently defined: The selection of policy options, the scope of the assessment and the definition of

13

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

boundaries of IA can be misused to neglect environmental aspects (or other unwanted issues) at an early point of time in the design of the IA. IAs do not always cover appropriately likely modifications at later stages of the decision making process and implementation in member states. Typically, the evaluations are based on an analysis of the IA reports according to a set of criteria or as case studies. A recent study by Ecologic and others studies also the practice in selected Member States and draws a more optimistic picture: Environmental aspects are considered in the majority of cases and the results have been considered in the decision making as well (Ecologic, Vrije Universiteit Amsterdam et al. 2007). It is open to investigation whether these results are generalisable as the number of cases is rather small and the selection may have favoured policies in which environmental aspects are a direct and obvious issue, while other studies reveal a sidelining of environmental effects if these are rather side effects. Until now, much attention has been paid to the improvement of the IA systems: Observers recommend the improvement of the IA system, placing more emphasis on requirements, to include environmental actors at every stage, to scrutinize the quality of IAs, and to spend more efforts in the development of tools and methods for assessing costs and benefits for the environment (Volkery and Jacob 2005, Wilkinson, Monkhouse et al. 2005, Jacob, Hertin et al. 2007). Other aspects that may explain the consideration of environmental aspects, such are the type of the policy proposal, the capacities available, etc. have received less attention. To analyse if the consideration of environmental aspects was taken seriously, and to understand if a lack of analysis can be attributed to the IA system, we develop a notion of quality of IAs with a focus of integration of cross cutting aspects and we develop a set of hypotheses about influencing factors of the quality.

6. QUALITY OF IA
How can we assess the quality of IA? Many IA evaluations focus on the quality of assessment reports - for example their scope, level of detail and methodological rigour - not least because it can be easily analysed on the basis of a document review. Empirical research has shown, however, that high-quality analysis is not a guarantee for impact on decision-making as it is not uncommon that IA reports are prepared at a late stage in the decision process. It is therefore important to evaluate the process as a whole. We have identified the following criteria to assess the quality and impact of IA (see Meuwese, Radaelli et al. 2006):

14

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

Process: Does the Process contribute to an integration of cross-cutting issues? Actual timing of IA. An IA conducted at the very end of the decision making is unlikely to have impacts on the proposal. A late IA indicates poor quality. Procedural integration. In how far were the different departments involved in a certain IA? At what point of time and on which issues (defining of problem, objectives, identification of options, assessment of options, etc.) have the different departments been involved? How is it justified to dismiss departments in the setup and conduct of an IA. Vertical integration. In how far are the requirements, constraints, interests, opportunities of the sub-national (or in case of EU: national level) dealt with? In how far were representatives from the sub national level being involved in the setup and the conduct of IA? Review. Did a review of the IA actually take place? Was the assistance offered by IA units used? Purpose: Is the integration of cross-cutting issues made explicit? Substantive consideration of cross-cutting issues. In how far are the different dimension of sustainable development considered, in how far are external effects and long term problems considered. In how far does the IA consider possible side effects of a planned policy and goes beyond the immediate objectives of a proposal? In how far is it justified if certain issue areas are not considered? Reference to specified problems and objectives. In how far does the IA/the proposal address clearly defined problems and refers to operationalised objectives. In how far is the baseline considered and is this done in a plausible manner. In how far is the identification of net benefit the explicit goal of the IA (versus the reduction of compliance costs for firms). Sophistication: Is the IA elaborated and scientifically grounded? Evidence base. Has evidence been collected and is it robust?. In how far is the selection of the favourite option based on the evidence collected (instead of e.g. interests of the target group). In how far are uncertainties addressed in the IA and how are risks dealt with arising from uncertainties? In how far is the lifetime of a policy considered?

15

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

In how far are possible negative effects considered and in how far are the benefits of a proposal considered? To what degree are costs and benefits quantified?

In how far are potential side effects on other policy areas and objectives considered?

Openness: Is the IA transparent and open for stakeholder involvement? Participation. In how far were stakeholders involved in the design and the conduct of IA. Is an involvement of non-governmental actors foreseen? In many countries only the results of the IA are made public.

7. IMPACTS OF IA
We assume that a good IA is likely to have impacts on the final policy decision. However, it is difficult to attribute changes to the policy to an IA, as there are many different factors, institutions, and actors that influence the final decision. Therefore, one appropriate way is to use interviews with desk officers to develop a counterfactual 'no-IA' scenario. Questions may refer to: Did the IA have a direct impact on the design of the IA? In what way? Has the IA process increased the knowledge available for the policy process? If the IA contributes to a more cooperative decision making is the evidence convincing for other institutions/actors? Did the IA contribute to a consensus/improved understanding among the actors involved? Was the IA used or referred to during the political process (in Parliament, by stakeholder?) Is the policy formulation more transparent with IA (question addressed to the stakeholders as well) Were the requirements of sustainable development (different stocks, long term focus and external effects) considered in the policy proposal and can this attributed to the IA?

8. DETERMINANTS OF QUALITY OF INTEGRATED IMPACT ASSESSMENTS

We expect the quality of IAs and the resulting impact on policy decisions to be a result of different groups of variables within the institutional framework of decision making, the design of the IA system, the decision making context, the type of policy, resources available and the use of analytical methodologies.

16

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

The variables can be divided in three different groups according to their relative stability and the opportunities to change and influence. Three different levels may be distinguished: a systemic level which are rarely and slowly changing, a policy level, i.e. the specific proposal and the context of an individual policy and thirdly the specific setup of an IA. Amongst the independent variables, we expect particularly strong links between the two groups of systems variables: the institutional framework and the design of the IA system. This relationship will therefore be analysed in more detail (see section 3 below).

Independent variables

Dependent variables

System variables

Institutional Framework

Design of IA System

Policy variables

Decision Making Context

Quality of IA

Type of Policy

IA variables

Resources for IA Impacts of IA Methodological Approach

Figure 1: Explaining quality of Impact Assessments

9. VARIABLES AND HYPOTHESES

In the following, the variables will be explained in further detail to develop a common understanding and terminology and to describe the underlying hypotheses. For some variables we have no explicit hypotheses but instead we have the opportunity to explore the relations between the respective variable and the quality of IA. The aim of this paper is to provide a rich overview of potentially relevant variables and relationships. We should keep in mind, however, that it will not be possible to

17

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

accurately measure all those variables, especially in a way that is consistent across jurisdictions and would allow robust comparative analysis. Once data collection has been completed we should therefore review the analytical framework with a view to narrowing it down to focusing on the more operational variables. Independent variables System Variables Institutional Framework The institutional framework includes the constitutional and basic legal framework of a jurisdiction, its political system, political and administrative culture, including the strength and the role of interest groups. We can expect the following factors to be particular relevant for IA: Degree of independence of individual departments from a central coordinating department. If a central department exists that may refuse individual proposals brought forward by the ministries, this may serve as a powerful institution for the control of quality. Furthermore, such a central unit may develop an own stake in IA, as this is a mean to oversee and to control the activities of the various departments. On the other hand, a large degree of independence of the ministries e.g. by guaranteeing them own rights to initiate proposals, by providing them with the power to veto proposals in the cabinet, etc. may weaken the interest to conduct integrated IA. Number of political institutions in the decision making context. The larger the number of institutions involved in the decision on policies, the less likely it is that IA plays a role, while other, more formalised mechanisms for coordination between these actors prevail. Federalism, Bicameralism, independent institutions such are national reserve bank or a constitutional court are examples for potential veto players in the political process. Judicial review and the existence of an administrative procedure act is on the one side an incentive to prepare well founded IAs to minimise the risk of refusal. On the other hand, the judicial review may be a functional equivalent to IA, as the officers in charge for the development of the proposal anticipate the review and consider possible side effects or other issues in early stages as if informed by an IA. The role of interests groups. The role of interest groups in the policy process varies not only for the political system as a whole, but also for the specific policy proposal. This will be captured in the Decision Making

18

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

Context (see below). However, the level of corporatism and the access of interest groups to the decision making is relevant also on the system level. In this case, IA may become subject of political bargaining. On the other hand, if there is a stronger inclusion of non-state actors, it is more likely that participatory approaches are applied.

Design of the IA system

IA is defined as a formalised procedure with a defined purpose beyond the individual proposal to systematically collect facts on the likely effects of a planned policy. While these elements are common, there are a number of differences concerning provisions in relation to timing, transparency, use of external expertise, transparency, quantification, scope, detail and quality control. It is important to note that these variables concern the system-level requirements (as for example expressed in guidance documents) rather than the actual characteristics of IAs which will be considered under 'Resources for IA', 'Methodological approach' and 'Quality of IA'). Basic orientation of the procedure: There is a considerable variation in the main function(s) attributed to IA in different jurisdictions. The main different orientations that can be observed are: improving the quality of regulation, reducing regulatory burden, enhancing transparency,

promoting sustainable development. We would expect the basic orientation has an influence on the dominant choice of methodologies and on a range of quality variables (level of integration, sophistication, openness etc.) Foreseen timing in the decision making process. Guidance documents make different provisions regarding the timing of the assessment. We expect an IA to be of higher quality if it is required at early stages of decision making. However, while some countries require the IA only when a proposal is forwarded to the parliament, others require it for the interdepartmental consultation (EC) or even before the consultation (Poland). Degree of transparency and disclosure. We expect a higher quality if the IA is subject of public review and the results have to be published. However, the disclosure of the IA may inhibit an open debate, encourage the lead ministry to use IA purely defensively, and opens up opportunities for interest-based lobbying by stakeholders. On the other hand, a high level of transparency may make it more difficult to pursue private interests

19

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

at the expense of public interests, therefore generic issues of SD are likely to be more relevant in transparent IA. Provisions for the involvement of external expertise. IA systems may be distinguished with regard to the degree to which external expertise is consulted. The use of data and consultancy may open up opportunities to influence decisions. However, in many cases external support is likely to improve the quality as officers lacking time and specialisation to gather and process the needed data on their own. Participation. In few countries the participation of stakeholders is foreseen not only in the policy proposal, but also part of the IA. This may take different forms, e.g. the participation in the design of the IA, the provision of data, the comments on the results. We expect participative approaches to be favourable to the consideration of SD issues in IA. Quantification. IA systems vary considerably regarding the emphasis placed on quantification of impacts. It is a question open to investigation if this contributes to an integration of the different dimensions of SD or if this narrows down the perspective on costs for business only. Scope of IA. While in most countries only laws and subordinate regulations are subject of IA, the EC system also requires IAs for Strategies. Some countries started experimenting with an application on existing law. As we decided to examine only laws in EVIA to keep the cases comparable we cannot explore if IA is suitable for other governmental activities. We may expect that strategies are more difficult to be assessed as they leave much room for interpretation on the actual implementation. Degree of detail: Guidelines for IA vary considerable regarding the requirements on the degree of detail. Often, a principle of proportionality is included in order not to overburden the officer with irrelevant requirements. If this principle of proportionality is not substantiated and if it leaves discretion to the officers, the IA is likely to be very short (with some cases in one sentence no unfavourable impacts are expected), and accordingly the quality is poor. Although lengthy IAs are not necessarily an indication of good quality, we expect good quality in more comprehensive IAs. Quality control and assistance. There are many different institutions and procedures developed for quality control. Three different types may be

20

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

distinguished: no control at all, supporting units without the power to sanctioning poor quality IA, and finally institutions that have an however varying degree of independence and possibilities for interventions. We expect the quality of IAs to be better if there is a stronger quality control and if there are more resources available for assistance. It is, however, important to keep in mind that guidelines and actual implementation often fall apart. This is partially due to the opportunities and restrictions to actually apply the generic framework on specific policies as well as to the resources available for a specific IA. Therefore, both factors will be elaborated in further detail in the following.

Policy variables
While the institutional framework and the design of the IA systems are rather static, IA has to be applied on a great variety of policy initiatives that are related to different problems and interests of actors. Two categories of variables may be distinguished: firstly we expect the decision making context, i.e. the definition of the problem, the actors involved, the specifics of the responsible unit and the expectations in a certain IA are likely to make a difference for the quality of IA, as well as the type of policy that is subject of the IA.

Decision making context

The decision-making context covers variables related to the origin of the policy problem (agenda setting), the relevant network of actors and the function of IA (i.e. the aims key actors try to achieve). Definition of the policy problem: The decision making context is firstly determined by the definition of the problem. Problems may firstly come on the agenda by scientific studies, by monitoring, scientific agencies, etc. The initiative comes most often from within the administration. Policies are hardly discussed in the media. Another mode of defining problems is driven by interests groups, the heads of departments, or the parliament. Policies are developed on response of political decisions. Thirdly, policy problems may be derived from external initiatives, i.e. the national implementation of international or European law. We expect IA to play a greater role when the definition of problem is controlled by the administration compared to political problems or external initiatives. In the latter cases, there is much less room for manoeuvre to assess different options.

21

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

Furthermore, the IA will be determined by the actors affected that are targeted or involved in the policy. If these are powerful and well organised interests, an IA is more often likely to be subject of bargaining. However, with interest groups having stakes in the process, this may serve as quality control, they may provide data and technical assistance, etc. In case of large, unorganised groups (e.g. consumers), there is neither support nor are interventions likely. No clear expectations can be formulated in this regard, but we have to explore the relationship in the case studies. Furthermore, we can expect the relationship between the regulators and the target groups of relevance: if an antagonistic relationship prevails, the quality of IA will be less good compared to an cooperative relationship

Use by lead department: Finally, the role an IA plays in practice will depend on how the lead department aims to use it. Research has shown the intuitive use of assessment - learning about policy design - only one possible function of knowledge in policy (Shulock 1999, Owens 2004). Empirically, four functions of IA have been observed (Hertin 2007): 1) symbolic use (using assessment as a substitute for policy), 2) strategic use (producing evidence to support a certain position, e.g. to defend the proposed policy) 3) learning (gathering knowledge to improve the design of the policy), and 4) formalistic use (fulfilling administrative obligations). We would expect that both quality and impact of IA is likely to be highest if it is used in a learning mode (3), and lowest if it only to fulfil an obligation (4). Although we located this variable at a policy-level, it should be explored to what extent it relates to departmental cultures as anecdotal evidence suggests, for example, that environment ministries tend to use IA strategically.

Type of policy
Type of Policy Instrument: a great number of typologies has been developed to categorise policy instruments. A standard classification distinguishes 1) information based instrument, 2) market based instruments, 3) regulation, 4) voluntary agreements/self regulation, 5) spending programmes, and 6) planning instruments. These instruments vary in their degree of bindingness for the target group. Accordingly, the impacts are more difficult to assess if the outcomes are not determined.

22

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

Rationale for intervention: Another dimension of the policy type refers to the objectives of the intervention. Policies that aim for a re-distribution of resources, or that are based on values that are not consensus in a society are more difficult to accomplish than policies that are distributive or that aim for a coordination of action and that are pareto efficient. The first type of difficult policies are likely to meet on resistance by affected groups, thereby, the IA will be subject of a political debate. The second category of easy policies is less likely to become subject of bargaining, and the mode of arguing will prevail.

Maturity of policies. Policies might be developed from the green table, in particular if this refers to a new policy field in which institutions have not been evolved. In these cases it is easier to develop possible alternatives and to compare impacts. The maturity of policies may be indicated by the age and the importance of the political institutions involved and their path dependency in developing new proposals. The type of proposals is another indicator, ranging from green paper, white paper, action plan, new policy to amendment of existing policy. We expect the quality of IA to be higher the less mature a policy field is as the assessment is likely to be open to alternative options. On the other hand, well developed institutions, their experience and their resources may improve the quality of IA as well.

IA variables
Finally we expect the operational design of IA to be of relevant for the quality of IA. The actual design is not only determined by the guidelines and the type of policy at stake. Instead, 1) the resources available and 2) the methodologies used are of importance in his regard.

Resources for IA
A number of constraints can be thought of when setting up an IA project within a department. IA might be limited by the availability of staff and their training and expertise, money to commissioning studies, availability of data and pervious studies. Furthermore, IAs are performed often in great time pressure. We expect that quality improves with budget, time, staff and their training.

23

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

Methodological approach
In our case studies we will explore in how far the actual application of formalised methodologies will influence the quality of IA. A number of different methodologies can be categorised: Modelling tools: three different families of models can be distinguished, i.e. economic models, biophysical models, social models. Tools to support stakeholder participation of stakeholders Comparison tools such as Monetary Assessment Tools and Multicriteria Analysis methods We expect particularly more stringent and rigid quantitative tools to favour greater depth of analysis, but broader and more flexible qualitative (or semi-qualitative) tools to be more akin to policy integration.

10.

CONCLUSIONS

In this paper we have set out a conceptual framework of and put forward hypotheses about integrated Impact Assessment. The aim was to develop an empirical research agenda for studying the contribution integrated IA may make to improving integration of cross-cutting policy objectives (such as the protection of the environment). In the next phase of the EVIA project, we will undertake to apply this framework to the study of IA practice in several EU countries through three methodologies: 1) a (largely desk-based) analysis of national-level IA procedures in all EU Member States, 2) a set of case studies of individual IAs in five selected jurisdictions and 3) a survey of desk officers and stakeholders in two countries.

11.

BIBLIOGRAPHY

A. Chaker, K. El-Fadl, et al. (2006). "A review of strategic environmental assessment in 12 selected countries." Environmental Impact Assessment Review 26: 15-56. Bartolomeo, Matteo, Piero Giugni, et al. (2005). Approaches to Impact Assessment in six OECD countries and at the European Commission. Findings and recommendations for the European Commission. Unpublished Report. Milano, Avanzi Ecologic, Vrije Universiteit Amsterdam, et al. (2007). Improving Assessment of the Environment in Impact Assessment http://www.ecologic-events.de/euimpact-assessment/en/documents/Env_in_IA_final.pdf. EEAC Working Group on Governance (2006). Impact Assessment of European Commission Policies: Acheivements And Prospects. Brussels http://www.eeac-

24

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

net.org/download/EEAC%20WG%20Gov_IA%20statement_final_18-506.pdf. European Commission (2002). Communication from the Commission on Impact Assessment. Brussels http://europa.eu.int/comm/secretariat_general/impact/docs/COM_2002_0276 _F_EN.pdf. European Commission (2005). Impact Assessment Guidelines. SEC (2005) 791. Brussels http://ec.europa.eu/governance/impact/docs/key_docs/sec_2005_0791_en.p df. Federal Office for Spatial Development ARE (CH) (2004). Sustainability Assessment. Conceptual framework and basic methodology http://www.are.admin.ch/are/en/nachhaltig/beurteilen/index.html. Hertin, Julia, Klaus Jacob, et al. (2007). Policy Appraisal. Innovation in Environmental Policy? Integrating environment for sustainability. A. Jordan and A. Lenschow. Institute for Miljovurdering / Environmental Assessment Institute (2006). Getting Proportions Right How far should EU Impact Assessments go? Copenhagen http://www.imv.dk/Files/Filer/IMV/Publikationer/Rapporter/2006/getting_propo rtions_right.pdf. Jacob, Klaus, Julia Hertin, et al. (2007). Impact Assessment and Sustainable Development: European Practice and Experience. Impact Assessment for a New Europe and Beyond. C. George and C. Kirkpatrick. Celtenham, Edward Elgar. Jacob, Klaus and Axel Volkery (2007). Instruments for Environmental Policy Integration in 30 OECD-Countries. Innovation in Environmental Policy? Integrating environment for sustainability. A. Jordan and A. Lenschow. Jordan, Andrew and Andrea Lenschow (2000). "'Greening' the European Union: What can be Learned from the 'Leaders' of EU Environmental Policy?" European Environment 10: 109-120. Marsden, Simon (1999). "Legislative EA in the Netherlands: The E-Test as a Strategic and Integrative Instrument." European Environment 9: 90-100. Meuwese, Anne, Claudio Radaelli, et al. (2006). Development of a preliminary set of indicators for the quality of IA. Exeter, University of Exeter OECD (2001). Improving Policy Instruments through Impact Assessment. Sigma Paper No. 31. Paris, OECD OECD (2004). Regulatory Impact Analysis (RIA) Inventory. Paris, OECD http://www.oecd.org/dataoecd/22/9/35258430.pdf. Radaelli, Claudio (2004). "The diffusion of regulatory impact analysis: Bestpractice or lesson drawing?" European Journal of Political Research 43(5): 723-747. Renda, Andrea (2006). Impact Assessment in the EU: The State of the Art and the Art of the State Brussels, CEPS. Scrase, Ivan J. and William R. Sheate (2002). "Integration and Integrated Approaches to Assessment: What Do They Mean for the Environment?" Journal of Environmental Policy and Planning 4: 275-294.

25

Jacob/Hertin: Evaluating Integrated Impact Assessments a conceptual framework

Volkery, Axel and Klaus Jacob (2005). The Environmental Dimension of Impact Assessment. Documentation of a Workshop organised together with the Federal Ministry for the Environment, Nature Conservation and Nuclear Safety, 17-18 June 2004, Berlin. Berlin, Environmental Policy Research Centre http://www.fu-berlin.de/ffu/download/Rep-01-2005.pdf. Wilkinson, David, Malcolm Fergusson, et al. (2004). Sustainable Development in the European Commission's Integrated Impact Assessments for 2003. London, IEEP http://www.ieep.org.uk/publications/pdfs//2004/sustainabledevelopmentineuc ommission.pdf. Wilkinson, David, Claire Monkhouse, et al. (2005). For Better or for Worse? The EUs Better Regulation Agenda and the Environment. London, Institute for European Environmental Policy http://www.ieep.eu/publications/pdfs/2005/forbetterorforworse.pdf.

26