Research Evidence and Policy Making

Administration, vol. 60, no. 2 (2012), pp.
119138
Research evidence and policymaking
in Ireland
Frances Ruane
Director, Economic and Social Research Institute
1
Introduction
The growing international literature on policymaking processes, which
draws on a range of different disciplines and perspectives, emphasises
the importance of governance and the use of evidence for
policymaking.
2
The literature on evidence for policy has been well
developed in the UK, particularly in the health and education areas.
3
There has been a growing emphasis on this topic in Ireland, with it
getting increased attention at conferences and in lectures over the past
decade.
4
However, many in the relevant research community
5
did not
engage actively with policy-related research or with policymakers until
very recently.
6
119
1
This paper draws on the content of a lecture delivered at the Royal Irish Academy
(RIA) to mark the fiftieth anniversary of the Economic and Social Research Institute
(ESRI). It has benefited from comments from attendees, from colleagues at the ESRI
and from a number of policymakers who read earlier drafts. The views expressed are
personal to the author and should not be interpreted as reflecting the views of the ESRI.
The usual disclaimer applies.
2
Some of this work involves the creation of data sets that can support further research.
See, for example, Hardiman et al. (2012).
3
The ESRC Centre for Evidence Based Policy and Practice has provided a focus for
these discussions in the UK, where much of the emphasis has been on health and
education. See Young et al. (2002).
4
See, for example, National Economic and Social Forum (2007) and the papers by
Ruane & Whelan (2011) and Hearne & Watt (2011).
5
These are social science researchers primarily based in higher education and research
institutions.
6
There have been several policy conferences since 2008 and these have been well
attended by both researchers and policymakers. As time has progressed, the range of
topics has widened from the macroeconomics and finance area to a range of other policy
areas.
09 Ruane forum_Admin 60-2 25/07/2012 09:32 Page 119
The need to ensure social cohesion in the face of difficult decisions
demands a robust policymaking process. Using research evidence to
help improve policymaking has much to commend it, especially as
tolerance for ineffective policies and wasteful use of resources is now
very low.
7
This paper looks at the research/policymaking interface.
The paper draws on personal experience in the research and
policymaking space.
8
For much of the past three decades, Irish policy
design appears to have been dominated by administrative concerns,
with less attention given to economic and social factors. The one major
exception to this was the set of programmes supported by EU
Structural Funds, where strong planning and evaluation processes
were put in place. The move away from robust ex ante and ex post
evaluation coincided with the ending of these projects. This, combined
with Irelands exceptionally poor record in physical planning
(especially during the years of very fast growth, referred to below as
boom years),
9
has left us with major policy liabilities that need
attention.
10
Additionally, with some notable exceptions, there was
little attempt in the early 2000s to measure robustly the impact of
policy and little obvious commitment to the use of data/analysis to
inform policy within line departments.
11
Furthermore, until recent
times, many stated policy objectives were poorly specified, and policy
interventions often conflicted with these objectives.
12
This paper is relevant to research in areas most directly linked to
policy economics, sociology, social psychology, political science,
120 FRANCES RUANE
7
This issue has been receiving increased media coverage since 2008/9.
8
This reflects over thirty years of engagement in different capacities. It also takes into
account my present perspective as Director of the ESRI, whose mission is to deliver
research that is relevant to policy.
9
The National spatial strategy (NSS) was launched in 2002, after a decade of rapid
growth, which had been supported by the EU Structural Fund Programmes. The NSS
appears to have had little positive impact on national physical planning in the mid 2000s.
Specifically, there seems to have been little coherence in the relationship between
housing, transport and land use. Furthermore, the gateway towns identified in the NSS
were ignored in the governments decentralisation plans, which were announced after it
had been published.
10
The built environment and the sustainability of some activities that were funded when
money was more plentiful (e.g. higher education) are particularly important areas
requiring attention.
11
The transfer of the Department of Finances evaluation unit to Tullamore as part of
the decentralisation plan was seen by many as providing a signal that such activities were
not being prioritised.
12
Specific examples can be found in the original RIA lecture in ESRI Working Paper
No. 354 (Ruane, 2010).
geography, demography, management, etc.
13
However, and while
recognising that multidisciplinary approaches are often required, the
papers main focus is on economics, as concepts and frameworks from
this discipline are of particular relevance in the present climate.
On the policymaking side, the focus is on those civil and public
servants who analyse policy possibilities and present options
(sometimes accompanied by recommendations) to politicians. The
term policymaker is used to refer to them, following common usage of
that term in Ireland.
14
The paper does not discuss the policy interface
between senior civil/public servants and politicians. Clearly, the
attitude of politicians towards the use of evidence to inform their
decisions impacts on the emphasis that is placed on such evidence
within the civil/public service. Finally, the paper does not discuss
either the engagement between researchers and politicians or the roles
played by researchers in policymaking bodies.
This discussion is set against a background where major policy
challenges must be addressed if Irelands economy is to get back on
track and social harmony is to be maintained. Even the most cursory
review of the rapid increases in current spending over the boom years
suggests that value for money played an inadequate role in decision-
making.
15
This was reflected in the growing gap between the rhetoric
and reality in the late 1990s/early 2000s in relation to evidence-based
policy and evidence-based approaches to evaluation.
16
Over this
period, money was spent on many nice to have rather than need to
have projects and programmes, with little apparent regard to
Research evidence and policymaking in Ireland 121
13
In practice, many policies cover areas where the content of other disciplines is highly
relevant, such as medical science, environmental science and engineering.
14
To people from outside Ireland, this term may look like a misnomer. However, it has
become common parlance in Ireland over the decades to refer to policymakers and
politicians. This term is not seen as denying in any way the constitutional role of
ministers (and the Dil) in the policy process, but it recognises that ministers bring
forward policies that derive strongly from options put forward by the policymakers.
15
As revenues were pouring into the exchequer, research or analyses that were critical
of policy were not welcome, and criticisms of some individual policies on the basis of
poor value for money (e.g. the Western Rail Corridor) were seen as petty. The
availability of revenues worked against cost being taken systematically into the decision-
making, as it would at a time of scarce resources.
16
Various presentations (including those from the Department of Finance) identified
the need for a more evidence-based evaluative approach but they were ignored.
See http://www.dcu.ie/education_studies/ien/iendata/David%20Doyle%20Keynote%
20Speaker.ppt; http://www.dcu.ie/education_studies/ien/iendata/FRuane%20 Evaluati
on%20Conference%20Presentation.ppt.
evidenced-based prioritisation in the decision-making process. For
example, there is no evidence of research or analysis to inform the very
significant decision made in December 2003 to decentralise large parts
of the civil/public service.
17
Indeed, it appears that this decision also
lacked any prior consultation with the senior public servants who
would have been able to provide advice if asked.
Todays tough decisions need to be grounded in evidence if they are
to command respect, acceptance and support. The research
community can contribute to this process so that evidence-informed
policymaking can become a reality. Recent policy initiatives in
government departments, led by the Department of Public
Expenditure and Reform (PER), have signalled support for this
change.
18
These are referenced further below.
The remainder of the paper addresses four questions:
i. What can research offer to the policymaking process?
ii. Why is greater use not made of research in the policymaking
process?
iii. Why is more policy-focused research not available?
iv. What is needed for a more productive engagement between
researchers and policymakers?
What can research offer to the policymaking process?
Policymaking is complex, reflecting historic patterns, institutional
structures, operational legacies, cultural influences and international
constraints. The policymakers role is to bring forward options for
political decision-making. The supporting analysis may be prepared by
analysts within the system or occasionally by consultants.
122 FRANCES RUANE
17
This contrasted with the UK decision in 2004 to decentralise the civil service further
out of London. This was based on a major government report, which focused on
a careful cost analysis and on plans to build regional labour markets that could
support skilled labour markets for public sector jobs. See http://www.hm-
treasury.gov.uk/consult_lyons_index.htm.
18
For example, in the Comprehensive Expenditure Report of December 2011, PER set
out a range of reforms, including a unified and updated Value-for-Money (VFM) Code
(to provide for appraisal (ex ante) and evaluation (ex post) of current as well as capital
expenditure), the creation of a Public Service Evaluation Network to provide a forum
for advancing the evidence-based policy agenda and a reformed medium-term budget-
formation process. Some of the thinking that informed this process is set out in Hearne
& Watt (2011).
Researchers can contribute most to the policy-formation process
where complex issues require comprehensive frameworks/models to
be developed or new empirical evidence to be analysed. Both
researchers and analysts need to understand from the outset that any
given piece of research is unlikely to provide a silver bullet solution,
and that the key role of the researcher is to draw out the implications
of the theoretical or empirical analysis for policy design or evaluation.
How can research help with policy design?
Economic theory can play an important role in clarifying objectives,
assumptions, rationale and mechanisms. For example, it can help
identify how policy can generate perverse incentives, resulting in
behaviour that is opposite to stated policy objectives. There have been
various cases where this has been seen to arise: university lecturers do
not focus on teaching if their promotion prospects depend solely on
research publications; patients do not use GP services if outpatient
costs are free and GP visits cost 50; government agencies do not use
funds efficiently if they know that their budgets will be cut if they have
unspent money at year end.
19
For new initiatives or major changes in the direction of policy,
20
researchers can provide additional independent, critical reviews of
empirical evidence both national and international.
21
This may
simplify the policymakers decision or may complicate it by drawing
out elements that could otherwise have been ignored.
22
To take an
international example, research in macroeconomics and finance in
recent years shows that researcher expertise has made a significant
contribution but has not necessarily helped to make the policymakers,
or indeed the politicians, immediate job any easier.
Researchers can generate evidence from existing or new data. For
example, in designing tax changes, existing data can be used to
measure likely responses. In some cases, policy experiments can be
undertaken to explore the impact of specific interventions, e.g.
19
This latter problem is now in the process of being dealt with as part of the recent
reforms, i.e. the introduction of multi-year expenditure ceilings, with allowance for
carry-over of unspent sums into the following year.
20
Where the literature is well developed and there is considerable research evidence
available, the researcher may provide little additional value and, for the most part, it will
be appropriate to undertake much of this analysis within PER as it builds up its skill sets.
21
See, for example, the two research reports prepared for the Expert Group on
Resource Allocation and Financing in the Health Sector (Brick et al., 2010).
22
In so doing, it may reduce future costs and difficulties.
through randomised control trials. Analysis of longitudinal data
provides a better basis for judging policy impact. The current
investment in longitudinal studies of children and the elderly are
examples of good national research planning that will provide much
better evidence in the future to inform policy. It will also provide data
for high-quality academic research.
The time horizon for good policy research can be very long. Today
we are reaping the benefits of decisions made more than a decade ago
to invest in our statistical infrastructure. These included new surveys
by the Central Statistics Office (CSO), such as the Quarterly National
Household Survey and the Survey of Irish Living Conditions,
23
and the
engagement of each government department with the CSO in relation
to obtaining statistical data from administrative records. The recent
government decision to extend departmental strategies for data to
include research is a very welcome development, and should help to
make sure that maximum return is generated by expenditure on policy
research.
24
How can research help to measure the effectiveness of policy?
Ensuring that decisions today are effective and efficient is crucial. This
means undertaking ex ante social cost-benefit analysis of capital
projects and regularly evaluating current programmes ex post to see if
they are meeting their stated objectives. In both cases, theoretical
insights help link policy objectives to policy actions and help define
coherent frameworks to measure their effectiveness. The use of
programme logic models is helpful in this regard and these are now
becoming part of the Irish policy-evaluation system.
25
However, some
areas require much more sophisticated analysis, with precise
measurement of the counterfactual in order to explore causal
relationships properly and ensure that the use of evidence is
124 FRANCES RUANE
23
This important study had its origins in the Living in Ireland Survey, which was
undertaken by the ESRI and funded in large measure by the European Commission.
24
The report by the Research Prioritisation Steering Group (2012) identified research
for policy as a specific area of research supported by government, different from basic
and applied research. It noted that much commissioned research was ad hoc rather
than part of an overall programme, and that peer review could assist in improving its
quality (pp. 3940). An example of a departmental data and research strategy can be
found at http://www.dcya.gov.ie/viewdoc.asp?fn=/documents/Publications/ndrs.htm.
25
The VFM Code (http://vfm.per.gov.ie) sets out this model, which connects inputs,
activities, outputs and results, and defines their linkage to strategic objectives.
unbiased.
26
Research for policy analysis must be of the highest quality
as the consequences of poor-quality research could contribute to
serious policy errors.
Many evaluation exercises are straightforward and are best done by
policy analysts within the civil/public service, unless the specific
context requires the use of independent analysts.
27
It is good practice
that such evaluations are undertaken outside the relevant line
department or, if undertaken within the department, are reviewed by
the relevant central government department, i.e. PER. This requires
building up expertise within PER and line departments and agencies.
Best practice is that these evaluations are published, and a similar
approach is needed for regulatory impact assessments.
28
External researchers, if they are involved, can best assist with the
methodology
29
and with evaluating the rigour of the process/analysis
followed, rather than with the detail.
30
In terms of evaluation, they can
help identify the additional data required; in terms of design, they can
help develop programme indicators, mindful of the dangers inherent
in the naive use of key performance indicators.
31
It is helpful to
recognise openly that those in the front line may be too close to the
coalface to be able to set objective indicators to measure policy
effectiveness. The recent reformatting of the Book of estimates to
relate expenditure subheads directly to strategic programmes is a
major step in increasing transparency in the public expenditure area,
and it could be a useful area for engagement between researchers and
policymakers. Drawing on theoretical insights and independent
analysis of evidence can help policymakers avoid becoming prisoners
26
For example, measuring the impact of policies to promote innovation on labour
productivity requires frameworks that take account of the effects of other factors, e.g.
internationalisation.
27
Sometimes external analysts are required to undertake the evaluations to guarantee
their independence, and where these are very complex, researcher inputs are required,
e.g. EU Structural Fund Programmes.
28
In the UK, for example, if impact assessments are to lead to primary legislation, they
are signed off by the relevant minister.
29
The ESRIs Renewal Series was developed to focus on international research that
could inform Irish policymaking. See http://www.esri.ie/publications/latest_publications/
economic_renewal_papers/.
30
Where the institutional or legal setting is complex, researchers from several different
disciplines (law, accounting, etc.) may be needed to ensure that the approach is holistic.
31
This issue was discussed in some detail by Canice Prendergast in the 2010 Geary
Lecture, published subsequently in The Economic and Social Review (Prendergast,
2011).
of their own thought processes and experiences, a danger which the
past decade illustrates only too well in many OECD countries.
Why is greater use not made of research in the policymaking
process?
To understand how greater use might be made of research in the
future, it is helpful to ask why, given the potential contribution from
research, so little use has been made of it in the past, and specifically
in the decade 19982008. Policy in this period was dominated by the
programme for government agreed by coalition partners and by the
social partnership framework. Major policy issues were decided in
settings where there was great pressure to obtain consensus and huge
media attention. The focus was on the decision-making process (the
winners and losers) at the expense of the content of the agreement. A
decision from such a process could come to be implemented even if
there had been no prior analysis or rigorous costing undertaken.
32
This
meant that some policy decisions did not involve the policymakers at
all, and their role in the process was simply to implement and evaluate.
What can be learnt from this? Unless analysis and research are publicly
available to inform the programme for government process, their
ability to inform policy is quite limited.
33
This means that unless
political parties indicate the policies they are considering well in
advance, the required analysis and research are unlikely to be
available.
34
In contrast, the social partners had an opportunity to draw on
research evidence through the underpinning work of the National
Economic and Social Council (NESC). However, over the course of
the 1990s, the role of the NESC increasingly became one of supporting
the partnership process by producing consensus documents, rather
than one of publishing objective policy analyses (a role partially
fulfilled by the National and Economic Social Forum). These
126 FRANCES RUANE
32
This issue was addressed in the Wright report of December 2010 (Independent
Review Panel, 2010). It described the budgetary process as being overwhelmed by the
combination of the programme for government and social partnerships systems.
33
Even if available, the analysis/research may be legitimately ignored by a
democratically elected government.
34
It has been suggested that ahead of New Labour taking power in the UK in 1997, the
Treasury had analysed and evaluated all of the policies it had been promoting in the
previous two years. This analysis provided evidence for Labour of what might or might
not work. Of course, there were no coalition complications in that setting.
consensus documents contributed significantly to industrial peace and
partnership harmony. However, the by-product of this was that the
policy arena had fewer analytical documents that drew on national and
international research evidence to set out the hard policy challenges,
choices and their consequences.
Another institutional factor impacting on the use of research was
the balance in analytical expertise between the civil service and the
wider public service. Whereas in most countries policy details are
designed by specialists within government departments, in Ireland the
specialist knowledge became fragmented across a growing number of
specialist agencies, and in some cases within private sector
consultancies. As a consequence, many government departments had
little specialist knowledge or resources to analyse and develop policy.
This further reduced the likelihood of research being used, with the
research agenda falling between agencies and departments. This often
led to potential for duplications and omissions, a problem that will be
reduced in the future if departments implement coherently their data
and research strategies.
The application of benchmarking is a positive feature of Irish
policymaking, when done objectively and rigorously.
35
It can be used
to challenge thinking and inspire action if the benchmark is the best
or the most relevant. However, even a good benchmarking exercise
must be viewed as the starting point rather than the end point for
comparative analysis, and policymakers must realise that comparative
results can be misleading, unless they are set in a properly defined
framework. With the exception of the original benchmarking report on
public sector wages and salaries in the early 2000s,
36
Ireland generally
has conducted this type of benchmarking very openly and trans -
parently. In recent times, developments in the websites of some
government departments, agencies and public bodies in Ireland have
increased transparency rapidly. These developments include reports,
statistical data, organisation charts and, in the case of one department
at least (PER), the minutes of its management meetings (with a two-
month lag).
A less-positive feature is the extensive use of expert groups
operating over relatively short time periods, often required to develop
major strategy positions. While other countries at our stage of
35
The Competitiveness Council is an example of a body that has used international
benchmarking to challenge actions and policies.
36
Benchmarking can be seen as the low waterline in Irish policymaking, with a
deliberate decision made not to place the evidence in the public domain.
economic development use such groups from time to time, Ireland
seems to use them more frequently. What exactly is their role? Are
they intended to generate new ideas, to challenge conventional
thinking, to build more policy coherence or to develop a wider
consensus? How independent are the experts and how do they make
use of existing research evidence? What is the quality standard for
these reports? Might it not be more useful and more transparent to
develop expertise within the relevant departments or agencies and
then issue published reports for wider consultation? What should the
official response process be to such expert reports? The extent of the
use of expert groups suggests that a systematic review of their impact
and effectiveness would be very informative.
37
Ireland has often called on the OECD for assistance. When and
why do we do this? Is it the lack of local expertise or desire to get the
OECD to deliver tough messages? The status of an OECD report
carries an exceptional amount of authority along with a presumption
that its recommendations should be accepted without further debate
or question.
38
The quality of reports depends on the skills of the team,
the validity of the international comparators, the terms of reference
set by the Irish policymakers and the quality of Irish research that can
feed into their work. Even if the report is excellent in its own terms,
there is still a need for a real debate on its conclusions and
recommendations before they are accepted or ignored. It would be
healthier if the role of the OECD were to become one of challenging
us and helping us to identify possible policy solutions rather than
providing us with the solutions.
Today, the troika is an additional source of external expertise. The
relationship between Irish policymakers and the troika is clearly
different to that between Ireland and bodies such as the OECD
because of the troikas specific oversight role. Nevertheless, there is a
possible gain from that engagement, notwithstanding the desire for
the policymakers to bring it to an end at the earliest possible date.
128 FRANCES RUANE
37
Their costs often greatly exceed what is measured by the departments that set them
up; in particular, no account is taken of the opportunity cost of the participants.
Furthermore, their time frame often means that only very narrow research questions can
be addressed and that commercial consultancies are the only bodies in a position to
undertake the work. This increases the costs of the expert group and reduces the
likelihood of knowledge transfers or of the analysis generated having future value.
38
While the calibre of OECD reports may be much higher than other local consultancy
reports, the latter do not enjoy the same status. However, unlike most other consultancy
reports, those by the OECD are always published.
There is a striking similarity between the content of the reform agenda
set out in the troika documents and what is contained in the National
recovery plan 20112014, which was published before the arrival of the
troika. Consequently, the troikas endorsement of that agenda, based
on analysis and international evidence, is effectively assisting the
government in tackling a range of powerful vested interests.
In considering the use of evidence, one should note the difference
between a process that uses independent, comprehensive and rigorous
analysis of the evidence to inform policy and a process based on the
selected use of external reference points (either research or policy)
without any rationale. Adopting a policy because it seems to work well
in country X is not prudent, especially when the policy setting is
different. The effectiveness of such a policy should have been fully
evaluated before being considered for adoption in Ireland.
39
A final explanation of Irelands low use of research in policymaking
more generally may be the poor respect for, and knowledge of,
statistics. This is, in part at least, due to how the education curriculum
at second level is delivered, with few candidates covering the statistical
sections of their mathematics courses.
40
This inhibits people engaging
with research results in the policy arena. Ireland is a nation of
storytellers and we all like good anecdotes; statistics leave people cold.
The research community could help overcome this by producing
statistically valid examples to illustrate findings. Otherwise, an
unpopular finding coming from high-quality research can be immed -
iately trumped by an unrepresentative anecdote, as most people are
not in a position to judge the difference between the representative
and unrepresentative anecdote.
41
Research findings could be more
informative if researchers, having undertaken robust analysis, improve
how they present research to the wider public.
42
While it is challenging
to simplify without being misleading, it would also assist journalists in
providing good material for the general public.
39
For example, there was much discussion about introducing the Dutch health
insurance system in Ireland while evidence was emerging from the Netherlands that it
was not yielding the expected results.
40
Recent changes in the mathematics curriculum under Project Maths should make a
contribution over coming decades but, on its own, the project will take a long time to
improve overall statistical literacy.
41
The comment I dont deal with statistics, I deal with real people is often uttered with
pride in reality this means I dont deal with real people, I deal with a few people I
know.
42
The mixed methods used in sociology play this role very effectively they combine
statistical analysis with qualitative analysis that provides case-study type examples.
Why is more policy-focused research not available?
In recent years academic institutions have increasingly emphasised the
importance of peer-reviewed, international journal articles in
promotion processes. While this emphasis has merit, it leaves little
incentive for academics to undertake policy research. This is not just
an Irish problem. In the UK the immediate impact of the research-
assessment exercise was a dramatic reduction in the volume of policy-
relevant research in the 1980s.
43
In response, the UK Government
established a system of funding research programmes that are relevant
to policy, mostly in cooperation with the Economic and Social
Research Council.
44
While this has yielded a positive response, it is
noteworthy that much of the policy research is being undertaken by
the older generation of researchers, who are less concerned with
promotion. These older academics have also built up the institutional
knowledge necessary to undertake good policy research.
45
Up to now, there has been no formal system for funding policy
research across the full policy spectrum in Ireland. However, there has
been public funding for programme-based social science research and
for programmes in specific policy areas. Examples of the former
include the creation of institutes in the universities, e.g. the Geary
Institute, University College Dublin (UCD); the Institute for
International Integration Studies, Trinity College Dublin (TCD); and
the National Institute for Regional and Spatial Analysis, NUI
Maynooth. Examples of the latter include funding for policy research
in Teagasc, the ESRI and the IPA (from the Committee for Public
Management Research).
46
The approach varies widely across
departments and agencies, as does the balance between project and
programme funding, and between research and consultancy.
47
Philanthropic funding for policy research, very common in the US, has
been modest in Ireland, with some striking exceptions, most notably
The Atlantic Philanthropies. Its support for research on children and
130 FRANCES RUANE
43
This exercise sought to measure research outputs of universities, which then became
the basis for funding higher-education research.
44
This involved setting up research centres and clusters with a specific policy focus.
45
One countervailing factor is the growth in the number of new policy journals. This has
increased the willingness of university academics to undertake policy research. In
Ireland The Economic and Social Review, the main local academic journal for economics
and empirical sociology articles, introduced a policy section in 2009.
46
See http://www.agresearch.teagasc.ie/rerc/research.asp, http://www.esri.ie/research/
research_areas/ and www.cpmr.gov.ie.
47
The ESRI is among a number of bodies that directly receive funds to support research
for policy, and its research focus reflects where such funds are available.
on ageing has demonstrated the research communitys willingness and
ability to respond when funding is available. In certain policy areas,
most notably health and children, charities have also funded research.
In contrast to some of the larger OECD countries, there is negligible
public good research funded by the private sector in Ireland.
48
In
terms of policy-relevant research, Ireland has benefited from EU
fund ing, which has mostly supported cross-country comparative
research.
Yet another reason for the low level of independent policy research
during the boom years was the widespread use of consultancy reports
to support policymaking, often where terms of reference ensured that
the outcome was virtually predetermined. The absence of budget
constraints and the increased emphasis on individual accountability
reinforced that pattern, as the reports could provide apparently
indepen dent support if required and otherwise were typically not
published.
One very positive development in the past decade is the availability
of high-quality Irish data, especially from the CSO. With
(anonymised) micro data, research that was previously not possible
can now be conducted, and this research has international journal
potential. Given the states large investment in data collection, it is
important that these data are being used in research.
49
Their use has
been slower than might have been expected, given the growing
numbers of academics and PhD students over the past decade. This
reflects the considerable investment of time required to use any given
data set and often the need for teams with mixed skill sets to realise
the potential benefits. The availability of these data enhances the
possible synergies between academic and policy research. While
researchers typically focus on the quality required for journal articles,
policy research needs to be just as robust as academic research and
requires a much greater investment in institutional knowledge. Some
policy research in Ireland is subject to peer review, with departments
and agencies sending out draft reports to independent researchers for
comment.
50
48
The ESRI has received some funding of this type for research but the general view of
the private sector is that it is the governments responsibility to fund policy research.
49
Many of these anonymised data sets are available in the data archive at UCD.
50
For example, the ESRIs review process involves seeking comments from internal and
external peers, as well as seeking comments from policy specialists in relevant agencies.
Frank Barry (TCD) recently suggested that one method of improving the quality of all
commissioned research by government (including that prepared by consultants) would
be to involve a seminar to peers as part of its process. In a similar vein, PER (contd.)
One of the possible silver linings in todays dark clouds is the
growing numbers of young academic researchers prepared to engage
in policy debates. This followed on from a decade of almost no
engagement with a younger generation of researchers. One can be
hopeful that this will translate over time into increased research across
the full range of policy issues.
What is needed for a more productive engagement between
researchers and policymakers?
The relationship between the research and policy communities
depends on the networks linking the individual researcher(s) and
policymaker(s). When the two communities in Ireland were very much
smaller, such networks emerged naturally; this does not happen
today.
51
Until 2011, when the incoming government committed to
building up specialist skills in the civil service, the policy of creating
strong generalists meant that there was little reason for engagement.
Relationships that had existed were continuously undermined by the
mobility of policymakers across areas, usually being replaced by
people with no specialist knowledge of the area and with a different set
of analytical skills. The increased specialisation in research in most
disciplines over the past two decades has compounded the problem.
Some ways of increasing productive engagement between
researchers and policymakers are addressed below for each group
separately and both together.
Researchers
Researchers can increase the accessibility of policy-relevant research
to policymakers by providing non-technical summaries/abstracts of
research findings, e.g. via research bulletins, policy briefs, web pages
and blog postings. This could cover research relevant to Ireland that
has been published internationally and is not readily accessible to
policymakers. Working papers could also have such non-technical
summaries, as could PhD theses undertaken at Irish universities.
52
It is
132 FRANCES RUANE
50
(contd.) has indicated its intention to publish economic evaluations in the future; this
is a natural extension of recent publications of reviews on its website.
51
Academic programmes, such as the MSc in Economic Policy Studies at TCD and the
various programmes offered by the IPA (more recently in conjunction with UCD), have
provided the main links into the policy community over the past decade.
52
Such summaries could be encouraged by research institutions and made mandatory
for research that is funded from the public purse (e.g. Irish Research Council for the
Humanities and Social Sciences).
arguably a good discipline for researchers to summarise findings in an
accessible form and, where appropriate, draw out their policy
relevance. The benefits to the policy community and to students could
be considerable. As noted above, economists engaged directly in
policy research could identify examples to illustrate the generality of
their findings.
Possibly the greatest challenge for researchers wishing to engage in
policy research is the scale and nature of the issues, which require
multi-person and increasingly multidisciplinary teams. While the
social sciences are moving towards more team-based research, the
tradition of the single scholar is still strong, as is the tradition of the
single discipline. Moving to multidisciplinary teams takes time and
effort, especially if programmes in the higher education system are
highly specialised.
53
What can research institutions do to support better engagement?
They can demonstrate that they value policy research by including
policy engagement in the criteria for promotion (under pro bono
headings). They can signal support through a positive attitude towards
postgraduate research and teaching programmes in policy areas.
Finally, they can ensure that their accounting systems and intellectual
property contracts do not hinder the development of policy research.
Policymakers
While the individual, rather than the institution, is the driver on the
research side, the opposite is the case on the policymaking side. The
individual policymaker can do very little to promote greater
engagement with the research community without institutional
endorsement. Consequently, the culture towards research within a
department or agency has a major influence on the behaviour of its
policymakers. So, what might help to stimulate greater engagement
with the research community?
It is something of a paradox that, in a country where public
discourse for two decades has emphasised the knowledge economy and
knowledge society, the civil service has, until very recently, pursued a
human resource strategy that effectively penalises people with
specialist knowledge by either narrowing their career prospects or
forcing them to deskill. What is needed is a combination of generalists
and specialists. This means developing and retaining the specialist
skills of people entering the civil service.
53
High levels of specialisation at undergraduate level result in little understanding of
the approaches and contributions of different disciplines.
The recent announcement by the government of its intention to
establish the Irish Government Economic and Evaluation Service
(IGEES) has been widely welcomed. The model is influenced by the
approach adopted in the UK. There, in the late 1990s, a Government
Economic Service was re-established to create government
economists, who combine high levels of economic skills with
institutional knowledge.
54
Central to the UK model (which operates in
Northern Ireland also) are the clusters of economists in different areas
who can share knowledge and engage in peer learning. The UK model
was designed to link the economists back to the Economic Service
even when employed in line departments. By having a system-wide
service, which includes agencies in the wider public sector, skilled
resources can be moved readily across departments and agencies.
These policymakers can directly access externally produced research,
whether national or international, and are able to commission
research very effectively.
55
They have also put much of their policy
documents into the public domain via their websites, a policy now
being followed in many Irish government departments.
56
In Ireland
the success of the IGEES will be crucial for policymaking.
Furthermore, its members will become the key links between the
policymaker and the research community.
57
A further attraction to researchers of undertaking policy research
can be the access to data sets that are within government departments
and agencies.
58
There are numerous examples where these data,
appropriately anonymised, have been used by researchers; for
example, the Forfs data on enterprises in the manufacturing and
service sectors. The new eGovernment strategy is increasing the pace of
development of departmental interconnections and, together with the
134 FRANCES RUANE
54
The UK created a Government Economic Service in the 1960s with specialist
economists, but these gradually disappeared during the budgetary cutbacks in the 1980s.
55
Policymakers need to develop commissioning research skills if they are to use their
budgets efficiently.
56
See, for example, reports on the PER website at http://per.gov.ie/reports/ and
http://www.hm-reasury.gov.uk/data_magentabook_index.htm for the UK Treasury
evaluation document.
57
For those with the requisite skills and interests, there may be occasional opportunities
for research analysts to engage in projects with researchers, as happens in the US and
Canada, for example.
58
Where research commissioned by government departments or agencies leads to peer-
reviewed academic publications, the value of the research itself is enhanced through
wider dissemination. Facilitating such publications also increases the attractiveness of
policy research to potential researchers.
planned data and research strategies, these should increase the
benefits of research, especially on cross-cutting issues, and should give
greater assurance that funds for policy research are productively and
coherently spent.
Policymakers and researchers
What extra can be achieved by both groups fostering the engagement?
Exchanges of personnel between the two sectors should be promoted
and welcomed, but these will take some time to develop. Meanwhile,
working closer together would help establish a clearer understanding
of roles, interests and the requirements of researchers and
policymakers.
59
This would allow more productive interactions,
thereby enhancing benefits and reducing potential tensions. It would
help in turn to foster commitment on both sides to building a
meaningful research agenda, which would see researchers producing
policy outputs (published reports) and academic outputs (peer-
reviewed papers).
60
Interactions with policymakers could stimulate
research into new issues and, when linked to pilot studies involving
multidisciplinary teams, could generate significant new research
possibilities.
By combining efforts, researchers and decision-makers would
inform, support and promote societal well-being in the face of
powerful, vested interest groups. The potential beneficiaries of many
policy improvements may be widely dispersed and without the
resources to promote such policies, e.g. a patient-centred health
system. In contrast, some policy changes, such as the abolition of the
bank levy in Budget 2007, benefit a small number of institutions at the
expense of the general taxpayer.
61
Ireland is good at bringing
stakeholders to the table. But this process needs to involve challenging
them with robust evidence and analysis so that they engage with the
totality of issues rather than simply their own agendas. If the
stakeholder process does not involve challenge, it may simply lead to
59
This involves understanding incentives and gaining clarity of and respect for roles and
governance by both sides.
60
If the researchers have no real commitment to policy research, the benefits will be
small in policy terms and the major benefit may be in terms of contributing to
institutional (university) overheads. Similarly, there is no point in policymakers
appearing to embrace research while believing that they know the answer already and
really have nothing to learn.
61
The banks were in a position to lobby the taxpayers were not.
no real engagement and possibly to solutions that are beneficial to
those at the table at a cost to society at large.
62
The engagement is not without possible risks and the most
obvious is that of promoting group think, especially if people were
only comfortable with consensus. One way of helping to avoid this is
by having open dialogue events, under the Chatham House Rule, and
greater acceptance of the value of critical thinking on both sides.
63
If
the debate is one-sided either with the researcher pronouncing and
the policymaker silent, or with the policymaker picking holes and the
researcher disengaging there will be no benefits from the
engagement. These Chatham House events will not happen unless
they are organised by some body or group and supported by
researchers and the leadership in the policy community.
The other large risk is that the researchers lose, or are perceived to
lose, their independence, by engaging with the policymakers. To avoid
this, an open and transparent process is needed. As part of this
process, policymakers would provide inputs to the research that seek
to inform and improve the analysis rather than influence the
conclusions. Similarly, researchers must engage with these
policymaker inputs and with standard peer-review processes. Finally,
the researchers must retain their right to publish the policy research
and commit to it being placed in the public domain so society at large
can examine the findings.
64
Concluding comments
This paper has explored some ways in which we might foster greater
engagement between researchers and policymakers in Ireland today.
This engagement would increase the return on the national investment
in research that is relevant to policy.
65
For the engagement to
136 FRANCES RUANE
62
The discussions around the renegotiation of the medical consultants contract in
200608 would appear to be one such example.
63
Such events were relatively limited in the past decade, and meetings of the Statistical
and Social Inquiry Society, which have provided a forum for policy debates for over 150
years, have been poorly attended by policymakers in recent years.
64
In very exceptional cases, this may not be practicable for confidentiality reasons, but
it should be the default, with the arguments for the exception clearly established. The
requirement to publish is a standard condition in ESRI contracts for all its
commissioned research.
65
As well as engaging with current policies, independent researchers are well placed to
contribute to identifying future problems and complexities, because of their longer time
horizons.
contribute to making the policy process more research- and evidence-
based, strong support is needed from across the full political system.
In a context where persistent inefficiencies in policy design and
delivery are now less acceptable, and definitely unaffordable, evidence
from research can help to
ensure that people understand why tough decisions are
necessary;
demonstrate that tough choices can be fair; and
show that lessons have been learnt from the past.
Recent developments point to possibilities of better engagement
between researchers and policymakers in the future. However, this will
not develop automatically unless it is welcomed and endorsed by the
political system and by public-sector leadership.
66
It also requires that
researchers who are interested in policy invest in understanding the
many recent changes in institutional systems and structures. The
IGEES will take time to build up its capacity and capability, and
interim measures will be required to speed up the engagement
process.
Despite the many weaknesses of policymaking over the boom
years, there were some good investments made that are relevant to
policymaking.
67
These included investments in the data infrastructure
(a sine qua non for empirical analysis) and investments in human
capital (increase in the number of MSc and PhD students graduating
from universities and in the numbers of civil/public servants pursuing
higher degrees). Furthermore, most departments and agencies have
had experience of investing in research to inform policymaking, albeit
on a somewhat ad hoc basis. What is required now is that the approach
to research becomes more systematic, more questioning, and with a
greater focus on the important policy questions facing Ireland today.
References
Brick, A., Nolan, A., OReilly, J., & Smith, S. (2010). Resource allocation,
financing and sustainability in health care: Evidence for the Expert Group on
Resource Allocation and Financing in the Health Sector (Vols III). Dublin:
Department of Health and Children & Economic and Social Research
Institute.
66
This means support by public representatives generally as well as by ministers.
67
There were good investments in many other areas also, e.g. parts of the road network.
Hardiman, N., MacCarthaigh, M., & Scott, C. (2012). The Irish state
administration database. Retrieved from http://www.isad.ie [5 July 2012].
Hearne, E., & Watt, R. (2011). Research and policy-making Strengthening
the link. Journal of the Statistical and Social Inquiry Society of Ireland, XL,
15266.
Independent Review Panel. (2010). Strengthening the capacity of the
Department of Finance: Report of the Independent Review Panel. Dublin:
Department of Finance.
National Economic and Social Forum. (2007). Evidence-based policy making:
Getting the evidence, using the evidence and evaluating the outcomes. Dublin:
National Economic and Social Forum.
Prendergast, C. (2011). What have we learnt about pay for performance?
The Economic and Social Review, 42 (2), 11334.
Research Prioritisation Steering Group. (2012). Report of the Research
Prioritisation Steering Group. Retrieved from http://www.forfas.ie/media/
ffs20120301-Research_Prioritisation _Exercise_Report.pdf [5 July 2012].
Ruane, F. (2010). Research and policy making (Working Paper No. 354).
Retrieved from http://www.esri.ie/UserFiles/publications/WP354/
WP354.pdf [5 July 2012].
Ruane, F., & Whelan, B. (2011). Building research capacity in the social
sciences Alternative approaches. Journal of the Statistical and Social
Inquiry Society of Ireland, XL, 13351.
Young, K., Ashby, D., Boaz, A., & Grayson, L. (2002). Social science and the
evidence-based policy movement. Social Policy and Society, 1 (3), 21524.
138 FRANCES RUANE

Research Evidence and Policy Making

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Research Evidence and Policy Making

Uploaded by

Copyright:

Available Formats

Administration, vol. 60, no. 2 (2012), pp.

You might also like