LSE - Small Logo
LSE - Small Logo

Blog Admin

January 10th, 2013

EU policy evaluation should make greater use of interpretative, qualitative research methods.

1 comment

Estimated reading time: 5 minutes

Blog Admin

January 10th, 2013

EU policy evaluation should make greater use of interpretative, qualitative research methods.

1 comment

Estimated reading time: 5 minutes

How should the EU’s policies be evaluated? Using the case of EU Cohesion Policy, Julian Hörner and Paul Stephenson outline some of the main theoretical principles that underline EU policy evaluation. They write that positivist approaches and quantitative research methods tend to be dominant, in part due to the pressures on European institutions to demonstrate the added-value of policies. Despite this, there may be benefits to adopting more interpretative and qualitative approaches, even if these are more difficult to administer.

Policy evaluation became particularly important in the 1980s for assessing EU Structural Funds, given the redistributive nature of what was then termed ‘Regional Policy’.  Most evaluation studies were carried out by external bodies and not by the Commission departments themselves, owing to a perceived need to ensure objectivity and neutrality, as well as a clear lack of human resources in the Commission Directorate-Generals (DGs). Today, Cohesion Policy evaluation is important not only for national, regional, and local authorities to improve implementation processes, but also for the Commission, in order to improve regulation.

Credit: European Parliament (Creative Commons BY NC ND)

EU Cohesion Policy may change in terms of thematic priorities and even functioning, but evaluation will continue to have serious political implications. An evaluation typically answers questions concerned with: first, relevance (do the objectives correspond to the needs and problems?); second, effectiveness (to what extent were the set objectives attained?); and third, efficiency/cost-effectiveness (were the results achieved at a reasonable cost?) The aim is to provide a reliable and objective assessment that is accessible to all stakeholders. Since 2010 the Commission also uses ‘fitness checks’ – comprehensive policy evaluations assessing whether the regulatory framework for a policy sector is fit for purpose. They are meant to identify excessive administrative burdens, overlaps, gaps, inconsistencies, and/or obsolete measures which may have built up over time, often owing to the cumulative impact of legislation.

A comparative analysis of the characteristics of evaluation may reveal similarities and differences in approach between, or even within, policy areas. A review of the evaluation literature reveals specific concerns for the evaluator – precisely those characteristics that may determine how evaluation is practised, and how practice (approach) influences findings (outcomes). First, let us consider the underlying epistemological assumptions and thus, the evaluator’s own philosophical approach to knowledge acquisition and the social scientific task of evaluation. This ultimately influences whether he/she is seeking to produce objective, generalisable results of universal value, or whether factors such as the subjective nature of human observation make such generalisations impossible. This important distinction mirrors the classic debate in the philosophy of science between logical empiricism and interpretivist theories. Indeed, different epistemological assumptions lie at the very core of the respective approaches; all other characteristics essentially stem from these fundamental perspectives.

Second, one must pay attention to the research methods used. A basic distinction can be made between quantitative methods – such as macroeconomic models, input–output analyses, statistics, and control groups – and qualitative methods, such as surveys of beneficiaries, stakeholder interviews, and case studies. Third, one needs to consider the direction of the evaluation research, which may determine whether data is gathered on an aggregate level from secondary sources, such as national public administrations and/or regional statistical agencies, or is collected directly from the stakeholders involved, in order to gauge experiences and opinions at the local level. Fourth, attention should be paid to the scope of stakeholder involvement, and whether evaluators only engage those directly participating in the programme or aspire to secure a more comprehensive, inclusive involvement of the community, however marginally affected by the programme. Choices will be made with time and financial constraints influencing the breadth or narrowness of the study.

Finally, the role of the evaluator in the evaluation process must be considered: i.e. whether he/she is a neutral, impartial observer, or considered an independent variable that influences the findings. Some take a ‘God’s Eye View’, as Putnam formulated it, while others consider the evaluator a policy stakeholder in their own right since he/she is ultimately a recipient of technical assistance budgets, and an agent of the principal who pays for the evaluation. The different theoretical approaches are summarised in Table 1 below.

Table 1: How the characteristics of practical approaches relate to theoretical perspectives

Theory
Positivism
Realism
Constructivism
 
Facts <———————————————————————> Values
Epistemological Assumptions
There is an objective reality
Programmes either work or do not work
Facts  / statistics / quantifiable data only matters
Several possible accounts, but not all equally valid
Facts and values should not be separated since both important for programme
Underlying structures governing workings of programme need to be deconstructed
No Objective ‘reality’ or ‘truth’Performance gauged by views, opinions and experiences of stakeholders involved
Methods / Data Collection
Quantitative methods favoured, especially econometric models
Quantitative or qualitative (especially in-depth cases studies), depends on programme requirements and specific context
Case Studies use the ‘hermeneutic-dialectical’ method, discussion with stakeholders in groups
Top-down/ bottom-up Approach
Top-down (macro-level, focus on generalisation and leave particularities largely aside)
Top-down/Bottom-up (macro-/micro level)
Bottom-up (micro-level, take into account local / regional particularities, limited possibility to generalize)
Stakeholder Involvement
Only passive as an economic actor Object of analysis
Evaluator and stakeholders form a teacher-learner relationshipSuggestionsof the stakeholders form an essential part of evaluation report
Very high, even involvingstakeholders reluctant to participate / unaware of their involvement
Role of the Evaluator
Does not get involved
Takes a ‘God’s eye perspective
The evaluator makes choices regarding categories and methods
Deconstructs underlying mechanism but not actively involved
Evaluator strongly involved in project, makes his own constructions explicit

There are a number of reasons why positivism is apparently dominant in the evaluation of EU policies. First, positivism fits the epistemological assumptions widespread in the Commission concerning what represents proper knowledge and sound data. Given the huge value of the Structural Funds it is essential to be able to demonstrate that programmes achieved (or not) what they set out to achieve. Quantitative data is suitable and convincing, allowing for easy aggregations, comparisons, and generalisations. Second, the Commission, like the EP and the Court of Auditors, favours quantitative data because of its usefulness in communicating the added-value of EU policies. Figures can be used to compare stakeholder involvement (i.e. how much money ‘flowed back’ to them as project partners), whereas words are opaque, can be interpreted differently, and create ‘shades of grey’.

Third, methods advocated by realist and constructivist theory, such as surveys, interviews, or meetings with stakeholders in small groups, are more expensive to carry out than positivist methods such as economic modelling – basically a desk-based activity seeking out pre-existing data. Fourth, it is arguably easier to carry out evaluation in a top-down/centralised manner. Some regional officials believe that the Commission sees evaluation as a ‘supranational instrument for controlling policy implementation’ and exerting pressure on regional public administrations. Numbers and percentages can be compared from one programming period to the next, and across geographical areas. Finally, the dominance of positivism may be intrinsically linked to how the evaluation community sees its own role. Many influential scholars in the field adhere to a ‘rationalist’ or positivist idea of evaluation. As a consequence, most leading evaluation firms, and companies offering evaluation services such as accountancies and consultancies, generally favour quantitative/ econometric approaches.

The negative consequence of an automatic and self-reinforcing focus on positivist methods can lead not only to unsatisfactory evaluation outcomes, but even to the failure of the programme itself. A stronger emphasis should arguably be placed on the various post-positivist approaches to policy evaluation, such as realism and constructivism, in order to make evaluation more effective and legitimate.

This article is based on a longer paper first published in Public Administration.

Please read our comments policy before commenting.

Note:  This article gives the views of the author, and not the position of EUROPP – European Politics and Policy, nor of the London School of Economics.

Shortened URL for this post: http://bit.ly/ZrUvHi

 _________________________________

About the author

Julian Hörner – LSE European Institute
Julian Hörner is a PhD candidate at the LSE European Institute.

_

_

Paul Stephenson – Maastricht University
Paul J. Stephenson is Assistant Professor in the Department of Politics at the Faculty of Arts and Social Sciences, at Maastricht University. His most recent book is the co-authored textbook ‘Analyzing the European Union Policy Process’ (Palgrave, 2011). His research interests include theories of integration, the policy-making process (particularly implementation and evaluation), institutions (European Commission, the European Court of Auditors and executive EU agencies), European policies (transport/space, regions/cohesion), as well as contemporary issues relating to the society and politics of France and Spain.

About the author

Blog Admin

Posted In: EU institutions, government and politics and enlargement | Julian Hörner | Paul Stephenson

1 Comments