Repost from EU Studies Association. Ex-post evaluation by the European Commission

by Stijn van Voorst. Original source: European Union Studies Association


The European Commission is often said to be a frontrunner in the field of ex-post evaluation (Stern, 2009, 69). Not only has it developed its own evaluation facilities, but it has also stimulated evaluation in the member states of the EU through joint evaluations of the EU’s spending programmes (Baslé, 2007, 226). While the Commission since at least the 1980s has conducted ex-post evaluations of such spending programs, evaluations of other activities are a more recent phenomenon. After being criticized for a lack of accountable government in the early 2000s, the Commission began to gradually extend its ambitions to evaluations of non-spending activities, including legislation (Fitzpatrick, 2012, 478). When the Juncker Commission entered office in 2014, its first vice president (Frans Timmermans) became responsible for the topic of ex-post evaluation. Under his guidance the Commission published new evaluation guidelines (2015), which included the ambition to conduct ex-post evaluations of entire regulatory frameworks – the so-called fitness checks. According to the new guidelines (European Commission, 2015, 259), ex-post evaluations help to improve the EU’s policies, strengthen accountable government and support strategic decision-making.

These developments show that the evaluation of a broad range of policies is becoming more and more important to the Commission, at least from a rhetorical point of view. Therefore, this is an opportune moment to reflect on the topic empirically: how does the Commission’s evaluation system function in reality?

This article (Closing the regulatory cycle? A meta evaluation of ex-post legislative evaluations by the European Commission, Journal of European Public Policy) first presents some key results of academic research about the Commission’s evaluation system.

Based on these findings, it then offers suggestions for future research, including some possibilities to provide academic studies on ex-post evaluations with a stronger theoretical background. To assess the performance of the Commission’s system for legislative evaluation, Mastenbroek et al. (2015) conducted a meta-study of 216 ex-post evaluations of regulations and directives published by the European Commission between 2000 and 2012.

They found that while the number of legislative evaluations has gradually increased over the years, the coverage of the Commission’s evaluation system is far from complete. Out of all major regulations and directives from the years 2000-2002, 33% had been evaluated by the end of 2012, meaning that almost seven of out ten major legal acts had not been evaluated ex-post.

The results of the meta-study also show that while more than half of the evaluations (52%) study the impact of EU legislation on society in terms of effectiveness, efficiency or side effects, the other evaluations are limited to process criteria such as transposition, implementation and enforcement. This second group of evaluatitions usually takes the form of a brief report to the Council and the European Parliament. The majority of evaluations concern directives (57%), with evaluations of regulations being slightly less common (43%). Besides looking at the coverage of the Commission’s ex-post legislative evaluations, Mastenbroek et al. (2015) also assessed their quality. They found that most evaluations of EU law formulate a clear problem definition (67%), contain a clear opera:onaliza:on (62%) and apply a mix of different research methods (65%) – usually content analysis combined with either interviews or surveys. About half of the evalua:ons clearly explain their case selection and have a response rate for their interviews or surveys of more than 50%, meaning that the external validity of the evaluations varies greatly. The reliability and methodological robustness of legislative evaluations do less well, as only 29% of the reports contain enough information to repeat the study if needed and only 15% of the evaluations offer a justification of their selected methods. The evaluations did beder when it comes to usability aspects: 76% of the reports contain clear recommendations, while 64% of the reports include a short executive summary. About 60% of all legislative evaluations mention some kind of stakeholder involvement, although this is mostly limited to using stakeholders as a source of information. Only 9% of all evaluations show stakeholders being more actively involved in the evaluation process. The authors conclude that the Commission has not yet lived up to its promises when it comes to providing systematic and high-quality legislative evaluations.

Continue reading the summary on European Union Studies Association

  • Ellen Mastenbroek, Stijn van Voorst & Anne Meuwese, Closing the regulatory cycle? A meta evaluation of ex-post legislative evaluations by the European Commission, Journal of European Public Policy, Published online: 16 Sep 2015 DOI:10.1080/13501763.2015.1076874