Research note by Pieter Zwaan, Stijn van Voorst and Ellen Mastenbroek based on the paper:
- Pieter Zwaan, Stijn van Voorst and Ellen Mastenbroek. Ex post legislative evaluation in the European Union. Questioning the use of evaluation as instruments for accountability, International Review of Administrative Sciences, 25 February 2016, doi: 10.1177/0020852315598389
In response to concerns about the democratic deficit of the EU, the European Parliament (EP) has increasingly gained powers to hold the European Commission accountable. This in turn has resulted in stronger demands on the Commission to provide the EP with information about EU policies (Curtin, 2009: 256-57). One tool for this purpose are ex-post evaluations which assess the fulfillment of policy goals and the behavior of the implementing actors involved (Mastenbroek et al., 2015). In our article “Ex post legislative evaluation in the European Union”, published in the International Review of Administrative Sciences, we turn to the question to what extent and under which conditions the EP uses ex post legislative (EPL) evaluations to hold the Commission accountable. We find that the EP makes little use of EPL evaluations to hold the Commission accountable. While evaluation are used, they are used mainly for agenda-setting purposes and steering the Commission. This usage is best explained by the degree of conflict between the Commission and the EP, a political factor.
In our research we tested four key hypotheses about evaluation use for accountability purposes. The first two hypotheses are rationalistic in nature. Firstly, evaluation use was expected to depend on the perceived quality of the evaluation (Cousins and Leithwood, 1986: 347). Since there have been doubts about the objectivity of the Commission’s evaluations in the EP (Poptcheva, 2013), Members of European Parliament (MEPs) were expected to be more likely to use external evaluations than internal ones. Secondly, communication quality matters: evaluation use might be affected by the clarity of the evaluation reports (ibid.: 347). Since MEPs are often overloaded with information, we expected that evaluations must either be short or have concise summaries (<10 pages) to be used.
Our other two hypotheses are more political in nature: we focused on the ‘decision setting’ and on how much importance was attached to the topic of the evaluation. In theory, this importance is affected by the risks for a principal (in this case the EP) that the agent (in this case the Commission) does not fulfill its obligations. Firstly, the significance of the principal’s risk matters: if the costs of shirking by the agent are higher, there is more need for control, which in turn increases the chance of evaluation use (Benjamin, 2008). We expected the significance of this risk for MEPs to depend on issue salience – their perceived importance of a piece of legislation – as measured by the number of recitals in the legislation being evaluated. Secondly, the accountability relationship was expected to be affected by the nature of a principal’s risk (Benjamin, 2008): if the conflicting interests between the EP and the Commission are stronger, the chance of evaluation use was expected to increase. The degree of conflict was measured by the the number of amendments proposed by the EP on the original legislative proposal.
The hypotheses were tested using a dataset of 220 EPL evaluations conducted or outsourced by the Commission (Mastenbroek et al, 2015). We measured our dependent variable ‘evaluation use for accountability purposes’ by checking whether or not each evaluation was referred to in parliamentary questions (EPQs) asked by MEPs, as EPQs are a key instrument for the EP to hold the Commission accountable (Wille, 2010: 60). For all EPL evaluations in our dataset, we searched for corresponding EPQs during three parliamentary terms (1999-2004, 2004-2009 and 2009-2014), checking each individual result to ensure that reference was indeed made to the evaluation in our dataset.
The results of our analysis show that out of 220 evaluations analyzed, 49 (22%) evaluations were referred to in EPQs in some way, out of which 34 evaluations (16%) were used to steer the behavior of the Commission. This percentage is above expectation when compared to the use of ex-ante evaluations by MEPs (Poptcheva, 2013). However, our analysis also revealed a forward-looking outlook rather than a backward-looking attitude of MEPs when it comes to using EPL evaluations: no retrospective use of evaluations was found at all.
Using binary logistic regression, our analysis showed that the variances in evaluation use can be explained best by the degree of conflict between the EP and the Commission, which is indicative for the nature of the principal’s risk. In line with our expectation, evaluations are more likely to be used in situations where there is a strong conflict, presumably because in such situations there is more need for control. The number of recitals, our indicator for the significance of the principal’s risk, turned out to have little impact.
Both the fact that our rationalistic factors had no impact on evaluation use and our outlier analysis point in the direction of political explanations for evaluation use. In our study we analyzed the impact of only a number of political factors; identifying other political variables which influence the use of EPL evaluations seems a worthy goal for future research. Given the forward-looking usage of evaluations, it is advised to turn to the literature on agenda-setting and policy change. A first avenue for further research is to investigate how the Commission responds to EPQs about evaluations: potentially, this traces evaluation impact at the political level.
- Benjamin LM (2008) Evaluator’s role in accountability relationships. Evaluation 14(3): 323-343.
- Cousins JB & Leithwood KA (1986) Current empirical research on evaluation utilization. Review of Educational Research 56: 331-365.
- Curtin DM (2009) Executive power of the European Union: Law, practices and the living constitution. Oxford: University Press.
- Mastenbroek, E., Voorst, S. van & Meuwese, A. (2015). Closing the regulatory cycle? A meta-evaluation of ex-post legislative evaluations by the European Commission. Journal of European Public Policy.
- Poptcheva, EM (2013) Library Briefing. Policy and legislative evaluation in the EU. Brussels: European Parliament.
- Wille AC (2010) Political–Bureaucratic Accountability in the EU Commission: Modernising the Executive. West European Politics 33(5): 1093–1116.