The German Institute for Development Evaluation (DEval) is mandated to conduct strategic evaluations aiming to improve the performance of German development cooperation and thereby support sustainable development in Germany’s partner countries.
Martin Bruder, head of DEval’s department for civil-society level development cooperation, and Marion Krämer, team leader of a BMZ-financed DEval project on the implementation and use of rigorous impact evaluation in German development cooperation, speak to the European Evaluation Society about the changing face of evaluation in Germany and the world.
Last year, Abhijit Banerjee, Esther Duflo and Michael Kremer received the Nobel Prize in Economics “for their experimental approach to alleviating global poverty.” For many, this marks the culmination of an “evidence revolution” within the profession of evaluation. What is the role of experimental and quasi-experimental evaluation designs at DEval?
DEval’s strategic evaluation of a comprehensive land-use planning approach in the Philippines includes a quasi-experimental design.
Martin Bruder: DEval’s mandate is to conduct strategic evaluations within German development cooperation with the aim of informing development policy and improving development practice. Our evaluations usually address complex topics such as Germany’s approach to supporting climate change adaptation in its partner countries or the way it integrates human rights and gender in its development interventions. Another focus is on the effectiveness of instruments such as particular technical cooperation approaches, budget support or structured funds as an instrument of blending financial resources from the private and the public sector. We almost always adopt a theory-based approach and try to improve on how we integrate quantitative and qualitative methods. That is to say: we start with the topic and our evaluation questions. If questions of effectiveness and the attribution of effects are key in a specific evaluation then we also use rigorous impact evaluation designs – usually of the quasi-experimental type – as part of a broader theory-based evaluation. We have done so, for example, when looking at the specific effects of participating in a development volunteer service, when analysing the effects of local planning in the Philippines and when examining the cash-for-work component of a broader initiative for refugees in Syria’s neighbouring countries. In these cases, rigorous impact evaluations have made a highly valuable contribution to the overall evaluation.
These innovations have been pioneered by organisations like the Abdul Latif Jameel Poverty Action Lab (J-PAL), the International Initiative for Impact Evaluation (3ie) and Innovations for Poverty Action (IPA). Sitting at the centre of a large network of partners, DEval must be in a good position to assess the spread of those innovations. How well do you think the broader evaluation community are managing to adopt these new methodologies?
Martin Bruder: We hope that the times of separated “method camps” within the evaluation community are over. Text mining and machine-learning are blurring the boundaries of qualitative research, process tracing is improving the rigour of small-n analyses, participatory statistics are engaging target groups in new ways, “big data” from satellites and a multitude of other sources provide rich data sources still largely untapped for evaluation purposes. Increasingly advanced methods of rigorous impact evaluation are therefore an important area of innovation but not the only one. All of these developments provide great opportunities for the evaluation profession and will become more commonplace with a new generation of evaluators entering the profession. Given that the speed of methodological innovation will remain high, evaluation capacity development is an increasingly pressing issue for the whole evaluation community. We also need to engage with these developments at a conceptual level. That is one reason why DEval has initiated the EES Thematic Working Group on “Methods and Designs” (TWG 6). We want to engage with others to improve how to integrate multiple methods in mixed- and multi-method designs to better assess complex evaluation objects in dynamic contexts. In particular, we are interested in identifying causal effects and causal mechanisms through integrated cross- and within-case designs. Rigorous impact evaluation designs are a very important piece of this puzzle.
Going forward, have you seen an awareness among donors and policymakers of the changing face of evaluation? To what degree do they adopt rigorous impact evaluation?
Martin Bruder: I worry that within development cooperation any form of complex social science-based evaluation methodology – be it qualitative or quantitative – is under an increasing level of pressure due to a more adverse political climate concerning aid budgets in many donor countries. As an evaluation community we should therefore be wary of pitching one approach against the other, but rather join forces to contribute to providing better evidence for better decisions in the interest of the target groups of development interventions.
As far as rigorous impact evaluation is concerned, we believe that the picture is very diverse. For some countries and organisations, rigorous impact evaluations are an integral part of their development practice – in some cases, the pendulum has even somewhat swung back from very high levels of enthusiasm. Others, for various reasons, remain reluctant to adopt rigorous impact evaluation in the first place. There is a third group of countries and organisations, including Germany, that recognise that they have not yet made full use of its potential. In Germany, the use of such methods has been rather sporadic thus far. We now see some positive dynamics, which we hope will lead to a more systematic use of rigorous impact evaluation designs. We believe there is substantial further potential in the application of those methods in German development cooperation for those questions they are suited to answer.
How will the more systematic adoption of impact evaluation designs affect the use of other evaluation approaches?
Marion Krämer: Rigorous impact evaluation designs are an important part of an evaluator’s toolbox but certainly not the only one. Rigorous methods are, for instance, powerful when it comes to questions concerning the effectiveness of interventions designed to reach a large number of beneficiaries. In such cases, rigorous impact evaluations allow valid causal attribution of effects and, when more than one treatment is tested, enable decision-makers to choose the most effective way to implement an intervention. They also have their limitations. For example, they are resource-intensive and struggle with challenges to external validity. These limitations need to be discussed openly in the same way as all alternative evaluation approaches. In good applied research, the method is derived from a practically relevant question or concern, not vice versa.
You are currently conducting a research project on rigorous impact evaluation in German development cooperation. The aim is to contribute to a more systematic use of these designs as well as the evidence they produce. How do you approach this goal?
Marion Krämer: We have four main work areas. First, in a research component, we investigate the status quo in terms of the implementation and use of results of rigorous impact evaluation in German development cooperation. We also look at reasons that prevent development professionals from implementing and using results of such evaluations in a more systematic way. We aim to formulate concrete advice on how to reduce barriers by looking at examples of best practice. Second, we are building national and international networks to engage with the latest developments in the field of impact evaluation, to create synergies and to act in a more coordinated way. Third, we provide advisory services for a small number of ongoing impact evaluations in German development cooperation. And, fourth, we support BMZ in establishing a mechanism to provide some seed funding for impact evaluation in German development cooperation.
On the basis of what you have learned in your project so far, what do you see as upcoming developments for the evaluation community?
Marion Krämer: I believe one area in which rigorous impact evaluation has a lot of potential to inspire new developments is when it comes to drawing lessons from different evaluations in one sector or field of activities. My hunch is that 90% of the evaluations conducted today are never used for learning outside the specific project or programme that they evaluated. The impact evaluation community is very active in creating new tools, such as evidence maps, evaluation syntheses and evidence portals, to make use of their evidence as easy as possible and put it at the fingertips of decisions makers. We all should think harder about how to create formats that allow us to make increased use of findings from other types of evaluations and do so in an intuitive, easy and fun way. Only when people use our evaluations can we contribute to better decision making and better lives of people around the world.
Disclaimer: The opinions expressed in this article are the author’s own and do not necessarily reflect those of the European Evaluation Society.