TWG 6 - Methods and Designs
Evaluations often deal with multiple and multi-dimensional evaluation questions and assess complex evaluation objects in changing contexts. This requires evaluators to apply different methods and to analyze different types of data. Additionally, the evaluation community is multi-disciplinary relying on a wide set of analytical procedures from the social sciences, natural sciences, applied sciences and humanities. The toolbox of available designs and methods is thus rich and diverse.
This EES Thematic Working Group deals with issues around evaluation methods and designs. Specifically, since the evaluation community agrees that there is a potential value added from integrating different methods in evaluations, the primary focus for the first four years (2019-2023) will be on how to apply and integrate multiple methods in mixed/multi-method designs. This concept note is therefore written with a focus on method integration in mind. The members of the TWG will decide in 2019 on a bi-annual work program (until 2021) and at the end of the second bi-annual (2021-2023) will decide whether to maintain or change the thematic focus of the TWG.
The relevance of setting up this working group builds upon the following observations:
1. No single method is able to cope with interventions that affect multiple levels of analysis, involve short- and longterm outcomes, and address complex systems with interacting components. The TWG supports the evaluation community in selecting and integrating suitable quantitative and qualitative methods.
2. Evaluation can benefit from rigorous causal inference. The TWG supports the identification of causal effects and causal mechanisms through integrated cross- and within-case designs.
3. There is potential in the increasing availability of mixed- and multi-method approaches to evaluation. The TWG contributes to clear evaluation designs by providing formal protocols.
4. Evaluations increasingly draw on new types of data (including big data such as geo-spatial imagery, transactional data, mobile phone records or internet/app data) and use of innovative analytical tools like machine-learning. The TWG promotes the integration of emerging data types and analytical techniques in complex evaluations.
The TWG sees an increased need for methodological exchange and learning on method integration.
With this Thematic Working Group we strive to improve the quality of evaluations and strengthen evidence-based decision making through better methods and designs. Specifically, our aim for the coming four years is advancing the systematic integration of different methods in evaluations in order to generate more valid and reliable evidence.
The TWG “Methods and Designs” will be a collaborative space to share experiences, good practices and lessons learned round evaluation methods and designs while at least initially a particular focus will lie on systematically integrating different methods in evaluations. It builds on ongoing debates in mixed- and multi-method research within the evaluation discipline and in the broader social sciences. The TWG aims at bringing together evaluators and academics to exchange ideas and experience on innovative ways of combining methods and to develop systematic approaches of method integration in evaluation. The main objectives of the TWG are to:
- Facilitate methodological debate and exchange among EES members, specifically on the matter of method integration.
- Bundle expertise and best practices in mixed-methods and multi-method approaches for evaluation.
- Reconcile different methodological cultures in the evaluation community.
- Transfer and adapt recent methodological innovations from the social sciences for the field of evaluation.
- Work and advance towards a more systematic integration of multiple methods in evaluation.
- Increase the availability of systematic approaches to method integration in evaluation.
- Provide guidance for evaluators on the selection of appropriate methods and their integration.
- Improve the integration of new types of data and data analysis in evaluation designs.
Accordingly the TWG helps bring together evaluators and academics interested in improving the use and integration of methods and evaluation designs.
In order to achieve these objectives, the TWG plans the following activities:
- Annual workshops of the TWG in order to enable evaluators to exchange experience, best practices and lessons learned knowledge.
- Develop a virtual platform for evaluators for knowledge exchange.
- Propose and agree with the EES Board ona methodological theme/strand for the biannual EES conference.
The workshops 2019-2021 will focus on the systematic integration of multiple methods in (complex) evaluations. Ideally, the 2019 workshop will result in a proposal for a methodological theme or strand for the 2020 EES conference. While the list of potential outputs is long and will need to be discussed and approved by the members of the TWG, EES members expressed particular interest in developing practical guidelines for the integration of methods in (complex) evaluations.
The TWG shall be open to all EES members. Feel free to join the working group and participate.
Sven Harten – Deputy director, DEval Sven.Harten@deval.org
Gerald Leppert – Senior evaluator, DEval
Martin Noltze – Senior evaluator, DEval
Mascha Rauschenbach – Evaluator, DEval
Johannes Schmitt – Evaluator, DEval
Thomas Wencker – Evaluator, DEval