Search on EES

Presentation of TWG

Evaluations often deal with multiple and multi-dimensional evaluation questions and assess complex evaluation objects in changing contexts. This requires evaluators to apply different methods and to analyze different types of data. Additionally, the evaluation community is multi-disciplinary relying on a wide set of analytical procedures from the social sciences, natural sciences, applied sciences and humanities. The toolbox of available designs and methods is thus rich and diverse.

This EES Thematic Working Group deals with issues around evaluation methods and designs. Specifically, since the evaluation community agrees that there is a potential value added from integrating different methods in evaluations, the primary focus for the first four years (2019-2023) will be on how to apply and integrate multiple methods in mixed/multi-method designs. This concept note is therefore written with a focus on method integration in mind. The members of the TWG will decide in 2019 on a bi-annual work program (until 2021) and at the end of the second bi-annual (2021-2023) will decide whether to maintain or change the thematic focus of the TWG.

RELEVANCE
The relevance of setting up this working group builds upon the following observations:
1. No single method is able to cope with interventions that affect multiple levels of analysis, involve short- and longterm outcomes, and address complex systems with interacting components. The TWG supports the evaluation community in selecting and integrating suitable quantitative and qualitative methods.
2. Evaluation can benefit from rigorous causal inference. The TWG supports the identification of causal effects and causal mechanisms through integrated cross- and within-case designs.
3. There is potential in the increasing availability of mixed- and multi-method approaches to evaluation. The TWG contributes to clear evaluation designs by providing formal protocols.
4. Evaluations increasingly draw on new types of data (including big data such as geo-spatial imagery, transactional data, mobile phone records or internet/app data) and use of innovative analytical tools like machine-learning. The TWG promotes the integration of emerging data types and analytical techniques in complex evaluations.
The TWG sees an increased need for methodological exchange and learning on method integration.

VISION
With this Thematic Working Group we strive to improve the quality of evaluations and strengthen evidence-based decision making through better methods and designs. Specifically, our aim for the coming four years is advancing the systematic integration of different methods in evaluations in order to generate more valid and reliable evidence.

Objectives of TWG

MISSION STATEMENT
The TWG “Methods and Designs” will be a collaborative space to share experiences, good practices and lessons learned round evaluation methods and designs while at least initially a particular focus will lie on systematically integrating different methods in evaluations. It builds on ongoing debates in mixed- and multi-method research within the evaluation discipline and in the broader social sciences. The TWG aims at bringing together evaluators and academics to exchange ideas and experience on innovative ways of combining methods and to develop systematic approaches of method integration in evaluation.

The main objectives of the TWG are to:

• Facilitate methodological debate and exchange among EES members, specifically on the matter of method integration.
• Bundle expertise and best practices in mixed-methods and multi-method approaches for evaluation.
• Reconcile different methodological cultures in the evaluation community.
• Transfer and adapt recent methodological innovations from the social sciences for the field of evaluation.
• Work and advance towards a more systematic integration of multiple methods in evaluation.
• Increase the availability of systematic approaches to method integration in evaluation.
• Provide guidance for evaluators on the selection of appropriate methods and their integration.
• Improve the integration of new types of data and data analysis in evaluation designs.

Accordingly the TWG helps bring together evaluators and academics interested in improving the use and integration of methods and evaluation designs.

STRATEGIC ACTIVITIES
In order to achieve these objectives, the TWG plans the following activities:

• Annual workshops of the TWG in order to enable evaluators to exchange experience, best practices and lessons learned knowledge.
• Develop a virtual platform for evaluators for knowledge exchange.
• Propose and agree with the EES Board ona methodological theme/strand for the biennial EES conference.

Specific Contact Person in the EES Board

Alena Lappo, alena.lappo@gmail.com