Almost two decades after the publication of the “Methodology Guidance” in 2006, the European Commission has released a comprehensive new manual, entitled the Evaluation Handbook. This Handbook is designed to prepare, launch and manage the evaluation process within European External Actions. What does the Handbook cover? What are its strengths and weaknesses? How useful is it in practice? These are the questions I will try to answer in this article.
- Introduction
Several international organisations publish evaluation handbooks from time to time, such as the OECD, the UNICEF, the UN or the World Bank. These handbooks are mainly used to measure the impact of development programmes and humanitarian projects. The handbooks are not binding but provide useful guidance to practitioners. They are therefore useful and increase predictability and legal certainty.
The EU seeks to build international partnerships that uphold and promote European values and interests and contribute to world peace and prosperity. Evaluations are at the heart of the EU’s institutional culture of international partnerships, and reliable evaluation results are important tools for building closer and more effective cooperation with partner countries.
The Commission is committed to the principle of “evaluation first” and uses evaluation as an essential tool for its various policies and decisions. Commission staff are constantly encouraged to make full use of the evaluation results already available to better support partner countries’ efforts to eradicate poverty, improve governance and achieve sustainable growth.
- Overview of the new Handbook
In July 2024, the European Commission’s Directorate-General for International Partnerships (DG INTPA) and the Service for Foreign Policy Instruments (FPI) published a revised Evaluation Handbook (“Handbook), updating its 2006 predecessor “Methodology Guidance” to reflect the significant advances in evaluation practice over the last two decades.
It incorporates contemporary evaluation literature, international good practice and valuable lessons learned from consultations with staff working on these issues. The Handbook is not only aimed at Commission staff, but also at involving evaluation experts and contractors in methodological discussions.
This comprehensive guide is divided into four main chapters:
- The role of evaluation: This chapter introduces the role of evaluation within DG INTPA and FPI, explaining the main concepts, types, timing and actors involved in evaluation, and how they fit together in evaluation planning.
- Managing evaluations: This part provides hands-on practical guidance on managing evaluations through the six-phases of the evaluation process: preparatory, inception, interim, synthesis, dissemination, and follow-up. The chapter details the tasks to be carried out, the steps to be followed and the mandatory or recommended templates for the different stages of the evaluation.
- Evaluation approaches and methods: This chapter deals with different evaluation approaches, methods and tools, providing detailed explanations and examples. It incorporates contemporary practices, discussing the use of big data, geospatial technology and participatory video, among other modern techniques. This inclusion ensures that users are equipped with the latest methodologies to conduct robust and relevant evaluations.
- Ethics in evaluation: It emphasises the fundamental importance of ethics in evaluations, detailing ethical principles and standards and stressing the importance of conducting evaluations ethically and responsibly. It provides guidance on how to deal with the most common ethical issues, especially in situations of fragility, crisis, conflict or violence.
The Handbook is available electronically, which facilitates easy access and regular updates, making it a living document that can be adapted to evolving practices and needs. Integration with platforms such as OPSYS and the Evaluation Wiki further enhances its accessibility and usefulness, providing users with a one-stop-shop for all evaluation-related guidance and resources.
- Strengths of the Handbook
3.1 Clarity and structure
One of the main strengths of the Handbook is its clarity and accessibility. Unlike some complex policy documents, the Handbook is well structured and easy to navigate. The step-by-step instructions, complemented by practical templates and checklists, make it an excellent resource for both emerging and experienced evaluators.
3.2 Integration of technological innovations
A notable improvement in this edition is the incorporation of big data, geospatial tools, and participatory video into evaluation methods. These additions reflect the evolving nature of data collection and analysis, making evaluations more accurate and insightful.
3.3 Emphasis on ethical considerations
The Handbook’s strong focus on ethics is commendable, particularly with regard to gender equality, human rights, and data protection. As digital tools become increasingly common in evaluation processes, protecting privacy and ensuring ethical data collection is more important than ever.
3.4 How evaluators can benefit from this Handbook
Evaluators will find this Handbook very useful for its structured approach, practical tools and emphasis on technological and ethical advances. The detailed guidelines, templates and checklists help to streamline evaluation processes and ensure consistency and efficiency. The inclusion of data-driven methodologies, such as big data and geospatial analysis, enables evaluators to make more accurate assessments. In addition, a strong focus on ethical considerations ensures that evaluations remain responsible and inclusive. While the Handbook provides a solid foundation, evaluators should remain flexible and adapt its recommendations to their specific contexts and available resources.
- Opportunities for further improvement
While the Handbook has considerable strengths, there is room for further improvement in order to contribute to more effective evaluations.
4.1 Inclusion of additional real-world case studies
While the Handbook effectively explains evaluation theories and frameworks, the inclusion of more practical case studies would enhance its applicability. Real-world examples of how different methodologies are implemented in different global contexts would provide more valuable insights for practitioners.
4.2 Greater emphasis on local adaptation
Given the diverse environments in which evaluations take place, additional guidance on how to adapt the frameworks to different local contexts would be beneficial. This could include specific recommendations for conducting evaluations in areas with particular social, economic or political challenges.
4.3 Providing alternative methods of data collection
While the integration of digital tools is a major strength, it is important to recognise that not all regions have equal access to technology and infrastructure. Providing alternative methods for data collection in low-resource settings would enhance the inclusiveness and practical relevance of the Handbook.
- Conclusion
The European Commission’s updated Evaluation Handbook is a valuable and practical resource, especially for those working on EU-funded projects. It provides clear guidance, integrates modern technology and prioritises ethical evaluation practices. While there is room for further improvement – such as including more real-world examples, increasing adaptability to different contexts and providing alternative methods for data collection – the Handbook is undoubtedly a significant step in the right direction.
The Handbook is a commendable effort to improve the field of evaluation. It strikes the right balance between clarity, technological advances and ethical considerations, making it a valuable tool for evaluators worldwide.
For evaluators looking for a structured and insightful reference, this Handbook is well worth exploring. By carefully adapting its recommendations to specific challenges, practitioners can make the most of its resources and contribute to more effective and ethical evaluations. With further refinement and adaptability, it has the potential to become an indispensable resource for the evaluation community.
Author information: Diana Kun works at the Hungarian Managing Authority in the Ministry of Public Administration and Regional Development (National Development Centre), dealing with European Union projects under the Human Resources Operational Programme. The views expressed in this article are the personal view of the author and should not be taken to reflect the views of the Ministry of Public Administration and Regional Development in Hungary.
References:
Evaluation Handbook (2024) European Commission, Directorate-General for International Partnerships, Hassnain, H., McHugh, K., Lorenzoni, M. et al., Publications Office of the European Union, 2024, Accessed from: https://op.europa.eu/en/publication-detail/-/publication/06350947-4d56-11ef-acbc-01aa75ed71a1/language-en
The European Commission’s Evaluation Handbook, Capacity4dev, Accessed from: https://capacity4dev.europa.eu/library/european-commissions-evaluation-Handbook_en?refgid=660