The EBRD-EvD Approach to Real-Time Evaluation

By Natalia Kryg(1), Gabriele Fattorelli(2) – EBRD

Crises prompt change. The COVID-19 pandemic tested the European Bank for Reconstruction and Development (EBRD) and its operations, prompting a unique evaluation response. The EBRD’s Independent Evaluation Department (EvD) adopted real-time evaluation (RTE) toolkit to provide rapid, evidence-based assessments of the crisis response. This innovative approach not only shaped decision-making but also led to insights for future crisis situations, namely in the EBRD’s response to the war in Ukraine.

 

RTEs: A Path in Crisis

RTEs, typically used by humanitarian organizations that have been conducting RTEs since the 1990s, offer real-time feedback during ongoing crisis responses, which is intended to contribute towards the decision-making process behind the crisis response. The EBRD’s EvD utilized RTEs for the first time to assess the crisis intervention of EBRD in times of the COVID-19 pandemic. The approach, driven by the necessity demanded by the EBRD Board of Directors, lacked a standardized definition within the international financial institution (IFI) community.

The evaluation offices from the IFI community faced the COVID-19 pandemic with limited or no experience in conducting RTEs. This led to decentralized approaches of tacking RTEs based on their own interpretation of RTEs tools. As a result, there is neither a unique codified definition nor a harmonized approach to RTEs among evaluation offices of IFIs today.

 

EBRD’s RTE Journey

During the pandemic, EvD conducted its first RTE, focusing on COVID-19 response assessment. It applied a phase evaluation approach, with its first product being a rapid assessment. This phase 1 product aimed at formative learning and identifying deviations from planned strategies by the Bank during the pandemic. Its success triggered the demand for more rapid assessments in response to the new crises, most recently in investigating the EBRD’s response to the war in Ukraine. The pandemic’s evolving nature required timely, adaptable methodologies. The subsequent evaluation phase delved into impact evaluation, marking EBRD’s debut in counterfactual impact evaluation (CIE).

 

Key Elements of RTE Methodology

RTEs demand flexibility. EvD adhered to OECD-DAC evaluation criteria while innovating within the RTE framework. The evaluation process remained ongoing, aligned with the programme’s duration. In-depth consultations with EBRD Board and Management ensured relevance and actionable insights. Mixed-methods approaches, including qualitative and quantitative proxies, enabled robust evaluation despite data and time constraints.

EvD’s RTE approach differs significantly from traditional ex-post evaluations in various aspects. The key distinctions include:

• Purpose: RTEs focus more on learning than accountability.
• Timing: RTEs start concurrently with the crisis response, with the evaluation of the COVID-19 program closing shortly before the Phase 1 report.
• Consultation: Closer engagement with EBRD Board, Management, and other IFIs to ensure collaboration and cooperation.
• Methodological tools: Utilizing familiar qualitative and quantitative methods but adapting them due to time and data constraints. Innovative use of proxies, such as surveys, when data is incomplete.
• Process: RTEs conducted swiftly within ongoing crises, leading to faster results compared to conventional evaluations.
• Resources: Phase 1 was internally handled by EvD’s team, whereas Phase 2 involved external consultants due to the new approach’s complexity.
• Skills: Expertise in quick, precise analysis and familiarity with EBRD systems and operations is crucial.
• Deliverables: Reports are concise, employing infographics and digital tools to cater to limited audience attention during crises.
• Recommendations: RTEs offer early lessons, some shaping subsequent crisis response strategies.
• Dissemination: Informal engagements with the EBRD Board and Management facilitate knowledge sharing, with summarized non-confidential findings distributed externally.

 

Success Factors Amid Challenges

Several key factors contributed to the success of the EBRD-EvD’s implementation of RTEs during crises:

• Methodological Soundness: The EvD maintained the fundamental principles of evaluation even within the constraints of real-time assessment, ensuring that the evaluations were credible, impartial, and evidence-based.
• Timely Delivery: The urgency of the crises required prompt evaluation. The EvD’s use of automated data updates and streamlined processes minimized time constraints.
• Actionable Insights: While the RTEs did not generate formal recommendations, they provided valuable lessons and insights that informed decision-making and program adaptation.
• Collaboration: Close collaboration with the EBRD’s Management and Board, as well as engagement with other IFIs, facilitated the successful implementation of RTEs.
• Effective Communication: Clear and concise communication of findings, aided by visual aids and presentations, ensured that evaluation insights were accessible and comprehensible to stakeholders.

 

Challenges and Trade-offs

Implementing RTEs during crises presented a set of challenges and trade-offs. The limitations of available data, the dynamic nature of crises, and the need for rapid assessments required EvD to strike a balance between rigour and timeliness. In some cases, the quality of evaluation results depended on expert judgment due to incomplete information. Additionally, ensuring that evaluation findings remained relevant and accurate as the crisis evolved was a delicate task. The absence of formal recommendations also required trust in the ability of management to act on early lessons.

 

Harmonizing RTEs in IFI Community

The IFI community faced the pandemic with varying RTE approaches. Recognizing the need for standardization, the Evaluation Cooperation Group (ECG) formed a working group to establish common RTE practices. This collective effort aims to share experiences, challenges, and best practices, culminating in a “good practice” document. The authors of this blog can share further details with anyone interested in learning more about this initiative.

 

Learning from each other

EBRD’s RTE experience highlighted benefits and challenges shared by other IFI evaluation offices:

• Benefits: Timely, relevant insights; formative learning; quicker production; evaluator flexibility; sharper focus.
• Challenges: Limited or preliminary data; time constraints affecting quality and collaboration; methodological demands; internal resource intensity; institutional culture for effective learning.

 

Conclusion: Revolutionizing Evaluations in Times of Crisis

In the crucible of crisis, the RTE approach shines as a beacon of proactive resilience and evolution. RTEs demand a balanced approach, blending learning and flexibility. EvD’s RTE journey demonstrated that adapting evaluation methodologies during crises fosters informed decision-making. As the IFI community navigates ongoing uncertainties, the harmonization of RTE approaches promises better evaluation outcomes and enhanced crisis responses within the international financial community.

As the world navigates an increasingly complex and interconnected landscape, the RTE model championed by EvD offers a beacon of hope and transformation. By placing learning at the heart of crisis responses, harnessing innovation in methodologies, and fostering collaboration and transparency, the EBRD not only evaluates its actions but also shapes a brighter, more adaptive future for itself and the organizations it inspires.

 

[1] EBRD-EvD, Principal Economist. Corresponding author: krygn@ebrd.com

[2] EBRD-EvD, Director of Corporate and Thematic Division