What did we read in May : Reforming ethics in Evaluation, Initiative-level Meta-theory

Search on EES

Three ways theories of change are helping evaluate a complex initiative to improve waste pickers’ lives | 3ie

Among challenging evaluation scenarios, few are more difficult and thankless than those that involve multiple overlapping interventions.  Anything less than the most rigorous approach is likely to render impact disaggregation practically impossible.  Unfortunately, partners still routinely mix (and remix) their projects, proceeding to involve an evaluator only at a late stage, and supplied frequently with only post hoc justifications.  While this interesting blog, from 3ie’s Evidence Matters series, does not provide a systematic method of overcoming this problem, it does outline how it was tackled in practice.  Four evaluators summarise how a highly complex task – evaluating several projects simultaneously that were all working towards improving the lives of urban waste pickers – was approached in a thorough manner. They are surely right to place a great deal of emphasis on partner engagement, especially since the projects involved a startling range of organising and supporting organisations.  This allowed them to reconstruct as far as possible the project’s theories of change.  As practitioners globally are discovering, an important part of this process was understanding the disruption wreaked by the pandemic and how the program processes adopted to manage it.  Where the evaluation strays into innovative work is its proposition of an “initiative-level meta-theory”, which combines common elements, outputs and outcomes in the projects.  The evaluation is ongoing and we eagerly await the conclusions of this approach.

Ethics for evaluation: three lessons from the essential new book for evaluators | Itad

This article is a dispatch from the recent annual conference of the UK Evaluation Society and covers a session dedicated to ethics in evaluation, based on the freshly released book ’Ethics for Evaluation: Beyond “doing no harm” to “tackling bad” and “doing good”’. Ethics is hardly a new subject in evaluation and most evaluators, although engaged with the ethics debate, will most value a concise summary of best practice, at least at the global level. Needless to say, with local contextualisation an ever more salient theme in evaluation, the idea of universal ethical standards is under pressure and this blog goes some way to acknowledging that.  The blog (and book’s) idea that ethics should facilitate a debate, rather than preclude it with sweeping conclusions, is a refreshing perspective.  The authors are at pains to show this does not mean an abnegation of responsibility but rather a testing and laborious commitment to evidence-based learning and inclusive discussion.  The authors also propose to partner professional conduct with meaningful actions and a broader notion of social responsibility above and beyond the immediate requirements of the evaluation.  Similarly, they suggest that the ‘value-free’ approach has reached its limits and should be actively replaced with ‘values-critical’ and ‘value-committed’ approaches, which privilege dialogue, deliberation, social justice and human rights.  We would expect these aspirations to be tempered at the proposal stage of the evaluation, especially by partners operating under unusual or pressurised circumstances, but as a starting position, these principles are worthy of practitioners’ consideration.

Building the evidence for more effective disaster risk reduction | Independent Evaluation Group

This blog from the World Bank’s Lauren Kelly and Stephen Hutton is a primer for the upcoming (and Covid-delayed) seventh session of the Global Platform for Disaster Risk Reduction in Indonesia.  Few people will need reminding of the intersecting disasters afflicting the world: climate hazards, war, famine, and disease.  The UN Office for Disaster Risk Reduction (UNDRR), which is organising the event, has incorporated these compounding threats into its program, without subtracting from the importance of long-standing natural disaster relief and adaptation.  The blog, based on an upcoming IEG evaluation, looks in some detail at the Bank’s performance in helping countries and regions address the risks associated with major natural hazards. The blog outlines the evolving nature of disaster risk reduction (DRR) and the way in which best practice has developed to critically weigh risks, prioritise those most vulnerable and learn from the long, not merely the immediate, aftermath of acute natural crises.  With the world’s resources – economic, political and social – already over-leveraged, such judicious treatment of “resilient development” is very welcome.

Want to get girls to school in sub-Saharan Africa? Tackle poverty

The J-Pal Poverty Action Lab has a strong track-record in education evaluation in the global south and this extended article from J-PAL’s Alessia Mortara, Primrose Adjepong and Onyinye Oguntoye provides a range of useful insights on education for women and girls in challenging circumstances. Investigations of multidimensional poverty are increasingly entering the workflow of evaluators and this article explores how such complex ideas are put into practice.  A dizzying array of factors can influence education decisions for girls and their families and the blog emphasises both rigour and sensitivity in dealing with them.  As per usual, the article is punctuated with the density of facts and reliance on strong study design that we have come to expect of J-PAL.  The piece is largely a spirited defence of cash transfers, an increasingly fashionable poverty-relief measure, and of lowering the conditionality attached, which recipients traditionally chafe against.  The article is unlikely to convince long-standing sceptics of cash transfers since it does not directly tackle the conventional reservations, but as a survey of impressively designed studies on the subject it is a useful article.  It also serves as an update to education-related poverty relief in light of COVID-19 and provides some illuminating first-hand evidence of the ways the pandemic has changed delicate household decision-making.