Advanced Content Analysis: Can Artificial Intelligence Accelerate Theory-Driven Complex Program Evaluation? | World Bank Independent Evaluation Group  

The spectre of artificial intelligence (AI) has haunted the evaluation profession for decades, evoking excitement and trepidation in nearly equal measure from practitioners anticipating a broad-based transformation of their work.  AI has seldom lived up to its promise, neither revolutionising nor destroying M&E.  However, the slow normalisation of large-scale and high-frequency data collection is widely predicted to herald a turning point in the adoption of AI solutions, which this report explores in some detail.  Abandoning the abstraction with which the issue is usually saddled, the report looks closely at AI-driven content analysis, a subfield of Natural Language Processing (NLP). The pace of NLP development has rapidly increased in step with data production, bringing with it such products as Google Translate.  There is evidently a gap to be filled as the authors acknowledge: with stagnant or falling budgets, increasing workloads and a rapid, unmanageable proliferation of new research, practitioners urgently require better tools to assimilate knowledge than traditional synthesis methods. This feasibility study focuses on the application of AI, specifically machine learning and knowledge graph methods, in identifying key aspects of an intervention in its documentation. The value of the report lies in how few assumptions it makes about the best practice for AI in M&E: although not exhaustive, it explores the main methods of machine learning without prejudice and draws on existing research without relying on it.  Predictably, the report finds that while a clear use case exists and the exploration is proof of concept, the effort involved in pioneering these methods obscures the time efficiencies that it may yield in the near future. While the report primarily looks ahead to future projects, automated tools already exist and the report might also provide practitioners with a view inside the black box of the computational social sciences that increasingly influence the profession.

United Nations contributions to national evaluation capacity development and the evolution of national evaluation systems  | United Nations Evaluation Group

Unsurprisingly, at the top of most evaluators’ reading list this month will be UNEG’s report on the United Nations contributions to national evaluation capacity development and the evolution of national evaluation systems.  The wordy title makes no secret of the reports’ exhaustive detail, which defies summary.  In all likelihood, the report will be studied and referenced for an extended period to come as institutions compare and contrast their investment in M&E.  

The report is notable not just for its contents but for being based on an agreed definition of national evaluation capacity development (NECD), which caps a lengthy process of agreeing common metrics. The report casts a wide net across the evaluation profession, drawing on the diverse cast of organisations, academics, activists constantly shaping M&E.  All of them are represented in the report through a range of case studies and interviews. 

How process evaluation can help improve program design and implementation | 3ie 

Process evaluation has emerged as an essential complement to its more traditional impact counterpart.  In essence, process evaluation examines the effectiveness of evaluation implementation, rather than the results of that implementation.  With standards of accountability to stakeholders advancing and the growing difficulty of disaggregating impact in a crowded field, process has taken on new importance.   

In this two-part blog from 3ie’s Michael Bamberger, the present state and future opportunities for process evaluations are analysed in detail. The first entry examines both the arguments for conducting process valuations and the persistent challenges in doing so. The second goes into practical detail on how a more comprehensive approach can be adopted.  

Emerging evaluator programme challenges notion that sustainable capacity development initiatives are lacking | EvalForward

This blog is another welcome addition to those shedding some light on the progress of emerging evaluators. As ever, there is an incongruity between the demand for evaluation and the development of professional evaluators. This is particularly true in the global south where there is an acute need for locally-based evaluators.  This blog, building on a theme of local expertise and local-led evaluation at evalforward, looks at an exemplary programme designed to cultivate young and emerging evaluators, bridging the divide from related professions and define a longer-term career path.  The blog is a useful entry point for those seeking inspiration for similar programs elsewhere in the world and for those looking to develop partnerships in the global south.

Transitioning to a Circular Econom | World Bank Independent Evaluation Group  

Unglamorous though it undoubtedly is, waste management is a core component of most comprehensive sustainability initiatives. A ‘circular economy’, in which systematic recycling ‘decouples economic activity from the consumption of finite resources’, has come to be seen as critically important. This blog examines the World Bank’s growing experience with the issue and takes a usefully wide view, covering the decade 2010-2020.  The World Bank has an unparalleled wealth of experience in implementing Municipal Solid Waste Management (MSWM) and its analysis is, if nothing else, a useful template for such broad retrospectives.