What kind of evaluator are you? Exploring different evaluator paradigms | Eval Forward
This blog explores the concept of “evaluator paradigms” and how different belief sets may influence an evaluator’s approach. Although inevitably generalising, any practising evaluator is likely to find some reflection of themselves and their colleagues. Conceived of by Egon Guba and Yvonna Lincoln in the late 1980s, four paradigms are described: transformational, (post-)positivist, utilitarian, and constructivist. The authors developed a quiz to determine how these paradigms apply in practice and if evaluators recognize themselves in them. The survey results showed that respondents did recognize themselves in these paradigms and often combined them, with the constructivist paradigm being the most common.
The study found that factors like gender, age, career level, and educational background had limited influence on paradigm choice, but specific academic background and work sector did influence choices, especially regarding the two less frequent paradigms. The findings also hinted at different national evaluation cultures and the possibility that preferences might be influenced by broader cultural aspects or an individual’s past.
Some experienced evaluators acknowledged they could align with all paradigms depending on the context, suggesting the existence of a “comfort zone” that might expand with experience. The research questions the notion of paradigms themselves and posits them as a topic for further study. In summary, the blog emphasises the multifaceted nature of evaluation paradigms and the influence of personal, professional, and cultural factors in shaping an evaluator’s approach.
J-PAL North America, ever the vanguard of evidence-based social science, has released a new Health Care Evaluation Toolkit aimed at guiding social scientists in conducting randomised evaluations of health care delivery interventions within the United States but with import for all evaluators in the health sphere. Drawing on a decade of experience and over 50 randomised evaluations, the toolkit provides resources that address real-world health care challenges. It includes guidance on working with hospital Institutional Review Boards (IRBs), assessing power and feasibility in study design, and collaborating with implementing partners to address unique challenges related to health impacts. The toolkit also explores the complex relationships between patients and providers, focusing on issues like informed consent and enrolling vulnerable individuals, and offers insights on how to foster a supportive environment among staff. Furthermore, it offers guidelines on navigating regulations such as HIPAA, understanding levels of identified health care data, and provides tips for publishing in medical journals, including understanding unique requirements and advanced planning. Practitioners will find in this blog a useful preview of the bureaucratic obstacles that, while maybe not immediately apparent, often pattern M&E in healthcare, as well as the sensitive outcomes peculiar to the sector. As policy-makers struggle with intensifying challenges in the post-covid healthcare landscape, pragmatic research such as this will be of increasing value.
World Bank Support for Domestic Revenue Mobilization
Since the World Conference on Financing for Development held in Addis Ababa in 2015, the focus on domestic revenue mobilisation (DRM) has intensified within the global development policy community. Existing official development assistance has long been recognised as insufficient to meet the Sustainable Development Goals, especially as the context of the goals radically changes with recent interlocking crises. Therefore, alternative resources, including domestic revenue, will have to be tapped into. The report gives a useful overview of the story of this response and its prospects going forward.
In the period leading to the COVID-19 pandemic, the emphasis on enhancing DRM grew, particularly for developing and lower-income nations. This emphasis was driven by mounting fiscal deficits and escalating debt levels. With the pandemic’s outbreak, there has been a 12% real-terms reduction in tax revenues, and in numerous countries, the tax to GDP ratio has fallen beneath the 15% threshold, deemed the minimum required to sustain a state’s fundamental operations.
The report examined the pertinence and efficacy of strategies and interventions supported by the World Bank to assist client nations in augmenting DRM between fiscal years FY16 and FY19. Although this time frame is relatively brief, it encompasses a significant increase in the international community’s attention to DRM and permits a comparison with the earlier FY12–15 period.
This evaluation draws on insights from country-specific studies in Chad, Colombia, Guatemala, Kenya, Madagascar, Pakistan, and Rwanda, and puts forth 4 specific recommendations for the World Bank to optimise its existing resources in support of DRM.
M&E has rightly focussed in recent years on local, specific, measurable, achievable, relevant, and time-bound outcomes but it does no disservice to the profession to keep more abstract concepts in mind, especially as these form import parts of the landscape of practitioners’ work. The blog post discusses one such concept and its evolution: democracy and its alarming decline. As noted by international observers such as the V-Dem Institute, there has been a measurable decline in competitive elections, political participation, and public accountability around the world. The 2023 Democracy Report found more closed autocracies than liberal democracies for the first time in two decades.
To combat this trend of democratic backsliding, many international actors are implementing interventions but often lack evidence of their effectiveness. 3ie, supported by the UK government’s Foreign, Commonwealth and Development Office (FCDO), is synthesising existing findings in a rapid evidence assessment.
The synthesis draws from six recently-published evidence gap maps on Democracy, Governance, and Human Rights. It has been observed that studies specifically focusing on democratic backsliding contexts are relatively small (197 studies) compared to the overall evidence base (1,867 studies). While some clusters of studies measure the effectiveness of accountable governance and civic space interventions, significant gaps remain, especially regarding inclusive politics, corruption, economic governance, and media freedom.
To quickly address this synthesis gap, the Rapid Evidence Assessment is focusing on relevant interventions in a subset of 13 countries across four regions, analyzing 64 of the 197 studies. This method is considered rigorous yet reduces the required time and cost by half compared to a standard systematic review.
The authors conclude by inviting suggestions, collaboration, and participation in their ongoing project, providing contact information for those interested in receiving preliminary findings and updates. We should expect that much of the feedback will urge caution – evaluators have won important victories for impactful policies in undemocratic environments and, while affirming their commitment principles of political participation and accountability, practitioners will be hesitant to jeopardise those roles.
Evaluation synthesis note – Targeting in IFAD-supported projects
This blog explores the vexed issue of targeting as part of IFAD’s mandate, and as it aligns (or conflicts) with the broader 2030 Agenda’s commitment to inclusivity. It brings to light several challenges within the current practice, including terminological confusions, misinterpretations surrounding target groups, and an often glaring gap between theory and practice. It finds that while there have been some improvements in targeting, IFAD has not fully capitalised on the demands of the 2030 Agenda or implemented universal principles across its diverse portfolio.
The blog also raises concerns about the drift away from people-centred development and the replacement of thoughtful analysis with a culture of compliance. Naturally, such allegations will be vigorously contested but likely resonate to some degree with many practitioners. Weak monitoring and evaluation systems, coupled with insufficient resources, have hindered the effective evaluation and implementation of targeting strategies. Some more promising practices and lessons learned are identified, including the need for collaboration, rigorous evaluation, and a renewed focus on participatory development, although in so doing the problematic preference for theory over practice is arguably repeated.
The piece portrays a complex picture of IFAD’s approach to targeting that is unlikely to please anyone completely but surely succeeds in capturing some practitioners’ most pressing concerns. While recognizing targeting’s centrality and the historical improvements made, it calls for significant changes in definitions, practice, evaluation, and resource allocation to fully realise the organisation’s potential. It underscores the necessity for a comprehensive overhaul of the existing systems, incorporating critical thinking, collaboration, and innovation to achieve better targeting outcomes.