Raghavan Narayanan, Senior Evaluation Officer, Finance and Private Sector Development Unit, World Bank and Tom Ling Head of Evaluation at RAND Europe and President of the European Evaluation Society.   The views expressed in this blog are the personal views of the authors and should not be understood to reflect the views of the World Bank, the Independent Evaluation Group, RAND Europe or the European Evaluation Society.

International Finance Institutions and the independent evaluation function

International Finance Institutions (IFIs) are financial institutions that are established by more than one country and most often by many nations. They are intended to strengthen trade and the economy, and this may include a concern with strengthening justice and fairness in that system. Therefore, they aim to minimize the tension between individual-country goals and collective-global goals, and to build trust in the global financial arrangements. The main types of IFIs are: Multilateral Development Banks, Bretton Woods institutions, regional development banks along with some bilateral banks and agencies. A key part of IFI’s effectiveness is that they should have legitimacy and authority. Holding authority and legitimacy requires transparent governance mechanisms and in turn this requires independent evaluations that are credible and relevant. Evaluations should ascertain the strength of evidence behind decision-making, actions taken, and results achieved. This includes, at a more organisation-wide level, evaluating the consistency and harmony among decisions, actions, and results. A first requirement is that this evaluation function should be independent and operate in an authorizing environment that is shielded from internal and external pressures and perceived or real threats to impartiality. Immunity from management capture is therefore fundamental to the evaluation process.  There are currently several perceived weaknesses in the IFIs, and that the independent evaluation systems suggest that IFIs are stuck in “active inertia” (G20, Brookings, 2022) and not delivering collectively as a system. For example, there is a growing sense, rightly or wrongly, that the evaluations are not sufficiently informing the IFIs to course correct and achieve the scale expected of them. At a time when international progress towards economic and development goals is stalling, there is a need for agile and responsive management and this, in turn, requires relevant, credible, and timely evaluative evidence. The independent evaluation function is therefore embedded within the wider functions of IFIs (innovation, accountability, monitoring, strategic decision-making and so forth). Furthermore, the independent evaluations of IFIs are part of a wider ecosystem of scholars, think tanks, not-for-profit consultants, the for-profit sector, and other providers. Evaluators both draw upon this ecosystem and are part of it. A strong and independent voice focused on the impacts achieved for development should be an important part of this both as providers of new knowledge and as knowledge brokers. Within this context, this blog explores three interrelated questions: (1) are independent evaluations still relevant in a post-pandemic, disinformation-filled world? (2) are independent evaluations getting marginalized in the IFI ecosystem due to the authorizing environment? and (3) can other external factors and new entrants in this area marginalize the independence function?

An independent and credible evaluation function

First, in today’s context of compounding or co-mingled crises (e.g., energy, food, war-and famine- linked migration, climate change), IFIs need to adapt and learn not just from lessons of the past (i.e., inter-crisis learning) but also learn in real-time from current engagement (i.e., intra-crisis learning). Such organizational learning may challenge current ideas and preconceptions. Learning uncomfortable lessons can be helped by an independent evaluation function for example by reducing asymmetries in information flows between sectors and strengthening system-wide learning. For example, successfully navigating the road to 2030 (SDGs, Net Zero) requires understanding tradeoffs, developing foresights to inform decisions, using empirically sound cost-benefit analysis, and generally supporting informed judgments from IFI managers. Hence, it is imperative for independent evaluations to assess whether unachieved goals were due to circumstances over which the IFIs had no control, or whether risks could have been managed differently and errors in judgement avoided (Arrow, 1974; Picciotto, 2004). To that extent, independent evaluations in IFIs can lead to new insights and encourage the pursuit of new mental models beyond the individual operators. This includes the approaches taken by independent evaluations of private sector operations. These need to address the many perceived gaps that exist today in self-evaluations. Currently, self-evaluations are conducted by decision makers and operators who may be incentivized to influence shareholders and minimize costs to their organisation. The current self-evaluation models in IFI private sector operations have elicited particular skepticism from public opinion and are often heavy on measuring inputs and light on measuring outcomes. Self-evaluations are being seen in some quarters as obligatory box-ticking, specifically in private-sector focused evaluations, rather than providing information of the true value added by the IFI and its project sponsors. For example, a cursory review of IFIs’ independent evaluations that studied the COVID-19 pandemic response focused on the private sector suggest that the IFIs struggled to target their response or operate in a coherent way. There was no good ‘crisis playbook’ at hand. Of course, such messages may not be easy to digest by the operators in the authorizing environment. Nor can it be easily reconciled with the large volume of ‘crisis lending’ that flowed to developing economies during the pandemic. Given the complexity of the causal chains involved, these often cannot be fully traced to their intended beneficiaries (Devex, Brookings, IMF, 2022). Independent evaluations are still relevant for IFIs and indeed they need to play a greater role in facilitating the rethinking of objectives, and goals, and adjusting business models, and identifying necessary reforms (ADB, 2022).

Perceived marginalization of the independent evaluation function

This takes us to the second issue of potential marginalization of independent evaluations by the authorizing environment, and in some cases, the perception of management capture. Independent evaluations do not facilitate improved decision-making if they fail to feed into strategy and policy formulation, to amplify the voice of legitimate stakeholders, and/ or to provide credible and reliable performance information to management and higher governance authorities (IMF, 1990). In particular, independent evaluation needs to leverage its impact through appropriate links to self-evaluation processes, including judicious protocols relating to professional interaction. Achieving and maintaining such relationships calls for careful institutional design of organizational structures and business processes. Finally, independent evaluators need to be adequately protected to deliver high quality, uncompromising reports and so must be shielded from capture and intimidation. For evaluation to be genuinely independent, all the essential enabling conditions—structural, behavioral, protection from external influences, and avoidance of conflict of interest—need to be in place. The enabling conditions for independent and credible evaluations have been declining in recent years at the IFIs, partly due to structural problems and partly due to incentive-related problems (Economist, 2022). For example, in some IFIs, managers of the independent evaluation function are frequently rotated out or phased out in the name of talent development, reducing institutional memory and the credibility of outputs. In other cases, new managers with no evaluation experiences are brought in to run the independent evaluation function, potentially leading to misalignment of incentives and creating new structural weaknesses. In some cases, the new evaluation managers may be further incentivized to “agree” or build consensus with management more than required to avoid the impression of over-reach. For example, the annual performance of IFI evaluation managers is determined by the 360-degree feedback[1] from the evaluands and their agents, and direct reports. Within this context, the performance of the managers is likely to be less motivated and incentivized by the quality of the evaluation or organizational learning. The evaluation managers’ performance is likely to be more motivated by bureaucratic processes, such as the level of engagement with the evaluands’ agents; processes, that the authors contest, add limited value to the evaluation quality and increase transaction costs. Such processes may further limit evaluation managers’ objectivity and create new disincentives. Similarly, the ‘revolving door’ approaches that exist within many IFIs, i.e., rotating evaluands into evaluation positions and vice-versa may impair the new evaluation managers’ judgement and create vested interests in the process. To date, no credible analysis (or human resources-oriented evaluation) exists within the IFIs to suggest that such revolving door policies have led to improvements in meaningful IFI activities (e.g., addressing development challenges at scale) or improved organizational learning. Further, the authors submit, the contestability of independent evaluation findings and principled, respectful disagreements about recommendations can make for a healthy organizational culture. Deep adversarial attitudes and related approaches rupture contacts with decision makers, restrict access to tacit knowledge, inhibit professional exchanges, and increase resistance to the adoption of evaluation recommendations and reduce organizational learning function. This is more of a problem in IFI private sector evaluations where decisions must be made in a time- and cost- efficient manner due to the underlying principles of private sector firms’ going concern, and the IFIs’ fiduciary responsibility (these challenges don’t typically exist in public sector interventions). Such adversarial approaches often lead to isolation and have a chilling effect on organizational learning.

New providers and the enabling external environment

The third issue relates to the external environment that can marginalize the independent evaluation function of IFIs, through the entry of new players in the impact measurement and impact assessment space. This is more of a challenge for private sector evaluations. Many new entrants to the impact measurement space, both from OECD countries and middle-income countries have emerged in recent years, spinning off from think tanks, private consulting firms, and data service providers. These entrants have added new business lines to their organization structure to cater to their clients’ activities in emerging markets and developing economies including in low-income countries (Rockefeller Foundation, 2015). Further, niche data and market intelligence service providers have built up their capabilities using new technologies (e.g., geospatial, artificial intelligence, social media analysis) and new ways of narrative building (e.g., emotional storytelling, short videos). In the case of climate change, the authors estimate that at least 30 high-quality, data service providers can provide the same services that evaluators can provide. These service providers have the necessary and sufficient reach (global), access (high quality data of beneficiaries), and authorizing environment (government-linked investors or foundations) to undertake large, high-quality evaluations. New entrants, such as consulting firms, are fully capable of developing sophisticated results frameworks for private sector interventions and provide the necessary tracking mechanisms as a service to investors in a time- and cost- efficient manner. Given the differences in authorizing environment, the new entrants are able to provide objective, independent views on results achieved. However, the new entrants challenge the IFIs and can marginalize the independent evaluators of IFIs in two ways: (a) connect directly with policy makers within the Governments, and decision makers of private sector firms and propose ways to improve ranking (e.g., Environment, Social and Governance), and improve the narrative (e.g., outputs presented as outcomes); or b) provide independent verification of outcomes (e.g., assess the extent to which a client has implemented policies and processes to execute on impact strategies) to help them report results (e.g., assess the extent to which a clients’ reporting of its impact performance is complete and reliable). However, in some circumstances, the activities of the new entrants may inadvertently, or not, lead to “Impact Washing”. Impact washing isn’t illegal. This is because there aren’t any public standards or laws currently governing sustainable investments. There are private governance measures in place, but they aren’t required for something to be identified as (for example) a sustainable or green investment. Some firms take advantage of this lack of legal governance to exploit the growing trend to label stocks, bonds, or other assets as sustainable or green without ensuring they will be used for such purposes. Many of the large firms in Emerging Markets and Developing Economies (EMDEs) also happen to be clients of IFIs. At some point in the near future, the authors submit, these large firms or clients are likely to start reporting SDG- linked outcomes and impacts achieved using bespoke self-evaluations. Such bespoke self-evaluations are likely to be filled with bespoke indicators and verified by a third-party validator chosen by the firm for a fee (for example, a consulting firm providing impact verification services). The outcomes and impacts reported by the firms are likely to be at-odds with the self-evaluations of IFIs sponsoring the same project since there are no globally accepted standards or frameworks for reporting on sustainable, private sector development today. (Within the IFIs, the OECD-DAC offers some guidance on private sector evaluation frameworks). Given the current incentives structures set up at IFIs, it would be more time- and cost- efficient for the IFIs, in the future, to lean on or rely on the client firms’ bespoke validations provided by the third-party, and reproduce the results achieved than conducting the IFIs’ self-evaluations. This can lead to significant principal-agent problems.

Concluding thoughts; three threats, three risks, three recommendations

In conclusion, the threat to the independent evaluation function within IFIs has three dimensions:

1. The authorizing and enabling environment of the independent evaluation function within IFIs is too weak to consistently support independent, credible and useful evaluations
2. The independent evaluation function risks becoming isolated from management decision-making
3. External providers lack perceived and real independence.

This carries three institutional risks:

1. IFIs are insufficiently learning organisations and so lose agility and responsiveness
2. IFIs become less accountable, and decision-making is insufficiently evidence based and forward-looking resulting in a long-term erosion of legitimacy and trust
3. Evaluative evidence is framed by external providers in ways that reinforce risks 1&2

This requires three responses from IFIs who should

1. Review their evaluation systems with a view to ensuring and authorizing their independence and their embeddedness in organizational learning and decision making
2. Reconnect accountability arrangements to evaluative evidence
3. Ensure that the market for new providers of evaluations is structured in such a way as to strengthen, not undermine, points 1&2

 

References

Bhattacharya, Kharas et al., The New Global Agenda and the Future of the Multilateral Development Bank System, February 2018. Bhattacharya, Kharas. Time to reform the multilateral development bank system., Brookings., February 2018. Boosting MDBs’ investing capacity. (2022). An Independent Review of Multilateral Development Banks’ Capital Adequacy Frameworks Chadwick. V, The big moment to reform development finance, Devex, September 2022 Good Governance: The IMF’s Role., 1990 Kenneth J. Arrow. The Limits of Organization. Pp. 86. New York: W.W. Norton, 1974 Organizational Review: A New Operating Model to Accelerate ADB’s Transformation Toward Strategy 2030 and Beyond, Asian Development Bank (ADB), October 2022 Picciotto. R, Lamdany, Edison et al., Independent Evaluation at the IMF: The First Decade, Chapter 4., 2014. Picciotto. R, The Institutional Role of Independent Evaluation, 2004. Rob van den Berg, Dennis Bours et al., What should evaluation learn from COP 26? Views of evaluation practitioners, February 2022. Schwartz. M, Rist et al., The International Monetary Fund and the Learning Organization: The Role of Independent Evaluation., 2016. Too important to fail: Is weak governance harming the African Development Bank? Economist, May 2022. Unlocking Capital, Activating a Movement: The Rockefeller Foundation’s Impact Investing Initiative, 2015. [1] 360 Degree Feedback is a human-resource management system or process in which employees receive confidential, anonymous feedback from the people who work around them. This typically includes the employee’s managers, peers, and direct reports.