Fishbowl Session, EES 2022, Copenhagen.  Thematic Working Group 2: Sustainability

By Leon Hermans, Wolfgang Meyer, Jindra Čekan, Ferko Bodnar, Carroll Patterson

 

Introduction

The last few years have seen a continued interest in evaluation of sustainable development, triggered by topics such as the Sustainable Development Goals (SDGs), climate change and adaptation. One of the classic complications in sustainability is dealing with short-term-long-term dilemmas. Sustainability evaluations need to adopt a time horizon that accommodates both short-term interventions and long-term ambitions, ensuring that the needs of future generations are addressed while responding to current short-term needs and decision-making requirements.

Learning about progress towards the SDGs, climate adaptation or the daunting task of keeping global temperature below 2 degrees Celsius above pre-industrial levels, requires us to move beyond traditional program evaluations and sectoral silos. At the EES Conference in Copenhagen these challenges were discussed, in a session hosted by the Thematic Working Group on Sustainability, to contribute to renewed debate, reflection and, ultimately, improvement.

Dimensions of Sustainable Development

Sustainable development uses System Thinking, considering systems and their main functions that would need to be maintained over time, as opposed to thinking in terms of independent sectors and isolated sectoral goals. Consequently, sustainable development engages with complexity, which in turn is also reflected in evaluation activities and methods. This is different from more reductionist approaches that seek to reduce complexity using segmentation and carved-out experimental designs. Sustainable development also looks at the interrelations between the factors in complex systems, and at the externalities that emerge from the interactions between those factors. This means evaluations would also need to look beyond only the intended or pre-designed causal pathways.

And of course, sustainable development is about linking short- and long-term objectives, which link current and future generations. This has in recent years given prominence to the concepts of transitions and resilience. These are multi-level phenomena, requiring linkages across levels, and vertical and horizontal integration, and an eye for cumulative results.

Very directly, one can see how these dimensions of sustainable development challenge some of the evaluation approaches and expectations that have evolved in the past decades. For instance, in a fund such as the Global Environment Facility, the four-year funding cycles require support from evaluation activities, but these evaluations cover projects that may only show some visible effects after at least ten years.

Delivering sustainable development is something that can most clearly be tested by evaluating projects ex-post, starting as early as two years ex-post project closure, up to 30 years. Currently, at the Adaptation Fund, we are evaluating climate projects that are at least 4 years of duration for sustained and closed for at least two years for the durability of outcomes and resilience to climate change. The lessons from these evaluations of assets are compelling for testing assumptions, and ratings and to feedback lessons into future funding, design, M&E learning, and knowledge management. Closing the learning loop involves participants and partners co-creating these evaluations with us.

National Monitoring and Evaluation Systems

For evaluation, these dimensions of sustainable development, and the insights from systems, transitions and resilience thinking, make it difficult to unambiguously measure impact. However, it may be possible to outline some contours of an “ideal-type” monitoring and evaluation system, which could help, for instance, to monitor national progress towards the SDGs.  Such monitoring and evaluation would look at the functionality of systems, identify the logic of required change, system needs, and leverage points, then try and deduce inputs from impacts. Evaluations within such designs would need to feature appropriate analytical methods, such as simulation methods and adaptive management analysis. In some countries, evaluation systems along these principles can be discerned, such as Nepal, Finland and South Africa.

At the same time, although the logic for such systems is convincing, the complex realities that characterize sustainability issues result in many caveats, limitations and challenges. One of those is ‘ownership’ of these systems, and who should drive their development and guard their use as integrative tools for different sustainability interventions and policies.

The multi-level nature of sustainability means that there are many international agreements, conventions and funds that require or support sustainability interventions. Ideally, national M&E systems would play a role in connecting different interventions, such as donor support for flood protection activities in small island development states. These systems do not seem to exist in many countries, although the multi-donor-government social protection programme in Rwanda was mentioned as a positive exception.

This need for more integrative national M&E systems is visible in relation to low-income countries and international donor agencies, but also in high-income countries with a need to coordinate activities that relate to sustainable development goals. Many countries seem to work from a collection of existing sectoral systems and statistics, rather than from integrated systems designed to fit sustainability thinking.

Policy coherence for sustainability

The push for more integrated approaches and a shift from a donor and project perspective to a national programme perspective have given more attention to policy coherence in evaluation. Internal coherence between different interventions of the same actor, external coherence between all actors intervening in the same area, coherence with long-term strategies, and coherence between local and global ambitions, turn out to be preconditions for sustainable system change. There are a growing number of efforts to move beyond a short-term project focus in evaluations, and towards a focus on national development and policy coherence. This also links to a trend to focus more on programmatic approaches, beyond more specific projects, for instance in deltas protected against floods and climate change in Bangladesh and Mozambique.

This seems a much needed and useful development. Yet at the same time, the larger story around policy coherence is also sobering. IMF figures show that the expenditures on fossil fuel subsidies are several orders of magnitudes higher than the climate finance investments – trillions versus billions of dollars annually.

Sustainability in highly volatile settings

Another type of challenge for sustainability evaluations are the highly dynamic, volatile and unstable settings that prevail in many parts of the world. What does sustainable development mean in countries such as South Sudan, Liberia and other fragile (post) conflict societies? In these settings, the notion of transition and transformation pathways may offer some useful starting points. Can we stay on a path of transition towards more stability or sustainability? For this, indicators can probably be developed but currently appear to be lacking. But if they are to be developed, who should define these indicators: are local people also being involved? The donor or international level notions of sustainability transitions may be different from those of local people. Even to the extent that for some, donor-driven monitoring is considered to be part of the problem, not the solution.

Towards learning systems and closing the loop

This brings about a last, key aspect, of sustainability evaluations. Eventually, more than being merely tightly designed and administered, sustainability evaluation systems should be learning systems. One critical element in such learning systems is to close the learning loops. Very often in current evaluations, local people may be involved in more participatory evaluation activities, but we hardly ever return to these local people with the evaluation findings. We don’t learn because we don’t return.

This is one illustration of how, directly and indirectly, relationships and power dynamics play a role, which is equally as true for sustainable development more broadly. The inclusive thinking that is at the heart of sustainability does not automatically come about, and is not always supported in existing evaluation practices. This is not easy to do but a change of perspective, moving away from perspectives of interventions as win/lose events, towards an understanding of longer-term “infinite” games among the stakeholders, focusing on underlying drivers and fundamental factors in systems and in transition trajectories, and doing more to include the relations and power dynamics as part of these, with methods and approaches available within and beyond the evaluation community, do offer hope.

In summary, the discussions in Copenhagen once more confirmed the difficulty and the critical importance of better sustainability evaluations. There are clearly many unresolved challenges, but also many useful starting points, ideas and approaches that can help us to move towards a better future. Within the Thematic Working Group we hope to continue the discussion – and welcome interested EES members to join!

 

Acknowledgements

The Fishbowl was organized as part of the EES2022 Biennial Symposium in Copenhagen, Denmark, and was attended by some 25 participants. As fishbowl organizers and rapporteurs, we want to thank them all for the lively discussion, especially the colleagues who contributed to the discussion “inside” the fishbowl: Nana Davies, Jonathan Stark, Carl Bruch, Daniella Ullemia, Juha Uitto, Zazie Tolmer.