What are you currently working on?

One of my activities concerns the book I am doing together with Sanjeev Sridharan (U of Hawaii) and with authors including Ray Pawson, Tom Ling, Patricia Rogers, Valerie Pattyn and others, titled From Silos to Ecologies of Evidence (Comparative Policy Series, Routledge, in press (2026).

An ecology of evidence refers to the interrelationships among different types of evidence, the interventions to which they relate, their underlying mechanisms, and the contexts in which those interventions operate. Put simply: rather than asking “which single piece of evidence is definitive?”, an ecology of evidence asks “how do different streams of knowledge, gathered through different means and from different people, come together to help us understand what is happening and why?” It explores how recognizing patterns across these streams deepens our understanding of what interventions do, for whom, under what conditions and in combination with other interventions.  

In one of my chapters I present evidence of the –indeed—booming business of evaluation in many countries over the last 50-60 years. Another chapter discusses that the very “success” of the evaluation business has gone hand in hand with the production of dark sides and serious challenges that may have made a genuine ecology of evidence harder, not easier, to achieve.  Some of these challenges are these:

• Institutionalization as a barrier to an ecology of evidence. The institutionalization of evaluation within governments, quango’s, development banks, and international organizations has created powerful incentives toward compliance, routine, and reputational risk management — what Dahler-Larsen calls “evaluation machines.” These bureaucratic approaches are structurally misaligned with the adaptive, pluralistic, and context-sensitive demands of an ecology of evidence. And they contribute to the danger of evaluation capture.  

• The orchestration of evaluation independence: as doing ‘independent evaluations’ is close to a kind of holy grail, orchestrated independence concerns situations where evaluations appear independent on the surface but are (or can be) actually subtly controlled or influenced by the commissioning and/or sponsoring organizations, stakeholders, management of evaluation and politicians and bureaucrats. Orchestrated independence is also fundamentally different from behavioural independence.
• Growth without diversity. I argue that the explosive growth in evaluation over sixty years — from a handful of studies in the 1960s to tens of thousands of reports annually — has not produced a corresponding growth in methodological diversity. Instead, it has produced more of the same: homogenized approaches dominated by experimental designs, logic models, and standardized evaluation systems. Growth, in other words, has reinforced hierarchy rather than ecology. This despite the increase in more than one hundred approaches, models and (evaluation) theories (Leeuw & Donaldson; Patton).  
• The forgotten purpose of betterment. The central provocation of the chapter is that six decades of growth in evaluation have drifted steadily away from the field’s foundational purpose: improving lives. The focus has shifted to governance, accountability, and institutional legitimation. An ecology of evidence, I argue, can only be meaningful if it reorients evaluation back toward social betterment (Mark, Henry and Julnes, Mark, Julnes and Henry)— asking not just whether programs were implemented well, but whether people and communities are genuinely better off.
• Atomization of knowledge. Rather than building cumulative, transferable understanding of how interventions work across contexts (‘consilience’, in Wilson’s terms), the booming evaluation industry has produced a landscape of disconnected, one-off studies — a cottage industry of atomized findings as Pawson calls it. 

• The political fragility of evidence ecosystems. In my chapters in the book I also note, with some urgency, that the current political environment in some countries has led to the dismantling of evaluation infrastructure, demonstrating how quickly even well-institutionalized evidence systems can collapse when political priorities shift (Lemire & Nielsen).

PS: In the book also, suggestions are given how to contribute to ecologies of evidence, including pathways how to do that. 

A second piece of work concerns algorithmization and evaluation: how does AI (agents) relate to evaluation; how can the benefits and costs of algorithmization of policies, programs, interventions for society be studied (which is more than adhering to ‘frameworks’); how applicable are our methods and theories and how will they develop when sentient (generative) AI ‘moves forward’ (or, what also might happen, if a new AI Winter might pop up). 

When were you EES President, and what are the key changes in our field and their context since then?

As far as I remember, I took over the Presidency in late 1999 and was active till 2002. 

Key changes: 

In the late 90’s/early 2000’s, the weight and importance of evaluations in governments, international organizations etc began to increase. At the same time, it was a struggle to give evaluators a ‘position’ alongside auditors, and alongside the social and economic sciences in academia. Very few professorships or senior lectureships existed for evaluation in Dutch (and European) universities. That also meant that EES had to work hard to — in my years — get a few hundred people to the bi-annual conference in Switzerland/Lausanne that I chaired. 

Now that has fundamentally changed. More participants, more orchestration, the influence of harmonization, evaluation machines, compliance and higher costs of participating in the EES bi-annual congresses. Compared to the 1970’s and 80s, evaluation now is very much more institutionalized in terms of regulations, TORs, governance documents, mostly if not all related to governments, quango’s and  qongo’s (Quasi Official Non-Governmental Organization). 

Personally, I also experience the negative impact of some developments, like woke thinking / language, the almost disappearance of hot, fierce and critical debates if not battles between evaluators on methods, theories, relevance etc. (related to the fear of being criticized as contributing to ‘unsafe environments’), as well as the very existence in (at least Western European governments) of victomocracy. 

What continues to be as challenges, are topics like these: 

• the problem of utilization of findings, 

• the problem of learning from evaluations (despite numerous speech acts to specify what ‘learning’ is or may be or should be), 

• the loss of the guiding principle that ‘social betterment’ at the level of civil society is one of the central goals of evaluation instead of contributing to governments and governance worlds 

• and how to reduce institutionalization within governments, quangos and lookalike organizations. 

The theme of the upcoming EES conference is ‘Evaluation for vibrant democracies’; what is your personal connection to this?

I can be short: if my impression is correct that ‘vibrant democracies’ addresses the role evaluation for and in (civil) society and is therefore much broader than only focusing on governments and governance, I am (very) positive about this theme. It then may stimulate to see evaluations as instruments for social betterment. 

We live in turbulent times with much change, including technological change: what do you see as the biggest challenges and opportunities for evaluation?

1. Reduce the existence of eval machines and the subsequent ‘love’ for governance, governments and (evaluation) regulations; 
2. Instead of passing out evaluation verdicts (as Pawson shows), be humble and try to take care of the Big Leap problem: how to evaluate the impact of the simultaneously operating and implemented tenths and tenths of interventions on civil society instead of addressing piece (=intervention/program) by piece (other interventions, programs etc)1; 
3. Learn from comparing different genAI agents and see their answers to prompts as ‘theories’ that have to be compared with others and with, if possible, other sources of evidence. 

What should the priorities for the EES be in the coming years?

See the points above. 

Finally: what ‘wise advice’ would you like to offer to young and emerging evaluators?

• Try to combine critical (realist) thinking and epistemology with perseverance and tenacity;
• Be aware of the Scylla and Charybdis temptations hanging around in our profession such as ‘capture’, performing speech acts telling everybody that “you are ‘independent’”, while de facto you are not; 
• Be aware of letting short-term successes prevail instead of contributing to consilience of knowledge;
• Be humble in your claims and accept skepticism.

1. Gantayat, N., Ashok, A., Manchi, P., Pierce-Messick, R., Porwal, R., & Gangaramany, A. (2024). Taking the big leap: Understanding, accessing and improving behavioural science interventions. Frontiers in Public Health, 12, 1355539. https://doi.org/10.3389/fpubh.2024.1355539