Search on EES

1.  Could you please introduce yourself and your specific areas of interest? (If relevant: and who will you be presenting this workshop with?)

SCOTT:  I have been working in international development for over 25 years, and evaluation for about 19 of those years. I spent over a decade as a Senior M&E advisor with the international Red Cross Red Crescent, and more recently I was the Director of Evidence, Measurement, and Evaluation for the climate change portfolio with the Children’s Investment Fund Foundation. I am currently a freelance consultant specializing in evaluation and strategy development, and I will be co-facilitating my session with Silva Ferretti.

Evaluation & Covid.

2. The theme for this year’s EES conference is ‘Actions and Shifting Paradigms for Challenging times’. The Covid-19 pandemic has significantly disrupted our ways of working and thinking as evaluators. What would you say were the biggest challenges you faced as an evaluator in the past two years?

SCOTT: Firstly, after working for organizations fulltime for over 15 years, I shifted to working independently, as well as moving with my family to another country. So, with all this change, there was a lot of challenge to adjust. But also rewards. I like working independently with the ability to be more selective in how and with whom I devote my resources.

Secondly, the shift to remote reality has presented inherent challenges. It is challenging to get to “know the place” when interfacing with people remotely, with little face2face interactions and direct observation of the evaluand.

SILVA: the main challenge is the lack of contact with reality. I always try to talk first and foremost with the protagonist of change, with the people on location. Usually you travel to a location, you get in touch with local partners of large INGOs, and, through them, to local actors, community leaders and a range of people experiencing and driving change.  You can then hear/share a plurality of voices. This was not possible remotely. Within remote evaluations it was often already hard to reach local partner staff! With some creativity sometimes you can reach a bit further (for example, a field worker made a participatory video with some of the kids he supported). But this was the exception, not the rule. So, I always had to be very clear about the limitations of this approach.

3.  … and how would you say the pandemic has incited you or further encouraged you to envision shifting approaches in evaluation?

SCOTT: The pandemic was as if Mother Nature said to humanity, “Okay, enough is enough. Everyone needs to take a time out and consider the consequences of your actions before it’s too late and I get really mad!” As the latest (April) report from the Intergovernmental Panel on Climate Change (IPCC) underscores, the science points to a near-term collapse in society if we do not urgently address the climate emergency. (278 scientists from 65 countries contributed to the report). In other words, we need to shape up or we will not be around as a species next Century, at least in tolerable, “civilized” existence that allows for ideals like equity and justice and basic human rights like education and health care.

The pandemic did not so much “awaken” me to this realization. Trained as a geographer with a focus on cultural and political ecology, I have for years been very aware of the limits of growth, and challenged the conventional perception of “development,” which is often a facade for the exploitation of human and natural resources. But the pandemic provided me the space to focus on evaluation at the intersection of the transformational. This was largely put into motion by the opportunity to lead author a chapter (in The Palgrave Handbook of Learning for Transformation) on evaluation’s potential to contribute to the transformational change.

SILVA: people were talking of a new normal, but it did not really materialize! The pandemic should have prompted more complex ways to look at reality…  adaptation, transformation in the face of new challenges. It was an immense missed opportunity. The big push was for restauration: pretending that nothing was happening, re-establishing the old normal. Of course many people are calling for new approaches, but, really, the pandemic did not really help change. On the contrary, it showed how resilient the old resistances to change are.

4.  Embracing change is never easy…. what piece of advice would you offer to those seeking to apply this new ‘paradigm shift’?

SCOTT: Business-as-usual will no longer float this spaceship Earth and we need a radical shift beyond just thinking into urgent action. Navigating this shift will be new territory, because we need to be more comfortable with uncertainty and acknowledge that change does not occur according to predetermined results identified in a ToC, logic model, or some other conceptual framework.

A good companion on this journey is systems thinking, that does not isolate or fixate on one intervention or agenda but connects the dots and stresses the interconnections and dependencies between phenomena. For example, sometimes I see people working on social justice shouting over those working on environmental sustainability or climate change. However, the destruction of the environment is integrally interlinked with social challenges, and to a large degree is a meta-driver exacerbating these problems. Environmental degradation stresses the very ecosystems that supports life on the planet, affecting the habitability and natural resource base needed to sustain humanity. In other words, environmental problems contribute to social problems, with the poor, marginalized, and most vulnerable bearing a disproportionate share of the burden. That is where environmental injustice collides with social injustice.

SILVA: it is becoming clearer that a paradigm shift is needed and urgent. Yet, look at how we are dealing with climate change: we should change our lifestyle, but nothing is happening. We need to build this sense of urgency. As evaluators, we cannot waste time looking at change in ways that do not really challenge the status quo. We have a moral duty to push new perspectives.

PDW & Conference

5. Speaking more specifically about (PDW topic), what prompted you to propose a workshop on this particular topic?

SCOTT: At the last (2021) EES and AEA conferences, I delivered a PDW titled, Evaluating Outside the Box – Evaluation and Transformation for the 21st Century. The workshop explored how evaluation, a profession in the business of assessment and advising, can inform and hasten transformation. The workshop this year will continue to do that, but it will zoom in on a particular area of evaluation that I felt is ripe to be reconsider: the separation of ‘evaluation” from “monitoring” in concept and practice.

As we can see all around us, our world is rapidly changing. For this reason, I always felt the concept of a “new norm” after the pandemic hit the fan was an oxymoron because it implies a new equilibrium where life will resume a degree of regularity. On the contrary, the “new norm” will be increasing frequency and magnitude of disruption. Given this, we need nimbler, alert, adaptive evaluation that supports real-time learning and course correction. This piggybacks on monitoring to support real time evaluative thinking, learning, and communication in what we call Monitoring as Evaluation.

Another inspiration for proposing this workshop was the opportunity to work with Silva, as we are both passionate about the topic. Cofacilitating  draws broader experience and perspective, style, and approach…not just for the learning of the workshop participants, but for Silva and I to nudge and learn from each other.

SILVA: If we already had the solutions for the problems we are confronting, life would be easy: let’s make a plan, let’s follow it. Then conventional M&E would be good enough: we could just occasionally report on adherence to the plan. But there are not-ready-made solutions for the challenges we are facing. We need to experiment, adapt, improve. This requires constantly asking ourselves: how are we doing? How could we do better? We need capacity to track change as it happens, and trigger thinking when needed, involving diverse actors of change. This is not how M&E works now.  Evaluations are not an afterthought. We need to acquire capacity to scan for change real time, and to constantly improve our solutions.

6. Could you briefly explain what attendants can expect from your professional development workshop?

SCOTT: As the witticism says, “The mind is like a parachute: it works best when it is open.” So come with an open mind, and given the professional background of attendees, we want to draw upon their experience and knowledge to support peer-to-peer and emergent learning as we navigate topics together.  We will cover the basics of real-time evaluation (RTE), which largely emerged out of the humanitarian sector, but then look at why it is receiving increasing attention as a complexity-friendly and systems-savvy method outside humanitarian contexts. We will then go beyond RTEs and their conventional conceptualization and practice to explore other  approaches that can support real-time evaluation, learning and adaptive management.

SILVA: Attendants can challenge the usual way of thinking about change and, hopefully!, get to some eureka! Moments, vividly showing why we need to think at change differently. We want them to understand that looking at the world with complexity lenses is not yet another approach, but rather thinking shift. And once you understand and experience it, the challenges of the current way of doing M&E become obvious.  We want to excite people about new possibilities. And have fun in doing so.

7. What other topics to be discussed at the 2022 EES Conference have captured your attention?

SCOTT: I am very pleased to see the inclusion of PDWs that embrace complex systems analysis. This includes Blue Marble Evaluation, Footprint Evaluation, Outcome Harvesting, Community-led, M&E, storytelling, and so much more.


8. What future role do you see for young evaluators in this new post-pandemic world?

SCOTT: YEEs are front and center, and have been a priority for me in the last few years. It is their future that we are seeking to transform towards, so their engagement, voice, and action are essential. Last year (about this time) I organized a virtual session with 7 YEEs for the gLOCAL Evaluation Week, and then another session for the American Evaluation Associate with YEEs that was selected for its Presidential Strand. It was a completely inspiring experience working with these YEEs, and I used to introduce them not so much as “young and emerging evaluators,” but “young, accomplished, and inspiring evaluators.”

SILVA: hard to say. The post pandemic world is terribly similar to the pre-pandemic one, just with more challenges and troubles. And we are going towards breaking point. I do not believe that evaluation, as it is, will be relevant as to whether this will happen. Other outlets will be.  Capacity to evaluate within movements – rather than organizations – will be crucial. Hybrid forms of research action will be needed. Capacity to share evidence and adjust practices of resistance across diverse stakeholders will be necessary. Young evaluators might bring in the needed fluidity and creativity in evaluation processes.

9. If you were to recommend this career path, what advice would you give young and emerging evaluators?

SCOTT: First off, refer to my response to Question #4 above. In addition, I would add that while there is an “urgency” in the world today, and people like me may “preach” for action, it is important that everyone finds their own path to “put skin in the game.” By this, I mean that they should find a way to engage that is meaningful and rewarding for them, that they enjoy, which will allow them to sustain their effort, and when people find their work fulfilling they typically make a bigger difference.

SILVA: be yourself. Use your skills, be creative in doing so. There is not one way to do evaluations. There are tons of approaches, there is a need to innovate. Also, do not be afraid to challenge things. The evaluation field is so conservative and technocratic! I cannot believe that we still stick to logframe thinking (which is a tool created by the army in the 70s… half a century ago!). We cannot just accept the status quo, the way in which we manage change is pushing us to dead ends. If evaluators do not challenge the way we think about change, who can then do so?

10. Looking back at your own career and lessons learnt from your own experience, are there any pitfalls/common mistakes young evaluators should avoid?

SCOTT: Think outside the box. Don’t just focus on evaluation, but expose yourself to and explore multi-disciplines and other worldviews (e.g., indigenous, feminist, “South,” etc.) that will challenge and disrupt yourself to look above the weeds and see and help save the forest while some of it is still standing.

SILVA: I cannot recall any spectacular failure, but I can share what kept me going: never stop challenging yourself. Always try to learn something new, whatever you are doing. Never miss on the the fun, on the joy of discovering and interacting with people generating change. And never forget that they might call you to judge change… But making it happen is a much harder job. Stay humble!

Scott Chaplowe & Silva Ferretti