My professional interest is to bridge the disciplinary boundary between Evaluation and Complexity Science. In this effort I am not alone (1, 2, 3, 4, 5). The bridge is important because programs and outcomes exhibit complex behavior. Without appreciating this reality, one cannot design effective programs or produce evaluations that can guide those programs.
The Place of Complex Behavior in Evaluation
I do not mean to imply that all evaluation must always recognize complexity. Quite the contrary. Much evaluation can serve stakeholders well with traditional approaches to developing models, designing methodologies, and interpreting data. (A small part of the reason I believe this can be found here. I have a fuller explanation, but it’s in a paper that is currently being reviewed.) What matters is knowing when complexity should be factored into understanding programs and their consequences. (Go here for a list of some questions that I think can benefit from an understanding of complexity.)
When complexity does matter, most of its influence is on models and data interpretation. For the most part, the methodologies we are familiar with, together with newer methodologies being advocated for all evaluation, will meet our needs. (Agent-based modeling [A, B, C] and network analysis are the exceptions, but those are seldom essential to any evaluation we may do.)
Evaluators are aware of complexity, but their engagement with the topic is metaphorical. I want to help effect a shift from the metaphorical to the technical. There is a parallel with statistics. I could say “I will analyze my data with statistics.” That statement is meaningful. It conveys a belief about the world with respect to true score and error, sampling, inference, and so on. But it says nothing about what I will do. For that, I would need to say something like: “My program is designed to reduce accident rates. To find out if it works, I will use a logistic regression to analyze the mean time between accidents.” That is the kind of shift I would like to see Evaluation make with respect to complexity.
I do not want to turn evaluators into complexity scientists, but I do want to see Evaluation more adjacent to Complexity Science. (How much about complexity should evaluators know? That is an open question, but the simple answer is that a little more is better than a little less.)
“Complex” and “Complex Behavior”
If I had my way, evaluators would not use the word “complex.” I don’t like this term because when used colloquially, it has useless connotations and incorrect technical assumptions:
From the Cambridge Dictionary:
- Involving a lot of different but related parts
- Difficult to understand or find an answer to because of having many different parts
Something that is complex has many different parts and is therefore often difficult to understand.
Colloquial: Everything evaluators deal with have lots of different but related parts, so how does the label “complex” help us? As for “difficult to understand,” that is also common in our line of work. Why do we need a special word to remind us that we may have a tough time?
Technical: The definitions contain the incorrect assumption that something is difficult to understand because it has many parts. Many phenomena with many parts can be easy to understand, and many phenomena with few parts can be difficult to understand.
The confusion can be resolved if we shift our focus from “complex” to “complex behavior.” This is because the shift in vocabulary focuses our attention to specific ways in which the world behaves. Those behaviors have specific meanings, and those meanings can be used by evaluators to develop models, metrics, and data interpretations.
Drawing on Complexity Science to do Evaluation
The field of Complexity Science is broad and deep (A, B). It is diverse with respect to disciplinary roots, the interests and backgrounds of its practitioners, theories that drive research, and phenomena studied. What can we draw from this intellectual terrain that will help us do better evaluation? Different people will have different answers to this question, but here is my simple answer.
Three constructs from complexity are sufficient to get us a long way towards understanding how programs and outcomes behave. These are: emergence, sensitive dependence, and attractors. Why these three? One answer is that they work for me when I develop models or methodologies, or when I interpret data. (There is much more in complexity that is useful, but these three concepts suffice for most purposes.)
But besides what works for me personally, there is a more profound answer. “Emergence” and “sensitive dependence” hold a special place in the population of complex behaviors because they have the potential to change how evaluators think about outcomes and causal change. “Attractors” help us visualize change and resistance to change, and to see why multiple paths through a system can lead to the same outcome.
“Emergence is a process by which a system of interacting subunits acquires qualitatively new properties that cannot be understood as the simple addition of their individual contributions.” The concept has implications for program theory and for constructing and choosing measurements.
“Sensitive dependence refers to the role that the starting configuration of that system plays in determining the subsequent states of that system. When this sensitivity is high, slight changes to starting conditions will lead to significantly different conditions in the future.” This concept implies that because local change can affect the long-term trajectory of a system, familiar patterns of extended if > then links may not be appropriate portrayals of connections between program and outcome. When causal chains do not work, our inclination is to fault ourselves. If only we knew more, we could figure it out. Sensitive dependence implies that the fault is in the system, not in ourselves.
Attractor can be used in two ways. One involves the behavior of dynamical systems and plays an important role in understanding what it means for a system to display regular or chaotic behavior. For our purposes though, the focus is on “social attractors” which “define a specific subset of states that a social system may take, which corresponds to its normal behavior towards which it will naturally gravitate.” Attractors have a topography. They can be described in terms of variations in depth, boundary shape, elevation contours, and area. As such they are a useful construct to understand change, resistance to change, and why multiple paths through a system can lead to the same outcome.
Each of these complex behaviors has its own value in doing evaluation but employing them also provides two other insights. One is what we can learn from combining their implications. The other is the understanding they provide with respect to three cross-cutting themes – pattern, predictability, and how change happens.
Jonny Morell