By TWG8 Co-leaders Barbara Schmidt-Abbey and Kirsten Bording Collins

 

The newly established EES Thematic Working Group 8 ‘Systems Approaches in Evaluation’ (TWG8) was honoured to host an online debate between two giants in their respective fields of evaluation practice and systems sciences, Michael Q. Patton and Mike C. Jackson, as TWG8’s public launch event on 27 February 2023. The recording of this debate is available here: https://www.youtube.com/watch?v=lnyfpC8E4O8

The debate, titled ‘In Search of a Golden Mean to Systemic Evaluation’, emerged out of a critical discussion between Michael Q. Patton and Mike C. Jackson.

The controversy was triggered by critical comments by Prof. Mike C. Jackson in his recent article series about Critical Systems Practice, notably part 4 of this series (Jackson, 2022b), in which Jackson criticised the use of the systems concepts in evaluation as presented in the SETIG principles developed by the AEA Systems TIG (SETIG, 2018): interrelationships, perspectives, and boundaries. In his article, Jackson found these concepts to be of ‘limited use to evaluation’ and as ‘philosophically untenable’ and lacking theoretical justification. As an alternative, Jackson proposed the use of Critical Systems Practice (CSP).

Michael Q. Patton responded to Jackson’s critique in an hour long YouTube video in December 2022 (Patton, 2022), followed by a Letter to the Editor (Patton, 2023) of the Journal (Systems Research and Behavioural Science) which published Jackson’s article.  In this Letter to the Editor, Patton defends the SETIG systems concepts and has the following key responses to Jackson’s critique:

• Patton recalls the origins of the SETIG systems concepts from concerted efforts by systems thinkers and evaluation practitioners going back to 2007 (Williams and Imam (2007). This was an effort to find crosscutting patterns and principles in the great variety of systems concepts, and make them practically useful for application in evaluation practice thereby basing them on pragmatism and grounded theory.
• The systems concepts may be pragmatically incorporated into evaluation frameworks that are not entirely systems oriented and may thereby be considered ‘minimum specifications’ for applying a systems approach. This makes them especially useful for many real-world evaluation contexts that are frequently constrained in terms of time and resources.
• Another strength of the systems concepts is that they may be understood by ordinary people without any experience in the systems traditions and approaches.

 

Patton in turn criticises Critical Systems Practice as proposed by Jackson as being too complicated, comprehensive and demanding to be useful for most real-world evaluations and considers CSP as an essentially linear and reductionist approach.

Michael Q. Patton’s letter in SRBS was followed by Mike C. Jackson’s response letter in the same journal, reacting to Patton’s counter-critique. In his response, Jackson states:-

• that other systems thinkers have come to different conclusions about the ‘core systems components’ than the ones described in the SETIG principles, going back for example to Checkland (1981).
• He rejects Patton’s claim of the SETIG principles as ‘minimum specifications’ for systemic evaluation as not being sufficient, saying that we need to “provide some clear direction”.
• Jackson maintains that CSP represents an ‘ideal-type’ of multi-methodological systemic intervention, and could be considered as a ‘maximum specification’ for systemic evaluation.

Jackson concludes by proposing that a possible “golden mean to systemic evaluation” may be found somewhere along the continuum between the ‘minimum specifications’ of the systems concepts approach on the one end, and the ‘maximum specifications’ of CSP on the other, thereby coining the title phrase for the ensuing debate.

The online debate on 27th February 2023 constituted the first time the two discussants Jackson and Patton exchanged their contentions and arguments directly face-to-face with each other in a structured conversation, following a classic debate format. In three rounds of exchanges, the opponents Jackson and Patton each laid out their respective perspectives and arguments. In the first round, Jackson summarised his initial critique of the systems approaches and principles as laid out in his original article and his response, and the background and reasoning for his criticism. Patton, in turn, explained the foundations of the SETIG principles and why they have proven so useful in evaluation practice.

In the second round, both discussants outlined their respective criticisms of the other’s position: Jackson’s critique of systems concepts in evaluation from a systems science point of view, and Patton’s critique of CSP from an evaluation practitioner perspective.

In the final round, both speakers engaged in an exploration of possible common ground, and their respective areas of agreement and disagreement.

In the ensuing question and answer session, drawing on pre-curated questions from members of TWG8 and live audience questions collected during the debate, the attention shifted to the relationship between the respective fields of evaluation and system sciences, which both speakers agreed were currently underdeveloped. There was a shared recognition by both that benefits could come from closer knowledge and appreciation of the others’ contributions. Jackson appreciated that there are evaluation principles and concerns that can be of benefit to the systems field. Patton appealed to the evaluation community needing to rise to the challenges faced in a period of poly-crises and climate change, which require us to move beyond the limitations of linear and reductionist programme and projects mentality.  This also requires a deep understanding of systems, and dealing with systems change. Evaluators need to get up to speed to learn to think and act systemically and need to be able to engage with systems thinkers like Jackson, who can help evaluation practitioners to look at how systems are evaluated and transformed.

In conclusion, this debate has seen a shift of engagement between the two respectively held positions by the discussants, moving initially from quite pointed and entrenched critiques of each of the other’s preferred approach and methodology, towards points of agreement.

The discussion exposed clear differences between the two opposing positions on several important points, but also exposed an emerging mutual appreciation of the value and contribution each field can bring to the other.  The benefits of a closer relationship between these two neighbouring meta-disciplines and practices can bring to each other, and the commonly held concerns. It also showed the importance and value of deeper engagements between systems-informed evaluation practice – as represented by Michael Q. Patton – and systems science academia – as represented by Mike C. Jackson, which benefits the development of both.

At the close of the debate there was a shared sense that this discussion was at a beginning rather than an end point, and there is a need to continue these important exchanges and further build on them.

EES TWG8 on ‘systems approaches in evaluation’ is a dedicated and committed platform to continue working on the important issues and concerns raised by this debate. TWG8  encompasses the full range of diverse systems approaches from all traditions of understanding of systems, to promote their development, use and further adoption in evaluation.

Further activities and developments are planned in the work plan the TWG is currently developing, which will include further panel discussions and exchanges. So watch this space, and we look forward to any suggestions, feedbacks and contributions to this important continuing debate!

References:

Checkland, P. (1981)  Systems thinking, systems practice. Wiley.

Jackson, M. C. (2020). Critical systems practice 1: Explore—Starting a multimethodological intervention. Systems Research and Behavioral Science, 37, 839–858. https://doi.org/10.1002/sres.2746

Jackson, M. C. (2021). Critical systems practice 2: Produce— Constructing a multimethodological intervention strategy. Systems Research and Behavioral Science, 38, 594–609. https://doi.org/10.1002/sres.2809

Jackson, M. C. (2022a). Critical systems practice 3: Intervene— Flexibly executing a multimethodological intervention. Systems Research and Behavioral Science, 39, 1014–1023. https://doi.org/10.1002/sres.2909

Jackson, M. C. (2022b). Critical systems practice 4: Check— Evaluating and reflecting on a multimethodological intervention. Systems Research and Behavioral Science, 39, 1014–1023. https://doi.org/10.1002/sres.2912

Jackson, M. C. (2023). In search of a golden mean for systemic evaluation: A response to Michael Quinn Patton. Systems Research and Behavioral Science, 40, 1092-7026. https://doi.org/10.1002/sres.2938

Patton, M. Q. (2022). Using systems concepts for evaluation: debate and dialogue. YouTube video: https://youtu.be/ZNgEtjAAizE

Patton, M. Q. (2023). Letter to the editor. Systems Research and Behavioral Science, 40, 1-3. https://doi.org/10.1002/sres.2927

SETIG (2018). Principles for Effective Use of Systems Thinking in Evaluation. SETIG-Principles-FINAL-DRAFT-2018-9-9.pdf (betterevaluation.org)

Williams, B. and Imam, I. (eds) (2007) Systems concepts in evaluation, an expert anthology. American Evaluation Association. http://www.bobwilliams.co.nz/ewExternalFiles/SYSTEMS%20CONCEPTS%20IN%20EVALUATION%20Handout.pdf