John has been with the European Stability Mechanism (ESM)’s Legal division since 2015, and was a member of the team preparing the below-mentioned evaluation report. Prior to joining the ESM, John worked in public administration in The Hague and Paris. He is a graduate from Leiden University in the Netherlands and has recently completed an impact evaluation certification programme from ITCILO in Turin.
Introduction
The European Stability Mechanism (ESM) was created by euro area member states in 2012. The ESM maintains financial stability in the euro area by helping countries avoid and overcome financial crises. In this way it helps safeguard the euro area currency. Following two independent evaluations on the relevance and effectiveness of past country programmes in 2017 and in 2020, the ESM recently published an internally commissioned evaluation on its analytical capabilities. This blog post explains how analytical capabilities are understood, what methodologies were used for the evaluation, and how limitations were dealt with.
Analytical capabilities
The Terms of Reference (contained in the Technical appendix) defined the overarching objective of the evaluation as assessing institutional capability to produce timely and relevant analytics and use the output for the benefit of the ESM mandate in both normal and crisis times. The evaluation process was to identify key achievements, relevant strengths, and remaining relative weaknesses. The Technical appendix adds some extra flavour by framing analytical capability as the institutional capability to deploy and integrate economic and macrofinancial analysis with market intelligence in a timely manner to identify relevant financial stability vulnerabilities, flag emerging risks, and consider policy options in preparation for a role in financial assistance. The analytical capability is thus harnessed to preparedness.
One problem the evaluation team ran into is that there is no well-established theoretical framework to measure or assess capability (though some specific capability evaluations mostly on innovation existed). The team drew on evaluation literature to develop and understanding of what analytical capability is in the specific ESM context and how to organise the evaluation and the findings (see Robeyns (2017), Wu, Ramesh, and Howlett (2015), Andrews, Pritchett, and Woolcock (2017), Bridoux et al. (2017), Escamila-Solano et al. (2017), Guo et al. (2022), Lado and Wilson (1994), Washington and Mintrom (2018) and Knobbout et al. (2024). The team understood the intervention to be evaluated as an attempt to establish and put into operation a repetitive process of regular analysis (macroeconomic, financial, market intelligence) and policy support, while maintaining a high level of preparedness for crisis response, as well as a capability to prioritise. Lastly, the evaluation also introduced sustainability as an evaluation criterion to examine the vulnerability of the analytical capability, to assess which activities have persisted, and how the institution is building for the long term.
Methodology
As with its first two evaluations, the evaluation team opted for a mixed methods approach with both quantitative and qualitative methods, with an emphasis on the latter. The choice of methods was selected to capture both wide and deep sources of findings as well as a broad variety of different perspectives and stakeholders, so as to ensure the highest quality output and avoid bias to the extent possible. The following table describes the various studies and surveys conducted, as well as other data sources collected to inform the evaluation findings.
Table 1: studies, surveys and other data sources used to produce evaluation findings
| Source | Description | |
| ESM staff self-assessment | Timeframe: October–November 2023.
Purpose: Gathering staff assessments of their individual skills and analytical capability (knowledge, applied skills, and behaviour). Tool/Platform: Netigate. Questionnaire based on “Policy Skills Framework”, The Policy Project (New Zealand government, 2023), see: Policy Skills Framework. |
|
| ESM staff survey | Timeframe: November–December 2023.
Purpose: Gathering staff assessments of the ESM’s analytical capability and framework. Tool/Platform: Netigate. |
|
| ESM board survey | Timeframe: December 2023.
Purpose: Gathering Members’ assessments of the ESM’s analytical capability and framework. Tool/Platform: Netigate. |
|
| External interviews | Timeframe: September 2023–March 2024.
Purpose: Gathering external views on the ESM’s analytical capability, role, and profile from the points of view of stakeholders, peer institutions, and the scientific community. Tool/Platform: Physical and online interviews. Coding in NVivo. |
|
| Focus group discussions | Timeframe: March–June 2024.
Purpose: In-depth discussions of ESM staff self-evaluation and ESM survey results. Further gathering of information on ESM analytical tools, processes, and products. Tool/Platform: Physical and online discussions. |
|
| Document analysis | Timeframe: August 2023–September 2024.
Purpose: Theoretical and empirical background for the evaluation. In-depth analysis of the ESM’s analytical framework and its development. Tool/Platform: NVivo. |
|
| Bibliometric study | Timeframe: January–May 2024.
Purpose: The bibliometric study was conducted to assess the productivity, quality, and impact of the ESM’s research and related publication activity for the period 2013 to 2023, reflecting both the individual analytical capabilities of staff and organisational capacity. Tools/Platforms: ESM publication database, ESM Communications division database, Google Scholar, Scopus, Overton database, SciVal, and the Elsevier Report (see below). |
|
| Elsevier Report | Timeframe: January–April 2024.
Purpose: Commissioned by ESM from Elsevier to assess quantity, quality, and impact (on policy and in media) of ESM articles (2014–2023) in peer-reviewed journals. Benchmarked against peer institutions. Tool/Platform: Elsevier Scopus database, Overton database, SciVal analytical tool. The full Elsevier Report is published with the evaluation report on the ESM’s website. |
|
| Shock event studies | Timeframe: December 2022–May 2023.
Purpose: Assessment of the ESM’s ability to detect early signals of shocks and to link market intelligence to economic analysis studying five crises selected through intensity sampling: Brexit; Covid-19; energy prices; monetary policy tightening; Credit Suisse. Tool/Platform: Document analysis, internal interviews to verify information. Assessment Framework drawn from New Zealand Government Policy Project: Policy Quality Framework and its paper-scoring template. |
|
| Staff skill profiles study | Timeframe: March 2023.
Purpose: Deeper understanding of ESM hiring strategy over time, regarding skills required and levels of experience. Tool/Platform: ESM Human Resources provided post descriptions, AI supported thematic analysis of new or updated post descriptions between 2016 and the first half of 2023. |
|
| Most Significant Change survey | Timeframe: April 2024.
Purpose: To obtain the perspective of the management of the Chief Economist department on the most significant and impactful changes that contributed to the new level of capabilities relevant to the enhanced mandate. Survey respondents were also asked to specify internal and external drivers and constraints for change. Tool/Platform: Netigate. Consultation of individual management team members. |
|
| Network analysis | Timeframe: April 2024.
Purpose: A small-scale network analysis to i) establish the role of the horizontal workstream in the CE department’s manner of organising and connecting with other departments, and ii) validate the role of the Review function in the analytical workstreams. Tool/Platform: Analysis was conducted initially using the Kumu.io network analysis platform to map connections and collect metrics. In a second round, another researcher verified these interpretations running the same dataset in R and extracting basic summary statistics (in- and out-degrees). Data was collected by the team by querying Chief Economist department staff and checking distribution lists. Dataset was anonymised when on external platform. In a third stage, AI was used to query the dataset for further insights. |
|
The findings obtained were triangulated between different data sources and data sets to bolster the reliability and depth of assessments. The majority of the findings was included in the main report, while some findings which were considered too detailed were included in the Technical annex. The findings were synthesised in both chapter conclusions and the overall conclusions, and four sets of recommendations to the ESM or (senior) management were drawn up on the basis thereof.
Limitations
An evaluation with this size and scope naturally runs into limitations. Practical limitations included scope limitations, coverage of the qualitative analysis, and time and resource constraints. This led, for example, to analytics on the Chief Financial Officer department receiving less coverage in the report. The qualitative analysis was constrained by time limitations, though the number of interviewees was considered adequate overall to inform the evaluation. Also, the team was unable to visit all member countries, but this was compensated through member states’ participation in the board survey.
To limit subjective bias in the qualitative approach, the team used organisational knowledge and stakeholder analysis, and contextualised the findings. Triangulation of findings among team members working on specific topics helped to disentangle facts and perceptions. Lastly, key person risks materialised as team members left the ESM during the drafting stage, and the work came on top of regular duties. This led to considerable delays and reduced the time available for synthesis.
What was found
On substance, the evaluation reveals that while commendable strides have been made, there are areas that require further attention. The analytical toolkit needs to be further codified and consolidated, and the integration of market intelligence with financial and macroeconomic monitoring is incomplete, which hampers effective crisis management and the ability to provide actionable policy advice. To enhance the ESM’s capabilities, a greater focus on proposing policy solutions to mitigate economic shocks is necessary, along with stronger internal coordination and independent analytical capabilities. Additionally, the ESM should share its assessments with concerned members and build capacity to assume a greater role in programme negotiation and design, which requires further contingency planning.
The ESM can ease constraints on its analytical capabilities by insourcing knowledge through external networks, which can also boost its influence. The evaluation highlights the need for timely and independent analytical capability, which is currently hampered by dependence on publicly available forecasts and data, and limited opportunities to leverage external expertise and partnerships. Addressing these constraints through collaboration arrangements with peer institutions, academia, and national authorities can alleviate these issues. Furthermore, the ESM’s ability to sustain progress and enhance analytical capability depends on tackling challenges such as the absence of key institutional policies, overly complex internal processes, staff retention issues, and inadequate arrangements for scaling up capabilities in response to sudden demand increases. Specific recommendations include consolidating the analytical toolkit, refining the analytical system, fostering synergies, and enhancing external engagement and policy advisory roles.
Figure 1: graphical depiction of how the shock event studies were conducted
Note: Different coloured boxes denote different team members involved.
Source: Evaluation team
Next steps
A management statement was included at the end of the report, welcoming the report, recognising progress made, acknowledging remaining gaps and challenges, and supporting the thrust of the report’s key recommendations. Management has committed to follow-up to the recommendations with the preparation of a medium-term implementation plan, prioritising the measures and taking into account progress on the initiatives that were already launched in the 2025 workplan.
More detail on the methodology used for this evaluation can be found in the Technical appendix to the evaluation report.
Acknowledgements
The author would like to thank his report co-authors Olga Francova, Kari Korhonen, Diana Zigraiova, Carsten Eppendorfer, Sotirios Giannoulis, Andres Willams, and Juergen Klaus, as well as the members of the external Evaluation Reference Group David Goldsbrough, John Hicklin, Barry Kolodkin, and Ritva Reinikka for their substantial guidance and ensuring the overall quality of the evaluation report.