Riitta Oksanen is the President of the EES. She works as a senior advisor at the Finnish Ministry for Foreign Affairs, Development Evaluation Unit. In this post Riitta discusses a new EES initiative – the ‘Voluntary Evaluator Peer Review’ (VEPR) process to promote evaluation professionalism – with Bob Picciotto who leads the EES Professionalizing Thematic Working Group and Pam Oliver who is leading the EES’s pilot of the VEPR.

Key Points

1. Evaluation standards and professional development mechanisms are the basis for evaluation professionalism. VOPEs have an important role to play.
2. The EES VEPR uses self-reflection supported by peers and focuses on practice areas selected by the evaluation practitioner – evaluation provider, manager, commissioner or educator/trainer – undertaking the review. The VEPR explicitly supports commitment to continuous learning. It is not a test.
3. Recruitment for EES VEPR pilot will start at the EES Conference in Maastricht and take place in November/early December. Join us to experience your own VEPR!

Riitta: Bob and Pam, thanks for joining me to talk on the EES VEPR initiative, and evaluation professionalization more generally. You are among the EES colleagues who have invested a lot of time and thinking into promoting the VEPR initiative. Professionalization, and particularly designation, is a controversial issue within the evaluation community. Why is promoting professionalization a priority?

Bob: As I see it, evaluation is duty-bound to strive towards professionalism in order to earn the public trust without which it cannot fulfil its potential. Ethical safeguards, adequate capabilities and occupational self-management characterize all professions. While evaluation has made giant strides in establishing itself as a discipline in its own right (as well as a trans-discipline that serves all disciplines), it has yet to be broadly perceived as a distinct profession. This is why the Global Evaluation Agenda features professionalization as a priority for the global community.

Pam: My original focus with the VEPR concept was on promoting safe evaluation practice – safe not only for evaluation providers/practitioners, but especially for evaluands. The only way that that can be ensured is to have clear standards and an effective way to facilitate evaluators to want to meet those standards. For the latter, we need a professional development and accountability mechanism that evaluators will find friendly, accessible and immediately valuable for them, and aspirational – evaluators will want to focus on practice areas that they need to strengthen. The VEPR concept achieves that by being voluntary, using peers as the facilitators of self-review through self-reflection, and having the reviewee identify the practice areas for the review, based on skills that they want to develop.

Riitta: I want to add that in this set-up, the role of the Voluntary Organisations of Professional Evaluation (VOPEs), such as national or regional evaluation societies, is crucial. It is fundamental that they are the institutions that set the standards as the basis for professionalisation. Any professionalisation process needs this foundation.

Without denying the priority of professionalisation, some of the concerns that sceptics have raised appear worthy of careful consideration. For example, shouldn’t professionalisation processes be inclusive? Shouldn’t they be sensitive to culture, language and context? Is there not a risk that designation will work against continuous learning and development?

Bob: These are precisely the issues that the Thematic Working Group worked hard to address. The process through which the EES capabilities framework involved the entire membership through two online surveys. We took great care in shaping it to reflect distinctive European values. We adopted an inclusive professionalization agenda focused on capability development rather than testing. Based on consultations at two EES Conferences we steered clear of a credentialing approach that would create an exclusive club. Instead we opted for an adaptable voluntary evaluation peer review (VEPR) concept grounded in reflective practice that allows each evaluator to focus on nurturing specific knowledge, skills and dispositions through dialogue with peers. Finally, we are intent on partnering with national evaluation societies to ensure full access to this initiative across all European languages.

Pam: I don’t see the VEPR as a ‘designation’ system. Its focus is on encouraging self-reflection as an everyday practice for evaluations, structured professional development as a regular practice, mentoring as a valuable medium for professional development, and the use of peer relationships for self-reflection, so that all parties – the reviewers as well as the reviewee – gain from the process. So it is both inclusive and focused purposely on continuous learning. And of course culture, context and language are all core aspects of the systems, as also is recognising that there are many different ‘evaluator’ roles that the VEPR needs to cater for. While the pilot of the VEPR will be initially in English, if the concept is found acceptable by members, then the national societies will be invited to develop and pilot their own versions, customiseded to culture and context. The VEPR application systems focus on good ‘matching’ of reviewers with the reviewee in terms of evaluation roles, level of experience, language preferences, and so on.

Riitta: During the preparations and consultations the VEPR Working Group has focused specifically on the importance, and some complexities, of language. It is fully acknowledged that anyone undertaking a VEPR review must be able to do so in a language of their preference; however VEPR also takes into account that they may want to nominate a peer reviewer with whom they have a language in common that is not the ‘first language’ of either person. Partnering with national VOPEs will help us offer a VEPR system that serves us all – I come from Finland, so it is naturally easy for me to be sensitive to this!

Bob: In 2016 we have refined the concept through real life experimentation. Pam and Riitta, both of you participated in testing the approach. What did you learn?

Pam: I was the first person on the VEPR Working Group to undergo a VEPR review as part of the pre-pilot. I’m from New Zealand, and my reviewers were from Poland and Finland. Nonetheless we discovered quite quickly that our practice contexts had significant commonalities as well as some important differences, and those differences were valuable in highlighting new ways of interpreting the practice issues I wanted to review. What I gained from that was, firstly, some very valuable ideas that I developed, in discussion with the two reviewers, related to improving the evaluation skills that I wanted to improve. However just as importantly, I developed a strong appreciation of at least two other processes: first, the value of identifying my own professional ‘weaknesses’ and of examining and researching those, as part of the process of completing the VEPR application; and second, the value of discussing those issues with two people whose skills I respected and who were not a part of my usual work team. The discussion was highly stimulating and resulted in me seeing the issues I had identified quite differently and developing some new ideas for addressing them. I also came away feeling that I had two new and independent colleagues with whom I could discuss challenging aspects of my practice in a truly supportive context.

Riitta: We are planning to launch the VEPR pilot in September at the EES Maastricht conference. The original purpose of the ‘pre-pilot’ of the VEPR systems that we undertook within the Working Group was to test technically the draft guidelines, forms and processes – but it has given us more than that. First, we are confident that, with some modifications, we now have robust and user-friendly systems for piloting with volunteers. Secondly, we recognized an important need to support all VEPR reviewers with training on key skills for facilitating the review process, and in particular understanding reflective practice and being clear about the VEPR reviewer role – that it is a supporting role, not supervision or coaching. It was evident that the Working Group members who took part in the ‘pre-pilot’ experienced a unique opportunity for learning.

Riitta: As a VEPR reviewee, I particularly appreciate that the VEPR is flexible in facilitating participation also of commissioners, managers and educators/trainers of evaluation. My own review gave me an opportunity to discuss and develop solutions for some of my hardest professional challenges. One of my colleagues has commented that the facilitated self-reflection “allows the participants to speak in first person”, and to me that captures well the essence of the VEPR process. As a result of my review, new approaches will be adopted at the Finnish Ministry for Foreign Affairs for a stronger evaluation culture. In addition to personal gains from the VEPR, my institution will also benefit.

I’m very much looking forward to providing the VEPR opportunity for a pilot group. Recruitment will start at the EES Maastricht Conference, and in early October all ESS members will receive an invitation by email to take part in the pilot.