Participatory evaluation guide - The approach
Approach: setting the ground for participation
In this section we share some of the approaches and methods we have found useful in establishing a good environment for open discussion and analysis, including of those people who are less powerful or confident in a group.
Evaluations can be broken down into several phases, and the values of equity, awareness of power and voice, should be fundamental to all of these stages.
- The initial TORs set out the scope and purpose of the evaluation, but an initial needs assessment is an opportunity for the evaluator to include other voices in selecting appropriate evaluation questions and methods.
- Alongside the desk review, this process will help the evaluator to understand the programme to the point of being able to describe it, and sketch out the underlying theory of change - whether clearly articulated or not.
- In this way the key stakeholders and their relationships to each other and the programme will become clearer, as a basis for selecting representative and relevant samples to consult and engage. With the questions and stakeholders, the appropriate tools and processes can be designed.
- This will usually include further desk review and research, as well as direct engagement with stakeholders through interviews, surveys and focus groups. This approach centres on a workshop with a small group of primary stakeholders, or beneficiaries, to understand the project first through their eyes and in relation to their realities and expectations. From there, the evaluator is able to select further stakeholders to engage and data to collect.
- With this 'grassroots' description and analysis of the programme relevance, impact and quality, the evaluator is then able to feed back to project staff and facilitate reflection on these inputs, and the experiences and learning coming from project implementation. We have found, and been told by project staff, that this is a very useful process helping to unearth challenges, opportunities and recommendations for future work and relationships.
From here, we are focusing on the workshop, research, analysis and feedback processes that happen 'in the field' or at the grassroots (although of course this may be local to the evaluator, in our cases it has always involved travelling and cross-cultural communication.
The approach requires the evaluator to facilitate processes of reflection, knowledge sharing (including storytelling) and collective analysis. There are plenty of resources available on the skills and tools of a good facilitator (and we would love you to add references to those you have found particularly helpful). In our experience, many of the qualities are personal and everyone has their own style, it is most important to be yourself. But there are some fundamental elements which underpin the relationships built between the evaluator and other participants in the evaluation.
These include independence and integrity, and also respect and open communication. We need to emphasise listening, critical questioning, and empathy. Participatory methods and tools can help facilitate this process, but the evaluator has not only to generate discussion and information, but also try to understand what it means. In the PLA issue 63  'How Wide are the Ripples' we explored the evaluator or researcher's role in interpretation, especially across cultural contexts, which is often not a subject of scrutiny or methodological attention. Here we looked at ways to increase opportunities to 'make sense together', allowing the informants to help the evaluator understand the meaning of the data collected.
Perhaps most controversial in relation to evaluations, where there is a sense that independence is equal to neutrality and objectivity, this approach requires us to be part of the situation, not above or outside it. The evaluator's position is to examine and understand 'the bigger picture', drawing together different elements and perspectives in order to inform the understanding of the underlying theory and a systemic analysis of the whole 'beast'. We have a lot of power in deciding who to consult, involve and listen to. We have to select who to listen to, and how to reconcile sometimes conflicting or varying accounts of the project and its value. As outsiders we can play an important role to bring different experiences, knowledge and opinions together to tell a story, but it is important to recognise and as far as possible counter the fact that our cultural and social background will inform our choices and interpretations. We have found it useful and effective to openly discuss with workshop particpants, and project staff, our own role in the evaluation and research process - exploring the potential benefits and challenges of having an outsider take the role of telling the story, or assessing the programme.
When introducing yourself as a facilitator and evaluator, it’s important to get an idea of who they think you are, what they think your role is and what they think your added value is. Some participants may be really clear about this and others may not have an answer, but it's likely that all will feel the need to know. One of the key roles of a facilitator is to create an environment of trust, openness and oneness or team. There are, of course, many activities used to do this and can include developing a team agreement as a group, exploring how you will work together and what each persons’ commitment is to the group and to the evaluation process itself. At this point we think it’s important for them to identify how this process can support them – how the process of reflection can be really helpful for them to progress but also to acknowledge their achievements and the impact they have made so far. It’s also useful with some groups to identify what skills they may learn from the workshops such as communication, teamwork, storytelling/presentation, reflection skills etc – this can really incentivise and encourage participation and engagement. Activity: Risk and skills graph. Draw a graph where one side is how many risks you are prepared to take and other side is how many skills you want to learn. Talk about the relationship between the two. Where they place themselves is where they place themselves and neither right nor wrong. The process is to allow them to see what they are up for and how when re-visited this might have changed and why. It’s a great way to encourage people to challenge themselves without pushing them and while creating a safe environment to do so.
Who tells your story
At this stage it is useful to have an open discussion of the role of the facilitator/ evaluator in the group. Why do you need an outsider to come and tell your stories to the organisation/ donor? What value does the role add to their participation in the project/ engagement with the NGO? How is the evaluator able to contribute to their social change processes?
The conversation can be open, guided by questions such as: - who needs to hear your stories/ perspectives? Or ... why do people in London/ New York/ Nairobi need to hear your stories/ perspectives on the project and its objectives? What do they need to understand and know in order to improve their support to you and your group/ community? What do you think they are like, know already? - what is the advantage of having an outsider? What do I bring? How should my role be understood? For example, which audiences to I have access to/ speak the same language as? What perspective can I bring? What kinds of support and processes do you need in order to present a collective analysis and combine different stories and perspectives from the group? - what are the challenges? How can they be managed? What might I not understand that is key to presenting the stories? How can we ensure that I represent your different voices effectively and fairly?
From this discussion you can set out some criteria for your participation in their storytelling, and your accountability to the group.
Effective cross-cultural communication
Why important: central to this evaluation approach is to collect people’s stories and insights into the project relevance, effectiveness and impact, and using this as inputs into both the final report and subsequent reflections and discussions with different stakeholders. It is always useful to prepare groups to think about how to communicate experiences and changes to people who are not familiar with their context. Much of the information which gives meaning to your story is assumed, unsaid and ‘between the lines’. Often (by no means always!) this can safely be assumed with audiences and correspondents who share your own culture and context. Writing a story down on paper for audiences from differnet social and organisational cultures requires strong awareness of different elements which need to be articulated to ensure the meaning is transmitted.
- inside outside is an exercise to explore effective communication of personal issues.
- Between the frames is an exercise to plan how to tell a story. Example: In a workshop with young people in Kenya, the participants collected photographs for a photostory showing their home lives. The next day they showed their pictures on a projector, keeping silence and allowing the audience to comment and describe what they had seen and felt. Afterwards the photographer was able to respond to the thoughts and comments of the audience. The facilitator was also able to make comments on differences between the way things were intended and perceived, or differences in the way someone from another place and culture might interpret them. For example, one young woman included a picture of an empty bus, which everyone agreed meant a bad feeling because it would signify significant delays while you wait for the bus to fill up. Meanwhile as a Londoner, I was able to share that for me, an empty bus might be a good feeling because our buses run to a timetable whether empty or full. Later a colleague from Nairobi commented that an empty bus would be very suspicious in a city where buses are always over-full. We were able to discuss the differences in the way we interpret the picture depending on context, and draw conclusions and insights for communication through photography.