I have been working on a paper that uses a specific episode of work done by the Nonviolent Peaceforce, as the basis for understanding some of the strengths and weakness of different evaluation methodologies. It is very telling that what might look like a failure using one system, can look like a success in another. For instance, if one of the indicators of an objective regarding building cross ethnic relationships is successful meetings, then the cancellation and lack of rescheduling of a critical meeting would appear to be a failure. However, if looked at via outcome mapping, the emerging connections created through the process of trying to hold the meeting and the tenuous new connections made outside of a formal meeting process, indicate some successful motion in changing behaviors and relationships.
What has started to interest me even more however, is that the different methodologies require different processes and therefore different actors to participate in the evaluation itself. For instance, a logical framework can be written in a central office, by a grant writer, based on field reports and senior program staff analysis. While there is no reason to exclude program staff or various partners to a project, they are not essential to the process. Similarly, an evaluation report can be written in the same office based on similar materials such as field reports, program staff analysis and the like. There is no need to involve others to do the evaluation, though of course some evaluations based on a logical framework do in fact involve input from many others. Similarly, various methods based on theories of change, while potentially reflecting a more complex and nuanced analysis of causality and relationships, do not require in the actual method, the input of others outside the implementing organization.
However, other methods such as outcome mapping and most significant change stories, require in the actual implementation process, the input not only of “field” or on the ground staff, but also of other partners, beneficiaries, target populations or other titles used to describe people outside of the implementing organization. “So what?” you might ask.
It seems to me that one of the deep and ongoing struggles in the world of conflict transformation, community building and many other arenas, is the question of who defines problems, solutions and methods. In other words, who makes meaning of situations, of phenomenon? This is a question addressed by philosophers and social change agents alike for many years. I don’t know that I have much to add to the broader discussion, but it seems very pressing in terms of evaluating transformational efforts.
If all the analysis can be done based on meaning assigned by the organization itself, then there is significant opportunity to distort the experiences of those in the actual change process. Even when, for instance, surveys of partners or beneficiaries are included, the questions on the survey are designed by the organization, and the results interpreted by that organization. When outside evaluators undertake the evaluation, it is often in part to avoid this situation. However, it is still an “outsider” that makes meaning of events, relationships, behaviors, or the change or lack thereof going on. In some cases an outside evaluator may be even less likely to access the many perspectives of the ongoing work, and therefore contribute to the inaccuracies in an evaluation.
So, it has struck me yet again, that the process of how we go about understanding our work needs to include as many perspectives as possible. A method such as most significant change stories, while it lacks many things, provides input from many places AND uses people in different positions to determine which story reflects the most significant change. The process clearly determines much of the content of an evaluation. This seems to me a clear argument for using several methodologies and at least one multi stakeholder narrative or other qualitative method in any serious evaluation.
Nothing new here, but much of the recent literature on evaluation conflict transformation has focused on the need for processes that reflect the complexity, lack of clear causality and non linearity of change. It is important to also focus on the issue of who makes meaning, of who decides what is important to focus on, what constitutes change, what constitutes success. Let’s be sure to include this aspect as well in our discussions.
Ellen Furnari
