Program Evaluation Toolkit for Harm Reduction Organizations
"[We’re] tired of people misrepresenting us or not putting drug users to the forefront in addressing what the problem is or what the solution is…[T]here’s not many people doing drug user-led evaluation policies or drug user-led stuff to be evaluated.”
—Nick Voyle, Executive Director of Indiana Recovery Alliance
Now that we have laid the foundation for the role of culture, equity, and inclusion in evaluation, let’s explore a community-centered approach that pulls all of those elements together.
Participatory evaluation is an evaluation approach that emphasizes involving in the evaluation process those individuals who will be directly impacted by how the evaluation is carried out and what it finds, and then using the results to strengthen programs. As the name suggests, this approach creates a pathway for PWUD to contribute as collaborative partners and work together with staff and volunteers to design and implement an evaluation process that is meaningful for all.
WHAT IS AN EVALUATION APPROACH?
Evaluation approaches are the “distinct ways to think about, design, and conduct evaluation efforts.”
There are many benefits to integrating this approach into your process, including:
So, how exactly does it work? There is no specific formula for fully integrating this approach into your evaluation process. There are four overarching steps:
Ensure you have the “right” people at the table. This is the most time-consuming element of the process because it requires identifying and training community members who have the ability to commit to a program evaluation process that might take up to a year to complete.
Establish evaluation “feedback loops.” This element focuses on determining where the community will be engaged in the process. Ideally, community will be able to contribute to core components of the program evaluation process, including:
Start simple. If this is your first time incorporating this approach, it is probably not the time to develop a complicated community evaluation. Throughout the process, it is important to ensure that all communication is accessible and easy to understand.
Build a culture of reflection over time … and stick with it! In the figure below you can see where in the continuum the approach makes space for a reflection period.
If you have been a part of a program evaluation in the past, it is likely that it was structured in a traditional or conventional way. You can think of the conventional approach as a composite of all of the traditional ways we have been taught evaluations should be carried out. The table below details some of the attributes of a conventional approach and how they differ from the participatory approach.
Table (2.5). Differences between conventional and participatory evaluation.
| CONVENTIONAL EVALUATION | PARTICIPATORY EVALUATION | |
|---|---|---|
| WHO | External evaluators, experts | Community members, project staff, facilitator |
| WHAT | Predetermined indicators of success, principally cost and activity/production outputs | People identify their own indicators of success |
| HOW | Focus on "scientific objectivity"; distancing of evaluator from other participants; uniform, complex procedures; delayed and limited access to results | Self-evaluation; simple methods adapted to local culture; open, immediate sharing of results through community involvement in evaluation processes |
| WHEN | Usually upon completion; sometimes also mid-term | Merging of monitoring and evaluation; hence frequent small-scale evaluations |
| WHY | Accountability, usually summative, to determine if funding continues | To empower local people to initiate, control and take corrective action |
"Any tool or technique can be participatory, or not, depending on how it is used. The key is who makes the decision and who is in control. Project staff and evaluators often find that one of the most difficult challenges in participatory evaluation is giving up total control, or 'letting go' of their notion of the right way, the right question, the right wording, the right order, or the right answer.”
—Deepa Narayan-Parker
It is also worth mentioning that, in most cases, it is not all or nothing when it comes to the two approaches. In fact, many programs have no choice but to strike a healthy balance between conventional and participatory approaches, particularly programs that have to incorporate funder-driven indicators for success or a specific funder-mandated data-collection process.
While there are many benefits to using the participatory evaluation approach, there are also some important challenges to consider, including:
Despite the challenges, participatory evaluation can be the right choice for programs that are committed to the ongoing engagement and inclusion of PWUD in their program activities.
