Program Evaluation Toolkit for Harm Reduction Organizations
When planning a program evaluation, it is important to consider the role of culture because of its influence on every aspect of the evaluation process. Culture influences how we conceptualize, plan for, and implement our program evaluations. It has the power to inform the types of questions we choose to ask, the people we choose to involve in the process, and even the value we place on certain feedback.
Examining culture. When many of us think of culture, we typically think of categories such as racial and ethnic groups, gender, and age. And, while those groups embody a wide range of cultures, they just scratch the surface when it comes to the complexities of culture and cultural identity. Broadly speaking, culture is the ever-changing, learned systems of knowledge that influence behavior, attitudes, perceptions, and one’s sense of belonging in both universal and specific ways.
Being mindful of the role culture plays in evaluation can help us to become more intentional about checking our perceptions and encourage us to facilitate a more inclusive process that elevates the viewpoints of the community we serve and the staff/volunteers we entrust to deliver those services. Below are some culture-specific questions that may be helpful to ask yourself as you reflect on the role of culture within your evaluation process to date:
Addressing our biases. Biases can be defined as “a tendency, or prejudice, toward or against something or someone.” Some of our biases are known to us, and other biases operate outside of our awareness. Those types of biases are commonly referred to as implicit biases, or blind spots. When it comes to your program evaluation, the presence of unchecked or unacknowledged biases can lead to inaccurate findings and ultimately can reduce the credibility of your program evaluation. For this reason, it is important to both acknowledge the presence of biases, and build processes to mitigate the impact of those biases on the integrity of our program evaluation effort.
There are many biases that can be present throughout an evaluation process. Of those, the following three biases often stand out within program evaluations:
Confirmation bias: This bias refers to the tendency to seek out or focus solely on the information that supports an existing belief or understanding. Within a program evaluation, confirmation bias may motivate an evaluator or an evaluation team to seek out data that aligns with their understanding of how their program has benefited the community, and disregard information that might contradict or refute that data.
Response bias: This bias refers to people's tendency to share incorrect or incomplete information due to fear or discomfort around sharing more truthful data that reflects their experience. If this bias is present, it can lead to the collection and analysis of inaccurate data and the development of incomplete or false conclusions. This type of bias is often found among evaluation respondents, but can also be held by members of the program team or the evaluation team who fear that sharing openly and honestly will negatively impact their employment or ability to contribute to the project.
Selection bias: This bias refers to selecting individuals to participate or contribute to the evaluation process who do not accurately or fully reflect the demographics of your target population. An example of this bias is seeking out specific clients to participate in an interview because it is known that they had a positive experience or are liked by staff.
Although biases are a part of human nature, they are not etched in stone. It is helpful to keep in mind that we each have the power to address our biases and mitigate the impact of biased decision making, particularly when it has the potential to negatively affect your program evaluation.
To learn more about bias that are common in program evaluations and how to address them, check out these resources:
Focusing on racial and gender equity. Incorporating a focus on racial and gender equity requires taking intentional steps toward eliminating gender and racial disparities within your program and committing to an evaluation process that will contribute to measurable change in the lives and experiences of program staff and participants who are people of color, as well as those who identify as transgender or non-binary.
WHAT IS SOCIAL JUSTICE?
“Social justice is the pursuit of equal rights and equitable opportunity for all.” —San Diego Foundation
To invest in racial and gender equity work means that you are investing in social justice. It also requires the normalization of conversations about race, white supremacy, transphobia, and discrimination as they exist within the larger community context and within the organization. These types of conversations are never comfortable and often require the assistance of an external facilitator who can help create a safe, constructive, and affirming environment for awareness building, dialogue, and decision making. The Road Map for Racial and Gender Equity offers a framework and a series of reflection questions to help you to assess where you currently fall in your racial and gender equity work journey.
Table (2.3). Road map for racial and gender equity.
| P | PROBLEM AWARENESS | CONDITION “Do I have an understanding of what the problems are and what the problems stem from?” |
|---|---|---|
| R | ROOT CAUSE ANALYSIS | |
| E | EMPATHY | CONCERN “How do I feel about the problems and the people who are harmed by those problems?” |
| S | STRATEGY | |
| S | SACRIFICE | CORRECTION “Do I know how to address the problems and am I willing to direct the needed resources and attention to addressing them?” |
Adapted from Harvard Business Review (2020). How to promote racial equity in the workplace. https://hbr.org/2020/09/how-to-promote-racial-equity-in-the-workplace
"There is a difference between Black harm reduction and White harm reduction. Oftentimes, harm reduction comes to us in a White model…what we’ve been finding in the Black harm reduction community is a need to educate our [white] counterparts that while you’re well intentioned, you still come in through the eyes of a racist society, and that’s going to affect the way you view the person [you’re engaging]."
—Al Forbes
Here are some additional resources to support your continued learning and strategy development:
Racial Equity
Gender Equity
Prioritizing inclusion. Inclusion can take many different forms but, for our purposes, we are referring to the meaningful involvement of PWUD and people with lived experience throughout the evaluation process. It involves establishing a low-barrier and accessible pathway to engagement and participation, while maintaining an openness to learning and receiving insights.
Table (2.4). PWUD inclusion model.
| INFORM | CONSULT | INVOLVE | COLLABORATE | EMPOWER |
|---|---|---|---|---|
| “Here is what’s happening.” | “Here are some options: what do you think?” | “Here is a problem: what idea(s) do you have?” | “Let’s work together to solve this problem.” | “We will implement the process that you decide is best.” |
| (One-directional information sharing) | (Information sharing and feedback) | (Bi-directional information sharing) | (Partnership and trust building) | (Shared leadership and ownership) |
| Providing information to PWUD that will support their understanding of the purpose and intended outcome of the evaluation process | Facilitating opportunities to obtain feedback from PWUD on the development of the evaluation process | Working with PWUD to ensure concerns and desires for the program are considered and reflected in the evaluation process | Engaging PWUD as partners and soliciting advice and recommendations on the design and implementation of the evaluation process | Placing the final decision making in the hands of PWUD |
Adapted from the IAP2 Spectrum of Public Participation
Now that you have reviewed the PWUD Inclusion Model, consider the following questions:
It is important to keep in mind that inclusion is not blind acceptance of what people ask for or recommend. Like many other program activities, your evaluation is going to have its limitations. When prioritizing inclusion, it is important to clearly communicate what the parameters, boundaries, and program capacities are, as well as what your policies will allow so that all parties understand what is and what isn't possible.
