Program Evaluation Toolkit for Harm Reduction Organizations
Barbie is a specialist in public health who works as the Deputy Director at the Virginia Harm Reduction Coalition. She is passionate about understanding the needs of program participants and advocating for them. While she acknowledges the importance of reporting data to funders, she believes evaluation should go beyond simple metrics and explore the actual impact of harm reduction programs.
Barbie doesn’t want evaluation data to show just superficial information or what could be learned from an intake form, she is passionate about building knowledge of participants' lives and their experiences, not just their interactions with interactions with harm reduction programs.
“We want to tap into your knowledge about this world so that we can better understand and do a better job of advocating for you and serving your needs.”
“I feel like this is a population that really needs to be studied…We have to - instead of making people feel like we're doing a study because we want to look at them - we want to understand them because they are people that we care about, we want to understand what we can do to serve them better. And we also want to make sure that they understand that we view them as experts in their own lives.”
Barbie also speaks to the limitations of simple metrics and the need for in-depth data that explores the actual impact and outcomes of a program on a community. She explains that simple evaluation data, such as the number of vaccines or supplies distributed, does not reveal how people heard about the supplies and can create a “black box” because it fails to demonstrate the process underlying the impact of the those supplies:
“We end up having a lot of black boxes where we don't know what happened, right?”
“We go from ‘this is what I want to have happened and then here we are’ and I have no idea to what extent any of these things was useful…all I know is that a whole load of people got vaccinated. My suspicion is that it had everything to do with the gift card we gave them. Do I know that for sure? Do I have any reason to believe that the flyers made any difference? I have no idea. I have no idea whether it's a complete waste of time or whether it's effective.”
“What we do is a lot of process evaluation and then we do some outcome evaluation like: this number of people got treated for hepatitis C, this number of people got connected and had at least one appointment with or initiated substance abuse treatment like this...so those are outcomes, and I totally appreciate that. But the thing that frustrates me is having too much of the black box stuff and too much of this stuff where we report what we did. We distributed, you know, fifty thousand condoms in a year, right? That's all fine. But did that change anything? Like how many pregnancies did you prevent? How many STIs did you prevent, that is the question.”
As for how to get that impact data, Barbie highly recommends focus groups with compensated, engaged participants who each bring a different perspective and expertise to the conversation. With compensation, setting up these conversations is often easy.
“I’m a really big believer in focus groups.”
“I don't think that they’re that hard. They're not. I mean, they're really not that hard. You really just need a few questions to like spark conversation and when you're talking about this population they love to talk.”
Now that we have gone over considerations for planning your evaluation, in the next module we review ways to set yourself and your team up for a successful harm reduction program evaluation process through preparation.
