Program Evaluation Toolkit for Harm Reduction Organizations
Laura Nagle is a Risk Reduction Specialist at Madison County Health Department in the state of Virginia. She detailed the opportunities and challenges faced by evaluation efforts in states with conservative public health traditions. With a grant provided by a national network of county and city health departments, she described how she and her team utilized evaluation tools and techniques to address an emerging issue and better serve rural participants:
“We got this grant to figure out how we're going to do better data collection. So we already use RedCap, you know, we collect a lot of quantitative stuff, but haven't done a lot of qualitative. We noticed in March last year that our numbers just went from [high] to straight down. I mean it was astonishing how our numbers dropped, we couldn't figure what was going on. So we were able to do incentives, a $20 gift card to do 30 minute interviews with folks, and we learned a couple of interesting things.”
“It was this perfect storm of things that caused our numbers to dive substantially.”
By providing structured, compensated point-in-time (PIT) surveys, Laura and her team quickly honed in on the cause of the sharp drop-off in participants. Using these simple surveys, her team traveled to various cities in the state and gathered direct feedback from program participants and those who would potentially benefit from being participants, discovering two primary reasons for the drop-off.
“A new law in Kentucky had got passed that said that folks can buy syringes at pharmacies without an ID or a prescription. That combined with the fact that we have had to use retractable syringes here, that's how our ordinance was written, and people, you know, hated our syringes. So, once word got out that they could buy them at Walmart, it took about eight months from the time the law passed until word got out.”
“We also learned how big of a barrier transportation is for people. We're a pretty large county...So people who live close to us, you know, could walk or bike, but a lot of people can't. So kind of the two things that came out of that is that we have to figure out how to get rid of these retractable syringes and we have to be able to start doing mobile exchange so that we can get to people who can't get to us. Because we're a public health department, you know it is our job to prevent being the next Austin, Indiana [a city that had a major HIV outbreak], you know that's right over the river from where we are, you know? So, if we want to prevent that, we've got to do those things.”
“It was fantastic to reach out to people.”
Laura and her team selected the right evaluation tool for their capacity and context. They needed a tool that could be developed quickly, was easy to use, and provided up-to-date information. For them, a PIT survey was the perfect fit not only for critical feedback, but also for outreach to new participants. Laura laid out some examples of questions they used for their survey:
“Do you know anyone who could use our services but hasn't come? What do you think are the reasons?”
“When there are times that you needed our services but didn't come, what were the reasons that you weren't able to be here?”
“How do you think people would feel about mobile exchange? What would it feel like to walk up to this van?”
“Most people's response was ‘thank you for asking.’ So that was very very positive.”
Delivering the PIT elicited positive responses from the community, as well as vital programmatic data and qualitative information that could be used when advocating for the program to be allowed to change their syringe type and begin mobile services.
“So it started with anyone who came in to our harm reduction program, we would ask if they had a little bit of time to do it and then it just was a snowball sample from there. That’s when we could start interviewing people who maybe didn't come to us but needed our services or people that we hadn't seen for a while - because we had the incentive - you know, we had a lot of people that were willing to talk to us. We would take the van out and do surveys with folks who didn't use our services but needed them... “
Unfortunately, the program met opposition from a single local official who disrupted their opportunity to put their information into action.
Now that we have gone over the final module on how to use your evaluation data evaluation, the conclusion includes a brief overview of the modules in this toolkit and provides tools and resources you can reference when conducting your own evaluation.
