Program Evaluation Toolkit for Harm Reduction Organizations
Al Forbes is an independent consultant and evaluator working with the Aliveness Project Northwest Indiana. He described his evaluation skills as “a continuation of my personal values and beliefs,” which have guided over two decades of experience working in the harm reduction movement and supporting people who use drugs, especially those in LGBTQ+ and Black communities. For Al, a lifelong practice and interest in internal and external improvement “helped me see that evaluation is an ongoing process and needs to be incorporated in every aspect of the conversation.”
“There was no one else to fill the role, the role needed to be filled.”
Al highlighted the key role that data and data capabilities play in harm reduction spaces, especially in spaces where improvement is critical but resources are limited, including small non-profits, minority non-profits, and LGBTQ+ organizations:
“They don't have a lot of money, they don't have a lot of resources, and when they do have money or resources, one of the ways to use that is to establish skills on evaluation or data collection… So they're doing the work. They're out there making a difference in the community, but they're not monitoring or measuring that. So I come along and help them to create a framework in which they can tell this story, this narrative. For me, evaluation is what comes after you've done the work to ensure you've done the work well, so that's what I typically do with my clients is just give them those tools so they can look back and say, ‘We did the job,’ and, ‘We did it well.’ The two aren’t always married to each other. Maybe it’s done, but it’s not done well, that’s why you do evaluation.”
“Let's make it about the organization.”
Al shared a recent success story highlighting the broad scope of evaluation thinking and the need for an organization to have a clear vision of its capacities before it can fully leverage evaluation data. Working with a client who had just received a grant to incorporate the RedCap data platform, Al explored the organization’s existing model as well as its existing questionnaires/forms, and soon recognized the need to step back and build a deeper framework to structure the program’s data collection:
“So we have to sit down and go back to the drawing board and say, ‘What are your programs? Not what are your services, but what are your programs?... Let's go back into evaluating what the organization is doing.’ So, we did that. We then created a list of services under each department and then we started collecting data using a simple Excel spreadsheet. Evaluation is not just the formal part of looking back at the data, but also going back and evaluating what the organization is doing.”
With Al’s support, the organization developed a more functional model and objectives, enabling them to use simple tools like Excel to start collecting and interpreting data. Empowered with this additional capacity, the organization was able to approach major stakeholders and continue growing their data capacity:
“We started doing that data collection, and they were able to start going to the City of Chicago and Chicago Department of Health and the state Department of Health with data in hand. So it changed their ability to then look at what their organization was doing from a statistical standpoint, from a numerical standpoint. Then—when you have the numbers—it allows you to go back and evaluate what is going on… I tell people that evaluation doesn’t have to be this long, complex process where you have to call [a specialist] in to do it. Evaluation is collecting the data and taking time to review the data.”
“We’re dealing with a traumatized population, our job is not to traumatize them further.”
Building an evaluation model within an organization requires internal dialogues with colleagues, but when an evaluation requires collecting data from participants Al stressed the need for programs to be selective and aware about the data they collect:
“When I look at data collection, which is an essential property of evaluation, I ask, ‘Why are we having people taking questions and surveys with questions that have nothing to do with the work we want to do?’ I tell my staff who do trainings, ‘If there’s a question on here that has no relevance to the work you’re doing, you’re going to find resistance from the client, because they’re going to want to know why you need to know that. If you can’t answer that question, that tells me it shouldn’t be on there’...Just make sure the questions are appropriate and that you have a reason for every question, that you’ve questioned the question, basically.”
Now that we have gone over considerations for making the case, in the next module we review ways to set yourself and your team up for a successful harm reduction program evaluation process through preparation.
