Developing Your Program Evaluation Questions

WHAT ARE EVALUATION QUESTIONS?

“Evaluation questions help to define the boundaries of an evaluation that are consistent with information needs, opportunities, and constraints related to data collection and available resources.”

Evaluation questions are used to frame your evaluation implementation effort and inform your data collection. When developing your evaluation questions, it is helpful to consider:

• The goal(s) of your harm reduction program
• The type(s) of evaluation you are conducting
• The intended outputs or outcomes of the program activities

Luckily, all this information can easily be found in your logic model. It is also helpful to keep in mind that you do not have to do this alone. At this stage of the planning, it is particularly beneficial to invite your evaluation team to contribute to the question development process. Working in partnership with your evaluation team will help ensure the questions you are developing are the right questions not only for your program team, but for your funder and community as well.

Evaluation Questions vs. Survey Questions. Upon first glance, it may seem like developing program evaluation questions and survey questions are one in the same, but that is not entirely true. Program evaluation questions are intentionally broad, while survey questions tend to be focused on assessing a specific behavior, feeling, or event.

Table (3.5) Here are some of examples that illustrate the difference.

EVALUATION QUESTION	SURVEY QUESTION
To what extent is our SSP providing satisfactory services to clients?	How satisfied are you with the services provided at the SSP today?
Are we providing the supplies that our participants need?	Did the syringe service program have the supplies you were looking for today?
Was our new outreach site effective at engaging new participants?	Have you ever visited our main clinic to access supplies?

Evaluators often advise teams to “question the question.” As you are working with your evaluation team to come up with a list of questions, ask yourself the following:

• Has this question been answered before? Have you searched for existing data inside and outside your program that could answer this question for you?
• Does this question reflect the goal(s) and objective(s) of your harm reduction program?
• Does this question reflect key elements of your harm reduction program logic model?
• Can the question be answered using available data/resources within the allotted timeframe?
• Is the question an "open-ended" question?
• Does your evaluation team feel good about the list of questions that have been developed?

If the answer is "no" for any of the evaluation questions that are developed, consider deprioritizing and/or reframing the question to better align with the criteria above.

Since every harm reduction program evaluation is unique, the number of evaluation questions you develop will all depend on what makes sense for your specific program. Depending on how robust the evaluation needs are, you may decide to develop anywhere from 3 to 10 evaluation questions. Keep in mind, the more evaluation questions you have, the more data collection, analysis, and/or reporting may be required.

Below are additional resources on program evaluation question development:

• Evaluation Questions Checklist for Program Evaluation by Western Michigan University, Evaluation Center
• Develop Evaluation Questions by Pell Institute.

The next step after finalizing your program evaluation questions is to establish indicators. Indicators are signs of progress that are used to determine if a program is meeting its objectives and goals. Indicators also help you understand what has changed or occurred as a result of your program activities. When developing your indicators, ensure that they are specific, observable, and measurable.

To learn more about how to develop indicators, check out these resources:

• What are Indicators by CDC
• Indicators for Evaluation by My Peer Toolkit
• Using Indicators: How to Make Indicators Work for You by CDC
• How to Write Good Evaluation Questions by Eval Academy
• Developing Evaluation Questions by CDC
• Programming Essentials, Monitoring and Evaluation by UN Women
• CDC Approach to Evaluation by CDC