Common Evaluation Planning Considerations
Selecting planning tools. The planning stage, much like evaluation more broadly, can be aided by selecting and using a small set of tools specifically to streamline and coordinate planning tasks. To make sure your team can collaborate effectively, it can be beneficial to utilize:
- A common timeline that can be referenced and updated
- A shared scheduling or calendar tool to ensure everyone is aware of meeting times and how to participate
- A collaborative writing platform so that everyone can view and contribute to written documents
- An established communication platform/channel to keep everyone engaged and ensure they have access to up-to-date evaluation activities and requests
Maintaining confidentiality. Most evaluation tasks do not require your program to track individual data, and the privacy and confidentiality of participant data must remain a constant priority. Ideally, someone looking at your evaluation data should not be capable of connecting a particular point of data with a particular person. To maintain the integrity of your program evaluation effort as well as the trust that people are putting into the process, it is important to devise a plan early in the process for how you will keep and maintain the confidentiality of the data you collect. Some general best practices include:
- Recognize what is called "personally identifiable data." This data includes names, addresses, any contact information, as well as personal details such as a participant’s date of birth. Unless it is absolutely necessary, do not collect this information. Data should be specific enough to be usable, but general enough that it cannot be linked with a single person. For example:
- Collect participants’ ages rather than birthdays.
- Collect their ZIP Code or neighborhood rather than their address.
- Assign random ID codes rather than using participant names.
- If your program collects any data that is unique to a participant, you now have "custody" of that sensitive information. This means you are responsible for safeguarding it, ensuring you know exactly who can access it, and being aware of where all copies are at any given time.
- Keep data in a locked cabinet or in a password-protected folder on your computer, and only allow access to the evaluators and key program staff. If you are using a web-based platform to share or store data, it must be kept in a private and protected format, and you should be aware of every individual who has access to those folders.
- Have all members of the evaluation team sign a statement that outlines the importance of confidentiality and ensure that they not share privileged information about the evaluation with anyone who is not a part of evaluation or program teams.
- Present the data in the aggregate, i.e., summarized with other data. This helps ensure that no individual data point can be identified independent of the others.
- The simplest way to avoid compromising a participant’s personal data is not to collect it in the first place. Very few evaluation tasks genuinely require individual data on participants.
Receiving clearance from your funder. Another planning consideration is related to clearance. Some federal programs require that you submit your evaluation plan and any accompanying tools through a clearance process. In a clearance process, an external party reviews planned evaluation activities to ensure they are in compliance with the terms of the grant award. This is not common, but can be a requirement for harm reduction programs that are directly funded by state or federal entities. If clearance is needed, you will not be able to begin your evaluation until you receive the green light from your funder. The clearance process is usually outlined in your grant award information and your project officer should be able to provide any additional clarification needed.
Encountering ethical problems. Harm reduction is both a practical system and an ethical framework. It is not possible to practice harm reduction without maintaining – at every moment – the ethical obligations which set harm reduction apart from other approaches. All evaluations should be designed and implemented in a way that fully respects the rights of the people whose data is being used. To accomplish this, it is important that all parties commit to causing no harm and/or minimizing the potential for harm to occur, not misrepresenting or misusing data, acquiring consent wherever possible, and not breaching confidentiality or specific requests for anonymity.
Despite these safeguards, it is not uncommon for ethical problems to emerge during a program evaluation. Some ethical problems may include:
- The evaluator or evaluation team has decided what the findings “should be” prior to the start of the program evaluation or plans to use the findings in an ethically questionable manner.
- The evaluator or evaluation team declares certain questions off limits in the evaluation, despite their relevance.
- Findings are modified by the evaluator, evaluation team, or program team before release.
- The evaluator is pressured to violate confidentiality.
- Legitimate clients, partners or staff are omitted from the planning process for fear of what they might share.
- The evaluator is no longer able to be objective or fair in presenting the findings.
- The evaluation results have the potential to expose participants to harm or are used to limit agency and/or compromise access to services for those who have provided evaluation information.
If an ethical matter does emerge during your program evaluation process, it is up to the program team, the evaluation team and/or the evaluator to name the issue and devise a plan to address it.
