Collecting Your Data

Within the context of program evaluation, data collection is the process of gathering and measuring information from various sources in a systematic way to answer a program evaluation question. Many evaluations will rely on both quantitative (numeric) and qualitative (non-numeric/descriptive) data to gain a fuller understanding of their program’s activities. The table below highlights some of the main differences between the two data types.

Table (4.1). Data Types: Quantitative and Qualitative Data

Quantitative Data (Numeric)	Qualitative Data (Non-Numeric)
Reveals quantities, such as counts and measurements.	Reveals qualities, such as descriptions, opinions, experiences, and assessments.
Less subjective	More subjective
Answers the questions: “How many?” “What?” and “Who?”	Answers the questions: “Why?” and “How?”
Survey question example: Did you receive a harm reduction kit during your visit? [yes/no/not sure]	Interview question example: What did you like most about your visit?
Data example: 57% of the 100 respondents received a harm reduction kit during their visit.	Data example: “I liked the fact that you treated me like a human being, and gave me things that I could use like the harm reduction kits and the Narcan.”

It is important to keep in mind that your data collection process should be informed by your evaluation needs, your timeline, and what has worked best for your program and program participants in the past. Qualitative and quantitative data collection usually work together to provide a robust set of information about your program’s activities and accomplishments.

Qualitative and Quantitative Data - Values with Values

When considering which type to collect or how to combine the two, remember you are representing/reporting on real lives and complex conditions. Your data should add detail and depth, not compress it.

Voices from the Field: Sam Armbruster

“Although it can be challenging to incorporate both quantitative and qualitative data in a report, I try to include as much detail and context as possible in the narrative sections of reports within the bounds provided. It significantly deepens the value of data to understand and discuss conditions that any harm reduction organization is operating within because geography, policy, demographic, and historical context all greatly impact the ways people can and cannot engage in Harm Reduction and related services.”

Commonly used data collection methods for program evaluations are:

Surveys
Interviews
Observations
Document review
Focus groups

The following tables provide an overview of the each of these data collection methods:

Table (4.2). Data Collection Method: SURVEYS

Surveys (also referred to as questionnaires) are written tools that are used to collect information from multiple respondents, for example, on their view of the program and how the program is being implemented.

Data Type	Primarily collects quantitative data but can also collect qualitative data through open-ended questions.
Examples of Data Sources	Participant questionnaires Surveys Checklists
Advantages of Surveys	They are effective tools for collecting a lot of data at once. It is easy to collect this form of data in a safe, non-threatening and unobtrusive way. Can be given at a single point in time or over time as a pre- and post-assessment. It is easy to maintain participant confidentiality or anonymity. There are many sample surveys available that can be modified to suit your needs.
Challenges of Surveys	The survey language can sway how participants respond to the questions (response bias). You may not be able to follow up with participants to clarify information that is submitted. Certain groups or individuals may be surveyed more than others, potentially resulting in data that reflects a subset of perspectives (sampling bias).

SURVEYS
collect a lot at once

Helpful Resources:
Collecting Evaluation Data: Surveys by University of Wisconsin
Program Evaluation Tip Sheet – Constructing Survey Questions by CDC
Data Collection Methods for Program Evaluation: Questionnaires by CDC
Checklist to Evaluate the Quality of Questions by CDC

Table (4.3). Data Collection Method: INDIVIDUAL INTERVIEWS

Individual interviews are conversational sessions that are conducted with program participants, either in person or virtually, to gain an in-depth understanding of their experiences and opinions.

Data Type	Primarily collects qualitative data but can also collect quantitative data by numerically coding responses.
Examples of Data Sources	Documented question-and-answer sessions with program participants or staff. Can be either formal or informal.
Advantages of Individual Interviews	You can receive a full range of information about your program, including participants' feelings, thoughts, beliefs, and opinions. You can clarify the information that is shared in real time.
Challenges of Individual Interviews	Interviewing is a skill that takes time to develop. Can be time consuming. Can be hard to analyze or compare data received across participants. The information received will likely reflect participant biases.

INDIVIDUAL INTERVIEWS
clarify info in real time

Helpful Resources:
Using Qualitative Interviews in Evaluations: Improving Interview Data by Westat
Data Collection Overview: Interviews by CDC

Table (4.4). Data Collection Method: OBSERVATIONS

Observations are written documentation (usually completed by staff) of program events and/or participant interactions.

Data Type	Primarily collects qualitative data but can also collect quantitative data by numerically coding responses.
Examples of Data Sources	Program participant case notes Incident reports
Advantages of Observations	You can document your observations as the event is happening (or shortly after). Can be a helpful way to understand an ongoing process or behavior. Can reveal things about your program that you had not thought of before.
Challenges of Observations	This data is often not standardized and may be hard to analyze across different times and observers. Can be difficult to interpret and categorize observations.

OBSERVATIONS
can help you understand an ongoing process

Helpful Resources:
Collecting Evaluation Data – Direct Observation by University of Wisconsin

Table (4.5). Data Collection Method: DOCUMENT REVIEW

Document review involves the gathering of existing program documents, monitoring data and records.

Data Type	Primarily collects quantitative data but can also collect qualitative data in the form of documented narratives.
Examples of Data Sources	Sign-in sheets Enrollment forms Needs assessments Meeting minutes
Advantages of Document Review	Provides comprehensive program information in a way that does not interrupt regular program activities. Information already exists or is captured as a routine part of program monitoring.
Challenges of Document Review	Information may be incomplete or unreliable. This process can be time consuming. Data is restricted to what already exists and is not flexible.

DOCUMENT REVIEW
is comprehensive

Helpful Resources:
Data Collection Method: Document Review by CDC

Table (4.6). Data Collection Method: FOCUS GROUPS

Focus groups are conversational sessions that are conducted with a group of program participants, either in person or virtually, to gain an in-depth understanding of their experiences and opinions.

Data Type	Primarily collects qualitative data but can also collect quantitative data by numerically coding responses.
Examples of Data Sources	Documented question-and-answer sessions with a group that represents a population or type of program participant.
Advantages of Focus Groups	You can quickly and reliably collect common impressions about the program or program experience. Can be an efficient way to acquire a range of information within a short span of time. You can capture the perspective of key participants.
Challenges of Focus Groups	Locating a trained facilitator to lead these sessions can be difficult and/or costly. It may be difficult to schedule the session(s) and/or facilitate the attendance of participants.

FOCUS GROUPS
quickly capture a range of thoughts

Helpful Resources:
Data Collection Overview: Focus Groups by CDC
Using Focus Groups in Program Development and Evaluation by University of Kentucky

Determine Collection Method by Evaluation Toolkit

<< Back to
What Are Data Sources?

Continue to
Promoting the Reliability of Your Data >>