Finding the Answers: A Look at Research Synthesis
By Erica Marsh | December 3, 2014
In my family, libraries were more than buildings that housed and loaned books. They were places with unrestricted access to incredible tools. They nurtured our passion for finding, organizing and sharing information.
Knowledge is serious stuff in my family. My grandmother, mother and sister were librarians. My father was an American Literature professor and author. My younger brother currently works at a library.
When I was growing up, whenever a question came up that my parents didn’t know the answer to, my mom would say, “Let’s find out!” She would call the reference desk at the local public library. We called this number so often it was posted by the phone.
This childhood ritual morphed into a guiding principle for my adult professional life: you don’t have to know everything, you just need to know where to look to find the answers. Nowadays, Google or Wikipedia are sufficient for basic questions. But those should only be starting points for research inquires, before more rigorous digging occurs.
From Card Catalogs to Web Content Management
Needless to say, I love information and the resources that provide it. I’ve come to have a special affinity for helping people find quality content through better organization, such as creating and using taxonomies to their fullest potential. To me, a taxonomy is no longer just the Dewey Decimal System from the public library of my youth (10 classes, each divided into 10 divisions, each having 10 sections). Today, every website navigation scheme has an underlying taxonomy. Whether it was strategically planned or organically grown, or is simply implied, it allows for easier and better navigating and searching.
In my 10 years of working at ETR, I’ve helped create and develop website taxonomies, as well as found, assessed and added, tagged, and shared thousands of content items for multiple sites.
Synthesizing Research Studies
Now I have the great pleasure of working with ETR’s Research Department on an exciting synthesis project funded by the National Science Foundation. It focuses on finding out more about the benefits for youth who design and program their own computer games, particularly in terms of their learning and motivation.
So far, some of my inherited skills (attention to detail) and interests (finding/organizing/analyzing information) have been useful. They also counterbalance my lack of familiarity with the content area. I’ve been told that my “beginner’s mind” allows me to garner “cleaner” data because it doesn’t have a layer of interpretation added.
Asking the Right Questions
Previously, our synthesis project has asked:
- What are the outcomes of children doing computer game design and programming (CGDP)?
- What strategies work best?
- What do the kids get (or not get) from certain pedagogies or learning systems?
- How do the outcomes and pedagogy interact? How do they inform or contradict each other?
- How do we boost the commitment and involvement of girls and other underrepresented youth?
Today, we are asking our audience of researchers, educators and youth advocates which they’d rather have:
- A synthesis of research on the kinds of learning and motivation CGDP is best suited to address.
-OR-
- A synthesis that shows whether there is evidence supporting common “lore” about CGDP (e.g., widely held beliefs about the benefits of the learning or the best ways to teach).
(If you have an opinion on this one, let us know!)
Another segment of ETR’s research and program work is in sexual and reproductive health. We’ve worked in this area for over 30 years. When we started out, we had approaches that seemed effective, but it wasn’t until they were heavily researched that the field could consider them true evidence-based strategies that definitely work.
It’s past time to do this same level of research for the field of CGDP for children. The first implementations of CGDP projects and programs for youth started decades ago. More educators are joining in all the time because there is a consensus that the programs are working. But what research exists, and what research still needs to happen, to prove that these strategies work?
Our synthesis team hopes to identify the gaps in the research and fill the role of proof-provider in the field.
Seeing Patterns in the Data
Since this unique synthesis project includes looking at data that relates to—and perhaps reinforces—the lore about CGDP and learning, we need to identify the claims that are frequently made (e.g., CGDP gets girls and other underrepresented groups interested in more computer courses and careers; CGDP engages kids more than just standard code programming; since kids like to play games, they’ll like to make games) and describe any current evidence that exists to support those claims.
Much of the research that’s been done is qualitative (descriptive) rather than quantitative (numbers based). This is a much more challenging body of data to wrestle with when trying to systematically review hundreds of studies. Luckily the same principles that help librarians sort and manage information can be useful in a synthesis approach to data, and in our case it’s particularly valuable with this overabundance of qualitative data.
Research syntheses often exclude qualitative data because incorporating it poses so many methodological problems. Many syntheses that do include qualitative data don’t document details about their methods.
But qualitative data acknowledges complexity and context. Excluding it from systematic reviews may neglect useful information or even distort the evidence base. So we are currently looking for themes to help us include qualitative data in our review. We’re also making sure to carefully detail our decisions along the way, so others can replicate a similar synthesis strategy in the future.
Understanding What the Answers Mean
Synthesis allows information to be analyzed that would not be available through standard statistical analysis. Although the synthesis process can be applied to any question or topic, we are taking these five steps toward answers to our initial research questions:
- Collect articles and reports that seem to address our questions. We’ve gathered around 400 total, but of course, have read hundreds more.
- Select those that meet the necessary criteria for data extraction (e.g., K–12 youth designing and programming their own computer games). We assessed and picked about 175 total.
- Extract the relevant data. For example, how many participants did the study have? What methods were used? Were there stated outcomes? Are learning outcomes included? Are the findings generalizable?
- Analyze. What inferences or conclusions can we draw from this synthesis? We are just starting this part of the process, beginning by discussing and making decisions about the quality of the extracted data.
- Share. Spread the results of our efforts in various ways (e.g., publish, present, blog).
This has been an exciting process so far. I look forward to continued work that contributes to the field’s understanding of the value of children learning through CGDP, and how to best approach the learning so it has the most positive and permanent impact on students.
Erica Marsh is a Content Specialist at ETR, and she carries at least three library cards with her at all times. She can be reached at ericam@etr.org.
