Why We Value Multiple Sources of Data

“. . . the facts are always friendly. Every bit of evidence that one can acquire, in any area, leads one that much closer to what is true.” 

-Carl Rogers

Photo by Hans-Peter Gauster on Unsplash

The push for open science points to this sentiment: more evidence, provided that we use it well, helps us learn and accomplish goals together. 

The open science movement calls for scientific information and data to be shared freely and transparently so scientists can use it to collaborate. Proponents of open science argue that freely shared data will lead to more efficient and cost effective studies, and more equity in the field of research.

Working to promote equity in schools presents a similar opportunity. Thankfully, with every passing year educators have more data available to them. But studies suggest that looking at data, on its own, isn’t enough. As one such study notes, “Technology is never the agent of change—people are.” So how do agents of change in our schools learn to use the growing amount of data they have available to them so change can happen? 

In his book Superforecasting, Philip Tetlock summarizes his research about experts and their ability to forecast future events. Tetlock categorizes forecasters into two groups. 

The first group focuses relentlessly on one big idea or ideology. The members of this group failed to beat a blind guess in their forecasts. The second group took the opposite approach. Rather than seek information based on a single philosophy, they drew from multiple sources of information and used a variety of analytic tools to solve problems. The members of this group did better than a blind guess in their forecasts, though only by a margin (the underlying message: forecasting is very, very difficult). 

Without a doubt, the members of this second group faced a cost for their superior forecasts: they had to make sense of multiple and possibly conflicting data points. When different data tools estimate the same phenomenon, they may get different results. Consider differing temperature estimates from weather forecasts in the same area. Or differing opinion polling on the same ballot issue. Or research studies that show that drinking coffee may have benefits, alongside research studies that show that drinking coffee may have health risks. 

Each of these challenging topics represents a version of what we’re seeing today with education data. And as agents of change that want to be more like the winning forecasters in Tetlock’s research, we’ll need to meet the growing volume of incoming education data with some new skills. 

A Key Question for Education Data Use

In the early 2000s, data became an important and more frequent part of conversations in education. State testing, formative assessments, and demographic data fueled the proliferation of dashboards and other reporting. Consequently, education now faces the same opportunities and challenges as the rest of the world: we have more data to support our decision-making and we’re learning the best ways to combine it all. 

Education-Trust West, the California School Dashboard, the Improvement Data Center, DataQuest, and our own Equity Dispro Data System project are just a few examples of data that California educators can use. The important question is less “How can we get more data?” and more “How can we make sense of all this data so it supports our equity work?” 

We can collectively take charge of where we go from here. As we learn how to use this new data to tell the story of our students, we might heed the advice of Brené Brown in Daring Bravely: “If you own this story you get to write the ending.”

Multiple Sources of Data in ED&D Projects

The Equity, Disproportionality, and Design project, like other leaders in the California System of Support, rely on workflows to deliver services that promote equity in schools. All these workflows have something in common. Whether it’s using design thinking to develop new services, using improvement science to understand systems, or collaborating with other teams to create exciting ideas, our work requires us to draw from evidence to drive our decisions and designs. 

The ED&D team has a number of projects underway that speak to this idea. We’re building a research portal on our website to gather studies we’ve found useful preventing disproportionality in schools. We’re exploring a collaboration with the Systems Improvement Leads to build capacity in SELPAs for synthesizing multiple data sources and data tools. And we’re including lessons about interpretation of multiple data points in our EDDS Onboarding training series. 

Using Multiple Sources of Data: An Example 

Here’s an example from the EDDS Onboarding training series. Inequality in special education outcomes are measured by a comparison of rates. This comparison of rates is communicated in a single number, called a risk ratio score. The higher the risk ratio score, the higher the level of disproportionality for the students in that subgroup. Here’s a hypothetical example: a risk ratio score of 2.5 for a student subgroup identified with a disability suggests that students in this subgroup have an identification rate that is 2.5 times higher than other students.

Let’s say you’re looking at risk ratio scores from three independent estimates of the same student subgroup. And let’s say these estimates are different. How do we make sense of this? 

When we train SELPA leaders on this topic, we boil it down to three questions: 

• How different are these numbers from each other? 

• Are these numbers pointing in the same direction, even if they’re different? 

• Can we act on these numbers?

When estimates are different but point in the same direction, we can still learn from them. Take these three hypothetical estimates of disproportionality, each a risk ratio score of the same student subgroup: 

4.0

6.2

3.4

These risk ratios are different, but their direction is similar. Each estimate suggests this hypothetical student subgroup is identified for special education at a rate that is higher than other student subgroups. Figuring out if it’s four times higher, six times higher, or three times higher requires more research. But all signs point to a conversation we need to have about why our system is producing results like these.

With that information, we can proceed by using any number of tools to examine how our systems produce these results: root cause analysis, process mapping, and empathy interviews are just some. 

When we relieve ourselves of the burden of picking the perfect source of data and instead practice the discipline of synthesizing multiple sources, we can access the real world problems we seek to solve more completely. And more importantly, we can move from analysis to action. 

Conclusion

In the 1960s, scientists set a record by developing a vaccine for the mumps in four years. About 60 years later, in September 2020, scientists uploaded 100,000 SARS-CoV-2 genomic sequences to a database so international colleagues could develop a vaccine for COVID-19. Consequently, in 2021, scientists shattered the previous record by developing multiple vaccines in less than one year. 

This is a story of how collaboration, open sharing of information, and full use of available data drove a record setting effort to address a global pandemic. But these practices are only exemplified in this remarkable accomplishment. Their use is not limited to the development of vaccines. 

We can apply the same commitment to using all the data we have and collaborating to develop services that give students access to their full social and academic potential. We can own the story of how data is used to promote equity in schools. 

Notes

Proponents of this movement: Ali-Khan, Sarah E. et al. “Defining Success in Open Science.” Bill and Melinda Gates Foundation. 9 Feb. 2018, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5852639/ 

The push for open science: Can “‘Open Science’ speed up the search for a COVID-19 vaccine? 5 things you need to know.” UN News. 10 Nov. 2020, https://news.un.org/en/story/2020/11/1077162

educators have more data: This forum guide cites research from IBM that showed 90 percent of the world’s data came to us in the last two years at the time of writing. National Forum on Education Statistics. “Forum Guide to Data Visualization.” US Department of Education. 2016,  https://nces.ed.gov/pubs2017/NFES2017016.pdf

looking at data, on its own, isn’t enough: Wayman, Jeffrey C. et al. “Longitudinal Effects of Teacher Use of a Computer Data System on Student Achievement.” AERA Open. 20 Jan. 2017. 

“Technology is never the agent of change—people are.” Wayman, Jeffrey C. et al. “Longitudinal Effects of Teacher Use of a Computer Data System on Student Achievement.” AERA Open. 20 Jan. 2017.

In his book Superforecasting: Tetlock, Philip. Superforecasting: The Art and Science of Prediction. 13 Sep. 2016.   

research studies that show that drinking coffee: Smith AP et al. “Investigation of the effects of coffee on alertness and performance during the day and night.” Neuropsychobiology. 1993, 27(4), 217-23, https://www.health.harvard.edu/blog/the-latest-scoop-on-the-health-benefits-of-coffee-2017092512429 

data became an important and more frequent part of our conversations: Many scholars have studied how the No Child Left Behind Act of 2001 influenced the use of data in schools, including Smith, E. “Raising standards in American schools: the case of No Child Left Behind.” Journal of Education Policy, 2005; Ravitch, Diane. The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education. 2011, Basic Books. 

“If you own this story you get to write the ending.” Brown, Brené. Daring Greatly. 2012, Avery.

just a few examples of data: Institute of Education Sciences maintains a list of different data tools that’s worth checking out. Institute of Education Sciences. “Data Tools.” Institute of Education Sciences, retrieved 16 Mar. 2021, https://nces.ed.gov/datatools/

any number of tools: For more on root cause analysis, please see the System Improvement Leads training series on root cause analysis. Systems Improvement Leads, “Professional Development.” Systems Improvement Leads, retrieved 16 Mar. 2021, https://systemimprovement.org/professional-learning.  Also see Fergus, Edward. Solving Disproportionality and Achieving Equity, Corwin, 2016; Bryk, Anthony S. et al. Learning to Improve: How America’s Schools Can Get Better at Getting Better, Harvard Education Press, 2015.

uploaded 100,000 SARS-CoV-2 genomic sequences: American Society of Microbiology, “SARS-CoV-2 Sequencing Data: The Devil Is in the Genomic Detail”, American Society for Microbiology, retrieved 16 Mar. 2021, https://asm.org/Articles/2020/October/SARS-CoV-2-Sequencing-Data-The-Devil-Is-in-the-Gen

global rollout of multiple vaccination initiatives: Ritchie, Hannah et al. “Coronavirus (COVID-19) Vaccinations.” Our World In Data, retrieved 16 Mar. 2021, https://ourworldindata.org/covid-vaccinations

scientists set a record by developing a vaccine for the mumps: Cohen, Sandy. “The fastest vaccine in history.” UCLA Health. 10 Dec. 2020, https://connect.uclahealth.org/2020/12/10/the-fastest-vaccine-in-history/