by Amanda Datnow and Vicki Park
Chapter One: The Promise and Pitfalls of Data-Driven Decision Making
Data are ubiquitous in our lives. Using the latest technologies, we can now quickly calculate how many steps we took in a day, how many calories we consumed, and how much money we spent and on what. Knowing this information will ideally help us make better decisions that will improve the quality of our lives. Businesses, health and education organizations, and governments can now quickly crunch big data to help them understand phenomena in ways never before possible.1 The power of data use is simple: armed with data, people will make better choices and organizations will function more effectively. This is the thinking behind a hot topic in educational reform: data-driven decision making.
A decade ago, data-driven decision making wasn’t on the radar of most educators or policymakers. Now it is difficult to imagine an educational reform agenda that does not include data use as a key pillar. The use of data has the potential to change teaching and learning. Teachers now have wider access to information learning and can address learning gaps before students fall behind. Data use can also build collective responsibility for all students. As student achievement results and teaching strategies are shared among teachers within and across grades, school cultures and routines are changing as well. Transparency is increasing and the culture of individualism that used to characterize classroom teaching is decreasing.
But how do we find the time to incorporate data use into the already incredibly busy professional lives of leaders and teachers? What gets pushed aside if teachers focus on data? How much training and support do educators need to use data effectively? How does data use fit with other reform agendas? Are we in danger of chasing the numbers and forgetting the central purpose of data use, which is to improve teaching and learning? Are we in danger of chasing the numbers and forgetting the central purpose of data use, which is to improve teaching and learning? Data-driven decision making is very popular in schools and districts across the United States, and there is also increasing emphasis on data use in other countries, including the Netherlands, Canada, Belgium, South Africa, Australia, and New Zealand.2 Although each place may take a different approach, the common idea is that when leaders and teachers become knowledgeable about how to use data in their work—when they collect and analyze data to guide educational decisions—they will become more effective in reviewing their existing capacities, identifying weaknesses, and charting plans for improvement. In the classroom, data can inform how teachers plan lessons, identify concepts for reteaching, and differentiate instruction.3
The push for educators to systematically gather and use data has brought with it a need to develop new competencies, skills, and cultures. But using data is not as straightforward as it seems. Leadership is essential in this endeavor. We can’t simply use data and expect good things to happen. Educational leaders play a critical role in shaping how and why data are used, what counts as data, and what people are aiming for when they push the use of data in schools. Although we titled this book Data-Driven Leadership, we strongly believe that data do not drive decisions by themselves. Individuals use data to engage in inquiry around current practices and inform courses of action. Data-informed leadership is thus a more appropriate term for what we’re asking leaders to do.4 And although the term data-driven decision making is commonly used in the field, from here onward we will refer to the practice as data-informed decision making to signal this important shift in thinking about data use. Leaders, we argue, should use data carefully to inform thoughtful decision making as part of an ongoing process of continuous improvement. Data use should not be seen as a passing fad or fancy. Leaders must take the initiative to assess what types of data are useful and for what purposes. Data-informed leadership aims to contribute to improving student achievement and teacher professionalism rather than threatening them.5
This book is written primarily for educational leaders at the school and district levels. It is geared toward leaders who are interested in becoming more data informed, as well as those who are well on the way and already feeling confident in their approach. Our aim is to provide a guide that will help build the reflective skills of leaders rather than offer a set of prescriptions about putting data use into practice. In order to help leaders get smarter about data use, we share research-based lessons learned from educators about how they have approached data use in their school systems. We examine how district and school leaders can create structures and cultures that support thoughtful engagement with data for continuous improvement. We also expose some of the potential land mines on the road to productive use of data for continuous learning and equity. Our intention is to help leaders avoid those problems and use data effectively and strategically in their decision making.
Perils and Perverse Incentives
All schools are already data informed in one sense or another. In the United States, the existing accountability system and its evaluation of schools based on student performance data expects and ensures it. Behind government accountability policies is the notion that educators need to know how to analyze, interpret, and use data so that they can make informed decisions about how to improve student achievement on state or national assessments. Within this, there is a strong policy emphasis on reducing the achievement gap, especially for historically underserved, low income students of color.
Data can be very powerful, but they also have hazards.6 There are some perverse incentives inherent in using accountability data within a high-stakes, limited-resource environment, which have led to some perilous practices and pitfalls, including these:
- Cheating on state tests
- Implementing quick fixes
- Targeting resources to students just below accountability thresholds
- Narrowing the curriculum
- Data overload
Cheating on State Tests
Accountability policies “invest faith in ever-increasing and voluminous amounts of numerical data collection. They can create an evidence base for an Orwellian system that can see everything, know everyone, and judge just when and where to intervene with any student, school or classroom, at any time.”7 Continual surveillance and high-stakes tests have resulted in a great deal of fear among many educators struggling to help their students show progress on state tests. In some cases, teachers and administrators have resorted to outright cheating by giving students the answers or changing students’ answers after they have completed the tests, before they are sent for scoring.8
Implementing Quick Fixes
Critics argue that the term data-driven decision making implies an overly technical model of professional action in which educators diagnose weaknesses and implement solutions in a linear fashion, ignoring the complexity of the teaching and learning process.9 As Andy Hargreaves and Dennis Shirley claim, this kind of focus on data use through numerical test score data can impair and impede the improvement of learning for all students:
With AYP [Adequate Yearly Progress] deadlines looming and
time running out, teachers have little chance to consider how
best to respond to figures in front of them. They find themselves
instead scrambling to apply instant solutions to all the students in the problematic cells—extra test-prep, new prescribed programs, or after-school and Saturday school sessions. There are few considered, professional developments here, just simplistic solutions driven by the scores and the political pressures behind them.10
This approach can lead to a focus on easy solutions rather than continuous development and substantive improvements.
Targeting Students Just below Accountability Thresholds
Accountability policies are meant to ensure that educators have high expectations for all students. They are intended to raise achievement across the board, regardless of students’ current ability levels. In what has been graphically termed “educational triage,” 11 some schools separate students they see as “nonurgent” from those who are “suitable for treatment” and those who are seen as hopeless cases.12 This is done in order to target resources and attention more economically and address the increasing emphasis on accountability. Educators frequently report focusing their efforts on students who are hovering near the cutoff point for proficiency.13 Teachers target for remediation and additional tutoring those on the cusp—or “bubble”—of scoring in the proficient range while the students below this level are sometimes considered lost causes.14 Similarly, observations of teachers’ data reflection meetings in four schools revealed that discussions overwhelmingly centered on helping students who were below 16 The resulting focus on specific groups of students at the expense of others has important implications for equity, especially with regard to equal opportunity to learn.
Narrowing of the Curriculum
A commonly documented concern arising from accountability policies is the narrowing of the curriculum. High-stakes testing has been found to sway schools to focus on math and language arts at the expense of other subjects, such as science or social studies.17 This occurs even when principals provide support for teacher professionalism and autonomy in instructional decision making.18 Some subjects get squeezed out altogether, and within targeted subjects, such as math and language arts, educators may spend their time teaching how to take tests—by emphasizing the styles and formats of state assessments, for example—rather than on what will be tested.19 Such strategies may produce short-term gains on test scores, but students ultimately learn less because
testing mastery is emphasized over learning.
The data collected and the expectations for using the data can far outweigh the supports necessary to make data use meaningful for instructional improvement and school change. Many schools are still at the basic stage of data use, relying on limited forms of data and simple processes of analysis.20 Yet school leaders are saturated with accountability data from state and federal systems and district benchmark assessments, schoolwide data on student and teacher performance, and data on school culture, pacing, curriculum, and resource allocation, not to mention data from research and program evaluations. When school administrators and teachers are drowning in a sea of data and lack the capacity to use them, they are more likely to discount data altogether or fall into a default mode of quick-fix decisions making without incorporating new evidence.
When educators are overloaded or focus on the use of data to avoid sanctions, they may inadvertently subvert the intended goals of data use and accountability policies. With these perils and perverse incentives in mind, let’s turn our attention to the promise of date use-the positives it can bring.
ABOUT THE BOOK:
This book is an essential guide to meeting the challenges of high-stakes accountability, building performance-based schools, and improving student outcomes.
By following the advice in this book, you’ll be able to transform data overload into a data-positive school culture. You’ll learn the difference between “data-driven leadership” and “data-informed leadership,” and how to use distributed leadership to inspire collaboration and guided analysis. Incorporating narrative reflections drawn from real educators and administrators, the authors refine their observations and interviews into practical conclusions that leaders can put to use immediately. This book empowers leaders to support inquiry, build trust in data-based initiatives, establish goals for evidence use, and provide educators with the skills they need to mobilize data for the good of all stakeholders.
“Datnow and Park’s ideas are easily accessible and grounded in clear examples, and their seven ‘calls’ about what needs to be done nail the problem and the solutions. Use this book as your action guide and you’ll be rewarded with better results in student learning.”
—Michael Fullan, professor emeritus, University of Toronto