CCE staff and partner reflections on our collaborative work to create schools where learning is engaging and rewarding, and every student is set up for success.
What to Ask Instead of "How Was School Today?": Reliable Measures for School Quality
Researchers have dedicated decades to developing assessments (e.g. SAT, ACT, MCAS) in an attempt to measure students’ academic abilities, and these surveys have become common currency in education policy and research. Notably, relatively little attention has been dedicated to measuring aspects of school quality beyond academic content knowledge. This has begun to change with more recent interest in constructs less like mastery of algebra, and more like the quality of student-teacher relationships. Yet some remain skeptical: can surveys reliably capture these less traditional qualities of education?
This is precisely the question undertaken by the Massachusetts Consortium for Innovative Education Assessment (MCIEA), a joint project in seven school districts and their teachers unions, working to develop measures that reflect community interests and priorities. And, according to a new white paper analyzing the survey questions on MCIEA’s teacher and student perception surveys—surveys that target roughly three dozen unique dimensions of school quality—the answer seems to be “yes, they can.”
During the 2016-2017 school year, MCIEA administered these surveys to 25,000 students and 5,500 teachers across six school districts, asking broadly about the social and material conditions of their schools. The purpose of the surveys was to measure the experience of schools beyond standardized test scores – an effort that has garnered increasing attention from policymakers and researchers in recent years. But regardless of how important or valuable measuring qualities like teacher-principal trust or student sense of belonging may be for educators and policymakers, the question remains whether a survey can accurately capture such qualities.
One standard way for researchers to address this concern is by measuring “internal coherence” – looking at how closely responses to a set of related questions align with each other. Take for example “teacher-principal trust”. On the MCIEA perception survey, we tried to capture this construct by asking teachers the following four questions:
|1||To what extent do you trust your principal at his or her word?|
|2||At your school, how comfortable are you raising concerns with the principal?|
|3||How much do you trust your principal to stand up for you in disagreements with parents?|
|4||To what extent do you think your principal has the best interests of the school in mind?|
If every teacher who answered positively on the first three questions answered negatively on the fourth question, or vice versa, we might worry that the first three questions are measuring something different from the fourth, rather than the single dimension of teacher-principal trust as intended. On the other hand, if teachers tended to respond similarly to all four of these questions, we would be more likely to believe the questions pinpointed the same quality.
Encouragingly, of the 36 surveyed dimensions, all but one (“Growth Mindset”) performed near or above standard thresholds for internal coherence. The high levels of reliability observed should bolster confidence that the survey questions coherently describe each of the dimensions and encourage further investigations into how these metrics can be meaningfully applied toward understanding school quality and contributing to school and district improvement.
We caution, though, that the results of this analysis do not suggest that the survey is ready for applications outside of research and development. We used the aforementioned threshold for internal coherence only as a signal for how to prioritize our attention while revising the survey for its next iteration (which has since been distributed for the 2017-2018 school year). The threshold for how reliable the survey must be to justify its use in policy decisions remains an open question that will require input from educators, administrators, students, and their families. The higher the stakes involved in the survey’s use, the higher our expectations should be that the results of the survey are trustworthy and meaningful.
Accurately measuring the various dimensions of school quality can help education systems improve student and teacher experiences and help stakeholders better understand the elements of school performance. Our analyses suggest that MCIEA’s School Quality Measures project has made promising inroads in addressing this gap in current educational scholarship and practice. We expect continued work on this project to provide important data on educational practices in MCIEA schools and valuable insight into the full range of factors that make schools good places to teach and learn.