Flaws of charter school studies

STUDIES THAT GRADE CHARTER SCHOOLS RELY ON IMPERFECT MATH; November 20, 2010; Wall Street Journal 

Charter schools are getting very confusing report cards.

Researchers have assessed the thousands of charter schools that have opened around the U.S. in the last two decades. The results from those studies are starting to flood in. But policy makers hoping to learn whether these scholastic experiments have been successful will be disappointed: Some studies say charter schools are outperforming their traditional counterparts. Other studies put charter and conventional schools on par, or even show charters trailing their peers.

The chief explanation for the lack of consensus is that the prominent studies on charter schools rely on different methodologies—all of which have flaws…

To study charters, researchers typically measure how well students score on standardized tests compared with their peers at nearby traditional public schools. And head-to-head comparisons give charters a big edge. In 2007, for instance, charter schools in Georgia had graduation rates of 90%, compared with 72% at conventional schools, according to data compiled by the Center for Education Reform, a pro-charter group.

Such comparisons rest on a major assumption: that students enter each type of school on equal footing. But many researchers note that students choose to apply for charters, which might indicate greater motivation.

Researchers have sought to control for the differences in student bodies using one of three approaches. The first compares students who won lotteries for placement in charter schools with lottery losers…

The drawback of using lottery data to gauge charter schools is they can be applied only to schools that are popular enough to require a lottery, and that keep good records of the process. Schools in such high demand could be better than other charter schools, which could help explain the strong results…

The think tank RAND used a different approach in its study last year of charters in eight states, examining students who switched between traditional and charter schools. Researchers estimated students' progress prior to enrolling in a charter school, then checked to see if their progress increased or fell compared with their peers after switching. They found no overall improvement from charter schools.

The advantage of this analysis is that it can look at schools without lotteries. But the method relies heavily on switching students, whose experience might not apply to all their peers.

One of the most controversial studies in the field is being run by Stanford University's Center for Research on Education Outcomes. The center doesn't take a position on charter schools but does receive funding from foundations that support charter schools.

Researchers examining test results of 70% of the students attending charter schools nationwide found comparable students in conventional schools, matching them based on demographics, English proficiency and participation in special-education or school-lunch programs. If these matching students did worse than the charter students, then the charter school was rated a success.

So far, the Stanford researchers have found that just 17% of charter schools posted academic gains compared with traditional schools, while 37% fared worse.

Charter-school skeptics have jumped on these results. But a closer look at the study reveals a potential methodological problem. There could be differences between the two pools of students, such as parental involvement or drug use, not accounted for in the study…