Study: Charters Boost College-Going — Even When Test Scores Fall

Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter

A new study of charter schools in Massachusetts has identified strikingly positive academic results.

The paper, released last week through the National Bureau of Economic Research, finds that charter students in the Bay State are significantly more likely to enroll in a four-year college and obtain a degree than their non-charter peers.

But an odd wrinkle emerged: Students in urban charters also experience a noticeable bump in their test scores, while those enrolled outside cities actually see their scores fall. 

The overall effects offer yet more evidence that the Massachusetts charter sector, cited by some researchers as the highest-performing in the country, substantially improves the life outcomes of its charges. The state has long won praise for holding choice schools to high standards, shuttering programs that fall short of expectations and allowing only charter organizations with a proven record of success to open new campuses. By the 2000s, charter students in Boston had begun out-scoring children in much more affluent towns on math and English. 

But the new research seems to indicate a paradox. Unlike in Boston, charters in suburban and rural areas boost their students’ chances of attending and graduating from college while also dragging their test scores downwards. The divergent measures of educational achievement make it unclear exactly how the schools are working and what truly matters for kids.

Sarah Cohodes, a professor at the University of Michigan and the paper’s lead author, said her work reflects the simple reality that schools can change students’ lives in a multitude of ways.

The whole premise of test-based accountability is that test-scores predict longer-term outcomes. But this situation shows it is not always the case.

Sarah Cohodes, University of Michigan

“The whole premise of test-based accountability is that test-scores predict longer-term outcomes,” Cohodes wrote in an email. “And that is likely still the case, writ large. But this situation shows it is not always the case, and other things are going on in schools.” 

Cohodes’s analysis revisits the conclusions of a study published in 2013 by, among others, Nobel Prize-winning economist Joshua Angrist. That experiment showed Massachusetts’s urban charter schools significantly beating the results of nearby public schools, while non-urban charters lagged far behind local competition. 

The latest study makes use of the same sample of 15 urban charter schools and nine non-urban charter schools. It also relies on the same identifying data from the schools’ attendance lotteries, which include information on student race, class background, special education status, and previous scores on the state’s annual standardized test, the Massachusetts Comprehensive Assessment System (MCAS). 

Through the use of the school lotteries, which randomly assigned similar students to either receive a slot at a charter school or not, both studies are able to pinpoint the effects of enrollment. But Cohodes extended her observations further in time, capturing high school graduating classes between 2006 and 2018, and gathered further figures on college enrollment and completion from the National Student Clearinghouse. 

Importantly, she identified large differences between charter students based on whether or not they lived in a city. Black and Latino students made up 54 percent and 27 percent of applicants, respectively, at urban charters, while fully 90 percent of applicants to non-urban charters were white. Urban applicants were also much more likely to qualify for free or reduced-price lunch (a common proxy for poverty), and had previously scored considerably below the state average on MCAS; non-urban applicants tended to score above that average.

After two years of attending their charter school, urban students saw their scores in both math and English leap upwards compared with students in traditional public schools. By comparison, those in non-urban charters fell by somewhat smaller, though still significant, amounts.

That finding replicates both the results from the 2013 study in Massachusetts and those of several other investigations, which have broadly pointed to a divide between urban charters and those in rural and suburban areas. The consistency of the result suggested to some observers that it could simply be easier to create a charter school that improves upon existing offerings; in more advantaged areas, however, newer alternatives must compete against schools where students already score fairly well.

Surprisingly, though, the same students whose scores fell in non-urban charters were also 11 percentage points more likely to enroll in a four-year college than their counterparts in traditional public schools. They were also 10 points more likely to attain a bachelor’s degree. Urban charter students also saw their college chances improve — 27 percent earned a bachelor’s degree within six years of graduating high school, compared with 23 percent of their peers in non-charters — but the effect was only about half that enjoyed by students outside of cities.

What could account for the difference? According to Jon Valant, a political scientist who leads the Brookings Institution’s Brown Center on Education Policy, charter-curious families in non-urban areas could be selecting for schools that don’t focus explicitly on raising test scores. Instead, their target schools might attempt to set themselves apart through a focus on the arts or social-emotional learning. Such an emphasis could boost chances of college completion while also leading to lower academic achievement in the short-run.

“In those areas, parents might not be looking for schools that are better at doing the same things as their local public schools,” Valant wrote in an email. “They might be looking for schools that do something different — even if that comes at the expense of their state test scores.”

Parents might not be looking for schools that are better at doing the same things as their local public schools. They might be looking for schools that do something different.

Jon Valant, Brookings Institution

That sentiment was echoed by Macke Raymond, the director of Stanford’s Center for Research on Education Outcomes (CREDO), which conducts comprehensive reviews of charter school performance around the United States. While cautioning that the Cohodes study’s sample of just a few dozen schools made its findings difficult to generalize, Raymond argued in an email that suburban parents often strike a bargain when selecting charters: The alternative school model might provide academic and social resources that help their children excel in college, even while their explicit focus on core subjects falls somewhat behind that of local schools.

“Our team has seen that many non-urban charter schools across the country intentionally offer students an experience that does not focus on maximizing academic gains,” Raymond wrote. “Knowing that families are well resourced, suburban charters may offer a different experience, either with thematic focus or emphasize an environment that stresses non-academic development of their students.”

Many non-urban charter schools intentionally offer students an experience that does not focus on maximizing academic gains. Knowing that families are well resourced, suburban charters may offer a different experience.

Macke Raymond, Stanford University

For her part, Cohodes said that while the correlation between test scores and later-life success is solidly established, it was important not to dismiss educational programs too hastily on the basis of setbacks on student assessments. She and her colleagues plan to conduct a follow-up study examining the practices in non-urban charters that might be contributing to their students’ post-secondary attainment, including smaller class sizes and college counseling.

“I think it’s important to find school models that work, and to define ‘work’ broadly such that it does not incorporate only test scores,” Cohodes said. “And I think we should replicate and expand school models that work, regardless of the sector.”

Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter