VOUCHING FOR COMPETITION:

Competition Passes the Test (JAY P. GREENE & MARCUS A. WINTERS, Summer 2004, Education Next)

Do public schools respond to competition from private schools by improving the quality of instruction? This is one of the key questions in the voucher debate. Advocates of vouchers believe that public schools facing the threat of losing students and funding to private schools will take the measures necessary to raise student performance. Opponents worry that vouchers will actually leave public schools worse off by draining them of funds and encouraging the best students and the most involved parents to flee a failing school.

Florida's A+ program affords a unique opportunity to test these competing predictions. The A+ program offers all the students in schools that chronically fail the Florida Comprehensive Assessment Test (FCAT) the opportunity to use a voucher to transfer to a private school. Schools face the threat of vouchers only if they are failing. They can remove the threat by improving their test scores. Comparing the performance of schools that were threatened with vouchers and the performance of those that faced no such threat gives a measure of how public schools respond to competition. [...]

Between the 2001-02 and 2002-03 administrations of the FCAT, voucher-eligible schools made the largest gain among the five categories of schools. In mathematics they improved by 15.1 scale-score points more than the rest of Florida's public schools. (Results on the FCAT are reported as the cohort change in mean scale score on a scale from 100 to 500. The median school in Florida had a mean scale score of 291 on the reading test and 300 on the math test. Schools at the 5th percentile of schools in Florida had a reading scale score of 243 and a math scale score of 247, while the 95th percentile school had a reading score of 327 and a math score of 328.) On the Stanford-9 math test, voucher-eligible schools achieved gains that were 5.9 percentile points greater than the year-to-year gains achieved by other Florida public schools. Results on the Stanford-9 are reported as the cohort change in national percentile rank.

Voucher-threatened schools made the next highest relative gains: 9.2 scale-score points on the math FCAT and 3.5 percentile points on the Stanford-9 in math. Each of these results is statistically significant at a very high level, meaning that we can be highly confident that the test-score gains made by schools facing the actuality or prospect of voucher competition were larger than the gains made by other public schools. As hypothesized, actual voucher competition produced the largest improvements in test scores, while the prospect of facing voucher competition produced somewhat smaller gains.

The results for the always-D and sometimes-D schools were also consistent with our hypotheses. Always-D schools, which, faced with the real danger of receiving their first F, had some incentive to improve, made a relative gain of 4.3 scale-score points on the math FCAT and 1.3 percentile points on the Stanford-9 math test. The sometimes-D schools experienced year-to-year changes in FCAT math scores that were only 2.4 points higher than all other Florida public schools, significantly less than the gains in both voucher-eligible and voucher-threatened schools. Their improvement relative to all public schools on the Stanford-9 was less than a percentile point. Formerly threatened schools saw no improvement in their math scores relative to all public schools.

The patterns were similar in reading, though the relative gains made by schools facing voucher competition were smaller and sometimes statistically insignificant. Overall on the FCAT reading test, voucher-eligible schools gained 5.2 points more than other schools gained. However, this gain fell barely short of a conventional standard for statistical significance, likely due to the very small number of schools in this category (only nine). Voucher-eligible schools also made a statistically insignificant relative gain of 2.2 percentile points on the Stanford-9.

Voucher-threatened schools actually made the greatest gains on the FCAT reading test: 6.1 points. Their relative gain on the Stanford-9 was a statistically significant 1.7 percentile points.

Always-D schools made no statistically significant gains on the FCAT or Stanford-9 reading tests, while sometimes-D schools experienced a decrease of 1.1 points on the FCAT and no significant change on the Stanford-9 reading test. We also found a relative loss of 3.8 points for formerly threatened schools on the FCAT and a relative loss of 1.6 percentile points on the Stanford-9 (both results were statistically significant).

Overall, the schools facing either the prospect or the reality of vouchers made substantial gains compared with the results achieved by the rest of Florida's public schools. They also made strong gains relative to those earned by schools serving similar student populations, which had nonetheless avoided receiving an F.

The smaller gains achieved by always-D and sometimes-D schools compared with the performance of voucher-eligible and voucher-threatened schools, despite the similar characteristics of all these schools, strengthen our confidence that voucher competition is the cause of the improvements. Always-D schools, in particular, are very similar to voucher-eligible and voucher-threatened schools in their initial test scores, student populations, and resources, as well as other unobserved factors for which we could not adjust the data. Since it is essentially by chance that always-D schools do not receive an F, the comparison approximates a randomized experiment. Yet the schools that faced voucher competition experienced much larger increases in test scores.

Moreover, the similarity of our findings on the Stanford-9 and FCAT math tests suggests that the gains being made by schools facing voucher competition are the result of real learning and not simply manipulations of the state's high-stakes testing system. If schools facing voucher competition were only appearing to improve by somehow manipulating Florida's high-stakes testing system, we would not have seen a corresponding improvement on another test that no one had incentives to manipulate.

Who'd have dreamt Adam Smith was right?

Posted by Orrin Judd at May 11, 2004 7:41 AM