This has been a good year for evidence on the effectiveness of charters, highlighted by a major national study from CREDO and a new study in the continuing work from New York City. Nonetheless, understanding and interpreting the scientific research within the political and media environment is made more difficult by the political context. Charter schools have received considerable attention since President Obama put them on the administration’s policy agenda. With the increasingly high stakes generated by inclusion under the Race to the Top, people have intensively searched the existing research evidence – sometimes with the intent of understanding the potential impacts but perhaps more frequently with an eye to supporting one or the other political side. The CREDO study assessed the performance of charter schools compared to traditional public schools across 15 states and the District of Columbia. It used an innovative matching technique to create control students in the traditional schools who were similar to those who chose charters. The results from this study showed a number of charters (17%) doing significantly better (at the 95% level) than the traditional public schools that fed the charters, but there was an even larger group of charters (37%) doing significantly worse in terms of reading and math. The remainder did not do significantly better or worse. These results were greeted with mixed emotions. The majority of researchers and policy makers were not overly surprised. They saw that there were success stories but that further work would need to be done to ensure that more of the good charters flourished and fewer of the bad charters remained (just as the case with traditional public schools). [Full disclosure: Macke Raymond, the lead author on the CREDO study, is my wife, so I know more about these studies than the random reader]. The study of New York City charters offered a different conclusion. The analysis by Hoxby, Murarka, and Kang (HMK) employed a different methodology. With sufficient popularity among parents so as to attract excess demand for enrollment, NYC charters are required to use a lottery to decide who is chosen to enroll (with exceptions for siblings and some other circumstances). HMK traced students who were “lotteried out” into the traditional public schools and compared their subsequent performance to those who had entered the charter schools. They found that the majority of students (86%) attended schools that had a positive effect (although it is not reported what this means for the number of schools or the statistical significance.) Thus, in New York City the charter experience appears notably more favorable than that for the rest of the nation. Some people believe that the results for these two studies should be the same. If not, one of the studies must be wrong. The media were further confused by a memo that Caroline Hoxby released that suggested an error in the statistical estimation used by CREDO and that this error would significantly bias downward the impact of charter schools. This memo, by implication, suggested that the rest of the country might actually look like the NYC results. The CREDO response to this points out, however, that the Hoxby memo is built upon an incorrect statement of the estimation approach by CREDO and an incorrect derivation of the statistical results – thus leaving the difference in results intact. (Both sides of the exchange can be found here.) In reality, expecting the results of the two studies to be the same is not the right way to look at them. The studies ask very different questions, and there is no reason to expect them to provide the same results. The CREDO study asks how well a typical charter school student across the sixteen separate state policy environments does compared to the counterfactual of attending a traditional public school. The HMK study investigates how well charter school students do when attending schools popular enough with parents to be oversubscribed compared to attending a traditional NYC public school. Thus, the NYC study can be thought of as proof that the best charter schools, as judged by parents, can dramatically outperform the alternative traditional school. That is important information, but it is impossible to know how to generalize it to other environments with different state laws, different union contracts, different district governance, different financing arrangements, and the like. Just on the surface, nobody would think that it was possible to generalize from NYC (one million students) to LA (700,000 students), let alone to Kansas City (20,000 students). Understanding the factors that make NYC charter schools perform so well relative to their traditional schools is an extraordinarily important research and policy question. It should clearly be at the top of the research agenda. Indeed, those charter school advocates who believe that there are important differences in state laws or that there is something special about KIPP schools already know that differences exist, even if the details are not well-understood. Unfortunately, this is also a subtle issue that the media and the policy community have trouble understanding. To many in the media, both studies sound like they are estimating the effectiveness of charter schools or maybe even the impact of school choice – so shouldn’t the answer be the same? While we have learned a lot from the new studies, we still remain in a situation with an unresolved key question about what policies, laws, and incentives lead some charters to flourish and others not.