School Sector and Academic Achievement: A Multilevel Analysis of NAEP Mathematics Data

The publication of the Christopher and Sarah Lubienski’s recent book, The Public School Advantage: Why Public Schools Outperform Private Schools, has sparked considerable debate on educational blogs. We take a look at the authors’ 2006 article, which provides the basis for a chapter in the book, considering the religious school effects. Are public schools outperforming religious schools in mathematics?

About the Study
Lubienski and Lubienski focus on the mathematics achievement of 4th and 8th graders in various school sectors in the United States, including traditional public, public charter, Catholic, Lutheran, “Conservative Christian,” and other private (presumably independent nonreligious schools). The study uses the 2003 National Assessment of Educational Progress (NAEP), which includes very large random samples of American students, allowing for an assessment of detailed school sector differences. In the analysis of the NAEP data, the study attempts to control for demographic and family background differences that may be related both to mathematics achievement and school sector. These include both individual characteristics as well as average values of socioeconomic variables at the school level, which attempts to control for “peer effects.”

Findings
The study finds some differences in fourth grade compared with eighth grade. In fourth grade, the non-public schools show higher levels of mathematics achievement before including statistical controls. But after adjusting these “raw” differences for the demographic and family background differences, which are likely to differ by school sector since families of higher socioeconomic status are more likely to choose private schools, the authors find that traditional public schools are doing slightly better than Catholic and Lutheran schools, and public charter schools. They are doing even better in comparison to “Conservative Christian” schools. The private-public school differences are not as stark when controlling for individual level demographic and family characteristics of students, but grow larger when the school-level averages for demographic and family background differences are included in the models (what the authors consider the “peer” effects).

Among eighth grade students, the sector differences are much narrower overall and the Lutheran and nonreligious independent school students are similar to public school students. Public charter schools appear to be slightly ahead of public school students on average, but this difference is not statistically significant (i.e., we cannot have high levels of confidence that it is not zero in the population). Again, public and private school differences are basically zero until the models include average values for demographic and family background in each school.

The analysis also finds evidence that racial and ethnic differences in achievement are large and similar across school sectors. They do not find evidence that any school sector does better or worse in reducing gaps between African Americans, Hispanics, and whites.

The authors conclude that school choice policies are highly suspect given the public school advantage in mathematics. They are particularly concerned that the “Conservative Christian” sector is the fastest growing private sector. (Actually, the evangelical Protestant sector has not seen much growth since the 1980s, though there has been significant growth of homeschooling among religious families.) If parents are choosing “Conservative Christian” schools that are low performing, the authors conclude, parents that have more choices under school choice plans will not necessarily send their children to schools that will improve the academic outcomes for their children.

Limitations
As the authors point out, this study is based on cross-sectional data. What we don’t have in this study is information about change or growth in mathematics achievement over a student’s career in a particular school. That makes it difficult to interpret the meaning of the findings, since we can’t say conclusively that these findings are truly school effects on achievement. A wide variety of alternative explanations need to be ruled out before drawing this conclusion with confidence.

And we should keep in mind that the study only considers mathematics outcomes, rather than a broader assessment of educational progress that includes reading, science, and civics education. As we will show in future reports, there are religious school differences in reading achievement for students in the elementary schools.

Other methodological concerns about the study and the authors’ subsequent book have been capably explained elsewhere (see http://educationnext.org/comparing-public-schools-private/). We focus on the religious school sector findings.

First, for those interested in the religious school findings, the relevant comparison in some cases may not be traditional public schools but all public schools, including charter schools and private independent schools. That analysis isn’t provided in this study.

The analysis also raises the question of what school controls should be included in the analysis, since only peer effects at the school level are considered in this study. The analysis controlled for the average socioeconomic status of the student body, under the argument that a higher average socioeconomic status in a school gives its individual students an advantage that should not be considered a true school effect. Of course, the analysis is not able to tell us whether changes in the composition of the student body because of a future expansion of school choice policies would lower religious school average scores as they appear to do in the public school sector. This is important to know, since the significant public and private school differences found in this study depend almost entirely on controls for “peer” effects.

Most likely, school choice plans would reduce socioeconomic differences across school sectors. But what other school sector differences would tend to become more equal under a greatly expanded school choice policy? What would religious schools do differently if they were funded at public school levels? Since the article attempts to draw implications of its findings for school choice debates, any difference between religious schools and public schools that is directly related to funding should be included as controls in the model. Hiring and retaining teachers, for example, may currently be more difficult in religious schools because of resource constraints. The authors mention that there is a large difference in the percentage of certified teachers in “Conservative Christian” compared to traditional public schools and that this variable is correlated with student mathematics achievement. That is, if greater public funding through school choice plans would increase the percentage of certified teachers in religious schools, then the appropriate analysis—if the authors want to speak to school choice debates—would include a control for the percentage of certified teachers in the school. That would allow an analysis that comes closer to what we would expect under a true “level playing field,” which would include equal funding. Other controls may also be necessary at the school level before considering school sector differences.

Are there other likely outcomes under expanded school choice plans? The study’s claim that a major outcome will be that parents pick “Conservative Christian” schools that are doing less well in math is overly simplistic to say the least and betrays a lack of serious engagement with the religious schooling literature.

That problem is reflected in the categories of religious schools used in this study, which are not discussed or defended in the text. Though the authors attempt to compare “Lutheran” and “Conservative Christian” sectors with traditional public schools it is not at all clear that these categories are meaningful. Most likely, the authors have adopted the National Center for Education Statistics definition of “Conservative Christian,” which is inadequate since it is based solely on the religious schooling association that is reported by each school in the Private School Survey. But not all religious schools are associated with a schooling association or would report this information in a government survey of private schools. By failing to use complete information on each religious school, the authors leave doubts as to whether the non-Catholic religious school findings are worth taking seriously. Further, as mentioned in the article, the participation rates for the religious schools are very low compared to other sectors, even falling below the standards for reporting set by the principal investigators of the NAEP.

In sum, the study is suggestive and important for stimulating new research that takes seriously the issues of diversity and measurement in the non-Catholic religious school sectors. The authors overplay their hand by assuming that their results undermine the entire case for increasing family choices in education, which does not rest on significantly higher private school scores on standardized tests in every grade. This serves as a reminder to advocates of increased schooling options for families to clarify the place test scores in a broader argument for considering school choice. And their findings on racial disparities in all schools call for continued and new efforts to provide equal educational opportunities for all. 

Keywords: sector effects, mathematics, elementary schools, school choice

Sector: Catholic,  Charter,  Evangelical Protestant,  Other Protestant,  Private,  Public

Outcomes: Academic,  School Choice,  School Organization

Date Posted: 2014-10-06