Are Catholic primary schools more effective than public primary schools?

There is considerable debate about whether Catholic schools in the U.S. impart academic or behavioral advantages to their students. Findings have been mixed. Though fairly strong and consistent positive findings have been found for Catholic high school outcomes, evidence for advantages of attending Catholic primary schools has been much more equivocal. This article attempts to take seriously the bane of school sector effects research, the issue of selection bias—that the population that ends up in Catholic schools is already advantaged and this explains away any positive findings for Catholic school outcomes. 

Data Notes
Using data from the Early Childhood Longitudinal Survey: Kindergarten Class of 1998-1999 (ECLS-K), Elder and Jepsen primarily consider the effects of Catholic primary schooling on eighth-grade math and science test scores as well as giving limited attention to outcomes of absenteeism, tardiness, and expulsion. The ECLS-K is a large national school-based survey of parents, students, teachers and administrators of randomly selected schools and is conducted by the National Center of Education Statistics (NCES), which is part of the Department of Education. The analysis is limited to a comparison of students in Catholic and public primary schools, and only considers students who spent their entire primary school career in the Catholic or in the public school sector. The full models include controls for kindergarten math and reading scores as well as extensive family and demographic controls, such as sex, race/ethnicity, age, birth weight, family income, parental education, marital status of child’s primary caregiver, and other family structure variables.

Catholic School Students Advantaged Before They Arrive at School
The descriptive findings show that in math and reading Catholic school students start well above public school students. This gap is not consistent throughout the primary school years and in fact shows some evidence of closing during primary school. After controlling for the advantages that Catholic school students have when they arrive at school, the analysis shows a negative effect of Catholic primary schools on mathematics test scores and a small positive effect of Catholic schools on reading scores. The authors argue, however, that the reading difference would likely disappear if full controls were available to eliminate the effects of selection bias in favor of Catholic schools. The reason the authors believe this is that there is a fairly strong correlation between the choice of Catholic school and variables that are available in the ECLS data. If we assume that there are important variables related to selection into Catholic schools that are not available in the ECLS-K (though the authors give us no theoretical reason to think that there are significant unobserved variables not captured by prior test scores and the extensive set of control variables), and if we assume that the correlation of Catholic school choice and these “unobserved” variables is significant but not overly large, we can infer that if we had measures of these “unobserved”  selection variables, the “real” difference in reading between Catholic and public school students would be zero at best. With these necessary controls, according to the authors, we may even find that public schools have a positive effect on reading scores compared to Catholic schools.

Five Conclusions
The article provides five key arguments regarding Catholic primary schooling. First, based on the descriptive data, Catholic school students enter schooling with large advantages in both mathematics and reading. Secondly, Catholic schools actually seem to decrease achievement in mathematics compared to public schools. Third, while there appears to be a small Catholic primary school advantage in reading outcomes, this is driven by selection bias. The fourth argument is that Catholic primary schooling does not appear to improve school-related behavior, such as absenteeism and tardiness. However, there does appear to be a small Catholic school advantage regarding expulsion. Finally, the authors offer an analysis of another national dataset, the National Education Longitudinal Survey, to support their claim that having controls for academic achievement prior to school entry is necessary to determine the presence or absence of a Catholic school effect.

Implications and Evaluation
The study suggests that any Catholic advantages over public school students seen in the eighth grade population are mostly due to differences that existed prior to kindergarten entry. The differences prior to entry, according to the authors, are likely related to the differences between families that are willing and able to pay tuition and fulfill the requirements for parental involvement in Catholic schools. A complete evaluation of this empirically sophisticated article will require more space than available here. A summary of some of our concerns follows.

On the plus side, it appears in their comparison of public and Catholic school student outcomes that the authors have conducted a detailed and thorough analysis of the ECLS-K data. This extensive work provides a very important contribution on the thorny issue of separating academic growth that is due to the school processes and organization from the effect of selection bias (which is essentially a parent or home effect rather than a school effect). The evidence is consistent with other analyses of Catholic primary school effects, which show no effect on academic outcomes or small negative effects on mathematics achievement.

The analysis is limited, however, to those that start and end in Catholic or public primary schools. Is there anything different (across sector) about stayers or leavers that might influence the results? Do school transitions vary across sectors and are they fully accounted for in this analysis? Further, the analysis is limited to Catholic school students who started in kindergarten. Is this an unusual group compared to all Catholic primary school students? And can we settle for an analysis that is unable to account for religious background of parents? We know that religious background influences educational outcomes, and conservative or traditional Catholic parents may be more likely to start in kindergarten and remain through 8^th grade. Would appropriate controls for family religious background lead to different conclusions?

More importantly, the article is hampered by the fact that it is focused on statistical analysis and almost devoid of theory. That is not necessarily a fundamental flaw, but it limits the broader implications of the piece, particularly regarding issues of school choice, such as whether government funding should flow to Catholic schools. We would need a new analysis that builds theories based on these findings and then tests these theories against another dataset in order to make more definitive conclusions useful to public policy debates.

The little theory that is offered on an ad hoc basis is underdeveloped. For example, to explain the negative effect of Catholic schools on math achievement, the authors mention at the end of the article that teacher pay differences between Catholic and public schools may lead to lower quality mathematics instruction in Catholic schools. Perhaps. But there is a lot of theoretical and empirical work to be done before we can make that conclusion, including research on the issue of whether quality teachers follow higher pay, on the extent to which “quality” teaching is an individual attribute that travels well, etc. Moreover, the authors do not consider what if anything lower teacher pay has to do with Catholic schooling per se. Does the Catholic mission and school organization insist on low pay for teachers? Are Catholic schools squandering their tuition dollars on administrator perks? Not likely. More likely is that schooling in the U.S. is expensive and religious schools must rely on donations and tuition dollars while competing with tuition-free public schools for students. Taking theory seriously would greatly increase the impact of the article’s findings.

It does not necessarily follow from these findings, then, that we should or should not allow government funds to flow to Catholic schools. If we only work with the theoretical tidbits offered in this article, we could as easily conclude that governments shouldn’t fund any school—whether public or private—that is not attracting high quality math teachers. Following the authors’ logic, we could as easily argue that government funding of students in Catholic schools will allow Catholic schools to blossom, since under the current system of funding Catholic schools lack adequate funding to attract good math teachers.

Our conclusion then is that we need better theoretical work to interpret the findings in this article. Finding positive correlations between attending Catholic school and tardiness is interesting, but without a theory about why Catholic schools would promote tardiness (a lack of seriousness about time on task? smaller schools don’t emphasize promptness? lack of public transportation? tardiness is measured more accurately in Catholic schools?), we are left calling for further research. This paper is a good start but does not settle the issues at stake.

Keywords: Catholic schools, public schools, achievement, selection effects, primary schools, mathematics

Sector: Catholic,  Public

Outcomes: Academic,  School Organization

Date Posted: 2014-06-11