Class size and student performance: Relying on naive regressions

Last month, bold headlines dominated the German media: "Learning outcomes do not depend on class size" (SPIEGEL), "Size does not matter" (SZ) or "Learning success does not depend on class size" (Focus), leading newspapers titled. 

Reading the articles, the reader is re-assured that the findings are scientific, based on a study headed by Prof Bos from the Technical University of Dortmund - and that the study has been presented to leading policy makers and circulated around various Ministries. While the report is not published yet, reviewing the older IGLU reports suggests that the inference is rather naive.

The SPIEGEL for instance, generally described the approach in these terms: "The education scientists calculated how X affected Y, given that the children had the same abilities and reading scores". Now in econometric terms, this is nothing else but regressing educational outcomes on class size, assuming that class size is exogenous and controlling for a bunch of socioeconomic indicators. 

Now why is this naive? Let me give you an anecdotal evidence based on my time as a pupil in primary schooling: Our school was based next to a public housing for asylum seekers. As schooling was mandatory, many children from the asylum housing were allocated to our elementary school. Because most of them (including myself) could not speak German fluently, we were all assigned to smaller classes. Obviously it is very hard to get any good scores (especially in reading) if you could barely speak the language, and the test results would be relatively poor. In this case, it it not surprising that a smaller class size is not necessarily associated with better performance.

In technical terms, the naive approach suffers an endogeneity problem as the class size could be associated with other factors (a caring teacher allocating "bad" pupils into smaller classes, in hope of devoting more intense tutoring to them) that could bias the estimated effect (in tech speak, the omitted variable would be captured in the error term, which would correlate with the dependent variable). What Prof Bos assumes, however, is that all students are randomly assigned to classes with different sizes - a rather problematic assumption, given that cases of endogenous class assignment are not only confined to my anecdotal evidence.

A more rigorous way to assess the impact of class size on student performance would be a randomized-controlled-trial (RCT), creating a truly random assignment to balance unobservable confounds in order to isolate the causal effect. This is for instance done by Banerjee, Duflo and Linden (2006) in India (albeit in a different context, the methodology remains the same), and the body of RCT papers is growing almost day by day.

The worrying thing is that naive regression analyses are often nevertheless re-interpreted (by the media) as causal relationships and scientific truth (presented in reduced form as an "executive summary"), which can have irresponsible effects: For sensitive issues such as education policy, the results are often used to inform policy makers and justify their decisions. Given the tendency of the currently ruling Conservative government to slash public expenditures, such a piece of "evidence" comes in very handy: Why employ more teachers? Research shows that it does not lead to better student performance!

Addendum: A not so up to date, but still comprehensive overview to the problems of identifying causality between class size and scores.