Why Good Schools And Poorly Performing Schools Stay That Way: Reprint of January 2010 Blog Post

A reprint of my January 25, 2010 blog post, "Why Good Schools And Poorly Performing Schools Stay That Way."

Monday, January 25, 2010

Why Good Schools And Poorly Performing Schools Stay That Way: Survivorship, Selection, Retention And Filtering Bias

Posted By Milton Recht

I am in favor of more information about schools, teachers and student educational performance because I believe there is not enough accountability by school systems for the educational performance of students and because I believe desirable outcomes should be measured to allow for corrective feedback at all levels of the organization, including school, teacher and student.

I would think that parents and educators would want information that measured performance and also allowed for corrective actions. Too much information or the wrong set of information is not helpful.

It is individual teacher and individual student level longitudinal (over time) information, which is almost never forthcoming in any disclosure by school systems, that best measures a student's and a teacher's performance. Has the teacher been able to improve the student's performance over the year? Yes, good school for that student: No, bad school for that student.

A large data set of information maybe interesting for conversational topics but is costly both to the parents and the schools, in the sense that both must expend effort in learning which data is useful information for improving student performance and which data is extraneous information.

The problem with aggregated non-specific student and non-specific teacher information is that over any time period the students and the teachers change and comparisons of increasing or decreasing test scores are made against two different groups. This year's graduating students with teacher group A are compared to last year's graduates with teacher group B, for example, without knowing if comparison of the two groups is valid.

There are indications in educational performance data that there is selection and filtering bias, which interferes with the usefulness of the information for parent decision-making. For example, schools with high scores tend to demand a commitment to more schoolwork than lower performing schools and also over other student interests. Do parents and children who want more schoolwork move into these school areas and produce higher test scores or do the schools and teachers. Whether better performing schools, in the sense of higher student scores, attract better achieving students or produce better achieving students is often ambiguous from most studies of performance. Do good students make schools look better on tests or do good schools make students look better on tests?

There are also indications that schools signal which students they want to enroll, which students they want to keep, and which students' families they want to move to another school area. In the US, it is sometimes as simple as whether the school highlights the winning science fair participants or the football team, but often the signals are much more subtle and less obvious. There is a survivability bias. Good schools may look better than other schools because they are better at enrolling, separating and retaining the higher performing students, and the poorer performing students move away from the school. Educational studies almost never control for survivorship bias.

Any educational information to be useful to parents has to allow them to choose a school that would benefit THEIR child. The current state of educational information about schools does not offer information to anyone that allows changes to be made to a school to improve its performance for any fixed student body. Yes, school scores can be improved but it is often by selecting a better performing group of students for testing and not by actually improving student performance. If it were easy to make improvement changes or if people really knew, what changes to make to improve schools, student outcomes at all schools, even poor performing schools, would be improved a long time ago.

The data is unreliable for recommending changes to schools to improve student outcomes and that is why student performance is deteriorating in many schools. It is also why every recommendation that has come out of previous data studies has failed to produce the desired results of significantly improving student test score outcomes.

I posted an almost identical comment on Core economics blog, "What has transparency ever done for us?" by Joshua Gans.