Reprint of my 2012 blog post, "NYC Teacher Evaluation Scores Show The Irrelevancy of Teachers On Student Test Performance."
Monday, February 27, 2012
NYC Teacher Evaluation Scores Show The Irrelevancy of Teachers On Student Test Performance
Two articles in The New York Times (mentioned below) about the recently released NYC teacher evaluations indicate that teacher quality is not an important factor in student performance. Parents, teachers, media and many educational researchers overemphasize the importance of teacher quality on student learning and student standardized test performance.
Of course, there is more to a good teacher's characteristics than just the ability to teach effectively. In lower grades, a nurturing, reassuring and encouraging personality is probably a trait that most parents and children would like. The ability to maintain discipline and safety in the classroom, without being overly harsh, is another trait parents and students would probably like to see in teachers at all grade levels. Similarly, teachers that instill curiosity or an interest in a particular subject or extra-curricular activity are valuable beyond their ability to improve reading or math scores.
As for student performance on reading and math tests, research indicates that teacher quality is not that important. For example, research shows that teacher credentials, such as certification, advanced degrees, and years of experience in the classroom, do not affect student educational performance in reading and math. See as an example my post, "Teacher Credentials Unrelated To Student Achievement." Also, see results from Teach for America and similar programs which place eager, but not trained in education, recent college graduates into poorly performing schools. These students are very effective in the limited time they are in the schools.
Gambler's Fallacy
Many educational researchers, because of their own prior beliefs in effective teaching, fall into the gambler's fallacy of believing in "hot hands" that certain decks of cards, roulette wheels or slot machines are paying out more than others, when what the gambler is seeing is a momentary random streak of winning hands.
Studies that evaluate teacher impact on student performance over time generally find wide variation in a teacher's effectiveness from one year to the next. If some teachers are more effective than others and the important factor, then why does student performance measurement vary much from year to year? (For example, a recent Harvard study that found that better teachers in the lower grades had an adulthood impact on students and their earnings stated "teacher value-added is not in fact a time-invariant characteristic." and "if true VA [value-added] is mean reverting, deselecting teachers based on their current VA [value-added] will yield smaller gains in subsequent years, because some of the low VA [value-added] teachers improve over time.") These studies find some teachers are better than others because over the time frame measured, on average, they are more effective than other teachers. It is the same as believing in winning slot machines. Over a short time frame, with wins and losses, the researchers see some teachers have a higher winning average than other teachers, confusing a random streak for a measure of the effectiveness of the gambler. Additionally, researchers have found and attempt to control in their studies, that there is student sorting in the schools where some teachers consistently get more of the higher test scoring, better students than other teachers.
NYC Evaluation Results
The data from the NYC teacher evaluations shows that high, average and low scoring teachers are about equally scattered among all the NYC schools, the high and the low performing schools.
From The New York Times, "Teacher Quality Widely Diffused, Ratings Indicate" by Fernanda Santos and Robert Gebeloff:
The article mentions that some of the higher income parents were surprised that teachers who they evaluated as effective did not produce test results beyond the expectation for the families' demographic group.
On average, independent of teacher quality, students from higher income families do better on standardized tests including SATs, graduate from high school at higher rates, go to and graduate from college at higher rates than students from poorer families.
Changes that have effect on student performance in school tend to be outside of the classroom, such as an increase in parental expectations of student educational performance, increase in parental literacy, decrease in teenage pregnancy rates, decrease in bullying in school and other factors which are unrelated to teacher effectiveness.
Of course, there is more to a good teacher's characteristics than just the ability to teach effectively. In lower grades, a nurturing, reassuring and encouraging personality is probably a trait that most parents and children would like. The ability to maintain discipline and safety in the classroom, without being overly harsh, is another trait parents and students would probably like to see in teachers at all grade levels. Similarly, teachers that instill curiosity or an interest in a particular subject or extra-curricular activity are valuable beyond their ability to improve reading or math scores.
As for student performance on reading and math tests, research indicates that teacher quality is not that important. For example, research shows that teacher credentials, such as certification, advanced degrees, and years of experience in the classroom, do not affect student educational performance in reading and math. See as an example my post, "Teacher Credentials Unrelated To Student Achievement." Also, see results from Teach for America and similar programs which place eager, but not trained in education, recent college graduates into poorly performing schools. These students are very effective in the limited time they are in the schools.
Gambler's Fallacy
Many educational researchers, because of their own prior beliefs in effective teaching, fall into the gambler's fallacy of believing in "hot hands" that certain decks of cards, roulette wheels or slot machines are paying out more than others, when what the gambler is seeing is a momentary random streak of winning hands.
Studies that evaluate teacher impact on student performance over time generally find wide variation in a teacher's effectiveness from one year to the next. If some teachers are more effective than others and the important factor, then why does student performance measurement vary much from year to year? (For example, a recent Harvard study that found that better teachers in the lower grades had an adulthood impact on students and their earnings stated "teacher value-added is not in fact a time-invariant characteristic." and "if true VA [value-added] is mean reverting, deselecting teachers based on their current VA [value-added] will yield smaller gains in subsequent years, because some of the low VA [value-added] teachers improve over time.") These studies find some teachers are better than others because over the time frame measured, on average, they are more effective than other teachers. It is the same as believing in winning slot machines. Over a short time frame, with wins and losses, the researchers see some teachers have a higher winning average than other teachers, confusing a random streak for a measure of the effectiveness of the gambler. Additionally, researchers have found and attempt to control in their studies, that there is student sorting in the schools where some teachers consistently get more of the higher test scoring, better students than other teachers.
NYC Evaluation Results
The data from the NYC teacher evaluations shows that high, average and low scoring teachers are about equally scattered among all the NYC schools, the high and the low performing schools.
From The New York Times, "Teacher Quality Widely Diffused, Ratings Indicate" by Fernanda Santos and Robert Gebeloff:
The controversial ratings of roughly 18,000 New York City teachers released on Friday showed that teachers who were most and least successful in improving their students’ test scores could be found all around — in the poorest corners of the Bronx, like Tremont and Soundview, and in middle-class neighborhoods of Queens, like Bayside and Forest Hills.From The New York Times, "In Teacher Ratings, Good Test Scores Are Sometimes Not Good Enough" by Sharon Ottyerman and Robert Gebeloff:*** They were in similar proportions in successful and struggling schools, and they were just as likely to have taught the most challenging of students and the most accomplished.
At Public School 234 in TriBeCa, where children routinely alight for school from luxury cars, roughly one-third of the teachers’ ratings were above average, one-third average and one-third below average.The NYC evaluations control and adjust for different student demographics, family income, etc. Students from better off families perform better on standardized tests and teachers should not get credit for the higher test scores of these students, especially since the higher scores are due to socio-economics and not teacher effectiveness.
At Public School 87 on the Upper West Side, where waiting lists for kindergarten spots stretch to stomach-turning lengths, just over half the ratings were above average. The other half were average or below average on measure, based on student test scores.*** The principal cause of the wide variation within schools is the methodology of the ratings, which compares teachers with similar student demographics and scores. For teachers in schools with high-achieving students, good test results are often not good enough, at least by the standards set by the formula.
The article mentions that some of the higher income parents were surprised that teachers who they evaluated as effective did not produce test results beyond the expectation for the families' demographic group.
On average, independent of teacher quality, students from higher income families do better on standardized tests including SATs, graduate from high school at higher rates, go to and graduate from college at higher rates than students from poorer families.
Changes that have effect on student performance in school tend to be outside of the classroom, such as an increase in parental expectations of student educational performance, increase in parental literacy, decrease in teenage pregnancy rates, decrease in bullying in school and other factors which are unrelated to teacher effectiveness.
