Filed under Disproof, Education, Psychology by aaronsw on April 22 | 0 comments
The experiment was stunning in its simplicity. A group of teachers at a low-income South San Francisco elementary school were asked to begin the year by administering the “Harvard Test of Inflected Acquisition” to their students. The results were processed and the teachers were given back a list of students whose intellectual abilities were expected to “bloom” that year. At the end of the school year the test was administered again and, sure enough, the bloomers were found to have bloomed, surpassing the other students. But there was just one catch: the test was actually a simple IQ test and the “bloomers” were actually chosen randomly.
The result was called the “Pygmalion effect”: teachers who expected their students to do better actually caused their students to do better. It was a classic self-fulfilling prophecy. The study (published as Pygmalion in the Classroom) was widely hailed. It made the front page of the Times, The Today Show, the New Yorker, and Time, among others. Teacher workshops in avoiding the effect spread from Puerto Rico to Saudi Arabia. LA banned IQ tests in its elementary schools. Presidents, textbooks, and Wikipedia articles repeat the notions to this day, over 30 years later.
Except none of it was true. The original study was conducted in first through sixth grades. The results were only statistically significant in grades one and two (where the alleged bloomers started with a 4-point advantage). The study was repeated in two Midwestern schools, where as statistically significant advantage was found in favor of the kids who weren’t expected to bloom. Psychologists who reviewed the analysis of the IQ test results found something was badly wrong. Some kids got lower IQ scores than they would have had they just filled out the test randomly. (It turned out the kids just didn’t fill out the test.)
All of this data was available before the Pygmalion book was ever published or promoted. Yet it was glossed over or otherwise ignored by the authors. And even when critics published articles spelling out the details, their critiques have been largely ignored by the public. Harvard’s Robert Rosenthal, the author of the original study, tried four more times to reproduce the effect, failing each time. A handful of studies did reproduce the effect, but they had incredibly small sample sizes. A meta-analysis found no overall effect when sample size was taken into account and showed that nearly half the replications had results that went in the opposite direction.
It might be nice if this Pygmalion effect were real, if students could do better on IQ tests simply by having their teachers think more highly of them. But, as best as we can tell, wishing doesn’t make it so.
Filed under Education, Psychology by aaronsw on April 22 | 4 comments
In 1972, Dr. Myron L. Fox, an authority on the application of mathematics to human behavior, gave a lecture to a group of educators — psychiatrists, psychologists, social workers, education students, and administrators — on the topic of “Mathematical Game Theory as Applied to Physician Education.” He spoke for an hour and took another half hour of questions. According to feedback forms distributed after the lecture, the talk was very well received. “Excellent presentation, enjoyed listening,” commented one. “Has warm manner,” added another. “Good flow, seems enthusiastic.” Not everyone was so positive, though. “Too intellectual a presentation,” complained one. “My orientation is more pragmatic.” Still, the majority of the responses were broadly favorable.
There was, however, a more serious problem. “Dr. Myron L. Fox” was actually an actor trained to give a speech consisting largely of “double talk, neologisms, non sequiturs, and contradictory statements … interspersed with parenthetical humor and meaningless references to unrelated topics.” The speech was actually an experiment conducted by a group of professors of medical education. They summarized the results by noting that “no respondents saw through the hoax of the lecture, [] all respondents had significantly more favorable than unfavorable responses, and [] one even believed he read Dr. Fox’s publications.”
“Given a sufficiently impressive lecture paradigm,” they concluded, “an experienced group of educators participating in a new learning situation can feel satisfied that they have learned despite irrelevant, conflicting, and meaningless content conveyed by the lecturer.”
Filed under Computer science, Education, Falsified? by pde on February 21 | 3 comments
I hesitated before posting this piece. Is computer science education really a subject in which any result could be of tremendous sociopolitical importance? But, upon consideration, the answer has to be “yes”.
The ability to understand how computer programs work is arguably a central part of a well-rounded modern education, in the same way that people once needed to know a bit about horses. Logically, as a friend of mine once argued, programming should probably be taught sooner than calculus, because it develops the same kinds of analytic reasoning skills but is useful to more people. Perhaps one of the reasons that programming isn’t more widely taught is that (apparently) many students have terrible trouble learning it. Others find it utterly straightforward.
A remarkable little working paper by Saeed Dehnadi and Richard Bornat provides intriguing data which illuminates the underlying cognitive step that is taken by those who can learn to program and missed by those who cannot. It turns out that while this mode of thinking is loosely correlated with general educational achievement, the link is not determinate. Some high achievers cannot program, and some low achievers can. Most remarkably, there are other kinds of abstract analytical tasks where performance is very well predicted by educational achievement. Programming just isn’t one of them.
The article is tentatively titled The Camel Has Two Humps. Be sure to get to the last set of graphs.
It appears that these results should open up an entire new field of research on testing the authors’ proposed explanation, polishing the tests that predict ability to learn programming, and — most importantly — figuring out ways to teach programmatic rule structures to those who do not understand them intuitively.