Monday, 16 May 2011

The Truth Wears Off

NTT has written extensively about the disproportionate effect of a black swan event. But it seems that there is a subtle trick in the Black Swan's armoury; it can appear white!  What is more, the transition from white to black is progressive!

Evidence is coming to light of a real phenomena where an experiment can capture a black swan event from the outset, and from which conclusions are drawn.  It is only on successive repetition of the experiment that the problem is identified, as the results of the initial experiment not only prove to be unrepeatable, but the initial 'effect' becomes less and less with each reptition of the experiment. 

This has been termed the Decline Effect, where the truth seems to 'wear off' and the black swan reveals himself.  The New Yorker carried this article titled 'The Truth Wears Off' which explores this phenomena, in the course of which, they provide a quite-staggering example:

  • "One of the first demonstrations of this mysterious phenomenon came in the early nineteen-thirties. Joseph Banks Rhine, a psychologist at Duke, had developed an interest in the possibility of extrasensory perception, or E.S.P. Rhine devised an experiment featuring Zener cards, a special deck of twenty-five cards printed with one of five different symbols: a card was drawn from the deck and the subject was asked to guess the symbol. Most of Rhine’s subjects guessed about twenty per cent of the cards correctly, as you’d expect, but an undergraduate named Adam Linzmayer averaged nearly fifty per cent during his initial sessions, and pulled off several uncanny streaks, such as guessing nine cards in a row. The odds of this happening by chance are about one in two million. Linzmayer did it three times."
The article acknowledges regression to the mean - the phenomena whereby you can have observe an increase and subsequent decline in a phenomena purely as a consequence of statistical fluke; the initial observation actually being an extreme event.  A good example of this is with speed cameras.  Local authorities like to identify stretches of road where there are clusters of RTAs.  They install speed cameras and the number of RTAs falls.  In such circumstances it is easy to conclude on a simple cause and effect; the speed cameras have made the road safer.  But it might just be that the decline in RTAs would have occured anyway, regardless of the speed cameras.  Perhaps the original cluster was actually due to 'random' phenomena such as driver errors from fatigue, fiddling with a radio/phone or due to wildlife running on to the road. 
But researchers aware of the Decline Effect believe it to be real, having fallen victim to it in their own research.  Jonathan Schooler is one such reseracher who experienced the Decline Effect when looking at 'verbal overshadowing';
  • "...Schooler has noticed that many of the data sets that end up declining seem statistically solid—that is, they contain enough data that any regression to the mean shouldn’t be dramatic. “These are the results that pass all the tests,” he says. “The odds of them being random are typically quite remote, like one in a million. This means that the decline effect should almost never happen. But it happens all the time! Hell, it’s happened to me multiple times.” And this is why Schooler believes that the decline effect deserves more attention: its ubiquity seems to violate the laws of statistics. “Whenever I start talking about this, scientists get very nervous,” he says. “But I still want to know what happened to my results. Like most scientists, I assumed that it would get easier to document my effect over time. I’d get better at doing the experiments, at zeroing in on the conditions that produce verbal overshadowing. So why did the opposite happen? I’m convinced that we can use the tools of science to figure this out. First, though, we have to admit that we’ve got a problem.”"
For the paleo crowd there is this bit on the relationship between symmetry and male reproductive success,
  • "It didn’t matter if scientists were looking at the hairs on fruit flies or replicating the swallow studies—females seemed to prefer males with mirrored halves. Before long, the theory was applied to humans. Researchers found, for instance, that women preferred the smell of symmetrical men, but only during the fertile phase of the menstrual cycle. Other studies claimed that females had more orgasms when their partners were symmetrical, while a paper by anthropologists at Rutgers analyzed forty Jamaican dance routines and discovered that symmetrical men were consistently rated as better dancers.

    Then the theory started to fall apart. In 1994, there were fourteen published tests of symmetry and sexual selection, and only eight found a correlation. In 1995, there were eight papers on the subject, and only four got a positive result. By 1998, when there were twelve additional investigations of fluctuating asymmetry, only a third of them confirmed the theory. Worse still, even the studies that yielded some positive result showed a steadily declining effect size. Between 1992 and 1997, the average effect size shrank by eighty per cent."
So, regression to the mean?  Publication bias?  Ancel Keys' bad science is one thing, but this seems something apart!    Lehrer nicley sums it up at the end,
  • "The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe"
Deja vu anyone?  A follow-up article, More Thoughts on the Decline Effect can be read here.


Chris said...

Were you listening to More or Less ladst night as well then?

Asclepius said...

I am a Radio 4 junky! I caught the episode of MOL on Friday but a weekend in Ireland prevented an earlier post.


Have you read "The Tiger That Isn't"?