Two years ago, researchers from 24 laboratories around the world set out to test one of the most influential psychology theories of the past 20 years - the "ego depletion" effect.
Based on the groundbreaking research of Roy Baumeister and Dianne Tice in 1996, the theory had become so embedded in the cultural lexicon, it was almost a truism. Its central thesis, that willpower is a limited resource which can be depleted, spawned nearly 500 other studies, became the subject of a best-selling book, inspired comic strips and informed the morning wardrobe choices of Barack Obama.
But it had recently come into doubt. A 2010 analysis of all published studies on the phenomenon confirmed its existence, but a subsequent analysis that included unpublished studies found a much smaller effect. Social psychologists began to ask whether ego-depletion had been exaggerated because the inconvenient studies had never come to light.
To settle the question, an international research collaboration attempted to re-run one of the seminal experiments - a computer-generated task that could easily be standardised across laboratories - to see if they could produce the same result. The answer fell into the field of psychology with a clang. Only three of the 24 replication studies reported a significant effect and one of these was in the wrong direction. Ego depletion - one of the most cherished theories in social psychology - was teetering.
Psychology is in the grip of a crisis. A groundswell of replication studies, which attempt to repeat cornerstone experiments, have eroded the integrity of numerous theories. A handful of high-profile data-falsifying cases have demolished others. Studies that have not withstood replication attempts include "goal priming", which formed the basis for 2000 subsequent studies and resulted from an experiment where people exposed to words about old age took longer to walk down a corridor, and a controversial study that found people were capable of seeing the future. A 2015 project that attempted to replicate 100 psychology experiments could not repeat the original studies in 40 per cent of cases. Some of these theories had been applied to human resources, dieting behaviour, motivational counselling and business management. The credibility of the entire discipline toppled in their wake.
Curtin University professor Martin Hagger, who was lead investigator on the ego depletion replication project, believes psychology has reached a branch in the road. "I think we're at the stage now where because of these high-profile replications, because people have mobilised and science has globalised ... I don't think we can walk away from this," he says.
Indeed, to psychologists the failure of the ego depletion replication was a major blow. Its genesis lay in an experiment that was delicious in every sense. Two groups of students were introduced to a room wafting with the aroma of freshly baked biscuits. One group was permitted to indulge, but the second could only eat radishes. Each student was then asked to complete a puzzle. The radish-eaters gave up much quicker than the biscuit-eaters. Baumeister and Tice concluded that people who exercise self-control find it more difficult to make decisions straight after.
Hagger attributes the popularity of ego depletion to its intuitive appeal and does not accept that the group's inability to repeat it means it is dead, though it certainly needs to be reconsidered. But others were less sanguine. University of Toronto professor Michael Inzlicht, who also participated in the project, wondered in a blog posted shortly before the results were published whether he had been "chasing puffs of smoke all these years".
"What other phenomena, which we now consider obviously real and true, will be revealed to be just as fragile?" he asked. "I'm in a dark place. I feel like the ground is moving from underneath me and I no longer know what is real and what is not."
The original authors, who consulted on the methodology for the replication, now labelled the project a "debacle", saying it had missed a crucial step. Baumeister, who is these days based at the University of Queensland, says the theory has been tested hundreds of times in multiple labs and to declare it terminal on the basis of a flawed project is "absurd". The replication crisis in general is "vastly overblown", he says.
"Mostly, replicators are not doing the studies properly," he explains in an email. "The hypothesis is not being tested, and the results are meaningless."
Hagger eventually became impatient with the controversy. "To be honest I got a bit sick of the attention it received because my motive was not to get that level of attention, it was to get to the truth," he says. "And it turns out that a lot of researchers couldn't handle that truth."
From journals to chat rooms
Psychologists are masters of navel-gazing and bickering. Paul Mehle, in a 1986 essay titled "Psychology: does our heterogenous subject matter have any unity?", wrote at times the question could better be framed: "Are we about to fall apart?" But the rise of social media has brought conversations about dubious research practices - conversations that were once held between like-minded colleagues at conferences - screaming into daylight in a way that is impossible to ignore.
In more conservative times, people who wanted to raise concerns about published research submitted their criticism to the journal where it originally appeared to be considered, moderated and peer-reviewed and published only if it brought down the entire thesis of the original article. The new media generation has upended this process in less than a decade, with continuous discussions on Facebook, Twitter and on blogs that occasionally poke holes in the evidence and force quick corrections.
These tactics have deepened the anxiety of some researchers, who accuse these critics of vigilantism and worry they are not subjected to the same level of scrutiny the original authors were required to meet. Western Sydney University school of social science and psychology dean Kevin Dunn says he is concerned about the migration of academic criticism from journals to chat rooms.
"It's a matter of balance and assuring ourselves that if we were to move away from peer review we would be moving towards a more robust system, and I'm not convinced that's been demonstrated yet," Dunn says.
The triumphalism of some researchers who have discredited original studies has only made the conversation more shrill. After one researcher labelled a study he had failed to replicate an "epic fail", defenders of the original researcher piled scorn on the "replication police". One prominent psychologist called them "shameless little bullies".
The central argument for replication studies is that when journals publish only original research they reward sexy headlines over academic rigour. This model gives scientists an incentive to conduct small studies that are more likely to throw up statistically significant results. It could mean, if the original hypothesis fails, that they run the data through different analyses until a desirable result is found, also known as "p-hacking". Or it could just mean studies that do not support a hypothesis are confined to the "file drawer", and the resulting publication bias in favour of positive results creates the illusion of an effect where there is none.
Baumeister accepts the value of replication studies, but argues that replicators have an equal incentive not to uphold the findings of the original study, because replication failures are more sensational and therefore likely to be published.
"We have created a career niche for bad experimenters," he wrote in an essay last year. "A broadly incompetent experimenter can amass a series of impressive publications simply by failing to replicate other work and thereby publishing a series of papers that will achieve little beyond undermining our field's ability to claim that it has accomplished anything."
He fears that it will lead to a push for bigger sample sizes, which will lead to studies that are more rigorous at the expense of those that are interesting.
"In general, more attention to replication is good," he says. "But the way this is unfolding, I fear it will do more harm than good."
Emperor's clothes 'a bit moth-eaten'
If the public dissection of psychological research has dialled up internal acrimony, another school of thought says it is part of a painful but necessary rebirth. An April study in the Journal of Personality and Social Psychology found psychologists were predominantly pessimistic about the state of their science, even though research practices had improved as a result of the introspection on social media.
"In many ways, this discussion could be tantamount to a revolution, with increasing numbers striving towards a new 'scientific utopia'," the authors found.
One of the biggest changes has been to the editorial policies in the major journals. Perspectives on Psychological Science now publishes replication studies. Other journals require psychologists to pre-register their studies so they cannot fudge the results through p-hacking.
Hagger believes it's making a tangible difference to the discipline of psychology. "It's happening slowly but even journal editors who were the last bastions of the previous approach, who were probably resistant to some of the changes that were being suggested as a result of the replication crisis, are now changing policy and changing their views."
University of Sydney psychologist Alex Holcombe is at the forefront of cultural change. In 2012 he co-founded PsychFileDrawer.org, an open access website for researchers to upload replication attempts. Two years later he co-founded the initiative Registered Replication Reports, which encourages laboratories around the world to replicate pre-determined studies, including the ego depletion replication project. When Holcombe joined Twitter in 2010 his tag was "Science is broke - let's fix it". But he recently changed it.
"I felt that we are fixing it and I don't know if we should be calling it broken any more," he says. "It's been really exciting."
Psychology is not the only field that has grappled with failed replication projects. Earlier this year a cancer study was unable to repeat the results in two of five papers, some of which involved treatments already being used in clinical trials. In economics, a student who attempted to replicate experiments that informed the austerity measures in Britain and Europe discovered those programs were premised on a spreadsheet error.
It took a new generation of psychologists to point out the emperor wore no clothes, Holcombe says.
He hastens to add: "I don't think the emperor has no clothes in this case, more that the emperor's clothes are a bit moth-eaten.
"Existing researchers have reconciled themselves to the holes in the emperor's clothes, while newly-trained PhDs are more likely to say, 'Why is the emperor wearing holey clothes? Let's get them stitched up - add some new policies to reduce publication bias and improve disclosure of methodology'."
