John Edward Terrell
Please note: this commentary, recovered on 3-Febr-2017, was originally published in Science Dialogues on 16-June-2014.
Abstract: According to some, the current debate in psychology about “direct replication” as a way of being vigilant against scientific fraud and sloppiness is devolving into a boxing match fostering snottiness, snark, and downright bullying. However, focusing on the downside of this call to arms may be sidetracking us from attending to a more fundamental question—when is research replication the right thing to do?
ONE OF THE THINGS I LEARNED while struggling to write a book about friendship, human nature, and evolution is that neuroscience and neurotic are not all that far apart. Before saying why I got this impression, however, I need to say something first about psychology today.
While reading journal articles garnered using Google Scholar I got the impression that different researchers working in different laboratories aided perhaps by different sorts of machinery are not only coming up with seemingly incompatible conclusions about how the human mind works (e.g., is there or isn’t there a lateral bias to creativity up there in the cranium?), but also that the left hand isn’t always aware of what the right hand is doing. Different research fiefdoms seem to be chugging along more or less unawares of how others are tackling the same issues. And I had the suspicion few are trying to replace the out-dated unity of wisdom of Sigmund Freud with anything approaching a holistic model of the mind. Why so, if this is true?
I am willing to admit my ignorance, but am I wrong to think experts in neuroscience nowadays are a lot like the famous blind men and the elephant? Each research team may have a firm grip on a piece of the puzzle, but does anyone really know how that beast called the brain actually works?
But wait a minute. What’s neurotic about the picture I am painting? A recent blog exchange between my friend Jim Coan at the University of Virginia and the anonymous science blogger Neuroskeptic has brought me some enlightenment on what sure seems like neuroticism to me.
According to Coan, there is currently a strong push within the field of neuroscience and psychology in general for something called “direct replication” (Klein et al. 2014)—a push that he finds both charming and naive. His real beef, however, is that some are taking this push to mean what might be called “replication failure” (my phrase, not Jim’s) is not just a worry confined (to succumb to a bit of word play) to the boudoir. Failure to replicate, real or perceived, evidently is giving rise to a rash of social nastiness he labels Negative Psychology that strikes me as being akin (or so it would seem) to the worst excesses of the post-modernist critique. “When we criticize each other using the tropes of Negative Psychology—that is, with moral outrage, hostile humor, and public shaming—we train the public to either disregard science altogether, or . . . to confuse outrage with rigor.”
While Coan points his finger as one case in point at Neuroskeptic anon., the latter in response has pleaded not guilty. In fact, Neuroskeptic anon. says he (or she) and Coan are on pretty much the same page and wavelength, and darnitall Jim would know this if he had bothered to read everything Neuroskeptic anon. has written in her (or his) blog over the years since ca. 2008.
I am not sure I should confess this, but I am not a great fan of blog sites. Until Jim’s entry into the fray (his first, by the way) I had paid scant attention to Neuorskeptic anon.’s corpus of writings on the web. Nor do I want to weigh in now as a qualified referee for minding the rules of the noble art of blog boxing. But I do agree with Coan on one thing.
He begins his own blog piece with this statement: “People on all sides of the recent push for direct replication—a push I find both charming and naive—are angry.” I think I know charm when I see it, and I don’t find much that is charming about what’s happening in the sciences of the psyche. But I do think the word naive is worth taking to heart.
According to some, skepticism is fashionable these days, and not just in psychology. One could argue, for example, that this is also a core tenet of climate-change deniers and the Tea Party in the U.S.A. Furthermore, who anywhere on earth could possibly deny that the replication of research results is the gold standard of scientific excellence?
Well maybe here and now and maybe me.
Perhaps more so than Jim Coan may be prepared to argue judging by his blog on Negative Psychology, I would at least like to cast a stone or two in that general direction. By focusing as he and Neuroskeptic anon. do on snark and snottiness at the core of modern skepticism in its many stripes, I think they may both be getting sidetracked from attending to a more central issue—namely why does anyone think research replication is such a good thing to do?
No doubt about it, failure to replicate research results may certainly be a flag on the field, but as Coan has said, anyone with a respectably nuanced view of why replications may fail knows they may do so for all kinds of reasons. What would be naive is to accept that not failing to replicate is proof of the pudding.
Why is this naive? Because doing the same thing over and over again in precisely the same way may amount to little more than making the same damn mistake over and over again–and thereby arriving at the same (erroneous) resolve over and over again. Said differently, direct replications that are just repetitions of the same-old same-old ought to be taken with a grain of salt and viewed with suspicion.
Now am I suggesting that like the United States in 1933 scientists should go off this gold standard? Maybe.
In 1966 the biologist Richard Levins published a paper on model building in population biology that has become a classic in the practice and philosophy of science (Levins 1966, 1993). I have long felt that Levins was leaning a lot on what Henri Poincaré (1905) and Alfred North Whitehead (1938) had written about such matters, and should have said so. Nonetheless, I am not alone in thinking what Levins wrote was inspirational and wise. And one of his main conclusions has become famous: “truth is the intersection of independent lies’’ (1966:423).
What he meant by this provocative statement has been richly discussed and debated (e.g., Levins 2006; Odenbaugh 2006; Orzack 2005; Orzack and Sober 1993; Weisberg 2006a, 2006b). One of the pragmatic lessons, however, taken home after reading his paper (as indeed after reading Poincaré and Whitehead) is that for all sorts of reasons there is no such thing as the definitive single approach, experiment, or scientific model capable of capturing reality in all its chameleon-like complexity.
Therefore, as Levins wrote retrospectively in 2006, we need different ways of converging on the truths we are looking for. Consider this:
In the dispute about climate change, a rising temperature in several cities is suggestive. Adding more cities to the list gives a diminishing return. But independent lines of evidence—ocean temperatures, cores from glaciers, decline of coral reefs, spread of species into places that had been too cold for them, accumulation of greenhouse gasses—each may have some separate idiosyncratic explanation or source of error but jointly converge on an unavoidable conclusion. We have to seek lines of evidence as independent as we can in order to support a large scale conclusion. (Levins 2006:753)
Where am I going with this? The strategy Levins is talking about (as did Poincaré and Whitehead before him) is not the one at the heart of the current drive in psychology and other sciences to replicate evidently successful experiments others have done. No, instead the take-home directive is this one: Can we do a different experiment to see if we get the same resolve? And if not, why?
If this strategy were routine, then there would be no doubt about it. To repeat earlier experiments that led to different results if nothing else is a way to become more confident before making headlines with what we have just done that we have given others due and proper benefit of the doubt. But this wouldn’t be something that might be called “knee-jerk direct replication.” This instead would doing something called “just good science.”
Bohannon, J. 2014. Replication effort provokes praise—and “bullying” charges. Science 344:788–789.
Klein, R. A. Klein, K. A. Ratliff, M. Vianello, R. B. Adams Jr., Š. Bahník, et al. 2014. Investigating variation in replicability: a ‘‘many labs’’ replication project. Social Psychology 45:142–152.
Levins, R. 1966. The strategy of model building in population biology. American Scientist 54:421–431.
Levins, R. 1993. A response to Orzack and Sober: formal analysis and the fluidity of science. Quarterly Review of Biology 68:547–555.
Orzack, S. H. and E. Sober. 1993. A critical assessment of Levins’s The strategy of model building in population biology (1966). 1993. Quarterly Review of Biology 68:533–546.
Odenbaugh, J. 2006. The strategy of ‘‘the strategy of model building
in population biology.’’ Biology and Philosophy 21:607–621.
Orzack, S. H. 2006. Discussion: what, if anything, Is “the strategy of model building in population biology?” A comment on Levins (1966) and Odenbaugh (2005). Philosophy of Science 72: 479–485.
Poincaré, H. 1905. Science and hypothesis, reprint ed. New York: Dover.
Weisberg, M. 2006a. Forty years of “the strategy”: Levins on model building and idealization. Biology and Philosophy 21:623–645.
Weisberg, M. 2006b. Richard Levins’ philosophy of science. Biology and Philosophy 21:603–605.
Whitehead, A. N. 1938. Modes of thought, reprinted ed. New York: Free Press.
© 2014 John Edward Terrell. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author(s) and source are credited. The statements and opinions expressed are those of the author(s) and do not constitute official statements or positions of the Editors and others associated with SCIENCE DIALOGUES.