Randomized Control Trials have recently been getting the heavy grills. On the one hand you have the likes of Ben Goldacre who advocate their adoption for just about anything (here’s his Cabinet Office paper on their potential role within policy making) while on the other there is a substantial philosophical literature undermining their claims to offer uniquely objective and reliable results. Nancy Cartwright has been doing this for a while and her latest book, co-written with Jeremy Hardie, was reviewed by James Wilsdon earlier this month. (For more on the philosophical end of the spectrum see also Paul Thompson’s chapter in this edited collection.) While these lines of enquiry are very valuable (and I have yet to see an adequate attempt to engage with these criticisms from the Goldacre camp) many of the questions they pose can also be pursued from a historical perspective. The latest chapter in my thesis has attempted to do just that, so here’s a very brief overview which I think emphasises a problem with RCTs that is too often overlooked.*
Firstly, RCTs and randomization have not been completely ignored by historians of science. It makes an appearance in Stigler’s Statistics on the Table and there has also been a small fight over the RCTs origins. Some, such as Dehue, have argued that it emerged primarily within psychology over an extended period of time. Hacking meanwhile, has acknowledged this prehistory but still gives credit for the widespread adoption of the RCT to its most vocal popularizer in the twentieth century, R.A. Fisher.
Indeed the majority of you that have heard of the RCT will know of its development by Fisher at the Rothamsted Experimental Station (now Rothamsted Research), an institute dedicated to agricultural science. The most recent historian to investigate the origins of the RCT at Rothamsted is Nancy Hall, who certainly sides with Hacking on the question of Fisher’s pivotal role. Unfortunately, her paper also completely overlooks the huge amount of criticism that thereby came Fisher’s way, painting a much rosier picture of the RCT’s reception.
In short, the RCT provoked a major bust up. Until now, in the few places in which this debate has been mentioned, it has largely been seen in personal terms, as a disagreement between Fisher and his otherwise very close friend William Sealy Gosset. This is how Fisher’s daughter saw the fight in her biography of her father (published in 1978 and still the most comprehensive study of the man) and largely also how Ziliak and McCloskey present the dispute in their remarkable book The Cult of Statistical Significance. I say remarkable not only for their sensitivity to the history of their subject matter (which as you might have guessed, is much more concerned with tests of significance than the closely aligned question of randomization) but also the humour with which they write. Go get it, it’s weird. Anyway, for these authors Gosset is the big hero (a veritable Woody Guthrie, as they write) to Fisher’s ‘waspish’ villain. I wish to introduce an institutional element to this discussion, seeing it also as a debate between Rothamsted’s chemistry and the National Institute of Agricultural Botany‘s biology (guess which one my thesis is about).
As this is a blog post, I’ll leave out all my pesky evidence/the foundations of my argument and just cut to the chase. Agricultural field trials, like those conducted by NIAB, played an important role within agricultural science and industry. They were not just about collecting reliable data. They were equally important as locations for discussion and demonstration, where methods familiar to the farmer were used to educate and persuade, and where in turn he could make observations and suggest improvements. They were a key symbol of intent, if you like, that agricultural scientists were working on problems directly related to the dilemmas of British farming (though of course the extent to which this was the case is a different question). It was also necessary that such trials were conducted across the country, due to the variability of the biological objects of NIAB’s research – vegetable and cereal crops – within different locations. This stands in stark contrast to the ‘classic’ field experiments at Rothamsted, the lengthy pedigree of which has this very day been trumpeted on the institute’s Twitter account (I will leave aside the near worthlessness of these non-randomized trials on a Fisherian view). While some farmers – particularly those local to the Harpenden area – no doubt made the trip to the institute, there was no attempt to sustain a direct relationship with the wider agricultural community. Rothamsted scientists were more directly focussed upon their research and the trials they conducted for agrochemical firms, the latter looking to establish the value of their latest fertilizer or soil treatment. This lack of extension across the country was also motivated by the chemical nature of their investigations. Certainly chemicals will vary in their effects depending on the circumstances in which they are used, but the point for these researchers (as opposed to those at NIAB) was to find ways to eliminate this variability, not capture it. Multiplying trial locations could only offer unnecessary complications.
In any trial (medical/social/agricultural) there are tensions over resource utility. In this instance, the primary resource was land. A trial can be set up in such a way as to answer a very specific set of questions (Fisher was always keen to stress that ‘either-or’ questions were far more complex than ‘simple’ questions about numerous variables) and answer them very well indeed. Such appears to be the impetus behind Goldacre’s randomized treatment allocation software as described in his latest book. Trouble is, that there are always other things you could be doing with a trial, always other questions you might pose, and even functions other than the collection of data that you might wish to perform (teaching farmers, helping the sick, governing the country). A researcher with infinite land and resources might never be forced to compromise. “Over here we have our demonstration fields for farmers, over here our trial grounds for botanists, and over here those for our chemists.” Yet even in this abstract idealisation compromises might be required if, for instance, the environmental conditions required for a trial only pertain in certain sections of the land. The decision to use an RCT is always just that, a decision. It is not always simply the best, or the most objective, or the most reliable. It is a choice, one for which experimenters must be held accountable. R.A. Fisher never recognized this. He was a bad farmer.
*I am still some way from submitting, so if you think I am talking shit, all criticisms will of course be very welcome and useful.