Last week we learned a famous 2010 academic paper, relied on by political big-hitters to bolster arguments for austerity cuts, contained significant errors; and that those errors came down to misuse of an Excel spreadsheet.
Sadly, these are not the first mistakes of this size and nature when handling data. So what on Earth went wrong, and can we fix it?
Harvard’s Carmen Reinhart and Kenneth Rogoff are two of the most respected and influential academic economists active today.
Or at least, they were. On April 16, doctoral student Thomas Herndon and professors Michael Ash and Robert Pollin, at the Political Economy Research Institute at the University of Massachusetts Amherst, released the results of their analysis of two 2010 papers by Reinhard and Rogoff, papers that also provided much of the grist for the 2011 bestseller Next Time Is Different.
Reinhart and Rogoff’s work showed average real economic growth slows (a 0.1% decline) when a country’s debt rises to more than 90% of gross domestic product (GDP) – and this 90% figure was employed repeatedly in political arguments over high-profile austerity measures.
During their analysis, Herndon, Ash and Pollin obtained the actual spreadsheet that Reinhart and Rogoff used for their calculations; and after analysing this data, they identified three errors.
The most serious was that, in their Excel spreadsheet, Reinhart and Rogoff had not selected the entire row when averaging growth figures: they omitted data from Australia, Austria, Belgium, Canada and Denmark.
In other words, they had accidentally only included 15 of the 20 countries under analysis in their key calculation.
When that error was corrected, the “0.1% decline” data became a 2.2% average increase in economic growth.
So the key conclusion of a seminal paper, which has been widely quoted in political debates in North America, Europe Australia and elsewhere, was invalid.
The paper was cited by the 2012 Republican nominee for the US vice presidency Paul Ryan in his proposed 2013 budget The Path to Prosperity: A Blueprint for American Renewal.
Undoubtedly, without Reinhart and Rogoff, Ryan would have found some other data to support his conservative point of view; but he must have been delighted he had heavyweight economists such as Reinhart and Rogoff apparently in his corner.
Mind you, Reinhart and Rogoff have not tried to distance themselves from this view of their work.
Keeping records
As said at the outset, this is not the first time a data- and/or math-related mistake resulted in major embarrassment and expense. In a summary of such historical clangers, Bloomberg journalist Matthew Zeitlin recently pointed to:
NASA’s Mariner 1 spacecraft, destroyed minutes after launch in 1962, thanks to “a missing hyphen in its computer code for transmitting navigation instructions”
an Excel spreadsheet error by a first-year law firm associate, who added 179 contracts to an agreement to buy the bankrupt firm Lehman Brothers’ assets, on behalf of Barclay’s bank
the 2010 European flight ban, following the eruption of Iceland’s Eyjafjallajokull, for which “many of the assumptions in the computer models were not backed by scientific evidence”
While many different types of errors were involved in these calamities, the fact that the errors in the Reinhart-Rogoff paper were not identified earlier can be ascribed by the pervasive failure of scientific and other researchers to make all data and computer code publicly available at an early stage – preferably when the research paper documenting the study is submitted for review.
We’ve discussed this topic in a previous article on Math Drudge and another in the Huffington Post – emphasising that the culture of computing has not kept pace with its rapidly ascending pre-eminence in modern scientific and social science research.
Most certainly the issue is not just one for political economists, although the situation seems worst in the social sciences. In a private letter now making the rounds – which we have read – behavioural psychologist Daniel Kahneman (a Nobel economist) has implored social psychologists to clean up their act to avoid a “train wreck”.
Kahneman specifically discusses the importance of replication of experiments and studies on priming effects.
Traditionally, researchers have been taught to record every detail of their work, including experimental design, procedures, equipment, raw results, data processing, statistical methods and other tools used to analyse the results.
In contrast, relatively few researchers who employ computing in modern science – ranging from large-scale, highly parallel climate simulations to simple processing of social science data – typically take such care in their work.
In most cases, there is no record of workflow, hardware and software configuration, and often even the source code is no longer available (or has been revised numerous times since the study was conducted).
We think this is a seriously lax environment in which deliberate fraud and genuine error can proliferate.
Raising standards
We believe, and have argued, there should be new and significantly stricter standards required of papers by journal editors and conference chairs, together with software tools to facilitate the storage of files relating to the computational workflow.
But there’s plenty of blame to spread around. Science journalists need to do a better job of reporting such critical issues and not being blinded by seductive numbers. This is not the first time impressive-looking data, later rescinded, has been trumpeted around the media. And the stakes can be enormous.
If Reinhart and Rogoff (a chess grandmaster) had made any attempt to allow access to their data immediately at the conclusion of their study, the Excel error would have been caught and their other arguments and conclusions could have been tightened.
They might still be the most dangerous economists in the world, but they would not now be in the position of saving face in light of damning critiques in the Atlantic and elsewhere.
As Matthew O’Brien put it last week in The Atlantic:
For an economist, the five most terrifying words in the English language are: I can’t replicate your results. But for economists Carmen Reinhart and Ken Rogoff of Harvard, there are seven even more terrifying ones: I think you made an Excel error.
Listen, mistakes happen. Especially with Excel. But hopefully they don’t happen in papers that provide the intellectual edifice for an economic experiment — austerity — that has kept millions out of work. Well, too late.
