Academic debates are rarely high stakes. Compared with the kind of work the Climate Change Commission is doing, debates among academics are of almost no consequence at all.
But the debate a decade ago on links between government debt and economic growth certainly was high stakes. Work by Reinhart and Rogoff in 2010 suggested that economic growth quickly falls off a cliff if government debt exceeds 90% of GDP. The work had policy implications – governments looked to it when thinking about how to respond to the financial crisis.
And it provides a critically important lesson about the Climate Change Commission’s approach to data sharing.
It turned out that Reinhart and Rogoff’s results were driven by some coding errors in Excel, and an error in one datapoint that misstated New Zealand’s growth rate in 1951.
The errors were found not because of journal refereeing processes – the work was published without its having been caught. It was found instead because the authors shared all of their workings with critics who were sceptical about the results.
Then others followed on.
Gerald Silverberg became convinced that some of the problem in the Reinhart and Rogoff work hinged on New Zealand data from around the time of the waterfront strike. The waterfront strike hammered New Zealand’s economic growth in 1951, and Reinhart and Rogoff had chosen the 1951 datapoint to represent New Zealand’s growth rate over the entire 1946-1951 period. And that wound up making the difference in whether high levels of debt caused a tipping point in economic growth.
The errors in coding would not have been found if Reinhart and Rogoff had shared only some of their assumptions and their results. Finding the error in the 1951 datapoint required knowing what value the researchers had used in that cell in the Excel sheet.
A series of webinars about the modelling work simply would not have helped anyone find the critical errors.
If you are working in Excel, the only way to find the errors is by drilling through the entire set of worksheets. That is one reason that serious modelling work simply is not done in Excel. Did somebody forget to put a dollar sign in a spot where a dollar sign was really needed? How could you tell without checking all the way through? It is too easy for a mistake in Worksheet 5, Tab 17, cell XC5840 to wind up making a hash of the whole thing.
Serious work instead is done in statistical packages like R or Stata, where the code that builds the results is easily documented and checked.
And that takes us back to the Climate Change Commission and its approach to its modelling.
The stakes in the Climate Change Commission’s work could not be higher. The government has already signalled that it will largely adopt whatever the Commission tells it to do. The entirety of New Zealand’s climate response hinges on it. The modelling underling the Commission’s sector-by-sector carbon budgeting drives a series of policy recommendations, including a ban on new household gas connections, a ban on petrol vehicle imports, and, surprisingly, bans on some forestry planting as carbon sinks.
Getting this right really really matters.
All of the Commission’s work, as best we have been able to make out, is driven by modelling work in a series of Excel spreadsheets.
And while the Commission has shared some of the assumptions and data underlying its work, and has held webinars with its modelling team, it has steadfastly refused to share its workings. Without the workings, it is impossible to see the formulas driving the results. Without seeing the formulas, it is impossible to tell whether there are any errors. And without the full set of worksheets, it is impossible to find all of the critical assumptions and test their importance.
To take one example, whether banning the import of petrol vehicles makes any sense depends on assumptions about the cost of that kind of ban. Remember that the Emissions Trading Scheme has a binding cap, and that transport is covered under the ETS. Every litre of petrol purchased comes with an emissions certificate purchased on your behalf for the 2.45 kilogrammes of emissions that come out of the tailpipe with every litre used. If you burn a litre of petrol, someone else reduces their emissions by 2.45 kilogrammes, because the permit that was bought on your behalf cannot be used by anyone else.
It is difficult, but not impossible, to make the case for carbon regulations in areas that are already covered by the Emissions Trading Scheme’s binding cap on emissions. The test of any policy is whether it delivers more benefits than costs, in this case after taking into account the ETS. But the Commission has failed to provide the workings necessary to show that they have gotten this right. We are instead asked to take it on faith: blessed are they that have not seen the model and its workings, and yet have believed.
If Reinhart and Rogoff had taken the Climate Commission’s approach to data sharing, the errors in their work would not have been found – or at least not nearly as easily. Governments might still be following their inadvertently faulty advice. Without publication, other researchers would have had to recreate the initial datasets and do their best to replicate the work. And it would be hard to tell whether differences in results were due to differences in method, or errors in one team’s workings.
Last week, Auckland University Professor of Statistics Thomas Lumley said that “the Climate Commission should just publish its models so people can see what they’re sensitive to.”
That they have not yet done so is worrying. Data scientist Peter Ellis warned that failure to publish the underlying models usually means one or more of a few bad options. It could mean that the underlying code is not properly managed or version controlled. It could mean that people have something to hide. Or it could mean that the organisation that built the model was not resourced sufficiently to meet professional standards. Or a combination of all three plus time constraints.
Getting this right is incredibly important. The Commission is laying the groundwork for the coming decades of New Zealand’s climate response. Their modelling team has been faced with a very difficult task on a very tight deadline. Errors could make net-zero billions of dollars more costly than it needs to be, could make getting to net zero slower than it needs to be, or both.
It is an urgent task, but not so urgent that the work should forgo basic scrutiny.
Had the Commission had time to do things properly from the outset, documenting the work as it went and following standards allowing for publishing the code and as much of the data as possible, the workings would have been published at the same time as its draft report. The work would have been done in R, or in Stata, so that the code could be checked for errors. Working in Excel makes it much easier to make mistakes and much harder to find errors.
There then would have been plenty of time for scrutiny.
Organisations including the New Zealand Initiative requested an extension to the submissions deadline to provide time for more rigorous testing of the Commission’s work. The Commission provided the extension, but not the workings, so the extra time will be of little help. The longer it takes for the Commission to release its workings, the easier it will be to conclude that the Commission is trying to avoid its work being checked before it turns into policy.
Is it too much to expect that a government agency tasked with the government’s most important long-term project be as open about its workings as Reinhart and Rogoff? It is the only real way to ensure we get this right.