“What do we want? Evidence-based policy! When do we want it? After peer review!”

I have seen pictures of such placards held up at a rally, a scientists’ gathering in Melbourne, if I remember correctly. However, given the sad state of our politics, it is actually a scene from a parallel universe.

Nevertheless, if we want to have productive debates on public policy, there is no alternative but to base them on evidence. This week, The New Zealand Initiative made such an attempt with the release of a research note on the performance of New Zealand’s schools.

The reactions to our research demonstrated both how wonderful it is to engage across ideological divides when the dialogue is based on evidence. It also showed how maddening it can be to resort to stereotypes even against evidence.

Our starting point was a frustration. For decades, New Zealand has debated the quality of our school system. We have heard much about the decline in our international test scores such as PISA and TIMMS. We have also learnt about “the long tail of underachievement,” that is, the uneven distribution of education success. We know there are massive education gaps between different ethnicities.

Despite knowing all that, we had no way of knowing how much of this state of affairs was due to differences between individual schools and across deciles. We had no reliable way of either telling whether the students’ backgrounds or their schools where the main drivers of the inequality in our education system.

Data treasure trove

New Zealand is fortunate to have a research database that allows us to find answers to such questions. Statistics New Zealand has faced a lot of negative press recently. However, its Integrated Data Infrastructure (IDI) is a treasure trove for social sciences research that is the envy of the world. No other country has brought together data from such a wide variety of government agencies as Stats NZ. We at the Initiative are grateful for the opportunity to use the IDI for our research.

For our education research, our econometrician compiled a dataset containing information about 400,000 New Zealanders who have sat NCEA exams between 2008 and 2017 (all anonymised, of course – we do not see anyone’s name but simply ID numbers). For these students, we could look at nearly every out-of-school factor with an impact on their education success.

In the Data Lab, we can see whether and how long their parents were employed, on benefits or in prison. We can see whether they came from two-parent families or solo parents. We know about their ethnicity, household incomes or any interactions with Child Youth Family services. We also know about their parents’ education record.

We have a comprehensive picture for each of the 400,000 people in our database with dozens of data points for each individual person. With this giant dataset, it was possible for the first time to calculate how much each factor contributes to education outcomes on average.

This exercise took us the better part of a year. Once that was done, we were able to state what level of education achievements we would predict for each student based on that student’s individual family background. In a second step, we could then compare this expected achievement with their actual achievement. Finally, by doing this for all students of a school, we determined whether the school added anything to that expected achievement.

Our results, once reviewed and released by Stats NZ, delivered an econometric answer to the age-old debate around school deciles. Once adjusted for student backgrounds, we could not find a significant difference between schools of different deciles on average. The vast majority of our schools lead to the education outcomes we would expect from students of different socio-economic backgrounds.

In a sense, this is good news. It shows there is no systemic crisis in the New Zealand education system. It also reveals that it is not the fault of lower-decile schools that they do not generate the NCEA credits or University Entrance qualifications that higher-decile schools achieve. It is a simple result of the lower decile schools dealing with students from more challenging backgrounds.

Still, this finding cannot be taken to say everything is fine and there is nothing for education policy to do. The opposite is true.

Variation by decile

Our research also found variation in performance within each decile. Separating out the effects of family background can help to find the real stars among lower-decile schools – and the laggards among higher-decile schools. This would remedy one of the real tragedies in the current system: High-decile schools able to coast because their task is easy and low-decile schools whose stellar performance would never be seen on an NCEA league table.

The challenge for education policy therefore is to identify struggling schools across the board and help them improve. To do that we also need to identify schools that are performing well and learn from their methods.

The response to our findings was mainly positive. Both Education Minister Chris Hipkins and National’s education spokesperson Nikki Kaye took our data analysis as evidence for moving away from the decile system. NZ Principals’ Council chairman James Morris said having data was useful to determine the impact of socio-economic factors on their students.

But it was also revealing how quickly you can fall back into the prejudice trap. Our chief economist Eric Crampton was interviewed on radio and TV about the report. On each and every occasion, he was challenged ad hominem and asked where he sent his children. That they are attending a decile 10 school was then interpreted as hypocrisy.

The actual story is different: As a Christchurch refugee, Eric was paranoid about earthquake risk when moving to Wellington. So the family bought a house in the place with the lowest earthquake risk – and sent their kids to the closest public school.

Despite that, and commenting on the Cramptons’ school choice, one interviewer even said, “I think what is going on here is the New Zealand Initiative’s attempts to say ‘it’s fine as it is and the decile thing is working’ is just an attempt to keep the status quo.” Never mind that all of the Initiative’s research is geared toward pointing out how to change that status quo for the better. And yes, I would have expected better from Heather du Plessis-Allan.

After that experience, I have another idea for a public rally: “What do we want? Evidence-based journalism. When do we want it? After switching off your prejudices.”

Dr Hartwich is the executive director of The New Zealand Initiative. The Initiative’s research note Tomorrow’s Schools: Data and evidence is available here.