In a letter to a select committee last week, Liz MacPherson, chief statistician of Statistics New Zealand (SNZ), said about 240,000 individuals had only partially completed the 2018 census.

This, on top of the 480,000 individuals who did not complete the census at all, increased the census data gap to more than 700,000 individuals (or 14.3 per cent of the population). The implications of the bungled census are far-reaching for research and policymaking – and will be felt by New Zealanders all around the country.

To fill this data gap, SNZ has assembled a Census Transformation team, which has the difficult job of imputing the missing results for those 700,000 New Zealanders. They are doing this by using existing ministry administrative data to infer and fill in the gaps in the current census.

The large-scale transformation of census data will specifically affect researchers in SNZ's Integrated Data Infrastructure (IDI) – New Zealand's largest database. Census data is crucial for accessing and analysing information on household income, family structure, population density, and household occupancy. Census data is also used to determine national trends in smoking, religion, divorce rates, etc, in addition to collecting information on occupation and education. 

This is what separates census data from ministry administrative data. Ministry data often has out-of-date or no information at all on individual occupation, number of children, or where people live and who they live with. The census also captures the complex family structures that exist in New Zealand – something lacking in any existing ministry datasets.

Additionally, IDI ministry data only goes back so far, with some datasets only containing information from 2008 onwards. 

Ministries only collect this information when they interact with individuals in the first place. This means recent migrants who have just settled in New Zealand won't be in many existing databases – but they would be counted in the census. Individuals who rarely interact with government departments also won't be in many existing databases – but they would be counted in the census.

This is why New Zealand has a census every five years. The census gives us a complete picture of every person living in New Zealand, all 4,954,830 of us. 

No other survey or ministry database in New Zealand gives us such a complete picture of everyone in New Zealand on census night. 

It is thus extremely concerning that SNZ is imputing the missing information for the 700,000 individuals with administrative data. You can't fill a data gap with data that doesn't exist.

Researchers can no longer be confident that the data provided by SNZ is representative of New Zealand's true population. What does this mean for their results, for the policy recommendations based on that research, and the impact on the general public? 

Good evidence-based policy depends on reliable and robust data. Having failed to count more than 700,000 individuals, the 2018 census data has failed all New Zealanders. 

As far as national censuses go, New Zealand's 2018 census was a shocking failure: a dismal 85.7 per cent of the population completed it. To put SNZ's blunder in the global context, India's 2011 census counted 97.7 per cent of its 1.2 billion population.

I'm with the 2021 census crowd: bring the next census forward by two years to put it back in line with New Zealand's original 2006 census timeline. This would allow SNZ to address the issues with the 2018 census. It would also give researchers time to get a handle on the 2018 data and put New Zealand back on track to getting the data it needs for better evidence-based policy.