Data Scientist
So the polls predicted a big swing to Labor but yesterday we saw the opposite: a swing away from them and a returned conservative government.
I wasn’t too surprised. I ran the numbers and found the same mismatch between experts and reality that I see in my consulting to major consumer brands:
The less educated polling booths were, the less they behaved as predicted
The above graph shows that the 20% of the polling booths with the fewest people with a graduate degree saw a swing of nearly 7% to the Coalition. On the other end of the spectrum, the most educated fifth of polling booths- actually swung 0.5% towards Labor (two party preferred).
How I did it: first I got this morning’s results for each polling booth in the country from the electoral commission and looked at the population in the Microburb directly surrounding it, making the assumption that if you live within a few blocks of a polling booth, that will be the one you vote at.
Then I gave it over to the machine to find the drivers of the swing. Out of the many tens of thousands of potential predictors I’ve collected over the years, university education came up #1. All the other top factors were education-related too. It has a correlation of -0.4 with a negligible P value (6061 polling booths could be matched and had enough data).
A pattern of bias among experts
The core problem is that when humans ask specific questions in surveys or of databases, they are starting with bias. Whereas the “big data” approach to throw in as much data as possible and let the machine scan for patterns. This empowers the human analyst to have a wider perspective in looking for core drivers.
In this case, survey targets no longer reliably answer land line calls and even when called on their mobiles, most don’t agree to be surveyed. So it was left to the experts to try and make the surveys representative. Chances are they weren’t entirely empirical about this and relied on their intuition.
My work analysing the data of big Australian consumer brands has routinely uncovered this systematic bias: market researchers and analysts make sense of numbers using a mix of stats and intuition. That intuition is always informed by their own life and experiences. They are mostly cosmopolitan, don’t encounter uneducated people so their voice is not properly extracted or smoothed over as noise.
There are many factors at play of course and I’m certainly not saying that all the Left has to do is send people to university. In the corporate world, I encounter lots of assumptions about them like that they are simply poorer versions of us and all they want to do is be like us. But these forgotten people have their own values, and don’t understand the world in the same way.
Nor am I saying intuition is worth nothing. It’s very valuable in coming up with hypotheses. But eventually these assumptions are tested in the marketplace and the polling booth.
The two Australias
In predicting behaviour these days, my models end up dividing Australia into two classes: a university educated, affluent, multicultural, cosmopolitan one on one side and on the other have to be very delicately described – the people who go against the brand’s values. But every mainstream brand should strive to understand the latter, not least ones that are supposed to champion the working class.
Luke Metcalfe is founder of Rapid Intelligence, a data science consultancy specialising in scanning for core business drivers across disparate datasets. He is also created Microburbs, a free service which uses big data to calculate liveability of Australian suburbs.
