Big Data’s dangling sword
Big Data There are two camps in the Big Data debate today: those who think it will solve the world’s problems, and those that don't.
Those that do, believe it will herald an age of machines that put humans out of work; and those who think it’s nothing new, believe it to be a mere hyperbole.
The truth is somewhere in between: a data-driven society changes economic fundamentals in unexpected ways.
First, big data makes it easy to surface correlations. Armed with abundant, cheap computing, we can quickly see which things are related. But correlation is not causality, and it takes careful investigation and experimentation, tempered with human insight, to solve problems.
Consider Boston’s Bump Map application, which uses smartphones to build a crowd-sourced map of the city’s potholes. Without context, the resulting data simply tells workers to patch the streets of affluent neighbourhoods—because that’s where car drivers with late-model phones and unlimited data plans live. Only once the data is weighted with demographic information like wealth do we get a more accurate map of city streets.
Second, big data makes it easy to improve the current model. Data-driven optimization assumes that the future is like the past, only more so. It’s great for what Sergio Zyman, the former CMO of Coca-cola, describes as marketing: selling more things to more people more often for more money more efficiently.
Where data disappoints, however, is in inspiration. Doing something different often looks sub-optimal at the outset, because it targets new, emerging needs—a problem captured in Clayton Christensen’s The Innovator’s Dilemma. As a result, machines need to be twinned with humans to truly improve things.
Finally, big data changes the process of finding things out. In the old, data-is-precious-and-expensive model of the world, we had to decide on the question first, then collect the data. If we wanted to analyse widget sales by colour, shape, and country, we defined the structure of the data up front, then collected it. Later, we were constrained by that structure, and couldn’t easily explore sales by, say, gender or price.
In the new, data-is-cheap model, we can collect data today and decide what we’re going to ask it tomorrow. Indeed, this is the sword of Damocles under which we wrestle today: we capture first and ask questions later, leading to concerns of government overreach and an Internet that never forgets.
Big data is neither sin nor savior. But it is a fundamental shift in how we make decisions, and what tomorrow will consider valuable. What’s clear is that in a connected, instrumented world, the way we make decisions as a species has forever changed—and with it many of our social contracts around privacy, prediction, free speech, and innovation.