You’ve been #hustling for the past year at your startup. Your team has doubled and doubled again. What was...
With all the euphoria / despondency from last night’s culmination of many months of electoral angst, many assume that the big story was that Barak Obama won election for a second term. You could be forgiven for thinking this is the true narrative. You may be lulled into the idea that somehow the next four years are going to be an idyllic period of sweeping liberalization and legislative purpose. However you are wrong. The big story is that big data won.
Nate Silver has been a bit of a lightning rod recently. His blog FiveThirtyEight on The New York Times website has been according to his own words “devoted to rigorous analysis of politics, polling, public affairs, sports, science and culture, largely through statistical means.” As such, many of his insights over the course of many months have run counter to many of the analyses of pundits and polls and political machines. When you put your neck out there against the tried and true “wisdom” of political punditry, you risk the ire and scorn of the establishment.
But the real question is, was he right? The image below should help you visualize Nate’s prediction versus the actual Electoral College results. Do you see any similarities?
How did he do it? He looked at cold, hard data and analyzed it using common statistical techniques. Some accused him of bias, especially Republicans, who viewed Nate as nothing more than an Obama supporter. While he does back Obama, his analysis was transparent and honest. If he had any sort of bias, it was a bias towards data.
There is something magical with the attraction of shooting from the hip and going with the gut. We are drawn to such stories of cavalier decision making. We attach great worth and trust in homespun bromides and “common sense”. Even in the face of strong evidence, solid data or verifiable results, we put more faith in so-called “experts” than clear and compelling data-supported arguments. We say we desire proof yet we end up listening to puffery.
This is not to say that data is spotless and irrefutable. Data can be corrupted, the datasets incomplete, the analysis methodologies flawed, and expansive conclusions incorrectly drawn from results. Data is not an excuse for lazy thinking or shortcuts. We need to dive in deep and take the time to understand fully what the numbers are truly telling us and if the numbers are even accurate. However, I would rather that we rely on data over mere conjecture and opinion spinning. At least one can correct for data veracity and methodology and present the process in a transparent manner.
Hopefully this election brought some sense to the media establishment. I am not all that optimistic though since they seem more interested in spectacle and entertainment than actual news. I do expect however that others took the cue and are beginning to seriously integrate more data-driven approaches to their own work. Big data came up big last night and will continue to become more of a fixture not only in political campaigns, but in every day life.