Information analytics is emerging as one of the most powerful tools to help companies make better decisions, plans and forecasts. But, like any such IT-based tools, it's most effective in the hands of people who understand its pitfalls and limitations as well as its potential. It's one of the key areas where CIOs can help their companies leverage advances in technology to improve their overall competitiveness.
Our recently concluded 2012 presidential campaign is an interesting case study in the effective use of information. Up to the very end, many pollsters and political pundits were saying that the election was too close to call. But, whenever friends and colleagues asked my opinion, I invariably told them to go look at FiveThirtyEight.com, the political polling website and blog created by Nate Silver.
Silver launched the FiveThirtyEight in March of 2008. He gained national attention when he correctly predicted the results of the 2008 Democratic Party presidential primaries. His final forecasts for the 2008 presidential elections predicted the winner in 49 of the 50 states, as well as the winner of every race in the Senate. The FiveThirtyEight has been affiliated with the NY Times since August of 2010.
Political forecasting attracts a variety of people. Many pollsters seem to still be using methodologies that worked well a decade or two ago, but look woefully behind the times in the emerging world of big data and advanced analytics. Most political pundits seem to inhabit a kind-of magical realism world of their own: if you say what you want to be true often enough and loud enough, it will eventually come to pass.
Silver, on the another hand, views information-based predictions, including political forecasting, as a scientific discipline. You use all available information; you analyze and extract insights out of all that information using sophisticated models and algorithms; you apply human judgment to make predictions based on those insights; and you keep evaluating and adjusting your models and predictions based on how well they perform in the real world.
But equally important, you must be aware not only of the possibilities but also the limitations and pitfalls inherent in any such predictions. In political elections, as is the case when dealing with any highly complex, fast changing, chaotic and unpredictable system, predictions can only be expressed in terms of probability distributions, and they are generally quite volatile as new information and unanticipated events are factored in. Moreover, predictions are based on models reflecting your views of how the future is likely to evolve. Different models applied to the same data can lead to widely different predictions.
Silver started tracking the 2012 election in June. In his initial forecast he estimated that Obama would win the election with 291.3 electoral votes, compared to 246.7 for Mitt Romney, which gave the President a 61.8% of reelection. As he explained in his blog, there are wide error margins inherent in such early forecasts, so they should only be taken as a pretty rough guide, not unlike forecasting the path of a hurricane five to seven days later. You can give an estimate, but the cone of uncertainty is very wide.
As it turned out, Silver correctly predicted the winner in all 50 states, including all nine highly contested swing states. He also correctly predicted the winner in 31 of the 33 Senate races.
Silver learned his craft in the new field of sabermetrics--the use of statistics in baseball to project a player's performance and career. Sabermetrics was popularized by Michael Lewis in Moneyball, his bestseller book--later turned into a film--about Billy Beane, the Oakland Athletics general manager who used such statistical techniques to make his small-market team highly competitive against teams with much larger budgets.
Sabermetrics combined Silver's love of statistics and baseball. He developed a sophisticated statistical system,PECOTA, for forecasting the future performance of baseball players by comparing the characteristics of the player being evaluated to the characteristics of all past and present players, and then looking at the actual career paths of those players with the most similar characteristics. This enabled the system to assign a probability to different career trajectories for the player being evaluated.
Sabermetrics is now widely used by every team in baseball. But, Silver asks in his book, can statistics alone tell you everything you want to know about a player? When Moneyball first came out, many viewed it as a story about the conflict between the traditional approach of the scouts versus the new approaches being introduced by the statheads. Is there still a valuable role for the scouts, the professional talent evaluators who personally travel to learn about the players first hand by actually watching them play and meeting them in person?
Billy Bean attributes the success of the Oakland A's not just to their statistical aptitude but to their careful scouting of amateur players. In fact, their scouting budget is now much higher than it has ever been as a way of complementing their statistical analysis. Scouting is particularly valuable when evaluating young amateur players, while they are still learning the game. This is when a good scout can spot certain intangibles that may make a player stand out, such as their mental makeup and overall attitude toward the game.
The Signal and the Noise describes a number of areas where information-based predictions have been successful, including baseball, political elections and the forecasting of hurricanes. Good hurricane forecasting has life-and-death consequences. Twenty five years ago, for example, you could only predict a hurricane's landfall 72 hours in advance to within 350 miles, an area way too large to evacuate if necessary. You can now come within a much more manageable 100 miles, and the forecasts are getting better every year. As we just saw with Hurricane Sandy, such accurate predictions truly help save lives.
But, good scientists and engineers are fatalistic by nature, always worrying about what can go wrong with their predictions and designs. We have gotten pretty good at predictions and designs in the more mature fields like physics and civil engineering, but we are just learning how to do so in sociotechnical systems, that is, systems involving people and organizations. Such systems generally exhibit a level of complexity that is often beyond our ability to understand and control. Not only do we have to have to deal with very tough mathematical problems, but with the even more complex issues involved in human and organizational behaviors. Our instincts will often lead us to see patterns where there might be none. We have to work very hard to become aware of and try to overcome our biases.
In business, for example, we are surprised when our strategies did not work in spite of being based on a careful analysis of market conditions. Often, the reason is that we use our analysis to confirm our biases and prejudices, - what we want to be true, - even without realizing we are doing so.