In the not too distant past, data were small and manageable. They (“data” is a plural word like numbers) were easily manipulated with a calculator or pencil and later by a computer. Small data were structured bits of information like tally marks, sales per quarter and percentages. All that information resided comfortably in a database until it was made visual by graphs and pie charts.
Big data differ from older, more familiar and traditional small data in four significant ways.
Volume: Walmart collects enough data in an hour to fill 50,000,000 filing cabinets. That volume exceeds the processing capacity of conventional databases. Without specialized software, storage, processing and querying capacity there is just too much big data to make sense of it all.
Velocity: Thanks to the Internet and mobile phones, all that volume is not just picked up and recorded. It streams into business systems on a river of bits and bytes. To be effective, business response must stream out equally fast.
After a terrorist attack in 2004, Madrid, Spain, completely revamped its emergency system. With big data monitoring points throughout the city, it is able to answer 81 percent of its police, fire and ambulance calls in less than eight minutes. That’s an extremely fast and tight feedback loop for a metropolitan area of over 230 square miles.
Variety: Some big data are tidy and structured like its smaller counterpart. These include a wide variety of test scores, financial data, personal information and the results of our last physical. Most big data are formless, messy, unstructured information like user clicks, GPS readings and social media text messages. If there is meaning here, it is not obvious. If there are trends, they are obscured by so much noise. Those who handle big data are able to extract meaning and importance from what resembles your grand-parent’s cluttered attic.
Veracity: Big, fast and formless data are also uncertain data. Uncertainties arise from incomplete data, entry errors, processing problems, sensor inaccuracies, social media, latency of information, modeling approximations and plain old deception. When big data managers speak of ensuring data quality, they are referring to its inherent lack of veracity.
A New Life Example
In the era of small data, scientists claimed in any discussion that, “Data wins.” Those with the numbers, observations and statistics trumped those that speculate from their cubicles. Today, the technology of information has expanded the old mantra; it is now, “Big Data wins.”
Here’s an example from the popular retail department store, Target that put it and statistician Andrew Pole in The New York Times Magazine and Forbes. Pole’s unique contribution to big data was that he developed a pregnancy predictor algorithm from Target’s purchase tracking card, the Guest ID, demographic data and Target’s baby registry database.
As every parent knows, pregnancy changes everything. What Target knew was that pregnancy changes and solidifies shopping habits. Pregnant women became company-loyal parents.
Pole and his team at Target’s Guest Data and Analytics Services examined all the purchases made by thousands of women who signed up for the company’s baby registry. Probing all the data at their disposal, Pole noticed the products these women purchased early in their pregnancy. The list included unscented lotion, vitamin supplements, hand sanitizers, scent-free soap and washcloths. Further study expanded the list to 25 key items.
Through statistical and other analytic tools, Pole could predict from a woman’s purchases and demographic data if she was pregnant and, within a small window of error, her due date.
He then applied his pregnancy predictor to every regular female shopper in Target’s national database. The result was a list of tens of thousands of women who were most likely pregnant. From its Guest ID data, Target knows exactly how to use coupons and ads to trigger purchases for each of those shoppers.
They then send coupons via mail, email or to the checkout counter for pregnancy- and baby-related items to