s, social media, latency of information, modeling approximations and plain old deception. When big data managers speak of ensuring data quality, they are referring to its inherent lack of veracity.
A New Life Example
In the era of small data, scientists claimed in any discussion that, “Data wins.” Those with the numbers, observations and statistics trumped those that speculate from their cubicles. Today, the technology of information has expanded the old mantra; it is now, “Big Data wins.”
Here’s an example from the popular retail department store, Target that put it and statistician Andrew Pole in The New York Times Magazine and Forbes. Pole’s unique contribution to big data was that he developed a pregnancy predictor algorithm from Target’s purchase tracking card, the Guest ID, demographic data and Target’s baby registry database.
As every parent knows, pregnancy changes everything. What Target knew was that pregnancy changes and solidifies shopping habits. Pregnant women became company-loyal parents.
Pole and his team at Target’s Guest Data and Analytics Services examined all the purchases made by thousands of women who signed up for the company’s baby registry. Probing all the data at their disposal, Pole noticed the products these women purchased early in their pregnancy. The list included unscented lotion, vitamin supplements, hand sanitizers, scent-free soap and washcloths. Further study expanded the list to 25 key items.
Through statistical and other analytic tools, Pole could predict from a woman’s purchases and demographic data if she was pregnant and, within a small window of error, her due date.
He then applied his pregnancy predictor to every regular female shopper in Target’s national database. The result was a list of tens of thousands of women who were most likely pregnant. From its Guest ID data, Target knows exactly how to use coupons and ads to trigger purchases for each of those shoppers.
They then send coupons via mail, email or to the checkout counter for pregnancy- and baby-related items to women with high potential pregnancy scores. The coupons are a reminder that Target has what pregnant women need at each stage of their pregnancy and when they became mothers.
In order to avoid the charge that Target was spying or invading their privacy, the newborn-related coupons were mixed with neutral discounts.
As The New York Times Magazine reported, in 2010 Target’s Mom and Baby sales increased dramatically and Pole was promoted.
How Big is Big?
Like maternity sweatpants, digital data are produced in a variety of sizes—small, medium, large, XL and Big. Knowing the digital data prefixes is an integral part of Informatics 101.
• Kilobytes: This article contains approximately 40 kilobytes of information. A kilobyte is 1,000 bytes and this article is 40 times that figure. A byte represents one character such as a single letter in this sentence.
• Megabytes: A good high resolution digital photograph suitable for framing at a large size would ordinarily contain one megabyte or more of data. That’s one million bytes.
• Gigabytes: Seven minutes of high definition television? That’s a gigabyte or one billion bytes.
• Terabytes: The total Internet traffic for the first quarter of 1993 was a terabyte or one trillion bytes. Today, the amount of Internet data used in one second exceeds four terabytes. In a 2011 report on big data, McKinsey Global Institute estimated that by 2009, nearly all sectors in the United States economy with 1,000 or more employees had, on average, 200 terabytes of stored data.
• Petabytes: A petabyte or 1,000 terabytes is what Google all by itself processes each hour.
• Exabytes: This year, one exabyte of digital data or one million terabytes is created every 9.6 hours by the vast worldwide array of electronic devices.
• Others: There are further extensions such as zettabytes and yottabytes all useful and mind boggling descriptors quantifying the enormity of big data.
Despite the ease of finding everyday examples of terabytes, petabytes and exabytes, these large datasets are beyond the ability 