<commit_before>---
title: Hip "Data" Nonsense
created_at: 2012-11-15
kind: article
---

I've been at loads of hip and non-hip data talks here and there and have come
up with some ideas about what this data nonsense is.

"Big data" and "data science"
---
Maybe there's a better term than "data nonsense". The two main ones I here that
relate to this beginning-of-a-field are "big data" and "data science". What do
these mean?

When people describe something as "big data", the thing is usually physically
big (in bytes). So a petabyte of data would might be big data.

Other people propose thresholds, like when the data don't fit in random-access
memory or when the data don't fit on one computer. (Some talk at Strata NY 2011
used this one.) More generally, we might say that data are "big" when they
don't physically fit the world's standard data analysis tool, [Excel](/!/excel/).

Judging data-bigness by physical size works today, but the size that seems big
today is different from what seemed big twenty years ago and from what will
seem big in twenty years.


In some
circles, one of these terms is more popular than the other<commit_msg>automatic<commit_after>---
title: Hip "Data" Nonsense
created_at: 2012-11-15
kind: article
---

I've been at loads of hip and non-hip data talks here and there and have come
up with some ideas about what this data nonsense is.

"Big data"
---
When people describe something as "big data", the thing is usually physically
big (in bytes). So a petabyte of data would might be big data.

Other people propose thresholds, like when the data don't fit in random-access
memory or when the data don't fit on one computer. (Some talk at Strata NY 2011
used this one.) More generally, we might say that data are "big" when they
don't physically fit the world's standard data analysis tool, [Excel](/!/excel/).

Judging data-bigness by physical size works today, but the size that seems big
today is different from what seemed big twenty years ago and from what will
seem big in twenty years. I present below two descriptions of big data that
get to the causes of data-bigness.

Some guy at Strata London 2012 proposed that big data come about when it
becomes less expensive to store data than it is to decide whether to delete
the data. We've recently moved from filing cabinets and libraries all the
way to Hadoop clusters and
[low-power hard drives](http://aws.amazon.com/glacier/), so it recently has
become reasonable to just save anything. Or so this guy might say.

Where do all of these data come from? I think part of this big data thing is
that the data are collected more automatically than they were before. Before
computers, if the post office wanted to study where mail is sent, I guess it
could sample letters at various points within the postal system and find their
destinations, return addresses and route through the postal system. There
might be tools to assist this data collection, but someone would still have
to take part in the data collection. Today, we already have all of our emails,
Twitter posts and other internet presense in reasonably standard digital
formats, so this process is much more automatic. I thus propose that automation
of data collection is part of the big data thing.
{'fraction_non_alphanumeric': 0.049846153846153846, 'fraction_numerical': 0.008615384615384615, 'mean_word_length': 4.160317460317461, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '348908', 'n_tokens_mistral': 873, 'n_tokens_neox': 810, 'n_words': 543}