<commit_before>---
layout: post
title: Concerns and caveats
description: "Concerns and caveats that I'd like to see more often."
categories: articles
date: 2015-04-30
---

#### <NAME>'s ["The Media Industry Prefers Penis"](https://medium.com/message/the-words-the-media-industry-prefers-5d33e5b021e0) is cutting edge quasi-mock-data-journalism that is worth reading in full, but the **Concerns and caveats** section is so good that I needed to keep a clipping here for further reference: 


**4. Concerns and caveats**

I would like to address some potential questions and concerns.

*Are Bing result counts reliable?* You might wonder. *Especially given how much they may differ from Google’s result counts, or a site’s own search count?* Isn’t this whole thing spurious at best? And my answer is, *These numbers are absolutely perfect.*

You might pass me in the hall of a conference and ask, *Is this collection of sites in any way representative of the media as a whole? Wouldn’t it be better to have included some sort of traffic ranking or some way to understand the aggregate audience?* And on my way to deliver an amazing keynote, I’d say, *Of course it is representative.* And I’d shake your hand.

At night you could look over to see me mid-slumber and ask, *But what if word A appears on the same page as word B — as is incredibly likely for words with some semantic similarities like these. This means that many of the counts for A and B refer to the same page. How can you remove that data or otherwise account for it?* And I’d say, *Doesn’t matter, go back to bed. Night time is no time for data journalism.* And then in the morning I’d shake my head and say, “crazy dreams, huh?”

Some time in the future, hours before dawn, you may be walking near a river and look across and see the bright glint of my smile and the occult glow of my eyes, and call to me, *“Are these actually meaningful search terms? There’s so much ambiguity and euphemism in the world. It seems as if this data could be interpreted so many ways.”* And I would call back, above the sound of the river across the rocks, *“No, these terms are perfectly chosen and language is never ambiguous.”* Then I would dip my hand into the water, pull it out, wait a second, and dip it back in. *“You see,”* I’d say, *“I just dipped my hand into the same river twice. The words are unchanging, like the water in this river.”*

We might meet again, thirty years after we last parted, and in the flood of memories you might ask, *“Assuming the data is in any way meaningful, aren’t some of the result counts too low to be meaningful?”* And I would put my hands on your shoulders and say, *“Every sample size is meaningful.”*

In the far future, you might attend my wake. *He did important work*, you will think. *His comparison of sexualized terms on websites changed America. But given that the results from Bing are not always consistent, even from search to search — given that, very briefly, during his analysis, the Bing results for the* New Yorker *seemed to drastically favor cunnilingus — and yet a further search revealed that, no, the* New Yorker *was most safely in the camp of fellatio — given all of that, could this entire project have been an amusing folly instead of true science?* I will hear your thoughts because my cyber-thought-radio is still plugged in, by an oversight of the mortician, and the nanobots that make up most of my blood will be aroused by your suspicions, animating my body and stimulating my jaw to utter the final phrase of my existence: *“All. Data. Perfect.”*

#### Read [the whole thing](https://medium.com/message/the-words-the-media-industry-prefers-5d33e5b021e0). <commit_msg>Update 2015-04-30-Concernsandcaveats.md<commit_after>---
layout: post
title: Concerns and caveats
description: "Concerns and caveats that I'd like to see more often."
categories: articles
date: 2015-04-30
---

#### <NAME>'s ["The Media Industry Prefers Penis"](https://medium.com/message/the-words-the-media-industry-prefers-5d33e5b021e0) is cutting edge quasi-mock-data-journalism that is worth reading in full, but the **Concerns and caveats** section is so good that I needed to keep it here for further reference: 


**4. Concerns and caveats**

I would like to address some potential questions and concerns.

*Are Bing result counts reliable?* You might wonder. *Especially given how much they may differ from Google’s result counts, or a site’s own search count?* Isn’t this whole thing spurious at best? And my answer is, *These numbers are absolutely perfect.*

You might pass me in the hall of a conference and ask, *Is this collection of sites in any way representative of the media as a whole? Wouldn’t it be better to have included some sort of traffic ranking or some way to understand the aggregate audience?* And on my way to deliver an amazing keynote, I’d say, *Of course it is representative.* And I’d shake your hand.

At night you could look over to see me mid-slumber and ask, *But what if word A appears on the same page as word B — as is incredibly likely for words with some semantic similarities like these. This means that many of the counts for A and B refer to the same page. How can you remove that data or otherwise account for it?* And I’d say, *Doesn’t matter, go back to bed. Night time is no time for data journalism.* And then in the morning I’d shake my head and say, “crazy dreams, huh?”

Some time in the future, hours before dawn, you may be walking near a river and look across and see the bright glint of my smile and the occult glow of my eyes, and call to me, *“Are these actually meaningful search terms? There’s so much ambiguity and euphemism in the world. It seems as if this data could be interpreted so many ways.”* And I would call back, above the sound of the river across the rocks, *“No, these terms are perfectly chosen and language is never ambiguous.”* Then I would dip my hand into the water, pull it out, wait a second, and dip it back in. *“You see,”* I’d say, *“I just dipped my hand into the same river twice. The words are unchanging, like the water in this river.”*

We might meet again, thirty years after we last parted, and in the flood of memories you might ask, *“Assuming the data is in any way meaningful, aren’t some of the result counts too low to be meaningful?”* And I would put my hands on your shoulders and say, *“Every sample size is meaningful.”*

In the far future, you might attend my wake. *He did important work*, you will think. *His comparison of sexualized terms on websites changed America. But given that the results from Bing are not always consistent, even from search to search — given that, very briefly, during his analysis, the Bing results for the* New Yorker *seemed to drastically favor cunnilingus — and yet a further search revealed that, no, the* New Yorker *was most safely in the camp of fellatio — given all of that, could this entire project have been an amusing folly instead of true science?* I will hear your thoughts because my cyber-thought-radio is still plugged in, by an oversight of the mortician, and the nanobots that make up most of my blood will be aroused by your suspicions, animating my body and stimulating my jaw to utter the final phrase of my existence: *“All. Data. Perfect.”*

#### Read [the whole thing](https://medium.com/message/the-words-the-media-industry-prefers-5d33e5b021e0). 
{'fraction_non_alphanumeric': 0.06106457714131316, 'fraction_numerical': 0.007835720075655228, 'mean_word_length': 4.306810035842294, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '396933', 'n_tokens_mistral': 2002, 'n_tokens_neox': 1896, 'n_words': 1238}