<commit_before>---
layout: post
title: "Data mining the New York Philharmonic performance history"
modified: 2017-01-19 17:02:00 -0500
image:
  feature: stoneStacks.jpg
  teaser: stoneStacks-teaser.jpg
  credit:
  creditlink:
share: true
categories: [musicology, data science, coding]
short_description: "How does war affect the music an orchestra plays?"
---

The New York Philharmonic has a public dataset containing <a href="https://github.com/nyphilarchive/PerformanceHistory/" target="blank_">metadata for their entire performance history</a>. I recently discovered this, and of course downloaded it and started to geek out over it. (On what was supposed to be a day off, of course!) I only explored the data for a few hours, but was able to find some really interesting things. I'm sharing them here, along with the code I used to do them (in <a href="https://www.r-project.org/" target="blank_">R</a>, using <a href="http://tidyverse.org/" target="blank_">TidyVerse</a> tools), so you can reproduce them, or dive further into other questions. (If you just want to see the results, feel free to skip over the code and just check out the visualizations and discussion below.)

## Downloading the data

First, here are the R libraries that I use in the code that follows. If you're going to run the code, you'll need these libraries.

~~~r
library(jsonlite)  
library(tidyverse)  
library(tidytext)  
library(stringr)  
library(scales)  
library(tidyjson)  
library(purrr)  
library(lubridate)  
library(broom)
~~~

To load the NYPhil performance data into R, you can download it from GitHub and load it locally, or just load it directly into R from GitHub. (I chose the latter.)

~~~r
nyp <- fromJSON('https://raw.githubusercontent.com/nyphilarchive/PerformanceHistory/master/Programs/json/complete.json')
~~~

Now their entire performance history is in a data frame called ```nyp```!
<commit_msg>added GitHub repo to nyphil blog post<commit_after>---
layout: post
title: "Data mining the New York Philharmonic performance history"
modified: 2017-01-19 17:02:00 -0500
image:
  feature: stoneStacks.jpg
  teaser: stoneStacks-teaser.jpg
  credit:
  creditlink:
share: true
categories: [musicology, data science, coding]
short_description: "How does war affect the music an orchestra plays?"
---

The New York Philharmonic has a public dataset containing <a href="https://github.com/nyphilarchive/PerformanceHistory/" target="blank_">metadata for their entire performance history</a>. I recently discovered this, and of course downloaded it and started to geek out over it. (On what was supposed to be a day off, of course!) I only explored the data for a few hours, but was able to find some really interesting things. I'm sharing them here, along with the code I used to do them (in <a href="https://www.r-project.org/" target="blank_">R</a>, using <a href="http://tidyverse.org/" target="blank_">TidyVerse</a> tools), so you can reproduce them, or dive further into other questions. (If you just want to see the results, feel free to skip over the code and just check out the visualizations and discussion below.)

<i>All scripts, extracted data, and visualizations in this blog post can also be found in the <a href="https://github.com/kshaffer/nyphil" target="blank_">GitHub repository</a> for this project.</i>

## Downloading the data

First, here are the R libraries that I use in the code that follows. If you're going to run the code, you'll need these libraries.

~~~r
library(jsonlite)  
library(tidyverse)  
library(tidytext)  
library(stringr)  
library(scales)  
library(tidyjson)  
library(purrr)  
library(lubridate)  
library(broom)
~~~

To load the NYPhil performance data into R, you can download it from GitHub and load it locally, or just load it directly into R from GitHub. (I chose the latter.)

~~~r
nyp <- fromJSON('https://raw.githubusercontent.com/nyphilarchive/PerformanceHistory/master/Programs/json/complete.json')
~~~

Now their entire performance history is in a data frame called ```nyp```!

{'fraction_non_alphanumeric': 0.09508440913604767, 'fraction_numerical': 0.00893743793445879, 'mean_word_length': 4.747503566333809, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 19, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2373413', 'n_tokens_mistral': 1191, 'n_tokens_neox': 1126, 'n_words': 515}