<commit_before>---
layout: post
title: "Consuming Open Data"
subtitle: "New Parser for the World Ocean Database"
date: 2015-07-27
permalink: wod-alpha
---

Last week had some great announcements in open data: [progress][fMRI] was reported in the OpenfMRI project, seeking in part to establish a standard data format for functional MRI brain scans, and K. Ramirez et al's [paper][Ramirez] was accepted for publication that describes a path to standardizing and distributing soil biodiversity data. Both these projects emphasize the distribution infrastructure for their data, but both also advance the conversation around *standardizing* data formats in their fields - one of the key elements of creating truly open data that I discussed [here][opendata] earlier this month.

Standardized data formats are worth the trouble because they are easy to read; my other recommendation in my last post was to distribute parsers for these formats, in order to take advantage of this ease of consumption. My colleagues working on the [International Quality Controlled Ocean Database][iquod] and I have done just that for the World Ocean Database format in our new Python package on PyPI, `wodpy`.

### wodpy

Here's some World Ocean Database data:

```
C41303567064US5112031934 8 744210374426193562-17227140 6110101201013011182205814
01118220291601118220291901024721 8STOCS85A3 41032151032165-500632175-50023218273
18117709500110134401427143303931722076210220602291107291110329977020133023846181
24421800132207614110217330103192220521322011216442103723077095001101818115508527
20012110000133312500021011060022022068002272214830228442684000230770421200000191
15507911800121100001333125000151105002103302270022022068002274411816302284426840
00230770426500000191155069459001211000013331250001511050021033011300220220680022
73319043022844268400023077042620000019116601596680012110000133312500021022016002
17110100220220680022733112830228442684000230770435700000181155088803001211000013
33125000210220160022022068002273311283022844268400023077042120000019115508880300
12110000133312500015110200210330535002202206800227441428030228442684000230770421
20000019115508880300121100001333125000152204300210220320022022068002273312563022
84426840002307704212000001911550853710012110000133312500015110200210220160022022
06800227331128302284426840002307704212000001100003328960044230900033267500222650
03312050033281000220100033289500442309000332670002227100331123003328100022025002
22900044231910033286200222900033115400332810002205000342-12300442324100332728003
32117003312560033280500 

```

Like I said, easy to read, right? If this is a bit opaque, there's a [170 page pdf][wod] that explains how to unpack this. Or, grab our new parser off of PyPI:

```
sudo pip install wodpy
```

throw that block of data into a file `data.dat`, and give this Python script a try:

```
from wodpy import wod

file = open("data.dat")
profile = wod.WodProfile(file)

print profile.latitude()
```

*et voila*: the latitude this measurement was taken at is returned. This data and toy demo are in [this repo][toy]; the README in the main [wodpy repo][wodpy] describes all the methods the `WodProfile` object provides for extracting usable information from the terse format it came in. Big thanks to Simon Good out of the UK Met Office for hammering out the first version of this class in our [AutoQC][autoqc] project.

`wodpy` is very alpha right now, and this is my first attempt at a serious package on PyPI. I'd appreciate feedback from *everyone* on basic operation - does the toy demo linked above work for you? Does the package actually check out and install? I pulled this together in cafes and airports while on vacation, so I haven't had access to a million different OS'es, python versions, etc; whether you're interested in WOD data or not, just letting me know when the demo falls over is a big help.

For those who actually use WOD-format data, what do you think? How can we make this as useful and convenient as possible for your research? There's information encoded in WOD data that isn't currently returned by a method on this class; it'll all get there *eventually*, but if someone were to call out their favorites, I'll send them to the top of the list.

By wrapping up our decoding in a something like the `WodProfile` class, that has easy-to-understand methods for pulling the information we actually want out of open data, I think we can lower the barriers to consumption and get closer to what the open data movement aspires to; beyond the World Ocean Database, I hope we can make `wodpy` a strong example of a modular parser that gets the boring munging out of the way, so we can get down to science faster. 

[fMRI]: http://www.livescience.com/51523-massive-brain-scan-database-goes-online.html
[Ramirez]: http://journal.frontiersin.org/article/10.3389/fevo.2015.00091/abstract
[opendata]: blog/truly-open-data
[iquod]: https://github.com/IQuOD
[wod]: http://data.nodc.noaa.gov/woa/WOD/DOC/wodreadme.pdf
[toy]: https://github.com/BillMills/woddemo 
[wodpy]: https://github.com/BillMills/wodpy
[autoqc]: https://github.com/IQuOD/AutoQC<commit_msg>wod post links<commit_after>---
layout: post
title: "Consuming Open Data"
subtitle: "New Parser for the World Ocean Database"
date: 2015-07-27
permalink: wod-alpha
---

Last week had some great announcements in open data: [progress][fMRI] was reported in the OpenfMRI project, seeking in part to establish a standard data format for functional MRI brain scans, and K. Ramirez et al's [paper][Ramirez] was accepted for publication that describes a path to standardizing and distributing soil biodiversity data. Both these projects emphasize the distribution infrastructure for their data, but both also advance the conversation around *standardizing* data formats in their fields - one of the key elements of creating truly open data that I discussed [here][opendata] earlier this month.

Standardized data formats are worth the trouble because they are easy to read; my other recommendation in my last post was to distribute parsers for these formats, in order to take advantage of this ease of consumption. My colleagues working on the [International Quality Controlled Ocean Database][iquod] and I have done just that for the World Ocean Database format in our new Python package on PyPI, `wodpy`.

### wodpy

Here's some World Ocean Database data:

```
C41303567064US5112031934 8 744210374426193562-17227140 6110101201013011182205814
01118220291601118220291901024721 8STOCS85A3 41032151032165-500632175-50023218273
18117709500110134401427143303931722076210220602291107291110329977020133023846181
24421800132207614110217330103192220521322011216442103723077095001101818115508527
20012110000133312500021011060022022068002272214830228442684000230770421200000191
15507911800121100001333125000151105002103302270022022068002274411816302284426840
00230770426500000191155069459001211000013331250001511050021033011300220220680022
73319043022844268400023077042620000019116601596680012110000133312500021022016002
17110100220220680022733112830228442684000230770435700000181155088803001211000013
33125000210220160022022068002273311283022844268400023077042120000019115508880300
12110000133312500015110200210330535002202206800227441428030228442684000230770421
20000019115508880300121100001333125000152204300210220320022022068002273312563022
84426840002307704212000001911550853710012110000133312500015110200210220160022022
06800227331128302284426840002307704212000001100003328960044230900033267500222650
03312050033281000220100033289500442309000332670002227100331123003328100022025002
22900044231910033286200222900033115400332810002205000342-12300442324100332728003
32117003312560033280500 

```

Like I said, easy to read, right? If this is a bit opaque, there's a [170 page pdf][wod] that explains how to unpack this. Or, grab our new parser off of PyPI:

```
sudo pip install wodpy
```

throw that block of data into a file `data.dat`, and give this Python script a try:

```
from wodpy import wod

file = open("data.dat")
profile = wod.WodProfile(file)

print profile.latitude()
```

*et voila*: the latitude this measurement was taken at is returned. This data and toy demo are in [this repo][toy]; the README in the main [wodpy repo][wodpy] describes all the methods the `WodProfile` object provides for extracting usable information from the terse format it came in. Big thanks to <NAME> out of the UK Met Office for hammering out the first version of this class in our [AutoQC][autoqc] project.

`wodpy` is very alpha right now, and this is my first attempt at a serious package on PyPI. I'd appreciate feedback from *everyone* on basic operation - does the toy demo linked above work for you? Does the package actually check out and install? I pulled this together in cafes and airports while on vacation, so I haven't had access to a million different OS'es, python versions, etc; whether you're interested in WOD data or not, just letting me know when the demo falls over is a big help.

For those who actually use WOD-format data, what do you think? How can we make this as useful and convenient as possible for your research? There's information encoded in WOD data that isn't currently returned by a method on this class; it'll all get there *eventually*, but if someone were to call out their favorites, I'll send them to the top of the list.

By wrapping up our decoding in a something like the `WodProfile` class, that has easy-to-understand methods for pulling the information we actually want out of open data, I think we can lower the barriers to consumption and get closer to what the open data movement aspires to; beyond the World Ocean Database, I hope we can make `wodpy` a strong example of a modular parser that gets the boring munging out of the way, so we can get down to science faster. 

[fMRI]: http://www.livescience.com/51523-massive-brain-scan-database-goes-online.html
[Ramirez]: http://journal.frontiersin.org/article/10.3389/fevo.2015.00091/abstract
[opendata]: http://billmills.github.io/blog/truly-open-data
[iquod]: https://github.com/IQuOD
[wod]: http://data.nodc.noaa.gov/woa/WOD/DOC/wodreadme.pdf
[toy]: https://github.com/BillMills/woddemo 
[wodpy]: https://github.com/BillMills/wodpy
[autoqc]: https://github.com/IQuOD/AutoQC
{'fraction_non_alphanumeric': 0.05208940719144801, 'fraction_numerical': 0.25578231292517006, 'mean_word_length': 6.221754385964912, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '219114', 'n_tokens_mistral': 4800, 'n_tokens_neox': 3094, 'n_words': 1176}