<commit_before>---
layout: article
title: Ten Simple Rules for Digital Data Storage
tags: []
bibliography: resources/manuscript.bib
csl: resources/plos.csl
author:
 - family: Hart
   given: Edmund
   affiliation: 1
   email: <EMAIL>
 - family: Barmby
   given: Pauline
   affiliation: 2
 - family: LeBauer
   given: David
   affiliation: 3
 - family: Mulrooney
   given: Patrick
   affiliation: 11
 - family: Michonneau
   given: François
   affiliation: 4,5
 - family: Mount
   given: Sarah
   affiliation: 6
 - family: Poisot
   given: Timothée
   affiliation: 7
 - family: Woo
   given: <NAME>.
   affiliation: 8
 - family: Zimmerman
   given: <NAME>.
   affiliation: 9
 - family: Hollister
   given: Jeffrey
   affiliation: 10
organization:
 - id: 1
   name: Univeristy of Vermont
   address: Department of Biology, Burlington
 - id: 2
   name: University of Western Ontario
   address: Department of Physics and Astronomy
 - id: 3
   name: University of Illinois at Urbana-Champaign
   address: National Center for Supercomputing Applications and Institute for Genomic Biology
 - id: 4
   name: University of Florida
   address: iDigBio, Florida Museum of Natural History
 - id: 5
   name: University of Florida
   address: Whitney Laboratory for Marine Bioscience
 - id: 6
   name: King's College London
   address: Department of Informatics
 - id: 7
   name: Université de Montréal
   address: Département de Sciences Biologiques
 - id: 8
   name: Washington State University
   address: Center for Environmental Research, Education, and Outreach
 - id: 9
   name: University of Arizona
   address: School of Plant Sciences
 - id: 10
   name: US Environmental Protection Agency
   address: Atlantic Ecology Division
 - id: University of California at San Diego
   address: San Diego Supercomputing Center
---

\linenumbers

# Introduction {-}

<!-- JWH Comments: May want to add a few more examples on top of DNA
sequencing and sensor networks.  I am thinking LHC and astronomy, but
outside my field so don't know good one.  Some references for those
too could also be added. I removed the second mention of
heterogeneity, seemed redundant. -->

Data is the central currency of science, but the nature of scientific data has
changed dramatically with the rapid pace of technology. This change
has led to the development of a wide variety of data formats, dataset sizes, data
complexity, data use cases, and data sharing practices. Improvements in high
throughput DNA sequencing, sustained institutional support for large sensor
networks [@Reid2014; @Hampton2013], and sky surveys with large-format digital
cameras [@Eisenstein2011] have created massive quantities of data. At the same time,
the combination of increasingly diverse research teams [@Adams2012] and data
aggregation in portals (e.g. for biodiversity data, [GBIF.org](http://gbif.org/) or
[iDigBio](https://www.idigbio.org/portal/)) necessitates increased coordination
among data collectors and institutions [@Fraser2013; @Robertson2014]. As a
consequence, "data" can now mean anything from petabytes of information stored
in professionally-maintained databases, through spreadsheets on a single
computer, to hand-written tables in lab notebooks on shelves. All remain
important, but data curation practices must continue to keep pace with the
changes brought about by new forms and practices of data collection and storage.
<commit_msg>Update manuscript.md<commit_after>---
layout: article
title: Ten Simple Rules for Digital Data Storage
tags: []
bibliography: resources/manuscript.bib
csl: resources/plos.csl
author:
 - family: Hart
   given: Edmund
   affiliation: 1
   email: <EMAIL>
 - family: Barmby
   given: Pauline
   affiliation: 2
 - family: LeBauer
   given: David
   affiliation: 3
 - family: Mulrooney
   given: Patrick
   affiliation: 11
 - family: Michonneau
   given: François
   affiliation: 4,5
 - family: Mount
   given: Sarah
   affiliation: 6
 - family: Poisot
   given: Timothée
   affiliation: 7
 - family: Woo
   given: <NAME>.
   affiliation: 8
 - family: Zimmerman
   given: <NAME>.
   affiliation: 9
 - family: Hollister
   given: Jeffrey
   affiliation: 10
organization:
 - id: 1
   name: Univeristy of Vermont
   address: Department of Biology, Burlington
 - id: 2
   name: University of Western Ontario
   address: Department of Physics and Astronomy
 - id: 3
   name: University of Illinois at Urbana-Champaign
   address: National Center for Supercomputing Applications and Institute for Genomic Biology
 - id: 4
   name: University of Florida
   address: iDigBio, Florida Museum of Natural History
 - id: 5
   name: University of Florida
   address: Whitney Laboratory for Marine Bioscience
 - id: 6
   name: King's College London
   address: Department of Informatics
 - id: 7
   name: Université de Montréal
   address: Département de Sciences Biologiques
 - id: 8
   name: Washington State University
   address: Center for Environmental Research, Education, and Outreach
 - id: 9
   name: University of Arizona
   address: School of Plant Sciences
 - id: 10
   name: US Environmental Protection Agency
   address: Atlantic Ecology Division
 - id: University of California at San Diego
   address: San Diego Supercomputer Center
---

\linenumbers

# Introduction {-}

<!-- JWH Comments: May want to add a few more examples on top of DNA
sequencing and sensor networks.  I am thinking LHC and astronomy, but
outside my field so don't know good one.  Some references for those
too could also be added. I removed the second mention of
heterogeneity, seemed redundant. -->

Data is the central currency of science, but the nature of scientific data has
changed dramatically with the rapid pace of technology. This change
has led to the development of a wide variety of data formats, dataset sizes, data
complexity, data use cases, and data sharing practices. Improvements in high
throughput DNA sequencing, sustained institutional support for large sensor
networks [@Reid2014; @Hampton2013], and sky surveys with large-format digital
cameras [@Eisenstein2011] have created massive quantities of data. At the same time,
the combination of increasingly diverse research teams [@Adams2012] and data
aggregation in portals (e.g. for biodiversity data, [GBIF.org](http://gbif.org/) or
[iDigBio](https://www.idigbio.org/portal/)) necessitates increased coordination
among data collectors and institutions [@Fraser2013; @Robertson2014]. As a
consequence, "data" can now mean anything from petabytes of information stored
in professionally-maintained databases, through spreadsheets on a single
computer, to hand-written tables in lab notebooks on shelves. All remain
important, but data curation practices must continue to keep pace with the
changes brought about by new forms and practices of data collection and storage.

{'fraction_non_alphanumeric': 0.060557184750733135, 'fraction_numerical': 0.01407624633431085, 'mean_word_length': 3.9824689554419286, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5361873', 'n_tokens_mistral': 1946, 'n_tokens_neox': 1760, 'n_words': 875}