<commit_before><!DOCTYPE html>
<html>
    <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
    <title>OCP</title>

    <!-- Bootstrap -->
    <link href="Web/css/bootstrap.min.css" rel="stylesheet">


    <style type="text/css">
      @import url(https://fonts.googleapis.com/css?family=Yanone+Kaffeesatz);
      @import url(https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic);
      @import url(https://fonts.googleapis.com/css?family=Ubuntu+Mono:400,700,400italic);
      @import url(http://fonts.googleapis.com/css?family=Varela+Round:regular,italic,bold,bolditalic);
      @import url(http://fonts.googleapis.com/css?family=Raleway:regular,italic,bold,bolditalic);

      body {
        font-family: 'Varela Round'; /*  */
        background-color: #fff;
        position: absolute;
        left: 20px;
        max-width: 640px;

      }
      h1, h2, h3, h4, h5, h6 {
        font-family: 'Raleway';
        font-weight: 400;
        margin-bottom: 0;
      }
      p.small {
        font-variant: small-caps;
      }
      .r { color: #fa0000; }
      .fig-container {
        font-size: small;
        border-top: 5px solid #efefef;
        border-bottom: 5px solid #efefef;
        padding: 1em;
      }

      thead {
          background: #ccc;
          font-weight: bold;
      }

      tr:nth-child(2n) {
          background-color: #efefef;
      }

      td, th {
          padding: 1em;
      }
      </style>

    </head>
    <body>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/markdown-it/5.0.2/markdown-it.js" charset="utf-8"></script>
<div id="md-raw">

# NeuroData: Enabling Petascale Neurosciene

## by, everyone
---


<h1>Abstract</h1>

<p>
Recent technical progress allows neuroexperimentalists to collect ever more detailed and informative anatomical and physiological data from brains of all sizes.  These datasets span experimentally accessible spatiotemporal scales, ranging from nanometer to meter, and millisecond to monthly sampling rates.  In classical neuroscientific experimental paradigms, it was feasible for neuroscientists to draw their results on paper. In contrast, many modern neuroscientific experimental paradigms break the classic data analysis workflow.  In particular, these large datasets create significant challenges for our community at every step of the data analysis pipeline: (1) storing, (2) exploring, (3) parsing, and (4)  analyzing.
</p>

<p>
*NeuroData* has been developed to lower the barrier to entry into big data neuroscience. We have designed and built a computational infrastructure to enable petascale neuroscience.  This includes our flagship project, the *NeuroData's Connectome Project* (previously called the Open Connectome Project). Our infrastructure enables anyone in the world with internet access to visualize, download, analyze, upload, or otherwise interact with a large number public datasets.  Moreover, all the results obtained using the NeuroData infrastructure are fundamentally reproducible and extensible.  We demonstrate the utility of these tools via two serial electron microscopy case studies.  First, we have reproduced many of the quantitative results from a recent landmark EM paper that included a saturated annotation of all objects. Second, we have tested a novel hypothesis about the distribution of synapse locations in cortex on a complementary much larger dataset.  Via the *NeuroData* infrastructure, the answers to these tools are both fully reproducible, and extensible to any other datasets that hold the answers to these questions. Moreover, anybody else can now also do neuroscience at scale, regardless of their background, computational resources, and expertise.
</p>
</div>

<!-- We are in the process of scaling up the number of datasets, the range of experimental modalities, and the Web-services we enable. Work underway will provide pre-packaged cluster environments—1-click deployable on local or commercial cloud computing infrastructures—so that others can replicate and modify our services internally. All of our code and data are available online at neurodata.io, in accordance with open science standards.  -->


<h1>Introduction</h1>




<h3>Opportunity</h3>

<p>In the 21st century, big data (data too large to fit on your workstation) is transforming industry, governments, and science.  The growth of data acquisition has greatly outpaced the growth of data analysis, rendering current computational and statistical tools insufficient for extracting meaning from large datasets in many domains.  In the sciences, the human genome project led to a big data revolution.  A consortium of government entities deploy a suite of Web-services that enables geneticists to upload their new genetic sequences to a centralized data, and search for similar sequences across the existing data.  These features are utilized nearly daily by  most molecular geneticists.  These data  are typically one-dimensional (1D) sequences.  Similarly, sky surveys, such as the Sloan Digital Sky Survey, have transformed cosmology.  Rather than reserving telescope time months in advance, and hoping the sky was clear that day, cosmologists can browse the sky survey at any time, looking for anomalies or insights. Again, most cosmologists use these services daily. The cosmology data is 2D, rather than 1D, and so, a different design strategy went in to developing the necessary resources to support the community.
</p>


<p>Neuroscience is now entering the age of big data, as laboratories from around the world are beginning to acquire data that exceeds their computational capabilities.  These data can come for dizzying variety of experimental paradigms, ranging from serial electron microscopy to calcium imaging to multimodal magnetic resonance imaging. For many of these paradigms the data are massive 3D image stacks.  Thus, the existing computational solutions for big data do not meet the requirements for these data.  Moreover, neuroscience is rife with >3D data, as certain modalities have many different channels (for example, array tomography datasets might have up to 24 channels), and includes functional data (which might have many thousands of time steps).
All these different datasets have troves of information about brains lurking within, waiting to be extracted by the community, which will reveal the underlying principles of normal, abnormal and exceptional mental function.
</p>



<p>The focus of this work is serial electron microscopy (EM) data, a relatively novel yet important experimental modality, as it is unique in its ability to reveal nanoscale anatomical structure [@Denk04,etc.].  Such datasets can obtain terabytes (TB) a day; multiple such datasets already exist with ~100 TB (personal communication, <NAME> and <NAME>).
And with the new IARPA program dubbed MICrONS, a 4-10 petabyte (PB) dataset is planned to be collected in the next three years [@MICrONS].
Moreover, because it is single channel (one color) and single time step (anatomical), addressing big data analysis on these datasets will be a crucial first step prior to addressing big data analysis on multi-spectral, multi-temporal, and multi-scale data.  </p>


<h3>Challenge</h3>

<p>
As mentioned above, existing computational ecosystems for data analysis are ill equipped to meet the challenges that arise with conducting science on big neuroscience.   To explain why, we consider one of the two cases studies that we will explore further below. In particular, imagine that we have collected 10 TB of serial EM data from a cortical column, and we have the very simple neuroanatomical question to answer: how uniform is the 3D distribution of synapses? For example, is it clustered in 3D space,  or layered?  We divide the data analysis requirements into four components, each of which faces unique challenges:
</p>

<ol start="1">
<!--   <li>Infrastructure: Without a terascale infrastructure, neuroscientists have to build custom hardware and software to load sub-<em>cubes</em> of data into RAM (for example, in MATLAB or Python), because RAM typically only holds up to 16 GB of data, and the data of interest is orders of magnitude larger. </li> -->  
 <li>Storing: Most labs do not have many terabytes of storage, and even if they did, each scientist would need to dynamically choose where to put data, how to organize it, what format to use, etc.  Moreover, others could not easily access or operate on it. </li>
  <li>Exploring: Even visualizing data that is larger than RAM is non-trivial.  Current local solutions typically involve sub-sampling or downsampling the data, and loading just that subset into RAM, and remote solutions are either 2D, or lack 3D annotation metadata support. </li>
  <!-- <li>Pre-Processing: The raw data is never collected in a form amenable to parsing into semantic objects, rather, images must be stitched together, chromatic aberrations must be adjusted, and data must be aligned to other references to compare. Currently solutions typically require all the data to be loaded into RAM, and must be local.</li> -->
  <li>Parsing: Manually identifying all the ~10,000,000 synapses, at a rate of about 1 synapse per second, would take over a year (40 hrs/week, 50 weeks/year). Existing computational solutions, such as ilastik [@XXX], do not scale well on their own.</li>
  <li>Analyzing: Analyzing the statistical regularities (and irregularities) from the resulting data derivatives requires statistics incorporating space, shape, and graphs/networks.  Existing packages for such analyses either have limited capabilities, are expensive, or do not scale sufficiently.</li>
</ol>

<p>
If one group did manage to overcome all of those barriers, then they could answer this question and publish the results. But science is fundamentally a collective endeavor, where each result builds upon the previous works. With small data, the full analysis can easily be reproducible using existing technology: deposit the digital data in a data repository (like FigShare) and the code in a code repository (like GitHub). Although even the above is not yet standard practice in neuroscience, it could be.  With big data, even the above suggestion would not work, for a variety of reasons:
<ol>
  <li>Most data repositories will not except multi-terabyte datasets.
  <li>Even if they did, how would other researchers access the data? Would they have to download the whole dataset, or could they specify a subset.  If they have to download the whole dataset, then they will need to build their own infrastructure to support it.
  <li>Code to analyze terascale data is not yet "turn-key", that is, it is not as simple as running a MATLAB script.  Running the code often means having installed a number of libraries, some of which may only be compatible with certain operating systems.  Running and debugging other people's code often takes weeks per tool, and in the above, we have described many different tools.
</ol>
</p>


<p>
Thus, in practice, the effort of simply reproducing a previous result might require many months or years of work, often work that is distant from neuroscienctists' core competencies.  Moreover, science proceeds not merely be reproducing previous results, but by confirming and extending them.  This includes simple modifications such as checking the robustness of previous analyses by modifying some of the assumptions, tests, or the data on which the results rely.
Therefore, to the extent that one can make relatively minor changes to a previous analysis, re-using as much infrastructure as possible, the barrier to confirming and extending previous results to novel findings can be significantly lowered.
</p>



<h3>Action</h3>

<p>
We have built the <em>NeuroData</em> computational ecosystem to address the above described problems,  therefore enabling <em>reproducible</em> and <em>extensible</em> neuroscience regardless of scale. In particular, we have made a large number of big neuroscience datasets open access.  By storing all of the datasets, regardless of scale and modality, in a common framework, a single infrastructure is sufficient for analysis on each of the datasets.  This includes each of the above four  steps (store, explore, parse, and analyze).  Moreover, all of the code we have developed is open source, and divided into two different components: backend and frontend. The backend code is deployed on our Web-accessible data cluster.  This means that anybody can store and explore existing datasets, or contribute new ones.  For the frontend code, we have configured an Amazon Machine Instance (AMI) that can run any of the remaining steps (pre-process, parse, and infer).  This means that anybody can create an Amazon account, boot this system, and run any of our analyses, and get the exact same results, without ever installing any software or learning anything about big data analytics.
</p>

<h3>Resolution</h3>



<p>
Here we demonstrate that the NeuroData computational ecosystem can enable big data neuroscience.  In particular, we have used this infrastructure in two different case studies.  In the first, we have reproduced a large fraction of the  quantitative claims made by a  recent landmark EM paper [@Kasthuri15], including a claim about the probability of synapsing given spatial proximity of its parent neurons.  In the second, we used a significantly larger dataset (~10 TB) from another recent landmark EM paper [@Bock11].  From that data, we characterize the three dimensional distribution of ~10 million synapses in mammalian cortex, to discover significant clustering and layering that was previously unknown.
Perhaps more important than the particular questions we have addressed in this work, is the capability we have enabled for the greater neuroscience community.  Anybody---professional or citizen scientists alike---can use our services to learn about the brain.  Power users can modify our scripts to answer different questions, and developers could add additional capabilities for the community.  As the community moves more and more towards collecting massive datasets, we hope these tools can be useful to address a wide range of scientific questions, and even other disciplines that are collecting 3D+ data.
</p>






<h1>Results</h1>


All of the results presented herein are fundamentally reproducible; the data used to generate each table and figure panel can be reproduced by running a single Python script located in http://docs.neurodata.io/ocp-journal-paper.  Figures are made using R; each R script for generating the figures is also available from the above URL.  We have designed all of our infrastructure to support three different types of data: images, annotations, and graphs.  Because the focus of this manuscript is on electron microscopy data, for which scant graphs are currently available, we focus the results on images and annotations, and in particular on single channel images.  However, see http://neurodata.io/graphs for more details about graph processing and analytics, and see http://at.neurodata.io for more updates supporting multichannel data.


<h2>Infrastructure</h2>

.TODO[@kl: make figs]

<p>
Our infrastructure is designed for fast reading and writing of terabytes of data.  Figure~\ref{fig:infra} shows the read and write speeds for our system, including writing both images and annotations.  Although we operate our own data cluster for the vast majority of our public facing data, we have built an Amazon Machine Instance (AMI) that is pre-configured with all the necessary software; all the results presented in this section are using that AMI, and are therefore wholly reproducible.
</p>

<p>
Importantly *NeuroData* is designed to easily enable reading or writing of arbitrarily sized image tiles or cubes using a single URL or Python call.  For example, to download the image in Figure~\ref{fig:infra}(A), one can simply type http://openconnecto.me/ocp/ca/kasthuri11/image/xy/0/9000,9500/12000,12500/50/ into any browser bar.  In contrast, if one desired to download an arbitrary image at an arbitrary zoom from data stored in files in a typical hierarchical folder system, one would have to write code to navigate to the right files, open them up, possibly downsample them, and stitch them back together.  Our infrastructure has built in support for all of that, significantly both speeding up and easing access to the data; Figure~\ref{fig:infra}(B) shows reading speed as a function of cube size, showing that reading speed is linear in the size of the data.    Because it is easy to implement different services for reading data into different file types, we have built support for a variety of exceptional 3rd party tools, to complement this ecosystem.  This includes ITK-SNAP [@XXX], CATMAID [@XXX], Viking [@XXX], Knossos [@XXX], VAST [@XXX], and BigDataViewer [@XXX].


<p>
Naively using the above described system, however, has the drawbacks just mentioned.  Therefore, we built *TileCache* which serves as a content distribution network.  More specifically, TileCache fetches the data and parses it into the appropriate format (e.g., a set of png files), and also pre-fetches data that it expects you will want next.  The implication is that for 1028x1028 frames, TileCache can load 60 frames per second in a streaming fashion (Figure~\ref{fig:infra}(C)).  In other words, using TileCache, Neurodata enables video rate data streaming.  And because TileCache is a stand-alone service, it can easily be deployed locally, thereby further eliminating the data transfer speeds to get data local to the machine on which you want the data.


<p>
Writing to our spatial database would be more time consuming than simply uploading to a file server, because we re-write the data into a format that is efficient for reading.  We combat that by using a distributed database management system (such as Cassandra), that automatically distributes writing jobs across nodes.  We further speed up writing by compressing the data using Blosc, a multi-threaded library.  The result is that we can write data in compressed cubes in time linear with the data size (Figure~\ref{fig:infra}(D)).  The implications are the writing a terabyte of data on a single node take XXX hours.

<p>
Writing annotations is a bit more complicated than simply writing images because we store annotations as objects with associated metadata, such as what type of object is it (e.g., synapse), who labeled it, what time, with what confidence, etc. Because each write object is therefore relatively small, and write speeds are determined largely by the number of files to write, we built NeuroBlaze, which caches writes in memory, only writing to disk a single large file joining all the small writes.  This enables us to write  annotations extremely efficiently (Figure~\ref{fig:infra}(E)).

<div class="fig-container">
Figure 1: Reading and writing to the database must be sufficiently fast to not be the primary bottleneck.  (A) OCP can write images faster than most current data collection speeds, meaning that data can be sent directly to OCP without requiring a large local storage buffer.  (B) Reading images from OCP using the tilecache is sufficiently fast to view the data at video rates. (C) Unlike many other scientific big data efforts, neuroscience requires many fast annotation writes.  NeuroBlaze writes annotation objects much more efficiently than our image writing by caching the writes in memory. (D) Although current EM derived neurographs are relatively small, and there are few, we expect that to change dramatically in the coming years. Therefore, we have built infrastructure to support downloading a large number of large graphs efficiently.
</div>


<h2>Store</h2>

<p>
We used the above described infrastructure to build a number of image and annotation databases to support a variety of operations. <a href="#tab:images">Table 1(A)</a> lists the  public image datasets that we are currently hosting, as well as several key properties of each.  Those properties that are required for efficient access of the data are stored in a Django
<a href="http://docs.neurodata.io/open-connectome/sphinx/datamodel.html">data model</a>, the remaining properties are stored in our custom Laboratory Information Management System called <a href="https://github.com/openconnectome/ndlims">ndlims</a>.  The image data in <a href="#tab:images">Table 1(A)</a>  comprise approximately 80 teravoxels of image data.  These datasets span spatial scales of experimental neuroscience, ranging from nanoscale with electron microscopy to milliscale with MRI, and many in between.  We also store time-series data, as exemplified by the "Freeman14" dataset, a calcium imaging time-series. By storing disparate datasets in the same format, anybody can access and analyze many different datasets with the same functionality and syntax.
</p>


<p>
@KL TODO Use case
In the project one of the workload is to run computer vision algorithms at scale to detect cell bodie in the data. This workload can be split into three sections to provide a better understanding of the problem. First section, reading sub volumes of data from the spatial database using multiple readers. The NDStore backend is scalable and is capable of upto 9GB/sec read throughput one a single node running MySQL. Usually, we run multiple nodes providing much higher read throughtput. Second section, to detect cell bodies in the data using a large compute cluster. We will talk more about this in the next section. Third section, to write the detected cell bodies as annotation back to the database. We can achieve upto 1.5GB/sec write throughput with 16 threads on a single nodes running MySQL. With high read and write throughput we avoid bottlenecks in the first and third section for large scale runs.

</p>

<div id="par:private-data">
In addition to the above described public datasets, we also host a large number of private datasets, including a variety of additional modalities. This includes >300 array tomography datasets associated with the <a href="http://synapto.me">Open Synaptome Project</a>; approximately 20 different whole brain CLARITY datasets, including several multispectral datasets; several expansion microscopy datasets; and several X-ray microscopy datasets.   In total, we have XXX private datasets from XXX different collaborators spanning XXX total voxels.  Thus, NeuroData is both the largest and most diverse public neuroscience data repository in the world.
</div>



<p>
To complement the image databases, we also created an annotation database management system called [RamonDB](http://docs.neurodata.io/nddocs/ramon.html) [@GrayRoncal15b], and a richly attributed graph database management system called [RagDB](http://openconnecto.me/graphs/).  RAMONdb stores multiple manually and automatically generated dense volumetric annotations, listed in Table~\ref{tab:anno}(B) along with several key features of each.  Collectively, we have XXX voxels manually annotated, and YYY machine annotated.

<p>
Additionally, RagDB currently holds XXX different neurographs, ranging in species from *C. elegans* to humans (see [here](http://openconnecto.me/graph-services/download/) for details).  RagDB supports a variety of different graph functions (see [here](XXX link to ndio API for graphsXXX)), including building graphs from DTI data and downloading graphs.  Much like one can download neuroimages in a variety of formats (for example, png and hdf5), we also enable graphs to be downloaded in a variety of formats (for example, MATLAB and R readable formats).


<p>
<a href="#tab:API">Supplementary Table #tab:API</a> lists all the functionality currently supported on all of the neuroimage, neuroannotation, and neurograph databases.  Importantly, we have both MATLAB (called [CAJAL](http://docs.neurodata.io/CAJAL/)) and Python (called [ndio](http://docs.neurodata.io/nddocs/ndio/)) bindings to support that functionality, without the user having to learn a new programming language.


<div class="fig-container" id="tab:images">
Table 1: List of all <em>public</em> current (as of 12/31/15) NeuroData Datasets.  Table 1A lists the <em>image</em> datasets and associated metadata.  Table 1B lists the volumetric *annotation* datasets, derived from a subset of the image datasets.
{% include table1b.html %}
</div>

<div class="fig-container", id="fig:benchmarks1">
<img src="http://docs.neurodata.io/ocp-journal-paper/Results/Figures/store/ReadThroughputPlot.png" alt="Drawing" style="width: 700px;"/>
<img src="http://docs.neurodata.io/ocp-journal-paper/Results/Figures/store/WriteThroughputPlot.png" alt="Drawing" style="width: 700px;"/>
<img src="http://docs.neurodata.io/ocp-journal-paper/Results/Figures/store/tile_read_speed.png" alt="Drawing" style="width: 612px;"/>

Benchmark Figures:

</div>

<h2>Explore</h2>

<p>
Interactively exploring these massive data requires tools that can automatically scale appropriately and efficiently.  For this reason, we have developed a number of *NeuroData Explorers*.   For images, we designed *Image Explorer* (IX; http://ix.neurodata.io).  It is the only visualization engine that has the following functionality: (i) loads data remotely from a server, (ii) works in Web-browsers, including mobile, (iii) natively supports volumetric annotations with associated metadata.  In particular, one can visualize  electron microscopy data, and annotations on top, and also "query" the metadata associated with each annotation object (Figure~\ref{fig:explorer}(A)).   IX also supports multiple color channels
(Figure~\ref{fig:explorer}(B)) and time-series data (Figure~\ref{fig:explorer}(C)).



 and time-series data (see [here](http://ix.neurodata.io/Ex10R55/) for an example array tomography dataset [@Weiler14], and [here](http://ix.neurodata.io/freeman14/) for an example time-series dataset [@Freeman14]).

<p>
To complement IX, we also built *Graph Explorer* (GX; http://gx.neurodata.io) and *Vector Explorer* (VX; http://vx.neurodata.io) (Figure~\ref{fig:explorer}(B) and (C)).  These two explorers enable users to upload either graphs or tabular data (where rows are subjects and columns are features, as in a csv file).  Both of these tools provide very basic functionality for graphs or tabular data that we find useful for exploring the relevant data types, including different normalization schemes, outlier detection, distributions of various features, and clustering.

<div class="fig-container">
Figure 2: Exploring the data is a crucial first step, regardless of the data type, therefore, we have built data explorers for images, vectors, and graphs. All subpanels are screenshots using these web-services.
(A) IX enables overlaying annotation layers and querying them for metadata details ( http://brainviz1.cs.jhu.edu/ndv/kharris15apical/em,ramon_test/3/507/469/90/),
(B) multispectral data (http://brainviz1.cs.jhu.edu/ndv/project/Aratome15c_S17_W10/1/5818/527/0/), including various blending and other color operations, and
(C) time-series data (http://viz.neurodata.io/freeman14/), including auto-play functions.

(D) GX facilitates computing various graph statistics, community detection, and plotting for arbitrary uploaded or generated graphs (http://gx.neurodata.io).

(E) Vector Explorer allows the user to upload a csv file and do basic exploratory data analysis (http://ix.neurodata.io).
</div>

<h2>Pre-process</h2>

<p>
Pre-processing of data, a process that converts raw data into a format ready for parsing, is an incredibly important and tedious task for essentially any data analysis task.  In electron microscopy, we can divide pre-processing into three steps: (i) geometric or (ii) chromatic alterations, and (iii) registration to a template or atlas. A number of groups are actively developing very nice geometric pre-processing packages, including Terastitcher [@XXX] and Saalfeld's Fiji plug-in [@XXX].  We therefore have focused our development efforts on the next two steps.  We built both a 2D and 3D chromatic pre-processing tool. The 2D tool, called *Distriubuted Multigrid Poisson* (DMG) addresses stitching artifacts from montaging multiple 2D tiles into a single large plane (Figure~\ref{fig:pre}(A)).  The 3D tool, called *Gradient Domain Fusion* (GDF) addresses histogram artifacts from different exposure times which can cause a strobe light effect across axial slices of the data (Figure~\ref{fig:pre}(B)).  Both tools were designed to stream the data efficiently, meaning that they can both operate on arbitrarily large data (Figure~\ref{fig:pre}(C)).

<p>
After addressing both geometric and chromatic aberrations, it is often desirable to align the image stack to a reference image, possibly in a different modality.  *ndreg* is our Python tool that sub-serves that functionality.  Because there does not yet exist public data set with both EM and another modality to align, we have aligned a whole mouse brain CLARITY image with the Allen Reference Atlas (Figure~\ref{fig:pre}(D)).  Despite the fact that the two images were collected at different scales, and using different modalities, we are able to successfully and automatically align the two using *ndreg*.

<div class="fig-container">
Figure 3: *NeuroData* provides fully automatic terascale tools for both 2D & 3D chromatic adjustments, as well as nonlinear volume registration.  (A) DMG Chromatic pre-processing adjusts for aberrations induces by data collection and/or geometric stitching. (Bi) 2D image before DMG. (Bii) 2D image after DMG.  (Biii) XZ projection of data before GDF. (Biv) XZ projection after GDF.  (Bv) Time per image size (red) and number of slices (black) on a particular AMI.  (C) Volume registration.
</div>

<h2>Parse</h2>


<p>
Given that the data has been sufficiently pre-processed, we now proceed to parse the scene, assigning voxels with semantic labels.  Parsing the scene involves three steps: (i) manually labeling a subset of training data, (ii) training and evaluating various machine vision algorithms on appropriate performance metrics, and (iii) deploying the algorithm (and parameter set) of choice at scale.  For manual labeling, we built [*manno*](http://docs.neurodata.io/manno/), a protocol that leverages ITK-SNAP by downloading image data from *NeuroData* in the appropriate format, uses ITK-SNAP to label a number of voxels, and then saves and uploads the annotations to the appropriate annotation database (with metadata; see Figure~\ref{fig:parse}(A) for an exemplar slice).  Given the training labels, [*macho*](http://docs.neurodata.io/macho/) uses ilastik to perform pixel level classification [@XXX], and then custom morphological operations to detect objects.  We choose the operating point that maximized f1-score on a held-out dataset located in a volume separated from the training volume (Figure~\ref{fig:parse}(B)).  Finally, we deployed our chosen classifier on the full dataset, using [*maxxo*](http://docs.neurodata.io/maxxo/), which utilizes the LONI Pipeline to distribute ilastik across many nodes of a HPC.  The result was ~11 million putative synapse detections (Figure~\ref{fig:parse}(C) shows approximately 2% of them).

<div class="fig-container" id="tab:images">
Figure 4: A schematic showing the synapses labeled with our manual annotation (mana) protocol (left) and the resulting parased scene detections using vesicle (right).  These figures, and associated associated error metrics can be reproduced using the Jupyter notebook provided with this manuscript. <br>

<img src="http://docs.neurodata.io/ocp-journal-paper/Results/Figures/parse/ndp_bock_truth_crop.png" alt="Drawing" style="width: 500px;"/>
<br>
<br>
<img src="http://docs.neurodata.io/ocp-journal-paper/Results/Figures/parse/ndp_bock_detect_crop.png" alt="Drawing" style="width: 500px;"/>
</div>


<div class="fig-container", id="fig:allsyn">
<div><img src="http://docs.neurodata.io/ndintro/images/rob2.png" alt="Drawing" style="height: 250px;"/></div>
Figure 4:

</div>


<h2>Infer</h2>

<p>
Inference has relatively task specific requirements, although statistical inference from anatomical data has some shared features.  In particular, in anatomy,  the questions typically include space, shape, and graphs.  Spatial statistics is a relatively developed branch of statistics, and we therefore can rely on previously developed methods [@Cressie15], though with novel implementations.  The statistics of shape is a much more recent sub-discipline in statistics [@Younes10], partially because it depends on the mathematics of nonlinear manifolds, which are still being actively developed [@Chikuse03].  We therefore developed novel implementations of several of these key quantities.  Finally, graph statistics is a similarly new field [@Kolaczyk09], as graphs are discrete combinatorial objects that require careful mathematical treatment [@Bollobas13].  Again, we have developed our own methods, and built our own scalable and open source implementations (http://flashx.io).  Below, we provide two case studies, one using spatial statistics, the other using graph statistics.
</p>




<h1>Case Studies</h1>


<h2>Reproducing Quantitative Claims from Kasthuri et al. Cell (2012)</h2>


<p>
Several months ago a landmark paper was published, Kasthuri et al. Cell (2015) [@Kasthuri15].  In it, the authors heroically manually annotated a subvolume of somatosensory cortex, and provided three things:
<ol>
  <li>a paragraphical "parts list" of all the anatomical objects in a small volume,
  <li>an excel file listing all 1,700 synapses, as well as over 20 properties of each (for example, post-synaptic density centroid location and size), and
  <li>a VAST remote volume file containing the manual labels, as well as some additional information (object type, and parents).
</ol>


We converted the latter two of those objects into the RAMON annotation data specification using CAJAL, and ingested the data into RamonDB.  One immediate implication of this is that we could check for consistency between the provided excel spreadsheet and the VAST data.  Indeed, we discovered that for XXX% of the synapses in the spreadsheet, the location provided did not match any synapses in the actual VAST export.  Similarly, the VAST export contained XXX synapses that we could not find in the spreadsheet.  While this may seem to be a banality, checking the data consistency is the first step for any data analysis, and often the first "result" is idiosyncracies that can be corrected.  Of the inconsistencies, we ignored some that we could not figure out how to correct, and corrected the rest to proceed.
</p>


<p>
We next proceeded to assess each of the quantitative claims from the paragraphical parts list that amonut to counting particular kinds of objects or computing volumes.  Within VAST, there is no way to compute these quantitites, and the spreadsheet does not contain enough information to extract them all (because it only includes synapses, and doesn't include tight boundaries).  Therefore, only via using RamonDB, can one automatically and reproducibly compute all these properties.  Table [#tab:parts] is the automatically generated table from our Python script, including all quantitative claims from the manuscript.
</p>

<p>
 In addition to a parts list, the authors of Kasthuri et al. tested a hypothesis about the relationship between spatial proximity of axon-dendrite pairs and the number of synapses between those pairs.  In particular, the question is: does spatial proximity on its own explain number of connections, or is that model insufficient to characterize the data? This question has recently been hotly debated [@Markram15].  Testing this hypothesis is difficult due to the difficulty of randomly sampling graphs with a given degree distribution [@http://arxiv.org/abs/1301.6635].  We therefore developed a novel methodology that is statistically principled, to obtain a p-value of XXX.  Our code for testing this hypothesis has been incoroprated into FlashX.
</p>




<h2>Discovering the Spatial Distribution of Synapses using the Data from Bock et al. Nature (2011)</h2>

The spatial distribution of synapses is a fundamental property of neural tissue, and one that has seen some study in the past.  In particular, two recent very nice studies examined the spatial distribution of synapses in rat cortex [@http://cercor.oxfordjournals.org/content/24/6/1579.long, @http://journal.frontiersin.org/article/10.3389/fnana.2014.00085/abstract].  Using a few thousand synapses, they fail to reject the hypothesis that synapses are distributed according to a random sequential adsoption model, implying that their synapses are distributed almost randomly, with the only constraint that they cannot overlap.  Their experimental design included multiple small volumes from different animals, which is often a wise experimental design choice, to mitigate batch effects [@http://www.nature.com/nrg/journal/v11/n10/full/nrg2825.html].  In contrast, we used a single large volume from Bock et al. Nature (2011).  Using the combination of (manno,macho,maxxo) described above, we detected ~10 million synapses, a number comensurate with the expected density of synapses per volume reported in the literature.  Our complementary experimental design decision enabled us to consider the spatial distribution across a much larger volume, in case parts of the volume look approximately uniform, and other parts do not.  Indeed, Figure [#fig:syn-clust](A) depicts the spatial distribution of synapses detected in our volume.  The size of the dots corresponds to the relative density of synapses.  It is immediately apparent that synapses are <em>not</em> distributed uniformly in space; rather, there is a clear cluster pattern.  The inset in the figure shows a volume of 200 cubic microns, the size of the volumes used in the previous publications.  It would be difficult indeed to detect anisotropies from volumes of this size, given the nature of the non-uniformity. Figure [#fig:syn-clust](B) shows the 2D histograms projected onto the three canonical planes.  Even ignoring the edge effects, again, it is clear that the synapses are not distributed uniformly in space.



<div class="fig-container", id="fig:allsyn">
<img src="http://docs.neurodata.io/ndintro/images/3D.png" alt="Drawing" style="width: 300px;"/>
<img src="http://docs.neurodata.io/ndintro/images/2Dproj.png"    alt="Drawing" style="width: 300px; "/>

Figure:

</div>


<h1>Discussion</h1>



<h1>Methods</h1>


<h2>Store</h2>

<h3>Databases</h3>

<p>
You can reproduce our Read/Write throughput benchmarks using the following steps:
<ul>
  <li>Launch the AWS AMI "ami-id Here".</li>
  <li>Run the <a href="https://github.com/neurodata/ndstore/tree/microns/benchmarks/ndstore_benchmark.py">ndstore_benchmark.py</a> in the <a href="https://github.com/neurodata/ndstore/tree/microns/benchmarks">benchmarks</a> folder. The commands to run the read and write benchmarks are mentioned in the <a href="https://github.com/neurodata/ndstore/blob/microns/benchmarks/README.md">README</a> file. These commands will generate csv files for the respective benchmarks.</li>
  <li>Run the <a href=" https://github.com/neurodata/ocp-journal-paper/blob/gh-pages/Code/Plots/ThroughputPlots.r">ThroughPlots.r</a> script in the <a href="https://github.com/neurodata/ocp-journal-paper/blob/gh-pages/Code/Plots/">Plots</a> folder. This script will generate the thorughput graphs included in the paper.</li>
</ul>
You can reproduce our Tilecache benchmarks using the following steps:
<ul>
  <li>Launch the AWS AMI "ami-id here".</li>
  <li>Run the <a href="https://github.com/neurodata/ndtilecache/blob/master/benchmarks/ndtilecache_benchmark.py">ndtilecache_benchmark.py</a> script in the <a href="https://github.com/neurodata/ndtilecache/blob/master/benchmarks/">benchmarks</a>folder. The command to run the benchmarks are mentioned in the <a href="https://github.com/neurodata/ndtilecache/blob/master/benchmarks/README.md">README file.</a> file. These commands will generate the csv file for the benchmark.</li>
  <li>Run the <a href="">TilecachePlots.r</a> script in the <a href="">Plots</a> folder. This script will generate the tilecache graph in the paper.</li>
</ul>
</p>

<h2>Explore</h2>

<h2>Parse</h2>

<h2>Analyze</h2>


---

<h1>Appendix</h1>



<div class="fig-container", id="fig:infra">
Supplementary Figure 1: Infrastructure supporting all components of the OCP data analysis pipeline.  (A) The OCP Data Cluster. (B) Software running on the backend. (C) Software running on the frontend.
</div>


<div class="fig-container", id="tab:api">
Supplementary Table 1: Functions supported in our databases.  (A) NeuroImages. (B) NeuroAnnotations. (C) NeuroGraphs.
</div>


<div class="fig-container", id="tab:parts">
Supplementary Figure 1: Infrastructure supporting all components of the OCP data analysis pipeline.  (A) The OCP Data Cluster. (B) Software running on the backend. (C) Software running on the frontend.
</div>



	<script type="text/javascript">
        m = markdownit({
                html: true,
                linkify: true,
                typographer: true
        });
        document.getElementById('md-raw').innerHTML = m.render(
                document.getElementById('md-raw').innerHTML);
     </script>

    <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
    <!-- Include all compiled plugins (below), or include individual files as needed -->
    <script src="/Web/js/bootstrap.min.js"></script>
    </body>
</html>
<commit_msg>re-arranged stuff<commit_after><!DOCTYPE html>
<html>
    <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
    <title>OCP</title>

    <!-- Bootstrap -->
    <link href="Web/css/bootstrap.min.css" rel="stylesheet">


    <style type="text/css">
      @import url(https://fonts.googleapis.com/css?family=Yanone+Kaffeesatz);
      @import url(https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic);
      @import url(https://fonts.googleapis.com/css?family=Ubuntu+Mono:400,700,400italic);
      @import url(http://fonts.googleapis.com/css?family=Varela+Round:regular,italic,bold,bolditalic);
      @import url(http://fonts.googleapis.com/css?family=Raleway:regular,italic,bold,bolditalic);

      body {
        font-family: 'Varela Round'; /*  */
        background-color: #fff;
        position: absolute;
        left: 20px;
        max-width: 640px;

      }
      h1, h2, h3, h4, h5, h6 {
        font-family: 'Raleway';
        font-weight: 400;
        margin-bottom: 0;
      }
      p.small {
        font-variant: small-caps;
      }
      .r { color: #fa0000; }
      .fig-container {
        font-size: small;
        border-top: 5px solid #efefef;
        border-bottom: 5px solid #efefef;
        padding: 1em;
      }

      thead {
          background: #ccc;
          font-weight: bold;
      }

      tr:nth-child(2n) {
          background-color: #efefef;
      }

      td, th {
          padding: 1em;
      }
      </style>

    </head>
    <body>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/markdown-it/5.0.2/markdown-it.js" charset="utf-8"></script>
<div id="md-raw">

# NeuroData: Enabling Petascale Neurosciene

## by, everyone
---


<h1>Abstract</h1>

<p>
Recent technical progress allows neuroexperimentalists to collect ever more detailed and informative anatomical and physiological data from brains of all sizes.  These datasets span experimentally accessible spatiotemporal scales, ranging from nanometer to meter, and millisecond to monthly sampling rates.  In classical neuroscientific experimental paradigms, it was feasible for neuroscientists to draw their results on paper. In contrast, many modern neuroscientific experimental paradigms break the classic data analysis workflow.  In particular, these large datasets create significant challenges for our community at every step of the data analysis pipeline: (1) storing, (2) exploring, (3) parsing, and (4)  analyzing.
</p>

<p>
*NeuroData* has been developed to lower the barrier to entry into big data neuroscience. We have designed and built a computational infrastructure to enable petascale neuroscience.  This includes our flagship project, the *NeuroData's Connectome Project* (previously called the Open Connectome Project). Our infrastructure enables anyone in the world with internet access to visualize, download, analyze, upload, or otherwise interact with a large number public datasets.  Moreover, all the results obtained using the NeuroData infrastructure are fundamentally reproducible and extensible.  We demonstrate the utility of these tools via two serial electron microscopy case studies.  First, we have reproduced many of the quantitative results from a recent landmark EM paper that included a saturated annotation of all objects. Second, we have tested a novel hypothesis about the distribution of synapse locations in cortex on a complementary much larger dataset.  Via the *NeuroData* infrastructure, the answers to these tools are both fully reproducible, and extensible to any other datasets that hold the answers to these questions. Moreover, anybody else can now also do neuroscience at scale, regardless of their background, computational resources, and expertise.
</p>
</div>

<!-- We are in the process of scaling up the number of datasets, the range of experimental modalities, and the Web-services we enable. Work underway will provide pre-packaged cluster environments—1-click deployable on local or commercial cloud computing infrastructures—so that others can replicate and modify our services internally. All of our code and data are available online at neurodata.io, in accordance with open science standards.  -->


<h1>Introduction</h1>




<h3>Opportunity</h3>

<p>In the 21st century, big data (data too large to fit on your workstation) is transforming industry, governments, and science.  The growth of data acquisition has greatly outpaced the growth of data analysis, rendering current computational and statistical tools insufficient for extracting meaning from large datasets in many domains.  In the sciences, the human genome project led to a big data revolution.  A consortium of government entities deploy a suite of Web-services that enables geneticists to upload their new genetic sequences to a centralized data, and search for similar sequences across the existing data.  These features are utilized nearly daily by  most molecular geneticists.  These data  are typically one-dimensional (1D) sequences.  Similarly, sky surveys, such as the Sloan Digital Sky Survey, have transformed cosmology.  Rather than reserving telescope time months in advance, and hoping the sky was clear that day, cosmologists can browse the sky survey at any time, looking for anomalies or insights. Again, most cosmologists use these services daily. The cosmology data is 2D, rather than 1D, and so, a different design strategy went in to developing the necessary resources to support the community.
</p>


<p>Neuroscience is now entering the age of big data, as laboratories from around the world are beginning to acquire data that exceeds their computational capabilities.  These data can come for dizzying variety of experimental paradigms, ranging from serial electron microscopy to calcium imaging to multimodal magnetic resonance imaging. For many of these paradigms the data are massive 3D image stacks.  Thus, the existing computational solutions for big data do not meet the requirements for these data.  Moreover, neuroscience is rife with >3D data, as certain modalities have many different channels (for example, array tomography datasets might have up to 24 channels), and includes functional data (which might have many thousands of time steps).
All these different datasets have troves of information about brains lurking within, waiting to be extracted by the community, which will reveal the underlying principles of normal, abnormal and exceptional mental function.
</p>



<p>The focus of this work is serial electron microscopy (EM) data, a relatively novel yet important experimental modality, as it is unique in its ability to reveal nanoscale anatomical structure [@Denk04,etc.].  Such datasets can obtain terabytes (TB) a day; multiple such datasets already exist with ~100 TB (personal communication, <NAME> and <NAME>).
And with the new IARPA program dubbed MICrONS, a 4-10 petabyte (PB) dataset is planned to be collected in the next three years [@MICrONS].
Moreover, because it is single channel (one color) and single time step (anatomical), addressing big data analysis on these datasets will be a crucial first step prior to addressing big data analysis on multi-spectral, multi-temporal, and multi-scale data.  </p>


<h3>Challenge</h3>

<p>
As mentioned above, existing computational ecosystems for data analysis are ill equipped to meet the challenges that arise with conducting science on big neuroscience.   To explain why, we consider one of the two cases studies that we will explore further below. In particular, imagine that we have collected 10 TB of serial EM data from a cortical column, and we have the very simple neuroanatomical question to answer: how uniform is the 3D distribution of synapses? For example, is it clustered in 3D space,  or layered?  We divide the data analysis requirements into four components, each of which faces unique challenges:
</p>

<ol start="1">
<!--   <li>Infrastructure: Without a terascale infrastructure, neuroscientists have to build custom hardware and software to load sub-<em>cubes</em> of data into RAM (for example, in MATLAB or Python), because RAM typically only holds up to 16 GB of data, and the data of interest is orders of magnitude larger. </li> -->  
 <li>Storing: Most labs do not have many terabytes of storage, and even if they did, each scientist would need to dynamically choose where to put data, how to organize it, what format to use, etc.  Moreover, others could not easily access or operate on it. </li>
  <li>Exploring: Even visualizing data that is larger than RAM is non-trivial.  Current local solutions typically involve sub-sampling or downsampling the data, and loading just that subset into RAM, and remote solutions are either 2D, or lack 3D annotation metadata support. </li>
  <!-- <li>Pre-Processing: The raw data is never collected in a form amenable to parsing into semantic objects, rather, images must be stitched together, chromatic aberrations must be adjusted, and data must be aligned to other references to compare. Currently solutions typically require all the data to be loaded into RAM, and must be local.</li> -->
  <li>Parsing: Manually identifying all the ~10,000,000 synapses, at a rate of about 1 synapse per second, would take over a year (40 hrs/week, 50 weeks/year). Existing computational solutions, such as ilastik [@XXX], do not scale well on their own.</li>
  <li>Analyzing: Analyzing the statistical regularities (and irregularities) from the resulting data derivatives requires statistics incorporating space, shape, and graphs/networks.  Existing packages for such analyses either have limited capabilities, are expensive, or do not scale sufficiently.</li>
</ol>

<p>
If one group did manage to overcome all of those barriers, then they could answer this question and publish the results. But science is fundamentally a collective endeavor, where each result builds upon the previous works. With small data, the full analysis can easily be reproducible using existing technology: deposit the digital data in a data repository (like FigShare) and the code in a code repository (like GitHub). Although even the above is not yet standard practice in neuroscience, it could be.  With big data, even the above suggestion would not work, for a variety of reasons:
<ol>
  <li>Most data repositories will not except multi-terabyte datasets.
  <li>Even if they did, how would other researchers access the data? Would they have to download the whole dataset, or could they specify a subset.  If they have to download the whole dataset, then they will need to build their own infrastructure to support it.
  <li>Code to analyze terascale data is not yet "turn-key", that is, it is not as simple as running a MATLAB script.  Running the code often means having installed a number of libraries, some of which may only be compatible with certain operating systems.  Running and debugging other people's code often takes weeks per tool, and in the above, we have described many different tools.
</ol>
</p>


<p>
Thus, in practice, the effort of simply reproducing a previous result might require many months or years of work, often work that is distant from neuroscienctists' core competencies.  Moreover, science proceeds not merely be reproducing previous results, but by confirming and extending them.  This includes simple modifications such as checking the robustness of previous analyses by modifying some of the assumptions, tests, or the data on which the results rely.
Therefore, to the extent that one can make relatively minor changes to a previous analysis, re-using as much infrastructure as possible, the barrier to confirming and extending previous results to novel findings can be significantly lowered.
</p>



<h3>Action</h3>

<p>
We have built the <em>NeuroData</em> computational ecosystem to address the above described problems,  therefore enabling <em>reproducible</em> and <em>extensible</em> neuroscience regardless of scale. In particular, we have made a large number of big neuroscience datasets open access.  By storing all of the datasets, regardless of scale and modality, in a common framework, a single infrastructure is sufficient for analysis on each of the datasets.  This includes each of the above four  steps (store, explore, parse, and analyze).  Moreover, all of the code we have developed is open source, and divided into two different components: backend and frontend. The backend code is deployed on our Web-accessible data cluster.  This means that anybody can store and explore existing datasets, or contribute new ones.  For the frontend code, we have configured an Amazon Machine Instance (AMI) that can run any of the remaining steps (pre-process, parse, and infer).  This means that anybody can create an Amazon account, boot this system, and run any of our analyses, and get the exact same results, without ever installing any software or learning anything about big data analytics.
</p>

<h3>Resolution</h3>



<p>
Here we demonstrate that the NeuroData computational ecosystem can enable big data neuroscience.  In particular, we have used this infrastructure in two different case studies.  In the first, we have reproduced a large fraction of the  quantitative claims made by a  recent landmark EM paper [@Kasthuri15], including a claim about the probability of synapsing given spatial proximity of its parent neurons.  In the second, we used a significantly larger dataset (~10 TB) from another recent landmark EM paper [@Bock11].  From that data, we characterize the three dimensional distribution of ~10 million synapses in mammalian cortex, to discover significant clustering and layering that was previously unknown.
Perhaps more important than the particular questions we have addressed in this work, is the capability we have enabled for the greater neuroscience community.  Anybody---professional or citizen scientists alike---can use our services to learn about the brain.  Power users can modify our scripts to answer different questions, and developers could add additional capabilities for the community.  As the community moves more and more towards collecting massive datasets, we hope these tools can be useful to address a wide range of scientific questions, and even other disciplines that are collecting 3D+ data.
</p>






<h1>Methods</h1>


All of the results presented herein are fundamentally reproducible; the data used to generate each table and figure panel can be reproduced by running a single Python script located in http://docs.neurodata.io/ocp-journal-paper.  Figures are made using R; each R script for generating the figures is also available from the above URL.  We have designed all of our infrastructure to support three different types of data: images, annotations, and graphs.  Because the focus of this manuscript is on electron microscopy data, for which scant graphs are currently available, we focus the results on images and annotations, and in particular on single channel images.  However, see http://neurodata.io/graphs for more details about graph processing and analytics, and see http://at.neurodata.io for more updates supporting multichannel data.


<h2>Store</h2>

<h3>Databases</h3>

<p>
Our infrastructure is designed for fast reading and writing of terabytes of data.  Figure~\ref{fig:infra} shows the read and write speeds for our system, including writing both images and annotations.  Although we operate our own data cluster for the vast majority of our public facing data, we have built an Amazon Machine Instance (AMI) that is pre-configured with all the necessary software; all the results presented in this section are using that AMI, and are therefore wholly reproducible.
</p>

<p>
Importantly *NeuroData* is designed to easily enable reading or writing of arbitrarily sized image tiles or cubes using a single URL or Python call.  For example, to download the image in Figure~\ref{fig:infra}(A), one can simply type http://openconnecto.me/ocp/ca/kasthuri11/image/xy/0/9000,9500/12000,12500/50/ into any browser bar.  In contrast, if one desired to download an arbitrary image at an arbitrary zoom from data stored in files in a typical hierarchical folder system, one would have to write code to navigate to the right files, open them up, possibly downsample them, and stitch them back together.  Our infrastructure has built in support for all of that, significantly both speeding up and easing access to the data; Figure~\ref{fig:infra}(B) shows reading speed as a function of cube size, showing that reading speed is linear in the size of the data.    Because it is easy to implement different services for reading data into different file types, we have built support for a variety of exceptional 3rd party tools, to complement this ecosystem.  This includes ITK-SNAP [@XXX], CATMAID [@XXX], Viking [@XXX], Knossos [@XXX], VAST [@XXX], and BigDataViewer [@XXX].
</p>

<p>
Naively using the above described system, however, has the drawbacks just mentioned.  Therefore, we built *TileCache* which serves as a content distribution network.  More specifically, TileCache fetches the data and parses it into the appropriate format (e.g., a set of png files), and also pre-fetches data that it expects you will want next.  The implication is that for 1028x1028 frames, TileCache can load 60 frames per second in a streaming fashion (Figure~\ref{fig:infra}(C)).  In other words, using TileCache, Neurodata enables video rate data streaming.  And because TileCache is a stand-alone service, it can easily be deployed locally, thereby further eliminating the data transfer speeds to get data local to the machine on which you want the data.
</p>

<p>
Writing to our spatial database would be more time consuming than simply uploading to a file server, because we re-write the data into a format that is efficient for reading.  We combat that by using a distributed database management system (such as Cassandra), that automatically distributes writing jobs across nodes.  We further speed up writing by compressing the data using Blosc, a multi-threaded library.  The result is that we can write data in compressed cubes in time linear with the data size (Figure~\ref{fig:infra}(D)).  The implications are the writing a terabyte of data on a single node take XXX hours.
</p>

<p>
Writing annotations is a bit more complicated than simply writing images because we store annotations as objects with associated metadata, such as what type of object is it (e.g., synapse), who labeled it, what time, with what confidence, etc. Because each write object is therefore relatively small, and write speeds are determined largely by the number of files to write, we built NeuroBlaze, which caches writes in memory, only writing to disk a single large file joining all the small writes.  This enables us to write  annotations extremely efficiently (Figure~\ref{fig:infra}(E)).
</p>

<div class="fig-container", id="fig:benchmarks1">
<img src="http://docs.neurodata.io/ocp-journal-paper/Results/Figures/store/ReadThroughputPlot.png" alt="Drawing" style="width: 640px;"/>
<img src="http://docs.neurodata.io/ocp-journal-paper/Results/Figures/store/WriteThroughputPlot.png" alt="Drawing" style="width: 640px;"/>
<img src="http://docs.neurodata.io/ocp-journal-paper/Results/Figures/store/tile_read_speed.png" alt="Drawing" style="width: 560px;"/>
<br>
Figure 1: Reading and writing to the database must be sufficiently fast to not be the primary bottleneck.  (A) OCP can write images faster than most current data collection speeds, meaning that data can be sent directly to OCP without requiring a large local storage buffer.  (B) Reading images from OCP using the tilecache is sufficiently fast to view the data at video rates. (C) Unlike many other scientific big data efforts, neuroscience requires many fast annotation writes.  NeuroBlaze writes annotation objects much more efficiently than our image writing by caching the writes in memory. (D) Although current EM derived neurographs are relatively small, and there are few, we expect that to change dramatically in the coming years. Therefore, we have built infrastructure to support downloading a large number of large graphs efficiently.
</div>



<p>
In the project one of the workload is to run computer vision algorithms at scale to detect cell bodie in the data. This workload can be split into three sections to provide a better understanding of the problem. First section, reading sub volumes of data from the spatial database using multiple readers. The NDStore backend is scalable and is capable of upto 9GB/sec read throughput one a single node running MySQL. Usually, we run multiple nodes providing much higher read throughtput. Second section, to detect cell bodies in the data using a large compute cluster. We will talk more about this in the next section. Third section, to write the detected cell bodies as annotation back to the database. We can achieve upto 1.5GB/sec write throughput with 16 threads on a single nodes running MySQL. With high read and write throughput we avoid bottlenecks in the first and third section for large scale runs.
</p>



<h3>Utilities</h3>


<p>
<a href="#tab:API">Supplementary Table #tab:API</a> lists all the functionality currently supported on all of the neuroimage, neuroannotation, and neurograph databases.  Importantly, we have both MATLAB (called [CAJAL](http://docs.neurodata.io/CAJAL/)) and Python (called [ndio](http://docs.neurodata.io/nddocs/ndio/)) bindings to support that functionality, without the user having to learn a new programming language.
</p>

<h3>Hardware</h3>


<h2>Explore</h2>


<h3>Images</h3>


<p>
Interactively exploring these massive data requires tools that can automatically scale appropriately and efficiently.  For this reason, we have developed a number of *NeuroData Explorers*.   For images, we designed *Image Explorer* (IX; http://ix.neurodata.io).  It is the only visualization engine that has the following functionality: (i) loads data remotely from a server, (ii) works in Web-browsers, including mobile, (iii) natively supports volumetric annotations with associated metadata.  In particular, one can visualize  electron microscopy data, and annotations on top, and also "query" the metadata associated with each annotation object (Figure~\ref{fig:explorer}(A)).   IX also supports multiple color channels
(Figure~\ref{fig:explorer}(B)) and time-series data (Figure~\ref{fig:explorer}(C)),
and time-series data (see [here](http://ix.neurodata.io/Ex10R55/) for an example array tomography dataset [@Weiler14], and [here](http://ix.neurodata.io/freeman14/) for an example time-series dataset [@Freeman14]).
</p>



<div class="fig-container">
Figure 2: Exploring the data is a crucial first step, regardless of the data type, therefore, we have built data explorers for images, vectors, and graphs. All subpanels are screenshots using these web-services.
(A) IX enables overlaying annotation layers and querying them for metadata details ( http://brainviz1.cs.jhu.edu/ndv/kharris15apical/em,ramon_test/3/507/469/90/),
(B) multispectral data (http://brainviz1.cs.jhu.edu/ndv/project/Aratome15c_S17_W10/1/5818/527/0/), including various blending and other color operations, and
(C) time-series data (http://viz.neurodata.io/freeman14/), including auto-play functions.

(D) GX facilitates computing various graph statistics, community detection, and plotting for arbitrary uploaded or generated graphs (http://gx.neurodata.io).

(E) Vector Explorer allows the user to upload a csv file and do basic exploratory data analysis (http://ix.neurodata.io).
</div>



<h3>Graphs</h3>

<p>
To complement IX, we also built *Graph Explorer* (GX; http://gx.neurodata.io) and *Vector Explorer* (VX; http://vx.neurodata.io) (Figure~\ref{fig:explorer}(B) and (C)).  These two explorers enable users to upload either graphs or tabular data (where rows are subjects and columns are features, as in a csv file).  Both of these tools provide very basic functionality for graphs or tabular data that we find useful for exploring the relevant data types, including different normalization schemes, outlier detection, distributions of various features, and clustering.
</p>

<h3>Matrices</h3>


<h2>Parse</h2>

<h3>Image Processing</h3>


<p>
Pre-processing of data, a process that converts raw data into a format ready for parsing, is an incredibly important and tedious task for essentially any data analysis task.  In electron microscopy, we can divide pre-processing into three steps: (i) geometric or (ii) chromatic alterations, and (iii) registration to a template or atlas. A number of groups are actively developing very nice geometric pre-processing packages, including Terastitcher [@XXX] and Saalfeld's Fiji plug-in [@XXX].  We therefore have focused our development efforts on the next two steps.  We built both a 2D and 3D chromatic pre-processing tool. The 2D tool, called *Distriubuted Multigrid Poisson* (DMG) addresses stitching artifacts from montaging multiple 2D tiles into a single large plane (Figure~\ref{fig:pre}(A)).  The 3D tool, called *Gradient Domain Fusion* (GDF) addresses histogram artifacts from different exposure times which can cause a strobe light effect across axial slices of the data (Figure~\ref{fig:pre}(B)).  Both tools were designed to stream the data efficiently, meaning that they can both operate on arbitrarily large data (Figure~\ref{fig:pre}(C)).
</p>

<p>
After addressing both geometric and chromatic aberrations, it is often desirable to align the image stack to a reference image, possibly in a different modality.  *ndreg* is our Python tool that sub-serves that functionality.  Because there does not yet exist public data set with both EM and another modality to align, we have aligned a whole mouse brain CLARITY image with the Allen Reference Atlas (Figure~\ref{fig:pre}(D)).  Despite the fact that the two images were collected at different scales, and using different modalities, we are able to successfully and automatically align the two using *ndreg*.
</p>


<div class="fig-container">
Figure 3: *NeuroData* provides fully automatic terascale tools for both 2D & 3D chromatic adjustments, as well as nonlinear volume registration.  (A) DMG Chromatic pre-processing adjusts for aberrations induces by data collection and/or geometric stitching. (Bi) 2D image before DMG. (Bii) 2D image after DMG.  (Biii) XZ projection of data before GDF. (Biv) XZ projection after GDF.  (Bv) Time per image size (red) and number of slices (black) on a particular AMI.  (C) Volume registration.
</div>





<h3>Object Detection</h3>

<p>
Given that the data has been sufficiently pre-processed, we now proceed to parse the scene, assigning voxels with semantic labels.  Parsing the scene involves three steps: (i) manually labeling a subset of training data, (ii) training and evaluating various machine vision algorithms on appropriate performance metrics, and (iii) deploying the algorithm (and parameter set) of choice at scale.  For manual labeling, we built [*manno*](http://docs.neurodata.io/manno/), a protocol that leverages ITK-SNAP by downloading image data from *NeuroData* in the appropriate format, uses ITK-SNAP to label a number of voxels, and then saves and uploads the annotations to the appropriate annotation database (with metadata; see Figure~\ref{fig:parse}(A) for an exemplar slice).  Given the training labels, [*macho*](http://docs.neurodata.io/macho/) uses ilastik to perform pixel level classification [@XXX], and then custom morphological operations to detect objects.  We choose the operating point that maximized f1-score on a held-out dataset located in a volume separated from the training volume (Figure~\ref{fig:parse}(B)).  Finally, we deployed our chosen classifier on the full dataset, using [*maxxo*](http://docs.neurodata.io/maxxo/), which utilizes the LONI Pipeline to distribute ilastik across many nodes of a HPC.  The result was ~11 million putative synapse detections (Figure~\ref{fig:parse}(C) shows approximately 2% of them).
</p>

<div class="fig-container" id="tab:images">

<img src="http://docs.neurodata.io/ocp-journal-paper/Results/Figures/parse/ndp_bock_truth_crop.png" alt="Drawing" style="width: 320px;"/>
<br>
<br>
<img src="http://docs.neurodata.io/ocp-journal-paper/Results/Figures/parse/ndp_bock_detect_crop.png" alt="Drawing" style="width: 320px;"/>
<br>
Figure 4: A schematic showing the synapses labeled with our manual annotation (mana) protocol (left) and the resulting parased scene detections using vesicle (right).  These figures, and associated associated error metrics can be reproduced using the Jupyter notebook provided with this manuscript. 
</div>



<h3>Graph Inference</h3>

@wg: claim 10 here?


<h2>Analyze</h2>


<p>
Inference has relatively task specific requirements, although statistical inference from anatomical data has some shared features.  In particular, in anatomy,  the questions typically include space, shape, and graphs.  Spatial statistics is a relatively developed branch of statistics, and we therefore can rely on previously developed methods [@Cressie15], though with novel implementations.  The statistics of shape is a much more recent sub-discipline in statistics [@Younes10], partially because it depends on the mathematics of nonlinear manifolds, which are still being actively developed [@Chikuse03].  We therefore developed novel implementations of several of these key quantities.  Finally, graph statistics is a similarly new field [@Kolaczyk09], as graphs are discrete combinatorial objects that require careful mathematical treatment [@Bollobas13].  Again, we have developed our own methods, and built our own scalable and open source implementations (http://flashx.io).  Below, we provide two case studies, one using spatial statistics, the other using graph statistics.
</p>


<h3>Images</h3>

<h3>Graphs</h3>

<h3>Matrices</h3>

<h3>Shapes</h3>

<h1>Results</h1>


<h2>Data</h2>

<h3>Image Datasets</h3>

<p>
We used the above described infrastructure to build a number of image and annotation databases to support a variety of operations. <a href="#tab:images">Table 1(A)</a> lists the  public image datasets that we are currently hosting, as well as several key properties of each.  Those properties that are required for efficient access of the data are stored in a Django
<a href="http://docs.neurodata.io/open-connectome/sphinx/datamodel.html">data model</a>, the remaining properties are stored in our custom Laboratory Information Management System called <a href="https://github.com/openconnectome/ndlims">ndlims</a>.  The image data in <a href="#tab:images">Table 1(A)</a>  comprise approximately 80 teravoxels of image data.  These datasets span spatial scales of experimental neuroscience, ranging from nanoscale with electron microscopy to milliscale with MRI, and many in between.  We also store time-series data, as exemplified by the "Freeman14" dataset, a calcium imaging time-series. By storing disparate datasets in the same format, anybody can access and analyze many different datasets with the same functionality and syntax.
</p>

<div class="fig-container" id="tab:images">
Table 1: List of all <em>public</em> current (as of 12/31/15) NeuroData Datasets.  Table 1A lists the <em>image</em> datasets and associated metadata.  Table 1B lists the volumetric *annotation* datasets, derived from a subset of the image datasets.
{% include table1b.html %}
</div>

<div id="par:private-data">
In addition to the above described public datasets, we also host a large number of private datasets, including a variety of additional modalities. This includes >300 array tomography datasets associated with the <a href="http://synapto.me">Open Synaptome Project</a>; approximately 20 different whole brain CLARITY datasets, including several multispectral datasets; several expansion microscopy datasets; and several X-ray microscopy datasets.   In total, we have XXX private datasets from XXX different collaborators spanning XXX total voxels.  Thus, NeuroData is both the largest and most diverse public neuroscience data repository in the world.
</div>


<h3>Annotation Datasets</h3>

<p>
To complement the image databases, we also created an annotation database management system called [RamonDB](http://docs.neurodata.io/nddocs/ramon.html) [@GrayRoncal15b], and a richly attributed graph database management system called [RagDB](http://openconnecto.me/graphs/).  RAMONdb stores multiple manually and automatically generated dense volumetric annotations, listed in Table~\ref{tab:anno}(B) along with several key features of each.  Collectively, we have XXX voxels manually annotated, and YYY machine annotated.
</p>


<h3>Graph Datasets</h3>


<p>
Additionally, RagDB currently holds XXX different neurographs, ranging in species from *C. elegans* to humans (see [here](http://openconnecto.me/graph-services/download/) for details).  RagDB supports a variety of different graph functions (see [here](XXX link to ndio API for graphsXXX)), including building graphs from DTI data and downloading graphs.  Much like one can download neuroimages in a variety of formats (for example, png and hdf5), we also enable graphs to be downloaded in a variety of formats (for example, MATLAB and R readable formats).
</p>


<h2>Applications</h2>

<h3>Reproducible Neurocartography</h3>


<p>
Several months ago a landmark paper was published, Kasthuri et al. Cell (2015) [@Kasthuri15].  In it, the authors heroically manually annotated a subvolume of somatosensory cortex, and provided three things:
<ol>
  <li>a paragraphical "parts list" of all the anatomical objects in a small volume,
  <li>an excel file listing all 1,700 synapses, as well as over 20 properties of each (for example, post-synaptic density centroid location and size), and
  <li>a VAST remote volume file containing the manual labels, as well as some additional information (object type, and parents).
</ol>


We converted the latter two of those objects into the RAMON annotation data specification using CAJAL, and ingested the data into RamonDB.  One immediate implication of this is that we could check for consistency between the provided excel spreadsheet and the VAST data.  Indeed, we discovered that for XXX% of the synapses in the spreadsheet, the location provided did not match any synapses in the actual VAST export.  Similarly, the VAST export contained XXX synapses that we could not find in the spreadsheet.  While this may seem to be a banality, checking the data consistency is the first step for any data analysis, and often the first "result" is idiosyncracies that can be corrected.  Of the inconsistencies, we ignored some that we could not figure out how to correct, and corrected the rest to proceed.
</p>


<p>
We next proceeded to assess each of the quantitative claims from the paragraphical parts list that amonut to counting particular kinds of objects or computing volumes.  Within VAST, there is no way to compute these quantitites, and the spreadsheet does not contain enough information to extract them all (because it only includes synapses, and doesn't include tight boundaries).  Therefore, only via using RamonDB, can one automatically and reproducibly compute all these properties.  Table [#tab:parts] is the automatically generated table from our Python script, including all quantitative claims from the manuscript.
</p>

<p>
 In addition to a parts list, the authors of Kasthuri et al. tested a hypothesis about the relationship between spatial proximity of axon-dendrite pairs and the number of synapses between those pairs.  In particular, the question is: does spatial proximity on its own explain number of connections, or is that model insufficient to characterize the data? This question has recently been hotly debated [@Markram15].  Testing this hypothesis is difficult due to the difficulty of randomly sampling graphs with a given degree distribution [@http://arxiv.org/abs/1301.6635].  We therefore developed a novel methodology that is statistically principled, to obtain a p-value of XXX.  Our code for testing this hypothesis has been incoroprated into FlashX.
</p>



<h3>Synapse Spatial Distribution</h3>




The spatial distribution of synapses is a fundamental property of neural tissue, and one that has seen some study in the past.  In particular, two recent very nice studies examined the spatial distribution of synapses in rat cortex [@http://cercor.oxfordjournals.org/content/24/6/1579.long, @http://journal.frontiersin.org/article/10.3389/fnana.2014.00085/abstract].  Using a few thousand synapses, they fail to reject the hypothesis that synapses are distributed according to a random sequential adsoption model, implying that their synapses are distributed almost randomly, with the only constraint that they cannot overlap.  Their experimental design included multiple small volumes from different animals, which is often a wise experimental design choice, to mitigate batch effects [@http://www.nature.com/nrg/journal/v11/n10/full/nrg2825.html].  In contrast, we used a single large volume from Bock et al. Nature (2011).  Using the combination of (manno,macho,maxxo) described above, we detected ~10 million synapses, a number comensurate with the expected density of synapses per volume reported in the literature.  Our complementary experimental design decision enabled us to consider the spatial distribution across a much larger volume, in case parts of the volume look approximately uniform, and other parts do not.  Indeed, Figure [#fig:syn-clust](A) depicts the spatial distribution of synapses detected in our volume.  The size of the dots corresponds to the relative density of synapses.  It is immediately apparent that synapses are <em>not</em> distributed uniformly in space; rather, there is a clear cluster pattern.  The inset in the figure shows a volume of 200 cubic microns, the size of the volumes used in the previous publications.  It would be difficult indeed to detect anisotropies from volumes of this size, given the nature of the non-uniformity. Figure [#fig:syn-clust](B) shows the 2D histograms projected onto the three canonical planes.  Even ignoring the edge effects, again, it is clear that the synapses are not distributed uniformly in space.



<div class="fig-container", id="fig:allsyn">
<div><img src="http://docs.neurodata.io/ndintro/images/rob2.png" alt="Drawing" style="height: 250px;"/></div>

<img src="http://docs.neurodata.io/ndintro/images/3D.png" alt="Drawing" style="width: 300px;"/>
<img src="http://docs.neurodata.io/ndintro/images/2Dproj.png"    alt="Drawing" style="width: 300px; "/>

Figure:

</div>


<h1>Discussion</h1>



<h1>Reproducability Guidelines</h1>


<h2>Store</h2>

<h3>Databases</h3>

<p>
You can reproduce our Read/Write throughput benchmarks using the following steps:
<ul>
  <li>Launch the AWS AMI "ami-id Here".</li>
  <li>Run the <a href="https://github.com/neurodata/ndstore/tree/microns/benchmarks/ndstore_benchmark.py">ndstore_benchmark.py</a> in the <a href="https://github.com/neurodata/ndstore/tree/microns/benchmarks">benchmarks</a> folder. The commands to run the read and write benchmarks are mentioned in the <a href="https://github.com/neurodata/ndstore/blob/microns/benchmarks/README.md">README</a> file. These commands will generate csv files for the respective benchmarks.</li>
  <li>Run the <a href=" https://github.com/neurodata/ocp-journal-paper/blob/gh-pages/Code/Plots/ThroughputPlots.r">ThroughPlots.r</a> script in the <a href="https://github.com/neurodata/ocp-journal-paper/blob/gh-pages/Code/Plots/">Plots</a> folder. This script will generate the thorughput graphs included in the paper.</li>
</ul>
You can reproduce our Tilecache benchmarks using the following steps:
<ul>
  <li>Launch the AWS AMI "ami-id here".</li>
  <li>Run the <a href="https://github.com/neurodata/ndtilecache/blob/master/benchmarks/ndtilecache_benchmark.py">ndtilecache_benchmark.py</a> script in the <a href="https://github.com/neurodata/ndtilecache/blob/master/benchmarks/">benchmarks</a>folder. The command to run the benchmarks are mentioned in the <a href="https://github.com/neurodata/ndtilecache/blob/master/benchmarks/README.md">README file.</a> file. These commands will generate the csv file for the benchmark.</li>
  <li>Run the <a href="">TilecachePlots.r</a> script in the <a href="">Plots</a> folder. This script will generate the tilecache graph in the paper.</li>
</ul>
</p>

<h2>Explore</h2>

<h2>Parse</h2>

<h2>Analyze</h2>



<h1>Appendix</h1>



<div class="fig-container", id="fig:infra">
Supplementary Figure 1: Infrastructure supporting all components of the OCP data analysis pipeline.  (A) The OCP Data Cluster. (B) Software running on the backend. (C) Software running on the frontend.
</div>


<div class="fig-container", id="tab:api">
Supplementary Table 1: Functions supported in our databases.  (A) NeuroImages. (B) NeuroAnnotations. (C) NeuroGraphs.
</div>


<div class="fig-container", id="tab:parts">
Supplementary Figure 1: Infrastructure supporting all components of the OCP data analysis pipeline.  (A) The OCP Data Cluster. (B) Software running on the backend. (C) Software running on the frontend.
</div>



	<script type="text/javascript">
        m = markdownit({
                html: true,
                linkify: true,
                typographer: true
        });
        document.getElementById('md-raw').innerHTML = m.render(
                document.getElementById('md-raw').innerHTML);
     </script>

    <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
    <!-- Include all compiled plugins (below), or include individual files as needed -->
    <script src="/Web/js/bootstrap.min.js"></script>
    </body>
</html>

{'fraction_non_alphanumeric': 0.07115400635494888, 'fraction_numerical': 0.009484820716657344, 'mean_word_length': 4.670805776757997, 'pattern_counts': {'":': 0, '<': 550, '<?xml version=': 0, '>': 554, 'https://': 28, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1584258', 'n_tokens_mistral': 22511, 'n_tokens_neox': 20653, 'n_words': 11123}