<commit_before># Brain-Observatory-Toolbox
A MATLAB toolbox for accessing and working with the Allen Brain Observatory \[1\] neural recording datasets

The Allen Brain Observatory resource comprises extensive public datasets of neural activity in the mouse visual cortex during visual stimulus presentation. These datasets are based on cellular-scale experimental neural recordings conducted at the Allen Institute for Brain Science: 
* the Visual Coding – 2P dataset \[2\], based on two-photon calcium imaging
* the Visual Coding – Neuropixels dataset \[3\], based on large-scale electrophysiological probes

The Brain Observatory Toolbox (BOT) provides a uniform interface allowing users to conveniently access and work with these two experimental neural datasets.

The BOT interface provides:
* Retrieval of public Visual Coding dataset directories describing contents and metadata
* Tabular data operations to inspect and select dataset items
* Object interfaces to inspect individual dataset items and retrieve underlying experimental data
* Useful methods to assist in analysing neural responses and behavioural data 
* Caching of retrieved directories and data, for best possible initial and repeat performance

![alt text](https://github.com/emeyers/Brain-Observatory-Toolbox/blob/backend/BOTdataSchematic.png?raw=true)

## Installation

1. Download the latest release as a `.zip` file.
1. Unzip using preferred system tool. 
1. Add the base directory of unzipped contents to the MATLAB path.  
<sup>Note it is unnecessary to add subdirectories to the MATLAB path; all contents of the `+bot` package are made available by adding the base directory</sup>

#### Optional requirements

Some aspects of the toolbox can make use of accelerated `.mex` files, which require a `mex` compiler to be configured in MATLAB. These files will be compiled automatically when needed. If a `mex` compiler is not configured, then MATLAB native versions will be used.

## Getting started

### Toolbox interface

The toolbox classes are contained within the `bot` namespace. Once the toolbox is added to the MATLAB path, you can access the data via the main factory functions:

`bot.fetchSessions()` — Retrieve a table of available EPhys or OPhys experimental sessions.

`bot.fetchExperiments()` — Retrieve a table of available OPhys experiments.

`bot.fetchProbes()` — Retrieve a table of individual probes used in the EPhys experiments.

`bot.fetchUnits()` — Retrieve a table of individual units (putative neurons) recorded in the EPhys experiments.

Analogous factory functions are used to obtain individual experiments or units, which can be manipulated to obtain experimental data:

`bot.session(session_id)` — Retrieve a single experimental session.

`bot.experiment(experiment_id)` — Retrieve a single experiment.

`bot.probe(probe_id)` — Retrieve a single experimental probe.

`bot.unit(unit_id)` — Retrieve a single experimental unit.

The toolbox manages downloading and caching of experimental data, in a "lazy access" fashion — only the minimal required data is downloaded.

### Example usage
See the [Ophys Quick Start Example](https://viewer.mathworks.com/?viewer=live_code&url=https%3A%2F%2Fwww.mathworks.com%2Fmatlabcentral%2Fmlc-downloads%2Fdownloads%2F6aee4c33-d05e-4715-82ab-748f121adcad%2Ff8904f7a-8904-2deb-4404-99caae194d40%2Ffiles%2FOphysQuickStart.mlx&embed=web) for an illustration of how these toolbox classes enable retrieving, filtering, and acccessing the Visual Coding – 2P dataset [2] from the Allen Brain Observatory. See also the Ophys demo for a more detailed dive into analysis of the 2P dataset.

See the Ephys Quick Start Example and the Ephys Demo for examples of how to access, filter and analyse the electrophysiology data from the Visual Coding – Neuropixels dataset [3].

----
#### References

[1] Copyright 2016 Allen Institute for Brain Science. Allen Brain Observatory. Available from: [portal.brain-map.org/explore/circuits](http://portal.brain-map.org/explore/circuits).

[2] Copyright 2016 Allen Institute for Brain Science. Visual Coding 2P Dataset. Available from: [portal.brain-map.org/explore/circuits/visual-coding-2p](http://portal.brain-map.org/explore/circuits/visual-coding-2p).

[3] Copyright 2019 Allen Institute for Brain Science. Visual Coding Neuropixels Dataset. Available from: [portal.brain-map.org/explore/circuits/visual-coding-neuropixels](https://portal.brain-map.org/explore/circuits/visual-coding-neuropixels).

#### Acknowledgements

Core engineering work was supported by the Foundation of Psychocultural Research and Sherman Fairchild Award at Hampshire College. <commit_msg> README release candidate for v0.9<commit_after># Brain-Observatory-Toolbox
A MATLAB toolbox for accessing and working with the public neural recording datasets obtained with the Allen Brain Observatory platform \[1\]. 

*NOTE*: Releases for the current version (0.9) represent a working prototype intended for evaluation, including functioning demos & tutorials. Feedback is encouraged and welcomed (one or more channels to be established soon). 

## About the Allen Brain Observatory datasets
[Data releases|https://portal.brain-map.org/latest-data-release] from the Allen Brain Observatory include two datasets of neural activity recorded from the mouse visual cortex during visual stimulus presentation, including:  

| Dataset | Recording Type | Nickname |
| --- | --- | --- |
| Visual Coding 2P \[2\] | Two-photon calcium imaging | "ophys" (optical physiology) |
| Visual Coding Neuropixels \[3\] | Large-scale neural probe recordings | "ephys" (electrophysiology) |

Technical white papers for each dataset provide information describing the experiments, recordings, and computational pipelines. 

## About the Brain Observatory Toolbox (BOT) 
 
The Brain Observatory Toolbox (BOT) provides a uniform interface allowing users to conveniently access and work with these Visual Coding neural datasets. 

The BOT interface provides a tabular representation of available dataset items and an object representation of specific dataset items: 
![alt text](https://github.com/emeyers/Brain-Observatory-Toolbox/blob/backend/BOTDataSchematic.png?raw=true)

*Key points:*
* Supported dataset items include experimental sessions (both 2P and Neuroxels datasets) as well as probes, channels, and units (for the Neuropixels dataset). 
* Tabular indexing or unique item identifiers can be used to select specific item(s) of interest from available items tables for item object creation. 
* Item object properties allow inspection and computation of direct, derived, and file-linked values associated to an item. 
* Item object methods allow computations of values determined with additional user-specified arguments. 
* The BOT provides local caching of retrieved item information, object representations, and file contents, to provide the fastest possible initial and repeat performance.

To preview the BOT in use, you can view the [Ephys Demo](https://viewer.mathworks.com/?viewer=live_code&url=https%3A%2F%2Fwww.mathworks.com%2Fmatlabcentral%2Fmlc-downloads%2Fdownloads%2F6aee4c33-d05e-4715-82ab-748f121adcad%2Fd61de411-5e28-4eba-8c36-c8b1df0435fc%2Ffiles%2FEphysDemo.mlx&embed=web) for the Visual Coding Neuropixels dataset and the [Ophys Demo](https://viewer.mathworks.com/?viewer=live_code&url=https%3A%2F%2Fwww.mathworks.com%2Fmatlabcentral%2Fmlc-downloads%2Fdownloads%2F6aee4c33-d05e-4715-82ab-748f121adcad%2Fd61de411-5e28-4eba-8c36-c8b1df0435fc%2Ffiles%2FOphysDemo.mlx&embed=web) for the Visual Coding 2P dataset. 

## Installation Instructions

1. Download the latest release, either by
   1. Downloading the `.zip` file from GitHub and unzipping via preferred system tool
   1. Cloning the GitHub `master` branch<sup>1</sup>
1. Add the root directory of the unzipped contents to the MATLAB path, either by:
   1. Navigating to root directory in the _Current Folder_ browser and selecting _Add to Path_<sup>2</sup> in the right-click context menu
   1. Selecting root directory in the Set Path dialog from the _Environment_ section of the _Home_ tab
   1. Calling `addpath(<root directory>)' in the _Command Window_
<sup>1. This branch will be renamed soon to remove abhorrent associations 
<sup>2. Note it is unnecessary to add subdirectories to the MATLAB path; all contents of the `+bot` package are made available by adding the base directory</sup>

#### Required products
* MATLAB (R2021a)
* Image Processing Toolbox (if running the Visual Coding 2P demonstration `OphysDemo.mlx`)

## Getting started
Four MATLAB live scripts are provided to help get started: 

* `EphysDemo.mlx` and `OphysDemo.mlx` demonstrate simple representative neuroscientific analysis using the BOT with the Visual Coding Neuropixels and Visual Coding 2P datasets, respectively
* `EphysTutorial.mlx` and `OphysTutorial.mlx` provide more step-by-step instruction and "under the hood" technical detail about using the BOT and the datasets 

Or to get a fast first look yourself, enter the following commands in MATLAB: 
```
sessions = bot.fetchSessions('ephys'); 
head(sessions)
```
----
#### References

[1] Copyright 2016 Allen Institute for Brain Science. Allen Brain Observatory. Available from: [portal.brain-map.org/explore/circuits](http://portal.brain-map.org/explore/circuits).

[2] Copyright 2016 Allen Institute for Brain Science. Visual Coding 2P Dataset. Available from: [portal.brain-map.org/explore/circuits/visual-coding-2p](http://portal.brain-map.org/explore/circuits/visual-coding-2p).

[3] Copyright 2019 Allen Institute for Brain Science. Visual Coding Neuropixels Dataset. Available from: [portal.brain-map.org/explore/circuits/visual-coding-neuropixels](https://portal.brain-map.org/explore/circuits/visual-coding-neuropixels).

#### Acknowledgements

Initial engineering work was supported by the Foundation of Psychocultural Research and Sherman Fairchild Award at Hampshire College. 
{'fraction_non_alphanumeric': 0.0733353480406971, 'fraction_numerical': 0.021658104160370707, 'mean_word_length': 5.26372239747634, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 8, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '845498', 'n_tokens_mistral': 2911, 'n_tokens_neox': 2654, 'n_words': 1134}