<jupyter_start><jupyter_text>Getting familiar with the electrophysiology (Ephys) pipelineIn this notebook, we will take a tour of the Moser group electrophysiolgoy (ephys) pipeline, taking a close look at how each table is defined and relate to each other to represent various experimental information. But before we begin, let's import a couple essential packages, in particular importing DataJoint (`datajoint`) package, and establish connection to the Moser lab database where the entire pipeline resides.<jupyter_code>import datajoint as dj
dj.config['display.limit'] = 10<jupyter_output><empty_output><jupyter_text>If you have followed all the setups for Python environment, DataJoint and the Ephys pipeline as described in [Zero to DataJoint Wiki page](https://github.com/kavli-ntnu/dj-elphys/wiki/Zero-to-datajoint), you should be able to just run the next line, importing pipeline definitions as found in the `ephys` package.<jupyter_code>from ephys import reference, acquisition, tracking, behavior, ephys
animal = dj.create_virtual_module('mlims', dj.config['custom'].get('mlims.database', 'prod_mlims_data'))<jupyter_output><empty_output><jupyter_text>If you experienced any issue in importing the pipeline, please refer to the [Wiki page and guide](https://github.com/kavli-ntnu/dj-elphys/wiki/Zero-to-datajoint) again or seek help from administrators. The entire pipeline is quite elaborate and contains many parts, grouping related tables together to facilitate navigation and understanding. Rather than looking at them all at once, we will now take a look at each of these groups of tables or **schemas**, one at a time.  Animal information Just like any experiment cannot occur without the animal, the entire pipeline also beging with capturing information about the animal. Naturally, information pertaining to the animal can be found under the `animal` schema.<jupyter_code>dj.Diagram(animal)<jupyter_output><empty_output><jupyter_text>Among other things, the `animal.Animal` table contains listing of all animals in the lab, and as we will see, serve as the starting point for the entire ephys pipeline.<jupyter_code>animals.Animal()<jupyter_output><empty_output><jupyter_text>Each animal is uniquely captured by a combination of `animal_id` and `datasource_id`. `datasource_id=0` indicates that these are animals found in MLIMS colony management system.  Querying tables In DataJoint, you can probe into or **query** tables to get a specific subset of data that you may be interested in. The most common type of query is **restriction**, where you'll subselect entries based on specific criteria.  Question: Find only rats<jupyter_code>mlims.Animal & 'animal_species = "rat"'<jupyter_output><empty_output><jupyter_text>Question: Find only animals born after 2017<jupyter_code>animal.Animal & 'animal_dob >= "2017-01-01"'<jupyter_output><empty_output><jupyter_text>Getting data out of the table Once you form your desired query result, you can **fetch** back the data from the tables by calling `fetch` on the query result.<jupyter_code># get all information about animals born on or after 2017
data = (animal.Animal & 'animal_dob >= "2017-01-01"').fetch()
data.shape<jupyter_output><empty_output><jupyter_text>Getting list of unique animals You can just get the identifying information of the table by grabbing (or **fetching**) the `KEY` of the table, and then using this to **restrict** the entries.<jupyter_code>keys = (animal.Animal & 'animal_dob >= "2017-01-01"').fetch('KEY')
keys[:5]  # first five keys<jupyter_output><empty_output><jupyter_text>Get specific animal back<jupyter_code># get the first animal back
animal.Animal & keys[0]<jupyter_output><empty_output><jupyter_text>Electrophysiology Session In order to understand how the entire `ephys` pipeline is organized, it is essential to first understand what we mean by "a session".The following figure illustrates the structure of an electrophysiology experiment session.  A single experimental **session** consists of one or more of the following:* tracking data stream (recording from animal tracker) -- referred to as **tracking** data* electrophsyiology data stream (recording from a single probe) -- referred to as **recording** data Within a single **session**, you can get any number of the above, with possibility of having multiple probe recordings occuring simultaneously (e.g. when recording from two or more probes at the same time).While the experimenter is free to split any number of recordings/trackings into a desired number of sessions, an **experimental session** is typically considered to be a single coherent collection of recordings, and is also typically the basic unit for the downstream analysis. Additionally, a single **session** may be assocciated with one or more **tasks**, usually labeling the particular behavioral task or environment that the animal is exposed to. To represent the above complexity, we make use of multiple related tables, starting with `Session`, representing a single experimental session in the pipeline.  An experimental `Session` can be uniquely identified by knowing the **animal** and **the exact date time** of the experiment session. This is reflected in the definition of the `Session` table.<jupyter_code>acquisition.Session()<jupyter_output><empty_output><jupyter_text>Looking at the *definition* of the table, we see that `Session` table **refers to `Animal` table** to indicate that each `Sessions` depends on an `Animal`. In other words, you cannot possibly have an experiment session without a corresponding animal.<jupyter_code>acquisition.Session.describe();<jupyter_output>-> animals.Animal
session_time         : datetime                     # start time of this session
---
session_note=null    : longblob<jupyter_text>This relationship is captured by the following table **diagram**<jupyter_code>dj.ERD(animal.Animal) + acquisition.Session<jupyter_output><empty_output><jupyter_text>Recording and Tracking As was mentioned above, we expect to find one or more electrode probe recordings and/or tracking data stream. These information are represented by `Recording` and `Tracking` tables, respectively.<jupyter_code>dj.Diagram(animal.Animal) + acquisition.Session + acquisition.Recording + tracking.Tracking + acquisition.ProbeInsertion<jupyter_output><empty_output><jupyter_text>The extended diagram now captures the relationship between `Session` and the two new tables `Recording` and `Tracking`. However, notice that `Recording` also *depends* on another table -- `ProbeInsertion`.  Understanding recording - working with probes<jupyter_code>acquisition.Recording.describe();<jupyter_output>-> acquisition.Session
-> acquisition.ProbeInsertion
recording_time       : datetime                     # start time of this recording
---
-> reference.RecordingSystem
-> reference.ElectrodeConfig
recording_order      : smallint                     # the ordering of this recording in this session
recording_duration   : float                        # (s) duration of this recording
recording_name=""    : varchar(40)                  # name of this recording (e.g. 27032019laserSess1)<jupyter_text>Looking at the definition of `Recording` table, we can see that it depends on two parent tables -- `acquisition.Session` and `acquisition.ProbeInsertion`. Dependency on `acquisition.Session` ensures that each recording must be associated with exactly one `Session` entry. What is the dependency on `ProbeInsertion` about? If you guessed that this captures exactly which electrode probe that the recording was performed on, then you guessed right! `ProbeInsertion` table actually represents the final table in a chain of tables capturing precise information about the probe that was inserted into the animal.<jupyter_code>dj.Diagram(acquisition.ProbeInsertion) - 1 + reference.ProbeModel + reference.ProbeType
acquisition.ProbeInsertion.describe();<jupyter_output>-> animals.Animal
insertion_time       : datetime                     # When this probe was inserted
---
-> reference.Probe
insertion_note=""    : varchar(1000)                # some notes<jupyter_text>Conceptually, an entry in `ProbeInsertion` table represents a particular event of probe insertion into the animal, and this is uniquely identified by knowing on **which animal** and **when** the insertion was performed. You can also see that `ProbeInsertion` *refers* to `Probe` table, thereby indicating what `Probe` was actually inserted.<jupyter_code>reference.Probe()
reference.Probe.describe();<jupyter_output># represent a physical probe
probe                : varchar(32)                  # unique identifier of this probe (e.g. serial number)
---
-> reference.ProbeModel<jupyter_text>An entry in `Probe` table refers to a specific *physical instance* of a probe, with some string of characters used to uniquely identify the probe (e.g. name or serial number of the probe). Again, `Probe` *refers* to `ProbeModel` table, that now captures the **model** of the probe, such as neuropixel probe version 1.0 or 4 probe tetrode array.<jupyter_code>reference.ProbeModel()
reference.ProbeModel.describe();<jupyter_output># represent a probe model
probe_model          : varchar(32)                  # nick name, or other user-friendly model name of this probe
---
-> reference.ProbeType<jupyter_text>Finally, each `ProbeModel` *refers* to the type of probe, which is a more general grouping of probe classifiation, such as neuropixels or tetrode array.<jupyter_code>dj.Diagram(acquisition.ProbeInsertion) - 1 + reference.ProbeModel + reference.ProbeType<jupyter_output><empty_output><jupyter_text>Put together, `ProbeInsertion` allows experimenters to capture the information about the probe that was inserted into an animal.  Recordings Now we understand what `ProbeInsertion` table captures, let's get back to `Recording` table.<jupyter_code>dj.Diagram(animal.Animal) + acquisition.Session + acquisition.Recording + tracking.Tracking + acquisition.ProbeInsertion
acquisition.Recording.describe();<jupyter_output>-> acquisition.Session
-> acquisition.ProbeInsertion
recording_time       : datetime                     # start time of this recording
---
-> reference.RecordingSystem
-> reference.ElectrodeConfig
recording_order      : smallint                     # the ordering of this recording in this session
recording_duration   : float                        # (s) duration of this recording
recording_name=""    : varchar(40)                  # name of this recording (e.g. 27032019laserSess1)<jupyter_text>Notice that a entry in `Recording` is uniquely identified by a unique combination of a `Session`, `ProbeInsertion`, **and recording time**! It is the presence of this last primary key attribute that enables more than one recording to be present for each combination of `Sessions` and `ProbeInsertion`. In other words, within a `Session`, you can have more than one recordings from the same probe, as long as they are separated in time (`recording_time`). Now you may have noticed that `Recording` also *refers* to a number of other tables. This is graphically depicted by the following diagram.<jupyter_code>dj.Diagram(acquisition.Recording) + animal.Animal + acquisition.Session + acquisition.ProbeInsertion + reference.RecordingSystem + reference.ElectrodeConfig<jupyter_output><empty_output><jupyter_text>`RecordingSystem`, as the name indicates, keeps track of different recroding systems.<jupyter_code>reference.RecordingSystem()<jupyter_output><empty_output><jupyter_text>`ElectrodeConfig` is a bit more complex, referring to the exact configurations of groupings of channels found in a probe that the recording occured for. We will revisit this table and other related tables later in this notebook.  Tracking data In contrast to the `Recording` table, `Tracking` table contains information about a particular tracking data stream found inside a session.<jupyter_code>tracking.Tracking().describe();<jupyter_output>-> acquisition.Session
tracking_time        : datetime                     # start time of this tracking recording
---
-> tracking.TrackingType
tracking_name        : varchar(40)                  # user-assign name of this tracking (e.g. 27032019laserSess1)
tracking_sample_rate : float                        # (Hz)
tracking_timestamps  : longblob                     # (s) timestamps of the frame in the tracking system clock, w.r.t the start of this tracking<jupyter_text>Again, `Tracking` *refers* to `Session` but also has it's own primary key attribute `tracking_time`, allowing for more than one tracking to be present per session, as long as they are separated in time.   Where is the data? While `Recording` and `Tracking` tables both represented data recordings, you may have noticed that we didn't see the actual data points present in the table. This is because these tables are used to describe information **about recording and tracking**, and the actual data are loaded and handled by downstream tables as we will see shortly.  Clustering While a stream of electrical activities in itself can be interesting (i.e. refer to `LFP` below), we are often interested in working with individual spikes, and thus want to perform spike detection/clustering on the raw signal to extract these spikes. The ephys pipeline is built so that it can express very complex configurations of clustering. Namely, **arbitrary time segments from one or more recordings across one or more sessions** can be combined to be the target of clustering. Let's now take a look at how that is represented in the tables.  Specification of clustering time windows To allow for flexible combination of one or more sessions, `ClusterSessionGrouping` table is used.<jupyter_code>dj.Diagram(acquisition.ClusterSessionGroup) + acquisition.Session + acquisition.ClusterSessionGroup.GroupMember
acquisition.ClusterSessionGroup.describe();<jupyter_output>group_name           : varchar(36)                  
---
group_desc=""        : varchar(128)<jupyter_text>`ClusterSessionGroup` table simply defines a group of one or more sessions, and the `ClusterSessionGroup.GroupMember` ties the member sessions to the group it belongs to. Taking a look at an example group.<jupyter_code>keys = acquisition.ClusterSessionGroup().fetch('KEY')
acquisition.ClusterSessionGroup() & keys[0]<jupyter_output><empty_output><jupyter_text>and members for this session group<jupyter_code>acquisition.ClusterSessionGroup.GroupMember & keys[3]<jupyter_output><empty_output><jupyter_text>Specifying time windows<jupyter_code>dj.ERD(acquisition.ClusterTimeWindows.TimeWindow()) + acquisition.ClusterTimeWindows + acquisition.ClusterSessionGroup +  acquisition.ClusterSessionGroup.GroupMember + acquisition.Session<jupyter_output><empty_output><jupyter_text>Once a group of one or more sessions is defined, `ClusterTimeWindows` and `ClusterTimeWindows.TimeWindow` are  used to subselect one or more time windows within the **concatenated session data**.<jupyter_code>acquisition.ClusterTimeWindows.TimeWindow.describe()<jupyter_output>-> acquisition.ClusterTimeWindows
window_start         : decimal(6,2)                 # (s) onset of this window, with respect to session onset
---
window_stop          : float                        # (s) offset of this window, with respect to session onset<jupyter_text>Electrode configurations  More about probes - location and adjustments In fact there are more information about the electrode probe that can be kept track of.<jupyter_code>acquisition.Pr
reference.Pro
acquisition.ProbeInsertion().describe()<jupyter_output>-> animals.Animal
insertion_time       : datetime                     # When this probe was inserted
---
-> reference.Probe
insertion_note=""    : varchar(1000)                # some notes<jupyter_text>In one ***Session***, there can be:+ multiple trackings (in time)+ multiple recordings (in time)    + one recording per probe+ multiple probes (different insertion sites)+ multiple tasks  Probe and Probe Insertion  Following the above experiment organization, the diagram below shows the design of the session related part of the ephys pipeline ![session overview](../images/session_overview_erd.png) An experiment ***Session*** can be uniquely identified by knowing the **animal** and **the exact date time** of the experiment session. This is reflected in the definition of the `Session` table.<jupyter_code>acquisition.Session.describe();<jupyter_output>-> mlims.Animal
session_time         : datetime                     # start time of this session
---
session_note=null    : longblob<jupyter_text>Additionally, each session can contain `session_note` where you can place arbitrary notes to describe the session.<jupyter_code>acquisition.Session()<jupyter_output><empty_output><jupyter_text>Probes and probe insertions<jupyter_code>dj.ERD(reference.ProbeModel) + 2
reference.ProbeModel()
reference.Probe().describe()
acquisition.ProbeInsertion.describe()
acquisition.ProbeInsertion()
dj.ERD(acquisition.ProbeInsertion) - 1 + acquisition.ProbeInsertion.InsertionLocation
acquisition.ProbeInsertion.InsertionLocation()<jupyter_output><empty_output><jupyter_text>In each ***Session***, a ***Recording*** corresponds to a single continuous recording from a single probe, and thus is uniquely identifed by the following factors:* The **session** that the recording belongs to* A particular instance of an **inserted electrode probe** that the recording was performed on* Exact start date and time of the recordingGiven this, we expect a recording to **depend on** both a `Session` and a `ProbeInsertion`, but also contain an additional primary key attribute/column for the recording start time. We can indeed see this in the definition of the Recording table:<jupyter_code>acquisition.Recording.describe();<jupyter_output>-> acquisition.Session
-> acquisition.ProbeInsertion
recording_time       : datetime                     # start time of this recording
---
-> reference.RecordingSystem
-> reference.ElectrodeConfig
recording_order      : smallint                     # the ordering of this recording in this session
recording_duration   : float                        # (s) duration of this recording
recording_name=""    : varchar(40)                  # name of this recording (e.g. 27032019laserSess1)<jupyter_text>Each ***Recording*** also contains additional information:+ recording system (e.g. Axona, Neralynx, Neuropixels, Neurologger)+ electrode configuration (e.g. subset of neuropixels sites, tetrode number)and depending on the recording system, information needed to perform synchornization to a common time clock spanning the whole session can be found in the `acquisition.Recording.RecordingSync` part table. ![recording](../images/recording_detail_erd.png)<jupyter_code>dj.ERD(acquisition.Session()) -1 + 1
acquisition.Session.Directories()
acquisition.Recording()<jupyter_output><empty_output><jupyter_text>Organization of Clustering Data For each ***Recording***, ***UnitSpikeTimes*** can be extracted, per ***Unit*** with spike times from this recording Clusterings can be performed on:+ Multiple concatenated sessions (although in most cases, just on one session)+ Different time window(s) over the selected session(s)+ Yielding units for this ***clustering*** ![](../images/clustering_erd.png) ***ClusteringSetup*** is for users to specify:+ the ***ClusterSessionGroup*** and ***ClusterTimeWindows***+ the probe this clustering is performed on - ***ProbeInsertion***+ the parameters to perform the clustering - ***ClusterParam***+ the directory to output this clustering results + the electrode configuration informationNote: all of this is taken care of in the ingestion routine (more on that later)***Clustering*** - a processing step to either trigger the clustering or perform ingestion on performed clustering***CuratedClustering*** - insert a new curation:+ the curator+ the time of curation+ the directory of the new curation results<jupyter_code>ephys.ClusteringSetup.describe();
ephys.CuratedClustering()<jupyter_output><empty_output><jupyter_text>A more detailed look to the ***Clustering*** pipeline ![](../images/clustering_detail_erd.png)  With flexibility comes complexity   Tracking  Toward the results - Spikes and Tracking ![](../images/spike_track_erd.png)<jupyter_code>ephys.UnitSpikeTimes()<jupyter_output><empty_output><jupyter_text>***UnitSpikeTimes*** and ***Tracking*** data can be matched together - resulting in the ***SpikesTracking***:> The spike times and tracking data (e.g. position, speed, head angle, etc.) associated with each spike, per unitAnd such ***SpikesTracking*** can then be further narrowed down to the task, in ***TaskSpikesTracking***<jupyter_code>ephys.SpikesTracking()
analysis.TaskSpikesTracking()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07882318952234206, 'fraction_numerical': 0.0053447611710323575, 'mean_word_length': 5.079918032786885, 'pattern_counts': {'":': 0, '<': 133, '<?xml version=': 0, '>': 160, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '410244', 'license': 'MIT', 'n_tokens_mistral': 5621, 'n_tokens_neox': 5229, 'n_words': 2188, 'path': 'notebooks/The_Ephys_Pipeline.ipynb', 'repo_name': 'eywalker/dj-docs', 'size': '20768'}