<commit_before># Software Explanation

## Video Compression

### COMPRESS

This step has the double function identifing candidate regions for the tracking and efficiently store data the using lossless compression by zeroing the background. 

The algorithm identifies dark particles on a light background or light particles on a dark background using [adaptative thresholding](http://docs.opencv.org/3.0-beta/modules/imgproc/doc/miscellaneous_transformations.html) together with a filtering particles by size. The filter parameters must be adjusted manually for a different setup but typically it is possible to use the same parameters under similar experimental contions. More information on how to setup this parameters can be found in the [GUI manual](manual.md#set-parameters).

![COMPRESS](https://cloud.githubusercontent.com/assets/8364368/8456443/5f36a380-2003-11e5-822c-ea58857c2e52.png)

The data is stored into a hdf5 container using a gzip filter. Some advantages of this format are:

* This format can significally reduce the size of a high resolution video since only the pixels corresponding to tracked particle are kept. This gain depends on codec used by the original video, and can be less important or even inexistent.

* Rapid access to specific video frames. Typically it is hard to rapidly access to a specific video frame. Most of the video readers do an approximative search when looking for a specific frame. To accurately retrive a video frame one has to read sequencially the whole video file, a process particuarly slow for large video files. The HDF5 format indexes the data in a way that makes trivial to access to specific frames.

* Metadata can be stored in the same file as the video. HDF5 format allows to store all kind of binary data into the same file. This allows to store the video metadata and analysis progress in the same file.

### VID_SUBSAMPLE


## Creating trajectories

### TRAJ_CREATE

We identify possible particles. The approach we follow is to divide the image in regions of non-zero connected pixels. For each candidate region we calculate a simple threshold and create binary mask. Then we calculate the centroid, area and bounding box of each connected-element in the binary mask. If the connected-element features are within the user defined ranges it is kept to create the trajectories. This information ins stored in the [plate_worms](#plate_worms) table.

### TRAJ_JOIN

We link the particles trajectories by using their closest neighbor in a consecutive frames. The closest neighbor must closer than `max_allowed_dist` and the change in area must be less than `area_ratio_lim` . A particle can only be joined to one particle in consecutive frames. If this conditions are not satisfied it means that there was a problem in the trajectory *e.g.* two worms colided and in the next frame the closest object is twice the area, or a worm disapear from the screen. Therefore the trajectory will be broken a new number will be assigned to any unassigned particle. In a subsequent step the program will try to join trajectories that have a small time gap between them *i.e.* the worm was lost for a couple of frames. Additionally we remove any spurious trajectory shorter than `min_track_size` .

Below there is an example of how the trajectories look.

![trajectories](https://cloud.githubusercontent.com/assets/8364368/26301795/25eb72ac-3eda-11e7-8a52-99dd6c49bc07.gif)

### SKE_INIT
Firstly, the center of mass and the threshold for each of the trajectories is smoothed.  This improves the estimation of the worm threshold, fills gaps where the trajectory might have been lost, and helps to produce videos where the ROI displaces gradually following individual worms.

This step create [trajectories_data](#trajectories_data) table, that contains

### BLOB_FEATS

We extract a series of features for each individual binary mask and store them in [blob_features](#blob_features).

## Extracting worm skeletons

### SKE CREATE
![skeletons](https://cloud.githubusercontent.com/assets/8364368/26309647/a6b4402e-3ef5-11e7-96cd-4a037ee42868.gif)

Secondly, a ROI is thresholded, a contour is calculated, and the worm is skeletonized. The key part of this step is the skeletonization code based on [segWorm](https://github.com/openworm/SegWorm). Since one has to deal with multiworm at a time speed becomes an important issue, therefore the code was optimized using Cython and C. The skeletons and contours are normalized to have the same number of points in order to store them in a simple table. The output is store in a file with the extension [basename_skeletons.hdf5](#basename_skeletonshdf5).
![skeletons](https://cloud.githubusercontent.com/assets/8364368/26309647/a6b4402e-3ef5-11e7-96cd-4a037ee42868.gif)

### SKE_FILT
Filter "bad worms", meaning any particle indentified and analyzed for the tracker that it is not a worm, or any trajectory that corresponds to two or more worms in contact.


### SKE_ORIENT
In a second part of the code the head and tail are identified by movement. Althought it is hard to determine the head and the tail from the contour, it is possible to assign "blocks" with the same orientation for skeletons in contingous frames, since the head in one skeleton will not suddenly jump to the other side of the worm within a few frames. We can then assign the relative standard deviation (SD) of the angular movement for the first and last part of the segment. If the blocks are large enough the section with the higher SD would be the head.
 
### STAGE_ALIGMENT
### CONTOUR_ORIENT
'ventral_side':['','clockwise','anticlockwise', 'unknown'],
<commit_msg>Update software_explanation.md<commit_after># Software Explanation

## Video Compression

### COMPRESS

This step has the double function identifing candidate regions for the tracking and efficiently store data the using lossless compression by zeroing the background. 

The algorithm identifies dark particles on a light background or light particles on a dark background using [adaptative thresholding](http://docs.opencv.org/3.0-beta/modules/imgproc/doc/miscellaneous_transformations.html) together with a filtering particles by size. The filter parameters must be adjusted manually for a different setup but typically it is possible to use the same parameters under similar experimental contions. More information on how to setup this parameters can be found in the [GUI manual](manual.md#set-parameters).

![COMPRESS](https://cloud.githubusercontent.com/assets/8364368/8456443/5f36a380-2003-11e5-822c-ea58857c2e52.png)

The data is stored into a hdf5 container using a gzip filter. Some advantages of this format are:

* This format can significally reduce the size of a high resolution video since only the pixels corresponding to tracked particle are kept. This gain depends on codec used by the original video, and can be less important or even inexistent.

* Rapid access to specific video frames. Typically it is hard to rapidly access to a specific video frame. Most of the video readers do an approximative search when looking for a specific frame. To accurately retrive a video frame one has to read sequencially the whole video file, a process particuarly slow for large video files. The HDF5 format indexes the data in a way that makes trivial to access to specific frames.

* Metadata can be stored in the same file as the video. HDF5 format allows to store all kind of binary data into the same file. This allows to store the video metadata and analysis progress in the same file.

### VID_SUBSAMPLE


## Creating trajectories

### TRAJ_CREATE

We identify possible particles. The approach we follow is to divide the image in regions of non-zero connected pixels. For each candidate region we calculate a simple threshold and create binary mask. Then we calculate the centroid, area and bounding box of each connected-element in the binary mask. If the connected-element features are within the user defined ranges it is kept to create the trajectories. This information ins stored in the [plate_worms](#plate_worms) table.

### TRAJ_JOIN

We link the particles trajectories by using their closest neighbor in a consecutive frames. The closest neighbor must closer than `max_allowed_dist` and the change in area must be less than `area_ratio_lim` . A particle can only be joined to one particle in consecutive frames. If this conditions are not satisfied it means that there was a problem in the trajectory *e.g.* two worms colided and in the next frame the closest object is twice the area, or a worm disapear from the screen. Therefore the trajectory will be broken a new number will be assigned to any unassigned particle. In a subsequent step the program will try to join trajectories that have a small time gap between them *i.e.* the worm was lost for a couple of frames. Additionally we remove any spurious trajectory shorter than `min_track_size` .

Below there is an example of how the trajectories look.

![trajectories](https://cloud.githubusercontent.com/assets/8364368/26301795/25eb72ac-3eda-11e7-8a52-99dd6c49bc07.gif)

### SKE_INIT
Firstly, the center of mass and the threshold for each of the trajectories is smoothed.  This improves the estimation of the worm threshold, fills gaps where the trajectory might have been lost, and helps to produce videos where the ROI displaces gradually following individual worms.

This step create [trajectories_data](#trajectories_data) table, that contains

### BLOB_FEATS

We extract a series of features for each individual binary mask and store them in [blob_features](#blob_features).

## Extracting worm skeletons

### SKE CREATE
![skeletons](https://cloud.githubusercontent.com/assets/8364368/26309647/a6b4402e-3ef5-11e7-96cd-4a037ee42868.gif)

Secondly, a ROI is thresholded, a contour is calculated, and the worm is skeletonized. The key part of this step is the skeletonization code based on [segWorm](https://github.com/openworm/SegWorm). Since one has to deal with multiworm at a time speed becomes an important issue, therefore the code was optimized using Cython and C. The skeletons and contours are normalized to have the same number of points in order to store them in a simple table. The output is store in a file with the extension [basename_skeletons.hdf5](#basename_skeletonshdf5).

### SKE_FILT
Filter "bad worms", meaning any particle indentified and analyzed for the tracker that it is not a worm, or any trajectory that corresponds to two or more worms in contact.


### SKE_ORIENT
In a second part of the code the head and tail are identified by movement. Althought it is hard to determine the head and the tail from the contour, it is possible to assign "blocks" with the same orientation for skeletons in contingous frames, since the head in one skeleton will not suddenly jump to the other side of the worm within a few frames. We can then assign the relative standard deviation (SD) of the angular movement for the first and last part of the segment. If the blocks are large enough the section with the higher SD would be the head.
 
### STAGE_ALIGMENT
### CONTOUR_ORIENT
'ventral_side':['','clockwise','anticlockwise', 'unknown'],

{'fraction_non_alphanumeric': 0.04784088872961835, 'fraction_numerical': 0.02365167532700233, 'mean_word_length': 4.823161189358372, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4199830', 'n_tokens_mistral': 2991, 'n_tokens_neox': 2722, 'n_words': 1584}