<commit_before># TrackR-CNN
Code for the TrackR-CNN baseline for the Multi Object Tracking and Segmentation (MOTS) task.

## Project website (including annotations)
https://www.vision.rwth-aachen.de/page/mots

## Paper
https://www.vision.rwth-aachen.de/media/papers/mots-multi-object-tracking-and-segmentation/MOTS.pdf

## mots_tools for evaluating results
https://github.com/VisualComputingInstitute/mots_tools

## Running this code
You'll need to install the following packages (possibly more):
```
tensorflow-gpu pycocotools numpy scipy sklearn pypng opencv-python munkres
```
Also create the following directories for logs, model files etc.:
```
mkdir forwarded models summaries logs
```
TODO

## References
Parts of this code are based on Tensorpack (https://github.com/tensorpack/tensorpack/tree/master/examples/FasterRCNN) and RETURNN (https://github.com/rwth-i6/returnn/blob/master/Log.py).

## Citation
If you use this code, please cite:
```
@inproceedings{Voigtlaender19CVPR_MOTS,
 author = {<NAME> and <NAME> and Aljo\u{s}<NAME>\u{s}ep and <NAME> and <NAME> and <NAME> and <NAME>},<commit_msg>Improved Readme<commit_after># TrackR-CNN
Code for the TrackR-CNN baseline for the Multi Object Tracking and Segmentation (MOTS) task.

## Project website (including annotations)
https://www.vision.rwth-aachen.de/page/mots

## Paper
https://www.vision.rwth-aachen.de/media/papers/mots-multi-object-tracking-and-segmentation/MOTS.pdf

## mots_tools for evaluating results
https://github.com/VisualComputingInstitute/mots_tools

## Running this code
### Setup
You'll need to install the following packages (possibly more):
```
tensorflow-gpu pycocotools numpy scipy sklearn pypng opencv-python munkres
```
Also create the following directories for logs, model files etc.:
```
mkdir forwarded models summaries logs
```

### Training
In order to train a model, run `main.py` with the corresponding configuration file. For the baseline model with two separable 3D convolutions and data association with learned embeddings, use
```
python main.py configs/conv3d_sep2
```
You'll need to adjust the `KITTI_segtrack_data_dir` and `load_init` flags to point to the KITTI MOTS data directory and the path to the pretrained model, respectively. Logs, checkpoints and summaries are stored in the `logs/`, `models/` and `summaries/` subdirectories.

### Forwarding and tracking
To obtain the model's predictions (we call this "forwarding") run:
```
python main.py configs/conv3d_sep2 "{\"task\":\"forward_tracking\",\"dataset\":\"KITTI_segtrack_feed\",\"load_epoch_no\":5,\"batch_size\":5,\"export_detections\":true,\"do_tracking\":false,\"video_tags_to_load\":[\"0002\",\"0006\",\"0007\",\"0008\",\"0010\",\"0013\",\"0014\",\"0016\",\"0018\",\"0000\",\"0001\",\"0003\",\"0004\",\"0005\",\"0009\",\"0011\",\"0012\",\"0015\",\"0017\",\"0019\",\"0020\"]}"
```
The json string supplied as an additional argument here overwrites the settings in the config file. Use `video_tags_to_load` to obtain predictions for specific sequences (in the example, all KITTI MOTS sequences are chosen). Output is written to the `forwarded/` subdirectory.

The model predictions as obtained by the previous command are not yet linked over time. You can use the following command to run the tracking algorithm described in the paper and to obtain final results in the `forwarded/` subdirectory which can be processed by the mots_tools scripts:
```
python main.py configs/conv3d_sep2
"{\"build_networks\":false,\"import_detections\":true,\"task\":\"forward_tracking\",\"dataset\":\"KITTI_segtrack_feed\",\"do_tracking\":true,\"visualize_detections\":false,\"visualize_tracks\":false,\"load_epoch_no\":5,\"video_tags_to_load\":[\"0002\",\"0006\",\"0007\",\"0008\",\"0010\",\"0013\",\"0014\",\"0016\",\"0018\"]}"
```
You can also visualize the tracking results here by setting `visualize_tracks` to true.

### Tuning
The script for random tuning will find the best combination of tracking parameters on the training set and then evaluate these parameters on the validation set. This is how the results in the MOTS paper are obtained.

To use this script, run
```
python scripts/eval/segtrack_tune_experiment.py /path/to/detections/ /path/to/groundtruth/ /path/to/precomputed_optical_flow/ /path/to/output_file /path/to/tmp_folder/ /path/to/mots_eval/ association_type num_iterations
```
where `/path/to/detections/` is a folder containing the model output on the training set (obtained by the forwarding command above); `/path/to/mots_eval/` refers to the official evaluation script (link see above); `association_type` determines the method for associating detections into tracks and is either `reid` (using the association embeddings), `mask` (using mask warping), `bbox_iou` (using bounding box warping with median optical flow) or `bbox_center` (nearest neighbor matching); `num_iterations` is the number of random trials (1000 in the paper).

## References
Parts of this code are based on Tensorpack (https://github.com/tensorpack/tensorpack/tree/master/examples/FasterRCNN) and RETURNN (https://github.com/rwth-i6/returnn/blob/master/Log.py).

## Citation
If you use this code, please cite:
```
@inproceedings{Voigtlaender19CVPR_MOTS,
 author = {<NAME> and <NAME> and Aljo\u{s}<NAME>\u{s}ep and <NAME> and <NAME> and <NAME> and <NAME>},
{'fraction_non_alphanumeric': 0.13246655360844167, 'fraction_numerical': 0.026380252496702467, 'mean_word_length': 5.274231678486998, 'pattern_counts': {'":': 16, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 10, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '391971', 'n_tokens_mistral': 1761, 'n_tokens_neox': 1580, 'n_words': 520}