Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation

Qi Yang, Xing Nie, Tong Li, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan and Shiming Xiang

This repository provides the pretrained checkpoints for the paper "Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation" accepted by CVPR 2024.

🔥What's New

(2024. 3.14) Our checkpoints are available to the public!
(2024. 3.12) Our code is available to the public🌲!
(2024. 2.27) Our paper(COMBO) is accepted by CVPR 2024!
(2023.11.17) We completed the implemention of COMBO and push the code.

🛠️ Getting Started

1. Environments

Linux or macOS with Python ≥ 3.6

# recommended
pip install -r requirements.txt
pip install soundfile
# build MSDeformAttention
cd model/modeling/pixel_decoder/ops
sh make.sh

Preprocessing for detectron2

For using Siam-Encoder Module (SEM), we refine 1-line code of the detectron2.

The refined file that requires attention is located at:

conda_envs/xxx/lib/python3.xx/site-packages/detectron2/checkpoint/c2_model_loading.py (refine the xxx to your own environment)

Commenting out the following code in L287 will allow the code to run without errors:

# raise ValueError("Cannot match one checkpoint key to multiple keys in the model.")

Install Semantic-SAM (Optional)

# Semantic-SAM
pip install git+https://github.com/cocodataset/panopticapi.git
git clone https://github.com/UX-Decoder/Semantic-SAM
cd Semantic-SAM
python -m pip install -r requirements.txt

Find out more at Semantic-SAM

2. Datasets

Please refer to the link AVSBenchmark to download the datasets. You can put the data under data folder or rename your own folder. Remember to modify the path in config files. The data directory is as bellow:

|--AVS_dataset
   |--AVSBench_semantic/
   |--AVSBench_object/Multi-sources/
   |--AVSBench_object/Single-source/

3. Download Pre-Trained Models

The pretrained backbone is available from benchmark AVSBench pretrained backbones[TODO].

|--pretrained
   |--detectron2/R-50.pkl
   |--detectron2/d2_pvt_v2_b5.pkl
   |--vggish-10086976.pth
   |--vggish_pca_params-970ea276.pth

4. Maskiges pregeneration

Generate class-agnostic masks (Optional)

sh avs_tools/pre_mask/pre_mask_semantic_sam_s4.sh train # or ms3, avss
sh avs_tools/pre_mask/pre_mask_semantic_sam_s4.sh val 
sh avs_tools/pre_mask/pre_mask_semantic_sam_s4.sh test

Generate Maskiges (Optional)

python3 avs_tools/pre_mask2rgb/mask_precess_s4.py --split train # or ms3, avss
python3 avs_tools/pre_mask2rgb/mask_precess_s4.py --split val
python3 avs_tools/pre_mask2rgb/mask_precess_s4.py --split test

Move Maskiges to the following folder Note: For convenience, we provide pre-generated Maskiges for S4\MS3\AVSS subset on the TODO hugging face link.

|--AVS_dataset
    |--AVSBench_semantic/pre_SAM_mask/
    |--AVSBench_object/Multi-sources/ms3_data/pre_SAM_mask/
    |--AVSBench_object/Single-source/s4_data/pre_SAM_mask/

5. Train

# ResNet-50
sh scripts/res_train_avs4.sh # or ms3, avss

# PVTv2
sh scripts/pvt_train_avs4.sh # or ms3, avss

6. Test

# ResNet-50
sh scripts/res_test_avs4.sh # or ms3, avss

# PVTv2
sh scripts/pvt_test_avs4.sh # or ms3, avss

🤝 Citing COMBO

@misc{yang2023cooperation,
      title={Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation},
      author={Qi Yang and Xing Nie and Tong Li and Pengfei Gao and Ying Guo and Cheng Zhen and Pengfei Yan and Shiming Xiang},
      year={2023},
      eprint={2312.06462},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}