Spaces:
Build error
title: Relate Anything
emoji: 👁
colorFrom: green
colorTo: red
sdk: gradio
sdk_version: 3.27.0
app_file: app.py
pinned: false
license: mit
RAM: Relate-Anything-Model
The following developers have equally contributed to this project in their spare time, the names are in alphabetical order.
Zujin Guo, Bo Li, Jingkang Yang, Zijian Zhou.
Affiliate: MMLab@NTU & VisCom Lab, KCL/TongJi
🚀 🚀 🚀 This is a demo that combine Meta's Segment-Anything model with the ECCV'22 paper: Panoptic Scene Graph Generation.
🔥🔥🔥 Please star our codebase OpenPSG and RAM if you find it useful/interesting.
[Dataset]
Relate Anything Model is capable of taking an image as input and utilizing SAM to identify the corresponding mask within the image. Subsequently, RAM can provide an analysis of the relationship between any arbitrary objects mask.
The object masks are generated using SAM. RAM was trained to detect the relationships between the object masks using the OpenPSG dataset, and the specifics of this method are outlined in a subsequent section.
Examples
Our current demo supports:
(1) generate arbitary objects masks and reason relationships in between.
(2) given coordinates then generate object masks and reason the relationship between given objects and other objects in the image.
We will soon add support for detecting semantic labels of objects with the help of OVSeg.
Here are some examples of the Relate Anything Model in action about playing soccer, dancing, and playing basketball.
Method
RAM utilizes the Segment Anything Model (SAM) to accurately mask objects within an image, and subsequently extract features corresponding to the segmented regions. Employ a Transformer module to facilitate feature interaction among distinct objects, and ultimately compute pairwise object relationships, thereby categorizing their interrelations.
Setup
To set up the environment, we use Conda to manage dependencies. To specify the appropriate version of cudatoolkit to install on your machine, you can modify the environment.yml file, and then create the Conda environment by running the following command:
conda env create -f environment.yml
Make sure to use segment_anything in this repository, which includes the mask feature extraction operation.
Download the pretrained model
Place these two models in ./checkpoints/ from the root directory.
Run our demo locally by running the following command:
python app.py
Acknowledgement
We thank Chunyuan Li for his help in setting up the demo.
Citation
If you find this project helpful for your research, please consider citing the following BibTeX entry.
@inproceedings{yang2022psg,
author = {Yang, Jingkang and Ang, Yi Zhe and Guo, Zujin and Zhou, Kaiyang and Zhang, Wayne and Liu, Ziwei},
title = {Panoptic Scene Graph Generation},
booktitle = {ECCV}
year = {2022}
}
@inproceedings{yang2023pvsg,
author = {Yang, Jingkang and Peng, Wenxuan and Li, Xiangtai and Guo, Zujin and Chen, Liangyu and Li, Bo and Ma, Zheng and Zhou, Kaiyang and Zhang, Wayne and Loy, Chen Change and Liu, Ziwei},
title = {Panoptic Video Scene Graph Generation},
booktitle = {CVPR},
year = {2023},
}