File size: 1,275 Bytes
d9f426e 190f88c 62187c4 190f88c 62187c4 8bda6ae aeef52d 8bda6ae 62187c4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 |
---
license: apache-2.0
datasets:
- 2toINF/IVM-Mix-1M
base_model:
- llava-hf/llava-1.5-7b-hf
---
## Quick Start
### Install
1. Clone this repository and navigate to IVM folder
```bash
git clone https://github.com/2toinf/IVM.git
cd IVM
```
2. Install Package
```bash
conda create -n IVM python=3.10 -y
conda activate IVM
pip install -e .
```
### Usage
```python
from IVM import load, forward_batch
ckpt_path = "IVM-V1.0.bin" # your model path here
model = load(ckpt_path, low_gpu_memory = False) # Set `low_gpu_memory=True` if you don't have enough GPU Memory
image = Image.open("image/demo/robot.jpg") # your image path
instruction = "pick up the red cup and place it on the green pan"
result = forward_batch(model, [image], [instruction], threshold = 0.99)
from matplotlib import pyplot as plt
import numpy as np
plt.imshow((result[0]).astype(np.uint8))
```
Citation
```
@misc{zheng2024instructionguided,
title={Instruction-Guided Visual Masking},
author={Jinliang Zheng and Jianxiong Li and Sijie Cheng and Yinan Zheng and Jiaming Li and Jihao Liu and Yu Liu and Jingjing Liu and Xianyuan Zhan},
year={2024},
eprint={2405.19783},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
``` |