File size: 3,842 Bytes
ab7f77b f296566 ab7f77b 2796d96 44eaebc 2796d96 90f9401 90fc30f 2796d96 f296566 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 |
---
license: llama3.1
datasets:
- remyxai/vqasynth_spacellava
language:
- en
library_name: transformers
tags:
- remyx
---
# Model Card for SpaceLlama3.1-hf
**SpaceLlama3.1-hf** uses [llama3.1-8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B) as the llm backbone along with the fused DINOv2+SigLIP features of [prismatic-vlms](https://github.com/TRI-ML/prismatic-vlms).
Uses [OpenVLA](https://github.com/openvla/openvla#converting-prismatic-models-to-hugging-face) to convert the [prismatic-vlm](https://huggingface.co/remyxai/SpaceLlama3.1) to a Huggingface model.
## Model Details
Uses a full fine-tune on the [spacellava dataset](https://huggingface.co/datasets/remyxai/vqasynth_spacellava) designed with [VQASynth](https://github.com/remyxai/VQASynth/tree/main) to enhance spatial reasoning as in [SpatialVLM](https://spatial-vlm.github.io/).
### Model Description
This model uses data synthesis techniques and publically available models to reproduce the work described in SpatialVLM to enhance the spatial reasoning of multimodal models.
With a pipeline of expert models, we can infer spatial relationships between objects in a scene to create VQA dataset for spatial reasoning.
- **Developed by:** remyx.ai
- **Model type:** MultiModal Model, Vision Language Model, Prismatic-vlms, Llama 3.1
- **Finetuned from model:** Llama 3.1
### Model Sources
- **Dataset:** [SpaceLLaVA](https://huggingface.co/datasets/remyxai/vqasynth_spacellava)
- **Repository:** [VQASynth](https://github.com/remyxai/VQASynth/tree/main)
- **Paper:** [SpatialVLM](https://arxiv.org/abs/2401.12168)
## Usage
Try the `run_inference.py` script to run a quick test:
```bash
python run_inference.py --model_location remyxai/SpaceLlama3.1
--image_source "https://remyx.ai/assets/spatialvlm/warehouse_rgb.jpg"
--user_prompt "What is the distance between the man in the red hat and the pallet of boxes?"
```
## Deploy
Under the `docker` directory, you'll find a dockerized Triton Server for this model. Run the following:
```bash
docker build -f Dockerfile -t spacellava-server:latest
docker run -it --rm --gpus all -p8000:8000 -p8001:8001 -p8002:8002 --shm-size 24G spacellama3.1-server:latest
python3 client.py --image_path "https://remyx.ai/assets/spatialvlm/warehouse_rgb.jpg" \
--prompt "What is the distance between the man in the red hat and the pallet of boxes?"
```
## Citation
```
@article{chen2024spatialvlm,
title = {SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities},
author = {Chen, Boyuan and Xu, Zhuo and Kirmani, Sean and Ichter, Brian and Driess, Danny and Florence, Pete and Sadigh, Dorsa and Guibas, Leonidas and Xia, Fei},
journal = {arXiv preprint arXiv:2401.12168},
year = {2024},
url = {https://arxiv.org/abs/2401.12168},
}
@inproceedings{karamcheti2024prismatic,
title = {Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models},
author = {Siddharth Karamcheti and Suraj Nair and Ashwin Balakrishna and Percy Liang and Thomas Kollar and Dorsa Sadigh},
booktitle = {International Conference on Machine Learning (ICML)},
year = {2024},
}
@article{kim24openvla,
title={OpenVLA: An Open-Source Vision-Language-Action Model},
author={{Moo Jin} Kim and Karl Pertsch and Siddharth Karamcheti and Ted Xiao and Ashwin Balakrishna and Suraj Nair and Rafael Rafailov and Ethan Foster and Grace Lam and Pannag Sanketi and Quan Vuong and Thomas Kollar and Benjamin Burchfiel and Russ Tedrake and Dorsa Sadigh and Sergey Levine and Percy Liang and Chelsea Finn},
journal = {arXiv preprint arXiv:2406.09246},
year={2024}
}
``` |