license:odc-by arxiv:2306.07279 arxiv:2212.08051 arxiv:2110.06199 region:us
---
license: odc-by
viewer: false
---

# Scalable 3D Captioning with Pretrained Models



## Dataset Description

- **Paper:** [Scalable 3D Captioning with Pretrained Models](https://arxiv.org/abs/2306.07279)
- **Repository**: [Github](https://github.com/crockwell/Cap3D)



This repository hosts data for [**Scalable 3D Captioning with Pretrained Models**](https://cap3d-um.github.io/), including descriptive captions for 3D objects in [Objaverse](https://arxiv.org/abs/2212.08051) and [ABO](https://arxiv.org/abs/2110.06199) ([**example captioning**](https://tiangeluo.github.io/projectpages/imgs/Cap3D/000.html)). It also includes point clouds and rendered images of Objaverse objects, as well as their Shap-E latent codes. **All captions and related 3D objects here have commercial-friendly licenses** (including CC-BY 4.0, CC-BY-SA 4.0, and CC0 1.0).

## Usage
Please download and unzip files from [**Page**](https://huggingface.co/datasets/tiange/Cap3D/tree/main) according to your usage. The different files, along with their descriptions, are listed in the table below. **Example python scripts to load our data are listed after the table.**

For **Point-E** usage, you can sample `1,024` points from `16,384` via FPS or randomly sampling `[:,torch.randperm(16384)[:1024]]` for finetuning the OpenAI diffusion model.


| Filename                               | Description                                                  |
| -------------------------------------- | ------------------------------------------------------------ |
| Cap3D_automated_{Objaverse, ABO}.csv   | Text descriptions generated by Cap3D. `661,577` and `6,440` 3D-caption pairs for Objaverse and ABO datasets, respectively. All 3D objects contained here have a commercial friendly license. |
| Cap3D_automated_Objaverse_no3Dword.csv | Our current 3D captions are densely packed with "3D-model" terminology, potentially limiting their utility in applications like embodied AI. As such, we've created a version with minimized 3D-related words. |
| Cap3D_automated_Objaverse_allviews.pkl | Objaverse text descriptions of  before putting into CLIP and GPT4. `pickle.load(open('Cap3D_automated_Objaverse_allviews.pkl','rb'))['view0'][UID]` |
| Cap3D_human_{Objaverse, ABO}.pkl       | Our collected human-authored text descriptions (Mostly English, very few non-English). The pickle file with `shape_id` as key. |
| PointCloud_zips                        | `661,577` PointClouds (`16,384` colorful points) extracted from Objaverse objects. Saved as `.ply` file. |
| PointCloud_pt_zips                     | PointClouds saved as torch.Tensor `.pt` files, providing faster loading speed than `.ply`. |
| ShapELatentCode_zips                   | Extracted latent codes for Objaverse objects by using pretrained [Shap-E transmitter](https://github.com/openai/shap-e/blob/main/model-card.md). |
| RenderedImage_zips                     | Rendered images for Objaverse objects: eight unique zip files each contain `661,577` JPEG images, representing different views. |
| RenderedImage_CamMatrix_zips           | The `3x4` transformation matrix of the camera used to render the image. |
| RenderedImage_Mask_zips                | Background mask (IMPERFECT) for rendered images: `bg_mask = np.load('xxx.npz')['bg_mask']` |
| our_finetuned_models                   | Model checkpoints for our finetuned models (point-E, shape-E, etc). Our also provide our finetune codes in [Github](https://github.com/crockwell/Cap3D/tree/main/text-to-3D). |
| blender.zip                            | The blender we used in this project.                         |
| text-to-3D_test                            | Including files that used to evaluate text-to-3D performance.                         |



```
# load our captions
import pandas as pd
captions = pd.read_csv('Cap3D_automated_Objaverse_no3Dword.csv', header=None)

## captions:
##                                        0                                                  1
## 0       ed51a51909ee46c780db3a85e821feb2                           A green and white rifle.
## 1       9110b606f6c547b2980fcb3c8c4b6a1c  a small building with a roof, accompanied by b...
## 2       80d9caaa1fa04502af666135196456e1  a pair of purple and black swords with white h...
## 3       28d43a218cd8466a8c1f82b29b71e314  a small house, island, road with trash, trash ...
## 4       75582285fab442a2ba31733f9c8fae66                              a small, grassy hill.
## ...                                  ...                                                ...
## 661572  ccee95eac2fb48ec92d357e3d853f2bd  a tall, white tower featuring a door, stairs, ...
## 661573  f02f574e85e94b879c1b54f4d3aa4b35           A yellow disc with a hole in the middle.
## 661574  f1d3d36114d34d29a18a8ed1516bf355        pink and white ball with a face and spikes.
## 661575  32d1927d6c0e4d9b9c445fc5988ec6c6  a white and pink bedroom and bathroom, featuri...
## 661576  2fa12ba0af5442c9af8f9bead1a7d020                  Monstera plant with green leaves.

## if u want to obtain the caption for specific UID
caption = captions[captions[0] == '80d9caaa1fa04502af666135196456e1'][1].values[0]


# load point clouds (unzip https://huggingface.co/datasets/tiange/Cap3D/tree/main/PointCloud_pt_zips)
import torch
pts = torch.load('Cap3D_pcs_pt/80d9caaa1fa04502af666135196456e1.pt')

## pts.shape == torch.Size([6, 16384])


```

If you have any questions, please contact [Tiange](mailto:tiange.cs@gmail.com) or [Chris](mailto:cnris@umich.edu).



## Citation Information

If you find our data or code useful, please consider citing:

```bibtex
@article{luo2023scalable,
      title={Scalable 3D Captioning with Pretrained Models},
      author={Luo, Tiange and Rockwell, Chris and Lee, Honglak and Johnson, Justin},
      journal={arXiv preprint arXiv:2306.07279},
      year={2023}
}
```

Please cite ***Objaverse*** and ***ABO*** paper accordingly, when using this data.

```
@inproceedings{deitke2023objaverse,
  title={Objaverse: A universe of annotated 3d objects},
  author={Deitke, Matt and Schwenk, Dustin and Salvador, Jordi and Weihs, Luca and Michel, Oscar and VanderBilt, Eli and Schmidt, Ludwig and Ehsani, Kiana and Kembhavi, Aniruddha and Farhadi, Ali},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={13142--13153},
  year={2023}
}
```

```
@inproceedings{collins2022abo,
  title={Abo: Dataset and benchmarks for real-world 3d object understanding},
  author={Collins, Jasmine and Goel, Shubham and Deng, Kenan and Luthra, Achleshwar and Xu, Leon and Gundogdu, Erhan and Zhang, Xi and Vicente, Tomas F Yago and Dideriksen, Thomas and Arora, Himanshu and others},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={21126--21136},
  year={2022}
}
```

