facebook/sparsh-gelsight-forcefield-decoder

Sparsh (DINO) + force field decoder for GelSight mini sensor

We decode the touch representations from Sparsh into normal and shear force fields. This allows us to gather an intuition about what the representations capture in terms of forces in a way that is interpretable for humans.

How to Use

For testing Sparsh(DINO) + force field decoder live, you only need a GelSight mini sensor. Follow these steps to run the demo:

1. Clone the sparsh repo

2. Create a folder for downloading the task checkpoints. For example, ${YOUR_PATH}/outputs_sparsh/checkpoints.

3. Download the Sparsh (DINO) base checkpoint

4. Download the decoder checkpoints from this repo.

5. Connect the sensor to your PC.

6. Make sure the device is recognized by the OS (you can use Cheese in Linux to see the video that the sensor is streaming).

7. Running the demo for GelSight mini (please refer to the Sparsh repo README for more information about how to setup the path configs):

python demo_forcefield.py +experiment=digit/downstream_task/forcefield/gelsight_dino paths=${YOUR_PATH_CONFIG} paths.output_dir=${YOUR_PATH}/outputs_sparsh/checkpoints/ test.demo.gelsight_device_id=${YOUR_GELSIGHT_VIDEO_ID}`

The GelSight mini is recognized as a webcam. You can get the video ID by checking in a terminal ls -l /dev/video*.

8. Take the sensor and slide it across the edge of a table, or across objects with interesting textures! Look at the normal field to localize where you're making contact on the sensor's surface. Look at the shear field to gather an intuition about the direction of the shear force that you applied while sliding the sensor. For example, slide the sensor over an edge up and down to get translational shear or rotate the sensor in place to see torsional slip!

BibTeX entry and citation info

    @inproceedings{
    higuera2024sparsh,
    title={Sparsh: Self-supervised touch representations for vision-based tactile sensing},
    author={Carolina Higuera and Akash Sharma and Chaithanya Krishna Bodduluri and Taosha Fan and Patrick Lancaster and Mrinal Kalakrishnan and Michael Kaess and Byron Boots and Mike Lambeta and Tingfan Wu and Mustafa Mukadam},
    booktitle={8th Annual Conference on Robot Learning},
    year={2024},
    url={https://openreview.net/forum?id=xYJn2e1uu8}
}