region:us
# Datacomp200m 

This is a smaller version of the [datacomp_1b](https://huggingface.co/datasets/mlfoundations/datacomp_1b) dataset. 

Filtering was done by taking all rows that had self similarity (inner product) above 0.32. This resulted in 213009083 (213 million) rows. 

The results of the datacomp paper suggest that filtering by CLIP score is better than random sampling. 

Included in this repo are search indices created using autofaiss, over the text and image embeddings. There are two ways to access metadata, either in .parquet files in the `./metadata` directory, or the `./index/metadata.hdf5` hdf5 file.

I would suggest using [embedding-reader](https://github.com/rom1504/embedding-reader) to load the text and image embeddings.