Any guidance on how to use this with sentence-transformers without downloading a bunch of extra stuff?

#3
by simonw - opened

This is a really cool model!

I'm using this with SentenceTransformers and it downloaded 314M of files. The big ones were:

model.safetensors 43M
pytorch_model.bin 43M
onnx/model.onnx 86M
onnx/model_optimized.onnx 86M
onnx/model_quantized.onnx 22M

Is there a way to use this with SentenceTransformers that only downloads the model file that I need?

hi @simonw ,

You can use huggingface_hub and snapshot only the needed files and pass the folder to the SentenceTransformer constructor

from sentence_transformers import SentenceTransformer
from huggingface_hub import snapshot_download

sentences = ["This is an example sentence", "Each sentence is converted"]
model_path = snapshot_download(
    repo_id="TaylorAI/gte-tiny", allow_patterns=["*.json", "pytorch_model.bin"]
)

model = SentenceTransformer(model_path)
embeddings = model.encode(sentences)
print(embeddings)

I'd also recommend the use of model.safetensors

from sentence_transformers import SentenceTransformer
from huggingface_hub import snapshot_download

sentences = ["This is an example sentence", "Each sentence is converted"]
model_path = snapshot_download(
    repo_id="TaylorAI/gte-tiny", allow_patterns=["*.json", "model.safetensors"]
)

model = SentenceTransformer(model_path)
embeddings = model.encode(sentences)
print(embeddings)

I can also release another minimal version without the onnx weights for SentenceTransformers.

txtai has built-in logic for mean/cls pooling using Transformers, which only downloads the files it needs.

For example:

import txtai

embeddings = txtai.Embeddings(path="TaylorAI/gte-tiny")
embeddings.batchtransform(["text1", "text2"])

very cool @davidmezzetti , just test it
image.png

Sign up or log in to comment