metadata

tags:
  - vision
  - clip
  - clip4clip
  - video
  - retrieval
pipeline_tag: text-to-video

Model Card

Details

This model underwent training using CLIP4Clip, a video retrieval method based on the CLIP framework, as described in the paper here and implemented in the accompanying code.

The training process involved 150,000 videos obtained from the WebVid Dataset, a comprehensive collection of short videos with corresponding textual descriptions sourced from the web.

To adapt the clip model obtained during training, we adjusted the weights and integrated them into the implementation of clip-vit-base-patch32, making certain modifications to the final layers.

Use with Transformers

import numpy as np
import torch
from transformers import AutoTokenizer, CLIPTextModelWithProjection


search_sentence = "a basketball player performing a slam dunk"

model = CLIPTextModelWithProjection.from_pretrained("Diangle/clip4clip-webvid")
tokenizer = AutoTokenizer.from_pretrained("Diangle/clip4clip-webvid")


inputs = tokenizer(text=search_sentence , return_tensors="pt", padding=True)

outputs = model(input_ids=inputs["input_ids"], attention_mask=inputs["attention_mask"], return_dict=False)

# Special projection and changing last layers:      
text_projection = model.state_dict()['text_projection.weight']
text_embeds = outputs[1] @ text_projection
final_output = text_embeds[torch.arange(text_embeds.shape[0]), inputs["input_ids"].argmax(dim=-1)]

# Normalizing the embeddings:
final_output = final_output / final_output.norm(dim=-1, keepdim=True)
final_output = final_output.cpu().detach().numpy()
sequence_output = final_output / np.sum(final_output**2, axis=1, keepdims=True)
print("sequence_output: ", sequence_output)

Model Use

Intended Use

This model is intended to use for video retrival, look for example this space.

Extra Information

For video embedding there is an extra notebook that describes how to embedd videos.

Performance and Limitations

Performance

We have evaluated the performance