
 
CLAP
	
Overview
	
The CLAP model was proposed in Large Scale Contrastive Language-Audio pretraining with
feature fusion and keyword-to-caption augmentation by Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov.
CLAP (Contrastive Language-Audio Pretraining) is a neural network trained on a variety of (audio, text) pairs. It can be instructed in to predict the most relevant text snippet, given an audio, without directly optimizing for the task. The CLAP model uses a SWINTransformer to get audio features from a log-Mel spectrogram input, and a RoBERTa model to get text features. Both the text and audio features are then projected to a latent space with identical dimension. The dot product between the projected audio and text features is then used as a similar score.
The abstract from the paper is the following:
Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zeroshot setting and is able to obtain performance comparable to models’ results in the non-zero-shot setting. LAION-Audio-6
This model was contributed by Younes Belkada and Arthur Zucker .
The original code can be found here.
ClapConfig
	
class transformers.ClapConfig
<
source
>
(
text_config = None
audio_config = None
logit_scale_init_value = 14.285714285714285
projection_dim = 512
projection_hidden_act = 'relu'
initializer_factor = 1.0
**kwargs
)
Parameters 
text_config (dict, optional) —
Dictionary of configuration options used to initialize ClapTextConfig.
audio_config (dict, optional) —
Dictionary of configuration options used to initialize ClapAudioConfig.
projection_dim (int, optional, defaults to 512) —
Dimentionality of text and audio projection layers.
logit_scale_init_value (float, optional, defaults to 2.6592) —
The inital value of the logit_scale paramter. Default is used as per the original CLAP implementation.
projection_hidden_act (str, optional, defaults to "relu") —
Activation function for the projection layers.
initializer_factor (float, optional, defaults to 1.0) —
Factor to scale the initialization of the model weights.
kwargs (optional) —
Dictionary of keyword arguments.
ClapConfig is the configuration class to store the configuration of a ClapModel. It is used to instantiate
a CLAP model according to the specified arguments, defining the text model and audio model configs. Instantiating a
configuration with the defaults will yield a similar configuration to that of the CLAP
laion/clap-htsat-fused architecture.
Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.
Example:
	Copied
 from transformers import ClapConfig, ClapModel
 # Initializing a ClapConfig with laion-ai/base style configuration
 configuration = ClapConfig()
 # Initializing a ClapModel (with random weights) from the laion-ai/base style configuration
 model = ClapModel(configuration)
 # Accessing the model configuration
 configuration = model.config
 # We can also initialize a ClapConfig from a ClapTextConfig and a ClapAudioConfig
 from transformers import ClapTextConfig, ClapAudioConfig
 # Initializing a ClapText and ClapAudioConfig configuration
 config_text = ClapTextConfig()
 config_audio = ClapAudioConfig()
 config = ClapConfig.from_text_audio_configs(config_text, config_audio)
from_text_audio_configs
<
source
>
(
text_config: ClapTextConfig
audio_config: ClapAudioConfig
**kwargs
)
→
ClapConfig
Returns
ClapConfig
An instance of a configuration object
Instantiate a ClapConfig (or a derived class) from clap text model configuration and clap audio model
configuration.
ClapTextConfig
	
class transformers.ClapTextConfig
<
source
>
(
vocab_size = 50265
hidden_size = 768
num_hidden_layers = 12
num_attention_heads = 12
intermediate_size = 3072
hidden_act = 'gelu'
hidden_dropout_prob = 0.1
attention_probs_dropout_prob = 0.1
max_position_embeddings = 514
type_vocab_size = 1
initializer_factor = 1.0
layer_norm_eps = 1e-12
projection_dim = 512
pad_token_id = 1
bos_token_id = 0
eos_token_id = 2
position_embedding_type = 'absolute'
use_cache = True
projection_hidden_act = 'relu'
**kwargs
)
Parameters 
vocab_size (int, optional, defaults to 30522) —
Vocabulary size of the CLAP model. Defines the number of different tokens that can be represented by the
inputs_ids passed when calling ClapTextModel.
hidden_size (int, optional, defaults to 768) —
Dimensionality of the encoder layers and the pooler layer.
num_hidden_layers (int, optional, defaults to 12) —
Number of hidden layers in the Transformer encoder.
num_attention_heads (int, optional, defaults to 12) —
Number of attention heads for each attention layer in the Transformer encoder.
intermediate_size (int, optional, defaults to 3072) —
Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.
hidden_act (str or Callable, optional, defaults to "relu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "relu",
"relu", "silu" and "relu_new" are supported.
hidden_dropout_prob (float, optional, defaults to 0.1) —
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
attention_probs_dropout_prob (float, optional, defaults to 0.1) —
The dropout ratio for the attention probabilities.
max_position_embeddings (int, optional, defaults to 512) —
The maximum sequence length that this model might ever be used with. Typically set this to something large
just in case (e.g., 512 or 1024 or 2048).
type_vocab_size (int, optional, defaults to 2) —
The vocabulary size of the token_type_ids passed when calling ClapTextModel.
layer_norm_eps (float, optional, defaults to 1e-12) —
The epsilon used by the layer normalization layers.
position_embedding_type (str, optional, defaults to "absolute") —
Type of position embedding. Choose one of "absolute", "relative_key", "relative_key_query". For
positional embeddings use "absolute". For more information on "relative_key", please refer to
Self-Attention with Relative Position Representations (Shaw et al.).
For more information on "relative_key_query", please refer to Method 4 in Improve Transformer Models
with Better Relative Position Embeddings (Huang et al.).
is_decoder (bool, optional, defaults to False) —
Whether the model is used as a decoder or not. If False, the model is used as an encoder.
use_cache (bool, optional, defaults to True) —
Whether or not the model should return the last key/values attentions (not used by all models). Only
relevant if config.is_decoder=True.
projection_hidden_act (str, optional, defaults to "relu") —
The non-linear activation function (function or string) in the projection layer. If string, "gelu",
"relu", "silu" and "gelu_new" are supported.
projection_dim (int, optional, defaults to 512) —
Dimension of the projection head of the ClapTextModelWithProjection.
This is the configuration class to store the configuration of a ClapTextModel. It is used to instantiate a CLAP
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the CLAP
calp-hsat-fused architecture.
Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.
Examples:
	Copied
 from transformers import ClapTextConfig, ClapTextModel
 # Initializing a CLAP text configuration
 configuration = ClapTextConfig()
 # Initializing a model (with random weights) from the configuration
 model = ClapTextModel(configuration)
 # Accessing the model configuration
 configuration = model.config
ClapAudioConfig
	
class transformers.ClapAudioConfig
<
source
>
(
window_size = 8
num_mel_bins = 64
spec_size = 256
hidden_act = 'gelu'
patch_size = 4
patch_stride = [4, 4]
num_classes = 527
hidden_size = 768
projection_dim = 512
depths = [2, 2, 6, 2]
num_attention_heads = [4, 8, 16, 32]
enable_fusion = False
hidden_dropout_prob = 0.1
fusion_type = None
patch_embed_input_channels = 1
flatten_patch_embeds = True
patch_embeds_hidden_size = 96
enable_patch_layer_norm = True
drop_path_rate = 0.0
attention_probs_dropout_prob = 0.0
qkv_bias = True
mlp_ratio = 4.0
aff_block_r = 4
num_hidden_layers = 4
projection_hidden_act = 'relu'
layer_norm_eps = 1e-05
initializer_factor = 1.0
**kwargs
)
Parameters 
window_size (int, optional, defaults to 8) —
Image size of the spectrogram
num_mel_bins (int, optional, defaults to 64) —
Number of mel features used per frames. Should correspond to the value used in the ClapProcessor class.
spec_size (int, optional, defaults to 256) —
Desired input size of the spectrogram that the model supports. It can be different from the output of the
ClapFeatureExtractor, in which case the input features will be resized. Corresponds to the image_size
of the audio models.
hidden_act (str, optional, defaults to "gelu") —
The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu",
"relu", "silu" and "gelu_new" are supported.
patch_size (int, optional, defaults to 4) —
Patch size for the audio spectrogram
patch_stride (list, optional, defaults to [4, 4]) —
Patch stride for the audio spectrogram
num_classes (int, optional, defaults to 527) —
Number of classes used for the head training
hidden_size (int, optional, defaults to 768) —
Hidden size of the output of the audio encoder. Correspond to the dimension of the penultimate layer’s
output,which is sent to the projection MLP layer.
projection_dim (int, optional, defaults to 512) —
Hidden size of the projection layer.
depths (list, optional, defaults to [2, 2, 6, 2]) —
Depths used for the Swin Layers of the audio model
num_attention_heads (list, optional, defaults to [4, 8, 16, 32]) —
Number of attention heads used for the Swin Layers of the audio model
enable_fusion (bool, optional, defaults to False) —
Whether or not to enable patch fusion. This is the main contribution of the authors, and should give the
best results.
hidden_dropout_prob (float, optional, defaults to 0.1) —
The dropout probabilitiy for all fully connected layers in the encoder.
fusion_type ([type], optional) —
Fusion type used for the patch fusion.
patch_embed_input_channels (int, optional, defaults to 1) —
Number of channels used for the input spectrogram
flatten_patch_embeds (bool, optional, defaults to True) —
Whether or not to flatten the patch embeddings
patch_embeds_hidden_size (int, optional, defaults to 96) —
Hidden size of the patch embeddings. It is used as the number of output channels.
enable_patch_layer_norm (bool, optional, defaults to True) —
Whether or not to enable layer normalization for the patch embeddings
drop_path_rate (float, optional, defaults to 0.0) —
Drop path rate for the patch fusion
attention_probs_dropout_prob (float, optional, defaults to 0.0) —
The dropout ratio for the attention probabilities.
qkv_bias (bool, optional, defaults to True) —
Whether or not to add a bias to the query, key, value projections.
mlp_ratio (float, optional, defaults to 4.0) —
Ratio of the mlp hidden dim to embedding dim.
aff_block_r (int, optional, defaults to 4) —
downsize_ratio used in the AudioFF block
num_hidden_layers (int, optional, defaults to 4) —
Number of hidden layers in the Transformer encoder.
projection_hidden_act (str, optional, defaults to "relu") —
The non-linear activation function (function or string) in the projection layer. If string, "gelu",
"relu", "silu" and "gelu_new" are supported.
layer_norm_eps ([type], optional, defaults to 1e-5) —
The epsilon used by the layer normalization layers.
initializer_factor (float, optional, defaults to 1.0) —
A factor for initializing all weight matrices (should be kept to 1, used internally for initialization
testing).
This is the configuration class to store the configuration of a ClapAudioModel. It is used to instantiate a
CLAP audio encoder according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the audio encoder of the CLAP
laion/clap-htsat-fused architecture.
Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the
documentation from PretrainedConfig for more information.
Example:
	Copied
 from transformers import ClapAudioConfig, ClapAudioModel
 # Initializing a ClapAudioConfig with laion/clap-htsat-fused style configuration
 configuration = ClapAudioConfig()
 # Initializing a ClapAudioModel (with random weights) from the laion/clap-htsat-fused style configuration
 model = ClapAudioModel(configuration)
 # Accessing the model configuration
 configuration = model.config
ClapFeatureExtractor
	
class transformers.ClapFeatureExtractor
<
source
>
(
feature_size = 64
sampling_rate = 48000
hop_length = 480
max_length_s = 10
fft_window_size = 1024
padding_value = 0.0
return_attention_mask = False
frequency_min: float = 0
frequency_max: float = 14000
top_db: int = None
truncation: str = 'fusion'
padding: str = 'repeatpad'
**kwargs
)
Parameters 
feature_size (int, defaults to 64) —
The feature dimension of the extracted Mel spectrograms. This corresponds to the number of mel filters
(n_mels).
sampling_rate (int, defaults to 48_000) —
The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). This only serves
to warn users if the audio fed to the feature extractor does not have the same sampling rate.
hop_length (int, defaults to 480) —
Length of the overlaping windows for the STFT used to obtain the Mel Spectrogram. The audio will be split
in smaller frames with a step of hop_length between each frame.
max_length_s (int, defaults to 10) —
The maximum input lenght of the model in seconds. This is used to pad the audio.
fft_window_size (int, defaults to 1024) —
Size of the window (in samples) on which the Fourier transform is applied. This controls the frequency
resolution of the spectrogram. 400 means that the fourrier transform is computed on windows of 400 samples.
padding_value (float, optional, defaults to 0.0) —
Padding value used to pad the audio. Should correspond to silences.
return_attention_mask (bool, optional, defaults to False) —
Whether or not the model should return the attention masks coresponding to the input.
frequency_min (float, optional, default to 0) —
The lowest frequency of interest. The STFT will not be computed for values below this.
frequency_max (float, optional, default to 14_000) —
The highest frequency of interest. The STFT will not be computed for values above this.
top_db (float, optional) —
The highest decibel value used to convert the mel spectrogram to the log scale. For more details see the
audio_utils.power_to_db function
truncation (str, optional, default to "fusions") —
Truncation pattern for long audio inputs. Two patterns are available:
fusion will use _random_mel_fusion, which stacks 3 random crops from the mel spectrogram and a
downsampled version of the entire mel spectrogram.
If config.fusion is set to True, shorter audios also need to to return 4 mels, which will just be a copy
of the original mel obtained from the padded audio.
rand_trunc will select a random crop of the mel spectrogram.
padding (str, optional, defaults to "repeatpad") —
Padding pattern for shorter audio inputs. Three patterns were originally implemented:
repeatpad: the audio is repeated, and then padded to fit the max_length.
repeat: the audio is repeated and then cut to fit the max_length
pad: the audio is padded.
Constructs a CLAP feature extractor.
This feature extractor inherits from SequenceFeatureExtractor which contains
most of the main methods. Users should refer to this superclass for more information regarding those methods.
This class extracts mel-filter bank features from raw speech using a custom numpy implementation of the Short Time
Fourier Transform (STFT) which should match pytorch’s torch.stft equivalent.
to_dict
<
source
>
(
)
→
Dict[str, Any]
Returns
Dict[str, Any]
Dictionary of all the attributes that make up this configuration instance, excpet for the
mel filter banks, which do not need to be saved or printed as they are too long.
Serializes this instance to a Python dictionary.
ClapProcessor
	
class transformers.ClapProcessor
<
source
>
(
feature_extractor
tokenizer
)
Parameters 
feature_extractor (ClapFeatureExtractor) —
The audio processor is a required input.
tokenizer (RobertaTokenizerFast) —
The tokenizer is a required input.
Constructs a CLAP processor which wraps a CLAP feature extractor and a RoBerta tokenizer into a single processor.
ClapProcessor offers all the functionalities of ClapFeatureExtractor and RobertaTokenizerFast. See the
__call__() and decode() for more information.
batch_decode
<
source
>
(
*args
**kwargs
)
This method forwards all its arguments to RobertaTokenizerFast’s batch_decode(). Please
refer to the docstring of this method for more information.
decode
<
source
>
(
*args
**kwargs
)
This method forwards all its arguments to RobertaTokenizerFast’s decode(). Please refer
to the docstring of this method for more information.
ClapModel
	
class transformers.ClapModel
<
source
>
(
config: ClapConfig
)
Parameters 
config (ClapConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.)
This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.
forward
<
source
>
(
input_ids: typing.Optional[torch.LongTensor] = None
input_features: typing.Optional[torch.FloatTensor] = None
is_longer: typing.Optional[torch.BoolTensor] = None
attention_mask: typing.Optional[torch.Tensor] = None
position_ids: typing.Optional[torch.LongTensor] = None
return_loss: typing.Optional[bool] = None
output_attentions: typing.Optional[bool] = None
output_hidden_states: typing.Optional[bool] = None
return_dict: typing.Optional[bool] = None
)
→
transformers.models.clap.modeling_clap.ClapOutput or tuple(torch.FloatTensor)
Parameters 
input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?
attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?
input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Input audio features. This should be returnes by the ClapFeatureExtractor class that you can also
retrieve from AutoFeatureExtractor. See ClapFeatureExtractor.__call__() for details.
return_loss (bool, optional) —
Whether or not to return the contrastive loss.
output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.
output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.
return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.
Returns
transformers.models.clap.modeling_clap.ClapOutput or tuple(torch.FloatTensor)
A transformers.models.clap.modeling_clap.ClapOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapConfig'>) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when return_loss is True) — Contrastive loss for audio-text similarity.
logits_per_audio:(torch.FloatTensor of shape (audio_batch_size, text_batch_size)) — The scaled dot product scores between audio_embeds and text_embeds. This represents the audio-text
similarity scores.
logits_per_text:(torch.FloatTensor of shape (text_batch_size, audio_batch_size)) — The scaled dot product scores between text_embeds and audio_embeds. This represents the text-audio
similarity scores.
text_embeds(torch.FloatTensor of shape (batch_size, output_dim) — The text embeddings obtained by applying the projection layer to the pooled output of ClapTextModel.
audio_embeds(torch.FloatTensor of shape (batch_size, output_dim) — The audio embeddings obtained by applying the projection layer to the pooled output of ClapAudioModel.
text_model_output(BaseModelOutputWithPooling):
The output of the ClapTextModel.
audio_model_output(BaseModelOutputWithPooling):
The output of the ClapAudioModel.
The ClapModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
	Copied
 from datasets import load_dataset
 from transformers import AutoProcessor, ClapModel
 dataset = load_dataset("ashraq/esc50")
 audio_sample = dataset["train"]["audio"][0]["array"]
 model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
 processor = AutoProcessor.from_pretrained("laion/clap-htsat-unfused")
 input_text = ["Sound of a dog", "Sound of vaccum cleaner"]
 inputs = processor(text=input_text, audios=audio_sample, return_tensors="pt", padding=True)
 outputs = model(**inputs)
 logits_per_audio = outputs.logits_per_audio  # this is the audio-text similarity score
 probs = logits_per_audio.softmax(dim=-1)  # we can take the softmax to get the label probabilities
get_text_features
<
source
>
(
input_ids: typing.Optional[torch.Tensor] = None
attention_mask: typing.Optional[torch.Tensor] = None
position_ids: typing.Optional[torch.Tensor] = None
output_attentions: typing.Optional[bool] = None
output_hidden_states: typing.Optional[bool] = None
return_dict: typing.Optional[bool] = None
)
→
text_features (torch.FloatTensor of shape (batch_size, output_dim)
Parameters 
input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?
attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?
output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.
output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.
return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.
Returns
text_features (torch.FloatTensor of shape (batch_size, output_dim)
The text embeddings obtained by
applying the projection layer to the pooled output of ClapTextModel.
The ClapModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
	Copied
 from transformers import AutoTokenizer, ClapModel
 model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
 tokenizer = AutoTokenizer.from_pretrained("laion/clap-htsat-unfused")
 inputs = tokenizer(["the sound of a cat", "the sound of a dog"], padding=True, return_tensors="pt")
 text_features = model.get_text_features(**inputs)
get_audio_features
<
source
>
(
input_features: typing.Optional[torch.Tensor] = None
is_longer: typing.Optional[torch.Tensor] = None
attention_mask: typing.Optional[torch.Tensor] = None
output_attentions: typing.Optional[bool] = None
output_hidden_states: typing.Optional[bool] = None
return_dict: typing.Optional[bool] = None
)
→
audio_features (torch.FloatTensor of shape (batch_size, output_dim)
Parameters 
input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Input audio features. This should be returnes by the ClapFeatureExtractor class that you can also
retrieve from AutoFeatureExtractor. See ClapFeatureExtractor.__call__() for details.
is_longer (torch.FloatTensor, of shape (batch_size, 1), optional) —
Whether the audio clip is longer than max_length. If True, a feature fusion will be enabled to enhance
the features.
output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.
output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.
return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.
Returns
audio_features (torch.FloatTensor of shape (batch_size, output_dim)
The audio embeddings obtained by
applying the projection layer to the pooled output of ClapAudioModel.
The ClapModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
	Copied
 from transformers import AutoFeatureExtractor, ClapModel
 import torch
 model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
 feature_extractor = AutoFeatureExtractor.from_pretrained("laion/clap-htsat-unfused")
 random_audio = torch.rand((16_000))
 inputs = feature_extractor(random_audio, return_tensors="pt")
 audio_features = model.get_audio_features(**inputs)
ClapTextModel
	
class transformers.ClapTextModel
<
source
>
(
config
add_pooling_layer = True
)
The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of
cross-attention is added between the self-attention layers, following the architecture described in Attention is
all you need_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
Kaiser and Illia Polosukhin.
To behave as an decoder the model needs to be initialized with the is_decoder argument of the configuration set
to True. To be used in a Seq2Seq model, the model needs to initialized with both is_decoder argument and
add_cross_attention set to True; an encoder_hidden_states is then expected as an input to the forward pass.
.. _Attention is all you need: https://arxiv.org/abs/1706.03762
forward
<
source
>
(
input_ids: typing.Optional[torch.Tensor] = None
attention_mask: typing.Optional[torch.Tensor] = None
token_type_ids: typing.Optional[torch.Tensor] = None
position_ids: typing.Optional[torch.Tensor] = None
head_mask: typing.Optional[torch.Tensor] = None
inputs_embeds: typing.Optional[torch.Tensor] = None
encoder_hidden_states: typing.Optional[torch.Tensor] = None
encoder_attention_mask: typing.Optional[torch.Tensor] = None
past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None
use_cache: typing.Optional[bool] = None
output_attentions: typing.Optional[bool] = None
output_hidden_states: typing.Optional[bool] = None
return_dict: typing.Optional[bool] = None
)
encoder_hidden_states  (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional):
Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
the model is configured as a decoder.
encoder_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional):
Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in
the cross-attention if the model is configured as a decoder. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
past_key_values (tuple(tuple(torch.FloatTensor)) of length config.n_layers with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)):
Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that
don’t have their past key value states given to this model) of shape (batch_size, 1) instead of all
decoder_input_ids of shape (batch_size, sequence_length).
use_cache (bool, optional):
If set to True, past_key_values key value states are returned and can be used to speed up decoding (see
past_key_values).
ClapTextModelWithProjection
	
class transformers.ClapTextModelWithProjection
<
source
>
(
config: ClapTextConfig
)
Parameters 
config (ClapConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.
CLAP Text Model with a projection layer on top (a linear layer on top of the pooled output).
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.)
This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.
forward
<
source
>
(
input_ids: typing.Optional[torch.Tensor] = None
attention_mask: typing.Optional[torch.Tensor] = None
position_ids: typing.Optional[torch.Tensor] = None
output_attentions: typing.Optional[bool] = None
output_hidden_states: typing.Optional[bool] = None
return_dict: typing.Optional[bool] = None
)
→
transformers.models.clap.modeling_clap.ClapTextModelOutput or tuple(torch.FloatTensor)
Parameters 
input_ids (torch.LongTensor of shape (batch_size, sequence_length)) —
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and
PreTrainedTokenizer.call() for details.
What are input IDs?
attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) —
Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:
1 for tokens that are not masked,
0 for tokens that are masked.
What are attention masks?
position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional) —
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].
What are position IDs?
output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.
output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.
return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.
Returns
transformers.models.clap.modeling_clap.ClapTextModelOutput or tuple(torch.FloatTensor)
A transformers.models.clap.modeling_clap.ClapTextModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapTextConfig'>) and inputs.
text_embeds (torch.FloatTensor of shape (batch_size, output_dim) optional returned when model is initialized with with_projection=True) — The text embeddings obtained by applying the projection layer to the pooler_output.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
The ClapTextModelWithProjection forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
	Copied
 from transformers import AutoTokenizer, ClapTextModelWithProjection
 model = ClapTextModelWithProjection.from_pretrained("laion/clap-htsat-unfused")
 tokenizer = AutoTokenizer.from_pretrained("laion/clap-htsat-unfused")
 inputs = tokenizer(["a sound of a cat", "a sound of a dog"], padding=True, return_tensors="pt")
 outputs = model(**inputs)
 text_embeds = outputs.text_embeds
ClapAudioModel
	
class transformers.ClapAudioModel
<
source
>
(
config: ClapAudioConfig
)
forward
<
source
>
(
input_features: typing.Optional[torch.FloatTensor] = None
is_longer: typing.Optional[torch.BoolTensor] = None
output_attentions: typing.Optional[bool] = None
output_hidden_states: typing.Optional[bool] = None
return_dict: typing.Optional[bool] = None
)
→
transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)
Parameters 
input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Input audio features. This should be returnes by the ClapFeatureExtractor class that you can also
retrieve from AutoFeatureExtractor. See ClapFeatureExtractor.__call__() for details.
is_longer (torch.FloatTensor, of shape (batch_size, 1), optional) —
Whether the audio clip is longer than max_length. If True, a feature fusion will be enabled to enhance
the features.
output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.
output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.
return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.
Returns
transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)
A transformers.modeling_outputs.BaseModelOutputWithPooling or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapAudioConfig'>) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — Last layer hidden-state of the first token of the sequence (classification token) after further processing
through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns
the classification token after processing through a linear layer and a tanh activation function. The linear
layer weights are trained from the next sentence prediction (classification) objective during pretraining.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
The ClapAudioModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
	Copied
 from datasets import load_dataset
 from transformers import AutoProcessor, ClapAudioModel
 dataset = load_dataset("ashraq/esc50")
 audio_sample = dataset["train"]["audio"][0]["array"]
 model = ClapAudioModel.from_pretrained("laion/clap-htsat-fused")
 processor = AutoProcessor.from_pretrained("laion/clap-htsat-fused")
 inputs = processor(audios=audio_sample, return_tensors="pt")
 outputs = model(**inputs)
 last_hidden_state = outputs.last_hidden_state
ClapAudioModelWithProjection
	
class transformers.ClapAudioModelWithProjection
<
source
>
(
config: ClapAudioConfig
)
Parameters 
config (ClapConfig) — Model configuration class with all the parameters of the model.
Initializing with a config file does not load the weights associated with the model, only the
configuration. Check out the from_pretrained() method to load the model weights.
CLAP Audio Model with a projection layer on top (a linear layer on top of the pooled output).
This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the
library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
etc.)
This model is also a PyTorch torch.nn.Module subclass.
Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage
and behavior.
forward
<
source
>
(
input_features: typing.Optional[torch.FloatTensor] = None
is_longer: typing.Optional[torch.BoolTensor] = None
output_attentions: typing.Optional[bool] = None
output_hidden_states: typing.Optional[bool] = None
return_dict: typing.Optional[bool] = None
)
→
transformers.models.clap.modeling_clap.ClapAudioModelOutput or tuple(torch.FloatTensor)
Parameters 
input_features (torch.FloatTensor of shape (batch_size, num_channels, height, width)) —
Input audio features. This should be returnes by the ClapFeatureExtractor class that you can also
retrieve from AutoFeatureExtractor. See ClapFeatureExtractor.__call__() for details.
is_longer (torch.FloatTensor, of shape (batch_size, 1), optional) —
Whether the audio clip is longer than max_length. If True, a feature fusion will be enabled to enhance
the features.
output_attentions (bool, optional) —
Whether or not to return the attentions tensors of all attention layers. See attentions under returned
tensors for more detail.
output_hidden_states (bool, optional) —
Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for
more detail.
return_dict (bool, optional) —
Whether or not to return a ModelOutput instead of a plain tuple.
Returns
transformers.models.clap.modeling_clap.ClapAudioModelOutput or tuple(torch.FloatTensor)
A transformers.models.clap.modeling_clap.ClapAudioModelOutput or a tuple of
torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various
elements depending on the configuration (<class 'transformers.models.clap.configuration_clap.ClapAudioConfig'>) and inputs.
audio_embeds (torch.FloatTensor of shape (batch_size, hidden_size)) — The Audio embeddings obtained by applying the projection layer to the pooler_output.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +
one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
The ClapAudioModelWithProjection forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module
instance afterwards instead of this since the former takes care of running the pre and post processing steps while
the latter silently ignores them.
Examples:
	Copied
 from datasets import load_dataset
 from transformers import ClapAudioModelWithProjection, ClapProcessor
 model = ClapAudioModelWithProjection.from_pretrained("laion/clap-htsat-fused")
 processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")
 dataset = load_dataset("ashraq/esc50")
 audio_sample = dataset["train"]["audio"][0]["array"]
 inputs = processor(audios=audio_sample, return_tensors="pt")
 outputs = model(**inputs)
 audio_embeds = outputs.audio_embeds
←Bark
EnCodec→
CLAP
Overview
ClapConfig
ClapTextConfig
ClapAudioConfig
ClapFeatureExtractor
ClapProcessor
ClapModel
ClapTextModel
ClapTextModelWithProjection
ClapAudioModel
ClapAudioModelWithProjection
