Warn Some weights of the model checkpoint at cointegrated/LaBSE-en-ru were not used when initializing BertModel:
#4
by
alashkov83
- opened
When I'm using LaBSE-en-ru with this code:
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
model = AutoModel.from_pretrained(model_name)
model.eval()
with torch.no_grad():
model = model.to(device)
model_output = model(**encoded_input)
embeddings = model_output.pooler_output
embeddings = torch.nn.functional.normalize(embeddings)
```,
I'm getting this warn:
Some weights of the model checkpoint at cointegrated/LaBSE-en-ru were not used when initializing BertModel:
['cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.bias
', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight']
Is this normal behavior?
I think this should be fine. The sentence embeddings are the outputs of the pooler; cls is supposed to be applied on top of the pooler (if I interpret the source code in https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py correctly), but is actually never used if you use the model to extract the embeddings.