ctheodoris

/

Geneformer

like 206

perturber and emb-extractor works even when provided pytorch_model.bin with the classification weights/biases (no need to remove classification heads to do downstream tasks:
newly initialized: ['cls.predictions.bias', 'cls.predictions.decoder.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight'] treated correctly with BertForMaskedLM