stefan-it
Abstract
We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both modeling paradigms within a single transformer stack: GPT-BERT can be transparently used like any standard causal or masked language model. We test the pretraining process that enables this flexible behavior on the BabyLM Challenge 2024. The results show that the hybrid pretraining outperforms masked-only or causal-only models. We openly release the models, training corpora and code.
Community
Hi Stefan, thanks for submitting this paper here, we will release the code in a few days, after some quick cleaning :) By the way, I'm one of the authors, I think you assigned it to a different David Samuel ;)
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data (2024)
- Are BabyLMs Second Language Learners? (2024)
- Generative Model for Less-Resourced Language with 1 billion parameters (2024)
- Adaptive Large Language Models By Layerwise Attention Shortcuts (2024)
- Exploring Pretraining via Active Forgetting for Improving Cross Lingual Transfer for Decoder Language Models (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 2
Datasets citing this paper 2
Spaces citing this paper 0
No Space linking this paper