Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model
Ensembling is a popular method used to improve performance as a last resort.
However, ensembling multiple models finetuned from a single pretrained model
has been not very effective; this could be due to the lack of diversity among
ensemble members. This paper proposes Multi-Ticket Ensemble, which finetunes
different subnetworks of a single pretrained model and ensembles them. We
empirically demonstrated that winning-ticket subnetworks produced more diverse
predictions than dense networks, and their ensemble outperformed the standard
ensemble on some tasks.