Prune the model for certain languages

#17
by fatshady - opened

What is the most efficient way to prune model for one or two languages so it is smaller and less memory intensive?

Cohere For AI org

If you only care about a subset of languages would recommend finetuning a new mt5 base using the released data filtered for only a subset of languages -- https://huggingface.co/datasets/CohereForAI/aya_collection. Typically models suffer from the curse of multilinguality -- so more capacity is needed to model more languages. If you reduce the number of languages and finetune on a smaller subset -- you will likely see bumps in performance for the languages you care about.

Modifying weights to selectively retain knowledge is an area of active research, but there are not yet very precise methods for ensuring when your prune you don't erode performance on languages you care about.

shivi changed discussion status to closed

Sign up or log in to comment