Flora: Low-Rank Adapters Are Secretly Gradient Compressors
Paper
•
2402.03293
•
Published
•
6
Yongchang Hao
yongchanghao
AI & ML interests
None yet
Recent Activity
Reacted to
their
post
with 🔥
20 days ago
We just released a paper (NeuZip) that compresses VRAM in a lossless manner to run larger models. This should be particularly useful when VRAM is insufficient during training/inference. Specifically, we look inside each floating number and find that the exponents are highly compressible (as shown in the figure below).
Read more about the work at https://huggingface.co/papers/2410.20650
authored
a paper
21 days ago
Teacher Forcing Recovers Reward Functions for Text Generation
posted
an
update
23 days ago
We just released a paper (NeuZip) that compresses VRAM in a lossless manner to run larger models. This should be particularly useful when VRAM is insufficient during training/inference. Specifically, we look inside each floating number and find that the exponents are highly compressible (as shown in the figure below).
Read more about the work at https://huggingface.co/papers/2410.20650
Organizations
