Li Zhang

Andcircle's activity

• All
• Models
• Datasets
• Spaces
• Papers
• Collections
• Community
• Posts
• Upvotes
• Likes

upvoted an article about 20 hours ago

view article

Article

Recoloring photos with diffusers

By •

Oct 9

• 28

Reacted to BramVanroy's post with 👍 6 months ago

view post

Post

2391

Reply

Does anyone have experience with finetuning Gemma? Even the 2B variant feels more memory heavy than mistral 7B. I know that its vocabulary is much larger (250k) but I'm a bit surprised that the max batch size that I can get in an A100 80GB is only 2 whereas I could fit 4 with mistral 7B - even though Gemma is much smaller except for the embedding layer. Both runs were using FA, same sequence length, same deepspeed zero 3 settings. Oh and yes I'm using the most recent hot fix of transformers that solves a memory issue with Gemma and others.

Any prior experience that you can share or suggestions to improve throughout?

• 
• 
• 4 replies

·

👍

8

8

👀

6

6

+

liked a model 6 months ago

google/paligemma-3b-mix-224

Image-Text-to-Text • Updated Jul 19 • 420k • 60

commented a paper 11 months ago

Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

Paper • 2401.01335 • Published Jan 2 • 64 •

2

New activity in mistralai/Mistral-7B-v0.1 about 1 year ago

Does Mistral support accelerate library?

4

#65 opened about 1 year ago by Sp1der

New activity in tiiuae/falcon-40b over 1 year ago

[Bug] Does not work

58

#3 opened over 1 year ago by catid

New activity in tiiuae/falcon-7b over 1 year ago

Run SFT using PEFT on single a10g with 24G mem, got out of mem problem

#35 opened over 1 year ago by Andcircle

Run SFT using PEFT got error with RotaryEmbedding

1

#34 opened over 1 year ago by Andcircle