Weight size VS VRAM requirements

#8
by mindkrypted - opened

Hello, I'd be interested to hear about what makes the model's inference require the large amount of VRAM ( 4 x 80GB )?
Weight is about 40GB and logically fits into 48GB of VRAM with enough leftover for a short* context.

Very promising model, great job from the team.
Thanks!

Sign up or log in to comment