Out of memory when using mergekit

#10

by kongzym - opened Sep 25

Sep 25

Hey, great work @mlabonne

I am currently using mergkit to merge two 70B models on a node with 8*A800, but got out of memory issues.
Do you know mergekit support loading a model on different GPUs? I checked the code, but not find the related part.

Thanks in advance!

mlabonne

Owner Sep 25

Hey, you shouldn't need any GPU to perform the merge. You might require additional RAM perhaps?

kongzym

Sep 26

Thanks for your reply! I am using merge-evolve, in which a CMA-ES was implemented. This requires GPUs, I guess.

mlabonne

Owner Sep 26

Oh alright, sorry I'm not familiar with this technique

kongzym

Sep 26

It's okay. Thank you!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment