Video memory related issues consultation
Because we rent graphics cards on the cloud platform to run experiments, I would like to ask what is the minimum video memory required to run OpenGVLab/InternVL2-Llama3-76B, so that I can rent a suitable graphics card.
If it is convenient, I hope you can tell me the minimum video memory required for the 26B or 40B model. Thank you.
Hello, you can roughly estimate the required GPU memory based on the model's parameter size. For example, to run the OpenGVLab/InternVL2-Llama3-76B model using fp16 or bf16, you'll need at least 76 x 2 = 152 GB of GPU memory. Since additional memory is needed for running the model, it is recommended to use at least three 80 GB GPUs. If you run the model with AWQ in 4-bit precision, you'll need at least 76 / 2 = 38 GB of GPU memory, so a single 80 GB GPU would suffice. The memory requirements for other model sizes can be estimated using a similar approach.
Thank you very much