Mistral-Nemo-Instruct-2407 just hangs on my Mac M3?

#53
by lancejpollard - opened

Here is the question I posted on StackOverflow, cross-posting here: https://stackoverflow.com/questions/78801372/huggingface-mistral-nemo-instruct-2407-python-script-for-text-generation-just-ha

It's basically that, when running on my Mac M3 with 4 TB hard drive and 36 GB RAM, it just hangs, I am not sure what's going on. Any ideas how to get this to work?

On my M1 Max with 32 GB I need to use the following command in the terminal:
sudo sysctl iogpu.wired_limit_mb=28000
This will allow more memory to be used for the GPUs. Maybe you will also have to limit the context size as the default size is very large.

Sign up or log in to comment