Running in CPU

by hedrergudene - opened Jun 26

Jun 26

As flash-attn is part of the requirements of this model, is there any chance of having a runtime version of this model in CPU? It would be particularly relevant given its size :)

atb29

Jun 30

As flash-attn is part of the requirements of this model, is there any chance of having a runtime version of this model in CPU? It would be particularly relevant given its size :)

did you find a way to run it in CPU

wambugu71

Jul 2

•

edited Jul 2

Check out a workaround for CPU inference

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment