Inference Time
#4
by
omarabb315
- opened
how is it possible for a 8B model to run in 0.25 seconds?
Natural Language Processing Group, Institute of Computing Technology, Chinese Academy of Science org
Thank you for your interest in our work! The latency calculated here refers to the time taken to generate the first audio chunk. Based on the results in Table 2 of our paper, when we set the chunk size to a smaller value like 10, this time corresponds to generating approximately 1.82 words. We measured this time on an L40 GPU, so it is possible to achieve latency of less than 0.25 seconds.