Batch inference on many images

#41
by yadavsaakash - opened

I have 1000's of images, how do i make proper use of gpu to do batch inference and reduce my total inference time?

I'd use vllm and send a bunch of async requests so that it can batch them for you

Sign up or log in to comment