Q:

Standard way to wait for all tasks to finish before exiting

I was wondering - is there a straightforward way to wait for all tasks to finish running before exiting without keeping track of all the ObjectIDs (and get()ing them)? Use case is when I launch @remotes for saving output, for example, where there is no return result needed. It's just extra stuff to keep track of if I have to store those futures.

A:

Currently there is no standard way to block until all tasks have finished.
There are some workarounds that can be used.

Keep track of all of the object IDs in a list object_ids and then call ray.get(object_ids) or ray.wait(object_ids, num_returns=len(object_ids)).
Loop as long as some resources are being used.
import time

while (ray.global_state.cluster_resources() !=
       ray.global_state.available_resources()):
    time.sleep(1)

The above code will loop until it detects that no tasks are currently being executed. However this is not a foolproof approach. It's possible that there could be a moment in time when no tasks are running but a scheduler a task is about to start running.

