process_read_pair = tasks.process_read_pair.s
g = group(process_read_pair(r1, r2) for r1, r2 in islice(zip(
    FastGeneralIterator(f1), FastGeneralIterator(f2)), 0, 1000))
result = g.delay()
Using celery to process huge text files