1. A method comprising: receiving, via a near real time publish application program interface (API), data to be indexed in a distributed file system; determining a shard responsible for the data to be indexed; and generating a message in a shard queue corresponding to the shard responsible for the data to be indexed, the message indicating that data needs to be urgently indexed, the detection of the message in the shard queue by a near real time manager corresponding to the shard responsible for the data to be indexed causing the near real time manager to cause the data to be indexed.