Can we parallelized the add_faiss_index process over dataset shards ?
Actually, you are right. I also had the same idea. I am trying this in the context of end-ton-end retrieval training in RAG.  So far I have parallelized the embedding re-computation within the training loop by using datasets shards. 

Then I was thinking of can I calculate the indexes for each shard and combined them with **concatenate**  before I save.
I am thinking of making the  **add_faiss_index** process faster. What if we run the add_faiss_index process on separate dataset shards and then combine them before (dataset.concatenate) saving the faiss.index file ?

I feel theoretically this will reduce the accuracy of retrieval since it affects the indexing process.

@lhoestq

Can we parallelized the add_faiss_index process over dataset shards ? 
 I am thinking of making the  **add_faiss_index** process faster. What if we run the add_faiss_index process on separate dataset shards and then combine them before (dataset.concatenate) saving the faiss.index file ?

I feel theoretically this will reduce the accuracy of retrieval since it affects the indexing process.

@lhoestq
 
 Actually, you are right. I also had the same idea. I am trying this in the context of end-ton-end retrieval training in RAG.  So far I have parallelized the embedding re-computation within the training loop by using datasets shards. 

Then I was thinking of can I calculate the indexes for each shard and combined them with **concatenate**  before I save.