add_faisis_index  gets very slow when doing it interatively  
@lhoestq  on a different note, I read about using Faiss-GPU, but the documentation says we should use it when the dataset has the ability to fit into the GPU memory. Although this might work, in the long-term this is not that practical for me.

https://github.com/matsui528/faiss_tips
As the below code suggests, I want to run add_faisis_index in every nth interaction from the training loop. I have 7.2 million documents. Usually, it takes 2.5 hours (if I run an as a separate process similar to the script given in rag/use_own_knowleldge_dataset.py). Now, this takes usually 5hrs. Is this normal?   Any way to make this process faster? 

@lhoestq 

```
   def training_step(self, batch, batch_idx) -> Dict:

    
        if (not batch_idx==0) and (batch_idx%5==0):

            print("******************************************************")
            ctx_encoder=self.trainer.model.module.module.model.rag.ctx_encoder
            model_copy =type(ctx_encoder)(self.config_dpr) # get a new instance  #this will be load in the CPU
            model_copy.load_state_dict(ctx_encoder.state_dict()) # copy weights and stuff


            list_of_gpus = ['cuda:2','cuda:3']
            c_dir='/custom/cache/dir'

            kb_dataset = load_dataset("csv", data_files=[self.custom_config.csv_path], split="train", delimiter="\t", column_names=["title", "text"],cache_dir=c_dir) 

            print(kb_dataset)

      
            n=len(list_of_gpus) #nunber of dedicated GPUs
            kb_list=[kb_dataset.shard(n, i, contiguous=True) for i in range(n)]

            #kb_dataset.save_to_disk('/hpc/gsir059/MY-Test/RAY/transformers/examples/research_projects/rag/haha-dir')


            print(self.trainer.global_rank)
            dataset_shards = self.re_encode_kb(model_copy.to(device=list_of_gpus[self.trainer.global_rank]),kb_list[self.trainer.global_rank])
            output = [None for _ in list_of_gpus]

            #self.trainer.accelerator_connector.accelerator.barrier("embedding_process")
            dist.all_gather_object(output, dataset_shards)
            

            #This creation and re-initlaization of the new index
            if (self.trainer.global_rank==0):  #saving will be done in the main process 
           
                combined_dataset = concatenate_datasets(output)
    
                passages_path =self.config.passages_path

                logger.info("saving the dataset with  ")
                #combined_dataset.save_to_disk('/hpc/gsir059/MY-Test/RAY/transformers/examples/research_projects/rag/MY-Passage')
                combined_dataset.save_to_disk(passages_path)
                logger.info("Add faiss index to the dataset that consist of embeddings") 

    
                embedding_dataset=combined_dataset
                index = faiss.IndexHNSWFlat(768, 128, faiss.METRIC_INNER_PRODUCT)
                embedding_dataset.add_faiss_index("embeddings", custom_index=index)

                embedding_dataset.get_index("embeddings").save(self.config.index_path)


add_faisis_index  gets very slow when doing it interatively   
 As the below code suggests, I want to run add_faisis_index in every nth interaction from the training loop. I have 7.2 million documents. Usually, it takes 2.5 hours (if I run an as a separate process similar to the script given in rag/use_own_knowleldge_dataset.py). Now, this takes usually 5hrs. Is this normal?   Any way to make this process faster? 

@lhoestq 

```
   def training_step(self, batch, batch_idx) -> Dict:

    
        if (not batch_idx==0) and (batch_idx%5==0):

            print("******************************************************")
            ctx_encoder=self.trainer.model.module.module.model.rag.ctx_encoder
            model_copy =type(ctx_encoder)(self.config_dpr) # get a new instance  #this will be load in the CPU
            model_copy.load_state_dict(ctx_encoder.state_dict()) # copy weights and stuff


            list_of_gpus = ['cuda:2','cuda:3']
            c_dir='/custom/cache/dir'

            kb_dataset = load_dataset("csv", data_files=[self.custom_config.csv_path], split="train", delimiter="\t", column_names=["title", "text"],cache_dir=c_dir) 

            print(kb_dataset)

      
            n=len(list_of_gpus) #nunber of dedicated GPUs
            kb_list=[kb_dataset.shard(n, i, contiguous=True) for i in range(n)]

            #kb_dataset.save_to_disk('/hpc/gsir059/MY-Test/RAY/transformers/examples/research_projects/rag/haha-dir')


            print(self.trainer.global_rank)
            dataset_shards = self.re_encode_kb(model_copy.to(device=list_of_gpus[self.trainer.global_rank]),kb_list[self.trainer.global_rank])
            output = [None for _ in list_of_gpus]

            #self.trainer.accelerator_connector.accelerator.barrier("embedding_process")
            dist.all_gather_object(output, dataset_shards)
            

            #This creation and re-initlaization of the new index
            if (self.trainer.global_rank==0):  #saving will be done in the main process 
           
                combined_dataset = concatenate_datasets(output)
    
                passages_path =self.config.passages_path

                logger.info("saving the dataset with  ")
                #combined_dataset.save_to_disk('/hpc/gsir059/MY-Test/RAY/transformers/examples/research_projects/rag/MY-Passage')
                combined_dataset.save_to_disk(passages_path)
                logger.info("Add faiss index to the dataset that consist of embeddings") 

    
                embedding_dataset=combined_dataset
                index = faiss.IndexHNSWFlat(768, 128, faiss.METRIC_INNER_PRODUCT)
                embedding_dataset.add_faiss_index("embeddings", custom_index=index)

                embedding_dataset.get_index("embeddings").save(self.config.index_path)

 
 @lhoestq  on a different note, I read about using Faiss-GPU, but the documentation says we should use it when the dataset has the ability to fit into the GPU memory. Although this might work, in the long-term this is not that practical for me.

https://github.com/matsui528/faiss_tips