Loading a faiss index KeyError
Ok in that case HF should fix their misleading example at https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index 

I copy-pasted it here.

> When you are done with your queries you can save your index on disk:
> 
> ```python
> ds_with_embeddings.save_faiss_index('embeddings', 'my_index.faiss')
> ```
> Then reload it later:
> 
> ```python
> ds = load_dataset('crime_and_punish', split='train[:100]')
> ds.load_faiss_index('embeddings', 'my_index.faiss')
> ```
I've recently been testing out RAG and DPR embeddings, and I've run into an issue that is not apparent in the documentation.

The basic steps are:

1. Create a dataset (dataset1)
2. Create an embeddings column using DPR
3. Add a faiss index to the dataset
4. Save faiss index to a file
5. Create a new dataset (dataset2) with the same text and label information as dataset1
6. Try to load the faiss index from file to dataset2
7. Get `KeyError: "Column embeddings not in the dataset"`

I've made a colab notebook that should show exactly what I did. Please switch to GPU runtime; I didn't check on CPU.

https://colab.research.google.com/drive/1X0S9ZuZ8k0ybcoei4w7so6dS_WrABmIx?usp=sharing

Ubuntu Version
VERSION="18.04.5 LTS (Bionic Beaver)"

datasets==1.4.1
faiss==1.5.3
faiss-gpu==1.7.0
torch==1.8.0+cu101
transformers==4.3.3

NVIDIA-SMI 460.56
Driver Version: 460.32.03
CUDA Version: 11.2    
Tesla K80           

I was basically following the steps here: https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index

I included the exact code from the documentation at the end of the notebook to show that they don't work either.

Loading a faiss index KeyError 
 I've recently been testing out RAG and DPR embeddings, and I've run into an issue that is not apparent in the documentation.

The basic steps are:

1. Create a dataset (dataset1)
2. Create an embeddings column using DPR
3. Add a faiss index to the dataset
4. Save faiss index to a file
5. Create a new dataset (dataset2) with the same text and label information as dataset1
6. Try to load the faiss index from file to dataset2
7. Get `KeyError: "Column embeddings not in the dataset"`

I've made a colab notebook that should show exactly what I did. Please switch to GPU runtime; I didn't check on CPU.

https://colab.research.google.com/drive/1X0S9ZuZ8k0ybcoei4w7so6dS_WrABmIx?usp=sharing

Ubuntu Version
VERSION="18.04.5 LTS (Bionic Beaver)"

datasets==1.4.1
faiss==1.5.3
faiss-gpu==1.7.0
torch==1.8.0+cu101
transformers==4.3.3

NVIDIA-SMI 460.56
Driver Version: 460.32.03
CUDA Version: 11.2    
Tesla K80           

I was basically following the steps here: https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index

I included the exact code from the documentation at the end of the notebook to show that they don't work either.
 
 Ok in that case HF should fix their misleading example at https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index 

I copy-pasted it here.

> When you are done with your queries you can save your index on disk:
> 
> ```python
> ds_with_embeddings.save_faiss_index('embeddings', 'my_index.faiss')
> ```
> Then reload it later:
> 
> ```python
> ds = load_dataset('crime_and_punish', split='train[:100]')
> ds.load_faiss_index('embeddings', 'my_index.faiss')
> ```