Loading a faiss index KeyError
{'login': 'nbroad1881', 'id': 24982805, 'node_id': 'MDQ6VXNlcjI0OTgyODA1', 'avatar_url': 'https://avatars.githubusercontent.com/u/24982805?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/nbroad1881', 'html_url': 'https://github.com/nbroad1881', 'followers_url': 'https://api.github.com/users/nbroad1881/followers', 'following_url': 'https://api.github.com/users/nbroad1881/following{/other_user}', 'gists_url': 'https://api.github.com/users/nbroad1881/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/nbroad1881/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/nbroad1881/subscriptions', 'organizations_url': 'https://api.github.com/users/nbroad1881/orgs', 'repos_url': 'https://api.github.com/users/nbroad1881/repos', 'events_url': 'https://api.github.com/users/nbroad1881/events{/privacy}', 'received_events_url': 'https://api.github.com/users/nbroad1881/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892861, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODYx', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/documentation', 'name': 'documentation', 'color': '0075ca', 'default': True, 'description': 'Improvements or additions to documentation'}]
[]
['In your code `dataset2` doesn\'t contain the "embeddings" column, since it is created from the pandas DataFrame with columns "text" and "label".\r\n\r\nTherefore when you call `dataset2[embeddings_name]`, you get a `KeyError`.\r\n\r\nIf you want the "embeddings" column back, you can create `dataset2` with\r\n```python\r\ndataset2 = load_from_disk(dataset_filename)\r\n```\r\nwhere `dataset_filename` is the place where you saved you dataset with the embeddings in the first place.'
 "Ok in that case HF should fix their misleading example at https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index \r\n\r\nI copy-pasted it here.\r\n\r\n> When you are done with your queries you can save your index on disk:\r\n> \r\n> ```python\r\n> ds_with_embeddings.save_faiss_index('embeddings', 'my_index.faiss')\r\n> ```\r\n> Then reload it later:\r\n> \r\n> ```python\r\n> ds = load_dataset('crime_and_punish', split='train[:100]')\r\n> ds.load_faiss_index('embeddings', 'my_index.faiss')\r\n> ```"
 'Hi !\r\n\r\nThe code of the example is valid.\r\nAn index is a search engine, it\'s not considered a column of a dataset.\r\nWhen you do `ds.load_faiss_index("embeddings", \'my_index.faiss\')`, it attaches an index named "embeddings" to the dataset but it doesn\'t re-add the "embeddings" column. You can list the indexes of a dataset by using `ds.list_indexes()`.\r\n\r\nIf I understand correctly by reading this example you thought that it was re-adding the "embeddings" column.\r\nThis looks misleading indeed, and we should add a note to make it more explicit that it doesn\'t store the column that was used to build the index.\r\n\r\nFeel free to open a PR to suggest an improvement on the documentation if you want to contribute :)'
 '> If I understand correctly by reading this example you thought that it was re-adding the "embeddings" column.\r\nYes. I was trying to use the dataset in RAG and it complained that the dataset didn\'t have the right columns. No problems when loading the dataset with `load_from_disk` and then doing `load_faiss_index`\r\n\r\nWhat I learned was\r\n1. column and index are different\r\n2. loading the index does not create a column\r\n3. the column is not needed to be able to use the index\r\n4. RAG needs both the embeddings column and the index\r\n\r\nIf I can come up with a way to articulate this in the right spot in the docs, I\'ll open a PR']
I've recently been testing out RAG and DPR embeddings, and I've run into an issue that is not apparent in the documentation.

The basic steps are:

1. Create a dataset (dataset1)
2. Create an embeddings column using DPR
3. Add a faiss index to the dataset
4. Save faiss index to a file
5. Create a new dataset (dataset2) with the same text and label information as dataset1
6. Try to load the faiss index from file to dataset2
7. Get `KeyError: "Column embeddings not in the dataset"`

I've made a colab notebook that should show exactly what I did. Please switch to GPU runtime; I didn't check on CPU.

https://colab.research.google.com/drive/1X0S9ZuZ8k0ybcoei4w7so6dS_WrABmIx?usp=sharing

Ubuntu Version
VERSION="18.04.5 LTS (Bionic Beaver)"

datasets==1.4.1
faiss==1.5.3
faiss-gpu==1.7.0
torch==1.8.0+cu101
transformers==4.3.3

NVIDIA-SMI 460.56
Driver Version: 460.32.03
CUDA Version: 11.2    
Tesla K80           

I was basically following the steps here: https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index

I included the exact code from the documentation at the end of the notebook to show that they don't work either.

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2029/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}