How to load a dataset with load_from disk and save it again after doing transformations without changing the original? 
{'login': 'shamanez', 'id': 16892570, 'node_id': 'MDQ6VXNlcjE2ODkyNTcw', 'avatar_url': 'https://avatars.githubusercontent.com/u/16892570?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/shamanez', 'html_url': 'https://github.com/shamanez', 'followers_url': 'https://api.github.com/users/shamanez/followers', 'following_url': 'https://api.github.com/users/shamanez/following{/other_user}', 'gists_url': 'https://api.github.com/users/shamanez/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/shamanez/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/shamanez/subscriptions', 'organizations_url': 'https://api.github.com/users/shamanez/orgs', 'repos_url': 'https://api.github.com/users/shamanez/repos', 'events_url': 'https://api.github.com/users/shamanez/events{/privacy}', 'received_events_url': 'https://api.github.com/users/shamanez/received_events', 'type': 'User', 'site_admin': False}
[]
[]
Hi ! That looks like a bug, can you provide some code so that we can reproduce ?
It's not supposed to update the original dataset Hi, I experimented with RAG. 

Actually, you can run the [use_own_knowldge_dataset.py](https://github.com/shamanez/transformers/blob/rag-end-to-end-retrieval/examples/research_projects/rag/use_own_knowledge_dataset.py#L80). In the 80 you can save the dataset object to the disk with save_to_disk. Then in order to compute the embeddings in this use **load_from_disk**. 

Then finally save it. You can see the original dataset object (CSV after splitting also will be changed)

One more thing- when I save the dataset object with **save_to_disk** it name the arrow file with cache.... rather than using dataset. arrow.  Can you add a variable that we can feed a name to save_to_disk function? @lhoestq   I also found that cache in tmp directory gets updated after transformations. This is really problematic when using datasets interactively. Let's say we use the shards function to a dataset loaded with csv, atm when we do transformations to shards and combine them it updates the original csv cache.  I plan to update the save_to_disk method in #2025 so I can make sure the new save_to_disk doesn't corrupt your cache files.
But from your last message it looks like save_to_disk isn't the root cause right ? ok, one more thing. When we use save_to_disk there are two files other than .arrow. dataset_info.json and state.json. Sometimes most of the fields in the dataset_infor.json are null, especially when saving dataset objects. Anyways I think load_from_disk uses the arrow files mentioned in state.json right?  > Anyways I think load_from_disk uses the arrow files mentioned in state.json right?

Yes exactly Perfect.  For now, I am loading the dataset from CSV in my interactive process and will wait until you make the PR!
I am using the latest datasets library.  In my work, I first use **load_from_disk** to load a data set that contains 3.8Gb information. Then during my training process, I update that dataset object and add new elements and save it in a different place.  

When I save the dataset with **save_to_disk**, the original dataset which is already in the disk also gets updated. I do not want to update it.  How to prevent from this?

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/1993/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}