[Question] How to move and reuse preprocessed dataset? 
{'login': 'AtmaHou', 'id': 15045402, 'node_id': 'MDQ6VXNlcjE1MDQ1NDAy', 'avatar_url': 'https://avatars.githubusercontent.com/u/15045402?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/AtmaHou', 'html_url': 'https://github.com/AtmaHou', 'followers_url': 'https://api.github.com/users/AtmaHou/followers', 'following_url': 'https://api.github.com/users/AtmaHou/following{/other_user}', 'gists_url': 'https://api.github.com/users/AtmaHou/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/AtmaHou/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/AtmaHou/subscriptions', 'organizations_url': 'https://api.github.com/users/AtmaHou/orgs', 'repos_url': 'https://api.github.com/users/AtmaHou/repos', 'events_url': 'https://api.github.com/users/AtmaHou/events{/privacy}', 'received_events_url': 'https://api.github.com/users/AtmaHou/received_events', 'type': 'User', 'site_admin': False}
[]
[]
@lhoestq @LysandreJik <s>Hi :) Can you share with us the code you used ?</s>

EDIT: from https://github.com/huggingface/transformers/issues/11665#issuecomment-838348291 I understand you're using the run_clm.py script. Can you share your logs ?
 Also note that for the caching to work, you must reuse the exact same parameters as in the first run. Did you change any parameter ? The `preprocessing_num_workers` should also stay the same > Also note that for the caching to work, you must reuse the exact same parameters as in the first run. Did you change any parameter ? The `preprocessing_num_workers` should also stay the same

I only changed the `preprocessing_num_workers` maybe it is the problem~ I will try again~
Hi, I am training a gpt-2 from scratch using run_clm.py.

I want to move and reuse the preprocessed dataset (It take 2 hour to preprocess),

I tried to :

copy path_to_cache_dir/datasets to new_cache_dir/datasets
set export HF_DATASETS_CACHE="new_cache_dir/"
but the program still re-preprocess the whole dataset without loading cache.

I also tried to torch.save(lm_datasets, fw), but the saved file is only 14M.

What is the proper way to do this?
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2345/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}