[Question & Bug Report] Can we preprocess a dataset on the fly?
{'login': 'shuaihuaiyi', 'id': 14048129, 'node_id': 'MDQ6VXNlcjE0MDQ4MTI5', 'avatar_url': 'https://avatars.githubusercontent.com/u/14048129?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/shuaihuaiyi', 'html_url': 'https://github.com/shuaihuaiyi', 'followers_url': 'https://api.github.com/users/shuaihuaiyi/followers', 'following_url': 'https://api.github.com/users/shuaihuaiyi/following{/other_user}', 'gists_url': 'https://api.github.com/users/shuaihuaiyi/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/shuaihuaiyi/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/shuaihuaiyi/subscriptions', 'organizations_url': 'https://api.github.com/users/shuaihuaiyi/orgs', 'repos_url': 'https://api.github.com/users/shuaihuaiyi/repos', 'events_url': 'https://api.github.com/users/shuaihuaiyi/events{/privacy}', 'received_events_url': 'https://api.github.com/users/shuaihuaiyi/received_events', 'type': 'User', 'site_admin': False}
[]
[]
We are very actively working on this. How does your dataset look like in practice (number/size/type of files)? It's a text file with many lines (about 1B) of Chinese sentences. I use it to train language model using https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm_wwm.py Indeed I will submit a PR in a fez days to enable processing on-the-fly :)
This can be useful in language modeling for tokenization, padding etc.
 any update on this issue? ...really look forward to use it  Hi @acul3,

Please look at the discussion on a related Issue #1825. I think using `set_transform` after building from source should do. @gchhablani thank you so much

will try look at it
I know we can use `Datasets.map` to preprocess a dataset, but I'm using it with very large corpus which generates huge cache file (several TB cache from a 400 GB text file). I have no disk large enough to save it.  Can we preprocess a dataset on the fly without generating cache?

BTW, I tried raising `writer_batch_size`. Seems that argument doesn't have any effect when it's larger than `batch_size`, because you are saving all the batch instantly after it's processed. Please check the following code:

https://github.com/huggingface/datasets/blob/0281f9d881f3a55c89aeaa642f1ba23444b64083/src/datasets/arrow_dataset.py#L1532
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/1776/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}