Regarding On-the-fly Data Loading
{'login': 'gchhablani', 'id': 29076344, 'node_id': 'MDQ6VXNlcjI5MDc2MzQ0', 'avatar_url': 'https://avatars.githubusercontent.com/u/29076344?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/gchhablani', 'html_url': 'https://github.com/gchhablani', 'followers_url': 'https://api.github.com/users/gchhablani/followers', 'following_url': 'https://api.github.com/users/gchhablani/following{/other_user}', 'gists_url': 'https://api.github.com/users/gchhablani/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/gchhablani/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/gchhablani/subscriptions', 'organizations_url': 'https://api.github.com/users/gchhablani/orgs', 'repos_url': 'https://api.github.com/users/gchhablani/repos', 'events_url': 'https://api.github.com/users/gchhablani/events{/privacy}', 'received_events_url': 'https://api.github.com/users/gchhablani/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['Possible duplicate\r\n\r\n#1776 https://github.com/huggingface/datasets/issues/\r\n\r\nreally looking PR for this feature'
 'Hi @acul3 \r\n\r\nIssue #1776 talks about  doing on-the-fly data pre-processing, which I think is solved in the next release as mentioned in the issue #1825. I also look forward to using this feature, though :)\r\n\r\nI wanted to ask about on-the-fly data loading from the cache (before pre-processing).'
 "Hi ! Currently when you load a dataset via `load_dataset` for example, then the dataset is memory-mapped from an Arrow file on disk. Therefore there's almost no RAM usage even if your dataset contains TB of data.\r\nUsually at training time only one batch of data at a time is loaded in memory.\r\n\r\nDoes that answer your question or were you thinking about something else ?"
 'Hi @lhoestq,\r\n\r\nI apologize for the late response. This answers my question. Thanks a lot.']
Hi,

I was wondering if it is possible to load images/texts as a batch during the training process, without loading the entire dataset on the RAM at any given point.

Thanks,
Gunjan
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/1827/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}