[Tensorflow] Use something else than `from_tensor_slices()`
{'login': 'astariul', 'id': 43774355, 'node_id': 'MDQ6VXNlcjQzNzc0MzU1', 'avatar_url': 'https://avatars.githubusercontent.com/u/43774355?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/astariul', 'html_url': 'https://github.com/astariul', 'followers_url': 'https://api.github.com/users/astariul/followers', 'following_url': 'https://api.github.com/users/astariul/following{/other_user}', 'gists_url': 'https://api.github.com/users/astariul/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/astariul/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/astariul/subscriptions', 'organizations_url': 'https://api.github.com/users/astariul/orgs', 'repos_url': 'https://api.github.com/users/astariul/repos', 'events_url': 'https://api.github.com/users/astariul/events{/privacy}', 'received_events_url': 'https://api.github.com/users/astariul/received_events', 'type': 'User', 'site_admin': False}
[]
{'login': 'lhoestq', 'id': 42851186, 'node_id': 'MDQ6VXNlcjQyODUxMTg2', 'avatar_url': 'https://avatars.githubusercontent.com/u/42851186?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/lhoestq', 'html_url': 'https://github.com/lhoestq', 'followers_url': 'https://api.github.com/users/lhoestq/followers', 'following_url': 'https://api.github.com/users/lhoestq/following{/other_user}', 'gists_url': 'https://api.github.com/users/lhoestq/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/lhoestq/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/lhoestq/subscriptions', 'organizations_url': 'https://api.github.com/users/lhoestq/orgs', 'repos_url': 'https://api.github.com/users/lhoestq/repos', 'events_url': 'https://api.github.com/users/lhoestq/events{/privacy}', 'received_events_url': 'https://api.github.com/users/lhoestq/received_events', 'type': 'User', 'site_admin': False}
[{'login': 'lhoestq', 'id': 42851186, 'node_id': 'MDQ6VXNlcjQyODUxMTg2', 'avatar_url': 'https://avatars.githubusercontent.com/u/42851186?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/lhoestq', 'html_url': 'https://github.com/lhoestq', 'followers_url': 'https://api.github.com/users/lhoestq/followers', 'following_url': 'https://api.github.com/users/lhoestq/following{/other_user}', 'gists_url': 'https://api.github.com/users/lhoestq/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/lhoestq/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/lhoestq/subscriptions', 'organizations_url': 'https://api.github.com/users/lhoestq/orgs', 'repos_url': 'https://api.github.com/users/lhoestq/repos', 'events_url': 'https://api.github.com/users/lhoestq/events{/privacy}', 'received_events_url': 'https://api.github.com/users/lhoestq/received_events', 'type': 'User', 'site_admin': False}]
["I guess we can use `tf.data.Dataset.from_generator` instead. I'll give it a try."
 'Is `tf.data.Dataset.from_generator` working on TPU ?'
 '`from_generator` is not working on TPU, I met the following error :\r\n\r\n```\r\nFile "/usr/local/lib/python3.6/contextlib.py", line 88, in __exit__\r\n    next(self.gen)\r\n  File "/home/usr/.venv/bart/lib/python3.6/site-packages/tensorflow_core/python/eager/context.py", line 1900, in execution_mode\r\n    executor_new.wait()\r\n  File "/home/usr/.venv/bart/lib/python3.6/site-packages/tensorflow_core/python/eager/executor.py", line 67, in wait\r\n    pywrap_tensorflow.TFE_ExecutorWaitForAllPendingNodes(self._handle)\r\ntensorflow.python.framework.errors_impl.NotFoundError: No registered \'PyFunc\' OpKernel for \'CPU\' devices compatible with node {{node PyFunc}}\r\n        .  Registered:  <no registered kernels>\r\n\r\n         [[PyFunc]]\r\n```\r\n\r\n---\r\n\r\n@lhoestq It seems you merged some changes that allow lazy-loading. **Can you give an example of how to use ?** Maybe the Colab notebook should be updated with this method as well.'
 'Could you send me the code you used to run create the dataset using `.from_generator` ? What version of tensorflow are you using ?'
 'I\'m using TF2.2\r\n\r\nHere is my code :\r\n```\r\nimport nlp\r\nfrom transformers import BartTokenizer\r\n\r\ntokenizer = BartTokenizer.from_pretrained(\'bart-large\')\r\n\r\ndef encode(sample):\r\n    article_inputs = tokenizer.encode_plus(sample["article"], max_length=tokenizer.model_max_length, pad_to_max_length=True)\r\n    summary_inputs = tokenizer.encode_plus(sample["highlights"], max_length=tokenizer.model_max_length, pad_to_max_length=True)\r\n\r\n    article_inputs.update({"lm_labels": summary_inputs[\'input_ids\']})\r\n    return article_inputs\r\n\r\ncnn_dm = nlp.load_dataset(\'cnn_dailymail\', \'3.0.0\', split=\'test\')\r\ncnn_dm = cnn_dm.map(encode)\r\n\r\ndef gen():\r\n    for sample in cnn_dm:\r\n        s = {}\r\n        s[\'input_ids\'] = sample[\'input_ids\']\r\n        s[\'attention_mask\'] = sample[\'attention_mask\']\r\n        s[\'lm_labels\'] = sample[\'lm_labels\']\r\n        yield s\r\n\r\ndataset = tf.data.Dataset.from_generator(gen, output_types={k: tf.int32 for k in [\'input_ids\', \'attention_mask\', \'lm_labels\']}, output_shapes={k: tf.TensorShape([tokenizer.model_max_length]) for k in [\'input_ids\', \'attention_mask\', \'lm_labels\']}\r\n```'
 "Apparently we'll have to wait for the next tensorflow release to use `.from_generator` and TPU. See https://github.com/tensorflow/tensorflow/issues/34346#issuecomment-598262489"
 'Fixed by https://github.com/huggingface/datasets/pull/339']
In the example notebook, the TF Dataset is built using `from_tensor_slices()` :

```python
columns = ['input_ids', 'token_type_ids', 'attention_mask', 'start_positions', 'end_positions']
train_tf_dataset.set_format(type='tensorflow', columns=columns)
features = {x: train_tf_dataset[x] for x in columns[:3]} 
labels = {"output_1": train_tf_dataset["start_positions"]}
labels["output_2"] = train_tf_dataset["end_positions"]
tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)
```

But according to [official tensorflow documentation](https://www.tensorflow.org/guide/data#consuming_numpy_arrays), this will load the entire dataset to memory.

**This defeats one purpose of this library, which is lazy loading.**

Is there any other way to load the `nlp` dataset into TF dataset lazily ?

---

For example, is it possible to use [Arrow dataset](https://www.tensorflow.org/io/api_docs/python/tfio/arrow/ArrowDataset) ? If yes, is there any code example ?
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/193/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}