Optimize int precision
{'login': 'albertvillanova', 'id': 8515462, 'node_id': 'MDQ6VXNlcjg1MTU0NjI=', 'avatar_url': 'https://avatars.githubusercontent.com/u/8515462?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/albertvillanova', 'html_url': 'https://github.com/albertvillanova', 'followers_url': 'https://api.github.com/users/albertvillanova/followers', 'following_url': 'https://api.github.com/users/albertvillanova/following{/other_user}', 'gists_url': 'https://api.github.com/users/albertvillanova/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/albertvillanova/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/albertvillanova/subscriptions', 'organizations_url': 'https://api.github.com/users/albertvillanova/orgs', 'repos_url': 'https://api.github.com/users/albertvillanova/repos', 'events_url': 'https://api.github.com/users/albertvillanova/events{/privacy}', 'received_events_url': 'https://api.github.com/users/albertvillanova/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['@lhoestq, are the tests OK? Some other cases I missed? Do you agree with this approach?'
 'I just tested this and it works like a charm :) \r\n\r\nHowever tokenizing and then setting the format to "torch" to feed the tokens into a model doesn\'t seem to work anymore, since the pytorch tensors have the int32/int8 precisions instead of int64 that is required as model inputs.\r\n\r\nFor example:\r\n\r\n```python\r\nimport torch\r\nfrom datasets import Dataset\r\nfrom transformers import BertModel, BertTokenizer\r\n\r\ntorch.set_grad_enabled(False)\r\n\r\ntokenizer = BertTokenizer.from_pretrained("bert-base-uncased")\r\nmodel = BertModel.from_pretrained("bert-base-uncased")\r\n\r\ndataset = Dataset.from_dict({"text": ["hello there !"]})\r\ndataset = dataset.map(tokenizer, input_columns="text", remove_columns=dataset.column_names)\r\ndataset = dataset.with_format("torch")\r\n\r\nprint(dataset.features)\r\n# {\'attention_mask\': Sequence(feature=Value(dtype=\'int8\', id=None), length=-1, id=None),\r\n#  \'input_ids\': Sequence(feature=Value(dtype=\'int64\', id=None), length=-1, id=None),  # this should be int32 though\r\n#  \'token_type_ids\': Sequence(feature=Value(dtype=\'int8\', id=None), length=-1, id=None)}\r\n\r\nmodel(**dataset[:1])\r\n# RuntimeError: Expected tensor for argument #1 \'indices\' to have scalar type Long; but got torch.CharTensor instead (while checking arguments for embedding)\r\n\r\ndataset = dataset.with_format("torch", dtype=torch.int64)\r\n\r\nmodel(**dataset[:1])\r\n# works as expected\r\n```\r\n\r\nPinging @sgugger here to make sure we take the right decision here.\r\n\r\nDo we want the "torch" format to always return int64 ? Or does it have to keep the precision defined by the `dataset.features` \r\n and therefore we would need to specify "torch" with `dtype=torch.int64` ?'
 'From a user perspective, I think it\'s fine if the "torch" format converts all ints types to `torch.int64` by default since it\'s what the model will need almost all the time. I don\'t see a case where you would want to keep the low precision at the top of my head, and one can always write a custom transform for an edge case.'
 'Sounds good to me !\r\nFor consistency maybe we should make the float precision fixed as well (float32, I guess)'
 'Yes, that would be the one used by default.'
 'Do we have the same requirements for TensorFlow?'
 'Yes I we should do the same for tensorflow as well since tf models would have the same issue\r\n\r\nThanks for adding this :)'
 '@lhoestq I think this PR is ready... :)']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/1985', 'html_url': 'https://github.com/huggingface/datasets/pull/1985', 'diff_url': 'https://github.com/huggingface/datasets/pull/1985.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/1985.patch', 'merged_at': datetime.datetime(2021, 3, 16, 9, 44)}
Optimize int precision to reduce dataset file size.

Close #1973, close #1825, close #861.
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/1985/reactions', 'total_count': 3, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 3, 'rocket': 0, 'eyes': 0}