Possible cache miss in datasets
{'login': 'ofirzaf', 'id': 18296312, 'node_id': 'MDQ6VXNlcjE4Mjk2MzEy', 'avatar_url': 'https://avatars.githubusercontent.com/u/18296312?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/ofirzaf', 'html_url': 'https://github.com/ofirzaf', 'followers_url': 'https://api.github.com/users/ofirzaf/followers', 'following_url': 'https://api.github.com/users/ofirzaf/following{/other_user}', 'gists_url': 'https://api.github.com/users/ofirzaf/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/ofirzaf/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/ofirzaf/subscriptions', 'organizations_url': 'https://api.github.com/users/ofirzaf/orgs', 'repos_url': 'https://api.github.com/users/ofirzaf/repos', 'events_url': 'https://api.github.com/users/ofirzaf/events{/privacy}', 'received_events_url': 'https://api.github.com/users/ofirzaf/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["Thanks for reporting !\r\nI was able to reproduce thanks to your code and find the origin of the bug.\r\nThe cache was not reusing the same file because one object was not deterministic. It comes from a conversion from `set` to `list` in the `datasets.arrrow_dataset.transmit_format` function, where the resulting list would not always be in the same order and therefore the function that computes the hash used by the cache would not always return the same result.\r\nI'm opening a PR to fix this.\r\n\r\nAlso we plan to do a new release in the coming days so you can expect the fix to be available soon.\r\nNote that you can still specify `cache_file_name=` in the second `map()` call to name the cache file yourself if you want to."
 'Thanks for the fast reply, waiting for the fix :)\r\n\r\nI tried to use `cache_file_names` and wasn\'t sure how, I tried to give it the following:\r\n```\r\ntokenized_datasets = tokenized_datasets.map(\r\n    group_texts,\r\n    batched=True,\r\n    num_proc=60,\r\n    load_from_cache_file=True,\r\n    cache_file_names={k: f\'.cache/{str(k)}\' for k in tokenized_datasets}\r\n)\r\n```\r\n\r\nand got an error:\r\n```\r\nmultiprocess.pool.RemoteTraceback:\r\n"""\r\nTraceback (most recent call last):\r\n  File "/venv/lib/python3.6/site-packages/multiprocess/pool.py", line 119, in worker\r\n    result = (True, func(*args, **kwds))\r\n  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 157, in wrapper\r\n    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)\r\n  File "/venv/lib/python3.6/site-packages/datasets/fingerprint.py", line 163, in wrapper\r\n    out = func(self, *args, **kwargs)\r\n  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1491, in _map_single\r\n    tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)\r\n  File "/usr/lib/python3.6/tempfile.py", line 690, in NamedTemporaryFile\r\n    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)\r\n  File "/usr/lib/python3.6/tempfile.py", line 401, in _mkstemp_inner\r\n    fd = _os.open(file, flags, 0o600)\r\nFileNotFoundError: [Errno 2] No such file or directory: \'_00000_of_00060.cache/tmpsvszxtop\'\r\n"""\r\n\r\nThe above exception was the direct cause of the following exception:\r\n\r\nTraceback (most recent call last):\r\n  File "test.py", line 48, in <module>\r\n    cache_file_names={k: f\'.cache/{str(k)}\' for k in tokenized_datasets}\r\n  File "/venv/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in map\r\n    for k, dataset in self.items()\r\n  File "/venv/lib/python3.6/site-packages/datasets/dataset_dict.py", line 303, in <dictcomp>\r\n    for k, dataset in self.items()\r\n  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1317, in map\r\n    transformed_shards = [r.get() for r in results]\r\n  File "/venv/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1317, in <listcomp>\r\n    transformed_shards = [r.get() for r in results]\r\n  File "/venv/lib/python3.6/site-packages/multiprocess/pool.py", line 644, in get\r\n    raise self._value\r\nFileNotFoundError: [Errno 2] No such file or directory: \'_00000_of_00060.cache/tmpsvszxtop\'\r\n```\r\n'
 "The documentation says\r\n```\r\ncache_file_names (`Optional[Dict[str, str]]`, defaults to `None`): Provide the name of a cache file to use to store the\r\n    results of the computation instead of the automatically generated cache file name.\r\n    You have to provide one :obj:`cache_file_name` per dataset in the dataset dictionary.\r\n```\r\nWhat is expected is simply the name of a file, not a path. The file will be located in the cache directory of the `wikitext` dataset. You can try again with something like\r\n```python\r\ncache_file_names = {k: f'tokenized_and_grouped_{str(k)}' for k in tokenized_datasets}\r\n```"
 "Managed to get `cache_file_names` working and caching works well with it\r\nHad to make a small modification for it to work:\r\n```\r\ncache_file_names = {k: f'tokenized_and_grouped_{str(k)}.arrow' for k in tokenized_datasets}\r\n```"
 "Another comment on `cache_file_names`, it doesn't save the produced cached files in the dataset's cache folder, it requires to give a path to an existing directory for it to work.\r\nI can confirm that this is how it works in `datasets==1.1.3`"
 'Oh yes indeed ! Maybe we need to update the docstring to mention that it is a path'
 'I fixed the docstring. Hopefully this is less confusing now: https://github.com/huggingface/datasets/commit/42ccc0012ba8864e6db1392430100f350236183a'
 'I upgraded to the latest version and I encountered some strange behaviour, the script I posted in the OP doesn\'t trigger recalculation, however, if I add the following change it does trigger partial recalculation, I am not sure if its something wrong on my machine or a bug:\r\n```\r\nfrom datasets import load_dataset\r\nfrom transformers import AutoTokenizer\r\n\r\ndatasets = load_dataset(\'wikitext\', \'wikitext-103-raw-v1\')\r\ntokenizer = AutoTokenizer.from_pretrained(\'bert-base-uncased\', use_fast=True)\r\n\r\ncolumn_names = datasets["train"].column_names\r\ntext_column_name = "text" if "text" in column_names else column_names[0]\r\ndef tokenize_function(examples):\r\n    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)\r\n# CHANGE\r\nprint(\'hello\')\r\n# CHANGE\r\n\r\ntokenized_datasets = datasets.map(\r\n    tokenize_function,\r\n    batched=True,\r\n...\r\n```\r\nI am using datasets in the `run_mlm.py` script in the transformers examples and I found that if I change the script without touching any of the preprocessing. it still triggers recalculation which is very weird\r\n\r\nEdit: accidently clicked the close issue button '
 "This is because the `group_texts` line definition changes (it is defined 3 lines later than in the previous call). Currently if a function is moved elsewhere in a script we consider it to be different.\r\n\r\nNot sure this is actually a good idea to keep this behavior though. We had this as a security in the early development of the lib but now the recursive hashing of objects is robust so we can probably remove that.\r\nMoreover we're already ignoring the line definition for lambda functions."
 'I opened a PR to change this, let me know what you think.'
 'Sounds great, thank you for your quick responses and help! Looking forward for the next release.'
 'I am having a similar issue where only the grouped files are loaded from cache while the tokenized ones aren\'t. I can confirm both datasets are being stored to file, but only the grouped version is loaded from cache. Not sure what might be going on. But I\'ve tried to remove all kinds of non deterministic behaviour, but still no luck. Thanks for the help!\r\n\r\n\r\n```python\r\n    # Datasets\r\n    train = sorted(glob(args.data_dir + \'*.{}\'.format(args.ext)))\r\n    if args.dev_split >= len(train):\r\n        raise ValueError("Not enough dev files")\r\n    dev = []\r\n    state = random.Random(1001)\r\n    for _ in range(args.dev_split):\r\n        dev.append(train.pop(state.randint(0, len(train) - 1)))\r\n\r\n    max_seq_length = min(args.max_seq_length, tokenizer.model_max_length)\r\n\r\n    def tokenize_function(examples):\r\n        return tokenizer(examples[\'text\'], return_special_tokens_mask=True)\r\n\r\n    def group_texts(examples):\r\n        # Concatenate all texts from our dataset and generate chunks of max_seq_length\r\n        concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}\r\n        total_length = len(concatenated_examples[list(examples.keys())[0]])\r\n        # Truncate (not implementing padding)\r\n        total_length = (total_length // max_seq_length) * max_seq_length\r\n        # Split by chunks of max_seq_length\r\n        result = {\r\n            k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)]\r\n            for k, t in concatenated_examples.items()\r\n        }\r\n        return result\r\n\r\n    datasets = load_dataset(\r\n        \'text\', name=\'DBNL\', data_files={\'train\': train[:10], \'dev\': dev[:5]}, \r\n        cache_dir=args.data_cache_dir)\r\n    datasets = datasets.map(tokenize_function, \r\n        batched=True, remove_columns=[\'text\'], \r\n        cache_file_names={k: os.path.join(args.data_cache_dir, f\'{k}-tokenized\') for k in datasets},\r\n        load_from_cache_file=not args.overwrite_cache)\r\n    datasets = datasets.map(group_texts, \r\n        batched=True,\r\n        cache_file_names={k: os.path.join(args.data_cache_dir, f\'{k}-grouped\') for k in datasets},\r\n        load_from_cache_file=not args.overwrite_cache)\r\n```\r\n\r\nAnd this is the log\r\n\r\n```\r\n04/26/2021 10:26:59 - WARNING - datasets.builder -   Using custom data configuration DBNL-f8d988ad33ccf2c1\r\n04/26/2021 10:26:59 - WARNING - datasets.builder -   Reusing dataset text (/home/manjavacasema/data/.cache/text/DBNL-f8d988ad33ccf2c1/0.0.0/e16f44aa1b321ece1f87b07977cc5d70be93d69b20486d6dacd62e12cf25c9a5)\r\n100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 21.07ba/s]\r\n100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 40/40 [00:01<00:00, 24.28ba/s]\r\n04/26/2021 10:27:01 - WARNING - datasets.arrow_dataset -   Loading cached processed dataset at /home/manjavacasema/data/.cache/train-grouped\r\n04/26/2021 10:27:01 - WARNING - datasets.arrow_dataset -   Loading cached processed dataset at /home/manjavacasema/data/.cache/dev-grouped\r\n```\r\n'
 'Hi ! What tokenizer are you using ?' "It's the ByteLevelBPETokenizer"
 'This error happened to me too, when I tried to supply my own fingerprint to `map()` via the `new_fingerprint` arg.\r\n\r\nEdit: realized it was because my path was weird and had colons and brackets and slashes in it, since one of the variable values I included in the fingerprint was a dataset split like "train[:10%]". I fixed it with [this solution](https://stackoverflow.com/a/13593932/2287177) from StackOverflow to just remove those invalid characters from the fingerprint.'
 "Good catch @jxmorris12, maybe we should do additional checks on the valid characters for fingerprints ! Would you like to contribute this ?\r\n\r\nI think this can be added here, when we set the fingerprint(s) that are passed `map`:\r\n\r\nhttps://github.com/huggingface/datasets/blob/25bb7c9cbf519fbbf9abf3898083b529e7762705/src/datasets/fingerprint.py#L449-L454\r\n\r\nmaybe something like\r\n```python\r\nif kwargs.get(fingerprint_name) is None:\r\n    ...\r\nelse:\r\n    # In this case, it's the user who specified the fingerprint manually:\r\n    # we need to make sure it's a valid hash\r\n    validate_fingerprint(kwargs[fingerprint_name])\r\n```\r\n\r\nOtherwise I can open a PR later"
 'I opened a PR here to add the fingerprint validation: https://github.com/huggingface/datasets/pull/4587\r\n\r\nEDIT: merged :)'
 'thank you!']
Hi,

I am using the datasets package and even though I run the same data processing functions, datasets always recomputes the function instead of using cache.
I have attached an example script that for me reproduces the problem.
In the attached example the second map function always recomputes instead of loading from cache.
Is this a bug or am I doing something wrong?
Is there a way for fix this and avoid all the recomputation?

Thanks

Edit:
transformers==3.5.1
datasets==1.2.0

```
from datasets import load_dataset
from transformers import AutoTokenizer

datasets = load_dataset('wikitext', 'wikitext-103-raw-v1')
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)


column_names = datasets["train"].column_names
text_column_name = "text" if "text" in column_names else column_names[0]
def tokenize_function(examples):
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)

tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
    num_proc=60,
    remove_columns=[text_column_name],
    load_from_cache_file=True,
)
max_seq_length = tokenizer.model_max_length
def group_texts(examples):
    # Concatenate all texts.
    concatenated_examples = {
        k: sum(examples[k], []) for k in examples.keys()}
    total_length = len(concatenated_examples[list(examples.keys())[0]])
    # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
    # customize this part to your needs.
    total_length = (total_length // max_seq_length) * max_seq_length
    # Split by chunks of max_len.
    result = {
        k: [t[i: i + max_seq_length]
            for i in range(0, total_length, max_seq_length)]
        for k, t in concatenated_examples.items()
    }
    return result

tokenized_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    num_proc=60,
    load_from_cache_file=True,
)
print(tokenized_datasets)

print('finished')
```
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/1718/reactions', 'total_count': 2, '+1': 2, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}