Use passed --cache_dir for modules cache
{'login': 'albertvillanova', 'id': 8515462, 'node_id': 'MDQ6VXNlcjg1MTU0NjI=', 'avatar_url': 'https://avatars.githubusercontent.com/u/8515462?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/albertvillanova', 'html_url': 'https://github.com/albertvillanova', 'followers_url': 'https://api.github.com/users/albertvillanova/followers', 'following_url': 'https://api.github.com/users/albertvillanova/following{/other_user}', 'gists_url': 'https://api.github.com/users/albertvillanova/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/albertvillanova/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/albertvillanova/subscriptions', 'organizations_url': 'https://api.github.com/users/albertvillanova/orgs', 'repos_url': 'https://api.github.com/users/albertvillanova/repos', 'events_url': 'https://api.github.com/users/albertvillanova/events{/privacy}', 'received_events_url': 'https://api.github.com/users/albertvillanova/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['I have a question: why not using a tmp dir instead, like the DummyDataGeneratorDownloadManager does?'
 'Hi @lhoestq, I am trying to understand better the logic...\r\n\r\nWhy do we have a `dynamic_module_path` besides the modules cache path?\r\n```python\r\nDYNAMIC_MODULES_PATH = os.path.join(HF_MODULES_CACHE, "datasets_modules")\r\n```\r\nMoreover, 2 subdirectories (for datasets and for metrics) were created inside it:\r\n```python\r\nDATASETS_PATH = os.path.join(DYNAMIC_MODULES_PATH, "datasets")\r\nMETRICS_PATH = os.path.join(DYNAMIC_MODULES_PATH, "metrics")\r\n```'
 'Hi :) \r\nThe modules cache path is the path added to `sys.path`.\r\nTherefore inside we need to have a folder that is going to be a package: `datasets_modules`.\r\nThis package will contain dynamic modules, i.e. datasets and metrics modules added on-the-fly.\r\nThen we have two sub-modules `datasets_modules.datasets` and `datasets_modules.metrics`.\r\n\r\nMaybe we can make things more explicit in the code with some comments explaining the structure, and maybe better variable naming as well..\r\n\r\nAlso I wanted to say that I started to work on offline loading of modules in #1726 and actually it lead to do similar changes to what you did to control the path where modules are stored.'
 'Hi @lhoestq, I see...\r\n\r\nIndeed I was also creating a draft for test_load, to clarify the expected behavior... ;)\r\n\r\nSo, for the command line:\r\n```sh\r\npython datasets-cli test datasets/<my-dataset-folder> --save_infos --all_configs --cache_dir <my-cache-dir>\r\n```\r\nthe `cache_dir` argument refers to dataset cache dir. We do not have control over the modules cache dir, but we would like to have. And if I understand well, you suggest adding another argument `dynamic_module_path`. Am I right?'
 "> So, for the command line:\r\n> \r\n> ```shell\r\n> python datasets-cli test datasets/<my-dataset-folder> --save_infos --all_configs --cache_dir <my-cache-dir>\r\n> ```\r\n> \r\n> the `cache_dir` argument refers to dataset cache dir. We do not have control over the modules cache dir, but we would like to have. And if I understand well, you suggest adding another argument `dynamic_module_path`. Am I right?\r\n\r\nYes the cache_dir is used to download files and also so save the dataset arrow files.\r\nThis is indeed different from the path for dynamic modules.\r\n\r\nI suggested to have `dynamic_module_path` as a parameter but actually this is the parent directory `hf_modules_cache` that we would need (it's the one that is passed to `init_dynamic_modules ` that we need to add to `sys.path`).\r\n\r\nCurrently it's already possible to override it using the env variable `HF_MODULES_CACHE` but we can imagine having it as a parameter as well.\r\n\r\nThis way the user controls both the `cache_dir` and the `hf_modules_cache` which are the two places used by the library to read/write stuff.\r\n\r\n"
 "I think #1726 is going to be merged pretty soon. Maybe can work on this as soon as it's merged to avoid doing the same things twice and to avoid conflicts ?"
 'I agree. Indeed I took some of your code in one of my last commit, to try to implement the logic you described.']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/1369', 'html_url': 'https://github.com/huggingface/datasets/pull/1369', 'diff_url': 'https://github.com/huggingface/datasets/pull/1369.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/1369.patch', 'merged_at': None}
When passed `--cache_dir` arg:
```shell
python datasets-cli test datasets/<my-dataset-folder> --save_infos --all_configs --cache_dir <my-cache-dir>
```
it is not used for caching the modules, which are cached in the default location at `.cache/huggingface/modules`.

With this fix, the modules will be cached at `<my-cache-dir>/modules`.
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/1369/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}