fix: files counted twice in inferred structure
{'login': 'borisdayma', 'id': 715491, 'node_id': 'MDQ6VXNlcjcxNTQ5MQ==', 'avatar_url': 'https://avatars.githubusercontent.com/u/715491?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/borisdayma', 'html_url': 'https://github.com/borisdayma', 'followers_url': 'https://api.github.com/users/borisdayma/followers', 'following_url': 'https://api.github.com/users/borisdayma/following{/other_user}', 'gists_url': 'https://api.github.com/users/borisdayma/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/borisdayma/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/borisdayma/subscriptions', 'organizations_url': 'https://api.github.com/users/borisdayma/orgs', 'repos_url': 'https://api.github.com/users/borisdayma/repos', 'events_url': 'https://api.github.com/users/borisdayma/events{/privacy}', 'received_events_url': 'https://api.github.com/users/borisdayma/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['I see it creates some errors in the tests.\r\n\r\nAnother solution if needed is to add something like `data_files = list(set(data_files))` after [this line](https://github.com/huggingface/datasets/blob/8555197a3fe826e98bd0206c2d031c4488c53c5c/src/datasets/data_files.py#L511)'
 'Hi ! Thanks for the correction :)\r\n\r\nYour change seems right, let me look at the errors and try to fix this'
 "Not sure if it's due to this change but I\xa0tested `load_dataset('dalle-mini/encoded-vqgan_imagenet_f16_16384', streaming=True)` and the `validation` set is empty."
 "So indeed there was an issue with the patterns `*` and `**/*` that would return some files twice. This issue came from the fact that we were not using the right `glob`.\r\n\r\nIndeed we were using `Path.rglob` for local files and `Path.match` for remote files. Since these two methods don't have the same behavior for such patterns, I decided to change that.\r\n\r\nIn particular, we now use `glob.glob` (same as `fsspec` glob) as a reference for data files resolution from patterns. This is the same as dask for example.\r\n\r\n/!\\ Here are some behaviors specific to `glob.glob` that are different from Path.glob, Path.match or fnmatch:\r\n- '*' matches only first level files\r\n- '**/*' matches only at least second level files\r\n\r\nThis way we have a consistent behavior with respect to other python data libraries and there's no overlap anymore between the two patterns.\r\n\r\nSome implementations details:\r\n\r\nTo ensure that we have the same behavior for local files and for files in a remote dataset repository, I decided to use `fsspec` glob for both. This was made possible by implementing the `HfFileSystem` class as a `fsspec` filesystem.\r\n\r\nI pushed those changes directly to your PR - I hope you don't mind. I'm still fixing the remaining tests.\r\nPlease let me know if that solves your problem, and then we can merge !"
 "There's still an issue with fsspec's glob - I'll take a look this afternoon"
 "I just found out that actually glob.glob and fsspec glob are different haha\r\nglob.glob needs `**/*` and recursive=True to look into deep subdirectories, while fsspec only requires `**`\r\n\r\nI think we can go with fsspec glob for consistency with dask and since it's our main tool for filesystems management"
 "To recap:\r\n```\r\nWe use fsspec glob as a reference for data files resolution from patterns.\r\nThis is the same as dask for example.\r\n\r\n/!\\ Here are some behaviors specific to fsspec glob that are different from glob.glob, Path.glob, Path.match or fnmatch:\r\n- '*' matches only first level items\r\n- '**' matches all items\r\n- '**/*' matches all at least second level items\r\n\r\nMore generally:\r\n- `*`` matches any character except a forward-slash (to match just the file or directory name)\r\n- `**`` matches any character including a forward-slash /\r\n```"
 'lol Windows… Maybe `Pathlib` for the tests?\r\n\r\nI tested streaming a repo and it worked perfectly now!']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/3309', 'html_url': 'https://github.com/huggingface/datasets/pull/3309', 'diff_url': 'https://github.com/huggingface/datasets/pull/3309.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/3309.patch', 'merged_at': datetime.datetime(2021, 11, 23, 17, 0, 58)}
Files were counted twice in a structure like:
```
my_dataset_local_path/
├── README.md
└── data/
    ├── train/
    │   ├── shard_0.csv
    │   ├── shard_1.csv
    │   ├── shard_2.csv
    │   └── shard_3.csv
    └── valid/
        ├── shard_0.csv
        └── shard_1.csv
```

The reason is that they were matching both `*train*/*` and `*train*/**/*`.

This PR fixes it. @lhoestq 
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/3309/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}