Support streaming for pubmed
{'login': 'abhi-mosaic', 'id': 77638579, 'node_id': 'MDQ6VXNlcjc3NjM4NTc5', 'avatar_url': 'https://avatars.githubusercontent.com/u/77638579?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/abhi-mosaic', 'html_url': 'https://github.com/abhi-mosaic', 'followers_url': 'https://api.github.com/users/abhi-mosaic/followers', 'following_url': 'https://api.github.com/users/abhi-mosaic/following{/other_user}', 'gists_url': 'https://api.github.com/users/abhi-mosaic/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/abhi-mosaic/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/abhi-mosaic/subscriptions', 'organizations_url': 'https://api.github.com/users/abhi-mosaic/orgs', 'repos_url': 'https://api.github.com/users/abhi-mosaic/repos', 'events_url': 'https://api.github.com/users/abhi-mosaic/events{/privacy}', 'received_events_url': 'https://api.github.com/users/abhi-mosaic/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['@albertvillanova just FYI, since you were so helpful with the previous pubmed issue :) '
 "IIRC streaming from FTP is not fully tested yet, so I'm fine with switching to HTTPS for now, as long as the download speed/availability is great"
 '@albertvillanova Thanks for pointing me to the `ET` module replacement. It should look a lot cleaner now.\r\n\r\nUnfortunately I tried keeping the `ftp://` protocol but was seeing timeout errors? in streaming mode (below). I think the `https://` performance is not an issue, when I was profiling the `open(..) -> f.read() -> etree.fromstring(xml_str)` codepath, most of the time was spent in the XML parsing rather than the data download.\r\n\r\n\r\nError when using `ftp://`:\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File "/Users/abhinav/Documents/mosaicml/hf_datasets/venv/lib/python3.8/site-packages/fsspec/implementations/ftp.py", line 301, in _fetch_range\r\n    self.fs.ftp.retrbinary(\r\n  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 430, in retrbinary\r\n    callback(data)\r\n  File "/Users/abhinav/Documents/mosaicml/hf_datasets/venv/lib/python3.8/site-packages/fsspec/implementations/ftp.py", line 293, in callback\r\n    raise TransferDone\r\nfsspec.implementations.ftp.TransferDone\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File "test_pubmed_streaming.py", line 9, in <module>\r\n    print (next(iter(pubmed_train_streaming)))\r\n  File "/Users/abhinav/Documents/mosaicml/hf_datasets/abhi-datasets/src/datasets/iterable_dataset.py", line 365, in __iter__\r\n    for key, example in self._iter():\r\n  File "/Users/abhinav/Documents/mosaicml/hf_datasets/abhi-datasets/src/datasets/iterable_dataset.py", line 362, in _iter\r\n    yield from ex_iterable\r\n  File "/Users/abhinav/Documents/mosaicml/hf_datasets/abhi-datasets/src/datasets/iterable_dataset.py", line 79, in __iter__\r\n    yield from self.generate_examples_fn(**self.kwargs)\r\n  File "/Users/abhinav/.cache/huggingface/modules/datasets_modules/datasets/pubmed/af552ed918e2841e8427203530e3cfed3a8bc3213041d7853bea1ca67eec683d/pubmed.py", line 362, in _generate_examples\r\n    tree = ET.parse(filename)\r\n  File "/Users/abhinav/Documents/mosaicml/hf_datasets/abhi-datasets/src/datasets/streaming.py", line 65, in wrapper\r\n    return function(*args, use_auth_token=use_auth_token, **kwargs)\r\n  File "/Users/abhinav/Documents/mosaicml/hf_datasets/abhi-datasets/src/datasets/utils/streaming_download_manager.py", line 636, in xet_parse\r\n    return ET.parse(f, parser=parser)\r\n  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/xml/etree/ElementTree.py", line 1202, in parse\r\n    tree.parse(source, parser)\r\n  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/xml/etree/ElementTree.py", line 595, in parse\r\n    self._root = parser._parse_whole(source)\r\n  File "/Users/abhinav/Documents/mosaicml/hf_datasets/abhi-datasets/src/datasets/utils/streaming_download_manager.py", line 293, in read_with_retries\r\n    out = read(*args, **kwargs)\r\n  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/gzip.py", line 292, in read\r\n    return self._buffer.read(size)\r\n  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/_compression.py", line 68, in readinto\r\n    data = self.read(len(byte_view))\r\n  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/gzip.py", line 479, in read\r\n    if not self._read_gzip_header():\r\n  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/gzip.py", line 422, in _read_gzip_header\r\n    magic = self._fp.read(2)\r\n  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/gzip.py", line 96, in read\r\n    self.file.read(size-self._length+read)\r\n  File "/Users/abhinav/Documents/mosaicml/hf_datasets/venv/lib/python3.8/site-packages/fsspec/spec.py", line 1485, in read\r\n    out = self.cache._fetch(self.loc, self.loc + length)\r\n  File "/Users/abhinav/Documents/mosaicml/hf_datasets/venv/lib/python3.8/site-packages/fsspec/caching.py", line 153, in _fetch\r\n    self.cache = self.fetcher(start, end)  # new block replaces old\r\n  File "/Users/abhinav/Documents/mosaicml/hf_datasets/venv/lib/python3.8/site-packages/fsspec/implementations/ftp.py", line 311, in _fetch_range\r\n    self.fs.ftp.getmultiline()\r\n  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 224, in getmultiline\r\n    line = self.getline()\r\n  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ftplib.py", line 206, in getline\r\n    line = self.file.readline(self.maxline + 1)\r\n  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socket.py", line 669, in readinto\r\n    return self._sock.recv_into(b)\r\nsocket.timeout: timed out\r\n```']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/3740', 'html_url': 'https://github.com/huggingface/datasets/pull/3740', 'diff_url': 'https://github.com/huggingface/datasets/pull/3740.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/3740.patch', 'merged_at': datetime.datetime(2022, 2, 18, 14, 42, 13)}
This PR makes some minor changes to the `pubmed` dataset to allow for `streaming=True`. Fixes #3739.

Basically, I followed the C4 dataset which works in streaming mode as an example, and made the following changes:

* Change URL prefix from `ftp://` to `https://`
* Explicilty `open` the filename and pass the XML contents to `etree.fromstring(xml_str)`

The Github diff tool makes it look like the changes are larger than they are, sorry about that.

I tested locally and the `pubmed` dataset now works in both normal and streaming modes. There is some overhead at the start of each shard in streaming mode as building the XML tree online is quite slow (each pubmed .xml.gz file is ~20MB), but the overhead gets amortized over all the samples in the shard. On my laptop with a single CPU worker I am able to stream at about ~600 samples/s.
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/3740/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}