[WIP] Add TEDLIUM dataset
{'login': 'sanchit-gandhi', 'id': 93869735, 'node_id': 'U_kgDOBZhWpw', 'avatar_url': 'https://avatars.githubusercontent.com/u/93869735?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/sanchit-gandhi', 'html_url': 'https://github.com/sanchit-gandhi', 'followers_url': 'https://api.github.com/users/sanchit-gandhi/followers', 'following_url': 'https://api.github.com/users/sanchit-gandhi/following{/other_user}', 'gists_url': 'https://api.github.com/users/sanchit-gandhi/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/sanchit-gandhi/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/sanchit-gandhi/subscriptions', 'organizations_url': 'https://api.github.com/users/sanchit-gandhi/orgs', 'repos_url': 'https://api.github.com/users/sanchit-gandhi/repos', 'events_url': 'https://api.github.com/users/sanchit-gandhi/events{/privacy}', 'received_events_url': 'https://api.github.com/users/sanchit-gandhi/received_events', 'type': 'User', 'site_admin': False}
[{'id': 2067376369, 'node_id': 'MDU6TGFiZWwyMDY3Mzc2MzY5', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/dataset%20request', 'name': 'dataset request', 'color': 'e99695', 'default': False, 'description': 'Requesting to add a new dataset'}
 {'id': 2725241052, 'node_id': 'MDU6TGFiZWwyNzI1MjQxMDUy', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/speech', 'name': 'speech', 'color': 'd93f0b', 'default': False, 'description': ''}]
[]
['_The documentation is not available anymore as the PR was closed or merged._'
 '```python\r\nfrom datasets import load_dataset\r\n\r\nds = load_dataset(\'./datasets/tedlium\', \'release1\', cache_dir=\'/home/sanchitgandhi/cache\')\r\n```\r\n\r\n```\r\nDownloading and preparing dataset tedlium/release1 to /home/sanchitgandhi/cache/tedlium/release1/1.0.1/5a9fcb97b4b52d5a1c9dc7bde4b1d5994cd89c4a3425ea36c789bf6096fee4f0...\r\nTraceback (most recent call last):\r\n  File "<string>", line 1, in <module>\r\n  File "/home/sanchit_huggingface_co/datasets/src/datasets/load.py", line 1703, in load_dataset\r\n    builder_instance.download_and_prepare(\r\n  File "/home/sanchit_huggingface_co/datasets/src/datasets/builder.py", line 605, in download_and_prepare\r\n    self._download_and_prepare(\r\n  File "/home/sanchit_huggingface_co/datasets/src/datasets/builder.py", line 1240, in _download_and_prepare\r\n    raise MissingBeamOptions(\r\ndatasets.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/\r\nIf you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). \r\nExample of usage: \r\n        `load_dataset(\'tedlium\', \'release1\', beam_runner=\'DirectRunner\')`\r\n```\r\nSpecifying the `beam_runner=\'DirectRunner\'` works:\r\n```python\r\nfrom datasets import load_dataset\r\n\r\nds = load_dataset(\'./datasets/tedlium\', \'release1\', cache_dir=\'/home/sanchitgandhi/cache\', beam_runner=\'DirectRunner\')\r\n```'
 'Extra Python imports/Linux packages:\r\n```\r\npip install pydub\r\nsudo apt install ffmpeg\r\n```'
 "Script heavily inspired by the TF datasets script at: https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/audio/tedlium.py\r\n\r\nThe TF datasets script uses the module AudioSegment from the package `pydub` (https://github.com/jiaaro/pydub), which is used to to open the audio files (stored in .sph format):\r\nhttps://github.com/huggingface/datasets/blob/61bf6123634bf6e7c7287cd6097909eb26118c58/datasets/tedlium/tedlium.py#L167-L170\r\nThis package requires the pip install of `pydub` and the system installation of `ffmpeg`: https://github.com/jiaaro/pydub#installation\r\nIs it ok to use these packages? Or do we tend to avoid introducing additional dependencies?\r\n\r\nThe TF datasets script also uses `_build_pcollection`:\r\nhttps://github.com/huggingface/datasets/blob/8afbbb6fe66b40d05574e2e72e65e974c72ae769/datasets/tedlium/tedlium.py#L200-L206\r\nHowever, I was advised against using `beam` logic. Thus, I have reverted to generating the examples file-by-file: https://github.com/huggingface/datasets/blob/61bf6123634bf6e7c7287cd6097909eb26118c58/datasets/tedlium/tedlium.py#L112-L138\r\n\r\nI am now able to generate examples by running the `load_dataset` command:\r\n```python\r\nfrom datasets import load_dataset\r\n\r\nds = load_dataset('./datasets/tedlium', 'release1', cache_dir='/home/sanchitgandhi/cache')\r\n```\r\n\r\nHere, generating examples is **extremely** slow: it takes ~1 second per example, so ~60k seconds for the train set (~16 hours). Is there a way of paralleling this to make it faster?"
 '> This package requires the pip install of pydub and the system installation of ffmpeg: https://github.com/jiaaro/pydub#installation\r\nIs it ok to use these packages? Or do we tend to avoid introducing additional dependencies?\r\n\r\nIt\'s ok, windows users will have have a bad time but I\'m not sure we can do much about it.\r\n\r\n> Here, generating examples is extremely slow: it takes ~1 second per example, so ~60k seconds for the train set (~16 hours). Is there a way of paralleling this to make it faster?\r\n\r\nNot at the moment. For such cases we advise hosting the dataset ourselves in a processed format. The license doesn\'t allow this since the license is "NoDerivatives". Currently the only way to parallelize it is by keeping is as a beam dataset and let users pay Google Dataflow to process it (or use spark or whatever).'
 'Thanks for your super speedy reply @lhoestq!\r\n\r\nI’ve uploaded the script and README.md to the org here: https://huggingface.co/datasets/LIUM/tedlium\r\nIs any modification of the script required to be able to use it from the Hub? When I run:\r\n```python\r\nfrom datasets import load_dataset\r\n\r\ntedlium = load_dataset("LIUM/tedlium", "release1") # for Release 1\r\n```\r\nI get the following error:\r\n```\r\n---------------------------------------------------------------------------\r\nTypeError                                 Traceback (most recent call last)\r\nInput In [2], in <cell line: 1>()\r\n----> 1 load_dataset("LIUM/tedlium", "release1")\r\n\r\nFile ~/datasets/src/datasets/load.py:1676, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)\r\n   1673 ignore_verifications = ignore_verifications or save_infos\r\n   1675 # Create a dataset builder\r\n-> 1676 builder_instance = load_dataset_builder(\r\n   1677     path=path,\r\n   1678     name=name,\r\n   1679     data_dir=data_dir,\r\n   1680     data_files=data_files,\r\n   1681     cache_dir=cache_dir,\r\n   1682     features=features,\r\n   1683     download_config=download_config,\r\n   1684     download_mode=download_mode,\r\n   1685     revision=revision,\r\n   1686     use_auth_token=use_auth_token,\r\n   1687     **config_kwargs,\r\n   1688 )\r\n   1690 # Return iterable dataset in case of streaming\r\n   1691 if streaming:\r\n\r\nFile ~/datasets/src/datasets/load.py:1502, in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, **config_kwargs)\r\n   1500     download_config = download_config.copy() if download_config else DownloadConfig()\r\n   1501     download_config.use_auth_token = use_auth_token\r\n-> 1502 dataset_module = dataset_module_factory(\r\n   1503     path,\r\n   1504     revision=revision,\r\n   1505     download_config=download_config,\r\n   1506     download_mode=download_mode,\r\n   1507     data_dir=data_dir,\r\n   1508     data_files=data_files,\r\n   1509 )\r\n   1511 # Get dataset builder class from the processing script\r\n   1512 builder_cls = import_main_class(dataset_module.module_path)\r\n\r\nFile ~/datasets/src/datasets/load.py:1254, in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_dir, data_files, **download_kwargs)\r\n   1249             if isinstance(e1, FileNotFoundError):\r\n   1250                 raise FileNotFoundError(\r\n   1251                     f"Couldn\'t find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "\r\n   1252                     f"Couldn\'t find \'{path}\' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"\r\n   1253                 ) from None\r\n-> 1254             raise e1 from None\r\n   1255 else:\r\n   1256     raise FileNotFoundError(\r\n   1257         f"Couldn\'t find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory."\r\n   1258     )\r\n\r\nFile ~/datasets/src/datasets/load.py:1227, in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_dir, data_files, **download_kwargs)\r\n   1225         raise e\r\n   1226 if filename in [sibling.rfilename for sibling in dataset_info.siblings]:\r\n-> 1227     return HubDatasetModuleFactoryWithScript(\r\n   1228         path,\r\n   1229         revision=revision,\r\n   1230         download_config=download_config,\r\n   1231         download_mode=download_mode,\r\n   1232         dynamic_modules_path=dynamic_modules_path,\r\n   1233     ).get_module()\r\n   1234 else:\r\n   1235     return HubDatasetModuleFactoryWithoutScript(\r\n   1236         path,\r\n   1237         revision=revision,\r\n   (...)\r\n   1241         download_mode=download_mode,\r\n   1242     ).get_module()\r\n\r\nFile ~/datasets/src/datasets/load.py:940, in HubDatasetModuleFactoryWithScript.get_module(self)\r\n    938 def get_module(self) -> DatasetModule:\r\n    939     # get script and other files\r\n--> 940     local_path = self.download_loading_script()\r\n    941     dataset_infos_path = self.download_dataset_infos_file()\r\n    942     imports = get_imports(local_path)\r\n\r\nFile ~/datasets/src/datasets/load.py:918, in HubDatasetModuleFactoryWithScript.download_loading_script(self)\r\n    917 def download_loading_script(self) -> str:\r\n--> 918     file_path = hf_hub_url(path=self.name, name=self.name.split("/")[1] + ".py", revision=self.revision)\r\n    919     download_config = self.download_config.copy()\r\n    920     if download_config.download_desc is None:\r\n\r\nTypeError: hf_hub_url() got an unexpected keyword argument \'name\'\r\n```\r\n\r\nNote that I am able to load the dataset from the `datasets` repo with the following lines of code:\r\n```python\r\nfrom datasets import load_dataset\r\n\r\nds = load_dataset(\'./datasets/tedlium\', \'release1\', cache_dir=\'/home/sanchitgandhi/cache\')\r\n```'
 'What version of `datasets` do you have ?\r\nUpdating `datasets` should fix the error ;)\r\n'
 '> This package requires the pip install of pydub and the system installation of ffmpeg: https://github.com/jiaaro/pydub#installation\r\nIs it ok to use these packages? Or do we tend to avoid introducing additional dependencies?\r\n\r\n`soundfile`, which is a required audio dependency, should also work with `.sph` files, no?'
 "> `soundfile`, which is a required audio dependency, should also work with `.sph` files, no?\r\n\r\nAwesome, thanks for the pointer @mariosasko! Switched `pydub` to `soundfile`, and having specifying the `dtype` argument in `soundfile.read` as `np.int16`, the arrays match with those from `pydub` ✅\r\n\r\nI also did some heavy optimising of the script with the processing of the `.stm` and `.sph` files - it now runs 2000x faster than before, so there probably isn't a need to upload the data to the Hub @lhoestq. The total processing time is just ~2mins now 🚀\r\n"
 'TEDLIUM completed and uploaded to the HF Hub: https://huggingface.co/datasets/LIUM/tedlium'
 'Awesome !']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/4309', 'html_url': 'https://github.com/huggingface/datasets/pull/4309', 'diff_url': 'https://github.com/huggingface/datasets/pull/4309.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/4309.patch', 'merged_at': None}
Adds the TED-LIUM dataset https://www.tensorflow.org/datasets/catalog/tedlium#tedliumrelease3 

TODO:

- [x] Port `tedium.py` from TF datasets using `convert_dataset.sh` script
- [x] Make `load_dataset` work
- [ ] ~~Run `datasets-cli` command to generate `dataset_infos.json`~~
- [ ] ~~Create dummy data for continuous testing~~
- [ ] ~~Dummy data tests~~
- [ ] ~~Real data tests~~
- [ ] Create the metadata JSON
- [ ] Close PR and add directly to the Hub under LIUM org
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/4309/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}