Update Open Subtitles corpus with original sentence IDs
{'login': 'Valahaar', 'id': 19476123, 'node_id': 'MDQ6VXNlcjE5NDc2MTIz', 'avatar_url': 'https://avatars.githubusercontent.com/u/19476123?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/Valahaar', 'html_url': 'https://github.com/Valahaar', 'followers_url': 'https://api.github.com/users/Valahaar/followers', 'following_url': 'https://api.github.com/users/Valahaar/following{/other_user}', 'gists_url': 'https://api.github.com/users/Valahaar/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/Valahaar/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/Valahaar/subscriptions', 'organizations_url': 'https://api.github.com/users/Valahaar/orgs', 'repos_url': 'https://api.github.com/users/Valahaar/repos', 'events_url': 'https://api.github.com/users/Valahaar/events{/privacy}', 'received_events_url': 'https://api.github.com/users/Valahaar/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892877, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODc3', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/good%20first%20issue', 'name': 'good first issue', 'color': '7057ff', 'default': True, 'description': 'Good for newcomers'}]
[]
["Hi ! You're right this can can useful.\r\nThis should be easy to add, so feel free to give it a try if you want to contribute :)\r\nI think we just need to add it to the _generate_examples method of the OpenSubtitles dataset builder [here](https://github.com/huggingface/datasets/blob/master/datasets/open_subtitles/open_subtitles.py#L103)"
 "Hey @lhoestq , absolutely yes! Just one question before I start implementing. The ids found in the zip file have this format: \r\n(the following is line `22497315` of the `ids` file of the `de-en` dump)\r\n\r\n\r\n`de/2017/7006210/7063319.xml.gz  en/2017/7006210/7050201.xml.gz  335     339 340` (every space is actually a tab, aside from the space between `339` and `340`)\r\n\r\n\r\nWhere filenames encode the information like this: `lang/year/imdb_id/opensubtitles_id.xml.gz` whereas the numbers correspond to the sentence ids which are linked together (i.e. sentence `335` of the German subtitle corresponds to lines `339` and `340` of the English file)\r\n\r\nThat being said, do you think I should stick to the raw sentence id (and replace the current sequential id) or should I include more detailed metadata (or both things maybe)?\r\n\r\nGoing with raw ID is surely simpler, but including `year`, `imdbId` and `subtitleId` should save space as they're just integers; besides, any operation (like filtering or grouping) will be much easier if users don't have to manually parse the ids every time.\r\nAs for the language-specific sentenceIds, what could be the best option? A list of integers or a comma-separated string?\r\n\r\n**Note:** I did not find any official information about this encoding, but it appears to check out:\r\nhttps://www.imdb.com/title/tt7006210/, https://www.opensubtitles.org/en/subtitles/7063319 and https://www.opensubtitles.org/en/subtitles/7050201 all link to the same episode, so I guess (I hope!) it's correct.\r\n\r\n"
 'I like the idea of having `year`, `imdbId` and `subtitleId` as columns for filtering for example.\r\nAnd for the `sentenceIds` a list of integers is fine.'
 'Thanks for improving it @Valahaar :) '
 'Something like this? (adapted from [here](https://github.com/huggingface/datasets/blob/master/datasets/open_subtitles/open_subtitles.py#L114))\r\n\r\n```python\r\nresult = (\r\n  sentence_counter,\r\n    {\r\n      "id": str(sentence_counter),\r\n      "meta": {\r\n        "year": year,\r\n        "imdbId": imdb_id,\r\n        "subtitleId": {l1: l1_sub_id, l2: l2_sub_id},\r\n        "sentenceIds": {l1: [... source_sids ...], l2: [... target_sids ...]},\r\n        # or maybe src/tgt? I\'d go with the first one for consistency with \'translation\'\r\n        "subtitleId": {"src": l1_sub_id, "tgt": l2_sub_id},\r\n        "sentenceIds": {"src": [... source_sids ...], "tgt": [... target_sids ...]},\r\n      },\r\n      "translation": {l1: x, l2: y},\r\n    },\r\n  )\r\n```\r\nOr at top level, avoiding nesting into \'meta\'?'
 'Merged in #1865, closing. Thanks :)']
Hi! It would be great if you could add the original sentence ids to [Open Subtitles](https://huggingface.co/datasets/open_subtitles).

I can think of two reasons: first, it's possible to gather sentences for an entire document (the original ids contain media id, subtitle file id and sentence id), therefore somewhat allowing for document-level machine translation (and other document-level stuff which could be cool to have); second, it's possible to have parallel sentences in multiple languages, as they share the same ids across bitexts.

I think I should tag @abhishekkrthakur as he's the one who added it in the first place.

Thanks!
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/1844/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}