Update Open Subtitles corpus with original sentence IDs
Hey @lhoestq , absolutely yes! Just one question before I start implementing. The ids found in the zip file have this format: 
(the following is line `22497315` of the `ids` file of the `de-en` dump)


`de/2017/7006210/7063319.xml.gz  en/2017/7006210/7050201.xml.gz  335     339 340` (every space is actually a tab, aside from the space between `339` and `340`)


Where filenames encode the information like this: `lang/year/imdb_id/opensubtitles_id.xml.gz` whereas the numbers correspond to the sentence ids which are linked together (i.e. sentence `335` of the German subtitle corresponds to lines `339` and `340` of the English file)

That being said, do you think I should stick to the raw sentence id (and replace the current sequential id) or should I include more detailed metadata (or both things maybe)?

Going with raw ID is surely simpler, but including `year`, `imdbId` and `subtitleId` should save space as they're just integers; besides, any operation (like filtering or grouping) will be much easier if users don't have to manually parse the ids every time.
As for the language-specific sentenceIds, what could be the best option? A list of integers or a comma-separated string?

**Note:** I did not find any official information about this encoding, but it appears to check out:
https://www.imdb.com/title/tt7006210/, https://www.opensubtitles.org/en/subtitles/7063319 and https://www.opensubtitles.org/en/subtitles/7050201 all link to the same episode, so I guess (I hope!) it's correct.


Hi! It would be great if you could add the original sentence ids to [Open Subtitles](https://huggingface.co/datasets/open_subtitles).

I can think of two reasons: first, it's possible to gather sentences for an entire document (the original ids contain media id, subtitle file id and sentence id), therefore somewhat allowing for document-level machine translation (and other document-level stuff which could be cool to have); second, it's possible to have parallel sentences in multiple languages, as they share the same ids across bitexts.

I think I should tag @abhishekkrthakur as he's the one who added it in the first place.

Thanks!
Update Open Subtitles corpus with original sentence IDs 
 Hi! It would be great if you could add the original sentence ids to [Open Subtitles](https://huggingface.co/datasets/open_subtitles).

I can think of two reasons: first, it's possible to gather sentences for an entire document (the original ids contain media id, subtitle file id and sentence id), therefore somewhat allowing for document-level machine translation (and other document-level stuff which could be cool to have); second, it's possible to have parallel sentences in multiple languages, as they share the same ids across bitexts.

I think I should tag @abhishekkrthakur as he's the one who added it in the first place.

Thanks! 
 Hey @lhoestq , absolutely yes! Just one question before I start implementing. The ids found in the zip file have this format: 
(the following is line `22497315` of the `ids` file of the `de-en` dump)


`de/2017/7006210/7063319.xml.gz  en/2017/7006210/7050201.xml.gz  335     339 340` (every space is actually a tab, aside from the space between `339` and `340`)


Where filenames encode the information like this: `lang/year/imdb_id/opensubtitles_id.xml.gz` whereas the numbers correspond to the sentence ids which are linked together (i.e. sentence `335` of the German subtitle corresponds to lines `339` and `340` of the English file)

That being said, do you think I should stick to the raw sentence id (and replace the current sequential id) or should I include more detailed metadata (or both things maybe)?

Going with raw ID is surely simpler, but including `year`, `imdbId` and `subtitleId` should save space as they're just integers; besides, any operation (like filtering or grouping) will be much easier if users don't have to manually parse the ids every time.
As for the language-specific sentenceIds, what could be the best option? A list of integers or a comma-separated string?

**Note:** I did not find any official information about this encoding, but it appears to check out:
https://www.imdb.com/title/tt7006210/, https://www.opensubtitles.org/en/subtitles/7063319 and https://www.opensubtitles.org/en/subtitles/7050201 all link to the same episode, so I guess (I hope!) it's correct.

