task_categories:translation task_categories:summarization task_categories:conversational size_categories:100M<n<1B language:en language:no region:us
---
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: test
    path: data/test-*
  - split: validation
    path: data/validation-*
dataset_info:
  features:
  - name: en
    dtype: string
  - name: 'no'
    dtype: string
  splits:
  - name: train
    num_bytes: 16980310.587072924
    num_examples: 253356
  - name: test
    num_bytes: 2183226.832496173
    num_examples: 32575
  - name: validation
    num_bytes: 5094106.580430903
    num_examples: 76007
  download_size: 17542044
  dataset_size: 24257644
task_categories:
- translation
- summarization
- conversational
language:
- en
- 'no'
pretty_name: Open Subtitles (EN-NO), similar and shorter texts
size_categories:
- 100M<n<1B
---
# Open subtitles (EN-NO): similar and shorter parallels

361938 samples of subtitles/sentences where the Norwegian parallel is a maximum of 60% of the size of the English text.
Additionally, the sentences are filtered on similarity scores with a [Norwegian SBERT model](https://huggingface.co/NbAiLab/nb-sbert-base), with a minimum of 0.7 similarity (score based on observations from 10k samples).
The purpose of this dataset is to identify how generative models perform joint operations on translation and reduction of sentences.


## Splits:
|   split   |  rows |
|:----------|------:|
|   train   | 253356 |
|   test    |  32575 |
| validation|  76007 |