task_categories:translation task_categories:summarization size_categories:100K<n<1M language:no language:nb language:en license:cc region:us
---
dataset_info:
  features:
  - name: en
    dtype: string
  - name: 'no'
    dtype: string
  splits:
  - name: train
    num_bytes: 44628652
    num_examples: 758144
  download_size: 33446436
  dataset_size: 44628652
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
license: cc
task_categories:
- translation
- summarization
language:
- 'no'
- nb
- en
pretty_name: Massive EN-NO shorter transfer
size_categories:
- 100K<n<1M
---
# Massive EN-NO shorter and similar transfer

A dataset of EN-NO translations comprised of the following sources:

- https://huggingface.co/datasets/opus100
- https://huggingface.co/datasets/opus_books
- https://huggingface.co/datasets/open_subtitles (https://huggingface.co/datasets/tollefj/subtitles-en-no-similar-shorter)
- https://huggingface.co/datasets/RuterNorway/Fleurs-Alpaca-EN-NO

And parsed by:
- simple preprocessing: stripping/misplaced punctuation
- computing all similarities with https://huggingface.co/NbAiLab/nb-sbert-base
  - effectively aligning the translations
- filters out where the length of the target language (norwegian) is less than 70% the length of the source language (english)
  - items with less than 6 words are passed regardless of length constraints
 
this results in a shorter and similar translation corpus.