region:us
---
dataset_info:
- config_name: default
  features:
  - name: slug
    dtype: string
  - name: text_chunk
    dtype: string
  - name: embedding
    sequence: float64
  splits:
  - name: train
    num_bytes: 17448677826
    num_examples: 3655376
  download_size: 14805980593
  dataset_size: 17448677826
- config_name: small
  features:
  - name: slug
    dtype: string
  - name: text_chunk
    dtype: string
  - name: embedding
    sequence: float32
  splits:
  - name: train
    num_bytes: 475656222.6698008
    num_examples: 99531
  - name: test
    num_bytes: 23459991.330199156
    num_examples: 4909
  download_size: 488406448
  dataset_size: 499116214.0
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
- config_name: small
  data_files:
  - split: train
    path: small/train-*
  - split: test
    path: small/test-*
---
# Dataset Card for "processed-old-with-embeddings"

## Dataset Description

- **Homepage:**
- **Repository:**
- **Paper:**
- **Leaderboard:**
- **Point of Contact:**

### Dataset Summary

Chunks of about 256 words split by whitespace and their embeddings computed with  the pretrained spacy model ["de_dep_news_trf"] (https://github.com/explosion/spacy-models/releases/tag/de_dep_news_trf-3.6.1).
The splits are created with respect to sentence boundaries parsed with the same model, sentences are concatenated if the result does not exceed max_words = 256, therefore the chunk length varies.

### Supported Tasks and Leaderboards

[More Information Needed]

### Languages

This dataset contains texts from the legal domain in German language. (German court decisions)

## Dataset Structure

[More Information Needed]

### Data Instances

{'slug': 'ag-pinneberg-2003-12-19-68-ii-9302-weg', 
'text_chunk': 'Die Berufung des Klägers gegen das am 23. April 2002 verkündete Urteil der 1. Zivilkammer des Landgerichts Wuppertal wird zurückgewiesen.\n\n Der Kläger trägt (...)',
'embedding': [-0.055155396461486816, -0.3904547095298767, -0.0033536632545292377, 0.8048776984214783, 0.30156993865966797, 0.5924882888793945, (...)]]}

### Data Fields
{
    'slug': data['slug'],
    'text_chunk': text,
    'embedding': embedding
}
### Data Splits

[More Information Needed]

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

This dataset contains texts from the legal domain in German language. (German court decisions)

### Citation Information

@inproceedings{10.1145/3383583.3398616,
author = {Ostendorff, Malte and Blume, Till and Ostendorff, Saskia},
title = {Towards an Open Platform for Legal Information},
year = {2020},
isbn = {9781450375856},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3383583.3398616},
doi = {10.1145/3383583.3398616},
booktitle = {Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020},
pages = {385–388},
numpages = {4},
keywords = {open data, open source, legal information system, legal data},
location = {Virtual Event, China},
series = {JCDL '20}
}