region:us
---
dataset_info:
  features:
  - name: input_ids
    sequence: int64
  - name: attention_mask
    sequence: int64
  - name: labels
    sequence: int64
  splits:
  - name: train
    num_bytes: 245408208
    num_examples: 60000
  - name: test
    num_bytes: 20286240
    num_examples: 5000
  - name: validation
    num_bytes: 20286240
    num_examples: 5000
  download_size: 27474371
  dataset_size: 285980688
---
# Dataset Card for "marc-multilingual-encodings-v4"
# marc-code-mixed-small

These encodings are based on the review_tokens of [msislam/marc-code-mixed-small](https://huggingface.co/datasets/msislam/marc-code-mixed-small).

It contains German (DE), English (EN), Spanish (ES), and French (FR) languages. 

The labels are 0 (DE), 1 (EN), 2 (ES), and 3 (FR). 

Each review contains all four languages.

Total number of tokens:
* In training set: 10195342
* In test set: 842760
* In validation set: 842760

The encodings are created using the tokenizer [xlm-roberta-base](https://huggingface.co/xlm-roberta-base).