region:us
---
dataset_info:
  features:
  - name: text
    dtype: string
  - name: context
    dtype: string
  - name: target_word
    dtype: string
  - name: metadata
    struct:
    - name: num_sents
      dtype: int64
    - name: target_word
      struct:
      - name: appeared_in_prev_sents
        dtype: bool
      - name: pos_tag
        dtype: string
    - name: title
      dtype: string
    - name: url
      dtype: string
    - name: word_type
      dtype: string
  splits:
  - name: test
    num_bytes: 16785187.523290113
    num_examples: 10246
  download_size: 10602825
  dataset_size: 16785187.523290113
configs:
- config_name: default
  data_files:
  - split: test
    path: data/test-*
---
# Lambada (Vietnamese)

## Install

To install `lm-eval` from the github repository main branch, run:

```bash
git clone https://github.com/hieunguyen1053/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .
```

## Basic Usage

> **Note**: When reporting results from eval harness, please include the task versions (shown in `results["versions"]`) for reproducibility. This allows bug fixes to tasks while also ensuring that previously reported scores are reproducible. See the [Task Versioning](#task-versioning) section for more info.

### Hugging Face `transformers`

To evaluate a model hosted on the [HuggingFace Hub](https://huggingface.co/models) (e.g. vlsp-2023-vllm/hoa-1b4) on `lambada_vi` you can use the following command:


```bash
python main.py \
    --model hf-causal \
    --model_args pretrained=vlsp-2023-vllm/hoa-1b4 \
    --tasks lambada_vi \
    --device cuda:0
```

Additional arguments can be provided to the model constructor using the `--model_args` flag. Most notably, this supports the common practice of using the `revisions` feature on the Hub to store partially trained checkpoints, or to specify the datatype for running a model:

```bash
python main.py \
    --model hf-causal \
    --model_args pretrained=vlsp-2023-vllm/hoa-1b4,revision=step100000,dtype="float" \
    --tasks lambada_vi \
    --device cuda:0
```

To evaluate models that are loaded via `AutoSeq2SeqLM` in Huggingface, you instead use `hf-seq2seq`. *To evaluate (causal) models across multiple GPUs, use `--model hf-causal-experimental`*

> **Warning**: Choosing the wrong model may result in erroneous outputs despite not erroring.