region:us
---
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: validation
    path: data/validation-*
  - split: test
    path: data/test-*
dataset_info:
  features:
  - name: id
    dtype: string
  - name: question
    dtype: string
  - name: choices
    struct:
    - name: label
      sequence: string
    - name: text
      sequence: string
  - name: answerKey
    dtype: string
  splits:
  - name: train
    num_bytes: 462541
    num_examples: 1118
  - name: validation
    num_bytes: 128948
    num_examples: 298
  - name: test
    num_bytes: 491761
    num_examples: 1170
  download_size: 511280
  dataset_size: 1083250
---
Reference: https://huggingface.co/datasets/ai2_arc

# ARC-Challenge (Vietnamese translation version)

## Dataset Summary
A dataset of grade-school level, multiple-choice science questions, assembled to encourage research in advanced question-answering.

## Install

To install `lm-eval` from the github repository main branch, run:

```bash
git clone https://github.com/hieunguyen1053/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .
```

## Basic Usage

> **Note**: When reporting results from eval harness, please include the task versions (shown in `results["versions"]`) for reproducibility. This allows bug fixes to tasks while also ensuring that previously reported scores are reproducible. See the [Task Versioning](#task-versioning) section for more info.

### Hugging Face `transformers`

To evaluate a model hosted on the [HuggingFace Hub](https://huggingface.co/models) (e.g. vlsp-2023-vllm/hoa-1b4) on `ai2_arc_vi` you can use the following command:


```bash
python main.py \
    --model hf-causal \
    --model_args pretrained=vlsp-2023-vllm/hoa-1b4 \
    --tasks ai2_arc_vi \
    --num_fewshot 25 \
    --batch_size auto \
    --device cuda:0
```

Additional arguments can be provided to the model constructor using the `--model_args` flag. Most notably, this supports the common practice of using the `revisions` feature on the Hub to store partially trained checkpoints, or to specify the datatype for running a model:

```bash
python main.py \
    --model hf-causal \
    --model_args pretrained=vlsp-2023-vllm/hoa-1b4,revision=step100000,dtype="float" \
    --tasks ai2_arc_vi \
    --num_fewshot 25 \
    --batch_size auto \
    --device cuda:0
```

To evaluate models that are loaded via `AutoSeq2SeqLM` in Huggingface, you instead use `hf-seq2seq`. *To evaluate (causal) models across multiple GPUs, use `--model hf-causal-experimental`*

> **Warning**: Choosing the wrong model may result in erroneous outputs despite not erroring.