region:us
BBH focuses on a suite of 23 challenging BIG-Bench tasks which we call BIG-Bench Hard (BBH). These are the task for which prior language model evaluations did not outperform the average human-rater. We find that applying chain-of-thought (CoT) prompting to BBH tasks enables PaLM to surpass the average humanrater performance on 10 of the 23 tasks, and Codex (code-davinci-002) to surpass the average human-rater performance on 17 of the 23 tasks. Since many tasks in BBH require multi-step reasoning, few-shot prompting without CoT, as done in the BIG-Bench evaluations (Srivastava et al., 2022), substantially underestimates the best performance and capabilities of language models, which is better captured via CoT prompting. As further analysis, we explore the interaction between CoT and model scale on BBH, finding that CoT enables emergent task performance on several BBH tasks with otherwise flat scaling curves.
@article{suzgun2022challenging,
  title={Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them},
  author={Suzgun, Mirac and Scales, Nathan and Sch{\"a}rli, Nathanael and Gehrmann, Sebastian and Tay, Yi and Chung, Hyung Won and Chowdhery, Aakanksha and Le, Quoc V and Chi, Ed H and Zhou, Denny and and Wei, Jason},
  journal={arXiv preprint arXiv:2210.09261},
  year={2022}
}
---
dataset_info:
- config_name: boolean_expressions
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 11790
    num_examples: 250
  download_size: 17172
  dataset_size: 11790
- config_name: causal_judgement
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 198021
    num_examples: 187
  download_size: 202943
  dataset_size: 198021
- config_name: date_understanding
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 54666
    num_examples: 250
  download_size: 61760
  dataset_size: 54666
- config_name: disambiguation_qa
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 78620
    num_examples: 250
  download_size: 85255
  dataset_size: 78620
- config_name: dyck_languages
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 38432
    num_examples: 250
  download_size: 43814
  dataset_size: 38432
- config_name: formal_fallacies
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 138224
    num_examples: 250
  download_size: 145562
  dataset_size: 138224
- config_name: geometric_shapes
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 68560
    num_examples: 250
  download_size: 77242
  dataset_size: 68560
- config_name: hyperbaton
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 38574
    num_examples: 250
  download_size: 44706
  dataset_size: 38574
- config_name: logical_deduction_five_objects
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 148595
    num_examples: 250
  download_size: 155477
  dataset_size: 148595
- config_name: logical_deduction_seven_objects
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 191022
    num_examples: 250
  download_size: 198404
  dataset_size: 191022
- config_name: logical_deduction_three_objects
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 105831
    num_examples: 250
  download_size: 112213
  dataset_size: 105831
- config_name: movie_recommendation
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 50985
    num_examples: 250
  download_size: 57684
  dataset_size: 50985
- config_name: multistep_arithmetic_two
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 12943
    num_examples: 250
  download_size: 18325
  dataset_size: 12943
- config_name: navigate
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 49031
    num_examples: 250
  download_size: 55163
  dataset_size: 49031
- config_name: object_counting
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 30508
    num_examples: 250
  download_size: 35890
  dataset_size: 30508
- config_name: penguins_in_a_table
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 70062
    num_examples: 146
  download_size: 74516
  dataset_size: 70062
- config_name: reasoning_about_colored_objects
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 89579
    num_examples: 250
  download_size: 98694
  dataset_size: 89579
- config_name: ruin_names
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 46537
    num_examples: 250
  download_size: 53178
  dataset_size: 46537
- config_name: salient_translation_error_detection
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 277110
    num_examples: 250
  download_size: 286443
  dataset_size: 277110
- config_name: snarks
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 38223
    num_examples: 178
  download_size: 42646
  dataset_size: 38223
- config_name: sports_understanding
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 22723
    num_examples: 250
  download_size: 28617
  dataset_size: 22723
- config_name: temporal_sequences
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 139546
    num_examples: 250
  download_size: 148176
  dataset_size: 139546
- config_name: tracking_shuffled_objects_five_objects
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 162590
    num_examples: 250
  download_size: 169722
  dataset_size: 162590
- config_name: tracking_shuffled_objects_seven_objects
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 207274
    num_examples: 250
  download_size: 214906
  dataset_size: 207274
- config_name: tracking_shuffled_objects_three_objects
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 122104
    num_examples: 250
  download_size: 128736
  dataset_size: 122104
- config_name: web_of_lies
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 47582
    num_examples: 250
  download_size: 52964
  dataset_size: 47582
- config_name: word_sorting
  features:
  - name: input
    dtype: string
  - name: target
    dtype: string
  splits:
  - name: test
    num_bytes: 60918
    num_examples: 250
  download_size: 66300
  dataset_size: 60918
---
# BIG-bench Hard dataset

homepage: https://github.com/suzgunmirac/BIG-Bench-Hard

```
@article{suzgun2022challenging,
  title={Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them},
  author={Suzgun, Mirac and Scales, Nathan and Sch{\"a}rli, Nathanael and Gehrmann, Sebastian and Tay, Yi and Chung, Hyung Won and Chowdhery, Aakanksha and Le, Quoc V and Chi, Ed H and Zhou, Denny and and Wei, Jason},
  journal={arXiv preprint arXiv:2210.09261},
  year={2022}
}
```