arxiv:2306.05301 arxiv:2307.16789 region:us
---
dataset_info:
- config_name: outputs_in_toolllm_format
  features:
  - name: response
    list:
    - name: function_call
      dtype: string
    - name: query
      dtype: string
    - name: task_id
      dtype: int64
    - name: timestamp
      dtype: float64
  splits:
  - name: train
    num_bytes: 303376
    num_examples: 348
  download_size: 83053
  dataset_size: 303376
- config_name: raw_api_list
  features:
  - name: dataset
    dtype: string
  - name: name
    dtype: string
  - name: description
    dtype: string
  - name: args_dicts
    list:
    - name: default
      dtype: 'null'
    - name: description
      dtype: string
    - name: name
      dtype: string
    - name: required
      dtype: bool
    - name: type
      dtype: string
  splits:
  - name: train
    num_bytes: 22276
    num_examples: 2
  download_size: 10949
  dataset_size: 22276
- config_name: raw_queries
  features:
  - name: dataset
    dtype: string
  - name: query_dict
    dtype: string
  splits:
  - name: train
    num_bytes: 466227
    num_examples: 339
  download_size: 98527
  dataset_size: 466227
- config_name: standardized_api_list
  features:
  - name: dataset
    dtype: string
  - name: name
    dtype: string
  - name: description
    dtype: string
  - name: args_dicts
    list:
    - name: default
      dtype: string
    - name: description
      dtype: string
    - name: name
      dtype: string
    - name: required
      dtype: bool
    - name: type
      dtype: string
  splits:
  - name: train
    num_bytes: 47776
    num_examples: 65
  download_size: 27751
  dataset_size: 47776
- config_name: standardized_queries
  features:
  - name: dataset
    dtype: string
  - name: prompt
    dtype: string
  - name: python_function_name
    dtype: string
  - name: python_args_dict
    dtype: string
  - name: context_functions
    sequence: string
  splits:
  - name: train
    num_bytes: 153860
    num_examples: 318
  download_size: 36721
  dataset_size: 153860
configs:
- config_name: outputs_in_toolllm_format
  data_files:
  - split: train
    path: outputs_in_toolllm_format/train-*
- config_name: raw_queries
  data_files:
  - split: train
    path: raw_queries/train-*
- config_name: standardized_api_list
  data_files:
  - split: train
    path: standardized_api_list/train-*
- config_name: standardized_queries
  data_files:
  - split: train
    path: standardized_queries/train-*
---

# NexusRaven API Evaluation dataset
Please see [blog post](http://nexusflow.ai/blog) or [NexusRaven Github repo](https://github.com/nexusflowai/NexusRaven) for more information. 


## License
The evaluation data in this repository consists primarily of our own curated evaluation data that only uses open source commercializable models. However, we include general domain data from the ToolLLM and ToolAlpaca papers. Since the data in the ToolLLM and ToolAlpaca works use OpenAI's GPT models for the generated content, the data is not commercially licensable, even if our own data is. As a result, the evaluation data used here is strictly non-commercial under [CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/). Thank you for understanding!


## References
We thank the following authors and entities for their evaluation data, which we leveraged to produce the results contained in this repository. Their citations can be found below

1. ToolAlpaca team
2. ToolLLM team

```
@misc{tang2023toolalpaca,
      title={ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases}, 
      author={Qiaoyu Tang and Ziliang Deng and Hongyu Lin and Xianpei Han and Qiao Liang and Boxi Cao and Le Sun},
      year={2023},
      eprint={2306.05301},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@misc{qin2023toolllm,
      title={ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs}, 
      author={Yujia Qin and Shihao Liang and Yining Ye and Kunlun Zhu and Lan Yan and Yaxi Lu and Yankai Lin and Xin Cong and Xiangru Tang and Bill Qian and Sihan Zhao and Runchu Tian and Ruobing Xie and Jie Zhou and Mark Gerstein and Dahai Li and Zhiyuan Liu and Maosong Sun},
      year={2023},
      eprint={2307.16789},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}
```


## Citation
```
@misc{nexusraven,
      title={NexusRaven: Surpassing the state-of-the-art in open-source function calling LLMs}, 
      author={Nexusflow.ai team},
      year={2023},
      url={http://nexusflow.ai/blog}
}
```

## Contact
Please reach out to info@nexusflow.ai for any questions!
