hsaest commited on
Commit
7244d87
1 Parent(s): 7ebfa1f

Update content.py

Browse files
Files changed (1) hide show
  1. content.py +11 -0
content.py CHANGED
@@ -43,6 +43,17 @@ Please refer to [this](https://huggingface.co/datasets/osunlp/TravelPlanner/reso
43
 
44
  Submission made by our team are labelled "TravelPlanner Team". Each submission will be automatically evaluated and scored based on the predefined metrics. You can then obtain the scores and download the detailed constraint pass rates after the evaluation.
45
 
 
 
 
 
 
 
 
 
 
 
 
46
  ## Show Your Results on Leaderborad
47
  If you are interested in showing your results on our leaderboard, we invite you to reach out to us. Please send an email to [us](mailto:[email protected]) including the following details: evaluation mode, fondation model, tool-use strategy, planning strategy, organization, and your paper link (if available), along with your submission files.
48
  """
 
43
 
44
  Submission made by our team are labelled "TravelPlanner Team". Each submission will be automatically evaluated and scored based on the predefined metrics. You can then obtain the scores and download the detailed constraint pass rates after the evaluation.
45
 
46
+ ## ⚠️Warnings
47
+
48
+ We release our evaluation scripts to foster innovation and aid the development of new methods. We encourage the use of evaluation feedback in training set, such as implementing reinforcement learning techniques, to enhance learning. However, we strictly prohibit any form of cheating in the validation and test sets to uphold the fairness and reliability of the benchmark's evaluation process. We reserve the right to disqualify results if we find any of the following violations:
49
+
50
+ 1. Reverse engineering of our dataset, which includes, but is not limited to:
51
+ - Converting our natural language queries in the test set to structured formats (e.g., JSON) for optimization and unauthorized evaluation.
52
+ - Deriving data point entries using the hard rules from our data construction process, without accessing the actual database.
53
+ - Other similar manipulations.
54
+ 2. Hard coding or explicitly writing evaluation cues into prompts by hand, such as direct hints of common sense, which contradicts our goals as it lacks generalizability and is limited to this specific benchmark.
55
+ 3. Any other human interference strategies that are tailored specifically to this benchmark but lack generalization capabilities.
56
+
57
  ## Show Your Results on Leaderborad
58
  If you are interested in showing your results on our leaderboard, we invite you to reach out to us. Please send an email to [us](mailto:[email protected]) including the following details: evaluation mode, fondation model, tool-use strategy, planning strategy, organization, and your paper link (if available), along with your submission files.
59
  """