niansong1996 commited on
Commit
33939c1
1 Parent(s): ff684a0

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +174 -0
README.md CHANGED
@@ -36,3 +36,177 @@ Load the model and tokenizer as follows:
36
  tokenizer = T5Tokenizer.from_pretrained("Yale-LILY/lever-spider-codex")
37
  model = T5ForConditionalGeneration.from_pretrained("Yale-LILY/lever-spider-codex")
38
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
  tokenizer = T5Tokenizer.from_pretrained("Yale-LILY/lever-spider-codex")
37
  model = T5ForConditionalGeneration.from_pretrained("Yale-LILY/lever-spider-codex")
38
  ```
39
+
40
+ # Model Details
41
+
42
+
43
+ ## Model Description
44
+ The advent of pre-trained code language models (CodeLMs) has led to significant progress in language-to-code generation. State-of-the-art approaches in this area combine CodeLM decoding with sample pruning and reranking using test cases or heuristics based on the execution results. However, it is challenging to obtain test cases for many real-world language-to-code applications, and heuristics cannot well capture the semantic features of the execution results, such as data type and value range, which often indicates the correctness of the program. In this work, we propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the CodeLM is correct or not based on the natural language input, the program itself and its execution results. The sampled programs are reranked by combining the verification score with the CodeLM generation probability, and marginalizing over programs with the same execution results. On four datasets across the domains of table QA, math QA and basic Python programming, LEVER consistently improves over the base CodeLMs (4.6% to 10.9% with code-davinci-002) and achieves new state-of-the-art results on all of them.
45
+
46
+
47
+ - **Developed by:** Yale LILY Lab and Meta AI
48
+ - **Shared by [Optional]:** Yale LILY Lab
49
+
50
+ - **Model type:** Text Classification
51
+ - **Language(s) (NLP):** More information needed
52
+ - **License:** Apache-2.0
53
+ - **Parent Model:** T5-base
54
+ - **Resources for more information:**
55
+ - [Github Repo](https://github.com/niansong1996/lever)
56
+ - [Associated Paper](https://arxiv.org/abs/2302.08468)
57
+
58
+
59
+ # Uses
60
+
61
+
62
+ ## Direct Use
63
+
64
+ This model is *not* intended to be directly used. LEVER is used to verify and rerank the programs generated by code LLMs (e.g., Codex). We recommend checking out our [Github Repo](https://github.com/niansong1996/lever) for more details.
65
+
66
+ ## Downstream Use
67
+
68
+ LEVER is learned to verify and rerank the programs sampled from code LLMs for different tasks.
69
+ More specifically, for `lever-spider-codex`, it was trained on the outputs of `code-davinci-002` on the [Spider](https://yale-lily.github.io/spider) dataset. It can be used to rerank the SQL programs generated by Codex out-of-box.
70
+ Moreover, it may also be applied to other model's outputs on the Spider dataset, as studied in the [Original Paper](https://arxiv.org/abs/2302.08468).
71
+
72
+
73
+
74
+ ## Out-of-Scope Use
75
+
76
+ The model should not be used to intentionally create hostile or alienating environments for people.
77
+
78
+ # Bias, Risks, and Limitations
79
+
80
+
81
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
82
+
83
+
84
+
85
+ ## Recommendations
86
+
87
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
88
+
89
+ # Training Details
90
+
91
+ ## Training Data
92
+
93
+ The model is trained with the outputs from `code-davinci-002` model on the [Spider](https://yale-lily.github.io/spider) dataset.
94
+
95
+ ## Training Procedure
96
+
97
+ 20 program samples are drawn from the Codex model on the training examples of the Spider dataset, those programs are later executed to obtain the execution information.
98
+ And for each example and its program sample, the natural language description and execution information are also part of the inputs that used to train the T5-based model to predict "yes" or "no" as the verification labels.
99
+
100
+
101
+ ### Preprocessing
102
+
103
+ Please follow the instructions in the [Github Repo](https://github.com/niansong1996/lever) to reproduce the results.
104
+
105
+
106
+
107
+ ### Speeds, Sizes, Times
108
+
109
+ More information needed
110
+
111
+
112
+
113
+ # Evaluation
114
+
115
+
116
+ ## Testing Data, Factors & Metrics
117
+
118
+ ### Testing Data
119
+
120
+ Dev set of the [Spider](https://yale-lily.github.io/spider) dataset.
121
+
122
+ ### Factors
123
+ More information needed
124
+
125
+ ### Metrics
126
+
127
+ Execution accuracy (i.e., pass@1)
128
+
129
+
130
+ ## Results
131
+
132
+
133
+ ### Spider Text-to-SQL Generation
134
+ | | Exec. Acc. |
135
+ |-----------------|------------|
136
+ | Codex | 75.3 |
137
+ | Codex+LEVER | 81.9 |
138
+
139
+
140
+
141
+
142
+ # Model Examination
143
+
144
+ More information needed
145
+
146
+
147
+ # Environmental Impact
148
+
149
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
150
+
151
+ - **Hardware Type:** More information needed
152
+ - **Hours used:** More information needed
153
+ - **Cloud Provider:** More information needed
154
+ - **Compute Region:** More information needed
155
+ - **Carbon Emitted:** More information needed
156
+
157
+ # Technical Specifications [optional]
158
+
159
+ ## Model Architecture and Objective
160
+
161
+ `lever-spider-codex` is based on T5-base.
162
+
163
+ ## Compute Infrastructure
164
+
165
+ More information needed
166
+
167
+ ### Hardware
168
+
169
+
170
+ More information needed
171
+
172
+ ### Software
173
+
174
+ More information needed.
175
+
176
+ # Citation
177
+
178
+
179
+ **BibTeX:**
180
+
181
+
182
+ ```bibtex
183
+ @inproceedings{ni2023lever,
184
+ title={Lever: Learning to verify language-to-code generation with execution},
185
+ author={Ni, Ansong and Iyer, Srini and Radev, Dragomir and Stoyanov, Ves and Yih, Wen-tau and Wang, Sida I and Lin, Xi Victoria},
186
+ booktitle={Proceedings of the 40th International Conference on Machine Learning (ICML'23)},
187
+ year={2023}
188
+ }
189
+ ```
190
+
191
+
192
+
193
+
194
+ # Glossary [optional]
195
+
196
+ More information needed
197
+
198
+ # More Information [optional]
199
+ More information needed
200
+
201
+
202
+ # Model Card Authors [optional]
203
+
204
+ Yale LILY Lab in collaboration with Ezi Ozoani and the Hugging Face team
205
+
206
+ # Model Card Contact
207
+
208
+ Ansong Ni, contact info on [personal website](https://niansong1996.github.io)
209
+
210
+ # How to Get Started with the Model
211
+
212
+ This model is *not* intended to be directly used, please follow the instructions in the [Github Repo](https://github.com/niansong1996/lever).