For comment updating, we report performance on the same metrics that have been
used previously to benchmark models for this
task [1]}. This includes: xMatch (does the model prediction exactly match the ground truth), common metrics
that measure lexical overlap for evaluating text generation (BLEU-4 [2]} and METEOR [3]}), and common metrics for measuring text editing (GLEU [4]} and SARI [5]}).
For bug fixing, we use
xMatch, as done in prior work [6]}. For automated code review, we
report performance on xMatch and BLEU-4, which have been used
previously to benchmark models for this task [7]}.
