codebleu with HF evaluate crashes

#3
by forwards - opened

Hi @k4black , Appreciate your work. I am trying to use codebleu with HF evaluate throws the following error on ubuntu. I am using evaluate==0.4.3. Also the code seems to always take language as JAVA in-spite of passing in python as the language in args.

>>> import evaluate
>>> metric = evaluate.load("dvitel/codebleu")
Downloaded keywords to /home/ubuntu/.cache/huggingface/metrics/codebleu/default/downloads/extracted/8962f4dbd65b3a6725b5f11dbd4887b3774cd113b9b7eabc913493ae88f41b7f
Downloaded languages.so to /home/ubuntu/.cache/huggingface/hub/spaces--dvitel--codebleu/snapshots/f8862f74b23bb1d0c383f31c32db54269b5ba104/my-languages.so
>>>
>>>
>>> prediction = "def add ( a , b ) :\n return a + b"
>>> reference = ["def add(x, y): return x + y"]
>>> results = metric.compute(references=reference, predictions=[prediction], lang="python")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/.local/lib/python3.10/site-packages/evaluate/module.py", line 467, in compute
    output = self._compute(**inputs, **compute_kwargs)
  File "/home/ubuntu/.cache/huggingface/modules/evaluate_modules/metrics/dvitel--codebleu/5fb798ec7da0c60e28e5abca10d6b84d2edbb430450c28bff545aa9e2d62659c/codebleu.py", line 118, in _compute
    res = calc_codebleu(
  File "/home/ubuntu/.cache/huggingface/modules/evaluate_modules/metrics/dvitel--codebleu/5fb798ec7da0c60e28e5abca10d6b84d2edbb430450c28bff545aa9e2d62659c/my_codebleu.py", line 57, in calc_codebleu
    syntax_match_score = syntax_match.corpus_syntax_match(
  File "/home/ubuntu/.cache/huggingface/modules/evaluate_modules/metrics/dvitel--codebleu/5fb798ec7da0c60e28e5abca10d6b84d2edbb430450c28bff545aa9e2d62659c/syntax_match.py", line 27, in corpus_syntax_match
    JAVA_LANGUAGE = Language(langso_dir, lang)
TypeError: __init__() takes exactly 1 argument (2 given)

Sign up or log in to comment