microsoft/phi-2 · reproducing gsm8k accuracy phi-2

Jan 21

hi .
i am trying to reproduce the gsm8k accuracy. in blog its mentioned that with 8-cot phi-2 was 68% on gsm8k.
but when i tired with FP16 accuracy was 50% (zero-shot). and FP4 accuracy was 30%(zero-shot). in phi-2 hugging face how to give 8-cot is missing.
model is unable to take the COT. any explanination. should we switch to phi-1.5 ????

gugarosa

Microsoft org Jan 22

Hello @karana657 !

I am not able to share the full GSM8k evaluation due to some internal imports, but this snippet might help you in using code for the evaluation:

def _timeout_handler(signum: int, frame: Any) -> None:
    raise Exception()

def _validate_completion(completion: str, label: str) -> bool:
    completion_lines = completion.split("TA:")[1].strip().split("\n")
    completion_code = "\n".join(completion_lines[1:] if ":" in completion_lines[0] else completion_lines)

    try:
        signal.signal(signal.SIGALRM, _timeout_handler)
        signal.alarm(2)

        try:
            stdout = io.StringIO()
            with contextlib.redirect_stdout(stdout):
                exec(
                    "import math\nfrom math import *\nimport numpy as np\nimport hashlib\n"
                    + completion_code
                    + "\n\n"
                    + "if type(result) == str:\n\tresult = result.replace(',', '')\n"
                    + f"assert(int(result) == {label})",
                    {},
                )
            signal.alarm(0)
            prediction = 1
        except Exception:
            prediction = 0
        finally:
            signal.alarm(0)

    except Exception:
        prediction = 0

    return prediction

The overall idea is to execute the code that was generated by the model and assert whether its outputs are equal to the ground-truth label. We also added some public imports to prevent many answers from failing.

gugarosa changed discussion status to closed Jan 22