return {"score": prediction == reference}
evaluation_config = smith_eval.RunEvalConfig(
    custom_evaluators = [MyStringEvaluator()],
)
await arun_on_dataset(
    client,
    "<my_dataset_name>",
    construct_chain,
    evaluation=evaluation_config,
)
Examples using arun_on_dataset¶
LangSmith Walkthrough