langchain.evaluation.parsing.base.JsonEqualityEvaluator¶
class langchain.evaluation.parsing.base.JsonEqualityEvaluator(operator: Optional[Callable] = None, **kwargs: Any)[source]¶
Evaluates whether the prediction is equal to the reference afterparsing both as JSON.
This evaluator checks if the prediction, after parsing as JSON, is equalto the reference,
which is also parsed as JSON. It does not require an input string.
requires_input¶
Whether this evaluator requires an
input string. Always False.
Type
bool
requires_reference¶
Whether this evaluator requires
a reference string. Always True.
Type
bool
evaluation_name¶
The name of the evaluation metric.
Always “parsed_equality”.
Type
str
Examples
>>> evaluator = JsonEqualityEvaluator()
>>> evaluator.evaluate_strings('{"a": 1}', reference='{"a": 1}')
{'score': True}
>>> evaluator.evaluate_strings('{"a": 1}', reference='{"a": 2}')
{'score': False}
>>> evaluator = JsonEqualityEvaluator(operator=lambda x, y: x['a'] == y['a'])
>>> evaluator.evaluate_strings('{"a": 1}', reference='{"a": 1}')
{'score': True}
>>> evaluator.evaluate_strings('{"a": 1}', reference='{"a": 2}')
{'score': False}
Attributes
evaluation_name
The name of the evaluation.
requires_input
Whether this evaluator requires an input string.
requires_reference
Whether this evaluator requires a reference label.
Methods
__init__([operator])
aevaluate_strings(*, prediction[, ...])
Asynchronously evaluate Chain or LLM output, based on optional input and label.
evaluate_strings(*, prediction[, reference, ...])
Evaluate Chain or LLM output, based on optional input and label.