langchain.evaluation.parsing.base.JsonValidityEvaluator¶
class langchain.evaluation.parsing.base.JsonValidityEvaluator(**kwargs: Any)[source]¶
Evaluates whether the prediction is valid JSON.
This evaluator checks if the prediction is a valid JSON string. It does notrequire any input or reference.
requires_input¶
Whether this evaluator requires an input
string. Always False.
Type
bool
requires_reference¶
Whether this evaluator requires a
reference string. Always False.
Type
bool
evaluation_name¶
The name of the evaluation metric.
Always “json”.
Type
str
Examples
>>> evaluator = JsonValidityEvaluator()
>>> prediction = '{"name": "John", "age": 30, "city": "New York"}'
>>> evaluator.evaluate(prediction)
{'score': 1}
>>> prediction = '{"name": "John", "age": 30, "city": "New York",}'
>>> evaluator.evaluate(prediction)
{'score': 0, 'reasoning': 'Expecting property name enclosed in double quotes'}
Attributes
evaluation_name
The name of the evaluation.
requires_input
Whether this evaluator requires an input string.
requires_reference
Whether this evaluator requires a reference label.
Methods
__init__(**kwargs)
aevaluate_strings(*, prediction[, ...])
Asynchronously evaluate Chain or LLM output, based on optional input and label.
evaluate_strings(*, prediction[, reference, ...])
Evaluate Chain or LLM output, based on optional input and label.
__init__(**kwargs: Any) → None[source]¶
async aevaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict¶
Asynchronously evaluate Chain or LLM output, based on optional input and label.
Parameters