langchain.evaluation.schema.EvaluatorType¶
class langchain.evaluation.schema.EvaluatorType(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]¶
The types of the evaluators.
QA = 'qa'¶
Question answering evaluator, which grades answers to questions
directly using an LLM.
COT_QA = 'cot_qa'¶
Chain of thought question answering evaluator, which grades
answers to questions using
chain of thought ‘reasoning’.
CONTEXT_QA = 'context_qa'¶
Question answering evaluator that incorporates ‘context’ in the response.
PAIRWISE_STRING = 'pairwise_string'¶
The pairwise string evaluator, which predicts the preferred prediction from
between two models.
LABELED_PAIRWISE_STRING = 'labeled_pairwise_string'¶
The labeled pairwise string evaluator, which predicts the preferred prediction
from between two models based on a ground truth reference label.
AGENT_TRAJECTORY = 'trajectory'¶
The agent trajectory evaluator, which grades the agent’s intermediate steps.
CRITERIA = 'criteria'¶
The criteria evaluator, which evaluates a model based on a
custom set of criteria without any reference labels.
LABELED_CRITERIA = 'labeled_criteria'¶
The labeled criteria evaluator, which evaluates a model based on a
custom set of criteria, with a reference label.
STRING_DISTANCE = 'string_distance'¶
Compare predictions to a reference answer using string edit distances.
EXACT_MATCH = 'exact_match'¶
Compare predictions to a reference answer using exact matching.
REGEX_MATCH = 'regex_match'¶
Compare predictions to a reference answer using regular expressions.
PAIRWISE_STRING_DISTANCE = 'pairwise_string_distance'¶
Compare predictions based on string edit distances.
EMBEDDING_DISTANCE = 'embedding_distance'¶