langchain.callbacks.tracers.evaluation.EvaluatorCallbackHandler¶
class langchain.callbacks.tracers.evaluation.EvaluatorCallbackHandler(evaluators: Sequence[RunEvaluator], client: Optional[Client] = None, example_id: Optional[Union[str, UUID]] = None, skip_unfinished: bool = True, project_name: Optional[str] = 'evaluators', **kwargs: Any)[source]¶
A tracer that runs a run evaluator whenever a run is persisted.
Parameters
evaluators (Sequence[RunEvaluator]) – The run evaluators to apply to all top level runs.
client (LangSmith Client, optional) – The LangSmith client instance to use for evaluating the runs.
If not specified, a new instance will be created.
example_id (Union[UUID, str], optional) – The example ID to be associated with the runs.
project_name (str, optional) – The LangSmith project name to be organize eval chain runs under.
example_id¶
The example ID associated with the runs.
Type
Union[UUID, None]
client¶
The LangSmith client instance used for evaluating the runs.
Type
Client
evaluators¶
The sequence of run evaluators to be executed.
Type
Sequence[RunEvaluator]
executor¶
The thread pool executor used for running the evaluators.
Type
ThreadPoolExecutor
futures¶
The set of futures representing the running evaluators.
Type
Set[Future]
skip_unfinished¶
Whether to skip runs that are not finished or raised
an error.
Type
bool
project_name¶
The LangSmith project name to be organize eval chain runs under.
Type
Optional[str]
Attributes
ignore_agent
Whether to ignore agent callbacks.
ignore_chain
Whether to ignore chain callbacks.
ignore_chat_model
Whether to ignore chat model callbacks.
ignore_llm