langchain.evaluation.criteria.eval_chain.CriteriaEvalChain¶
class langchain.evaluation.criteria.eval_chain.CriteriaEvalChain[source]¶
Bases: StringEvaluator, LLMEvalChain, LLMChain
LLM Chain for evaluating runs against criteria.
Parameters
llm (BaseLanguageModel) – The language model to use for evaluation.
criteria (Union[Mapping[str, str]]) – The criteria or rubric to evaluate the runs against. It can be a mapping of
criterion name to its description, or a single criterion name.
prompt (Optional[BasePromptTemplate], default=None) – The prompt template to use for generating prompts. If not provided, a
default prompt template will be used based on the value of
requires_reference.
requires_reference (bool, default=False) – Whether the evaluation requires a reference text. If True, the
PROMPT_WITH_REFERENCES template will be used, which includes the
reference labels in the prompt. Otherwise, the PROMPT template will be
used, which is a reference-free prompt.
**kwargs (Any) – Additional keyword arguments to pass to the LLMChain constructor.
Returns
An instance of the CriteriaEvalChain class.
Return type
CriteriaEvalChain
Examples
>>> from langchain.chat_models import ChatAnthropic
>>> from langchain.evaluation.criteria import CriteriaEvalChain
>>> llm = ChatAnthropic(temperature=0)
>>> criteria = {"my-custom-criterion": "Is the submission the most amazing ever?"}
>>> evaluator = CriteriaEvalChain.from_llm(llm=llm, criteria=criteria)
>>> evaluator.evaluate_strings(prediction="Imagine an ice cream flavor for the color aquamarine", input="Tell me an idea")
{