langchain.smith.evaluation.runner_utils.run_on_dataset¶
langchain.smith.evaluation.runner_utils.run_on_dataset(client: Optional[Client], dataset_name: str, llm_or_chain_factory: Union[Callable[[], Union[Chain, Runnable]], BaseLanguageModel, Callable[[dict], Any], Runnable, Chain], *, evaluation: Optional[RunEvalConfig] = None, concurrency_level: int = 5, project_name: Optional[str] = None, project_metadata: Optional[Dict[str, Any]] = None, verbose: bool = False, tags: Optional[List[str]] = None, **kwargs: Any) → Dict[str, Any][source]¶
Run the Chain or language model on a dataset and store traces
to the specified project name.
Parameters
dataset_name – Name of the dataset to run the chain on.
llm_or_chain_factory – Language model or Chain constructor to run
over the dataset. The Chain constructor is used to permit
independent calls on each example without carrying over state.
evaluation – Configuration for evaluators to run on the
results of the chain
concurrency_level – The number of async tasks to run concurrently.
project_name – Name of the project to store the traces in.
Defaults to {dataset_name}-{chain class name}-{datetime}.
project_metadata – Optional metadata to add to the project.
Useful for storing information the test variant.
(prompt version, model version, etc.)
client – LangSmith client to use to access the dataset and to
log feedback and run traces.
verbose – Whether to print progress.
tags – Tags to add to each run in the project.
Returns
A dictionary containing the run’s project name and the resulting model outputs.
For the (usually faster) async version of this function, see arun_on_dataset().
Examples
from langsmith import Client
from langchain.chat_models import ChatOpenAI