generated text so far
param stop: Optional[List[str]] = None¶
List of strings that stop the generation when they are generated.
param tags: Optional[List[str]] = None¶
Tags to add to the run trace.
param temperature: float = 1.0¶
Float that controls the randomness of the sampling.
param tensor_parallel_size: Optional[int] = 1¶
The number of GPUs to use for distributed execution with tensor parallelism.
param top_k: int = -1¶
Integer that controls the number of top tokens to consider.
param top_p: float = 1.0¶
Float that controls the cumulative probability of the top tokens to consider.
param trust_remote_code: Optional[bool] = False¶
Trust remote code (e.g., from HuggingFace) when downloading the model
and tokenizer.
param use_beam_search: bool = False¶
Whether to use beam search instead of sampling.
param verbose: bool [Optional]¶
Whether to print out response text.
param vllm_kwargs: Dict[str, Any] [Optional]¶
Holds any model parameters valid for vllm.LLM call not explicitly specified.
__call__(prompt: str, stop: Optional[List[str]] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → str¶
Check Cache and run the LLM on the given prompt and input.
async abatch(inputs: List[Union[PromptValue, str, List[BaseMessage]]], config: Optional[Union[RunnableConfig, List[RunnableConfig]]] = None, *, return_exceptions: bool = False, **kwargs: Any) → List[str]¶