param rope_freq_scale: float = 1.0¶
Scale factor for rope sampling.
param seed: int = -1¶
Seed. If -1, a random seed is used.
param stop: Optional[List[str]] = []¶
A list of strings to stop generation when encountered.
param streaming: bool = True¶
Whether to stream the results, token by token.
param suffix: Optional[str] = None¶
A suffix to append to the generated text. If None, no suffix is appended.
param tags: Optional[List[str]] = None¶
Tags to add to the run trace.
param temperature: Optional[float] = 0.8¶
The temperature to use for sampling.
param top_k: Optional[int] = 40¶
The top-k value to use for sampling.
param top_p: Optional[float] = 0.95¶
The top-p value to use for sampling.
param use_mlock: bool = False¶
Force system to keep model in RAM.
param use_mmap: Optional[bool] = True¶
Whether to keep the model loaded in RAM
param verbose: bool = True¶
Print verbose output to stderr.
param vocab_only: bool = False¶
Only load the vocabulary, no weights.
__call__(prompt: str, stop: Optional[List[str]] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → str¶
Check Cache and run the LLM on the given prompt and input.