param do_sample: bool = False¶
Activate logits sampling
param inference_server_url: str = ''¶
text-generation-inference instance base url
param max_new_tokens: int = 512¶
Maximum number of generated tokens
param metadata: Optional[Dict[str, Any]] = None¶
Metadata to add to the run trace.
param model_kwargs: Dict[str, Any] [Optional]¶
Holds any model parameters valid for call not explicitly specified
param repetition_penalty: Optional[float] = None¶
The parameter for repetition penalty. 1.0 means no penalty.
See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.
param return_full_text: bool = False¶
Whether to prepend the prompt to the generated text
param seed: Optional[int] = None¶
Random sampling seed
param server_kwargs: Dict[str, Any] [Optional]¶
Holds any text-generation-inference server parameters not explicitly specified
param stop_sequences: List[str] [Optional]¶
Stop generating tokens if a member of stop_sequences is generated
param streaming: bool = False¶
Whether to generate a stream of tokens asynchronously
param tags: Optional[List[str]] = None¶
Tags to add to the run trace.
param temperature: Optional[float] = 0.8¶
The value used to module the logits distribution.
param timeout: int = 120¶
Timeout in seconds
param top_k: Optional[int] = None¶
The number of highest probability vocabulary tokens to keep for
top-k-filtering.
param top_p: Optional[float] = 0.95¶
If set to < 1, only the smallest set of most probable tokens with probabilities
that add up to top_p or higher are kept for generation.
param truncate: Optional[int] = None¶
Truncate inputs tokens to the given size