langchain.llms.vllm.VLLM¶
class langchain.llms.vllm.VLLM[source]¶
Bases: BaseLLM
VLLM language model.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param best_of: Optional[int] = None¶
Number of output sequences that are generated from the prompt.
param cache: Optional[bool] = None¶
param callback_manager: Optional[BaseCallbackManager] = None¶
param callbacks: Callbacks = None¶
param download_dir: Optional[str] = None¶
Directory to download and load the weights. (Default to the default
cache dir of huggingface)
param dtype: str = 'auto'¶
The data type for the model weights and activations.
param frequency_penalty: float = 0.0¶
Float that penalizes new tokens based on their frequency in the
generated text so far
param ignore_eos: bool = False¶
Whether to ignore the EOS token and continue generating tokens after
the EOS token is generated.
param logprobs: Optional[int] = None¶
Number of log probabilities to return per output token.
param max_new_tokens: int = 512¶
Maximum number of tokens to generate per output sequence.
param metadata: Optional[Dict[str, Any]] = None¶
Metadata to add to the run trace.
param model: str = ''¶
The name or path of a HuggingFace Transformers model.
param n: int = 1¶
Number of output sequences to return for the given prompt.
param presence_penalty: float = 0.0¶
Float that penalizes new tokens based on whether they appear in the
generated text so far
param stop: Optional[List[str]] = None¶