langchain.llms.deepsparse.DeepSparse¶
class langchain.llms.deepsparse.DeepSparse[source]¶
Bases: LLM
Neural Magic DeepSparse LLM interface.
To use, you should have the deepsparse or deepsparse-nightly
python package installed. See https://github.com/neuralmagic/deepsparse
This interface let’s you deploy optimized LLMs straight from the
[SparseZoo](https://sparsezoo.neuralmagic.com/?useCase=text_generation)
.. rubric:: Example
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param cache: Optional[bool] = None¶
param callback_manager: Optional[BaseCallbackManager] = None¶
param callbacks: Callbacks = None¶
param generation_config: Union[None, str, Dict] = None¶
GenerationConfig dictionary consisting of parameters used to control
sequences generated for each prompt. Common parameters are:
max_length, max_new_tokens, num_return_sequences, output_scores,
top_p, top_k, repetition_penalty.
param metadata: Optional[Dict[str, Any]] = None¶
Metadata to add to the run trace.
param model: str [Required]¶
The path to a model file or directory or the name of a SparseZoo model stub.
param model_config: Optional[Dict[str, Any]] = None¶
Key word arguments passed to the pipeline construction.
Common parameters are sequence_length, prompt_sequence_length
param streaming: bool = False¶
Whether to stream the results, token by token.
param tags: Optional[List[str]] = None¶
Tags to add to the run trace.
param verbose: bool [Optional]¶
Whether to print out response text.