Construct the pipeline remotely using an auxiliary function.
The load function needs to be importable to be imported
and run on the server, i.e. in a module and not a REPL or closure.
Then, initialize the remote inference function.
param cache: Optional[bool] = None¶
param callback_manager: Optional[BaseCallbackManager] = None¶
param callbacks: Callbacks = None¶
param device: int = 0¶
Device to use for inference. -1 for CPU, 0 for GPU, 1 for second GPU, etc.
param hardware: Any = None¶
Remote hardware to send the inference function to.
param inference_fn: Callable = <function _generate_text>¶
Inference function to send to the remote hardware.
param load_fn_kwargs: Optional[dict] = None¶
Key word arguments to pass to the model load function.
param metadata: Optional[Dict[str, Any]] = None¶
Metadata to add to the run trace.
param model_id: str = 'gpt2'¶
Hugging Face model_id to load the model.
param model_kwargs: Optional[dict] = None¶
Key word arguments to pass to the model.
param model_load_fn: Callable = <function _load_transformer>¶
Function to load the model remotely on the server.
param model_reqs: List[str] = ['./', 'transformers', 'torch']¶
Requirements to install on hardware to inference the model.
param tags: Optional[List[str]] = None¶
Tags to add to the run trace.
param task: str = 'text-generation'¶
Hugging Face task (“text-generation”, “text2text-generation” or
“summarization”).
param verbose: bool [Optional]¶
Whether to print out response text.