langchain.llms.xinference.Xinference¶
class langchain.llms.xinference.Xinference[source]¶
Bases: LLM
Wrapper for accessing Xinference’s large-scale model inference service.
To use, you should have the xinference library installed:
pip install "xinference[all]"
Check out: https://github.com/xorbitsai/inference
To run, you need to start a Xinference supervisor on one server and Xinference workers on the other servers
Example
To start a local instance of Xinference, run
$ xinference
You can also deploy Xinference in a distributed cluster. Here are the steps:
Starting the supervisor:
$ xinference-supervisor
Starting the worker:
$ xinference-worker
Then, launch a model using command line interface (CLI).
Example:
$ xinference launch -n orca -s 3 -q q4_0
It will return a model UID. Then, you can use Xinference with LangChain.
Example:
from langchain.llms import Xinference
llm = Xinference(
    server_url="http://0.0.0.0:9997",
    model_uid = {model_uid} # replace model_uid with the model UID return from launching the model
)
llm(
    prompt="Q: where can we visit in the capital of France? A:",
    generate_config={"max_tokens": 1024, "stream": True},
)
To view all the supported builtin models, run:
$ xinference list --all
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param cache: Optional[bool] = None¶
param callback_manager: Optional[BaseCallbackManager] = None¶