<story><title>An LLM playground you can run on your laptop</title><url>https://github.com/nat/openplayground</url></story><parent_chain><item><author>d4rkp4ttern</author><text>This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?</text></item></parent_chain><comment><author>MacsHeadroom</author><text>Yes, there are python bindings for llama.cpp and the text-generation-webui already uses them for local inference. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;oobabooga&amp;#x2F;text-generation-webui&amp;#x2F;wiki&amp;#x2F;llama.cpp-models&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;oobabooga&amp;#x2F;text-generation-webui&amp;#x2F;wiki&amp;#x2F;llam...&lt;/a&gt;&lt;p&gt;&amp;quot;pip install llamacpp&amp;quot; or &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;thomasantony&amp;#x2F;llamacpp-python&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;thomasantony&amp;#x2F;llamacpp-python&lt;/a&gt;</text></comment>
<story><title>An LLM playground you can run on your laptop</title><url>https://github.com/nat/openplayground</url></story><parent_chain><item><author>d4rkp4ttern</author><text>This is very neat, thanks for sharing. I was wondering about a related thing — is there a way to query a llama.cpp (or other such local model) via an API from Python? In other words, I see a lot of cool applications being built with langchain + ClosedAPI, so I’m wondering if an API call to a local model could be a drop-in replacement for the ClosedAPI call?</text></item></parent_chain><comment><author>versteegen</author><text>Maybe TextSynth server? It has a REST JSON API to pretty much every major LLM running locally with minimal dependencies (no python&amp;#x2F;pytorch&amp;#x2F;CUDA). (And I see that since last week it now has an HTML GUI too.) &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;bellard.org&amp;#x2F;ts_server&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;bellard.org&amp;#x2F;ts_server&amp;#x2F;&lt;/a&gt;&lt;p&gt;However, the GPU version is only available commercially. I&amp;#x27;d like to see someone compare the speed of the CPU version against PyTorch or llama.cpp. (Edit: llama.cpp&amp;#x27;s author wrote &amp;quot;I expect LibNC [used by ts_server] will be better in every aspect: performance, accuracy, determinism. But hopefully with time we will close the gap.&amp;quot; [1])&lt;p&gt;EDIT: But if you wish, here&amp;#x27;s a Python interface to llama.cpp: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;PotatoSpudowski&amp;#x2F;fastLLaMa&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;PotatoSpudowski&amp;#x2F;fastLLaMa&lt;/a&gt;&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;context?id=35195270&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;context?id=35195270&lt;/a&gt;</text></comment>