<story><title>OpenAI compatibility</title><url>https://ollama.ai/blog/openai-compatibility</url></story><parent_chain><item><author>ultrasaurus</author><text>The improvements in ease of use for locally hosting LLMs over the last few months have been amazing.  I was ranting about how easy &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;Mozilla-Ocho&amp;#x2F;llamafile&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;Mozilla-Ocho&amp;#x2F;llamafile&lt;/a&gt; is just a few hours ago [1]. Now I&amp;#x27;m torn as to which one to use :)&lt;p&gt;1: Quite literally hours ago: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;euri.ca&amp;#x2F;blog&amp;#x2F;2024-llm-self-hosting-is-easy-now&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;euri.ca&amp;#x2F;blog&amp;#x2F;2024-llm-self-hosting-is-easy-now&amp;#x2F;&lt;/a&gt;</text></item></parent_chain><comment><author>keriati1</author><text>I think it is even easier right now for companies to self host an inference server with basic rag support:&lt;p&gt;- get a Mac Mini or Mac Studio
- just run ollama serve, 
- run ollama web-ui in docker
- add some coding assitant model from ollamahub with the web-ui
- upload your documents in the web-ui&lt;p&gt;No code needed, you have your self hosted LLM with basic RAG giving you answers with your documents in context.
For us the deepseek coder 33b model is fast enough on a Mac Studio with 64gb ram and can give pretty good suggestions based on our internal coding documentation.</text></comment>
<story><title>OpenAI compatibility</title><url>https://ollama.ai/blog/openai-compatibility</url></story><parent_chain><item><author>ultrasaurus</author><text>The improvements in ease of use for locally hosting LLMs over the last few months have been amazing.  I was ranting about how easy &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;Mozilla-Ocho&amp;#x2F;llamafile&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;Mozilla-Ocho&amp;#x2F;llamafile&lt;/a&gt; is just a few hours ago [1]. Now I&amp;#x27;m torn as to which one to use :)&lt;p&gt;1: Quite literally hours ago: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;euri.ca&amp;#x2F;blog&amp;#x2F;2024-llm-self-hosting-is-easy-now&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;euri.ca&amp;#x2F;blog&amp;#x2F;2024-llm-self-hosting-is-easy-now&amp;#x2F;&lt;/a&gt;</text></item></parent_chain><comment><author>xyc</author><text>The pace of progress here is pretty amazing. I loved how easy it is to get llamafile up and running, but I missed feature complete chat interfaces, so I built one based off it: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;recurse.chat&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;recurse.chat&amp;#x2F;&lt;/a&gt;.&lt;p&gt;I still need GPT-4 for some tasks, but in daily usage it&amp;#x27;s replaced much of ChatGPT usage, especially since I can import all of my ChatGPT chat history. Also curious to learn about what people want to do with local AI.</text></comment>