<story><title>Show HN: An open source framework for voice assistants</title><url>https://github.com/pipecat-ai/pipecat</url><text>I&amp;#x27;ve been obsessed for the past ~year with the possibilities of talking to LLMs. I built a bunch of one-off prototypes, shared code on X, started a Meetup group in SF, and co-hosted a big hackathon. It turns out that there are a few low-level problems that everybody building conversational&amp;#x2F;real-time AI needs to solve on the way to building&amp;#x2F;shipping something that works well: low-latency media transport, echo cancellation, voice activity detection, phrase endpointing, pipelining data between models&amp;#x2F;services, handling voice interruptions, swapping out different models&amp;#x2F;services.&lt;p&gt;On the theory that something like a LlamaIndex or LangChain for real-time&amp;#x2F;conversational AI would be useful, a few of us started working on a Python library for voice (and multimodal) AI assistants&amp;#x2F;agents.&lt;p&gt;So ... Pipecat: a framework for building things like personal coaches, meeting assistants, story-telling toys for kids, customer support bots, virtual friends, and snarky social bots.&lt;p&gt;Most of the core contributors to Pipecat so far work together at our day jobs. This has been a kind of &amp;quot;20% time&amp;quot; thing at our company. But we&amp;#x27;re serious about welcoming all contributions. We want Pipecat to support any and all models, services, transport layers, and infrastructure tooling. If you&amp;#x27;re interested in this stuff, please check it out and let us know what you think. Submit PRs. Become a maintainer. Join the Discord. Post cool stuff. Post funny stuff when your voice agent goes completely off the rails (as mine sometimes do).</text></story><parent_chain></parent_chain><comment><author>awenix</author><text>Nice to see an open source implementation, i have been seeing many startups get into this space like &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.retellai.com&amp;#x2F;&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.retellai.com&amp;#x2F;&lt;/a&gt;, &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;fixie.ai&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;fixie.ai&amp;#x2F;&lt;/a&gt; etc. They always end up needing speech-to-speech models (current approach seems speech-text-text-speech with multiple agents handling 1 listening + 1 speaking), excited to see how this plays with recently announced gpt-4o</text></comment>
<story><title>Show HN: An open source framework for voice assistants</title><url>https://github.com/pipecat-ai/pipecat</url><text>I&amp;#x27;ve been obsessed for the past ~year with the possibilities of talking to LLMs. I built a bunch of one-off prototypes, shared code on X, started a Meetup group in SF, and co-hosted a big hackathon. It turns out that there are a few low-level problems that everybody building conversational&amp;#x2F;real-time AI needs to solve on the way to building&amp;#x2F;shipping something that works well: low-latency media transport, echo cancellation, voice activity detection, phrase endpointing, pipelining data between models&amp;#x2F;services, handling voice interruptions, swapping out different models&amp;#x2F;services.&lt;p&gt;On the theory that something like a LlamaIndex or LangChain for real-time&amp;#x2F;conversational AI would be useful, a few of us started working on a Python library for voice (and multimodal) AI assistants&amp;#x2F;agents.&lt;p&gt;So ... Pipecat: a framework for building things like personal coaches, meeting assistants, story-telling toys for kids, customer support bots, virtual friends, and snarky social bots.&lt;p&gt;Most of the core contributors to Pipecat so far work together at our day jobs. This has been a kind of &amp;quot;20% time&amp;quot; thing at our company. But we&amp;#x27;re serious about welcoming all contributions. We want Pipecat to support any and all models, services, transport layers, and infrastructure tooling. If you&amp;#x27;re interested in this stuff, please check it out and let us know what you think. Submit PRs. Become a maintainer. Join the Discord. Post cool stuff. Post funny stuff when your voice agent goes completely off the rails (as mine sometimes do).</text></story><parent_chain></parent_chain><comment><author>ilaksh</author><text>This is great but we really need an audio-to-audio model like they demoed in the open source world. Does anyone know of anything like that?&lt;p&gt;Edit: someone found one: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=40346992&quot;&gt;https:&amp;#x2F;&amp;#x2F;news.ycombinator.com&amp;#x2F;item?id=40346992&lt;/a&gt;</text></comment>