<story><title>Inversion: Fast, Reliable Structured LLMs</title><url>https://rysana.com/inversion</url></story><parent_chain></parent_chain><comment><author>AmazingTurtle</author><text>Most likely a small retrieval optimized model with focus on JSON tokens and a combination of the technique involved in grammar&amp;#x2F;guidance&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;guidance-ai&amp;#x2F;guidance&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;guidance-ai&amp;#x2F;guidance&lt;/a&gt;&lt;p&gt;Same principle</text></comment>
<story><title>Inversion: Fast, Reliable Structured LLMs</title><url>https://rysana.com/inversion</url></story><parent_chain></parent_chain><comment><author>fzysingularity</author><text>I was initially impressed with the landing page, but it does look a bit suspect when things are claimed to be 100x faster without much info on the HW acceleration or the model sizes.&lt;p&gt;My best guess is that they&amp;#x27;re using two approaches to get this running faster:&lt;p&gt;- structured generation techniques from sglang (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;sgl-project&amp;#x2F;sglang&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;sgl-project&amp;#x2F;sglang&lt;/a&gt;) that allow them to generate faster JSON (with look-ahead &amp;#x2F; pre-fill) with strong guarantees on the output (i.e. 100% reliable, without requiring any retries).&lt;p&gt;- distilling a gpt-3.5 turbo-esque model from GPT-4 JSON outputs, and using it in conjuction with above to give the additional performance boosts on inference.&lt;p&gt;It doesn&amp;#x27;t seem like they&amp;#x27;re deploying on any custom silicon, nor have they optimized GPU kernels to suggest that the speed ups came there.</text></comment>