<story><title>High-Throughput Generative Inference of Large Language Models with a Single GPU</title><url>https://arxiv.org/abs/2303.06865</url></story><parent_chain><item><author>TaylorAlexander</author><text>I genuinely believe LLMs are not enough on their own to generate the most interesting and humanlike interactions, but it is clear to me that we are on this path.&lt;p&gt;I was watching Westworld the TV series recently, and it made me think of what it would be like to be in a VR space with a bunch of characters that all had their own back story and personalities, which were sufficiently humanlike that it was fun to talk to them for long periods of time. I feel like we will see this in maybe ten years, and it&amp;#x27;s going to be pretty cool! I mean I am sure we will see attempts at this sooner but I think it will take a while to get all the pieces working so that a conversational AI really feels human. Maybe 5 years, who knows.</text></item><item><author>georgehill</author><text>&amp;gt; FlexGen lowers the resource requirements of running
175B-scale models down to a single 16GB GPU and reaches a generation throughput of 1 token&amp;#x2F;s with an effective batch size
of 144.&lt;p&gt;I can&amp;#x27;t imagine what will be happening in LLM space next year this time. Maybe LLM natively integrated into games and browsers.</text></item></parent_chain><comment><author>noduerme</author><text>If something like GPT-4 can batch the language tokens in with multi modal tokens (e.g. the character LLM gets a low-res &amp;quot;video&amp;quot; feed of you, the player, from the character&amp;#x27;s perspective)... you could have it a lot sooner.</text></comment>
<story><title>High-Throughput Generative Inference of Large Language Models with a Single GPU</title><url>https://arxiv.org/abs/2303.06865</url></story><parent_chain><item><author>TaylorAlexander</author><text>I genuinely believe LLMs are not enough on their own to generate the most interesting and humanlike interactions, but it is clear to me that we are on this path.&lt;p&gt;I was watching Westworld the TV series recently, and it made me think of what it would be like to be in a VR space with a bunch of characters that all had their own back story and personalities, which were sufficiently humanlike that it was fun to talk to them for long periods of time. I feel like we will see this in maybe ten years, and it&amp;#x27;s going to be pretty cool! I mean I am sure we will see attempts at this sooner but I think it will take a while to get all the pieces working so that a conversational AI really feels human. Maybe 5 years, who knows.</text></item><item><author>georgehill</author><text>&amp;gt; FlexGen lowers the resource requirements of running
175B-scale models down to a single 16GB GPU and reaches a generation throughput of 1 token&amp;#x2F;s with an effective batch size
of 144.&lt;p&gt;I can&amp;#x27;t imagine what will be happening in LLM space next year this time. Maybe LLM natively integrated into games and browsers.</text></item></parent_chain><comment><author>macrolocal</author><text>Nb. the non-humanlike interactions they&amp;#x27;ll enable are equally (maybe more) exciting.</text></comment>