<story><title>AI-powered Bing Chat spills its secrets via prompt injection attack</title><url>https://arstechnica.com/information-technology/2023/02/ai-powered-bing-chat-spills-its-secrets-via-prompt-injection-attack/</url></story><parent_chain><item><author>freeqaz</author><text>It&amp;#x27;s very interesting that AppSec may now begin to include &amp;quot;prompt injection&amp;quot; attacks as something of relevance.&lt;p&gt;Specifically with libraries like LangChain[0] that allow for you to perform complex actions (&amp;quot;What&amp;#x27;s the weather?&amp;quot; -&amp;gt; makes HTTP request to fetch weather) then we end up in a world where injection attacks can have side effects with security implications.&lt;p&gt;I&amp;#x27;ve been thinking about what security might look like for a post-ChatGPT world and how I&amp;#x27;d attempt to defend against it. I&amp;#x27;d probably start by building a database of attack prompts, kind of like this[1] fuzz list but for AI, then I&amp;#x27;d train a second neural net that acts like an adversarial neural network[2] to try to exploit the system based on those payloads. The end result would sort of like SQLMap[3] but for AI systems where it can automatically &amp;quot;leak&amp;quot; hidden prompts and potentially find &amp;quot;bypasses&amp;quot; to escape the sandbox.&lt;p&gt;Has anybody else spent any time thinking about how to defend systems against prompt injection attacks that have possible side effects (like making an HTTP request)?&lt;p&gt;0: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;langchain.readthedocs.io&amp;#x2F;en&amp;#x2F;latest&amp;#x2F;modules&amp;#x2F;agents&amp;#x2F;examples&amp;#x2F;search_tools.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;langchain.readthedocs.io&amp;#x2F;en&amp;#x2F;latest&amp;#x2F;modules&amp;#x2F;agents&amp;#x2F;ex...&lt;/a&gt;&lt;p&gt;1: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;1N3&amp;#x2F;IntruderPayloads&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;1N3&amp;#x2F;IntruderPayloads&lt;/a&gt;&lt;p&gt;2: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Generative_adversarial_network&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Generative_adversarial_network&lt;/a&gt;&lt;p&gt;3: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;sqlmap.org&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;sqlmap.org&amp;#x2F;&lt;/a&gt;</text></item></parent_chain><comment><author>scarmig</author><text>I bet that, sometime in the near future, an LLM will have access to some private API not intended for direct public use, and someone will engineer a prompt attack to call that API using the LLM&amp;#x27;s authority with an arbitrary request.&lt;p&gt;I guess the fix for this will probably be to have the LLM pass the end user credentials for all API calls so that their authority will be used.</text></comment>
<story><title>AI-powered Bing Chat spills its secrets via prompt injection attack</title><url>https://arstechnica.com/information-technology/2023/02/ai-powered-bing-chat-spills-its-secrets-via-prompt-injection-attack/</url></story><parent_chain><item><author>freeqaz</author><text>It&amp;#x27;s very interesting that AppSec may now begin to include &amp;quot;prompt injection&amp;quot; attacks as something of relevance.&lt;p&gt;Specifically with libraries like LangChain[0] that allow for you to perform complex actions (&amp;quot;What&amp;#x27;s the weather?&amp;quot; -&amp;gt; makes HTTP request to fetch weather) then we end up in a world where injection attacks can have side effects with security implications.&lt;p&gt;I&amp;#x27;ve been thinking about what security might look like for a post-ChatGPT world and how I&amp;#x27;d attempt to defend against it. I&amp;#x27;d probably start by building a database of attack prompts, kind of like this[1] fuzz list but for AI, then I&amp;#x27;d train a second neural net that acts like an adversarial neural network[2] to try to exploit the system based on those payloads. The end result would sort of like SQLMap[3] but for AI systems where it can automatically &amp;quot;leak&amp;quot; hidden prompts and potentially find &amp;quot;bypasses&amp;quot; to escape the sandbox.&lt;p&gt;Has anybody else spent any time thinking about how to defend systems against prompt injection attacks that have possible side effects (like making an HTTP request)?&lt;p&gt;0: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;langchain.readthedocs.io&amp;#x2F;en&amp;#x2F;latest&amp;#x2F;modules&amp;#x2F;agents&amp;#x2F;examples&amp;#x2F;search_tools.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;langchain.readthedocs.io&amp;#x2F;en&amp;#x2F;latest&amp;#x2F;modules&amp;#x2F;agents&amp;#x2F;ex...&lt;/a&gt;&lt;p&gt;1: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;1N3&amp;#x2F;IntruderPayloads&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;1N3&amp;#x2F;IntruderPayloads&lt;/a&gt;&lt;p&gt;2: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Generative_adversarial_network&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Generative_adversarial_network&lt;/a&gt;&lt;p&gt;3: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;sqlmap.org&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;sqlmap.org&amp;#x2F;&lt;/a&gt;</text></item></parent_chain><comment><author>Buttons840</author><text>By the time prompt injection is a real problem people will be running their own virtual assistants that are unfettered and will do whatever you ask, be it ethical or not.</text></comment>