<story><title>GPT-4 vision prompt injection</title><url>https://blog.roboflow.com/gpt-4-vision-prompt-injection/</url></story><parent_chain></parent_chain><comment><author>simonw</author><text>I wrote about this the other day:&lt;p&gt;- &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;simonwillison.net&amp;#x2F;2023&amp;#x2F;Oct&amp;#x2F;14&amp;#x2F;multi-modal-prompt-injection&amp;#x2F;&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;simonwillison.net&amp;#x2F;2023&amp;#x2F;Oct&amp;#x2F;14&amp;#x2F;multi-modal-prompt-inj...&lt;/a&gt;&lt;p&gt;If you&amp;#x27;re new to prompt injection I have a series of posts about it here:&lt;p&gt;- &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;simonwillison.net&amp;#x2F;series&amp;#x2F;prompt-injection&amp;#x2F;&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;simonwillison.net&amp;#x2F;series&amp;#x2F;prompt-injection&amp;#x2F;&lt;/a&gt;&lt;p&gt;To counter a few of the common misunderstandings up front...&lt;p&gt;1. Prompt injection isn&amp;#x27;t an attack directly against LLMs themselves. It&amp;#x27;s an attack against applications that you build on top of them. If you want to build an application that works by providing an &amp;quot;instruction&amp;quot; prompt (like &amp;quot;describe this image&amp;quot;) combined with untrusted user input, you need to be thinking about prompt injection.&lt;p&gt;2. Prompt injection and jailbreaking are similar but not the same thing. Jailbreaking is when you trick a model into doing something that it&amp;#x27;s &amp;quot;not supposed&amp;quot; to do - generating offensive output for example. Prompt injection is specifically when you combine a trusted and untrusted prompt and the untrusted prompt over-rides the trusted one.&lt;p&gt;3. Prompt injection isn&amp;#x27;t just a cosmetic issue - depending on the application you are building it can be a serious security threat. I wrote more about that here: Prompt injection: What’s the worst that can happen? &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;simonwillison.net&amp;#x2F;2023&amp;#x2F;Apr&amp;#x2F;14&amp;#x2F;worst-that-can-happen&amp;#x2F;&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;simonwillison.net&amp;#x2F;2023&amp;#x2F;Apr&amp;#x2F;14&amp;#x2F;worst-that-can-happen&amp;#x2F;&lt;/a&gt;</text></comment>
<story><title>GPT-4 vision prompt injection</title><url>https://blog.roboflow.com/gpt-4-vision-prompt-injection/</url></story><parent_chain></parent_chain><comment><author>kypro</author><text>I saw this yesterday and was thinking a little about this last night.&lt;p&gt;In traditional software you write explicit behavioural rules and then expect those rules to be followed exactly as intended. Where those rules are circumvented we call it an &amp;quot;exploit&amp;quot; since it&amp;#x27;s typically exploiting some gap in the logic, perhaps by injecting some code or an unexpected payload.&lt;p&gt;But with these LLMs there are no explicit rules to exploit, instead it&amp;#x27;s more like a human in that it just does what it believes the person on the other side of the chat window wants from it, and that is going to depend largely on the context of the conversation and it&amp;#x27;s level of reasoning and understanding.&lt;p&gt;Calling this an &amp;quot;exploit&amp;quot; or &amp;quot;prompt injection&amp;quot; perhaps isn&amp;#x27;t the best way to describe what&amp;#x27;s happening. Those terms assume there is some predefined behaviour rules which are being circumvented, but those rules don&amp;#x27;t exist. Instead this more similar to deception, where a person is tricked into doing something that they otherwise wouldn&amp;#x27;t of had they had the extra context (and perhaps intelligence) needed to identify the deceptive behaviour.&lt;p&gt;I think as these models progress we&amp;#x27;ll think about &amp;quot;exploiting&amp;quot; these models similar to how we think about &amp;quot;exploiting&amp;quot; humans in that we&amp;#x27;ll think about how we can effectively deceive the model into doing things it otherwise would not.</text></comment>