Prompt injection

Prompt injection is a security attack that happens when someone intentionally manipulates the input to a Generative AI system like a chatbot or code generator to make it behave in ways the designer didn’t intend.

It’s done by crafting inputs to Gen AI systems in order to confuse, hijack, or redirect the AI’s response by messing with its underlying structure.

For software testers, it's a way to test for input attacks on LLM-based systems. Just like a SQL injection or XSS, but here the payload is language and words designed to interfere with the model or system prompts.

Prompt injection is defined as a security exploit whereby an attacker embeds malicious instructions within user-provided input to override or subvert the original, intended prompt given to a language model. In essence, it's akin to SQL injection—but instead of manipulating a database query, you're manipulating a GPT-style prompt.

Prompt injection = malicious input that "injects" new directives into a prompt to trick the model into doing something unintended, sidestepping the instructions it was originally given.

Prompt injection involves: