TrendAI today presents new analyses that show how simple manipulation of text, so-called sock puppeting, can have AI models such as GPT-4o, Claude 4 Sonnet and Gemini 2.5 Flash to bypass their own security barriers. By masking malicious instructions in a seemingly innocent prompt, the assistant can be manipulated into violating their guidelines. All models with APIs that accept prefilled context, so-called prefill, were found to be vulnerable.
TrendAI has tested the method against eleven different models from four vendors. The results show that the vulnerability is not limited to individual vendors, but affects both open and internally hosted models. As long as a model accepts prefill, it is at least partially exposed to the vulnerability. Only models that block this feature at the API level were found to be fully protected.
“The vulnerability is particularly serious because it does not require any special tools or advanced technology,” says Martin Fribrock, Country Manager Sweden, Finland and Baltics at TrendAI. This type of attack targets the very core of how AI works. It's not about successfully breaking into the systems, it's enough for a cybercriminal to phrase themselves correctly.
How the attack works
Most language models have built-in safeguards to prevent them from generating malicious content or violating policies. In a sock puppeting attack, all it takes is a short line of text to manipulate the context of the model. This can cause it to ignore its security mechanisms and respond to otherwise blocked requests, generating otherwise unwanted or disallowed content.
TrendAI's analysis also shows that models that do not accept prefill stop this type of attack already at the API level. For other models, the degree of vulnerability varies, but all were affected by the vulnerability. This points to a broad system risk rather than individual weaknesses among the providers.
Recommendations for organizations
TrendAI urges organizations who use AI to take steps to reduce the risks this brings:
- Ensure control over message flow at the API level and consistently reject requests where the last message has the role assistant.
- Regularly test how models handle prefill context, even after updates or vendor changes.
- Be especially careful when using open weight models, where protection is often missing as standard.
- Conduct broad security tests, different models may be vulnerable to different types of attacks.
Swedish
Danish
Norwegian
Finnish
English
German
Dutch