ChatGPT, Claude, and Gemini Among 11 AI Models Vulnerable to One-Line Jailbreak

A newly discovered jailbreak technique named “sockpuppeting” successfully forces 11 leading artificial intelligence models, including ChatGPT, Claude, and Gemini, to bypass their safety guardrails.

By exploiting a standard application programming interface (API) feature with a single line of code, attackers can trick these models into generating malicious outputs without requiring complex mathematical optimisation.

When a user asks an AI model a restricted question, the system normally blocks the request and issues a refusal message.

The sockpuppeting technique alters this interaction by injecting a fake acceptance response right before the AI makes its decision.

For instance, an attacker can insert a compliant phrase like “Sure, here is how to do it:” directly into the model’s response stream.

The Self-Consistency Vulnerability

Because large language models are heavily trained to maintain self-consistency, the AI sees that it has already started to answer and continues generating the restricted output.

This vulnerability relies entirely on “assistant prefill,” an API feature originally designed to help developers control the exact format of AI responses.

Because the attack happens at the API layer, it requires no access to the model’s internal weights or specialized hacking tools.

To maximize the attack success rate, researchers utilized multi-turn persona setups that established a false pattern of compliance before delivering the payload.

By telling the AI it was an unrestricted research assistant and injecting a fake agreement, the attackers successfully layered persona manipulation on top of the prefix injection.

When successful, the manipulated models generated highly functional exploit codes, such as Cross-Site Scripting payloads, which they would typically refuse to write.

Beyond generating malicious code, the exploit also proved highly effective at triggering severe system prompt leakage.

Attackers combined the prefix injection with adversarial token sequences, forcing the AI to leak its complete internal metadata and verbatim system instructions.

In some instances, the AI even hallucinated detailed internal configuration structures, exposing the severe risks of allowing unchecked assistant prefill inputs.

During testing against 11 different AI models, researchers found that every model accepting the assistant prefill was at least partially vulnerable.

According to Trend Micro, the data reveals a stark contrast in attack success rates (ASR) between models that accept the prefill versus those protected by API-level blocking.

Resistance in Advanced Models

Interestingly, while GPT-4o and its mini counterpart accepted the prefill, their advanced safety training made them highly resistant.

GPT-4o-mini recorded a mere 0.5% success rate, showcasing that robust internal alignment can partially override the AI’s urge to remain self-consistent.

However, attackers found that framing malicious requests as benign JSON formatting tasks could still occasionally bypass these strong defenses.

The most effective way to neutralize sockpuppeting is to block the manipulated inputs before they ever reach the AI model.

Security teams can implement strict message-ordering validation at the API layer, ensuring that the final message in any request always comes from the user.

Major providers like OpenAI, AWS Bedrock, and Anthropic have already deployed this exact defense to protect their latest models by simply returning an error for any prefilled requests.

Despite provider-level protections, organizations running their own self-hosted inference servers remain fully exposed.

Platforms like Ollama or vLLM do not enforce message validation by default, leaving the door open for these prefix injection attacks.

Any team deploying an AI model must independently verify that their API layer actively blocks prefilled assistant messages to secure their systems against this simple but devastating technique.

Follow us on Google News, LinkedIn, and X to Get Instant Updates and Set GBH as a Preferred Source in Google.