Tuesday, April 29, 2025
HomeCyber AIResearchers Uncovered a New Flaw in ChatGPT to Turn Them Evil

Researchers Uncovered a New Flaw in ChatGPT to Turn Them Evil

Published on

SIEM as a Service

Follow Us on Google News

LLMs are commonly trained on vast internet text data, often containing offensive content. To mitigate this, developers use “alignment” methods via finetuning to prevent harmful or objectionable responses in recent LLMs.

ChatGPT and AI siblings were fine-tuned to avoid undesirable messages like hate speech, personal info, or bomb-making instructions.

However, security researchers from the following universities showed recently how a simple prompt addition breaks defenses in multiple popular chatbots:-

- Advertisement - Google News
  • Carnegie Mellon University (Andy Zou, J. Zico Kolter, Matt Fredrikson)
  • Center for AI Safety (Zifan Wang)
  • Bosch Center for AI (J. Zico Kolter)

New Flaw in AI Chatbots

Non-adversarially aligned LLMs fall victim to a single universal adversarial prompt, evading state-of-the-art commercial models, including:-

  • ChatGPT
  • Claude
  • Bard
  • Llama-2 

These outputs prove potential misuse with high probability, achieved by the “Greedy Coordinate Gradient” attack on smaller open-source LLMs.

Flow chain (Source – Arxiv)

New adversarial attacks exploit aligned language models to generate objectionable content by adding an adversarial suffix to user queries. 

However, the attack’s success lies in the careful combination of three key elements, previously seen in the theories but now reliably effective in practice.

Here below we have mentioned those three key elements:-

  • Initial affirmative responses.
  • Combined greedy and gradient-based discrete optimization.
  • Robust multi-prompt and multi-model attacks.

Clever AI chatbots’ tendency to go off the rails is not a minor problem but a fundamental weakness, challenging advanced AI deployment.

Adding specific information prompts the chatbots to generate harmful responses which bypasses the restrictions and leads to disallowed content.

Researchers alerted OpenAI, Google, and Anthropic of the exploit before publishing the findings. While the companies blocked specific exploits but still struggle to prevent adversarial attacks overall. 

Negative ChatGPT Prompt (Source – Arxiv)

Since Kolter discovered strings affecting ChatGPT and Bard, claiming to possess thousands of such strings.

Anthropic actively research stronger defenses against prompt injection and adversarial measures. They aim to make base models safer and explore additional layers of protection. 

While the OpenAI’s ChatGPT and similar models completely rely on vast language data to predict such characters.

Language models excel in generating intelligent output but are prone to discrimination and fabricating information. 

Adversarial attacks exploit data patterns, causing aberrant behaviors, like misidentification in image classifiers or responding to inaudible messages in speech recognition. The attack highlights the inevitability of AI misuse. 

AI safety experts should focus on safeguarding vulnerable systems like social networks from AI-generative disinformation rather than solely trying to “align” models.

Also Read

Keep yourself informed about the latest Cyber Security News by following us on GoogleNews, Linkedin, Twitter, and Facebook.

Tushar Subhra
Tushar Subhra
Tushar is a Cyber security content editor with a passion for creating captivating and informative content. With years of experience under his belt in Cyber Security, he is covering Cyber Security News, technology and other news.

Latest articles

Researchers Uncover SuperShell Payloads and Various Tools in Hacker’s Open Directories

Cybersecurity researchers at Hunt have uncovered a server hosting advanced malicious tools, including SuperShell...

Cyber Espionage Campaign Targets Uyghur Exiles with Trojanized Language Software

A sophisticated cyberattack targeted senior members of the World Uyghur Congress (WUC), the largest...

Konni APT Deploys Multi-Stage Malware in Targeted Organizational Attacks

A sophisticated multi-stage malware campaign, potentially orchestrated by the North Korean Konni Advanced Persistent...

Outlaw Cybergang Launches Global Attacks on Linux Environments with New Malware

The Outlaw cybergang, also known as “Dota,” has intensified its global assault on Linux...

Resilience at Scale

Why Application Security is Non-Negotiable

The resilience of your digital infrastructure directly impacts your ability to scale. And yet, application security remains a critical weak link for most organizations.

Application Security is no longer just a defensive play—it’s the cornerstone of cyber resilience and sustainable growth. In this webinar, Karthik Krishnamoorthy (CTO of Indusface) and Phani Deepak Akella (VP of Marketing – Indusface), will share how AI-powered application security can help organizations build resilience by

Discussion points


Protecting at internet scale using AI and behavioral-based DDoS & bot mitigation.
Autonomously discovering external assets and remediating vulnerabilities within 72 hours, enabling secure, confident scaling.
Ensuring 100% application availability through platforms architected for failure resilience.
Eliminating silos with real-time correlation between attack surface and active threats for rapid, accurate mitigation

More like this

Researchers Uncover SuperShell Payloads and Various Tools in Hacker’s Open Directories

Cybersecurity researchers at Hunt have uncovered a server hosting advanced malicious tools, including SuperShell...

Cyber Espionage Campaign Targets Uyghur Exiles with Trojanized Language Software

A sophisticated cyberattack targeted senior members of the World Uyghur Congress (WUC), the largest...

Konni APT Deploys Multi-Stage Malware in Targeted Organizational Attacks

A sophisticated multi-stage malware campaign, potentially orchestrated by the North Korean Konni Advanced Persistent...