Friday, May 24, 2024

AI Package Hallucination – Hackers Abusing ChatGPT, Gemini to Spread Malware

The research investigates the persistence and scale of AI package hallucination, a technique where LLMs recommend non-existent malicious packages. 

The Langchain framework has allowed for the expansion of previous findings by testing a more comprehensive range of questions, programming languages (Python, Node.js, Go,.NET, and Ruby), and models (GPT-3.5-Turbo, GPT-4, Bard, and Cohere). 

The aim is to assess if hallucinations persist, generalize across models (cross-model hallucinations), and occur repeatedly (repetitiveness). 

Langchain Default Prompt
Langchain Default Prompt

2500 questions were refined to 47,803 “how-to” prompts fed to the models, while repetitiveness was tested by asking 20 questions with confirmed hallucinations 100 times each.  

Results of GPT 4
Results of GPT 4

A study compared four large language models (LLMs)—GPT-4, GPT-3.5, GEMINI, and COHERE—for their susceptibility to generating hallucinations (factually incorrect outputs). 

GEMINI produced the most hallucinations (64.5%), while COHERE had the least (29.1%). Interestingly, hallucinations with potential for exploitation were rare due to factors like decentralized package repositories (GO) or reserved naming conventions (.NET). 

Results of Gemini
Results of Gemini

Lasso Security’s study also showed that GEMINI and GPT-3.5 had the most common hallucinations, which suggests that their architectures may be similar on a deeper level. This information is essential for understanding and reducing hallucinations in LLMs. 

Run Free ThreatScan on Your Mailbox

AI-Powered Protection for Business Email Security

Trustifi’s Advanced threat protection prevents the widest spectrum of sophisticated attacks before they reach a user’s mailbox. Try Trustifi Free Threat Scan with Sophisticated AI-Powered Email Protection .

Multiple large language models (LLMs) have been used to study hallucinations. This is done by finding nonsensical outputs (hallucinated packages) in each model and then comparing these hallucinations to see what they have in common.

Multiple LLM analyses reveal 215 packages, with the highest overlap between Gemini and GPT-3.5 and the least between Cohere and GPT-4. 

Result of Cross-Model hallucinations
Result of Cross-Model hallucinations

This cross-model hallucination analysis offers valuable insights into the phenomenon of hallucinations in LLMs, potentially leading to a better understanding of these systems’ internal workings.  

There was a phenomenon where developers were unknowingly downloading a non-existent Python package called “huggingface-cli,”  which suggested a potential issue where large language models might be providing users with inaccurate information about available packages. 

Screenshot of ChatGPT
Screenshot of ChatGPT

To investigate further, the researchers uploaded two dummy packages: “huggingface-cli” (empty) and “blabladsa123” (also empty). 

They then monitored download rates over three months; the fake “huggingface-cli” package received over 30,000 downloads, significantly exceeding the control package “blabladsa123.”. 

fake and empty package got more than 30k authentic downloads
fake and empty package got more than 30k authentic downloads

It suggests a possible vulnerability where developers rely on incomplete or inaccurate information sources to discover Python packages. 

The adoption rate of a package was believed to be a hallucination (not an actual package), and to verify its usage, they searched GitHub repositories of major companies as the search identified references to the package in repositories of several large companies. 

installing the packages found in the README
installing the packages found in the README

For example, a repository containing Alibaba’s research included instructions on installing this package in its README file. 

These findings suggest that either the package is accurate and used by these companies or there’s a widespread phenomenon of including instructions for non-existent packages in documentation. 

Are you from SOC and DFIR Teams? – Analyse Malware Incidents & get live Access with ANY.RUN -> Start Now for Free


Latest articles

Hackers Weaponizing Microsoft Access Documents To Execute Malicious Program

In multiple aggressive phishing attempts, the financially motivated organization UAC-0006 heavily targeted Ukraine, utilizing...

Microsoft Warns Of Storm-0539’s Aggressive Gift Card Theft

Gift cards are attractive to hackers since they provide quick monetization for stolen data...

Kinsing Malware Attacking Apache Tomcat Server With Vulnerabilities

The scalability and flexibility of cloud platforms recently boosted the emerging trend of cryptomining...

NSA Releases Guidance On Zero Trust Maturity To Secure Application From Attackers

Zero Trust Maturity measures the extent to which an organization has adopted and implemented...

Chinese Hackers Stay Hidden On Military And Government Networks For Six Years

Hackers target military and government networks for varied reasons, primarily related to spying, which...

DNSBomb : A New DoS Attack That Exploits DNS Queries

A new practical and powerful Denial of service attack has been discovered that exploits...

Malicious PyPI & NPM Packages Attacking MacOS Users

Cybersecurity researchers have identified a series of malicious software packages targeting MacOS users.These...
BALAJI is an Ex-Security Researcher (Threat Research Labs) at Comodo Cybersecurity. Editor-in-Chief & Co-Founder - Cyber Security News & GBHackers On Security.

Free Webinar

Live API Attack Simulation

94% of organizations experience security problems in production APIs, and one in five suffers a data breach. As a result, cyber-attacks on APIs increased from 35% in 2022 to 46% in 2023, and this trend continues to rise.
Key takeaways include:

  • An exploit of OWASP API Top 10 vulnerability
  • A brute force ATO (Account Takeover) attack on API
  • A DDoS attack on an API
  • Positive security model automation to prevent API attacks

Related Articles