AI Package Hallucination – Hackers Abusing ChatGPT, Gemini to Spread Malware

The research investigates the persistence and scale of AI package hallucination, a technique where LLMs recommend non-existent malicious packages. 

The Langchain framework has allowed for the expansion of previous findings by testing a more comprehensive range of questions, programming languages (Python, Node.js, Go,.NET, and Ruby), and models (GPT-3.5-Turbo, GPT-4, Bard, and Cohere). 

The aim is to assess if hallucinations persist, generalize across models (cross-model hallucinations), and occur repeatedly (repetitiveness). 

2500 questions were refined to 47,803 “how-to” prompts fed to the models, while repetitiveness was tested by asking 20 questions with confirmed hallucinations 100 times each.  

A study compared four large language models (LLMs)—GPT-4, GPT-3.5, GEMINI, and COHERE—for their susceptibility to generating hallucinations (factually incorrect outputs). 

GEMINI produced the most hallucinations (64.5%), while COHERE had the least (29.1%). Interestingly, hallucinations with potential for exploitation were rare due to factors like decentralized package repositories (GO) or reserved naming conventions (.NET). 

Lasso Security’s study also showed that GEMINI and GPT-3.5 had the most common hallucinations, which suggests that their architectures may be similar on a deeper level. This information is essential for understanding and reducing hallucinations in LLMs. 

Run Free ThreatScan on Your Mailbox

AI-Powered Protection for Business Email Security

Trustifi’s Advanced threat protection prevents the widest spectrum of sophisticated attacks before they reach a user’s mailbox. Try Trustifi Free Threat Scan with Sophisticated AI-Powered Email Protection .

Run Free Threat Scan

Multiple large language models (LLMs) have been used to study hallucinations. This is done by finding nonsensical outputs (hallucinated packages) in each model and then comparing these hallucinations to see what they have in common.

Multiple LLM analyses reveal 215 packages, with the highest overlap between Gemini and GPT-3.5 and the least between Cohere and GPT-4. 

This cross-model hallucination analysis offers valuable insights into the phenomenon of hallucinations in LLMs, potentially leading to a better understanding of these systems’ internal workings.  

There was a phenomenon where developers were unknowingly downloading a non-existent Python package called “huggingface-cli,”  which suggested a potential issue where large language models might be providing users with inaccurate information about available packages. 

To investigate further, the researchers uploaded two dummy packages: “huggingface-cli” (empty) and “blabladsa123” (also empty). 

They then monitored download rates over three months; the fake “huggingface-cli” package received over 30,000 downloads, significantly exceeding the control package “blabladsa123.”. 

It suggests a possible vulnerability where developers rely on incomplete or inaccurate information sources to discover Python packages. 

The adoption rate of a package was believed to be a hallucination (not an actual package), and to verify its usage, they searched GitHub repositories of major companies as the search identified references to the package in repositories of several large companies. 

For example, a repository containing Alibaba’s research included instructions on installing this package in its README file. 

These findings suggest that either the package is accurate and used by these companies or there’s a widespread phenomenon of including instructions for non-existent packages in documentation. 

Are you from SOC and DFIR Teams? – Analyse Malware Incidents & get live Access with ANY.RUN -> Start Now for Free