Developers Beware! Malicious ML Models Found on Hugging Face Platform

In a concerning development for the machine learning (ML) community, researchers from ReversingLabs have uncovered malicious ML models on the Hugging Face platform, a popular hub for AI collaboration.

Dubbed “nullifAI,” this novel attack method leverages vulnerabilities in the widely used Python Pickle serialization format to execute malicious code on unsuspecting systems.

The discovery highlights growing cybersecurity risks in AI development as attackers exploit open-source platforms to distribute malware.

Pickle Files: A Double-Edged Sword

Pickle files are commonly used by Python developers to serialize and deserialize ML models, enabling easy sharing and reuse of pre-trained models.

However, their ability to execute arbitrary Python code during deserialization makes them inherently risky.

Despite warnings in Hugging Face’s documentation and the implementation of a security tool called Picklescan, attackers have managed to bypass these safeguards by embedding malicious payloads into corrupted Pickle files.

The ReversingLabs team identified two such models on Hugging Face that contained a reverse shell payload, allowing attackers to gain remote access to compromised systems.

These models were compressed using non-standard formats like 7z, which prevented detection by default security tools.

The payloads were strategically placed at the beginning of the serialized data stream, ensuring execution before any security checks could flag them as unsafe.

Security Implications for Developers

This incident underscores a critical gap in current security measures for collaborative AI platforms.

While Hugging Face has implemented scanning tools like Picklescan, these rely on blacklists of known dangerous functions and fail to account for evolving threats or corrupted files.

The inability of Picklescan to detect malicious functions in broken Pickle files further exacerbates the risk.

Researchers warn that such vulnerabilities could lead to severe consequences, including data breaches, system corruption, and unauthorized access to sensitive environments.

With over 100 instances of malicious ML models reported on Hugging Face in recent months, the threat is far from isolated.

Hugging Face has taken steps to address these issues by removing the identified malicious models within 24 hours of notification and updating its security tools to better detect corrupted files.

Additionally, the platform is promoting the adoption of Safetensors, a secure serialization format that eliminates executable code in model files.

For developers, experts recommend exercising caution when downloading models from open-source platforms.

Security best practices include:

• Verifying the source and integrity of downloaded models.
• Avoiding reliance on inherently unsafe formats like Pickle when possible.
• Incorporating advanced scanning tools into MLOps workflows to detect potential threats proactively.

The discovery of nullifAI serves as a wake-up call for the AI community to prioritize security in an era where open-source collaboration is both a strength and a vulnerability.

As AI continues to drive innovation across industries, ensuring robust safeguards against malicious actors will be essential to maintaining trust and integrity in the ecosystem.

Investigate Real-World Malicious Links & Phishing Attacks With Threat Intelligence Lookup - Try for Free