Hackers Can Hijack Your Terminal Via Prompt Injection using LLM-powered Apps

Researchers have uncovered that Large Language Models (LLMs) can generate and manipulate ANSI escape codes, potentially creating new security vulnerabilities in terminal-based applications.

ANSI escape sequences are a standardized set of control characters used by terminal emulators to manipulate the appearance and behavior of text displays.

They enable features such as text color changes, cursor movement, blinking text, and more. Terminal emulators interpret these sequences to provide dynamic functionality, but they’ve also historically been a source of vulnerabilities.

This discovery, initially reported by Leon Derczynski and further investigated by security researchers, raises important concerns about the security of LLM-integrated command-line tools.

Leveraging 2024 MITRE ATT&CK Results for SME & MSP Cybersecurity Leaders – Attend Free Webinar

ANSI escape codes, which are special character sequences used to control terminal behavior, can be exploited by LLMs in several concerning ways:

• Generating flashing text and color changes
• Manipulating cursor position and screen content
• Creating hidden text in responses
• Copying text to clipboards without user consent
• Executing denial of service attacks
• Creating potentially malicious clickable hyperlinks
• Triggering DNS requests on macOS systems

To test these vulnerabilities, a researcher created a Python-based app, dillma.py, that integrates with LLMs.

In one demonstration, a malicious file was fed into the app, which then used ANSI codes to produce flashing, colored text and auditory beeps in the terminal.

Another test showcased how LLM-generated output could add clickable links that potentially leak user data, particularly in environments like Visual Studio Code’s terminal, which supports hyperlink rendering.

Security experts have demonstrated that these vulnerabilities can be exploited through two primary methods: direct prompting of LLMs to generate control characters, and utilizing code interpreter tools.

The implications are particularly serious for LLM-powered CLI applications that don’t properly sanitize their output.

“It’s important for developers and application designers to consider the context in which they insert LLM output, as the output is untrusted and could contain arbitrary data,” notes the research.

This discovery follows previous findings about Unicode Tags enabling hidden communication in web applications, suggesting a pattern of legacy features creating unexpected attack surfaces in AI applications.

To address these security concerns, researchers recommend implementing several protective measures:

• Encoding ANSI control characters by default
• Adding specific options to enable control characters when necessary
• Implementing allow-listing for approved characters
• Conducting thorough end-to-end testing of applications

This discovery serves as a reminder that as AI technology continues to evolve, security practitioners must remain vigilant about potential vulnerabilities, especially when integrating LLMs with existing systems and protocols.

The research community expects more hidden vulnerabilities to be discovered as investigation into LLM security continues, highlighting the ongoing need for robust security measures in AI-powered applications.

Investigate Real-World Malicious Links,Malware & Phishing Attacks With ANY.RUN - Tree for Free