OWASP Released Top 10 Critical Vulnerabilities for LLMs(AI models)

OWASP Foundation has released the 0.9.0 version of Critical Vulnerabilities in LLMs (Large Language Models).

A groundbreaking initiative has emerged to address the pressing need for educating developers, designers, architects, and other professionals involved in AI models.

AI-based technologies are currently being developed across various industries with the goal of revolutionizing long-standing traditional methods that have been in use for over three decades.

The scope of these projects is not just to ease the work but also to learn the potential capabilities of these AI-based models.

Organizations working on AI-based projects must understand the potential risks they can create and work on preventing the loopholes in the near future.

Open Web Application Security Project (OWASP) has expanded its focus beyond web applications to release its inaugural Top 10 Critical Vulnerabilities for Large Language Models (LLMs) and broader AI models.

This list serves as a testament to the burgeoning importance and potential risks associated with the deployment of AI, particularly as it becomes a cornerstone in diverse industries from healthcare to finance.

Threat actors leverage every piece of information they collect to conduct cybercriminal activities.

https://gbhackers.com/owasp-zap-2-13-0

Table of Contents

OWASP Top-10 for LLMs
LLM01: Prompt Injection
LLM02: Insecure Output Handling
LLM03: Training Data Poisoning
LLM04: Model Denial of Service
LLM05: Supply Chain Vulnerabilities
LLM06: Sensitive Information Disclosure
LLM07: Insecure Plugin Design
LLM08: Excessive Agency
LLM09: Overreliance
LLM10: Model Theft
Conclusion

OWASP Top-10 for LLMs

As per the recent publishing of the OWASP 0.9.0 version, the top 10 critical vulnerabilities are as follows,

LLM01: Prompt Injection

This vulnerability arises if an attacker manipulates an LLM’s operation through crafted inputs, resulting in the attacker’s intention to get executed.

There are two types of prompt injections direct prompt injection and indirect prompt injection.

• Direct Prompt Injection
• Indirect Prompt Injection

Direct Prompt Injection which is otherwise called “jailbreaking” arises if an attacker overwrites or reveals the underlying system prompt resulting in the attacker interacting with insecure functions and data stores that are accessible by the LLM.

Indirect Prompt Injection occurs if the LLM accepts external source inputs that are controlled by the attacker resulting in the conversation being hijacked by the attacker.

This can give the attacker the ability to ask the LLM for sensitive information and can get severe like manipulating the decision-making process.

LLM02: Insecure Output Handling

An Insecure Output Handling vulnerability is a form of prompt injection vulnerability that occurs when a plugin or application accepts large language model (LLM) output without sufficient scrutiny and then directly feeds it to backend, privileged, or client-side operations.

This type of vulnerability can result in a security breach. This behavior is analogous to providing users with indirect access to more functionality.

This is due to the fact that LLM-generated material can be controlled by prompt input.

Exploitation of a vulnerability known as Insecure Output Handling that is successful can lead to cross-site scripting (XSS) and cross-site request forgery (CSRF) in web browsers, as well as SSRF, privilege escalation, or remote code execution on backend systems.

The severity of this vulnerability grows when the application permits LLM material to carry out operations that are beyond the scope of what the intended user is authorized to do.

Additionally, this can be used with other types of attacks, such as agent hijacking attacks, to grant an attacker privileged access to the environment of a target user.

LLM03: Training Data Poisoning

This vulnerability occurs if an attacker or unaware client poisons the training data, which can result in providing backdoors, and vulnerabilities or even compromise the LLM’s security, effectiveness, or ethical behavior.

Large language models, also known as LLMs, make use of a wide variety of source text in order to learn and produce outputs.

Nevertheless, training data poisoning, which occurs when an adversary inserts flaws, might corrupt the model, leaving users vulnerable to receiving wrong information.

The OWASP List for LLMs draws attention to the potential danger of placing excessive reliance on AI-generated content.

Common Crawl, which is utilized for models like T5 and GPT-3; WebText and OpenWebText, which contain public news and Wikipedia; and books, which make up 16% of GPT-3’s training data. These are some of the most important data sources.

LLM04: Model Denial of Service

An attacker with potential skills or a method can interact with the LLM model to make it consume a high amount of resources resulting in exceptionally high resource costs. It can also result in the decline of the quality of service of the LLM.

LLM05: Supply Chain Vulnerabilities

This vulnerability arises if the supply-chain vulnerabilities in LLM applications affect the entire application lifecycle including third-party libraries, docker containers, base images, and service suppliers.

The supply chain in LLMs can be susceptible to vulnerabilities, which can compromise the integrity of training data, machine learning models, and deployment platforms, and result in biased results, security breaches, or even entire system failures.

Traditionally, vulnerabilities concentrated on software components; but, with AI, this focus has expanded because of the prevalence of transfer learning, the re-use of pre-trained models, and crowdsourcing data.

This vulnerability can also manifest itself in public LLMs like OpenGPT’s extension plugins, which are another area of potential exposure.

LLM06: Sensitive Information Disclosure

This vulnerability arises if the LLM reveals sensitive information, proprietary algorithms, or other confidential details by accident, resulting in unauthorized access to Intellectual Property, piracy violations, and other security breaches.

LLM07: Insecure Plugin Design

LLM plugins have less application control as they are called by the LLMs and are automatically invoked in context and chained. Insecure plugin Design is characterized by insecure inputs and insufficient access control.

LLM08: Excessive Agency

This vulnerability arises when the LLMs are capable of performing damaging actions due to unexpected outputs from the LLMs. The root cause of this vulnerability is excessive permission, functionalities, or autonomy.

LLM09: Overreliance

This vulnerability arises when the LLMs are relied on for decision-making or content generation without proper oversight.

Though LLMs can be creative and informative, they are still in the developmental phase and provide false or inaccurate information. If used without a background check, this can result in reputational damage, legal issues, or miscommunication.

LLM10: Model Theft

This refers to unauthorized access and exfiltration of LLMs when threat actors compromise, physically steal, or perform theft of intellectual property.

This can result in economic losses, unauthorized usage of the model, or unauthorized access to sensitive information. 

OWASP has released a complete report about these vulnerabilities which must be given a high priority for organisations that are developing or using LLMs. It is recommended for all organizations to take security into consideration when building application development lifecycles.

Conclusion

The move taken by OWASP to draw attention to vulnerabilities in LLMs is an important milestone in the progression of the technological landscape.

As artificial intelligence continues on its path of transforming industries, common knowledge of its vulnerabilities and the methods to prevent them will ensure that its benefits are realized without compromising security or ethics.

Such a list, if ever officially created, would be intended to guide AI researchers, developers, and stakeholders in identifying and addressing the primary security and ethical considerations related to deploying LLMs in real-world scenarios.

Always consult the official OWASP website or trusted AI research communities for the most recent updates.

FAQ

1.Who is the primary audience for the OWASP Top 10 for LLMs?

The OWASP (Open Web Application Security Project) Top 10 for LLMs (Legal, LegalTech, and Legal InfoSec workers) is mostly for the following people:

• Lawyers and other legal professionals who work on cases or advice connected to technology.
• LegalTech workers are in charge of making or putting in place legal software solutions.
• Legal Information Security (InfoSec) experts worked to protect private legal information.
• Anyone who works in a law company or legal department and wants to understand and reduce possible technology risks.
• Regulatory bodies and lawmakers are making rules for legal technologies that are safe.
• Legal educators and trainers are making sure that the next crop of lawyers knows how to use technology.

2. What is the OWASP Top 10 for Large Language Models (LLMs)?

The Open Web Application Security Project (OWASP) has come up with a list of the ten most important security risks to web applications. Large Language Models like GPT-4 are based on natural language processing, which is different from web application security.

• OWASP is mostly concerned with flaws in web applications.
• Large word Models are systems that work with natural words.
• There isn’t an “OWASP Top 10” for LLMs that everyone knows about.
• It’s important to make a clear distinction between web security and ethics or worries about AI.
• Concerns like data bias and misuse are unique to LLMs.
• Concerns about LLM might be addressed in future versions of OWASP or by other groups.

3. Will the OWASP Top 10 for LLMs be updated in the future?

There was no special “OWASP Top 10 for Large Language Models (LLMs)”. But, just to play with words:

• If OWASP or another group made such a list, it’s likely that it would be updated.
• Threat areas and technologies are always changing, so it’s important to look at them every so often.
• The best ways to handle security and ethics in LLM would change over time.
• Guidelines would change based on what the community says and what new study finds.
• Because LLMs are becoming more important in the tech world, they might need to be updated more often.
• Like the standard OWASP Top 10 for web apps, a list based on LLM would need to be updated regularly to stay useful.