Why Red Team Operators Should Think Twice Before Using Commercial AI Tools

Red teaming runs on being discrete. Everything, from initial reconnaissance to exploitation, depends on information staying with only the people who are supposed to have it.

And yet, more and more, professionals in the field are sharing prompts full of sensitive client information with a cloud server owned by a company whose data policy runs to forty-odd pages of carefully worded terms and conditions that probably works against the user.

The most popular commercial AI tools collect conversations by default to improve their models. Some let you disable this, but that’s generally an ‘opt out’ option that most users forget (or don’t realise they should be doing). 

The result? All of that sensitive information about the client’s infrastructure ends up logged on servers that are outside the user’s – and the client’s – control.

It poses an immediate new privacy risk that will easily go unnoticed, but the even more immediate problem is contractual, since most NDAs in red team engagements explicitly prohibit sharing client information with third parties. 

Using ChatGPT, Copilot, or Gemini with data from the target environment could technically constitute a breach of that agreement, even if it feels totally private at the time.

Part of the problem is the speed with which LLMs have become a key part of our everyday life.

Whereas, before, we would generally search broad queries on Google that didn’t explicitly identify a client, their infrastructure, or any other sensitive information, LLMs enable us to flesh-out queries with plenty of context and unique information.

We can add more and more, while the LLm continues to ask probing questions that may seem – and may well be – purely inquisitive, but that mean, over time, an entire project is shared with a third party.

But that’s only part of the problem. The key issue here is that LLMs are set up in a way that promotes trust from users, while storing and using shared data and entire chats in ways that are only spelled out in the dense terms and conditions that nobody ever looks at anyway. 

What’s the solution? The extreme option is to rule-out the use of AI tools completely, keeping all information securely behind company walls and reverting back to a form of life that feels almost archaic at this point in the AI revolution.

It’s certainly doable, but with accelerating workloads and the standardisation of AI tools across industries, that’s bound to put you behind your competitors. 

The alternative? Switching to an AI assistant that has been designed with privacy and data security at the forefront, rather than just a formality that has to be adhered to.

Encryption should be a non-negotiable for companies utilising third-party tools that require large amounts of sensitive data, and along with chats not being logged, data not being shared with anyone else, and open source code for full visibility.

Too many of the most prominent LLMs are unable to offer any of this, because it isn’t hardwired into their business models. 

AI usage isn’t the problem for red team operators. The problem stems from using the wrong AIs tools, and handing confidential information over as if no one else is going to see or learn from it.