Copilot, Studio bots are woefully insecure, says Zenity CTO

Black Hat One hopes widely used enterprise software is secure enough. Get ready for those hopes to be dashed again, as Zenity CTO Michael Bargury today revealed his Microsoft Copilot exploits at Black Hat.

“It’s actually very difficult to create a [Copilot Studio] bot that is safe,” Bargury told The Register in an interview ahead of his conference talks, “because all of the defaults are insecure.” 

Bargury is speaking twice about security failings with Microsoft Copilot at Black Hat in Las Vegas this week. His first talk focused on the aforementioned Copilot Studio, Microsoft’s no-code tool for building custom enterprise Copilot bots. The second covered all the nasty things an attacker can do with Copilot itself if they manage to break into the systems of an organization that uses the tech, as well as how to use Copilot to gain that initial access. 

Zenity, for what it’s worth, offers among other things security controls for Copilot and similar enterprise-level assistants. Bear that in mind. It warns of the risks of using Microsoft’s AI services here.

Your Copilot bots are quite chatty

If you don’t have much exposure to Copilot Studio, it’s a tool for non-technical people to create simple conversational bots, using Microsoft’s Copilot AI, that can answer people’s questions using internal business documents and data. This is made possible by what’s called retrieval-augmented generation, or RAG.

It’s Microsoft’s way “to extend [Copilot’s] tentacles into other business areas, such as CRM and ERP,” as we wrote here. Companies can create customer and/or employee-facing bots that provide a natural-language interface to internal information.

Unfortunately for all the Copilot Studio customers out there, we’re told the default settings in the platform are entirely insufficient. Combine those with what Zenity marketing chief Andrew Silberman told us is nearly 3,000 Copilot Studio bots in the average large enterprise (we’re talking Fortune 500-level companies), along with research indicating that 63 percent of those are discoverable online, and you have a potential recipe for a data exfiltration.

Specifically, if these bots are accessible to the public, and we’re told a good number of them are, they can be potentially tricked into handing over, or simply hand over by design, information to people that should not have been volunteered during conversations, it’s claimed.

As Copilot bots frequently have access to internal company data and sensitive documents, it’s a matter of figuring out how to fool or prompt them into disclosing that data, we’re told. Bargury said he was able to do that by configuring ChatGPT to fuzz Copilot bots with automated, malformed prompts.

“We scanned the internet and found tens of thousands of these bots,” Bargury said. He blamed the high online availability of these agents on default Copilot Studio settings that published them to the web without any need to authenticate to access them – an oversight Microsoft has since fixed after the Zenity team brought it to their attention. 

Unfortunately, new default settings that keep Copilot Studio bots off the public internet by default currently only apply to new installations, Bargury said, so users of the suite who installed it before now should check their deployments to be sure.

Bargury and his team have released a new tool to detect and exploit Copilot bot vulnerabilities. Dubbed CopilotHunter, it’s now available as a module in PowerPwn, a tool Zenity released at Black Hat last year for testing abuses of Microsoft 365 guest accounts. 

Copilot, please breach my target for me

While Bargury told The Reg he may have overextended himself by planning two Black Hat talks this year, his second shows no less effort – or devastating effect – than the first. 

Copilot, Bargury demonstrated this week, is quite susceptible to indirect prompt injection attacks, which he argues rise to the severity of remote code execution (RCE) when performed against an enterprise target with access to sensitive data. 

“An RCE is simply, from a remote location, being able to execute code that does something on your machine,” Bargury said. “Indirect prompt injection that makes an AI do something on your behalf is the exact same thing with the same impact.” 

With access to a compromised environment, Bargury said he can jailbreak Copilot, make it visit phishing sites to force it to feed malicious information to users, control references, display arbitrary information while secretly exfiltrating encrypted data, conduct operations without user approval and the like. 

To top it all off, Copilot can also be tricked into granting initial access to a network, and conduct other malicious activities, with nothing but an email, direct message, calendar invite or other common phishing tactic, but this one even works without the user needing to interact with it or click a link because of how Copilot scans messages. 

“Microsoft Copilot is built on the enterprise graph,” Bargury explained. Once a message, email or invite is sent it hits the graph, Copilot scans it, “and that’s a path for me to start with prompt injection.” 

In one example, Bargury demonstrated how he was able to change banking information to intercept a bank transfer between a company and client “just by sending an email to the person.” 

An AI bot feature

Bargury explained to us that he sees these discoveries as indicative of the industry still being in the very early days of artificial intelligence in the enterprise, and having to face the fact that AI is changing our relationship with data. 

“There’s a fundamental issue here,” he said. “When you give AI access to data, that data is now an attack surface for prompt injection.” 

When you give AI access to data, that data is now an attack surface for prompt injection

If that’s true, Copilot bots are by their very nature insecure since many are publicly accessible, they’re tied closely to enterprise data, and are ready to spill secrets with a bit of hidden HTML or a ChatGPT-powered fuzzing bot. 

“It’s kind of funny in a way – if you have a bot that’s useful, then it’s vulnerable. If it’s not vulnerable, it’s not useful,” Bargury said. 

The Zenity CTO noted that Microsoft has been incredibly responsive to his reports, and said several of the faults he found have been addressed, albeit within limits.

“[AI] apps are basically changing in production because AI chooses to do what it wants, so you can’t expect to have a platform that’s just secure and that’s it,” Bargury said. “That’s not going to happen because these platforms have to be flexible, otherwise they’re not useful.” 

If you have a bot that’s useful, it’s vulnerable. If it’s not vulnerable, it’s not useful

Bargury believes that securing AI software like Copilot requires real-time monitoring of memory, monitoring conversations and tracking potential prompt-injection RCEs, but even that can be difficult in closed-off enterprise environments. 

The bottom line is that businesses are the guinea pigs testing an experimental drug called “artificial intelligence,” and we’re not at a point where we know how to make it safe yet.

Bargury and team have released another testing kit called “LOLCopilot” for organizations that want to test their setups for vulnerability to his exploits. 

“Copilot has great skills. It can search, it can enable your employees to find data they have access to but didn’t know they did … those things are important,” Bargury told us. “But that’s not as important as preventing remote code execution.”

We’re seeking a response from Microsoft direct about Zenity’s findings, and will let you know if we hear back from the Windows giant. ®

Updated to add

Spokespeople for Microsoft have been in touch to tell us the Azure giant believes it can keep its Copilot users protected by itself, thank you, though it appreciated Zenity’s reports:

As we understand it, Bargury disclosed various ways to meddle with Copilot at Black Hat, some that Microsoft fixed such as the insecure defaults, and some that require compromising a victim’s environment. Redmond acknowledged as much, telling us: “Similar to other post-compromise techniques, these methods require prior compromise of a system or social engineering.”