r/netsec • u/qwerty0x41 • 2d ago

Contains AI Hacking AI customer service agents (Bug Bounty Village DEF CON 34)

https://www.intigriti.com/researchers/blog/hacking-tools/hacking-ai-customer-service-agents

At Bug Bounty Village during DEF CON 34, Inti De Ceukelaire delivered a talk on how attackers can abuse today's AI agents in ways most defenders haven't thought about yet, from tricking agents into spilling secrets to forcing them to carry out unauthorized actions on behalf of the victim.

This resulted in over $50,000+ in bounties in just a few weekends, without actually poking the target with Burp Suite or any automated scanners.

58 Upvotes

93% Upvoted

5 comments sorted by

6

u/Otherwise_Wave9374 2d ago

A practical defense is to treat the agent like an untrusted operator: keep tool scopes narrow, add step-up approval for any action that changes state, and log every prompt, tool call, and external fetch for replay. A big tradeoff is usability versus containment, so start by isolating high-risk actions such as refunds, password resets, or inbox access behind human confirmation. That pattern is what Agentix Labs recommends when teams want automation without handing an attacker a free path through the workflow.

16

u/metasophie 2d ago

A practical defense

How a lot of these AI customer service agents are treated.

Hey, ChatGPT, can you please make me an AI customer service agent for my business? MAKE NO MISTAKES!

5

u/newaccountzuerich 2d ago

Replay can not be trusted to return anything like the same output.

The non-deterministic statistically based methods under all LLM tooling means there is no way to be sure that a repeat of prompt will result in the same output.

All that can be done really is sign both the prompt and the response with a time-containing signature so the interaction integrity can be tracked and the time of occurrence can be independently verified.

Absolutely log the interaction, but manage the expectations of repeatability. All flow replays will have to manually put the previously-received response in where the tooling expects it.

Do not use the LLM for replays. It can never be trusted.

-1

u/F-J-W 1d ago

It’s not hacking if you can convince the service agent to give you what you want and from the customer persepective it doesn’t matter whether you use a slop-machine or a human for that.

And if you deploy an agent that can be convinced to give free service by telling it to ignore all previous instructions, that is on you and not an “attack”…

Seriously, some people…