Building an AI Detection Engine That Understands Agent Intent
Building an AI Detection Engine That Understands Agent Intent
AI agents are doing real work in production: shipping code and managing infrastructure with broad access and few guardrails. Unlike deterministic software, agents act on intent, which can be hijacked by a poisoned input or can drift toward a goal the agent was never meant to pursue. To effectively secure AI agents, we can no longer evaluate outputs in isolation. Instead, we must analyze an agent’s full reasoning and execution path. 🚀
In this blog, we explore how a benign task assigned to an OpenAI agent escalated into a security breach, and why model input and output telemetry is key to detecting intent manipulation. We’ll also break down how we’re building a specialized detection engine to distinguish routine operations from malicious attacks, and how we validate it against real-world threat scenarios.
Historically, security tools could never peek inside an attacker’s thought process. With AI agents, we have a new opportunity: a direct window into live reasoning. Detecting shifts in intent as they happen lets us capitalize on this visibility, tackling a challenge legacy security was never built to address. The need for this visibility was on full display during the Hugging Face incident, disclosed by OpenAI in the summer of 2026. During a cybersecurity evaluation, AI agents driven by a combination of OpenAI models circumvented the isolation controls designed to keep them sandboxed, established an unauthorized communication channel using a package manager as an improvised message board, and coordinated over several days to penetrate Hugging Face’s production infrastructure.
In a detailed analysis of the incident, OpenAI shared the full incident analysis, including the messages published by the swarm of agents in the make-shift message board. In the full report, we can clearly see how intent gradually shifted from solving ExploitGym to searching for the answer key to the benchmark on third-party services, including HuggingFace. This is a clear example of what makes AI agents different from traditional applications. The agents were not simply executing a predefined set of actions. They were reasoning about how to accomplish a goal, adapting their approach, and taking actions that, from their perspective, made sense.
To understand intent and reasoning patterns, defenders need telemetry coming directly from these models. Model I/O logs from providers give us the opportunity to access the agent’s train of thought. This telemetry record captures every layer of agent interactions, including user requests, system prompts defining intent, tool specifications, and tool results, as well as the model’s full output and internal reasoning.
To demonstrate both the power and complexities of this log, let’s take an example from our own internal attack simulations: an organization deploys an autonomous AI agent to triage and resolve customer support tickets. In its daily workflow, the agent receives a simple instruction: A new support ticket has arrived. Please handle it. The customer support ticket the agent will action on seems typical, but deeper investigation reveals some suspicious activity. The ticket directs the agent to send an email to customers, with a link asking them to “reconfirm” their details. The link looks completely legitimate and comes from a trusted source. With the context of the model logs, we can see that the link actually leads to a website controlled by an external attacker.