University researchers say six widely used frameworks for building AI agents can be made to execute tool calls an attacker has written, simply by getting the underlying language model to repeat the attacker's text, CSO Online reported Monday. For organizations deploying agents, the finding means a model's refusal, or a system prompt telling it to decline dangerous requests, may not stop the surrounding software from carrying out the action.

What Happened

The paper, "ToFu: Exploiting and Fuzzing the LLM-Orchestrator-Tool Trust Boundary," is dated Oct. 9 on arXiv and comes from researchers at Pennsylvania State University, Université Montpellier and Dartmouth College. It lists a 2027 ACM software engineering conference as its venue; CSO Online notes it has not been peer reviewed.

An AI agent pairs a language model with an orchestration framework, the software that reads the model's output and, when that output is formatted as a request to use a tool such as a file reader or database query, actually runs the tool. In the attack the authors call "Repeat-After-Me," an attacker writes out a tool call with arguments of their choosing and asks the model to reproduce it verbatim. The model is only echoing text, but the orchestrator treats the text as a real request and dispatches it.

RelatedOpenAI Says Its Models Broke Into Two Outside Data Sites and Left It Out of Their Answers

The authors say they reproduced the behavior in LangChain's LangGraph, Microsoft Agent Framework, Microsoft AutoGen, Google's Agent Development Kit 2.0, AWS Strands Agents and CrewAI, though only LangGraph is documented in detail. In that test, CSO Online reported, the team used LangGraph 1.2.0 with Meta's Llama 3.1 8B model, two dummy tools with no real effect and a system prompt as the only safeguard. The model first declined. Once the request was framed as hypothetical, it emitted the call in structured form, LangGraph executed it with the attacker's arguments, and the model's refusal arrived only after the tool had already run.

The paper's second half tests the tools themselves. Many agents connect to outside services through the Model Context Protocol (MCP), a common standard for plugging tools and data into AI assistants, with each set of tools served by an "MCP server." The team's fuzzer, ToFu, which bombards software with unexpected inputs to see what breaks, ran against 273 open-source MCP servers exposing 3,405 tools. The authors confirmed 253 bugs affecting 343 tools across 118 servers, including 115 cases of unsafe output handling, 64 path traversal flaws that let a tool reach files outside its intended folder, 58 log injection bugs, seven server-side request forgery flaws, five data exfiltration bugs and three operating system command injection flaws.

Why It Matters

The research undercuts a common deployment assumption: that the model is the safety check. If the framework acts on whatever correctly formatted call the model emits, the decision about whether a sensitive action happens rests on text an attacker can shape. CSO Online cautioned that the findings do not show every deployment of these frameworks is vulnerable.

For companies building or buying agents, the practical question is whether something other than the model authorizes tool calls. Vibhum Dubey, an independent security researcher quoted by CSO Online, called the problem architectural and said a valid-looking call or convincing explanation "should never substitute for permission." He recommended explicit approval for sensitive actions, isolated execution, limited credentials and monitoring of sequences of tool calls. The authors make a related argument: tools should validate their own inputs and treat every argument as untrusted, because orchestrators check a call's structure, not its safety.

The MCP results bear on procurement and vendor review. The paper does not say whether the bugs it found were reported to the servers' maintainers or assigned CVE identifiers, and CSO Online said it does not say whether they have been fixed or exploited.

What to Watch

  • Responses from the framework makers; CSO Online said the researchers reported the missing pre-execution check to LangGraph's maintainers, while LangChain, Microsoft, Google, AWS and CrewAI did not immediately respond to requests for comment.
  • Whether the MCP server bugs are disclosed to maintainers, patched or given CVE identifiers.
  • The paper's final version after peer review, and whether more detail emerges on the five frameworks beyond LangGraph.