Tool and function call arguments are validated before execution

When a model is given tools (function calling, tool use), a successful prompt injection no longer just changes the model's words—it causes the model to call real functions with attacker-specified arguments. OWASP LLM06:2025 (Excessive Agency) directly addresses this: an AI agent that can take consequential actions without independent authorization checks is a force-multiplier for any injection. MITRE ATLAS AML.T0051 and CWE-1427 both apply to the unvalidated dispatch of tool calls. A tool that performs database writes, sends emails, or makes API calls on the model's instruction—without re-verifying authorization at the execution layer—can be triggered by an injected instruction to perform actions the authenticated user never authorized. NIST AI RMF GOVERN 1.1 requires clear boundaries around what an AI system is authorized to do.

Validate tool calls at three layers: allowlist the tool name, schema-validate the arguments, and enforce authorization inside the tool function independent of the model's request.

// src/lib/tools.ts
const ALLOWED_TOOLS = ['search_docs', 'get_user_profile', 'create_draft'] as const
type AllowedTool = (typeof ALLOWED_TOOLS)[number]

for (const toolCall of completion.choices[0]?.message?.tool_calls ?? []) {
  // Layer 1: allowlist check
  if (!(ALLOWED_TOOLS as readonly string[]).includes(toolCall.function.name)) {
    throw new Error(`Blocked unknown tool: ${toolCall.function.name}`)
  }

  // Layer 2: argument schema validation
  const schema = ToolSchemas[toolCall.function.name as AllowedTool]
  const args = schema.parse(JSON.parse(toolCall.function.arguments))

  // Layer 3: authorization inside the tool function (never trusts caller intent)
  await TOOL_HANDLERS[toolCall.function.name as AllowedTool](args, session)
}

The authorization check inside each tool handler is non-negotiable: it must verify the acting user has permission for the specific operation, not just that the model requested it.

• ID: ai-prompt-injection.output-filtering.tool-call-validation

• Severity: high

• What to look for: Count all tool/function definitions available to the LLM. For each tool, if the project uses tool use, function calling, or agent patterns with AI models, examine the code that handles tool call results. Look for code that reads the tool_calls or function_call field from the model response and executes the referenced function with the model-supplied arguments. Check whether: (1) the tool name is validated against an allowlist, (2) the arguments are validated against a schema before the tool function is called, and (3) the tool function itself has authorization checks independent of the model's request.

• Pass criteria: Tool calls are validated on three levels: the tool name matches an allowlist of registered tools, the arguments conform to the tool's schema, and the tool function performs its own authorization check before taking any action — 100% of tool calls must validate arguments against a schema before execution. Report: "X tools registered, all Y validate arguments before execution."

• Fail criteria: Tool names are dispatched without allowlist validation, OR tool arguments are used without schema validation, OR tools take actions without independent authorization checks (trusting only that the model requested it).

• Skip (N/A) when: No tool use, function calling, or agent pattern detected in the AI integration. No tools or functions parameter in AI API calls.

• Detail on fail: "lib/tools.ts dispatches tool calls by passing the model-returned tool name directly to a handler map without allowlist validation" or "Tool arguments from model are passed directly to database operations without schema validation or authorization check"

• Remediation: An attacker who successfully injects into your prompt can cause the model to call tools with arbitrary arguments. Defense must exist at the execution layer, not just the prompt layer:

  const ALLOWED_TOOLS = ['search_docs', 'get_user_profile', 'create_draft'] as const
  type AllowedTool = typeof ALLOWED_TOOLS[number]
  
  function isAllowedTool(name: string): name is AllowedTool {
    return (ALLOWED_TOOLS as readonly string[]).includes(name)
  }
  
  // When processing a tool call from the model:
  if (!isAllowedTool(toolCall.function.name)) {
    throw new Error(`Unknown tool: ${toolCall.function.name}`)
  }
  
  // Validate arguments with the tool's schema
  const args = ToolSchemas[toolCall.function.name].parse(
    JSON.parse(toolCall.function.arguments)
  )
  
  // Execute with authorization check inside the tool function