How Does AI Agent Governance Differ From Observability?

Forrester observes that most enterprises govern the systems around their agents. They control credentials, log tool calls, and route high-risk actions to a human approver, which are controls designed for software that follows instructions [1]. A log entry proves that a tool call happened, not that the agent should have made it. Teams conflate governance and observability because both operate on the same tool calls and both produce logs. AI agent governance sets and enforces what an agent may do: which tools it can call, under whose authority, and when a human has to approve the action first. AI agent observability records what the agent actually did, step by step, so that someone can inspect it, score it, and reconstruct it later. The core tension in AI governance vs observability is enforcement against visibility.

Governance vs. Observability: The Short Answer at a Glance

Governance and observability differ on every dimension that matters to an audit:

How the Two Layers Differ in Purpose and Timing

What AI Agent Governance Enforces

Governance is the management of authority that has been delegated to software. In the OWASP Top 10 for LLM Applications 2025, the authors identify three root causes of damaging agent actions in the Excessive Agency (LLM06) entry: excessive functionality, excessive permissions, and excessive autonomy [2]. The OWASP mitigations correspond directly to governance controls:

• Least privilege: Limit the extensions an agent can call to the minimum it needs, and scope each tool’s permissions to its task.
• Complete mediation: Implement authorization in downstream systems rather than letting the LLM decide whether an action is allowed, and validate every request against security policy.
• Human approval: Require a person to approve high-impact actions before they run [2].
• Role-based access control (RBAC): Tie each agent identity to a role, and tie each role to an accountable owner.
• Runtime guardrails: Check the request before the model is invoked, and check the response before it reaches the user or a downstream system.

Each of these controls produces its decision before execution. When a control works, the prohibited action never happens. That is why governance cannot be reduced to written policy. A policy that teams do not enforce at inference time is only a statement of intent.

What AI Agent Observability Records

Observability captures the reasoning chain, the tool calls, and the decisions an agent makes, with enough structure to replay a run. The core signals are:

• Traces: One end-to-end record of a request, from user input to final output.
• Spans: The individual steps inside a trace, such as an LLM call, a tool invocation, a retrieval, or a handoff to a sub-agent.
• Sessions: Groups of traces that belong to one multi-turn interaction.
• Logs and events: Timestamped records, including prompt and response content when content capture is enabled.
• Evaluations: Scores attached to spans for qualities such as faithfulness, toxicity, or tool-selection correctness.
• Cost: Token counts per call, rolled up by session, agent, and team.

The OpenTelemetry GenAI semantic conventions standardize much of this. They record the model called and the input and output token counts, and, only when content capture is opted in, the full text of prompts, completions, tool calls, and tool results [3]. Observability runs in real time for alerting and post-hoc for investigation. A dashboard is only one view of it. The traces are the underlying record.

Why Agent Telemetry Needs More Than APM

Application performance monitoring (APM) was built for deterministic services. A request arrives, known code runs, a response returns, and failures surface as exceptions or 5xx status codes. Agents break each of those assumptions. The same input can produce a different plan. One request can expand into several model calls, a retried tool call, and a sub-agent handoff. A run that returns HTTP 200 can still contain a fabricated tool result or a wrong action.

Agent telemetry needs a hierarchy that follows the agent’s structure rather than the service’s. OpenTelemetry describes a trace whose top-level invoke_agent span holds child chat spans for each LLM call and execute_tool spans for each tool invocation [3]. The agentic hierarchy extends this into five levels:

1. Application: The deployed agentic product.
2. Session: One user’s multi-turn interaction.
3. Agent: Each agent participating in the session, including sub-agents.
4. Trace: One request through those agents.
5. Span: One LLM call, tool call, or retrieval inside the trace.

With this hierarchy, an alert raised at the application level can be followed down to the exact span where a bad tool argument first appeared.

How Observability Data Feeds Governance (and Governance Shapes Telemetry)

Governance and observability meet at the decision record. A policy engine evaluates a request before execution and returns allow, block, or redact. That verdict has to be written into telemetry as a logged signal, attached to the same trace as the tool call it ruled on. Otherwise an auditor sees the action but not the control that governed it.

Data flows in both directions:

• Telemetry into policy: Traces show which tools agents actually call, how often guardrails fire, and where false positives cluster. Policy owners use that evidence to tighten or loosen rules.
• Policy into telemetry: Governance requirements determine what has to be captured. That includes the agent identity behind each call, the policy version applied, and whether content capture is enabled for a regulated workflow.
• Audit trails: The combined record answers three questions together: what the agent attempted, what the policy decided, and what happened next.

Apple Machine Learning Research describes the cost of keeping these layers apart. When teams collect telemetry but treat governance as a downstream analytics concern, they detect policy violations only after damage is done, an “observe-but-do-not-act” gap [4]. Apple’s researchers propose an architecture connecting telemetry collection to automated policy enforcement [4]. To close that loop, teams need a pre-execution policy engine that writes its decisions into telemetry, plus a telemetry pipeline that policy owners review.

A Tool Call Through Both Layers: A Worked Example

Take a customer-support agent that can call an issue_refund tool. A user asks for a refund and pastes a full email thread into the chat, including another customer’s address. The call passes through both layers in sequence:

1. Request check before invocation: The guardrail scans the prompt and finds personal data that belongs to a third party. Policy owners require redaction, so the guardrail masks the address before the model sees it.
2. Tool-call check: The model proposes issue_refund with an amount above the agent’s authorized limit. The policy engine blocks the call and routes the request to a human approver.
3. Response check: The agent’s reply to the user is scanned before delivery. It contains no prohibited content, so it is allowed.
4. Recording: Each step produces a span. The execute_tool span carries gen_ai.tool.name and, if content capture is on, gen_ai.tool.call.arguments. The redact, block, and allow decisions travel as custom attributes or as linked log records on the same trace.
5. Audit: A reviewer opens the session and sees the redacted prompt, the blocked tool call, the approver’s later decision, and the final response, all tied to one trace ID.

Reviewers can use observability to investigate three agent-specific risks:

• Cascading errors: A degraded tool result at step two becomes the premise for every later step. A reviewer may see no problem in individual spans but identify where the error entered by examining the chain.
• Silent failures: The run completes with no error, but the outcome is wrong. Reviewers can use evaluation scores on spans to identify incorrect outcomes that status codes do not reveal.
• Drift: Gartner warns that traditional agent observability creates a blind spot, because agents can meet service level agreements while drifting from business intent or compliance [5]. Detecting drift requires comparing current behavior against a baseline over time.

Who Owns Governance vs. Observability in the Enterprise

Policy and telemetry usually sit with different teams. The IAPP AI Governance Profession Report 2025 found that privacy (22%) and legal and compliance (22%) are the functions most often given primary responsibility for AI governance, followed by IT at 17% [6]. Production telemetry is typically run by engineering. The ownership map looks like this:

• Legal, risk, privacy, and GRC teams: Own the policy itself, including risk tiers, approval thresholds, prohibited data classes, and the mapping of each agent to the regulations it must satisfy.
• AI governance or model risk leads: Own the registry of live, testing, and retired agents, plus the named business owner for each.
• Platform engineers and SREs: Own instrumentation, trace pipelines, retention configuration, alert routing, and the gateway where enforcement runs.
• Agent builders: Own tool definitions, permission scopes, and the evaluations that gate releases.

Handoffs are where controls tend to fail, so each one needs a named owner on both sides:

• Compliance writes the policy rule, and platform engineering translates it into enforcement configuration and deploys it.
• Agent builders define evaluation thresholds, and platform engineers configure production alerts.
• Platform engineers export enforcement decisions for compliance reviewers to use as audit evidence.
• Compliance and platform engineering review policies after incidents, with both teams reading the same trace.

Mapping Regulatory Requirements to Controls vs. Evidence

Each major framework imposes two kinds of obligation. Control obligations require something to be enforced. Evidence obligations require something to be recorded and kept. The following table sorts each framework’s requirements into those two groups:

The EU AI Act sets both obligation types out in the text itself:

• Automatic logging: Article 12(1) states that high-risk AI systems “shall technically allow for the automatic recording of events (logs) over the lifetime of the system” [7].
• Human oversight: Article 26(2) requires deployers to “assign human oversight to natural persons who have the necessary competence, training and authority, as well as the necessary support” [7].
• Log retention: The six-month log floors in Articles 19 and 26(6) apply to logs under the provider’s or deployer’s control, and other Union or national law, particularly data-protection law, can change the period [7].
• Documentation retention: Article 18 requires providers to keep technical documentation for 10 years after the system is placed on the market [7].
• Application dates: Timing changed in 2026. Regulation (EU) 2026/1744, the Digital Omnibus on AI, moved the obligations in Chapter III, Sections 1 to 3 to 2 December 2027 for Annex III high-risk systems and to 2 August 2028 for systems covered by Annex I [8]. Articles 12 and 14 fall within Section 2, so logging and human oversight follow those dates.

NIST’s AI RMF 1.0, dated January 2023, assigns inventory, roles, and controls to the Govern function. Production monitoring sits under Measure 2.4: “The functionality and behavior of the AI system and its components... are monitored when in production” [9]. ISO/IEC 42001 lists policies and lifecycle controls alongside performance evaluation and monitoring as requirements of a single AI management system [10]. That structure supports the view that each standard needs both layers to be satisfied.

These mappings organize obligations for planning purposes. They are not legal advice. Which articles apply depends on risk classification, role (provider or deployer), and jurisdiction, and those determinations belong to counsel.

When Separate Tools Work and When One Platform Pays Off

Most enterprises begin with separate tools. Open-source tracing frameworks and APM incumbents record agent behavior. Policy engines and guardrail libraries enforce rules. That split works while a small number of agents run on one stack. Teams need to connect the layers at the handoff so guardrail verdicts reach the trace store and compliance can query traces without help from engineering.

Start with the simplest setup that closes that handoff. OpenTelemetry and its GenAI semantic conventions give both layers a shared vocabulary, so a policy decision and the tool span it governs can carry the same trace ID. OpenTelemetry describes the conventions as in use today and under active development [3], so attribute names can change between versions. Teams should pin a version and plan for migrations. Adding a unified platform makes sense once agents span several frameworks and teams, because at that point maintaining joins between separate systems costs more than consolidating them.

The Fiddler AI Observability and Security platform enables governance by placing enforcement and evidence on the same telemetry foundation:

• Continuous Monitoring: Agentic Observability provides OpenTelemetry-native instrumentation across the application, session, agent, trace, and span hierarchy. It offers SDKs for LangGraph, LangChain, AWS Strands Agents, and Google ADK, plus REST endpoints, and gateway-based deployment needs no agent-side instrumentation.
• Enforceable Policy: Fiddler Guardrails applies inline enforcement on the agent request and response path, deployed directly or through the existing LLM or MCP gateway, such as LiteLLM, AgentGateway, or Kong Gateway. Requests are checked before the model is invoked and responses before they reach the user or a downstream system, with allow, block, or redact actions depending on the policy and under 80ms response time.
• Auditable Governance: Governance records capture each enforcement decision with contextual metadata, including who triggered the event and what action or policy was applied. An AI registry covers live, testing, and retired models and applications along with their owners and lifecycle status.

Deployment options include software as a service (SaaS), virtual private cloud (VPC), on-premise, and Amazon Web Services (AWS) GovCloud, for regulated workloads that cannot send inference data out of the network.

Request a demo to review how inline enforcement and audit records would fit the existing agent architecture and deployment environment.

References

[1] L. Joseph, “Does Your ‘Agent Governance’ End Up Governing Everything But The Agent Itself?,” Forrester Blogs, Forrester Research, Aug. 2026. Accessed: Oct. 28, 2026. [Online]. Available: https://www.forrester.com/blogs/does-your-agent-governance-end-up-governing-everything-but-the-agent-itself

[2] OWASP GenAI Security Project, “LLM06:2025 Excessive Agency,” OWASP Top 10 for LLM Applications 2025. Accessed: Oct. 28, 2026. [Online]. Available: https://genai.owasp.org/llmrisk/llm062025-excessive-agency

[3] OpenTelemetry, “Inside the LLM Call: GenAI Observability with OpenTelemetry,” OpenTelemetry Blog, 2026. Accessed: Oct. 28, 2026. [Online]. Available: https://opentelemetry.io/blog/2026/genai-observability

[4] A. Pathak and N. Jain, “Governance-Aware Agent Telemetry for Closed-Loop Enforcement in Multi-Agent AI Systems,” Apple Machine Learning Research, Apr. 2026. Accessed: Oct. 28, 2026. [Online]. Available: https://machinelearning.apple.com/research/governance-aware-agent-telemetry

[5] Gartner, “AI Governance Requires Agent Drift Detection,” Gartner Research, May 26, 2026. Accessed: Oct. 28, 2026. [Online]. Available: https://www.gartner.com/en/documents/7912009

[6] IAPP, “AI Governance Profession Report 2025,” IAPP Resources. Accessed: Oct. 28, 2026. [Online]. Available: https://iapp.org/resources/article/ai-governance-profession-report

[7] European Parliament and Council of the European Union, “Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act),” Official Journal of the European Union. Accessed: Oct. 28, 2026. [Online]. Available: https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng

[8] European Union, “Regulation (EU) 2024/1689, consolidated text as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI),” EUR-Lex, version of Jul. 27, 2026. Accessed: Oct. 28, 2026. [Online]. Available: https://eur-lex.europa.eu/legal-content/EN/TXT/HTML?uri=CELEX%3A02024R1689-20260727

[9] National Institute of Standards and Technology, “Artificial Intelligence Risk Management Framework (AI RMF 1.0),” NIST AI 100-1, Jan. 2023. Accessed: Oct. 28, 2026. [Online]. Available: https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf

[10] International Organization for Standardization, “ISO/IEC 42001 explained,” ISO Insights. Accessed: Oct. 28, 2026. [Online]. Available: https://www.iso.org/insights/iso-42001-explained