Leading Financial Services Institution Scales Trusted Compound AI with Continuous Evaluations
One of the world's largest financial services firms operates an AI portfolio that spans customer-facing assistants for brokerage and retirement services, agentic workflows for portfolio managers and analysts developed through its innovation lab, a RegTech platform for compliant marketing content, and machine learning models powering fraud detection, personalization, and portfolio automation such as tax-loss harvesting. Fiddler runs in the firm's VPC as an Amazon SageMaker Partner App, integrated with its proprietary LLM gateway, Amazon Bedrock, and OpenTelemetry-instrumented agent stack.
Results at a Glance
Fiddler's partnership delivered measurable results for the platform, including:
- Accelerated Time-to-Market: GenAI applications and agents stalled by compliance concerns moved into production faster by evaluating in-environment.
- Faster Root Cause Resolution: Step-level diagnostics trace an application-level metric drop to the specific LLM call, ML inference, or tool call responsible.
- Eliminated Data-Sharing Risk: Hallucination, PII, and safety scoring run inside the firm's VPC with no external API calls.
- Predictable Total Cost of Ownership: In-built Centor Models eliminate unpredictable per-token LLM judge fees at enterprise interaction volume.
- Audit-Ready Governance: A continuous, independent evidentiary record mapped to NIST AI RMF, ISO 42001, and the firm's model risk standards.
The Challenge: Evaluating Compound AI in a Highly Regulated Environment
The firm's AI applications are compound AI systems. A single customer request may trigger multi-step reasoning, document retrieval, market data tool calls, inference from underlying ML models, and response generation through an LLM gateway, producing hundreds of spans per interaction. Evaluating these systems reliably presents four obstacles that traditional monitoring tools cannot address:
- Failures can originate at any layer of the stack. An incorrect answer may be caused by an LLM hallucination, a stale retrieval index, a drifted personalization model, or a tool call that returned bad data. Evaluations that score only the final response cannot identify the source.
- Each application carries distinct regulatory obligations. Chatbot responses are firm communications under FINRA's fair-and-balanced standard. Marketing content is governed by FINRA 2210 and the SEC Marketing Rule. Data handling falls under Reg S-P and GLBA. ML models are subject to SR 11-7-style model risk management.
- FSI compliance requirements demand that no customer data leave the secure environment. This rules out evaluation solutions that rely on external API calls for scoring, including LLM-as-a-judge services.
- Token-based judge pricing makes evaluation costs unpredictable. At enterprise interaction volume, per-call fees create budget uncertainty and pressure teams to downsample the very evidence regulators expect.
The Solution: The Fiddler AI Control Plane
The firm's constraint was non-negotiable: no customer data leaves the environment. So Fiddler runs inside it, deployed as an AWS SageMaker Partner App in the firm's VPC, and everything else follows from that decision.
Because Fiddler sits in the environment, it can ingest OpenTelemetry traces from the entire agent stack, not just final responses. Those traces are organized into four levels: application, session, agent, and span. A degraded application metric traces down through the session and agent to the exact LLM call, ML inference, or tool call that caused it. That answers the first problem, failures that could originate anywhere in the stack.
The same traces feed evaluation on both sides of the release gate. Offline, golden datasets per agent catch regressions before a model, prompt, or retrieval index change ships. Online, production traffic is scored against the baselines set during offline evaluation. One scoring substrate, so the evaluation that approved a release is the evaluation watching it in production.
What makes in-environment scoring possible at this volume is Fiddler Centor Models, purpose-built evaluation models that run in the VPC alongside everything else. Hallucination detection, PII identification, and safety scoring happen with no external API calls and no per-token judge fees. The compliance blocker and the cost unpredictability disappear together.
And because those models are fast, the same scoring moves from observation into enforcement: Fiddler Guardrails moderates risky prompts and responses in the LLM gateway at sub-100ms, before they reach customers.
Independent Evaluation for Audit and Governance
The firm built its RegTech compliance business on the principle that compliance review requires an independent, evidence-backed layer rather than self-certification. The same principle applies to agent behavior. When regulators, auditors, or the board risk committee ask how the firm validates its AI applications, audit evidence carries more weight when the evaluation layer is independent of the teams building and operating the agents. Fiddler provides that independence while running entirely within the firm's environment, scoring and retaining every production interaction as a complete evidentiary record.
Looking Ahead: Building on a Trusted Foundation
With the observability and governance infrastructure in place, the firm is positioned to expand GenAI across additional business functions while maintaining the security and compliance standards its industry demands. As it explores more sophisticated multi-agent systems, Fiddler's visibility across the agentic hierarchy ensures oversight and governance scale with the complexity.