Fiddler AI Observability Platform
Fiddler is the pioneer in AI Observability, offering a full stack platform that enables Data Science, Engineering, Trust and Safety, Analytics, and LOB teams to streamline end-to-end MLOps and LLMOps workflows. Actionable AI Observability provides teams with actionable model insights to improve predictive models, large language models (LLMs), and generative AI models in pre-production and in-production.
Fiddler provides a full stack AI Observability that aligns teams across the organization with real-time alerts and model monitoring, rich model diagnostics and protection that help you identify the underlying issues impacting model outcomes, increase trust and transparency, and build a framework for responsible AI practices. AI Observability is reliant not only on metrics but also on how actionable model insights are when something eventually goes wrong.
How it Works
Getting started with Fiddler is easy. Simply upload your baseline dataset and add your model with Fiddler. Publish production events and start analyzing your model’s performance, accuracy, and safety.
- Fiddler is model, framework, and data-agnostic.
- Upload training and production data from any local or cloud data source of your choice, including Snowflake, AWS, Google, PostgreSQL, or Databricks
- Publish data with real-time event streaming or schedule batches of events that fit your business needs
Monitor performance, detect bias, and analyze models trained using Amazon SageMaker, TensorFlow, PyTorch, SAS, Python, H20.ai, and Scikit Learn
Fiddler gives you the flexibility to deploy your models wherever you need them. Run your models in a dedicated VPC cluster in Fiddler’s AWS cloud or your private cloud of choice, such as AWS, Azure, Google, and IBM.
A Full Stack AI Observability Platform
Predictive Models
Monitoring
Fiddler helps you reduce costs and increase efficiencies by tracking how all your ML deployments are performing in pre-production and production. Mitigate model performance issues before they impact your business with actionable model monitoring.
- Performance Monitoring: Track your predictive model’s performance with out-of-the-box metrics, including accuracy, recall, precision, F1-score, regression, mean absolute error (MSE), and mean average precision (MAP) for binary classification, multi-class classification, regression, and ranking models
- Unstructured Monitoring: Detect drift in natural language processing (NLP) and computer vision models to improve model performance
- 3D UMAP Visualizer: Gain contextual insights into complex data drift by locating and identifying drift in high dimensional spaces
- Data Drift: Easily monitor data drift, and compare data distributions between baseline and production datasets to assess how shifts in data impact model outcomes
- Prediction Drift and Impact: Use popular model drift metrics like Jensen-Shannon Divergence (JSD) and Population Stability Index (PSI) to uncover any data drift and help calculate which drifting features are impacting your model’s predictions
- Data Integrity: Uncover data integrity issues in your data pipeline causing models to underperform, such as missing data, range violations, and data type mismatches
- Feature Quality: Pinpoint key features attributing to model outcomes for further deep dive into the model and inputs/outputs
- Class Imbalance: Detect changes in low-frequency predictions due to class imbalance in each stage of your ML workflow
- Ground Truth Updates: Update ground truth labels in a delayed, asynchronous fashion
- Alerts: Configure and receive real-time model monitoring alerts to identify and troubleshoot high-priority issues caused by performance, data drift, data integrity, and traffic metrics
Analytics
Fiddler’s model analytics tool provides you with rich model diagnostics to perform root cause analysis and gain actionable insights to create a continuous feedback loop for MLOps.
- Dashboards: Increase business alignment and confidence in decision-making by enabling teams across the organization to glean insights and connect ML metrics to business KPIs in a unified view
- Insights: Build custom reports with the insights you need to gain deep understanding of your models and their impact on business outcomes, from monitoring metrics, feature impact, correlation, and distribution to partial dependence plot (PDP) charts
- Root Cause Analysis: Drill down on problem areas to uncover the root cause of underperforming segments
- Slice and Explain: Drill down into specific segments to perform exploratory or targeted analysis, and find underperforming cohorts
- Model Validation: Evaluate your model’s performance and validate it before deploying it into production
- Fiddler Report Generator: Create and share custom reports for periodic risk and compliance reviews
Fairness
Build transparent, accountable, and ethical practices for your business with responsible AI. Fiddler enables effective model governance with continuous monitoring, while detecting and mitigating bias in datasets and predictive models.
- Algorithmic Bias Detection: Detect algorithmic bias using powerful visualizations and metrics
- Intersectional Bias Detection: Discover potential model bias by examining intersectional fairness across multiple dimensions simultaneously (e.g. gender, race, etc.)
- Model Fairness: Obtain fairness information by comparing model outcomes and model performance for each subgroup of interest
- Dataset Fairness: Check for fairness in your dataset before training your model by catching feature dependencies and ensuring your labels are balanced across subgroups
- Fairness Metrics: Use out-of-the-box fairness metrics, such as disparate impact, demographic parity, equal opportunity, and group benefit, to help you increase transparency in your models
Generative AI Models and Applications
Fiddler Trust Service
The Fiddler Trust Service enables enterprises to efficiently and securely score and monitor LLM applications at scale.. The Fiddler Trust Service is comprised of proprietary, fine-tuned Fiddler Trust Models, designed for task-specific, high accuracy scoring of LLM prompts and responses with low latency. Detect hallucinations and PII leakage, prevent prompt injection attacks, and more, safeguarding LLM applications and end users.
- Fiddler Trust Models: Quickly calculate Trust Scores on user prompts and LLM responses for LLM metrics monitoring
- Fiddler Trust Scores: Evaluate multiple trust-related dimensions of prompts and responses, including: faithfulness, legality, hateful, racist, sexist, violent, harassing, sexual, harmful, unethical, jailbreaking content
LLM-Based Embedding Monitoring
Monitor drift of LLM-based embeddings in production and detect issues as soon as drift happens to minimize risks impacting users from adversarial model outcomes.
- Actionable Alerts: Get early warnings on performance of embeddings
- Drift Monitoring: Continuously detect dips in performance caused by data drift
- Dashboards and Charts: Pinpoint performance issues for deeper analysis on LLM outputs. Measure metrics, such as toxicity, costs, and safety
- 3D UMAP Visualizer: Gain contextual insights into complex data drift by locating and identifying drift in high dimensional spaces
Enterprise-level Scale
Fiddler helps companies securely operationalize and scale model deployment.
- Scalability: Accelerate the deployment of advanced models that require large volumes of data ingestion. Gain deeper insights with scalable baseline dataset ingestion.
- Single Pane of Glass: Monitor, analyze, and improve your models within a unified platform with a single pane of glass. Easily view all your models in production and track what’s most important to your business.
- Choose Your Deployment: Choose where you deploy your models to meet your company’s needs — it can be in the Fiddler cloud or you can bring your own cloud.
- Security: Fiddler provides SOC 2 Type 2 security and HIPAA compliance. Users across the business will receive level-specific permissions to access protected environments through role-based access control (RBAC), and SSO.
