Fiddler Auditor:
Ship More AI Models into Production

Test, evaluate, and optimize NLP, LLMs, and generative AI models in pre-production
Try Fiddler Auditor
Star
Fiddler Auditor correctness report assessing a model's generations for pertubations.

Red-Teaming of LLMs and Generative AI Models Using Fiddler Auditor

Fiddler Auditor is the open-source robustness library for red-teaming of LLMs. Whether AI teams are launching AI applications using open source, in-house built LLMs or LLMs provided by OpenAI, Anthropic, or Cohere, Fiddler helps them evaluate LLMs before they go into production to mitigate risks of hallucinations, adversarial responses, prompt injection attacks, and biases.

As part of the Fiddler AI Observability solution for LLMOps, AI teams use:
01
The Fiddler Auditor to evaluate prompts and LLMs for robustness, correctness, and safety.
02
The Fiddler AI Observability platform to:
  • Monitor production LLMs for performance (drift, feedback), correctness (hallucination), safety (toxicity), and privacy (PII) metrics
  • Visually analyze trends in prompts and responses and drift
  • Gain insights from dashboards and custom metrics
Diagram flow of Fiddler Auditor applying perturbations to test and evaluating the robustness of LLMs
Fiddler Auditor supports red-teaming LLMs with by evaluating the robustness, correctness, and safety of prompts and LLMs

Fiddler Auditor Benefits

Ship More AI Models and Applications

Evaluate and improve pre-production models for optimal outcomes.
Evaluate the robustness of prompts and LLMs.

Prevent Prompt Injection Attacks

Identify weaknesses and vulnerabilities in LLMs and prompts.
Prevent prompt injection attacks.

Improve LLM Outcomes

Mitigate adversarial outcomes, provide helpful and safe responses.
Optimize and fine-tune for better LLM outcomes.

Core Capabilities

LLM and Prompt Robustness Scoring

Assess models by generating perturbed data and comparing model outputs to expected benchmarks, assigning robustness scores, and revealing actionable insights.

A/B Testing

Use more powerful models like GPT-4 to conduct thorough and nuanced evaluations of smaller models, optimizing your resources and picking the best model for your use case.
Safety and Correctness Evaluations
Evaluate for toxicity and hallucination, addressing potential harm and ensuring your LLMs are aligned with the highest safety standards before deployment​​.

Hugging Face Dataset Support

Integrate with Hugging Face datasets to refine your AI's capabilities, making the most of a vast array of data for comprehensive training and evaluation.

Integration Ready

Seamlessly integrate Fiddler Auditor with your existing workflows, ensuring a smooth evaluation process, such as DataStax, Pinecone, LangChain, LlamaIndex, and more.
Evaluating OpenAI with Fiddler Auditor using prompt evaluation with robustness report.
Try Fiddler Auditor
Star

Featured Resources

Blog
Evaluate LLMs Against Prompt Injection Attacks Using Fiddler Auditor
Blog
Introducing Fiddler Auditor: Evaluate the Robustness of LLMs and NLP Models
On-demand webinar
Al Forward - Evaluating LLMs with Fiddler Auditor
Video
Fiddler Auditor: Evaluate LLMs to Prevent Prompt Injection Attacks
Video
Fiddler Auditor: The Open Source Robustness Library for Red-teaming of LLMs
Video
Fiddler Auditor: How to Improve LLMs with Robustness Testing in Pre-Production

Frequently Asked Questions

What is Fiddler Auditor?

Fiddler Auditor is the open-source robustness library for red-teaming of LLMs. Evaluate the robustness, correctness, safety, and privacy of LLMs and prompts in pre-production.

What type of LLMs does Fiddler Auditor support?

We support AI applications using open source, in-house built LLMs or LLMs provided by OpenAI, Anthropic, or Cohere.

Can Fiddler Auditor support in production LLMs?

Fiddler Auditor supports NLP, LLMs, and generative AI models in pre-production. We support end-to-end LLMOps using the Fiddler AI Observability platform to monitor LLM metrics, including performance (drift), correctness (hallucination, faithfulness), safety (toxicity), and privacy (PII), for production LLMs. Learn more about how Fiddler supports LLMOps.

How do I get access to Fiddler Auditor?

Get access to Fiddler Auditor in GitHub.

Start Evaluating LLMs in Pre-Production
Try Fiddler Auditor
Monitor LLMs and Visualize Data Patterns
Request demo