Introducing Fiddler Guardrails: The Fastest in the Industry

[00:00:00]

[00:00:02] Welcome back to Safe and Sound AI. So get ready because today we're diving into a major development in AI safety.

[00:00:08] It's a brand new tool and it's designed to keep those powerful large language models, LLMs, on track, the ones that are generating all this text, translating languages, even writing different kinds of creative content.

[00:00:18] But we've all seen it. They can go off the rails, sometimes in pretty dramatic ways.

[00:00:22] Yeah, it's true. I mean, we're really seeing LLMs being adopted across industries. But with that, obviously comes a whole new set of risks. These models can sometimes generate incorrect or misleading information.

[00:00:32] Exactly, and that's where I think this idea of Fiddler Guardrails comes in. It's like a, you know, a safety net for those LLM applications we're talking about.

[00:00:39] The speed is really impressive. We're talking under 100 milliseconds latency. I mean, that's the fastest in the industry. So it can keep pace with even the most demanding applications like chatbots or content generation platforms.

[00:00:51] Okay, so let's break this down a little so it sounds like Guardrails builds on the existing Fiddler Trust Service. Can you give us a quick refresh on what that is exactly?

[00:00:58] Sure. So the Fiddler Trust Service, think of it as a foundation. It provides this comprehensive evaluation of both the prompts that are given to an LLM and the responses that it generates.

[00:01:08] And then it scores these interactions against key trust dimensions. So things like hallucinations, toxicity. Kind of like a multifaceted risk assessment for every single LLM interaction.

[00:01:18] So if I'm understanding this correctly, the Trust Service provides the scoring and a Guardrails is what steps in to actually moderate the prompts and responses.

[00:01:28] Exactly. It takes those scores from the Trust Service and then it uses those scores to determine, okay, should a particular prompt or response be allowed through ? So it's almost like having a sophisticated security checkpoint built right into your LLM workflow.

[00:01:43] It sounds like a pretty robust system, but one thing I'm always curious about is the level of control.

[00:01:47] Can organizations customize Guardrails to match their own specific risk tolerance? Because not every use case is going to have the same level of sensitivity, right?

[00:01:56] Absolutely. And that's actually one of the key features of Guardrails.

[00:01:59] Okay.

[00:01:59] You're not stuck with this like one size fits all approach.

[00:02:02] Right.

[00:02:02] Companies can actually define their own risk tolerance, right? It's all about deciding what level of risk you're comfortable with.

[00:02:08] So it's not a one size fits all solution. Organizations have the power to define their own safety boundaries. That's pretty impressive.

[00:02:15] Exactly.

[00:02:16] That kind of flexibility is amazing. But, you know, setting up all those rules and thresholds, it sounds like it could get pretty complex, especially for larger organizations with multiple LLM deployments.

[00:02:25] You know what? Fiddler's actually made it surprisingly easy.

[00:02:28] And the integration with their AI Observability platform gives you this awesome visual dashboard for managing everything.

[00:02:34] So it's not just about setting up the Guardrails, it's about having the visibility and control to manage them effectively over time.

[00:02:41] Talk about peace of mind.

[00:02:42] Exactly. You can monitor all those key metrics we talked about, see if any flags get tripped, any violations, and even drill down into specific incidents to see why they happen.

[00:02:51] Well said. You mentioned earlier deployment environments. We know a lot of organizations are working with very sensitive data and have to meet some very strict security requirements.

[00:03:00] How do Fiddler Guardrails address those concerns?

[00:03:03] That's a great question, and it's something Fiddler took very seriously when designing Guardrails. They can actually be deployed in lots of different environments, including like virtual private clouds, and even air-gapped systems.

[00:03:13] So even for those organizations operating in those highly regulated industries, you know, like healthcare or finance, they can use Guardrails without having to compromise on their security protocols.

[00:03:23] That's a big relief for anyone dealing with sensitive data.

[00:03:26] Exactly. It makes sure that all that data processing and analysis happens in a secure and controlled environment. So you minimize the risk of any, you know, unauthorized access or breaches.

[00:03:36] Okay, I think we've laid a pretty solid foundation here about what Fiddler Guardrails are and why they matter so much.

[00:03:42] I want to go back to something we talked about earlier, the importance of speed. You mentioned that Fiddler Guardrails are the fastest in the industry with a response time of under 100 milliseconds .

[00:03:51] Right. That combination of speed and security is only going to become more important as AI keeps advancing. We're going to see LLMs being used in even more sensitive and mission critical applications.

[00:04:02] And having these robust security measures, like Fiddler Guardrails in place, is going to be essential for making sure that those deployments are safe, reliable, and trustworthy.

[00:04:10] Absolutely, it's about striking that balance between innovation and responsibility, pushing the boundaries of what's possible while also protecting against those potential risks.

[00:04:19] I love that. What can organizations working with LLMs actually take away from this conversation?

[00:04:24] Okay, first and foremost, I think it's crucial to understand that, that LLM security, it's not a one and done thing. It's this ongoing process, this constant vigilance, this, this need to adapt and be willing to embrace those new tools and strategies as they emerge.

[00:04:40] It's almost like an arms race, right? A cybersecurity arms race, but for LLMs.

[00:04:44] You got it. Exactly that. That threat landscape, it never sits still. It's always changing, always evolving. And those attackers, they're getting smarter, more creative with their methods. Organizations need to be proactive, always assessing their security, always tweaking, adjusting as needed.

[00:04:59] And that's exactly where solutions like Fiddler come in, they give you that essential protection, but they also offer, you know, that flexibility, that visibility you need to keep pace with those, those ever changing threats.

[00:05:10] Okay, so , let's say an organization's on board, they're convinced, ready to implement Fiddler Guardrails. What are some practical first steps? Where do they even begin?

[00:05:18] I'd say start with a really solid risk assessment, figure out what LLM applications you're running, the types of data they're handling, , and really think about what are the potential consequences , if a breach were to happen.

[00:05:29] So basically, know your weaknesses , before you jump in.

[00:05:32] Exactly. Once you've got that clear picture of your risk profile, then you can start looking at Fiddler Guardrails and see how they fit into your specific needs. Fiddler has a ton of resources, documentation, tutorials,

[00:05:43] And as you start implementing Guardrails, remember, it's not, you know, a one shot deal. It's an iterative process. Don't be afraid to experiment, tweak those thresholds, you know, really tailor the system as your security needs evolve.

[00:05:57] So it's not a set it and forget it kind of thing. You've got to stay engaged, keep refining it.

[00:06:00] Yeah, exactly. And that's why those monitoring and analysis capabilities of Fiddler are so powerful. You learn from those incidents, you spot those patterns. You're constantly improving your LLM security posture.

[00:06:12] So we've covered a lot of ground here today, but before we wrap things up, I want to leave our listeners with something to think about. As LLMs become more powerful, more integrated into our lives , those security stakes, they're only getting higher, right? So what role do you think, individual developers, researchers, even just everyday users have in making sure that AI is developed and deployed responsibly?

[00:06:34] That's a really deep question and I don't think there's any easy answer. I think it starts with, awareness. We all need to understand those potential risks that come with AI, not just the shiny, exciting benefits. And we have to be thoughtful about the choices we make, both as, as the people creating this technology and as the people using it.

[00:06:51] It's like recognizing that AI isn't just , this cold, hard tool. It's a reflection of us, of our values, our hopes for the future.

[00:06:58] And on that note, I think it's time to wrap up our deep dive into Fiddler Guardrails. It's been a fascinating look at this game changing technology that has the potential to really shape the future of AI security.

[00:07:09] This podcast is brought to you by Fiddler AI. For more on Fiddler Guardrails, see the article in the description.

‍