Compare and Analyze Cohorts Using Segment Monitoring
Discover how Fiddler lets you compare and contrast specific segments of data that contributed to model outcomes.
In this product tour, see how to track model outcomes by specific user segments — such as geography or custom attributes — revealing insights that global metrics may miss. Learn how to set targeted alerts for critical segments, such as a high-value customer cohort vs low-value customer cohort, and apply dynamic filters for live data exploration. Fiddler’s segment monitoring enables you to detect and analyze segments of users, and act on those insights.
[00:00:00] I want to share with you how you can use the Fiddler platform to add more granularity to your model monitoring practices by not just tracking the global accuracy for your model over time or drift over time, but instead tracking that at a very granular level by looking at the data across specific segments of the users.
[00:00:21] In my case, I'm looking at geography. And you see by doing this granular analysis, I find out where my model is having issues, like with this cohort of Hawaii users, which is seeing a drop in accuracy, that is not registered in my system when I'm tracking global accuracy.
[00:00:39] I can set specific alerts to make sure when the accuracy for this team drops, it's detected. And I can do the similar kind of tracking and alerting over different segments like complex segments where users geography and age is combined together.
[00:00:54] All of this behind the scenes is driven by the data you're already bringing to Fiddler for observability.
[00:01:00] We can use any inputs, outputs, targets, metadata, or custom features if your model has that to build out specific cohorts that you need to track for your users or your model outcomes.
[00:01:12] So in my case, I have a specific segment of Hawaii users over 60 because this is a high value customer base for us that we want to track, but also things like their credit card statuses. If the customers have a specific gender, or they're from a specific geography. All of these can be used to segment any chart, any dashboard, any metric in Fiddler that can then be plotted on these charts to give your team this granular view.
[00:01:38] You can actually even play around with this by picking up a new metric. Let's say I'm going to continue working with accuracy here, and define a segment on the fly and basically treat these dashboards like your live filtering stage. So, let's say I'll pick those Hawaii users again and I want to add the condition where they also have an age span that belongs to under 60. I could optionally save this to my model for future use, but in this case, I'm just going to apply this to the chart. And we have that new trend line show up and you see that drop in accuracy here.
[00:02:12] Now, you can take this chart or any chart with the filters that you've created live or saved in your model already as a segment and pin them to any dashboard. So teams working with you can track the same level of granularity over time and be alerted when situations change for users or model outcomes across these segments.
