Leading Consumer Lending Platform Delivers Trusted ML Solutions Using AI Observability

Fiddler AI Observability has helped this Leading Consumer Lending Platform accelerate the development of their MLOps framework, deliver mission-critical ML solutions, save costs, and gain deep insights from explainability and fairness that empower the business to confidently make business decisions.

The Growth Challenge and the Need for Scalable ML Monitoring

As a leader in AI-driven consumer loans, the Leading Consumer Lending Platform relies on ML models for multivariable risk assessment that goes deeper than FICO scores. Within a single year, the company launched a dozen ML models. They rely on those models for millions of transactions and lending decisions worth several hundred million dollars every month. But maintaining and running their existing stack of home-grown and off-the-shelf tools was an increasing strain on resources — especially on data scientists who were focused on model development. 

While their ML workflows worked as expected, the dollar-volume demands on their models was only increasing, and overloading their team with operational tasks wasn’t scalable. Data scientists needed an alternative solution that could free up their time from maintaining homegrown tools to building models that support the company’s growth. 

Evaluating Open-source and Managed Options for an Effective MLOps Framework

The company used their existing IT tool stack to address some ML use cases. While the tools provided limited monitoring functionality, their dashboards lacked the sophistication, flexibility, and ease of use required for their team to effectively track model performance — a non-negotiable requirement for their investors and banking partners.

Determined to avoid a protracted evaluation period, the company had all but committed to proceeding with the open-source application ExploreAI. However, the new approach was still resource-intensive, requiring data scientists to maintain the application. “Implementing open-source sounds easy enough,” said the Head of Data Science, “but if you're running a data science team, then you want to do data science, and the infrastructure piece is something that you'd rather have somebody else handle for you.”

The Head of Data Science quickly found Fiddler, the leader in AI Observability for LLMOps and MLOps, as the company’s standardized model monitoring solution in their MLOps framework.

“Not only did I find the UI really appealing,” said the Head of Data Science, but the Fiddler AI Observability Platform also checked some important boxes for the company, including explainable AI — a critical element in their ML roadmap. Moreover, “Fiddler has powerful ML monitoring capabilities we didn’t expect — a unified dashboard to track concept drift, data drift, and performance, glean on global and point explanations, and also assess for fairness and bias,” they explained.

Increased Business Context with Model Visibility and Transparency

Fiddler not only serves as the Consumer Lending Platform’s tool for ML monitoring metrics but also enables them to glean rich insights from Fiddler’s explainability and fairness capabilities. “Banking partners are deeply interested in explainability, and by speaking the language of finance right out-of-the-box, Fiddler helps the consumer lending company fit into those ecosystems,” said the Head of Data Science.

“ML means different things to different people,” the Head of Data Science explained, “and banking is a special challenge. For us, it was the ability of Fiddler to tailor products to speak their language and present data in a manner they are used to seeing.”

The data science team is now better equipped to discover feature impact and how features contributed to drift using population stability index (PSI), and calculating SHAPley values to understand differing recommendations among similar data subjects. Fiddler’s ability to provide explanations using various explainability methods further cemented its position as the model-monitoring platform of choice for the consumer lending company.

Implementing the Vision

Since 2015, the consumer lending company’s unique approach to risk assessment has enabled a broader segment of borrowers to access financial services while managing risk through continually evolving ML models. From the outset, the Fiddler team worked closely with the consumer lending company to establish a structure around MLOps instrumentation, helping to operationalize model monitoring by setting up an AI Observability pipeline to upload data and using Airflow for scheduling. For the team, this meant they could upload data once in the early morning, while Fiddler’s solution provided end-to-end MLOps instrumentation on the pipeline.

Fiddler also implemented push alerts to automatically notify key stakeholders when model drift is detected, eliminating the need for staff to log in, pull metrics from dashboards, or manually monitor performance. In the first months of implementation, the team transitioned from “running around logging into dashboards,” scanning through metrics, and performing maintenance on their tool stack, to a nearly hands-free MLOps workflow requiring manual interaction from data scientists just once a day.

The Way Forward: Advancing AI Responsibly with ML Monitoring

While Fiddler quickly operationalized model monitoring for performance and data drift, that was just the initial risk-assessment use case; the consumer lending company was just getting started. The Head of Data Science is confident that the data science team can now support additional business-critical use cases thanks to Fiddler, after saving over 2,000 hours per year from maintaining their home-grown monitoring tool. The team plans to grow with Fiddler, deploying fair and ethical models that offer better customer experiences, deliver more transparent responses, and reduce fraudulent transactions that could negatively impact customers and investors alike.

Responsibly managing these models will involve ramping up explainability tools and enhancing the delivery of model insights produced by Fiddler, expanding the reach of reports to internal users and automating their distribution. Assessing AI fairness is also critical for achieving responsible AI. Thanks to Fiddler, the consumer lending company can infer fairness from the mosaic of available data, including limited personal and demographic data often required for explicit bias comparison, and alert stakeholders when potential model bias is detected.

With models waiting in the wings and plans to bring several more online this quarter, the Fiddler team continues to build out the company’s MLOps lifecycle.

While Fiddler’s solution fits seamlessly into the consumer lending company’s ML framework, the partnership isn’t built on technology alone. “Fiddler’s response to meet our needs has been the most impressive experience yet,” said the Head of Data Science. “They listen to us and continuously roll out new capabilities and product upgrades. It feels like we are all one team.”

To learn more about how the Fiddler AI Observability Platform can help you ship more LLM deployments into production, book a demo, or read additional case studies.