With high-profile snafus with black box AI making headlines recently, more and more organizations are thinking about how to invest in explainable AI. In February 2021, we sat down with a panel of explainable AI experts to discuss the future of this field. What kinds of interpretability tools should we expect to see on the horizon? What is at the cutting edge of research, and how should organizations and AI practitioners be planning to stay ahead of the curve? Mary Reagan, Data Scientist at Fiddler, moderated the discussion. We’ve summarized the top takeaways below, and you can watch the full video below.
1. Going beyond bias and fairness
The need for explainable AI is accelerating rapidly, particularly in certain fields, like insurance, finance, and healthcare. Merve Hickok, Founder at AIEthicist, noted that these are “high-stakes” situations due to consumer pressure and existing regulations. Explainability is the way to prove that the model’s decisions are not biased or unfair. And new regulations and laws may be on the horizon that will make this model auditability a requirement across more kinds of AI systems.
However, explainable AI is not just about bias and fairness. As Hickok noted, explainability can help with the other kinds of high stakes that companies face when working with black box AI. For instance, understanding the model’s decisions can prevent the wrong predictions from reaching client applications and end-users. And explainable AI can help monitor the model’s performance to protect against adversarial attacks and data drift.
2. Providing different techniques for different kinds of users
Furthermore, we need a more nuanced definition of interpretability that is tailored to specific use cases. “When you’re deploying a model, it’s likely you want to understand the relative distribution of your dataset,” explained Sara Hooker, Artificial Intelligence Resident at Google Brain. On the other hand, “For a consumer, you are always going to want to know for your prediction, why did the model perform the way it did?” Different users need different explainability tools and techniques.
In the future, we might expect to see more investment in end-user explainability. User feedback is essential for building software, but right now it’s hard to collect feedback on a model’s predictions. Also, showing “just the explanation itself is not going to be enough for users,” said Hickok. They will want more control to be able to change their settings or delete the information that is leading the model to make a certain prediction.
3. Building “inherently interpretable” models
As Narine Kokhlikyan, Research Scientist at Facebook, explained, “Moving forward, I think the model developers—the ones who put the architecture together—will put more emphasis on inherently interpretable models.” Some components will be interpretable by design, while others will have to remain black box and in need of post-hoc explanations. Hickok said that especially in high-stakes industries, we might see a shift to building “inherently gray or white box” models. But as Hooker pointed out, it’s important to keep in mind that when you cascade several white box models, the result is a black box—so we need techniques to explain how models work together, too.
It’s important that AI explainability isn’t just used after the fact when something has gone wrong. Hooker sees an opportunity to build interpretability into model training. “Most interpretability methods have taken it as a given that the model is trained, it’s done, and then you’re trying to do acrobatics after training to introduce interpretability.” One exciting research question is, how do we treat training examples in a way that results in a more interpretable model?
4. Developing a portfolio of AI explainability methods
In the future, AI pipelines will need to contain many different techniques that offer different perspectives on explainability. Kokhlikyan believes that people “will not rely on one explanation or one technique, but will explore various different techniques and look at [the model] from different lenses.”
A “toolbox” of AI explainability techniques is needed because humans tend to struggle when there are more than two dimensions involved in a problem, and machine learning models are a very high dimensional space. If we just look at a single interpretation, it’s like “trying to summarize or explain something as complex as a neural network with a score,” as Kokhlikyan explained. Instead, we should have ways to look at the interaction of features in relation to the model, and do various different kinds of analyses before drawing a conclusion.
5. Expanding into multimodal explanations
Kokhlikyan works on Captum, a generic, unified open-source model interpretability library. The core philosophy behind it is that it’s scalable and multimodal. Multimodal explanations work across different types of model data (e.g. text, images, video, audio).
“The majority of interpretability research has been focused on computer vision up until very recently,” Hooker explained, with little research into making audio data interpretable, for example. We need to expand into more types of modalities because AI applications are expanding as well. “The way that we’re applying these models is often across very different tasks, with an emphasis on transfer learning—fine-tuning existing weights for unexpected use cases.” And because of that, we have to expect surprising results that we’re going to want to be able to explain.
Conclusion
We’ll tease a few other priorities for the future: (1) research into viewing subsets of model distributions and (2) improving the performance of interpretability algorithms. But for more information on these and many other interesting topics, you’ll have to watch the full conversation with our panelists. We look forward to continuing to share fascinating perspectives on the future of AI.
