The interpretability problem, also known as AI explainability, is the challenge of understanding and predicting what AI is doing. Deep learning systems, which are the technology that undergird Large Language Models, image generators, deepfake apps and AlphaFold, are artificial neural networks modeled after human neural networks. As it stands, our understanding of these systems is so opaque, AI models are commonly referred to as “black boxes.” Aliya Babul, an AI/quant expert, defines the AI interpretability problem as “what is the AI doing and why is it doing it?”
