This article discusses the impact of deep learning on modern AI applications and how it has revolutionized many fields. It also examines the disconnect between theory and practice in deep learning and the role of multi-layer perceptrons (MLPs) in modern settings. Researchers from ETH Zürich conducted a series of experiments to evaluate MLPs’ performance in modern settings and explore the role of inductive bias and the impact of scaling compute resources.