Information-theoretic Methods In Deep Learning: Theory And Applications

Information theory is a mathematical infrastructure to deal with manipulation of information and has a significant influence on the design of efficient and reliable communication systems. Information theoretic learning (ITL) has been used for different deep learning problems, such as mutual information neural estimator for representation learning and the principle of relevant information for redundancy compression and graph sparsification. ITL has applications in supervised, unsupervised and reinforcement learning problems, such as classification, clustering, and sequential decision making. The Information Bottleneck (IB) theory helps to better understand the basic limits of learning problems, such as the learning performance of deep neural networks, geometric clustering, and extracting the Gaussian part of a signal. In recent years, researchers have revealed that ITL provides a powerful paradigm for analyzing neural networks. Divergence estimation is another approach with a broad range of applications.