This thesis explores the potential of machine learning to augment the capacity of digital communication infrastructures. Autoencoders are a particularly promising avenue for this purpose, as they can optimize both the transmitter and receiver tailored to a specific channel model and performance metric. The main contributions of this thesis are twofold. Firstly, reinforcement learning-based transmitter optimization is explored to overcome the challenge of requiring a differentiable channel model for end-to-end autoencoder learning. Secondly, supervised learning is applied to optimize the receiver for a given channel model and performance metric. Simulation and experimental results demonstrate the effectiveness of the proposed methods.
