Collage: A New Machine Learning Approach To Deal With Floating-point Errors In Low-precision To Make Llm Training Accurate And Efficient

COLLAGE is a novel approach that uses a Multi-Component Float (MCF) representation to optimize efficiency and memory usage during training of large language models. It addresses challenges related to numerical inaccuracies and restricted representation ranges, resulting in significant improvements in training throughput and memory savings.