How Sure Is Sure? Incorporating Human Error Into Machine Learning

Researchers from the University of Cambridge, The Alan Turing Institute, Princeton, and Google DeepMind are attempting to bridge the gap between human behavior and machine learning by incorporating uncertainty into AI systems. They adapted a well-known image classification dataset so that humans could provide feedback and indicate their level of uncertainty when labeling a particular image. The results of their research will be reported at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society (AIES 2023) in Montréal. Their findings suggest that training with uncertain labels can improve the performance of these hybrid systems, although humans also cause the overall performance of these systems to drop.