Data Engineering For Hpc With Python

Researchers from the Luddy School of Informatics, Computing and Engineering, Digital Science Center, Bloomington, IN, Indiana University, and the Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka, have developed a Python API that uses table abstraction to represent and process data using high-speed compute kernels via C++. This paper compares the proposed solution with existing data engineering libraries in Python and big data, and discusses the challenges and approaches to implementing big data systems, high-performance computing (HPC) for data engineering, and Python for data engineering. Cylon and PyCylon libraries and frameworks can be effective resources for high-performance data engineering.