Colorectal cancer (CRC) is the second leading cause of cancer death in the world, so digital pathology is essential for assessing prognosis. Deep learning (DL) methods can provide a viable alternative to histopathology image (HI) analysis, however, DL architectures may not be sufficient to classify CRC tissues based on anatomical histopathology data. To address this issue, a hybrid deep learning method was proposed which combines a dilated ResNet (dResNet) structure and attention module with neighborhood component analysis (NCA) and a deep support vector machine (SVM) based on an ensemble learning algorithm called DeepSVM. The hybrid model achieved 98.75% and 99.76% accuracy on CRC datasets, outperforming state-of-the-art approaches in terms of computational efficiency and time. This hybrid deep learning method can be used to accurately predict CRC based on pathology image classification.