Rural building is important to the well-being of rural residents, leading to a significant need to carry out extensive surveys and retrofits of many rural buildings. On-site surveys by expert surveyors are currently the main approach, but this traditional method is often expensive and laborious, especially for large-scale survey tasks. Therefore, this study explores an alternative workflow based on deep learning (DL) methods to apply automatic classification of rural building characteristics. Taking four villages in Jizhou District of Tianjin, China as research samples, we tested selected convolutional neural network (CNN) architectures through the establishment of the training database containing 3258 labeled images, under the performance metrics of accuracy, recall and F1 score. The results showed that ResNet50 is the CNN architecture with the best performance, with the comprehensive consideration of overall metrics. Taking accuracy as the performance metric to test the generalization ability of ResNet50, the prediction results for seven building characteristic indicators from low to high are as follows: building function (0.827); building style (0.863); building quality (0.871); building age (0.880); building structure (0.891); abandoned or not (0.959); the number of storeys (0.995). Due to simplicity, accuracy and effectiveness, this workflow is transferable and cost-effective to investigate large-scale villages.