Motion tracking via Inertial Measurement Units (IMUs) on mobile and wearable devices has attracted significant interest in recent years. High-accuracy IMU-tracking can be applied in various applications, such as indoor navigation, gesture recognition, text input, etc. Many efforts have been devoted to improving IMU-based motion tracking in the last two decades, from early calibration techniques on ships or airplanes, to recent arm motion models used on wearable smart devices. In this paper, we present a comprehensive survey on IMU-tracking techniques on mobile and wearable devices. We also reveal the key challenges in IMU-based motion tracking on mobile and wearable devices and possible directions to address these challenges.