Recognizing dynamic variations on the ground, especially changes caused by various natural disasters, is critical for assessing the severity of thedamage and directing the disaster response. However, current workflows for disaster assessment usually require human analysts to observe and identify damaged buildings, which is labor-intensive and unsuitable for large-scale disaster areas. In this paper, we propose a difference-aware attention network (D2ANet) for simultaneous building localization and multi-level change detection from the dual-temporal satellite imagery. Considering the differences in different channels in the features of pre- and post-disaster images, we develop a dual-temporal aggregation module using paired features to excite change-sensitive channels of the features and learn the global change pattern. Since the nature of building damage caused by disasters is diverse in complex environments, we design a difference-attention module to exploit local correlations among the multi-level changes, which improves the ability to identify damage on different scales. Extensive experiments on the large-scale building damage assessment dataset xBD demonstrate that our approach provides new state-of-the-art results. Source code is publicly available at https://github.com/mj129/D2ANet.