Recently, steganalytic methods based on deep learning have achieved much better performance than traditional methods based on handcrafted features. However, most existing methods based on deep learning are specially designed for one image domain (i.e., spatial or JPEG), and they often take long time to train. To make a balance between the detection performance and the training time, in this paper, we propose an effective and relatively fast steganalytic network called US-CovNet (Universal Steganalytic Covariance Network) for both the spatial and JPEG domains. To this end, we carefully design several important components of US-CovNet that will significantly affect the detection performance, including the high-pass filter set, the shortcut connection and the pooling layer. Extensive experimental results show that compared with the current best steganalytic networks (i.e., SRNet and J-YeNet), US-CovNet can achieve the state-of-the-art results for detecting spatial steganography and have competitive performance for detecting JPEG steganography. For example, the detection accuracy of US-CovNet is at least 0.56% higher than that of SRNet in the spatial domain. In the JPEG domain, US-CovNet performs slightly worse than J-YeNet in some cases with the degradation less than 0.78%. However, the training time of US-CovNet is significantly reduced, which is less than 1/4 and 1/2 of SRNet and J-YeNet respectively.