The Internet of Things (IoT) has grown rapidly due to artificial intelligence driven edge computing. While enabling many new functions, edge computing devices expand the vulnerability surface and have become the target of malware attacks. Moreover, attackers have used advanced techniques to evade defenses by transforming their malware into functionality-preserving variants. We systematically analyze such evasion attacks and conduct a large-scale empirical study in this paper to evaluate their impact on security. More specifically, we focus on two forms of evasion attacks: obfuscation and adversarial attacks. To the best of our knowledge, this paper is the first to investigate and contrast the two families of evasion attacks systematically. We apply 10 obfuscation attacks and 9 adversarial attacks to 2870 malware examples. The obtained findings are as follows. (1) Commercial Off-The-Shelf (COTS) malware detectors are vulnerable to evasion attacks. (2) Adversarial attacks affect COTS malware detectors slightly more effectively than obfuscated malware examples. (3) Code similarity detection approaches can be affected by obfuscated examples and are barely affected by adversarial attacks. (4) These attacks can preserve the functionality of original malware examples.