With the rapid development of cyberspace and smart home technology, human life is changing to a new virtual dimension with several promises for improving its quality. Moreover, the heterogeneous, dynamic, and internet-connected nature of smart homes brings many privacy and security difficulties. Unauthorized access to the smart home system is one of the most harmful actions and can cause several trust problems and relationship conflicts between family members and invoke home privacy issues. Access control is one of the best solutions for handling this threat, and it has been used to protect smart homes and other Internet of Things domains for many years. This survey reviews existing access control schemes for smart homes, which concern the essential authorization requirements and challenges that need to be considered while designing an authorization framework for smart homes. Furthermore, we note the most critical challenges that other access control solutions neglect for smart homes.