With the advancements in Artificial Intelligence (AI) technology, Large Language Models (LLMs) provide outstanding capabilities for natural language understanding and generation, enhancing various domains. In psychiatry, LLMs can empower healthcare by analyzing vast amounts of medical data to improve diagnostic accuracy, enhance therapeutic communication, and personalize patient care with their strength in understanding and generating human-like text. In clinical AI, developing and utilizing robust and interpretable models has been a longstanding challenge. This survey investigates the current psychiatric practice of LLMs, along with a series of corpus resources that could be used for training psychiatric LLMs. We discuss the limitations concerning LLM reproducibility, capabilities, usability, interpretability in clinical settings, and ethical considerations. Additionally, we propose potential future directions for research, clinical application, and education in psychiatric LLMs. Finally, we discuss the challenge of integrating LLMs into the evolving landscape of healthcare in real-world scenarios.