Dexterous hand manipulation is a crucial ability for robots in various applications. However, ensuring safety and reliability during manipulation poses significant challenges. Safe Reinforcement Learning (Safe RL) algorithms are important to ensure robust performance and prevent damage to the robotic hand, manipulated object, or environment. Realistic and complex simulation platforms are needed to develop and evaluate such algorithms. Unfortunately, existing platforms have limitations in terms of realism, complexity, and customizability. To address these issues, we introduce ReDMan, an open-source simulation platform that provides a standardized implementation of safe RL algorithms for Reliable Dexterous Manipulation. ReDMan features challenging tasks based on real-world scenarios that require safety awareness, such as Jenga, as well as multi-modal observations and customizable robotic hardware. This platform facilitates the replication and comparison of experimental results and demonstrates the effectiveness of safe RL methods compared to classical RL algorithms. ReDMan is the first benchmark for safe dexterous manipulation and aims to bridge the gap between safe RL and dexterous manipulation research.