The Department of Defense and Air Force depend on cyber systems for mission enablement and delivery. Maintaining the core cyber capabilities of the DoD and Air Force requires high-fidelity security assessments of their software systems, as well as cyber security solutions. Several Cyber Test and Evaluation (T&E) infrastructures (cyber ranges) provide experimenters with the testing capabilites to conduct emulated or real cyber exercises. However, current cyber ranges and their toolsets require significant manpower and time, and the T&E exercises need to be more effective. To address these critical needs, IAI and its partner, Raytheon BBN Technologies (Raytheon BBN), are developing Hermes, an innovative framework and toolkit to prepare, visualize, automate, and debug cyber security T&E exercises. It assists cyber security researchers in test planning, preparation, visualization, automation, debugging, report generation, and integration with existing third-party tools. Hermes reduces the complexity and time required to create cyber tests and provides a powerful tool for analyzing both attacking and defending approaches. Hermes uses automation to increase efficiency, reduce costs, and provide a customizable real-time and post-test visualization of the T&E process to enable improved situational awareness for cyber security testers.