Paving the Way for Reinforcement Learning in Smart Grid Co-simulations

This paper identifies and addresses a gap in research on using reinforcement learning (RL) in co-simulation. Co-simulation is an effective simulation paradigm for systems of systems such as smart grids. It relies on combining heterogeneous simulators into a coupled simulation. RL is a promising machine-learning tool for complex grid applications—for instance, demand-side management. However, existing literature does not specifically address challenges of integrating RL with a co-simulation environment. Therefore, we focus on two challenges: how an RL agent is best integrated into a co-simulation architecturally, and to what extent typical RL frameworks are interoperable with orchestrated co-simulation tools. First, we introduce, categorize, and evaluate four approaches of architecturally integrating RL into co-simulation. Additionally, we provide guidance on selecting an appropriate approach. Second, we conduct a case study where we use and incorporate a framework-based RL agent into a co-simulation framework for a simple demand-side management scenario; we identify the need to change the control flow traditionally used in RL frameworks to achieve interoperability. In conclusion, our work is a basis for future academic or industrial applications of RL in co-simulation. Our architectural and framework-specific advice facilitates the implementation of RL in smart-grid co-simulations.