An Architecture for Deploying Reinforcement Learning in Industrial Environments

G. Schäfer; R. Kozlica; S. Wegenkittl; S. Huber

doi:10.1007/978-3-031-25312-6_67

An Architecture for Deploying Reinforcement Learning in Industrial Environments

G. Schäfer^*
, R. Kozlica
, S. Wegenkittl
, S. Huber

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Industry 4.0 is driven by demands like shorter time-to-market, mass customization of products, and batch size one production. Reinforcement Learning (RL), a machine learning paradigm shown to possess a great potential in improving and surpassing human level performance in numerous complex tasks, allows coping with the mentioned demands. In this paper, we present an OPC UA based Operational Technology (OT)-aware RL architecture, which extends the standard RL setting, combining it with the setting of digital twins. Moreover, we define an OPC UA information model allowing for a generalized plug-and-play like approach for exchanging the RL agent used. In conclusion, we demonstrate and evaluate the architecture, by creating a proof of concept. By means of solving a toy example, we show that this architecture can be used to determine the optimal policy using a real control system. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Original language	English
Title of host publication	Computer Aided Systems Theory – EUROCAST 2022
Subtitle of host publication	18th International Conference, Las Palmas de Gran Canaria, Spain, February 20–25, 2022, Revised Selected Papers
Publisher	Springer Nature
Number of pages	8
Volume	13789 LNCS
ISBN (Electronic)	978-3-031-25312-6
ISBN (Print)	978-3-031-25311-9
DOIs	https://doi.org/10.1007/978-3-031-25312-6_67
Publication status	Published - Feb 2023
Event	18th International Conference on Computer Aided Systems Theory, EUROCAST 2022 - Las Palmas de Gran Canaria, Spain Duration: 20 Feb 2022 → 25 Feb 2022 https://eurocast2022.fulp.ulpgc.es/

Publication series

Name	Lecture Notes in Computer Science
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	18th International Conference on Computer Aided Systems Theory, EUROCAST 2022
Abbreviated title	EUROCAST 2022
Country/Territory	Spain
City	Las Palmas de Gran Canaria
Period	20/02/22 → 25/02/22
Internet address	https://eurocast2022.fulp.ulpgc.es/

Keywords

Cyber physical system
Digital twin
Hardware-in-the-loop simulation
Industrial control system
OPC UA
Reinforcement learning
Computer architecture
Control systems
Cyber Physical System
Embedded systems
Learning systems
Cybe-physical systems
Cyber-physical systems
Hardwarein-the-loop simulations (HIL)
Industrial control systems
Industrial environments
Mass customization
OPC UA
Product sizes
Reinforcement learnings
Time to market

Classification according to Österreichische Systematik der Wissenschaftszweige (ÖFOS 2012)

202022 Information technology

Applied Research Level (ARL)

Not applicable

Research focus/foci

Industrial Informatics

Access to Document

10.1007/978-3-031-25312-6_67

https://www.scopus.com/inward/record.uri?eid=2-s2.0-85151127093&doi=10.1007%2f978-3-031-25312-6_67&partnerID=40&md5=11d96536ab559d64dd3d604dd7d1dfe2

JRZ ISIA: Josef Ressel Centre for Intelligent and Secure Industrial Automation
Huber, S. (PI), Hoher, S. (CoPI), Kranzer, S. (CoI), Schäfer, G. (CoI), Siebenhandl, N. (CoI), Uray, M. (CoI), Saßnick, O. (CoI), Unger, H. (CoI), Brettfeld, K. (CoI), Schindler, S. (CoI), Enzensberger, L. (CoI), Unger, A. (CoI), Rosenstatter, T. (CoI), Reich, E. S. (CoI), Nosrati, K. (CoI), Lürzer, L. (CoI), Zeng, S. (CoI), Sain, D. (CoI), Messineo, S. (CoI), Entleitner, F. (CoI), Haratzmüller, S. M. (CoI), Huber, L. (CoI), Pop, A.-I. (CoI), Rehrl, J. (CoI), Rosenstatter, T. (CoI), Schäfer, G. (CoI), Uray, M. (CoI) & Langschwert, J. (CoI)
1/07/22 → 30/06/27
Project: Funded research

Cite this

Schäfer, G., Kozlica, R., Wegenkittl, S., & Huber, S. (2023). An Architecture for Deploying Reinforcement Learning in Industrial Environments. In Computer Aided Systems Theory – EUROCAST 2022: 18th International Conference, Las Palmas de Gran Canaria, Spain, February 20–25, 2022, Revised Selected Papers (Vol. 13789 LNCS). (Lecture Notes in Computer Science). Springer Nature. https://doi.org/10.1007/978-3-031-25312-6_67

Schäfer, G. ; Kozlica, R. ; Wegenkittl, S. et al. / An Architecture for Deploying Reinforcement Learning in Industrial Environments. Computer Aided Systems Theory – EUROCAST 2022: 18th International Conference, Las Palmas de Gran Canaria, Spain, February 20–25, 2022, Revised Selected Papers. Vol. 13789 LNCS Springer Nature, 2023. (Lecture Notes in Computer Science).

@inproceedings{5926a6beca9d46b59a522d65e25f4e1f,

title = "An Architecture for Deploying Reinforcement Learning in Industrial Environments",

abstract = "Industry 4.0 is driven by demands like shorter time-to-market, mass customization of products, and batch size one production. Reinforcement Learning (RL), a machine learning paradigm shown to possess a great potential in improving and surpassing human level performance in numerous complex tasks, allows coping with the mentioned demands. In this paper, we present an OPC UA based Operational Technology (OT)-aware RL architecture, which extends the standard RL setting, combining it with the setting of digital twins. Moreover, we define an OPC UA information model allowing for a generalized plug-and-play like approach for exchanging the RL agent used. In conclusion, we demonstrate and evaluate the architecture, by creating a proof of concept. By means of solving a toy example, we show that this architecture can be used to determine the optimal policy using a real control system. {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.",

keywords = "Cyber physical system, Digital twin, Hardware-in-the-loop simulation, Industrial control system, OPC UA, Reinforcement learning, Computer architecture, Control systems, Cyber Physical System, Embedded systems, Learning systems, Cybe-physical systems, Cyber-physical systems, Hardwarein-the-loop simulations (HIL), Industrial control systems, Industrial environments, Mass customization, OPC UA, Product sizes, Reinforcement learnings, Time to market",

author = "G. Sch{\"a}fer and R. Kozlica and S. Wegenkittl and S. Huber",

note = "Conference code: 291639 Cited By :3 Export Date: 14 December 2023 Correspondence Address: Sch{\"a}fer, G.; Salzburg University of Applied SciencesAustria; email: [email protected] Funding details: AB292 KI-Net Funding details: 20102-F1901166-KZP Funding details: Christian Doppler Forschungsgesellschaft, CDG Funding text 1: Georg Sch{\"a}fer and Stefan Huber are supported by the European Interreg {\"O}sterreich-Bayern project AB292 KI-Net and the Christian Doppler Research Association (JRC ISIA). Reuf Kozlica is supported by the Lab for Intelligent Data Analytics Salzburg (IDA Lab) funded by Land Salzburg (WISS 2025) under project number 20102-F1901166-KZP. References: Push-ing the Limits of the Maximum Punch-Through Design with an Advanced Buffer for Thin Wafer IGBTs (2020) Proc. Int. Symp. Power Semicond. Devices Ics, pp. 509-512; Kaelbling, L.P., Littman, M.L., Moore, A.W., Reinforcement learning: A survey (1996) J. Artif. Intell. Res., 4 (1), pp. 237-285; Kober, J., Bagnell, J.A., Peters, J., Reinforcement learning in robotics: A survey (2013) Int. J. Robot. Res., 32 (11), pp. 1238-1274; Kritzinger, W., Karner, M., Traar, G., Henjes, J., Sihn, W., Digital twin in manufacturing: A categorical literature review and classification (2018) Ifac-Papersonline, 51 (11), pp. 1016-1022; Mahnke, W., Leitner, S.H., Damm, M., (2009) OPC Unified Architecture, , https://doi.org/10.1007/978-3-540-68899-0, Springer Science \& Business Media; Mai, V., Mani, K., Paull, L., Sample efficient deep reinforcement learning via uncertainty estimation (2022) International Conference on Learning Representations; Nian, R., Liu, J., Huang, B., A review on reinforcement learning: Introduction and applications in industrial process control (2020) Comput. Chem. Eng., 139; (2020) OPC Foundation: IEC-62541, 14. , PubSub; Foundation, O.P.C., IEC-62541 (2020) Part 5: Information Model; Sutton, R.S., Barto, A.G., Reinforcement learning: An introduction. MIT press (2018) 2Nd Edn; Watkins, C.J.C.H., (1989) Learning from Delayed Rewards; 18th International Conference on Computer Aided Systems Theory, EUROCAST 2022, EUROCAST 2022 ; Conference date: 20-02-2022 Through 25-02-2022",

year = "2023",

month = feb,

doi = "10.1007/978-3-031-25312-6\_67",

language = "English",

isbn = "978-3-031-25311-9",

volume = "13789 LNCS",

series = "Lecture Notes in Computer Science",

publisher = "Springer Nature",

booktitle = "Computer Aided Systems Theory {\textendash} EUROCAST 2022",

address = "Switzerland",

url = "https://eurocast2022.fulp.ulpgc.es/",

}

Schäfer, G, Kozlica, R, Wegenkittl, S & Huber, S 2023, An Architecture for Deploying Reinforcement Learning in Industrial Environments. in Computer Aided Systems Theory – EUROCAST 2022: 18th International Conference, Las Palmas de Gran Canaria, Spain, February 20–25, 2022, Revised Selected Papers. vol. 13789 LNCS, Lecture Notes in Computer Science, Springer Nature, 18th International Conference on Computer Aided Systems Theory, EUROCAST 2022, Las Palmas de Gran Canaria, Spain, 20/02/22. https://doi.org/10.1007/978-3-031-25312-6_67

An Architecture for Deploying Reinforcement Learning in Industrial Environments. / Schäfer, G.; Kozlica, R.; Wegenkittl, S. et al.
Computer Aided Systems Theory – EUROCAST 2022: 18th International Conference, Las Palmas de Gran Canaria, Spain, February 20–25, 2022, Revised Selected Papers. Vol. 13789 LNCS Springer Nature, 2023. (Lecture Notes in Computer Science).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - An Architecture for Deploying Reinforcement Learning in Industrial Environments

AU - Schäfer, G.

AU - Kozlica, R.

AU - Wegenkittl, S.

AU - Huber, S.

N1 - Conference code: 291639 Cited By :3 Export Date: 14 December 2023 Correspondence Address: Schäfer, G.; Salzburg University of Applied SciencesAustria; email: [email protected] Funding details: AB292 KI-Net Funding details: 20102-F1901166-KZP Funding details: Christian Doppler Forschungsgesellschaft, CDG Funding text 1: Georg Schäfer and Stefan Huber are supported by the European Interreg Österreich-Bayern project AB292 KI-Net and the Christian Doppler Research Association (JRC ISIA). Reuf Kozlica is supported by the Lab for Intelligent Data Analytics Salzburg (IDA Lab) funded by Land Salzburg (WISS 2025) under project number 20102-F1901166-KZP. References: Push-ing the Limits of the Maximum Punch-Through Design with an Advanced Buffer for Thin Wafer IGBTs (2020) Proc. Int. Symp. Power Semicond. Devices Ics, pp. 509-512; Kaelbling, L.P., Littman, M.L., Moore, A.W., Reinforcement learning: A survey (1996) J. Artif. Intell. Res., 4 (1), pp. 237-285; Kober, J., Bagnell, J.A., Peters, J., Reinforcement learning in robotics: A survey (2013) Int. J. Robot. Res., 32 (11), pp. 1238-1274; Kritzinger, W., Karner, M., Traar, G., Henjes, J., Sihn, W., Digital twin in manufacturing: A categorical literature review and classification (2018) Ifac-Papersonline, 51 (11), pp. 1016-1022; Mahnke, W., Leitner, S.H., Damm, M., (2009) OPC Unified Architecture, , https://doi.org/10.1007/978-3-540-68899-0, Springer Science & Business Media; Mai, V., Mani, K., Paull, L., Sample efficient deep reinforcement learning via uncertainty estimation (2022) International Conference on Learning Representations; Nian, R., Liu, J., Huang, B., A review on reinforcement learning: Introduction and applications in industrial process control (2020) Comput. Chem. Eng., 139; (2020) OPC Foundation: IEC-62541, 14. , PubSub; Foundation, O.P.C., IEC-62541 (2020) Part 5: Information Model; Sutton, R.S., Barto, A.G., Reinforcement learning: An introduction. MIT press (2018) 2Nd Edn; Watkins, C.J.C.H., (1989) Learning from Delayed Rewards

PY - 2023/2

Y1 - 2023/2

N2 - Industry 4.0 is driven by demands like shorter time-to-market, mass customization of products, and batch size one production. Reinforcement Learning (RL), a machine learning paradigm shown to possess a great potential in improving and surpassing human level performance in numerous complex tasks, allows coping with the mentioned demands. In this paper, we present an OPC UA based Operational Technology (OT)-aware RL architecture, which extends the standard RL setting, combining it with the setting of digital twins. Moreover, we define an OPC UA information model allowing for a generalized plug-and-play like approach for exchanging the RL agent used. In conclusion, we demonstrate and evaluate the architecture, by creating a proof of concept. By means of solving a toy example, we show that this architecture can be used to determine the optimal policy using a real control system. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

AB - Industry 4.0 is driven by demands like shorter time-to-market, mass customization of products, and batch size one production. Reinforcement Learning (RL), a machine learning paradigm shown to possess a great potential in improving and surpassing human level performance in numerous complex tasks, allows coping with the mentioned demands. In this paper, we present an OPC UA based Operational Technology (OT)-aware RL architecture, which extends the standard RL setting, combining it with the setting of digital twins. Moreover, we define an OPC UA information model allowing for a generalized plug-and-play like approach for exchanging the RL agent used. In conclusion, we demonstrate and evaluate the architecture, by creating a proof of concept. By means of solving a toy example, we show that this architecture can be used to determine the optimal policy using a real control system. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

KW - Cyber physical system

KW - Digital twin

KW - Hardware-in-the-loop simulation

KW - Industrial control system

KW - OPC UA

KW - Reinforcement learning

KW - Computer architecture

KW - Control systems

KW - Cyber Physical System

KW - Embedded systems

KW - Learning systems

KW - Cybe-physical systems

KW - Cyber-physical systems

KW - Hardwarein-the-loop simulations (HIL)

KW - Industrial control systems

KW - Industrial environments

KW - Mass customization

KW - OPC UA

KW - Product sizes

KW - Reinforcement learnings

KW - Time to market

U2 - 10.1007/978-3-031-25312-6_67

DO - 10.1007/978-3-031-25312-6_67

M3 - Conference contribution

SN - 978-3-031-25311-9

VL - 13789 LNCS

T3 - Lecture Notes in Computer Science

BT - Computer Aided Systems Theory – EUROCAST 2022

PB - Springer Nature

T2 - 18th International Conference on Computer Aided Systems Theory, EUROCAST 2022

Y2 - 20 February 2022 through 25 February 2022

ER -

Schäfer G, Kozlica R, Wegenkittl S , Huber S. An Architecture for Deploying Reinforcement Learning in Industrial Environments. In Computer Aided Systems Theory – EUROCAST 2022: 18th International Conference, Las Palmas de Gran Canaria, Spain, February 20–25, 2022, Revised Selected Papers. Vol. 13789 LNCS. Springer Nature. 2023. (Lecture Notes in Computer Science). doi: 10.1007/978-3-031-25312-6_67