Stability of intelligent energy system and intelligent control methods

In modern power systems, a variety of both objects and the tools of control is expected to be much larger than before. As a result, the dynamic properties of these systems are complicated, and the issues of maintaining stability come to the fore. The paper provides a brief overview of the types of stability, including those that, until recently, were considered local in the electric power systems of Russia. It is shown that in today’s conditions the violation of these types of stability affects the operation of the electric power system as a whole. Therefore, the coordination of control of both normal and emergency modes of the systems takes on a special role and should become more intelligent. In this regard, a brief overview of machine learning developments of control agents at different levels of the control hierarchy is


Stability and instability of the power system
Electric power system (EPS) stability is usually defined as the ability of a system to return to its original or close state after a disturbance. Any system has a "natural" margin of stability, and during its functioning tends to a state of equilibrium -that is, a balance (zero sum) of the forces (in the broad sense) at any point (both at nodes and at any points on the links). Mathematically, this is expressed by the equation of the first Kirchhoff law. However, in the process of its functioning, the system is always subjected to external and internal disturbances. With strong perturbation, the "natural" resource of stability may not be sufficient. In this case, to bring the system to an equilibrium state, admissible by the values of its state parameters, control actions are required, which should be provided by devices and control systems.
Traditionally, in the EPSs of Russia (and earlier, the USSR), under the stability of the system was understood, mainly and primarily, angle stability. Other types of stability have historically been regarded as having a local (non-systemic) character. The traditional principles of controlling the EPS modes were based on the use of the regulatory effect of the load and the frequency characteristics of the generation. It is precisely due to these effects that a "natural" resource of stability was provided.
However, over the past decades, the structure and operation modes of Russian EPSs have undergone significant transformations in all segments -generation, consumption, and the electric grid.
At present and in the near future, the use of power electronics, rectifier-inverter devices, photovoltaic installations, storage devices and other similar devices significantly reduces the regulatory effect of the load in voltage and frequency and the frequency regulatory effect of generation and significantly change the properties and controllability of EPS. Connection of distributed generation units to a distribution electric network radically changes its properties, creating stability problems, forming the need for significant development and fundamental reconstruction of relay protection and automation systems at this level [1].
In addition, the increase in power consumption with the dispersal of generating sources and consumers throughout the territory leads to an increase in the density of transmission and distribution electric networks [1]. A dense network (or, equivalently, a concentrated node) is an electric network that has very strong cross sections that do not limit the exchange of power between its parts. Moreover, if the instability in the angle (instability of the parallel operation of the generators) is characteristic of the connection or cross section between two parts of the EPS (including between concentrated nodes of the EPS), then in a dense network the types of instability are much more diverse (see Table 1). It is in a dense network that such types of stability as voltage, frequency and thermal stability acquire a system-wide nature.
Based on the classical diagram of the stability structure of P. Kundur [2] and taking into account the analysis of the types of stability according to [3], Fig. 1 presents a simplified structure of cause-effect relationships between stability disorders of different types and the corresponding states of the EPS. Violations of stability are closely intertwined and difficultly connected by causal relationships with each other and with the phenomena (events) in the EPS. Some of these phenomena are local in nature (for example, an increase in voltage that is dangerous for equipment), while others affect the fundamentals of the operation of EPS, the stability of the entire EPS or a significant part of it. Table 1. Types of instability characteristic of a dense network.

By frequency
Separation of the dense network from the rest of the EPS with an emergency power imbalance exceeding the regulatory capabilities of the generating sources and the load of the separated part.

By voltage (voltage reduction)
• severe short circuit in a dense network or a slow decrease in voltage due to a shortage of reactive power; • asynchronous mode in section, to which a dense network adjoins as an intermediate load node or additional generation; • synchronous swings in a section close to the unit with the load.

By angle
• "density loss" -that is loss of a number of network elements with a significant weakening of some of its sections compared to other sections; • a heavy protracted short circuit near the power plant tires, which can cause a rapid violation of the dynamic stability of the station generators relative to the rest of the EPS.

By heating (violation of thermal stability)
Overloading the line with current creates heating wires. The most common cause of current overload is the disconnection of parallel or shunt lines. For a power system, a cascade process of overloading and shutting off several shunting lines is especially dangerous.
Adapted from [2], [3] Note to the table. Like stability in angle, it is also advisable to subdivide voltage stability into state stability at small perturbations (static, Long-Term) and transition stability between states at large perturbations (dynamic, Short-Term). Note. Frequency instability or voltage instability can lead to shutdown of all thermal power plants, the first -due to the shutdown of auxiliary engines, the second -sometimes for the same reason and more often -due to overload of generators with reactive power.
Any of the mentioned phenomena can develop into another, even more dangerous for EPS, spread in one form or another to other parts of EPS, and ultimately lead to its catastrophic collapse [3].

Hierarchy of control and hierarchy of intelligence
In the EPS, the control systems (operational dispatch and automatic, normal and emergency modes) are organized according to the hierarchical principle: control of the energy object (generating, consuming, network) -control of the energy district (district power system) -control of the interconnected power system -control of a unified energy system.
If the control in a certain area is performed locally, then the control action performed in the interests of the section of the given region can cause the section to be overloaded in another region of this EPS. This drawback of local control can be eliminated by adding it to the coordination system (the simplest is hierarchical) in order to take into account the real multi-node nature of the network, covering all areas and sections of EPS [3].
Intelligence (or its absence) is manifested in the control of the system, aimed at ensuring the functioning of the system in the field of admissible values of its state parameters. More strictly, the intelligence of the unified can be defined as the intelligence of its devices and control systems, and the ability of higher levels of control to coordinate and adjust the actions of lower levels. In this case, it is advisable to consider devices and systems that are part of the control hierarchy as agents and use existing and intensively developing approaches to agent control and multi-agent systems. An "agent" (in general terms) is usually defined as a certain entity that (regardless of its physical nature) may possess some or all of the following properties [4]: adaptability (learning ability), autonomy (ability to conduct independent purposeful actions), communicative ability (ability to communicate with other agents), collaboration (ability to collective behavior -that is, coordinated interaction with other agents to achieve common goals), ability to reason (possess of partial knowledge and / or inference mechanisms).
In relation to the power system, the use of agents listed in Table 2 was proposed in [5]. Moreover, the control levels (types) are based on the multilevel organization of complex control systems for a dynamic system proposed in [6], which was used in the development of the IES Concept [7] and detailed in relation to the IES in [5]. Agents, depending on the level of intelligence, take pre-fixed actions, or formed actions with a fixed goal, or carry out adjustment (revision) of goals for lower levels. 1. Starting with positional control (level 2), feedback from the control object is taken into account, and intelligent control (level 6) takes into account the influence of the external environment.
2. The key difference between intelligent control (level 6) from intelligent (level 5) is the presence of goal-setting. 3. Starting from level 4, the agent must assess the state of the groups of managed objects, from level 6 -the state of the managed system, at level 7 -also evaluate the possible consequences of decisions. 4. Agents of level 5 and above must have the ability to learn.
According to [7], the existing automated technological control system of the Unified national electric grid of Russia is responsible for the implemented automatic control functions at levels 1-2 and partially at levels 3-4. The active and adaptive control system currently being created for the implemented automated and automatic control functions will correspond to levels 1-6.
It should be noted that the hierarchy of intellectual levels of agents from Table 2 does not necessarily coincide with the hierarchy of EPS control given at the beginning of this section. On the contrary, agents of different intellectual levels can function at almost any level of this hierarchy. Moreover, depending on the presence and orientation of inter-agent relationships, a wide variety of architectures of a multi-agent system is possible (from vertically integrated to fully distributed).
It is clear that the larger the range of control tasks to be solved at a lower level, the less information will have to be exchanged with a higher one (respectively, the speed of information exchange will increase) and the lower will be the load of computing power of a higher level (accordingly, the speed of decision-making on it will increase ) At the same time, the more control functions can be delegated from a higher level of energy system control to a lower one, the more "intelligent" a lower level should be.
Thus, the presence of highly intelligent agents is desirable at any level of the control hierarchy. The degree of intelligence of the agent is determined by the variety of information with which the agent is able to work. Increasing the intellectual level of the agent can be carried out in the process of its (agent) functioning through machine learning.
The next section presents a brief overview of ways to creating intelligent agents using machine learning.

Algorithms and models of machine learning in the tasks of intelligent power systems control
The development of artificial intelligence (AI) methods has made it possible to significantly accelerate and automate the solution of a whole range of problems in the control of electric power systems (EPS) [8].
One of the actively developed areas is the application and implementation of machine learning (ML) technology, including methods for constructing algorithms that can be trained, such as artificial neural networks (ANNs), reference vector machines, decision trees, etc.
The use of various types of training models of ML: supervised learning, unsupervised learning, reinforcement learning, deep learning, etc., allowed to Their main advantages are speed, high adaptability, the ability to approximate nonlinear functions and the presence of a certain kind of machine intelligence, which allows to develop the most autonomous systems capable of independent decision making based on experience and original properties for generalization. 2. Isolated power system and micro-grid control [11] 3. Voltage and reactive power regulation [14] [12] [13] 4. Energy consumption control [15] Emergency mode 5. Power system blackout prevention [16] 6. Peak load and overload control [17] 7. Control in transition modes [18] [19], [20] Recovery mode 8. Restoration of the power system [21] Abbreviations: RL -Reinforcement Learning; SL -Supervised Learning; DL -Deep Learning; DRL -Deep Reinforcement Learning; USL -Unsupervised Learning. Table 3 presents some of the tasks of control the EPS modes that are successfully solved on the basis of the Ministry of Defense, including in combination with other AI technologies.
Studies show that agents trained offline based on the methods of reinforcement learning, successfully control the individual components of the EPS in the control of normal and emergency conditions.
For example, in [22], the global generator excitation regulator WASCCO, implemented on the basis of reinforcement learning for damping electromechanical vibrations and controlling the transient voltage, allows the excitation system to "learn" when interacting with EPS and to effectively predict its future states.
Tests have shown that this approach gives better damping and a transition characteristic than traditional system stabilizers and allows damping interregional mode oscillations better than a regulator based on heuristic programming, acting independently.
Good results were also obtained when Q-agents were a dynamic brake, a thyristor-controlled series capacitor, synchronous generators, individual or aggregated loads, etc., to implement optimal control strategies [23]. Moreover, the use of reinforcement learning showed good results in the whole spectrum of tasks of operational dispatch and emergency control.
Recent developments indicate that the use of promising technology of deep machine learning allows the development of fully autonomous intelligent control systems. For example, Siemens is already applying deep ANNs in various projects: improving the operation of SIPROTEC multifunctional relay protection devices, optimizing the operation of gas turbines to reduce emissions of toxic nitrogen oxides, intelligently adjusting the position of the wind turbine rotors depending on the wind direction [24].
In the general case, deep machine learning is a part of a wider family of ML methods of teaching representations, where the feature vectors are located simultaneously on many levels.
As a rule, we are talking about special ANNs with many hidden layers of neurons (levels), which are based on special building blocks, for example, restricted Boltzmann machine, which allow for network pretraining, training each layer separately.
However, on the basis of the combination of reinforcement learning methods and deep ANNs, technologies have been created that have become breakthroughs of recent times in the field of AI, such as AlphaGo, Atari Deep Q-learning and others. These developments have shown that such models are able to solve problems better than the person himself (an expert in the subject area) or completely displace the classic, traditional algorithm for solving the problem. According to some experts, this particular technology will allow in the near future to create a full-fledged machine intelligence similar to human [25].

Conclusion
Historically, in the power systems of the USSR, and then Russia, the angle stability was considered as the main type of system stability, and the principles of EPS operation control were based on the use of the regulatory effect of the load and the frequency characteristics of generation. Due to these effects, traditional EPSs had a "natural" resource of stability, and control systems acted when parameters of EPS condition exceeded certain boundaries.
However, the new characteristics of consumers, storages and the generators of future EPSs significantly change the properties and controllability of systems, and the increase in power consumption with the distribution of generating sources and consumers across the territory leads to an increase in the density of transmission and distribution electric networks. Therefore, traditionally "local" types of stability (in terms of voltage, frequency, and heating) acquire a system-wide nature in the EPSs of Russia, which previously was typical only for "western" EPSs with a dense electric network.
Already now there are, and in the future will be aggravated, new problems associated with the need to strengthen coordination of control of EPS conditions at various levels, improve control efficiency, ensure the reliability of control systems themselves. Therefore, the traditional principles of control of EPS conditions need to be substantially modified and developed.
The intelligence of a power system can be defined as the intelligence of its control devices and systems, and the ability of higher levels of control to coordinate and adjust the actions of lower levels. In this case, it is advisable to consider devices and systems that are part of the control hierarchy as agents, and use existing and intensively developing approaches to agent control and multi-agent systems.
The presence of highly intelligent agents is desirable at any level of the control hierarchy. Increasing the intellectual level of an agent can be carried out in the process of its functioning through machine learning. Currently, various machine learning technologies are being intensively developed for use in the control of intelligent EPS modes, including both normal mode control and emergency control, as well as system recovery control after accidents. Recent developments indicate that the use of deep machine learning technology allows the development of fully autonomous intelligent control systems that have already found application at power facilities.
The work was carried out within the framework of the scientific project of SB RAS III.17.4.2. "`Theory and methods of substantiating the development and control of intelligent electric power systems"' (No. AAAA-A17-117030310438-1).