Expert assessment systems to support decision-making for sustainable development of complex technological and socio-economic facilities

. In this paper, we investigate problems of decision making in management systems for the sustainable development of complex technological and socio-economic facilities. We show both the limitations of traditional expert systems and decision support systems, and the necessity of using expert evaluation technologies to find possible development strategies. Based on that we substantiate the need of creating a new class of systems, i.e. Automated eXpert Assessment Systems, and propose their organizational structure and design principles. W e substantiate the level of automation of the work performed during the examinations and describe the composition of models and computer programs we recommend for creating effective automated expert assessment systems and corresponding technology. In the paper, we give examples of using the proposed method for various areas of human activity, in the management of urban infrastructure and e-learning at the universities, and show the effectiveness of the developed approach.


Introduction
World leaders increasingly declare the need to protect the Earth from degradation by [1][2][3][4][5]: • Rational use of natural resources, • Introducing rational models of production and consumption, • Taking urgent action due to climate change.
Features of the modern stage of development of society such as [13][14][15][16]: • Transition to a digital economy, • Widespread use of e-government and e-society technologies, • Introduction of complex hierarchical systems for managing enterprises, corporations, territories, technologies for ensuring the vital functions of regions, actualize the problem of finding quick optimal solutions for managing actions [1,10,12].
For such new complex organizational and technical systems that operate under high risks and uncertainty conditions, it becomes almost impossible to create a unified global mathematical model that is convenient for decision support.
In these conditions, it is increasingly necessary to use hybrid intelligence technologies and attract highly qualified specialists for expert assessment of possible scenarios for the sustainable development of production, society, region, state [1,8,17].

Problem analysis and research goals setting
Some simple choice problems one may reduce to mathematical models that allow finding the optimal solution for the problems. Unfortunately, the number of well-formalized problems is significantly inferior to the number of poorly formalized or non-formalized ones, for the solution of which the operation analysis methods are not suitable [8,19]. The poor formalizability of many choice problems is associated with the presence of socalled uncertainty, which means that decision-making is influenced by factors that cannot be unambiguously defined and described for one reason or another [1,18].
The nature and causes of the uncertainty can be very different, including [18,20]: • Incomplete or unclear information available, • Stochastic nature of factors, • Heterogeneity and subjectivity of the criteria for evaluating alternatives. Therefore, experts in the decision-making theory are constantly exploring and developing methods for describing and accounting for uncertainties of various kinds. For example, Lotfi Zadeh created the theory of fuzzy logic [21], which allows us to describe the ambiguity of statements. Currently, various logical-probabilistic and logical-linguistic models are widely used for modelling uncertainty [22].
Nevertheless, there are choice problems in which the uncertainty factor has not yet received a model that allows it to be adequately described and taken into account when searching for a solution.
In such cases, they speak of tasks of unique choice, i.e., arising for the first time in a sense [1,23].
Often, when modelling complex hierarchical socioeconomic systems, a task can contain several different types of uncertainty at once, which greatly complicates the search for a solution by formal methods focused on interacting with uncertainties of one particular type.
There is a specific area for such problems in decision theory -expert assessment methods, the cornerstone of which is the person himself with his subjective preferences as the primary source of the choice problem. The expert assessment methods are based on the assumption that it is possible to find, if not the optimal, then at least a good solution, analyzing the judgments and preferences of a person interested in choosing the best alternative.
In some cases, a subject in a situation of choice cannot independently sort his preferences, and then you need to help him in this. In other cases, the subject simply does not have the necessary level of knowledge, that is, cannot solve the task at all.
Then, just as when modelling a real object is replaced by its model, the original subject is replaced by an expert -specialist who has the necessary level of knowledge so that, based on the judgments and preferences analysis, the choice problem can be solved. As a rule, instead of one, a whole group of experts is involved in order to reduce the influence of the subjectivity factor and collect more information.
Unfortunately, it is believed that the presence of a large library of mathematical methods for expert assessment of alternatives, for example [22,[24][25][26], completely solves the problem of choice in managing complex objects.
In fact, the incorrect organization of expert assessment, especially in the tasks of planning the sustainable development of complex organizational, technical and social systems, can lead to incorrect guidelines and huge losses.
We set up an experiment in relation to: • Planning socio-economic measures for the city development; • Examination of educational electronic modules in the e-learning system; • Selecting a vector for the development of an industrial enterprise (mechanical production).
The use of different expert groups and different methods for evaluating alternatives led to fundamentally different results.
In this regard, in this article we set the goal of determining design principles and system of measures that are necessary for automated expert assessment systems for decision-making to manage the sustainable development of complex technological and socioeconomic facilities.

Intelligent agent-manager for solving local problems of expert assessment
To solve local management problems, we propose using the idea of an intelligent agent-manager [27], which uses the most powerful intellectual resources (for today)people. This agent, through interaction with millions of Internet users, provokes interest in specialists and seeks a problem to be solved, presenting the problem in a natural form (Fig. 1, Fig. 2).  Agent-manager searches for users via the Internet and establishes communication with them. As a result of the agent's communication with users, a database is formed. Further, potential solvers, who fall into the database, participate in solving problems that the agent or decision maker (DM) offers them to solve for a financial compensation.
The problem description presented in natural language, as well as information about the material and time resources allocated for solving the problem, is fed to the system input. In some cases, before solving a problem, it can be convenient to decompose it. After the task is divided into k subtasks, the most suitable solvers are selected in the database of potential solvers. Selected solvers are given the subtasks, after which the agentmanager monitors the execution of work and after some time receives (ideally) k ready-made solutions. The k received decisions are analysed and, with the help of the agent or decision maker, are transformed into the final solution of the problem.
The advantage of the agent-manager technology is its promise.
The disadvantages of the agent-manager technology: • The inability (at the present time) to solve complex tasks of a global level, • The inability to attract outsiders to the solution of strategic tasks, security tasks, secret tasks, • High risks associated with cybercrime.

The concept of an automated expert assessment system
For global complex tasks involving high risks and management costs, agent-manager technology cannot always be recommended for use. In this regard, we justify the need to create a fundamentally new class of systems -Automated eXpert Assessment Systems (AXAS).
Methods of expert assessment are not always directly aimed at solving the choice problem, and the purpose of the assessment can be associated with this task only indirectly.
The indirect goals of expert assessment may include: • Assessment of the products quality and their compliance with certain standards and requirements, • Forecast of the development dynamics and future state of the object of assessment, • Development of a system of criteria and methods for object assessment.
In addition to the expert group acting as a collective decision-maker, the working group should play an active role in the assessment.
The working group is responsible for organizational aspects and the conduct of assessment, ensures the effective work of experts and the timely results presentation.
In accordance with the functions performed, one may distinguish the subgroups within the working group: • managers -specialists who are directly responsible for organizing and conducting the assessment, • cognitive scientists -specialists who are responsible for extracting expert information and communicating both the working group with experts and experts among themselves, if it is necessary to manage communication within the expert group, • analysts -specialists who are responsible for the analysis and processing of information received from experts, • technical staff responsible for supporting functions.
We can say that expert assessment methods are based on the principle: "A bad plan is better than its complete absence, because even a bad plan makes it possible to meaningfully move towards a goal, controlling the process of achieving it".
The process of achieving the assessment objective is divided into several successive (iteratively repeated) stages. Some of the stages can be completely formalized, and some of them cannot be formalized (at the current level of science).
Thus, assessment is a complex organizational and analytical activity, the implementation of which requires resources: specialists, premises, hardware and software, finance, and time.
The need for the effective use of all the resources leads to the formulation of the task of automating activities related to conducting expert assessment, i.e. the development and implementation of AXAS.
Non-specialists sometimes mistakenly believe that such systems have long been functioning. Unfortunately, most often they mean local (limited) automation. Indeed, various kinds of Expert Systems (ES) and Decision Support Systems (DSS) have been used for several decades. However, they have significantly limited capabilities.
ES are intended for mass use and enable specialists of insufficiently high qualification to use the experience and knowledge of highly qualified expert to solve the problems they face. ES can be used to perform diagnosis, training, interpretation, design, development of alternative solutions. The basis of any ES is knowledge about a specific subject area, which is structured, encoded in a certain way, accompanied by a set of inference rules and ready for use. Distinctive features of ES: • The possibility of substantiating and explaining the obtained solution in a human-understandable manner, • Focus on the application in a specific subject area for which it was specially developed.
DSS are computer systems which purpose is to support the activities of decision-makers managing decisions, in particular by providing a comprehensive and objective analysis of the problem being solved. This type of systems arose as a result of the merge of management information systems (IS) and database management systems.
The difference between DSS and ES is that the key role is played by the decision maker, who bears full responsibility for the decisions made and, therefore, needs complete control over the process of decision making and decision implementation.
Thus, neither ES nor DSS are an adequate answer to the issue of automating the assessment of decisions for sustainable development management of complex technological and socio-economic facilities.
The disadvantages of ES and DSS are that the both types of systems do not have the necessary properties of adaptability and universality, since they are initially oriented toward solving problems of a specific subject area. The specified properties should be possessed by a new generation system -AXAS.
AXAS should be much more than ES and DSS in terms of functionality, adaptability, coverage of the processes of assessment, intelligence.
AXAS main goal is to support all stages of the assessment, regardless of: • Objectives, • Assessment programs and techniques, • Subject area, • Selected assessment criteria and scales, • Type of expert assessments and the method of obtaining evaluations.
The idea and organizational structure of AXAS are shown in Fig. 3. AXAS is a set of flexible customizable and arbitrarily connected software modules from which the system user will be able to assemble and configure a version of the system that fully meets the goals and objectives of a particular assessment.
The core of AXAS is an extensive set of models, methods and criteria used in data collection, analysis and processing for expert assessments (this is not just about an intelligent system, but about an ES development framework).
In addition to that, the practical feasibility of an AXAS design is largely determined by the degree of achievable formalization of the individual stages of the assessment, since it is hardly possible to automate what cannot be represented as a formalized model. Therefore, we further consider the main phases of assessment in regards with the possibility of their formalization and automation.

The phases of the assessment within an Automated eXpert Assessment System
Phase 1: Setting the goal of the assessment.
Participants and performers: assessment customer, working group (managers).
The phase is the foundation of the entire assessment and is carried out in the process of concluding a contract with the assessment customer -an individual or collective person asking for the help of third-party specialists.
The Customer and the assessment managers communicate, and administrative and financial issues are resolved.
Means ( This phase begins with the fact that the working group, including not only the managers, but also analysts and cognitive scientists, develops an assessment plan based on the purpose of the assessment, customer requirements, deadlines and available resources (financial, human, computational, etc.).
The working group determines: a list of intermediate tasks to complete (to achieve the global objective of the assessment), how the list items are interconnected, what resources are required, timelines, who is in charge for each task, etc. The assessment program should answer the following questions: • What shall be done? • When shall this be done? • Who shall do this?
The determination of the assessment methods is carried out either in parallel with the assessment program development, or after the program has already been prepared.
For each intermediate task, working group selects an adequate tool from the extensive arsenal of expert assessment methods, taking into account resource limitations. The assessment methods describe how exactly each item of the assessment program shall be implemented.
As one can see, the phase of program development and determination of the assessment methods is a creative process and involves a large number of people.
It is unlikely to exclude human in the implementation of this phase (in the near future). Nevertheless, one may formalize the process of developing the program and determining the assessment methods, by presenting it in the form of constructing an assessment algorithm from a certain set of basic blocks that are adaptable to a specific set of requirements. What prevents the development of IS to support the work of the working group at this phase? Firstly, it is the scale of the work to be done by the developers of such an IS: the range of methods and types of expert assessments is very wide and diverse, and creating a computer system reflecting this diversity requires considerable work. Secondly, the selected methods and algorithms of expert assessment have to be adapted to a specific task or subject area, and it is simply impossible to take into account all the nuances in advance.

Means (tasks) of automation (AXAS elements):
Visual editor allowing the working group to: • Build and edit the assessment program in the form of a certain visual structure (using the project management methodology, including Gantt charts, etc.), • Carry out meaningful filling of the program's blocks.

Phase 3: Expert group formation.
Participants and performers: working group, Customer (if needed).
The working group solves the tasks of examination aimed at the selection of experts. The selected experts will work to achieve the main goal of the assessment.
To accomplish the experts selection, at the previous phase should be defined: • The structure of the expert group-homogeneous or heterogeneous, localized in time and space or not, • The methods and the program of experts selection; • The motivation of experts for the conscientious performance of their duties.
In a homogeneous expert group, all members have the same right to vote (number of votes). In heterogeneous group, experts differ from each other in the weight of their judgments, depending on their specialization, competence, objectivity, certain personal qualities, etc.
Sometimes, when conducting assessments, one may use the so-called expert assessments of the second kind are used: the judgment of each expert corresponds to a weight coefficient reflecting the degree of confidence in this judgment. Usually, they call such weighting factors "experts' competence coefficients", and their determination is an important task, since the quality of the examination results directly depends on the competence of experts.
The structure of the expert group also depends on the communication methods of the working group and the experts (first of all, the method of interviewing them), as well as among the experts. In some cases, it is necessary to bring all experts together to conduct a personal debate, in others, interaction with experts can be carried out remotely. Some types of interviews are based on individual work with each expert.
To determine the expert group composition, we propose the "snowball" method, when the core of the expert group is determined by the working group (for example, as a result of analysis of scientific publications of potential experts), and then each expert names specialists who, in his opinion, could be experts in this case. The new experts nomination is repeated until the expert group is fully formed.
Obviously, the choice of the structure of the expert group is ambiguous, depends on various factors, and therefore is hardly formalized.
On the other hand, the task of assessing the competence of experts and selecting candidates that meet the requirements is entirely conducted by the AXAS, but two conditions must be met for this: • Firstly, such a system should contain models and methods for assessing various characteristics of experts: competence, objectivity, etc., • Secondly, there should be a database containing all the necessary information about experts, analyzing which, the system selects suitable candidates.

Means (tasks) of automation (AXAS elements):
Automatic multi-criteria selection of candidates meeting the specified quality criteria to form an expert group of a user-defined structure.

Phase 4: Retrieving expert information.
Participants and performers: cognitive scientists, working group.
The defining aspects of this phase are: • The form of expert information presentation; • The polling procedure.
The most convenient for formalization and processing are quantitative point estimates. Less convenient are interval estimates. This is due to the fact that the burden of formalizing the opinions rests with the experts themselves, albeit receiving assistance from information extraction specialists.
However, cognitologists together with analysts have to develop a system of criteria to be used by experts. This requires: • Compile a complete (but not redundant) list of criteria, • Determine the structure of relationships between the criteria, • Assess the relative importance of the criteria (usually for this one needs to involve experts), • Develop an adequate scale for each criterion and interpret its values.
The AXAS should support the development of a system of assessment criteria similar to that described at the phase of assessment program development (using the theory of project management methodology).
Various kinds of qualitative assessments (rankings, verbal estimates, logical judgments, etc.) are much more convenient and understandable for experts, however, formalizing them within the AXAS framework and processing accordingly is much more difficult.
The methods of interviewing experts range from extremely formal to completely free: from closed questionnaire methods to open debates between experts.
Therefore, human-machine methods and procedures for extracting expert knowledge, involving direct interaction between the expert and AXAS, cannot always be practically implemented, since this approach requires a high level of development and formalization of the survey procedure.
Means (tasks) of automation (AXAS elements): • A visual editor that provides the working group with tools for developing and further modifying the system of assessment criteria; • An interactive system that allows for an effective survey of experts; • A database for storing expert information.

Phase 5: Analysis and processing of expert information.
Participants and performers: analysts and expert group.
Analysts analyse the information received from experts and process it in accordance with the methods, models and criteria provided by the assessment program and methods.
The degree of the tasks formalization at this phase primarily depends on the type or format in which the information received from experts. Analysis and processing of expert assessments is the most elaborated and convenient for automation, since most of the methods that comprise it are of a mathematical or statistical nature.
The complexity, as already noted in the consideration of the second phase, lies in the scale of the work associated with the implementation in AXAS of all possible methods and models used when working with different types of expert assessments.
Means (tasks) of automation (AXAS elements): • A set of computer tools for the analysis and processing of expert assessments; • A database storing the results of the examination. Phase 6: Summarizing the results of the assessment.
Participants and performers: managers, analysts. Based on the results of the previous phase, managers and analysts issue the necessary reports and transfer it to the Customer in the form initially established at the first phase.
In this case, the AXAS must generate flexible reports on the results of the assessment (using technologies such as OLAP).

Means (tasks) of automation (AXAS elements):
Digital reporting system with the possibility of flexible data visualization (such as OLAP).

Approbation
The experimental version of AXAS was fragmentarily used for: • Development of urban infrastructure development strategies, • Ergonomic assessment of complex systems including educational systems, • A number of other complex objects.
The essence of determining priority areas for the sustainable development of the city is shown in Fig. 4.
Obviously, the budget allocation task is of interest to a huge number of participants in the budget process. The use of AXAS technology elements allowed for one of the regional centres of Ukraine to reduce social tension in both the city Council and the city as a whole, and to distribute budget money in rational way.
The idea of using AXAS for e-learning modules certification in a university educational environment [28][29] is shown in Fig. 5.
The use of AXAS technology made it possible, according to [30], to increase the design efficiency of the certified digital training materials database: • In comparison to IS technology the efficiency raised 4.73 times, • In comparison to non-automated technology -17.5 times.
As a result, the quality of the educational process and the attractiveness of new forms of e-learning have significantly improved.

Conclusions
Problems of sustainable management of complex socioeconomic and technological systems require high-quality expert assessment of possible development scenarios.
There is an opinion that it is possible to automate the expert sphere of decision-making by replacing experts with databases of expert knowledge.
We showed that in many cases this is not entirely true, since many assessments are aimed at solving poorly studied or even unique problems, which only after they are studied and structured, can be solved by other more rigorous methods.
An effective solution to the issues of expert assessment is possible with the new class of systems proposed in this work -Automated eXpert Assessment Systems, which differ significantly from existing expert systems and decision support systems oriented to a narrow class of systems.
A justified degree of automation of expert assessment, as well as the set of methods and software tools can be recommended for designing effective automated systems for expert assessment.
The scientific novelty of the results lies in the fact that, in contrast to local models for obtaining and processing expert assessments focused on a usually narrow problem area, the developed technology allows to integrate existing methods and tools into one system. The system is flexibly configured for a specific problem situation and takes into account the capabilities of both human and computer decision support systems.
This creates the prerequisites for creating a fundamentally new generation of expert assessment systems based on hybrid human-machine procedures for making collective decisions (hybrid intelligence systems).