Dual approach to the assessment of the project management system performance and the results of its application in the ITER Project

. The article considers the current problem of low efficiency of tools and methods used to assess the project management systems of complex engineering facilities. The problem is important today because engineering companies strive to increase their rating in a highly competitive market, and this issue is of a special concern in the international market of goods, works and services. The article presents the methodology for the assessment of the project management system, based on the simultaneous cross-section use of approaches to the assessment of project management and personnel management processes (the so-called "processes – personnel" pair) with the use of questionnaire-dialog forms of data collection. The essence of the study was to search for patterns of cross-assessment results, their comparison and determination of algorithms of mutual influence on each other, and what is necessary for a more accurate assessment of the management system. The scientific novelty of the research is the development of a set of interrelated indicators of the management system behavior in a company. The presented method was tested by the team of authors of this article during the management assessment of the international project on the construction of a thermonuclear reactor (ITER, France) in 2020. The results of the study revealed the comparability of the assessment results, confirmed the impact of the assessment results on each other and determined the possibility of predicting the assessment results per individual indicators, depending on the state of the management system in the “process-personnel” pair. This method can be used to assess the management systems of any project-oriented companies implementing capital construction projects. In the future, it can become the basis for creating a specialized standard for assessing the construction project management systems.


Introduction
In recent years, the activity of Russian business in the world market of goods, works and services has increased to a large extent. Russian companies are trying to significantly E3S Web of Conferences 258, 12017 (2021) UESF-2021 https://doi.org/10.1051/e3sconf/202125812017 accelerate the process of adapting their business to the demands of foreign markets, in particular consumers, competitors, state and supervisory bodies. The growth of Russian companies is fueled by a significant growth in the implementation of large construction, infrastructure and other types of projects. As a rule, the implementation of such projects involves the establishment of individual enterprises, branches and representative offices, which, to a certain extent, are standalone and have their own management system.
In order to assure successful development in the world's markets, under conditions of open competition, companies must be ready to face challenges that are alien to the Russian mentality -strict regulation and standardization of business requirements, as well as their mandatory execution that is prescribed in the contracts and tied to fines. The combination of the above factors is fully expressed by the general trend of the global market -special attention must be given to the management system of an organization, especially if it is implementing a large scale project. One of the most important aspects of improving the project management system is the assessment of its effectiveness, allowing for monitoring and timely adjustments in case of deviations [1].
Assessment is an applied science, an everyday tool for running the modern business. The methods and approaches of assessment are constantly changing and improving.
The emergence of new technological companies, the development of new markets, has led to the emergence of new, "creative" assessment methodologies, which place the utmost importance on the factors of risk and uncertainty [2].
Assessment of the effectiveness of the management system involves a number of complex tasks. The main task of any management system is to ensure active influence on the managed object in order to improve its performance. The various properties and parameters of a management system's subject and object are often inconsistent with each other, are in dialectical contradiction, and therefore there is a problem of determining a generalized indicator that would be an index of the management effectiveness. However, the characteristics of the management systems and the conditions under which they are operating are so diverse that it is advisable to apply different efficiency criteria to the different systems. Indicators can have quantitative and qualitative content, which, in turn, has subject-production, economic, statistical and mathematical interpretations [3].
In the economic literature relating to management systems [4,5,6,7,8,9], it is recommended to apply an integrated approach for assessing the effectiveness of management tools. The essence is the need to identify a set of key indicators for individual management functions, such as planning, organization, motivation, control, etc.
While studying the above methods and their application experience, it is worth noting that the results obtained with the use of these methods of the effectiveness evaluation consider this very effectiveness locally, "on the one hand," and moreover, the results of the methods have no logical connection, have different nature and do not give an unambiguous assessment of the system as a whole.
Moreover, there is no clear leader or outsider among these methods in terms of their weight in the evaluation of a system . A detailed review of the above methods shows their bias either toward the project management system or toward business management.
It is obvious that at the moment, especially for the companies operating in the global market, it is necessary to apply some universal methodology of the management system assessment, suitable for both the project-oriented and functionally oriented companies. Such methodology will allow companies to conduct assessment of the management system from different perspectives. And what is critical, the indicators for evaluating various aspects of management should be mapped and allow one aspect of a management system to be predicted from the other.
Any management in global terms is based on two things: management technology, in other words the sequence of actions, and the actual management subject or employee who has the knowledge, skills, experience and many other characteristics and implements this very management technology. The experience of management systems assessment shows that, as a rule, most of the techniques are aimed at assessing exactly the management technology, study of processes of organizations and improvements in this area. Such an approach is undeniably successful, but it may be difficult to dispute that even the most advanced technology, including management technology, handled by unpracticed hands, is ineffective and even capable of doing harm. In order to exclude the above scenario, the assessment of the project management system, business management system, and the management system in general, especially in large organizations, must be carried out in a comprehensive manner.

Theoretical basis of methodology
In developing the methodology of comprehensive assessment, the hypothesis was based on the fact that a set of assessed factors (indicators) describing the status of project management processes and HR processes form an interconnected system.
In case of analytical or mathematical determination of this system in the future, it is possible to set the highly probable predictions about how, with a certain change in one element of the system (the assessment factor) the other factors and the system as a whole will behave.
To test the hypothesis, the technique was thoroughly explored and first tested during the assessment of the management system of the international project for the construction of ITER thermonuclear reactors.
The comprehensive approach to project performance assessment is based on the principles of PMI PMBOK International Project Management standards [10], IPMA ICB [11], P2M Innovation Project and Program Management Guidelines [12], and HRCI HR practices. The uniqueness of the proposed methodology is that it is based on a dual approach to the analysis of all aspects of the project-oriented enterprise: all the aspects are simultaneously considered through two types of values -value for business processes, value for the people -the participants and stakeholders of the project. success of each particular project and the balance of the organization's activities as a whole directly depends on the quality of the functional link "PROCESSES -PEOPLE".
In assessing the project effectiveness according to the proposed methodology, each stage of the capital construction project implementation is analyzed for the presence of management processes in all areas of management, then each process is examined according to the management cycle (initiation, planning, organization and control, analysis and regulation, completion), as well as in terms of values, products and people. The areas of management to be analyzed are scope, time, cost, quality, people, risk, requirements, changes, non-conformities, safety, and communication.

Stages of methodology application
Practical application of the developed methodology on the pilot mega-project ITER was implemented in three stages:

Stage 1
During this stage, two detailed lists of factors to be assessed were developed, separately to assess the project management processes and separately to assess the human resources management system. For the purity of the experiment, the work was conducted independently by two groups of experts. In the course of this stage, there were also prepared tools and techniques through which the information on the assessed factors was further collected.

Stage 2
In the course of this stage, information on all the assessed factors was collected using the data acquisition tools prepared through the first stage. Such tools include questionnaires, interviewing, surveys and group work. Several hundred regulatory documents covering various areas of processes and personnel management were examined and analyzed as part of the management assessment. The use of various information collection tools allowed the authors of the methodology to cover the largest sample of interviewees, to verify and detail the results obtained in the survey forms by conducting face-to-face interviews, and to interpret the data obtained from various sources as correctly as possible putting the data into concrete proposals for improving the quality of processes and management.
During the survey, to get the maximum objectivity of the data, 888 questions were prepared (162 for processes and 726 for personnel). The said questions were divided among 3 categories of personnel (TOP management, engineering staff, workers and specialists), according to the specifics of their activities (subject areas). As a result, each respondent received a unique set of questions, which was most relevant with their level of position and the scope of the tasks to be solved. The number of questions for each category varies and ranges from 130 to 500 questions. Open-end to close-end questions ratio is 1:4. The methodology implies the study coverage of at least 10% of the personnel, while maintaining real proportions in the number of each of the categories of employees at the enterprise. In the assessment of large-scale international and cross-cultural projects, it is expedient to take into account and ensure the proportional distribution of respondents per their working experience in the company or at the project, per the level of integration with the company (work on permanent or temporary contract, part-time employment contracts), as well as the pace of career growth of an employee, his national and gender affiliation, family situation, etc. To ensure that the findings were not biased, the study was conducted simultaneously by two independent teams (processes and personnel) who were not allowed to discuss the preliminary findings directly during interviews and in-group analysis of the findings.

Stage 3
During this stage, the results of the management system condition were summarized per two sections: processes and personnel. At this, 55 key process recommendations and 61 personnel recommendations were generated from the several hundred identified "growth areas". The cross-examination of problem areas based on a dual approach has resulted in 13 key areas of development of the management system as a whole.
In the course of research based on the dual methodology, the following general aspects of management are analyzed: organizational structure, presence of processes and their formalization, roles and responsibilities, tools.

Values for the processes
In the scope of the first area -values for the processes, the below topics are addressed for each management area: What is the level of implementation of these tools?

Values for people
The second area covers the values for personnel, HR strategy is assessed with the use of HR audit tools per seven key units. The review of the HR management function in view of the main stages of the HR cycle allows to increase the transparency of the methodology and the results obtained in its process part. The personnel is the main resource of any project, and the assessment of the effectiveness of work with it at different stages of project implementation is an objective and indicative identifier of the quality of the current state of the project, possible risk events and the formation of measures for their prevention.

Analysis of Business
The unit consists of the following subsections: Compliance with legislation (including countries legislation).

Audit of HR metrics
This will answer the questions such as whether HR processes are being effectively built, whether managers are paying sufficient attention to the key HR performance indicators, and whether they can respond to changes in key performance indicators in a timely manner and take corrective action? 2.5. Audit of corporate culture The company's mission, prospects and values, their conformity to the business goals of the project, their acceptance and use by employees in everyday work activities are analyzed. It is important to assess not only the missions and values that are being declared, but also how they are being translated from senior management to each employee.

Audit of Employer Value Proposition Audit (EVP)
To complete the analysis, the focus was on the following key subtopics:

Results
Based on the results of practical application of the dual approach at the ITER pilot megaproject, it is possible to confidently state that the hypothesis, which is the basis of the methodology, is confirmed. A set of assessed factors obtained from the project management assessment presented comparable results, which equally characterize the condition of individual elements of the management system. The use of a dual approach to management assessment allowed our team to systematically, objectively and comprehensively examine the unique features of a particular project management system and develop the most accurate and effective proposals for its improvement. The proposals were carefully considered and accepted by the Council of the ITER Project for implementation in the near term.

Conclusions
In the course of the study, the results that coincided under the independent application of the two assessment approaches were analyzed in detail. The main emphasis was on a detailed review of the assessed factors, in order to obtain the final result. This review resulted in the identification of sets of assessed factors that led to the same assessment results. It was determined that the results had been verified and comprehensively validated. Based on the obtained results, the impact analysis of factors was performed, in particular the possible behavior of the system of factors while their changing, and the influence of the change of factors on the final result. The basic interface algorithms and the impact of assessment factors on each other were formulated. An algorithm for predicting the condition of related assessment factors was also developed.
In the course of the methodology application, sufficient information was obtained to confirm the hypothesis, but for the mathematical determination of the system of factors and algorithms of the system behavior it is necessary to further apply the methodology and develop the database of the obtained results (confirmation and refinement of the data obtained during the assessment of the ITER project). At the same time, based on the results obtained from the initial validation of the dual assessment approach, the team of authors succeeded to form a detailed and synchronized in the field of duality, practical and coordinated pool of proposals for optimizing the process and management component.
In the course of the methodology validation, one can confidently conclude that the application of a complex or "dual" methodology of assessing the project management systems and organization management systems will greatly benefit the client due to a much deeper assessment mechanism.
Existing assessment methodologies, as discussed above, tend to focus on one-way assessment of a system, in particular on assessment of management processes. These methodologies undoubtedly produce good static results and evaluate the management system from a processes perspective. The same can be said of the management system assessment while applying the methods of personnel evaluation only. The results of the application of the "dual" methodology showed that, first, its application allows to reveal cause-and-effect relations of the assessment results in the "process-personnel" pair, second, the established dependencies of assessment indicators allow to verify the conclusions of the assessment, third, they allow to answer the questions "What to do? How to do it? "Who should do it?" The answer to all of these questions is the real practical value obtained from the assessment.
In the framework of further development of the developed and tested dual approach to the assessment of the efficiency of the activity of project-oriented companies, it is planned to conduct a series of assessments at enterprises with different types of activities, in different branches of industry and on projects of different scale and complexity, after which it is planned to formalize the method and legalize this method as a standard.