Development of 3D workspace and methodology for assessing operator’s functional state during the work in virtual environment

The article presents the current results of a research project aimed at developing software and hardware solutions that optimize an operation of dispatch services based on immersive technologies. The results of the virtual operator’s office creation for the subsequent program-protocol communication of the displayed information with the data sources of the dispatching object are presented. The virtual workspace is implemented in the form of large-scale objects, textures, lighting, etc. “animated” in virtual reality glasses. The results of the initial approbation of the methodology for assessing the psychophysiological state of operators during virtualization of the working information space are also presented in framework of multistage studies of effects of virtual reality on the user.


Introduction
Trends of science and technology are aimed at reducing the use of human labor in face space during the extraction of minerals. Mining is an industry in which industrial automation is widely used. The increase in the volume of automated and remotely performed operations will reduce the risk of injury and death of personnel in the event of emergencies. Also, to improve the labor safety of mining workers, to positively influence the situation with the development of chronic diseases in them.
Production operator's workplace virtualization is becoming relevant due to the spread in the world of remote operation for many services. Also, the Russian development strategy of the mining industry today is aimed at digitalization and automation of both main and auxiliary production processes. Operator workplace virtualization is effective and promising for the tasks of training, dispatching, streaming support (supervising) of the main technological, assembly, commissioning, and repair work.
The relevance of research aimed at studying the impact of virtual reality (VR) on the psychophysiological indicators of the state of operator-users is increasing. This will ensure the future safety of personnel working with immersive reality displays.
Virtual environment is bright, interactive, and facilitates concentration of attention. In VR, it is unlikely that attention will be diverted to stimuli from the real environment [1]. This aspect is of great importance in the creation of VR for technological process control, but it is not quintessential. From the point of view of a number of scientists, finding a person in a virtual environment under certain conditions not only does not impair a person's ability to function effectively, but, on the contrary, can activate the capabilities of his body [2]. Scientists working with large data sets publish the results of studies comparing human performance when processing such data sets in virtual reality and ordinary two-dimensional space [3,4]. The results of these studies show that human performance when working with data in virtual reality is not worse, and often even better, than when working with a familiar monitor. Some scientists opine that gamification of many routine tasks can positively affect productivity due to greater involvement in the process [5]. In addition, immersive reality increases the convenience of working with already processed data prepared for work in VR (graphs, diagrams, voluminous tables etc.) [6], especially in terms of comparing graphical information in front of our usual monitor. All the listed advantages, combined with the ability to change the very way of handling data that the virtual environment gives us, pushed the team of authors to develop a full-fledged virtual environment, in which an operator will be able to fully solve tasks assigned to him.
Perception largely depends on picture quality, matrix frequency and resolution, operator's head tracking system accuracy and speed. In addition, VR technologies are constantly evolving in terms of content and quality of content, devices and software are being improved, the performance of graphic data processing is increasing, and the response time to feedback is reduced.

Research objective
These studies suggest a method for increasing an information capacity of interfaces of industrial centres and equipment management systems at enterprises, which differs from those used. It is based on three-dimensional graphics and the means of its volumetric display and is divided into two subtasks.
The first is a software and hardware, interface research subtask. The task of developing VR environment consisted in analysis and selection of the display hardware, software graphical development environment, and subsequent creation of the test virtual dispatcheroperator workspace, which allows the most efficient implementation of interface interaction with data.
The second subtask of research is psychophysiological -the development and testing of methods for assessing the cognitive load during immersion and work in VR on the psychophysiological state of operators.

Development of a test virtual space
The results of the development environments analysis, hardware for displaying immersive environments, their functionality, were published by the authors earlier [7]. At the current stage of the project, an analytical review of world practice on the theoretical aspects of designing a virtual environment (operator position) was carried out, taking into account the requirements of ergonomics, including approaches to transferring objects into virtual space.
Approaches to the design and placement of objects from the real world in a threedimensional virtual environment have been scientifically substantiated. The requirements of mental and physiological ergonomics are taking into account. As a result, a specialized software and hardware complex with an integrated test virtual space was developed for the future study of the parameters of the virtual cockpit elements. The complex includes a personal computer with the Unity virtual environment development software, Oculus store, Oculus Link software, Oculus Quest and Quest 2 VR glasses.
The test virtual space was created using graphical design techniques for a virtual environment using UNITY software version 2019.4.1.f1. It is created in the form of a threedimensional room equivalent in size to a room with dimensions L×W×H 15×15×4 meters. The room is assembled from the primitive "cube", the floor is made from the primitive "plane". When an attempt was made to use a "plane" primitive for the ceiling, which is more convenient in terms of three-dimensional design, the ceiling was "translucent" with an external built-in light source. This entailed the formation of parasitic shadows from objects inside the room. The use of counter-light exposure techniques with additional light sources solved the problem, but partially. Applying the "cube" primitive for the ceiling cut off the external light flux from the virtual light source and made it possible to create diffuse local lighting inside the test space.
On the recommendation of a neuropsychologist, textures for the walls were chosen in a matte turquoise shade. To create a more comfortable internal environment, the usual (expected) for the brain colour colouring of the ceiling and floor was implemented, a texture imitating a laminate covering was applied to the floor, and the ceiling was painted in light blue, close to white. The white ceiling does not look natural when the scene is rendered using spotlights. To create a room with soft diffused light that does not overexpose the objects inside the room, but sufficient to "illuminate" the virtual room, one virtual source of directional lighting was used, and 73 virtual point light sources of different intensities were distributed.
After the creation of the virtual room, the concept of placing information panels was developed. For this, a survey was conducted of the dispatch and operator staff of the largest mining and industrial enterprises of Kemerovo region -Kuzbass. According to the survey, it is generalized that dispatcher use large-format interactive information panels for monitoring and control, use spreadsheets for reporting, text editors, instant messengers and e-mail to communicate with remote production sites. Thus, all these element's skin will have to be in the virtual test space. In addition, an important aspect for ensuring mental ergonomics is the creation of a familiar environment for perception by the brain -with the presence of reference that provide an unambiguous position of the body in space on the correct scale.
The current version of the test virtual workspace has three large and three small information panels attached to the wall opposite to the operator at an equivalent distance of 8.4 meters from the operator's eyes. It is possible to change the content, both by the size of the dashboards and by their number (Fig. 1).
To work with text documents and spreadsheets, two separate information panels are provided, located closer to the operator with an angle of inclination of 30 degrees to the horizontal plane, if necessary, it is possible to change the size of the panels and their number. Directly in front of the operator there is a control panel consisting of a keyboard, an information panel of instant messengers on the left side, a telephone module and buttons for activating instant messengers and additional applications on its right side. Also, in the upper left part there is an imitation of a push-button control panel (Fig. 2).
The imitation of the control panel is made in such a way that it is convenient to interact with it using the standard Oculus Quest controllers, and then control your hands using the Quest Hand-tracking technology.
In the test virtual space, two modes of interaction between the operator and the virtual test space are implemented: 1. "Review mode" -the operator can inspect the room by turning and tilting his head, the direction of his gaze is controlled using the internal sensors of the virtual reality glasses.

Assessment of the operator's state during virtualization of the workspace
A separate block of research was the subtask of developing and testing a methodology for assessing the psychophysiological state of an operator during virtualization of the working information space. The situation is complicated by the fact that studies of the influence of VR on the human psyche are currently mainly studying the influence of computer games, where the virtual environment is highly dynamic. The influence of VR on a person under static conditions has practically not been studied.
To solve this problem, an experimental methodology for the work of subjects in VR glasses was developed, including an assessment of their psychophysiological state before, during the work and after its completion (Fig. 3). The approbation of the methodology was carried out on the basis of the Federal State Budgetary Scientific Institution NII KPSSZ. In particular, an experiment was carried out in which 10 dispatchers of right-handed men aged 25 to 45 years without experience in using VR tools took part. Their psychophysiological state was assessed before, during and after the completion of work in VR. A number of studies were carried out to evaluate the subjects. The first stage -examination by a neurologist with conducting vestibular tests, then an initial electroencephalographic (EEG) study was performed. The next stage was the initial assessment of psychoemotional and cognitive status, including the assessment of complex visual-motor response (reaction time, error rate), brain performance, assessment of attention functions, short-term memory and spatial perception. After that, the subjects put on VR glasses and for three 20-minute sessions performed tasks for verbal counting, reading and analyzing text, analyzing a voice task with composing and sending an answer as a cognitive load. The type of tasks is selected based on the specifics of the dispatching staff work, and the method and order of stimulus presentation is due to the need to ensure multitasking work with a high degree of cognitive shifting. To perform the tasks, we used the preinstalled (manual) interface software of the used VR headset Oculus Quest and an Internet browser. Third-party applications used free software for viewing PDF files, VLC media player for playing audio tasks, and Polaris Office for reading and analyzing some text documents. Intermediate EEG recording was performed without taking off the VR glasses. After the VR sessions, a neurologist was examined, an output EEG, and a reassessment of the psychophysiological state.
The results of the task success within the framework of the cognitive load are presented in Table 1, where MA -mental arithmetic, IS -information search, ST -search in the text, AT -audio task.
According to results of methodology approbation on a sample of ten subjects after cognitive load in VR conditions using the methods of psychometric testing and EEG registration, both positive and negative changes in psychophysiological status were revealed [8]. The indicator of spatial perception after being in a virtual environment improves, and the deterioration of psychometric indicators is observed only in the most difficult tasks to perform (brain performance and attention distribution). The numerical indicators of the research results are not given by the authors due to the small sample and the importance of correct results interpretation. These results should not be considered definitive data on the impact of the virtual environment on the operator's state. The results should be considered as confirmation of the developed research paradigm quality. According to the results obtained, the research methodology with the definition of the psychophysiological state of a person before and after cognitive load showed acceptable subjective difficulty and tolerance of the virtual environment. Solving mental arithmetic, information search, search in the text, listening and analyzing an audio task in VR glasses are considered as a cognitive load. When considering the changes in EEG parameters before and after the experiment on virtualization of the working information space, it was found that the EEG changes observed during the experiment in subjects after immersion in VR correspond to the activation patterns of the cerebral cortex associated with the performance of cognitive load and the resulting fatigue. It will be possible to determine the degree of influence of the virtual environment and cognitive load as sources of fatigue after creating conditions for comparing the activities of operators in conditions of VR and without it. For this, further application of the developed methodology with an assessment of the psychophysiological state of operators before, during work and after its completion implies the formation of a control group, the working conditions of which will be identical, with the exception of immersion in VR.

Conclusion
During the implementation, the fundamental and applied aspects of three-dimensional virtualization of the working environment were studied. The principles of transferring control objects from a real environment to a three-dimensional interface are substantiated taking into account the mental ergonomics of the operator, as well as the effectiveness of his interaction with virtual objects. Based on these principles, a virtual test space has been developed to study the parameters of the virtual cockpit elements. In the future, this will help improve the management of industrial technological processes, including the processes of coal mining, and ensure the transition to advanced digital, intelligent production technologies. The results of testing the technique will subsequently help determine the degree of influence of the virtual environment on the psychophysiological state of the operator, as well as formulate recommendations for the mental ergonomics of the virtual workspace.