Models and algorithms for managing the emotional state of customers in commercial banks using deep convolutional neural networks

. In this research, a model for managing the emotional state of customers in a commercial bank has been developed using a deep convolutional neural network (DCNN) and algorithms for distributing conflicting customers along the routes to the certain operator, depending on this emotional state. In order to route a customer to the certain operator, it was necessary to develop a mathematical model of emotional target routing based on the Newton interpolation polynomial. The developed model has four classes [angry, happy, neutral, and sad], trained and tested on the well-known FER2013 dataset using machine learning and computer vision. Finally, the model validation accuracy of 70.35% has been achieved.


Introduction
Emotions occupy an important place in a person's life [1], and their influence on society members manifests itself in different ways, both in communication between people and during decision-making processes.Emotions can be expressed in various ways: facial expression, posture, speech reactions, and others.Nevertheless, it is the human face that can reflect emotions more truly impressively .Facial expressions consist of one or more movements or positions of muscles on the face [2].These muscles in a certain visual structure on the face are associated with a specific emotion.A person recognizes an emotion by determining the structure of a facial expression.In order for machines to learn how to recognize and classify emotions, they need to imitate the structure of neurons in the human brain in a similar way.Researchers in [3,4,5] note that activity in banks is characterized by a high stress and extremity, greater responsibility for decision-making, and the severity of consequences in case of mistakes.In this regard, it should be noted that the recognition of the emotional state of the customer should be used in the access control system in banks.Moreover, in public service systems, there is an increase in the number of emotional and mental abnormalities among customers [6] who need to identify the customer's condition in the bank as soon as possible so that automated decisions can be made about the distribution of these customers to the certain operator.Only using modern results of scientific and technological progress can help to solve this problem quickly.Most of the available resources are not released to the public, which complicates the research process as a whole.Accordingly, there is a need to develop programs and software complexes that automatically Corresponding author: guedes.soma@mail.rucontrol the emotional state of customers and allow them to make decisions quickly without any human help.In this work, one of the most important excited emotions is considered -"anger".By its semantic structure, the word "anger" is understood as "indignation, a feeling of strong indignation".This emotion can lead to serious potential conflicts in the bank, to prevent which we have proposed the development of a system with the following hypotheses: 1.
Application of machine learning method and computer vision to develop a DCNN model.

2.
Using a balanced dataset with a large number of images.

3.
Adjustment of network parameters during model training in order to reduce information loss and overfitting.4.
Using the evaluate method, classification_report and confusion matrix to check the model prediction results after training.
The novelty of the research lies in the fact that we have developed a system for recognizing the most important emotions from the view point of public safety based on a visual channel for tracking the emotional state of customers and a mathematical model based on the Newton interpolation polynomial for automatic distribution of customers along the route to the certain operator in a commercial bank.The relevance of this study was due to the need of improvement the effectiveness of managing the emotional state of customers and preventing conflicts in a commercial bank.

Methodology
To develop the emotional model in this study, we used the Python programming language as well as the machine learning method and computer vision.As shown in Fig. 1, at the first stage a dataset (FER-2013) [7] from our computer to the Kaggle platform is uploaded, and then all the code in Kaggle is written.At the second stage we prepare a dataset for training and make it class-balanced; at the third stage we develop our DCNN architecture, extract characteristic features from images in the Dataset, adjusting weights to minimize errors during the training.At the fourth stage we train our model and test it on the data that the model has not seen before; at the last stage (the fifth) the accuracy of the model's predictions is checked, using the well-known evaluate method, classification report and confusion matrix.We make up a function for calculating the waiting time of the customer depending on their emotional state.This function is necessary because the initial waiting time is three minutes (i.e., 180 seconds).The waiting time should vary depending on the current average level of anger and sadness of an individual customer.The average value of the anger and sadness level is the sum of both levels in all frames for a certain time period (in our case, the program recognizes twenty frames per second, and six seconds are set to determine the average level).To create such a function, experimental interpolated nodes are determined.
The Newton interpolation polynomial of n degree is written as: where  separated differences -are: (2) In this work, in order to distribute the customer along routes to a certain operator, an interval of values of the anger and sadness level ([from 0% to 100%]) was determined and divided into four parts.The resulting mathematical model of emotional target routing based on the Newton interpolation polynomial.This function determines how many units of time passes in a period of one second, depending on the emotional state of the customer who came.The areas of definition of this function are percentages that determine the anger and sadness level of the customer.For example, if this level is, according to the definition of the visual channel, in the interval [0.2,0.6[, it means that the waiting time (distance from the customer to the operator) will be longer (that is, increases to 180s) than the customer whose anger and sadness level is determined in the interval [0.6,0.9[.For customers in the interval [0,0.2[ the waiting time increases, since being in this emotional state, such customers will undergo services in the order of the usual queue.The system does not react to customers with a happy and neutral emotional state, thus they do not present any conflict, but it will be necessary to demonstrate the impact of the bank operator on the emotional state of the customer after the service.
Fig. 3 illustrates the developed emotion recognition algorithm which can work in real time in a commercial bank using a visual channel.The development of this algorithm allows identifying the emotional state and tracking the customer within the bank.When a customer enters the bank, our system records the values of the initial emotional state of the customer for a short period of time (six seconds), saves the customer's face in the database and provides him\her with an identifier (ID) so that the algorithm can follow him\her in the bank.Our system takes a frame as an input, and then processes the frame to detect the faces that appear in it.After that, it applies face recognition to match the faces in the current frame with previously saved faces in order to update their emotional state values based on the new one received.

Emotional state management model
The management model is understood as a set of theoretical and practical ideas about how the management system should look like, how it should work, how it affects the managed object and how it adapts to the changes in the external environment so that the managed object can achieve its goals, develop sustainably, and ensure its viability [8].Fig. 4 shows the model we have developed for managing the emotional state of a customer in order to prevent conflicts in a commercial bank by distributing customers along routes depending on the emotional state of the customer.The model consists of seven parts, such as: 1.The customer's digital profile model is a model containing all the information about the customer, it consists of customer sources and customer characteristics .In the customer sources we find an image from a video camera and a terminal for issuing coupons.The characteristic depends on the source.
2. The digital operator profile model is a model containing all information about the operator, it consists of sources and characteristics of operators.The bank operators are required to approve such personal qualities as the level of training (trainee, experienced operator, psychologist operator), high communicative competence, competence on credit and salary, behavioral flexibility, and conflict-free communication skills.
3. Customer external characteristics represent a number of tasks to determine the characteristics of the customer, the main ones are: -Assign identifier -is used to track the customer in the bank, we assign it to the customer when he\she enters the bank -The initial emotional state of the customer is analyzed when the customer enters the bank before servicing, in order to obtain information about the first initial emotional state of the customer -The conflict state is analyzed in all departments where cameras with our system are installed 4. Information about the senior operator or psychologist is the current information of a particular operator provided by the system about employment and window number in order to create a short-term queue if the operator is busy.In this case, the service of each client does not exceed ten minutes.
5. The database was used to store personal data about the customer obtained using a visual channel, such as face, initial emotional state, report on the impact on the emotional state of the customer before and after the service.
6. Emotion model is a model developed using the FER-2013 dataset with four emotions (angry, happy, neutral, and sad).
7. Routing is the direction of a customer in a state of anger or sadness to a certain operator.Routing is developed on the basis of the Newton interpolation polynomial.
Since the accuracy of facial expression recognition algorithms partly depends on the dataset used for training and testing, it becomes correct to compare the use of different methods on the same dataset.Thus, in order to compare the methods of facial expression recognition on still images, the results were examined by the FER-2013 dataset.The authors in [9] used transfer learning to train two networks to classify facial expressions: ResNet50 and MobileNet, and proposed a new classification architecture based on a convolution neural network.Compared to ResNet50 and MobileNet, the proposed classifier received the best result in the validation accuracy (60.13%).In [10], the development of an intelligent system that recognizes and classifies emotions by facial expressions was proposed.The model of a multilayer convolutional neural network was used.In addition, another learning method using transfer learning technology on a pre-trained ResNet50 model was investigated.This model was trained on the FER-2013.A basic live video application was developed to demonstrate the effectiveness of using a trained model that could track and record facial emotions in real time during a live video broadcast and then summarize the overall responses at the end of the broadcast.After training and testing, the validation accuracy of 61.47% was achieved.
According to Fig. 5, we can observe a good result, since after training, the value of the loss function of the developed classifier decreases by 0.5136, and the validation accuracy increases to 70.35%.Table 1 shows a comparison of two published scientific results on the classification of emotions on the FER-2013 dataset, and our classifier appears to have the best validation accuracy.

Truth
As shown in Table 2, we note that the classifier we developed performed well with an unweighted average recall (UAR), according to the resulting confusion matrices.The confusion matrix allows us to understand the performance of a classifier according to the table.In the confusion matrix, rows correspond to actual labels, and columns correspond to predicted labels.The correctly classified samples are shown on the diagonal (bold numbers), while the standard text numbers represent incorrectly classified samples.

Conclusion
In this study, a mathematical model based on the interpolation of the Newton polynomial for a commercial bank was created.A DCNN-based model has been developed using the FER-2013 dataset containing four classes with facial expressions ("angry", "happy", "sad", and "neutral"); the validation accuracy of 70.35% was obtained.In order to prevent conflicts in commercial banks, a system for managing the emotional state of customers, directing them to a certain operator in case of emotion "anger" or "sadness", has been developed.In the future, it is planned to improve our models in order to achieve the validation accuracy above 90%, as well as expand the work not only to be limited to banks, but also to other areas such as airports, supermarkets, and call centers.

Fig. 1 .
Fig. 1.The process of developing of an emotion recognition model.

Fig. 5 .
Fig.5.Training and testing of loss and accuracy: developed emotion classifier (a, b).

Table 1 .
Results of testing the classification model.

Table 2 .
Confusion matrix of the developed model.