Development of a somatosensory controller for positioning a manipulator device for picking apples

. The aim of the study is to develop a somatosensory controller for remote control of a manipulator for picking apples using resistive sensors, a gyroscope and an accelerometer. A computer simulation of the process of picking fruits from a tree crown was carried out in the Gazebo simulator using a somatosensory controller. An environment has been created, including a model of a manipulator for picking apple fruits and a fruit tree, and the properties of the environment have been set. To conduct a comparative experiment to evaluate the results of computer simulation of the process of picking apple fruits using a self-sensor controller and a manufactured sample of a manipulator, artificial models of trees with an identical crown and the location of fruits in the crown at a distance of no more than 1,5 meters from the zero point of the manipulator were used. According to the results of the research, it was found that the average time for picking one apple fruit by a manufactured manipulator in an automated mode is 12,9 seconds, the minimum time was 9,4 seconds, the maximum 15,9 seconds. Gazebo environment using a self-touch controller was 11,7 seconds. The minimum time was 8 seconds, the maximum 14,9 seconds. The development of digital intelligent systems, telematic services will ensure the widespread use of the proposed method for monitoring and managing the group work of robotic tools in the cultivation of fruit and berry crops.


Introduction
The rapid development of robotic technologies has made it possible to create highprecision robotic systems to perform complex tasks in real-time remote control mode. Such tasks include capturing and manipulating objects. The use of machine learning techniques, object recognition, and motion planning algorithms allow robotic devices to perform more human-like tasks with greater autonomy, allowing robots to operate in unknown environments without human supervision, assistance, and intervention [1,2]. An analysis of research by well-known scientists has shown that non-repetitive manipulation tasks, limited in time, requiring high-level decision making, due to unpredictability in a non-deterministic environment, require human participation [3]. Fully autonomous robots, which perform the tasks specified by the program without human intervention, have their advantages, but in some cases, remote control of the robot can be more efficient and convenient. For example, if a robot needs to perform a complex and unpredictable task that requires decision-making based on external conditions and the environment, then remote control of the robot can allow a human user to quickly and flexibly respond to changes and make optimal management decisions based on their experience [4].
However, this control strategy may not be accurate enough as it is based on open loop control where the only feedback the human operator receives is visual feedback. Humans are able to complete tasks in minimal time due to a combination of multiple sensory feedbacks, including force feedback. Thus, the lack of force perception in telecontrol systems leads to increased task time due to reduced feedback that human users can use as clues. To increase the realism of the control system for the operator, telepresence allows information perceived in the environment to be transmitted back to the human user so that the user feels his physical presence at a remote object [5,6]. The information sent back to the operator allows him to make adjustments to his inputs to the system depending on how well he is able to control his outputs, thus closing the control loop.
The aim of the study is to develop a somatosensory controller for remote control of a manipulator for picking apples using resistive sensors, a gyroscope and an accelerometer.

Materials and methods
The methods of computer modeling, differential and integral calculus and programming were used in the work. When modeling the design of the manipulator device, agrotechnical requirements for the basic processes of machine technologies in horticulture and physical and mechanical features of the structure of the crowns of intensive-type fruit trees were taken into account [7][8][9]. To substantiate the parameters of the manipulator for removing fruits, the following initial technical requirements are presented.
1. The manipulator must have the properties of autonomy, be able to analyze the properties of the external environment and automatically change the operating modes of operation.
2. The manipulator must be able to capture the fruit within the working area on the surface of the tree crown and in the depth of the crown no more than 200 mm. The positioning accuracy of the links of the robotic arm relative to the fruit should be 10-20 mm.
3. After successfully grasping the fetus, the manipulator must rotate the device by at least 45 degrees.4. The manipulator must carry out fruit picking in the zone of a flat tree crown with the following dimensions: vertically from the soil from 1000 to 3500 mm, horizontally 1500 mm.
Requirements for the weight and size characteristics of the manipulator device: the mass of the manipulator should not exceed more than 45 kg, the weight of the object (fetus) being lifted with the full reach of the arm should be at least 2 kg, the distance of the upper arm should not exceed 700 mm, the distance of the upper and lower arms should not exceed 2100 mm.
As a result of the research, a somatosensory controller has been developed for remote control of a manipulator device. To read the parameters of the position of the controller in space, as well as to control the bending of the fingers, an electronics complex was used. The core of the complex is the Arduino Nano ATmega328P microcontroller. It processes data from sensors and transmits the information received to a computer. For data transmission, the Bluetooth wireless communication protocol for wireless data transmission in the 2,4 GHz band, JDY-16 module, based on the nRF51822 chip, was used. The UART protocol was used to communicate the JDY-16 module with the microcontroller. To determine the position of the controller in space, an MPU-6050 gyroscope-accelerometer module with six measurement axes was used: three axes for measuring acceleration and three axes for measuring angular velocity. The module communicates with the microcontroller using the I2C protocol. To determine the bend of the controller fingers, resistive sensors FLX-03 are used, which, depending on the bend, change their resistance in the range of 25-60 kOhm. A resistor divider is assembled to convert the resistance of the sensors into voltage. Sensors are connected to pins A0-A3 and A6 of the microcontroller. To ensure design flexibility, all modules were connected using a surface mounting method. A module has been developed to obtain acceleration values for moving the "arm" of the manipulator device in space, reading values from the gyroscope-accelerometer and a module for analyzing data from resistive sensors to obtain finger bend values on the hand to implement the movement of the manipulator links.
The Gazebo simulator was used to simulate the process of picking apple fruits. The architecture of the Gazebo simulator allows the use of various high-level programming languages and various types of sensors and RGB sensors, as well as connecting common packages. An environment has been created, including a model of a robotic device for picking apple fruits and a fruit tree, and the properties of the environment, such as gravity and atmosphere, have been set (Fig. 1). The movement of the manipulator links along the X axis is carried out with the help of an accelerometer, by tilting the controller with a sensor along the axis, in connection with which the effect of gravity on the sensor plane, on which it was calibrated, is reduced. The movement of the manipulator links along the Z axis is carried out using resistive bend sensors on the index and middle fingers; when the sensor is bent, the manipulator links begin to move. The movement of the manipulator links along the Y-axis is carried out using a resistive bend sensor on the ring finger.
The model of a manipulator for picking apple fruits was developed in KOMPAS-3D. Taking into account the requirements for the design of the manipulator, a simulation model was created, including the base, lower arm, upper arm, stepper motor with an angular sensor, rotary mill with gear, lower actuator, upper actuator, upper arm movement actuator, gripper compression and unclamping actuator , capture, magnetic angle sensor. The manipulator has 4 degrees of freedom, 3 of which are rotational and one translational, while the second and third degrees of freedom are controlled by linear actuators. The fruit tree model was developed using the SDF (Simulation Description Format) modeling tool. Defined the shape of an apple tree using the built-in geometric primitive sphere. Added physical properties to models, materials and textures. A Python script for control has been developed, which includes the ability to determine the distance between the gripper and the fruit of an apple tree using a camera and a YOLOv5 convolutional neural network. The script can use various algorithms such as PID (Propotional-Integral-Derivative) to control the movement of the manipulator. The YOLOv5 model divides the image into a grid of cells, each of which is responsible for detecting and classifying an apple fruit if it is inside the cell. This method allows you to significantly speed up the process of image processing and achieve high accuracy in detecting objects. The YOLOv5 model used in research uses lightweight CNN architectures such as CSPNet, which significantly reduce the number of network parameters and training time, while maintaining high accuracy [10][11][12]. To train the used neural network model, the public dataset COCO (Common Objects in Context) was used. The Gazebo Simulator tool was used to test the operation of the models and analyze their behavior. To recognize objects in the image, a ZED Stereo 3D camera was used, which performs three-dimensional shooting at a resolution of up to 2K, suppresses «noise» at low light levels, and has a wide viewing angle of up to 110°.
At the first stage of the model implementation, using the camera and the YOLOv5 convolutional neural network in the Gazebo environment, the coordinates of the object (fetal model) are determined. At the second stage, the inverse problem of kinematics is solved according to the developed method, as a result of which the manipulator starts moving to the target point. Based on the developed models of the object being assembled (apple fruit) and the objects on which it is located (fruit trees), a scene was created using the elements of Joints-links and Shapes-forms. The apple model is loaded into the simulator as a file with the OBJ extension, the texture is specified from the file with the .jpeg extension. The simulator implements the interaction of the assembled object with the gripping device.

Results and discussion
To conduct a comparative experiment to evaluate the results of computer simulation of the process of picking apple fruits using a self-sensor controller and a manufactured sample of a manipulator, artificial models of trees with an identical crown and the location of fruits in the crown at a distance of no more than 1,5 meters from the zero point of the manipulator were used. The experiment was carried out in triplicate. The input parameters were the numbers of apples with different locations. The distance from the zero point of the manipulator to the crown varied within 0,3-0,5 meters. The output parameter was the time of separation of the apple from the stem from the moment the manipulator began to move. The aiming of the manipulator was carried out according to the principle of capturing an object in the field of view and moving it to the center of the image of the video sequence, after which the boom was extended with the capture. In the case of a successful capture of the desired object (fetus), the end sensor located on the grip is triggered, which immediately after that is pulled back, while it closes, due to which the fetus is torn off. After performing this operation, the return of the manipulator boom to its original position and its opening with the subsequent dumping of the fetus is implemented. If there are other identified objects in the frame, the procedure is repeated, while aiming occurs at the nearest object with the highest degree of recognition. The stages of operation of the manipulator device include the start of movement, the moment of aiming the gripping device, separation of the fetus and transportation of the fetus (Fig. 2). According to the results of the research, it was found that the average time of harvesting one apple fruit by a manufactured manipulator in an automated mode is 12,9 seconds, the minimum time was 9,4 seconds, the maximum 15,9 seconds. Gazebo environment using a self-touch controller was 11,7 seconds. The minimum time was 8 seconds, the maximum 14,9 seconds (Fig. 3).   Fig. 3. The results of a comparative experiment of computer simulation of the process of picking apple fruits using a self-sensor controller and a manufactured sample of a manipulator device.
Comparison of the results of computer simulation and the experimental data obtained to determine the time to capture it confirmed the time to capture one fetus is no more than 16 seconds. Deviations of the trajectories of the movement of the manipulator were revealed when working in the middle and upper tiers of the crown: the maximum values of deviations of the trajectory of the movement of the manipulator grip from the calculated one did not exceed 5,8 cm.
The self-sensor controller showed more efficient operation in the process of picking apple fruits in comparison with the manufactured sample of the manipulator device operating offline. The average time of harvesting one apple fruit when using a self-sensor controller is 1,5 seconds less than when using a manufactured sample of a manipulator device. The average positioning accuracy of the gripping device is 1,2 cm, and computer simulation in the Gazebo environment made it possible to achieve a positioning accuracy of 0,8 cm. zero point of the manipulator, while computer simulations using a somatosensory controller showed more stable results at longer distances. However, it should be noted that the accuracy and speed of the manipulator device when using the developed somatosensory controller depend on the experience of the operator and his motor skills.

Conclusions
As a result of the conducted research, a simulation model has been developed for controlling a manipulator device using a somatosensory controller in a Gazebo environment. The generalized coordinates of the center of the grip of the manipulator device are established, the parameters and the working space in the half-plane crown of the fruit tree with the location of the fruits along the crown height from 600 to 1800 mm are determined.
The results obtained demonstrate the potential of using artificial intelligence technologies in the cultivation of agricultural crops. The development of digital intelligent systems and telematics services will ensure the wide application of the proposed method for monitoring and managing the group work of robotic tools in the cultivation of fruit and berry crops.
The transmission of a sense of presence at a remote object, when picking apple fruits, can be supplemented with the help of various technologies, including virtual reality (VR), augmented reality (AR) and feedback. VR and AR allow you to create a visual presence, while feedback will allow you to transmit tactile sensations, such as resistance, vibration or pressure, to provide the necessary effort to tear off the fruit of the apple tree. The proposed technology can also be supplemented with machine learning to automatically adjust the movements of a robotic manipulator based on feedback. The use of a somatosensory controller with feedback will significantly improve the efficiency and accuracy of remote control of a robotic manipulator, which in turn will ensure timely response to changes in environmental conditions, and will also allow using the experience of operators to make optimal decisions in real time.