Chaotic System Prediction Using Data Assimilation and Machine Learning

. Atmospheric systems are typically chaotic and their chaotic nature is an important limiting factor for weather forecasting and climate prediction. So far, there have been many studies on the simulation and prediction of chaotic systems using numerical simulation methods. However, there are many intractable problems in predicting chaotic systems using numerical simulation methods, such as initial value sensitivity, error accumulation, and unreasonable parameterization of physical processes, which often lead to forecast failure. With the continuous improvement of observational techniques, data assimilation has gradually become an effective method to improve the numerical simulation prediction. In addition, with the advent of big data and the enhancement of computing resources, machine learning has achieved great success. Studies have shown that deep neural networks are capable of mining and extracting the complex physical relationships behind large amounts of data to build very good forecasting models. Therefore, in this paper, we propose a prediction method for chaotic systems that combines deep neural networks and data assimilation. To test the effectiveness of the method, we use the model to perform forecasting experiments on the Lorenz96 model. The experimental results show that the prediction method that combines neural network and data assimilation is very effective in predicting the amount of state of Lorenz96. However, Lorenz96 is a relatively simple model, and our next step will be to continue the experiments on the complex system model to test the effectiveness of the proposed method in this paper and to further optimize and improve the proposed method.


Introduction
The prediction of chaotic systems is an important area of research [1][2][3]. Predicting chaotic systems is a very difficult task because the essential laws of chaotic systems are still unknown and are extremely sensitive to initial values. For complex systems, such as the atmosphere and oceans, even small perturbations can set off a chain of events with very different consequences. While it is not possible to make long-term predictions about chaotic systems, it is possible to make short-term predictions about chaotic systems. Over the years, many researchers have been modeling and predicting chaotic systems through known physical laws [4][5][6][7]. Currently, many numerical simulation methods have been used for the numerical simulation of chaotic systems. In the case of weather systems, for example, many numerical weather prediction models, as well as climate models, have now been used for weather forecasting and climate prediction, such as the Weather Research and Forecasting Model (WRF), the Global/Regional Assimilation and Prediction System model (GRAPES), and the Community Earth System Model (CESM) [8][9][10][11]. These numerical prediction models have been highly successful and have greatly improved weather and climate prediction. Besides, they have improved the understanding of atmospheric chaotic systems. However, they also face many problems, such as error accumulation, irrational parameterization of physical processes, and huge computational effort.
In recent years, with the advent of big data, datadriven approaches have received more and more attention and research [12][13][14]. Driven by big data, machine learning has achieved great success, especially deep neural networks have had a great impact on many fields [15][16][17]. Studies have shown that neural networks have powerful nonlinear fitting capabilities and can learn the complex physical relationships from massive amounts of data to build corresponding predictive models [18][19][20]. Since deep neural networks have performed well in a large number of prediction tasks, many researchers have started to try to use deep neural networks to predict chaotic time series, and a lot of work has been carried out. This study attempts to apply them to the prediction of chaotic atmospheric systems, using neural networks instead of traditional physics-based models, which can avoid the problems of discretization of control equations and make better use of observed information, making the prediction process more flexible and efficient. However, the problem of initial value sensitivity still exists in the prediction of chaotic systems, so this study still adopts the data assimilation method [21][22] to make full use of the observed data, construct a reasonable initial field, and improve the prediction results.

Proposed Method
Artificial Neural Network (ANN) is a mathematical model that mimics biological neural networks [23][24]. ANN consists of a large number of artificial neurons linked together and is used to estimate or approximate functions. Like other machine learning methods, deep neural networks have been used as a nonlinear modeling tool to solve a wide variety of problems, such as image classification and speech recognition. Figure 1 shows a schematic diagram of an artificial neural network. As shown in Figure 1, an artificial neural network contains an input layer, one or more hidden layers, and an output layer. A deep neural network is capable of learning the complex physical relationships behind massive amounts of data. In this study, we train the neural network using historical observations of the chaotic system to obtain a predictive model of the neural network instead of traditional physics-based models. In addition, we use EnKF-N [25][26] to utilize the observational information to provide a reasonable input to the neural network. EnKF-N is a sequential ensemble data assimilation method. In a concrete implementation, the forecast matrix and analysis matrix for time t is as follows  (2) In this study, the neural network is trained by assimilating the forecast state values to the data, while the data assimilation is trained by fusing the observed data to provide reasonable inputs to the forecast model. The training of the neural network is achieved by minimizing the following loss functions where W is the target parameter to be continuously optimized. Through the above combination of neural networks and data assimilation, we predict the state changes of the chaotic system.

Experiments and Results
In order to analyze the performance of the method proposed in this study, the Lorenz-96 system [27] was selected for this experiment. Specifically, the Lorenz-96 system is controlled by a set of differential equations, which are mathematically formulated as follows: where i x (0<i<m) is the value of the state variable.
F  . The model is integrated using a fourth-order Runge-Kutta scheme with a time step dt=0.001, and at the values of m and F given above, lorenz96 exhibits chaos. In this study, the number of sets used for data assimilation is N = 30. The model noise is taken as a normal distribution with a mean of zero and a standard deviation of 0.01. We constructed a neural network containing 6 hidden layers, each containing 80 neurons, and chose ReLU as the activation function. The input and output of the neural network are 40 variables. Specifically, we use the Tensorflow [28] framework to build and train the neural network. During the training process, we set the learning rate to 0.001 and selected Adam optimizer to optimize the loss function. Adam has been widely used as an effective stochastic optimization method. Therefore, by using Adam optimizer in the process of training the model, we can converge to good results faster. Moreover, to prevent the overfitting problem, we used dropout and set the dropout rate to 0.1. Figure 2 shows the prediction results of the proposed method in this paper for the Lorenz96 system. The green solid line is the true state value, the red dashed line is the prediction result, and the blue scatter is the observed data. From Figure 2, it can be seen that the prediction model combining data assimilation and machine learning can predict the lorenz96 system well. In this experiment, the prediction time can reach two thousand steps, and the prediction results are still very close to the real results. This is due to the strong fitting ability of the neural network and the data assimilation method can make full use of the observed information to provide high quality input data for the neural network prediction model. The experimental results show that the combination of machine learning and data assimilation has great potential in the prediction of chaotic systems and deserves further improvement and development.

Conclusions
To improve the prediction of chaotic systems, a hybrid prediction model combining machine learning and data assimilation is proposed in this paper. The method uses neural networks to construct prediction models instead of physics-based models, in addition, data assimilation provides reasonable input for neural network, and the output of neural network model is used for the next round of data assimilation. We select the Lorenz-96 system to test the method proposed in this paper. The test results show that the method proposed in this paper can predict the amount of state of the lorenz96 system well, and the prediction results are close to the true value over a certain time scale. However, it should be noted that the method in this paper needs to be validated on more chaotic systems. Moreover, there are many details to be further improved in this method, which has a lot of room for improvement. However, it is clear that the combination of machine learning and data assimilation methods is important to improve the prediction of chaotic systems and deserves continuous in-depth research.