Neural Network Predictions Can Be Misleading Evidence From Predicting Crude Oil Futures Prices

This paper applies Neural Network to predict WTI crude oil futures prices on basis of intraday minutely high frequency data. We looked into the recent market crash in the WTI crude oil futures market in April before the Delivery day. The results indicate that Neural Network could be misleading. More specifically, the paper shows that in normal situations Neural Network works well in sample and out of sample but it could give predictions with the opposite signs when the there exists a crash such as the one happened on April 20th, 2020. The evidence demonstrates that the prediction based on Neural Network may not be suitable to predict the market crash which is due to extreme shocks or financial or economic crises. This study gives a new insight in the relation between short term price discovery and the extreme market crash movement.


Introduction
WTI crude oil futures have been one of the highest market capitalizations in the commodities asset space, and due to increasing speculation, it is now gradually becoming one of the highest traded Commodities and attracts the attentions from both industrial and financial investors and speculators. The price prediction has been a hot topic as well.
Garbade and Silber developed a simple but innovative model of simultaneous price dynamics to examine the ability of futures markets [1]. Oellermann, Wade, and Farris examined the feeder cattle market and found that futures prices lead spot prices but that the strength of this lead became weaker in more recent periods [2]. Quan applied the Garbade-Silber model to the crude oil market and found contrasting results with respect to the price discovery function [3]. From the past research it can be concluded that the crude oil futures market performs the functions of price discovery and risk transfer rather well.
The prediction methods on financial products have been gathering great attention as well. With the development of prediction methods and skills, there are plenty of price prediction methods. Price prediction methods in the finance areas are mainly technical analytical methods, fundamental analysis, and prediction based on time series and machine learning. The machine learning and time series based statistical modeling are among the most popular quantitative prediction methods.
For deep Neural Network, Sun and Vasarhelyi confirmed that deep Neural Networks have a better overall predictive performance compared with machine-learning algorithms of logistic regression, naive Bayes, traditional artificial neural networks, and decision trees [4]. The results indicated that the SVM could yield more precise results.
Yim predicted the return of the daily index of Brazil stock and compared the results of prediction using the artificial Neural Networks with the results of generalized autoregressive conditional heteroskedasticity and ARMA and structural prediction models and showed the superiority of artificial Neural Networks [5].
Deep learning is one of the machine learning models that has recently been innovated, using learning in various levels and layers. Learning could be supervised, semisupervised, or unsupervised [6][7] [8]. The most important advantages of deep learning are automatic learning of features, multilayer learning of features, high precision in results, high generalization power, and identification of new data.
As for prediction of crude oil prices, Lin gives a grey prediction model of monthly intentional crude oil prices and show that the model of GM (1,1) is suitable for crude oil prices forecast [9]. The weak-form efficiency of energy futures markets has long been studied and empirical evidence is mixed. Jiang, Xie, and Zhou use nonparametric methods to estimate the Hurst indexes of the WTI crude oil futures prices (1983-2012) and applies a strict statistical test in the spirit of bootstrapping to verify the weak-form market efficiency hypothesis [10].
Karali, Ye, and Ramirez extend the distributional event response model of Rucker, Thurman, and Yoder by developing a mixed event response model (MERM) to allow for possible asymmetric effects and apply the model to the crude oil futures market and found out that the truly unanticipated events, such as the September 11 terrorist attacks, having short-lived impacts and suggest that simply using an event-day dummy variable would hinder discovering overall market responses to slowly evolving information events [11] [12].
Yu, Wang and Lai proposed an empirical mode decomposition (EMD) based neural network ensemble learning paradigm for world crude oil spot price forecasting [13]. In particular, they first decompose the original crude oil spot price series into a finite, and often small, number of intrinsic mode functions (IMFs). Secondly, the tendencies of these IMFs could be accurately predicted based on a three-layer feed-forward neural network (FNN) model which was used to model each of the extracted IMFs. Lastly, they combine the prediction results of all IMFs with an adaptive linear neural network (ALNN), to formulate an ensemble output for the original crude oil price series. They demonstrate the effectiveness of the proposed EMD-based neural network ensemble learning method by applying the approach to two main crude oil price series, West Texas Intermediate (WTI) crude oil spot price and Brent crude oil spot price.
Jammazi and Aloui use three variants of activation function namely sigmoid, bipolar sigmoid and hyperbolic tangent in order to test Yonaba, Anctil and Fortin model's flexibility [14] [15]. Jammazi and Aloui check the forecasting robustness through several levels of inputhidden nodes and their results of HTW-MBPNN perform better than the conventional BPNN. Their conclusions add a major attribute to the previous studies corroborating the Occam razor's principle, especially when simulations are constructed through training and testing phases simultaneously.
Chiroma, Abdulkareem, and Herawan proposes an alternative approach based on a genetic algorithm and neural network (GA-NN) for the prediction of the West Texas Intermediate (WTI) crude oil price and show that the proposed GA-NN approach is better than the baseline algorithms in terms of prediction accuracy and computational efficiency and the WTI crude oil price predicted by the proposed GA-NN and the observed price are statistically equal [16].
Omole, Falode, and Deng use artificial neural network (ANN) to address the inaccuracy of empirical correlations used for predicting crude oil viscosity [17]. The study predicts the crude oil viscosity using 32 data sets collected from the Niger Delta Region of Nigeria and finds that the back propagation neural network model (BPNN) were better than the empirical correlations in terms of average absolute relative error and correlation coefficient.
Tang and Zhang constructed a multiple wavelet recurrent neural network (MWRNN) simulation model, in which trend and random components of crude oil and gold prices were considered [18]. The study utilized the wavelet analysis to capture multiscale data characteristics, while utilizing a real neural network (RNN) to forecast crude oil prices at different scales and adding a standard BPNN to combine these independent forecasts from different scales into an optimal prediction of crude oil prices. The study's simulation results showed that the model has high prediction accuracy.
In this paper we apply Neural Network to predict WTI crude oil futures prices on basis of intraday minutely high frequency data. We show that the Neural Network can work well in sample and out of time when the prices work "normally", that is, the volatility is low and there does not exist crashes. However, the prediction from Neural Network can be very wrong and even gives the wrong signs. Thus, we conclude it should be more cautious to apply Neural Network to predict crude oil future prices. From this point, we contribute to the literature which studies the prediction and price discovery of futures prices using machine learning methods.
The rest of the paper is organized as follows. Section II describes data information. The empirical results are shown in Section III. Section IV states the conclusions, a few concerns and proposes several directions for future research.

Data
Our primary goal of this study is to explore Neural Network to predict WTI crude oil futures prices on basis of intraday minutely high frequency data, especially during the market crash. We looked into the recent market crash in the WTI crude oil futures market in April before the Delivery day.
We have minutely data on WTI crude oil futures prices and its volume on April 17, 19 and 20th, 2020. Figure 1 and 2 plot the price and volume series by days separately. We create price and volume variables with lag of one. We also create predictors which are computed by taking differences in prices and volume, respectively. To capture the potential nonlinear relationship, we compute other variables by squaring the variables constructed above. In all, we have price and volume with lag of one, differences in prices and volumes, and squares of the above variables. We standardize all predictors. The outcome variable of our interest is prices. In this paper, we train the Neural Network using data of April 17 and 19. The data of April 20 is used as test data. Figure 3 shows the observed prices and predicted prices in training dataset from Neural Network. We can see that it fits pretty well with root mean squared errors (RMSE) of 0.052 (as shown in Table 1). We note that the standard deviation of prices in training dataset is 1.166. The ratio of RMSE to SD is only 0.045. Figure 4 shows the out of sample prediction results. We can see that it does not fit well across all time period with a very high RMSE of 18.328. And the ratio of RMSE to SD is 1.338 (as shown in Table 1).

Results
Especially, the Neural Network predicts badly during the crash period (on the right side of the dashed line) in which the RMSE is 33.648 and the ratio of RMSE to SD is 2.424 much greater than 0.824 in the normal period (shown in Table 1). It even gives the opposite signs as seen in Figure 4 when the price plummets.
However, the left side of Figure 4 shows that Neural Network predicts well out of sample when the time is normal (on the left side of the dashed line) with RMSE of 2.107 and RMSE to SD of only 0.824 (shown in Table 1). We define the normal time in the test sample as following. We compute the rolling standard deviations for prices with the time window of 120 seconds. The first time when the rolling SD meets the threshold of 2 SDs of prices in training dataset (which is 2.332) is defined as a threshold between normal and non-normal times. With other rolling time windows such as 60 seconds, we have similar results.
To check the robustness of our results, we change the rolling windows for computing the rolling SDs used to as the cutoff for the normal and non-normal periods in test sample and find that the results are robust. Also, the main qualitative results are unchanged when Neural Network with different layers and hidden neurons is applied. This indicates that it is not a overfitting problem. We separate the datasets into training and test samples in different ways and the main results are robust. That is, Neural Network is not unable to predict the crash out of sample in our data.

Conclusion
We looked into the recent market crash in the WTI crude oil futures market in April before the Delivery day. The analysis conducted shows that as opposite to the ordinary market situation where the prediction can be well performed, Neural Network could be misleading during the market crash period. More specifically, we find that in normal situations Neural Network works well in sample and out of sample but it could give predictions with the opposite signs when the there exists a crash such as the one happened on April 20th, 2020. The evidence demonstrates that that the prediction based on Neural Network may not be suitable to predict the market crash which is due on extreme shocks or financial or economic crises. Our study gives a new insight in the relation between short term price discovery and the extreme market crash movement. It indicates that price discovery on Neural Network is focused on the past data and it needs to combine with the external extreme shocks information to keep with the prediction on the extreme market movements.
This finding is important for investors who are interested in trading WTI Crude Oil Futures using Neural Network for price discovery. It is important to understanding the respective price movement during the normal time and market crash period. However, there are some issues in this paper. Firstly, conditional on the data available we not do have a long data to train the model. In particular, during the train sample we have there do not exist crashes. Thus, longer data may help train a better Neural Network model. Yet at least, it is confirmed by our evidence that training a Neural Network without crashes is possibly unable to predict crashes and it could lead to misleading predictions. Secondly, all the predictors used in this paper are constructed from past volume and prices. One might doubt that these predictors used may not be enough to characterize the price movement of crude oil futures prices. This is a reasonable concern especially it may need more predictors related to extreme shocks to capture the price movement when the market faces crashes. However, it could be difficult to collect data about other potential predictors in high-frequency basis.
Based on the issues stated above, for future investigations it will be straightforward to expand the research if there exists data with longer time periods. Also, it will be interesting to study how to combine the Neural Network price discovery and external shock news (more information and predictors) for the WTI Crude Oil Futures. It also may be preferable to investigate the Neural Network price discovery with other financial products during the market crash periods.