Daily natural gas load prediction method based on APSO optimization and Attention-BiLSTM

Attention mechanism

The attention mechanism was first used in the domains of computer vision and natural language processing, inspired by the human attention mechanism to sift through a large amount of information and identify the most valuable content. It automatically assigns different weights to various features at different time stages (Yang et al., 2022; Luo et al., 2018). The principles of the attention mechanism used in this article are illustrated in Fig. 1.

The whole computational process consists of three stages. In the initial stage, the similarity between the query and the key is calculated using the function: (1)${\displaystyle F\left(Q,K\right)=Q^T{K}_i=Si{m}_i\left(i=1,2,3,\dots ,n\right)}$

The second stage involves a scoring mechanism for attention. It primarily uses a Softmax operation to normalize the similarity derived in the previous stage: (2)${\displaystyle A_i=Softmax\left(Si{m}_i\right)=\frac{F\left(Q,K\right)}{\sqrt{d_k}}}$

In the third stage, the computed weights are used to perform a weighted summation of all the values in V, resulting in the final attention vector: (3)${\displaystyle Attention=\sum _i^n{A}_i{V}_i.}$

Basic principles of BiLSTM

Long short-term memory (LSTM) is a type of recurrent neural network (RNN) designed for time series prediction. It exploits the long-term memory capabilities of RNNs and overcomes challenges in traditional neural networks such as gradient explosion and vanishing gradient (Wang, Mu & Liu, 2021; Lu, Rui & Ran, 2020). Unlike the single network structure of traditional RNNs, the LSTM unit (Ballesteros Martín et al., 2010) consists of four network layers, which provides a significant advantage in capturing long-term dependencies in temporal data. In addition, the LSTM includes a gating structure that evaluates and filters incoming information to determine whether it should be retained or discarded. A standard LSTM unit is equipped with input, output, and forget gates. The detailed structure of the LSTM is shown in Fig. 2.

The BiLSTM model consists of both a forward LSTM and a backward LSTM, stacked on top of each other. Its hidden layer consists of two units, both of which receive the same input, process time series in both forward and backward directions, and connect to the same output. This design enables BiLSTM to analyze load data bi-directionally and compute the forward and backward propagation states separately. The structure of the BiLSTM is shown in Fig. 3.

The BiLSTM can analyze the load data bidirectionally and compute the forward and backward propagation states using the BiLSTM network, as follows: (4)${\displaystyle {\overrightarrow{C}}_t=LSTM\left(x_t,{\overrightarrow{\mathrm{h}}}_t-1,{\overrightarrow{C}}_t-1\right)}$ (5)${\displaystyle {\overleftarrow{C}}_t=LSTM\left(x_t,{\overleftarrow{h}}_t-1,{\overleftarrow{C}}_t-1\right)}$ (6)${\displaystyle C_t=W_T{\overrightarrow{C}}_t+W_V\overleftarrow{C_t}}$

where $\overrightarrow{{\mathit{C}}_{\mathit{t}}}$ and $\overrightarrow{{\mathit{C}}_{\mathit{t}}}$ denotes the memory cell state of forward and backward LSTM at time t, W_T is the forward and moment unit state weight coefficients, W_V is the backward moment unit state weight coefficients, x_t is the input information and t is the current LSTM unit output hidden state.

Adaptive particle swarm optimization

The concept of particle swarm optimization (PSO) is inspired by the migratory movements of flocks of birds and fish in the animal kingdom (Kennedy & Eberhart, 1995). Similar to genetic algorithms (Shneiderman, 1996), each individual in the swarm has two characteristics: position and velocity. The fitness of a particle is determined by an objective function corresponding to its position coordinates, and this fitness measures the merit or quality of the particle. While the particle swarm algorithm is characterized by its simple principle and fast convergence speed, it also has notable shortcomings, including poor local search ability and premature convergence. The adaptive particle swarm optimization (APSO) method improves upon the basic PSO. It incorporates concepts similar to mutation in genetic algorithms and probabilistically re-initializes specific variables to increase the likelihood of finding the optimal value. Its core algorithm is similar to the particle swarm optimization algorithm, which involves updating the speed and position of particles based on specific rules. These rules are articulated in Eqs. (7) and (8). (7)${\displaystyle v_{i,j}^{t+1}={\omega }^t{v}_{i,j}^t+{\varphi }_p{u}_{p,j}^t\left(x_{p,j}^t-x_{i,j}^t\right)+{\varphi }_g{u}_{g,j}^t\left(x_{g,j}^t-x_{i,j}^t\right)}$ (8)${\displaystyle x_{i,j}^{t+1}=x_{i,j}^t+v_{i,j}^{t+1}}$

where $v_{i,j}^t$ is the velocity of the i particle in the j dimension of the t generation, $x_{i,j}^t$ is the position of the i particle in the j dimension of the t generation, $u_{p,j}^t$ and $u_{g,j}^t$ are two random numbers, ω^t is the inertia weight, ϕ_p andare the learning factor, $x_{p,j}^t$ and $x_{g,j}^t$ represent the position of the first particle on the j dimension of the individual optimal solution and the global optimal solution of the t generation.

The results show that the convergence of PSO is significantly influenced by the inertia weight. To improve searchability, APSO introduces adaptive adjustment of the inertia weight, which allows the inertia weight to vary with the objective function of each particle. This helps to overcome the premature convergence shortcomings of the PSO algorithm. The flow of the APSO algorithm is shown in Fig. 4. The updated adaptive inertia weight formula is shown in Eq. (9). (9)${\displaystyle \omega =\left\{\begin{array}{c}\omega \left[\min \right]-\frac{\left(\omega \left[\max \right]-\omega \left[\min \right]\right)\times \left(f-f_{\min }\right)}{f_{avg}-f\min },f\le f_{avg}\hfill \\ \omega \left[\max \right],f>f_{avg}\hfill \end{array}\right.}$

where ω_max and ω_min are the maximum and minimum of inertial weights, f is the objective function value of the particle, f_avg is the average objective value, and f_min is the minimum objective value.

Another key to APSO optimization is the adaptive adjustment of the learning factors. In each iteration, APSO dynamically adjusts the learning factor based on the variance of the calculated adaptive degree of each particle. When the variance of the adaptive degree is low, the learning factor is reduced to facilitate convergence to the optimal solution.

Experimental study

Data pre-processing

(1) Data cleaning: Outliers are frequently generated during data collection due to machine failures, manual errors, and so on. To mitigate the impact of missing and noisy data, polynomial interpolation is employed to fill in the missing values, while random forest algorithms are utilized to identify and remove outliers from the dataset, thereby improving the accuracy of the forecast (Qiu et al., 2020). After filling in all the missing data points, the dataset is resampled to represent daily load data.

(2) Normalization: To avoid the negative impact of different eigenvalue scales on prediction accuracy and to speed up the gradient descent process, the load data are normalized to the same order of magnitude (Sun et al., 2020). The formula is shown in Eq. (10): (10)${\displaystyle x_i=\frac{x-x_{\min }}{x_{\max }-x_{\min }}}$

where x_i is the normalized data, x is the original data, x_max is the maximum values in the actual data, and x_min is the minimum values in the actual data.

Model evaluation metrics

To accurately assess the prediction accuracy of the model, this article utilizes mean absolute percentage error (MAPE), root mean square error (RMSE), and coefficient of determination (R²) to evaluate the performance of the prediction model.

The MAPE model represents the total error of the model by using a percentage to measure the deviation’s size. RMSE indicates the level of discrepancy between the predicted data and the original data; the lower its value, the higher the prediction accuracy of the model. R² is the coefficient of determination and reflects the proportion of features that can be accounted for by the prediction model for the variation in the daily natural gas data. A value closer to 1 indicates a better fit of the model. The formulas are as follows: (11)${\displaystyle RMSE=\sqrt{\frac{1}{n}\sum _{i=1}^n{\left(y_i-\widehat{y_i}\right)}^2}}$ (12)${\displaystyle MAPE=\frac{100\text{\%}}{n}\sum _{i=1}^n\left|\frac{y_i-\widehat{y_i}}{y_i}\right|}$ (13)${\displaystyle R^2=1-\frac{\sum _{i=1}^n{\left(y_i-\widehat{y_i}\right)}^2}{\sum _{i=1}^n{\left(y_i-\stackrel{\text{\_}}{y_i}\right)}^2}}$

where y_i and $\widehat{y_i}$ represent actual and projected values, respectively. $\stackrel{-}{y_i}$ represents the average value of the data.

Experiment design and result analysis

Experiment design

The complexity of natural gas load forecasting is multivariate, non-linear, and intrinsically unstable. Accurate predicting results are frequently not obtained when relying just on single model. An innovative APSO-Attention-BiLSTM model is proposed in this article to enhance forecasting accuracy and reliability. The Attention-BiLSTM technique emphasizes assigning different weights to hidden layers to facilitate bidirectional learning of the data, thus addressing the challenges posed by the complexity of the data. The hyperparameter optimization of the hybrid models aims to improve the prediction accuracy using the APSO algorithm. The flowchart of the proposed model is shown in Fig. 5.

First, the raw daily load data is cleaned and normalized in the data pre-processing stage. Then, partition the processed sequence into training and test sets. The training set samples are fed into the Attention BiLSTM model where features are extracted and the hidden layer state weights of the model are computed. The Attention-BiLSTM was optimized using APSO, and the optimal solution of BiLSTM weight and threshold was obtained. The test set samples are fed into the optimized Attention-BiLSTM model to obtain the final prediction results.

To validate the effectiveness of the model proposed in this article, it is compared and analyzed with several other prediction models. The comparison models include the BP model, RF model, LSTM model, BiLSTM model, Attention-BiLSTM model, and PSO-Attention-BiLSTM model. Our approach uniquely exploits and automatically optimizes the model’s hyperparameters using the adaptive particle swarm algorithm. The key hyperparameters that are adjusted include the number of nodes in the hidden layer, the initial learning rate, the number of layers in the neural network, and the number of training rounds. The optimal hyperparameters are shown in Table 1.

Attention-BiLSTM model experimental results analysis

The algorithmic experiments compare and validate five single models, including the BP, RF, CNN, LSTM, and BiLSTM models. The predicted and actual value curves for the five single prediction models are shown in Fig. 6 and the prediction accuracies are shown in Table 2. Figure 6 makes it clear that, in comparison to the BP, RF, and CNN models, the predictions made by the LSTM and BiLSTM models more closely matched the actual values. Table 2 shows that the fit indices for the CNN, RF, and BP models are, respectively, 0.68, 0.69, and 0.77, which are all less than 0.8. The R² of LSTM and BiLSTM are both higher than 0.8, being 0.82 and 0.86 respectively. Compared to LSTM, the index of BiLSTM is improved by 4.88%. This suggests reduced prediction bias and a better match to the data. It has the highest prediction accuracy of the five individual models.

Based on the favorable prediction accuracy of BiLSTM, the attention mechanism is incorporated to compute the hidden state of BiLSTM. The two subgraphs in Fig. 7 illustrate how the attention mechanism influences the hidden layer weight and compare the predictions of the attention-BiLSTM and BiLSTM models, respectively. It is clear from the graphic that the attention mechanism emphasizes the learning of the various hidden levels by giving them varied weights. It also compares the prediction results of the combined model with those of the BiLSTM model. Table 3 compares the prediction results of the combined model. As can be seen from Table 3, the BiLSTM model considering the attention mechanism has a MAPE value of 4.20% and an R² of 0.90. In combination with Table 2, the MAPE of the BiLSTM model is 4.95% and R² is 0.86. The MAPE of the Attention-BiLSTM model decreased by 15.15% in comparison to the BiLSTM model, while its R² increased by 4.5%. This suggests that the Attention-BiLSTM model fits more accurately and has a lower prediction error index than the BiLSTM model.

APSO optimization model experimental results analysis

To further explore the characteristics and advantages of the Attention-BiLSTM model optimized by the adaptive particle swarm algorithm, it is compared and analyzed with the PSO-Attention-BiLSTM model. The same set of data was employed, and the particle swarm algorithm’s parameters were set exactly like those of the adaptive particle swarm method, to guarantee the experiment’s rigor. The specific parameters are shown in Table 1.

The prediction curves of each combined model for natural gas daily load data are depicted in Fig. 8, and the prediction accuracies are shown in Table 3. Upon closer examination of the detailed graph, it is evident that the Attention-BiLSTM model, optimized by the adaptive particle swarm algorithm, demonstrates superior adaptability to the fluctuating trend of the data. This is especially true at the turning point where the data undergoes multiple changes. In conjunction with Table 3, the PSO-Attention-BiLSTM model has a fitting accuracy of 0.95 and a MAPE of 2.53%, but the APSO-Attention-BiLSTM model has a fitting accuracy of 0.99 and a MAPE of only 0.90%. (1) Compared with the PSO-Attention-BiLSTM model, the enhancements were 64.43%, 67.15%, and 4.64% in the respective metrics. (2) In contrast to the Attention-BiLSTM model, the improvements were 78.57%, 79.37%, and 9.86%. (3) Compared to the BiLSTM model, the improvements were significant at 81.82%, 81.16%, and 14.81%. (4) The LSTM model showed significant increases of 78.42%, 84.68%, and 20.68% in the evaluation metrics. (5) Compared to the CNN model, the evaluation metrics improved by 78.42%, 84.68%, and 29.17%. (6) Compared to the RF model, there was an improvement of 87.68%, 89.30%, and 43.64% in each of the metrics. (7) In comparison to the BP model, it showed improvements of 92.48%, 92.34%, and 46.52% in each of the indicators. The results show that the combined model proposed in this article has the highest prediction accuracy. It is higher than other models. The improvement percentage of the model optimization is shown in Fig. 9.

The models have known accuracy. We evaluate the efficiency of the combined models after adding the attention mechanism, the particle swarm algorithm, and the adaptive particle swarm algorithm. This study calculates the average running time of each combined model over 100 epochs. The results are shown in Table 4. As shown in Table 4, the APSO- Attention-BiLSTM model takes 266.63s to run 100 epochs. There is minimal variation in the model’s running time. The prediction accuracy has significantly increased, though.

Performance analysis of the APSO-Attention-BiLSTM model prediction

The comparative analysis between the predicted results and the actual data shows that all models track the actual data trend reasonably well. However, their accuracy varies. The relative error of each model is shown in Fig. 10. The absolute error box plot of each model is shown in Fig. 11. The absolute error represents the absolute value of the difference between the measured value and the true value. The relative error expresses the absolute error as a percentage of the true value. Absolute errors cannot be used to compare the reliability of different measurements. In terms of relative error, the relative error of the APSO-Attention-BiLSTM model fluctuates within the range of ±0.1%. It has the smallest range of variation. In terms of absolute error, the box chart displayed in Fig. 11 reveals the presence of outliers for each of the five single models. Among the three combined models without outliers, the model proposed in this article has the smallest error range. This shows that the model has high prediction accuracy. It also has a stable distribution range.

We made a violin chart, which display the relative errors, to look at the model’s stability. The violin plot is a combination of the box plot and the kernel density plot. The box plot indicates the quartile locations, while the kernel density plot shows the density at each point. A larger area indicates more aggregated data and more stable distributions. Fig. 12 shows that the prediction error of the APSO-Attention-BiLSTM model falls within the range of ±0.1% with the highest probability. This indicates that the model proposed in this article performs well in terms of both prediction accuracy and stability.

The great accuracy and stability of the prediction model suggested in this research are confirmed by the previously mentioned analysis. In addition to predictive accuracy, the robustness of a predictive model is typically assessed by evaluating the resilience of the methodology and the explanatory power of the metrics. In other words, when certain parameters are altered, will the evaluation methodology and indicators still provide a consistent and stable interpretation of the evaluation results? There are many ways to test robustness. Outlier removal in the previous data processing also maintains model robustness. Considering that since the outbreak of the epidemic in 2019, the daily natural gas load data will inevitably be influenced by its fluctuations. In this study, we have chosen to adjust the sample size for selecting subsamples from the dataset to conduct validation of the combined model predictions. We have selected the load data for the two-year period of 2017/7/1 to 2019/6/30 as a sub-sample, and have also predicted the load for June during the summer to mitigate the impact of seasonal variations. Divide the subsamples into training and test sets. The training set: 2017/7/7–2019/5/30, test set: 2019/6/1–2019/6/30. Figure 12 displays the prediction results of the subsample combination model, and Table 5 displays the evaluation indicators. As indicated in Table 5, the Attention-BiLSTM model’s R² is 0.85, the PSO-Attention-BiLSTM model’s R² is 0.88, and the APSO-Attention-BiLSTM model suggested in this work has an R² of 0.93. Compared to the other two combined models, the model presented in this research has substantially higher decision coefficients.

As can be seen from the above analysis, the APSO-Attention-BiLSTM model, which is the subject of this research, has a higher prediction accuracy than both single models and other combined models. When compared to the single models, the improvement percentage is higher than when compared to the other combined models. This shows that the APSO algorithm excels in parameter optimization, positioning the proposed method above its counterparts in terms of data fitting and stability. As a result, our model forecasts the daily natural gas load data with greater precision.