Prediction of stock price direction using the LASSO-LSTM model combines technical indicators and financial sentiment analysis

LASSO algorithm

Lasso logistic regression is a model that incorporates the L₁ norm as a penalty function, in contrast to ordinary logistic regression models, which yield a sparse solution. Here we assume that the predictor variables is X = (x₁, x₂, …, x_n), response variables is Y = (y₁, y₂, …, y_n).c_t represents the closing price of a given stock in t-th day, z_t+1 = c_t+1 − c_t represents the stock return in t + 1-th day. The stock movement direction function can be expressed as follows (1)${\displaystyle y_{t+1}=\left\{\begin{array}{c}\hfill 1,z_{t+1}>0\hfill \\ \hfill 0,z_{t+1}⩽0\hfill \end{array},t=1,\dots ,n-1.\right.}$ where y_t+1 represents the stock prices movement direction in t + 1-th day. Then, the conditional probabilities of up and down trends for stock prices can be expressed as: (2)${\displaystyle P\left(y_{t+1}=1|x_t\right)=\frac{e^{{\beta }_0+x_t\beta }}{1+e^{{\beta }_0+x_t\beta }},}$ generally, the β₀ is intercept of the predictor variables, β = (β₁, …, β_p)^⊤ is the parameter vector of predictor variables, and x_t = (x_t,1, …, x_t,p) is the predictor variables in t-th day. This function is the sigmoid function and can be used to describe binary-classification datasets. Based on the above probabilities function and data set, we get the log-likelihood function as (3)${\displaystyle L\left({\beta }_0,\beta \right)=\sum _{i=1}^{n}log\left(P\left(y_{t+1}|x_t\right)\right).}$

Compared with standard logistic regression, the LASSO algorithm introduces the penalty function, and the LASSO criterion is (4)${\displaystyle Q\left({\beta }_0,\beta ;\lambda \right)=L\left({\beta }_0,\beta \right)+\lambda P_{\alpha }\left(\beta \right),}$

where $P_{\alpha }\left(\beta \right)={\sum }_{j=1}^p\left|{\beta }_j\right|$, $\left|{\beta }_j\right|$ is the penalty function, λ⩾0 is a tuning parameter that controls the intensity of penalization, p is the number of indicator. Breheny & Huang (2011) introduced the coordinate descent algorithm to solve the penalized weighted least-squares problem and get the iterative estimation of LASSO. (5)${\displaystyle \underset{\left({\beta }_0,\beta \right)\in {\mathbb{R}}^{p+1}}{min}\left\{-L\left({\beta }_0,\beta \right)+\lambda P_{\alpha }\left(\beta \right)\right\}.}$ The LASSO algorithm is characterized by the ability of convergence and variable selection. It shrinks the range of parameters to be estimated and only a few parameters are estimated at each step. In addition, the LASSO algorithm automatically selects the critical variables to build the model. In the stock prediction problem, the LASSO algorithm can eliminate the multi-collinearity among predictor indicators and avoid model over-fitting, disease the calculation time of the model. Here we provide the pseudo code on how to apply the LASSO algorithm to obtain estimators.

 
_______________________ 
 Algorithm 1: The solution process of LASSO algorithm                      ____ 
    Input: The training set {(xt,1,xt,2,...,xt,p),yt+1}N 
                                     t=1, the optimal λ, 
               the maximum iteration number M and the tolerance limit ɛ 
 1  Initialization : Set m = 0, ^ β = (^β0, ^ β1,..., ^ βp)⊤, ^ β (0) be the initial 
    value of the parameter estimated vector, ^ β (0) ← 0, 
 2  repeat 
     3   for j = 1,2,...,p do 
    4   the training set X, where the x⋅j = (x1,j,x2,j,...,xN,j)⊤, the 
   xt⋅ = (xt,1,xt,2,...,xt,p)⊤, 
5    ^ Pt =   e^β0(m)+xt⋅^ β (m) __ 
1+e^β0(m)+xt⋅^ β (m) , 
6   ωt = ^ Pt ( 
   1 − ^ Pt) 
        ,W = diag(ω1,ω2,...,ωN), 
7   zj = n−1x⊤⋅jWx⋅j + vj ^ βj(m), 
8   vj = n−1x⊤⋅jWx⋅j, 
9    ^ βj (m + 1) ← S(zj,λ) 
   vj 
10   where S(zj,λ) = sign(zj)(|zj|− λ)+ = ( 
{ 
( zj − λ,zj > 0,λ < |zj|, 
    zj + λ,zj < 0,λ < |zj|, 
           0,λ ≥|zj|. 
11   end 
12  until ∥ ∥ 
∥^ β(m + 1) −^ β(m)∥ ∥ 
    ∥2 
                                  2 ≤ ɛ or m = M; 
    Output: ^β(m + 1)    

LSTM model

Based on the above work, we obtain the indicators after variable selection and could predict the stock movement direction. LSTM has the gating structure to control the transmission of information. Figure 4 is the architecture of the LSTM model.

In Fig. 4, X_t is the input vector at t moment, h_t is the external state, and C_t is the internal state, which performs circular messaging and also outputs information to the external state h_t of the hidden layer. The C_t calculated as follows: (6)${\displaystyle C_t=f_t\odot C_{t-1}+i_t\odot {\overline{C}}_t,}$ (7)${\displaystyle h_t=O_t\odot tanh\left(C_t\right),}$

where C_t−1 represents the memory unit in t − 1 moment, ${\overline{C}}_t$ is the candidate state calculated by no-linear function: (8)${\displaystyle {\overline{C}}_t=tanh\left(W_c{X}_t+U_c{h}_{t-1}+b_c\right),}$

where W_c and U_c represent the weight matrices, and b_c represents the bias values vector. Tanh function is a activation function. LSTM control the path of message delivery by gating mechanism, the function as follows: (9)${\displaystyle i_t=\sigma \left(W_i{X}_t+U_i{h}_{t-1}+b_i\right),}$ (10)${\displaystyle f_t=\sigma \left(W_f{X}_t+U_f{h}_{t-1}+b_f\right),}$ (11)${\displaystyle O_t=\sigma \left(W_O{X}_t+U_O{h}_{t-1}+b_O\right)}$

where σ is the sigmoid function, which output is 0 to 1. Forget gate f_t controls the information needs to be forgotten from last internal state C_t−1; input gate i_t controls the information needs to be input from candidate state ${\overline{C}}_t$. At moment t, the mechanism of the lstm model as follows: (1) Calculating the i_t, f_t and o_t based on last external state h_t−1 and input vector; (2) Combining f_t and i_t to update memory units C_t; (3) Passing information from the internal state to the external state h_t.

LASSO and LSTM integrated forecasting algorithm

We combine the LASSO and LSTM models for short-term stock movement forecasting. Set $D={\left\{X_t,Y_t\right\}}_{t=1}^N$ represent the stock data set. The LASSO-LSTM can be divided into stages: first, we take the stock data set to the LASSO algorithm to select the important variables and eliminate the multi-collinearity among predictor variables. Then, we adopt the grid search and 10-fold cross-validation to tune the parameter value in LASSO model; Finally, we apply the predictor variables after selection as the input vector into LSTM model and predict the stock movement direction. We summarize the procedure of the LASSO-LSTM model in Algorithm2, which can execute variables selection and effectively process the time-series data set due to the penalized function and gate structure.

 
_______________________________________________________________________________________________________ 
  Algorithm 2: The algorithm produce of LASSO-LSTM model              ____ 
  1  Initialization : The predictor variables {Xt}N 
        t=1 are normalized to 
    eliminate scale effects. 
  2  Divide the data set D = {Xt,Yt}N 
            t=1 to the training set and testing set; 
  3  Adopt the LASSO algorithm to variables selection, eliminate the 
    multi-collinearity among predictor variables; 
  4  Set the alternative parameter values {λ1,...,λK}, where 
    0 ≤ λk ≤ 1.λk < λk+1,k=1,...,K. Tuning the λ parameter values in the 
    LASSO algorithm by grid search and 10-fold cross-validation method 
    based on training set; 
  5  Select the λ value with the best performance and retain the corresponding 
    critical predictor variables in the training set and testing set; 
  6  Predict the stock movement direction based on the LSTM model and 
    variables after processing;    

Note that to make sure the parameter tuning is effective, we rank the alternative parameters from smallest to largest to get $\left\{{\lambda }_1,\dots ,{\lambda }_K\right\}$. The selection result of each lambda value was evaluated by grid search and 10-fold cross-validation. Too small lambda values will weaken the variable selection ability of the lasso algorithm, and too large lambda values will lead to the loss of important information in the data. Such a parameter tuning method could guarantee reasonable results and efficient computation. In this way, we construct the LASSO-LSTM model that could reduce the dimensionality of the predictor variables and capture key information in time-series data.

Compared with traditional machine learning models, the proposed LASSO-LSTM has advantages when applied to stock prediction problems: (1) The technical indicators and sentiment indicators in stock prediction always have multi-collinearity, different from standard LSTM, LASSO-LSTM could eliminate these factors on prediction performance; (2) In the prediction phase, the LASSO-LSTM model predicts the time-series stock data by the LSTM model rather standard LASSO model, which can capture the time-series more effectively; (3) The LASSO-LSTM model has fewer hyperparameters, so it does not require several numbers of computational resources to achieve good prediction accuracy.

Data crawling

In this article, we hope to fuse the finical textual data and stock historical transaction data and forecast stock movement more accurately. To this goal, we need to select some companies and collect their transaction and financial news data. First, we select three representative stocks as analysis subjects from NYSE/NASDAQ: AAPL, MNST, and BAC.

In stock history data collection phase, we apply the E-finance packages in Python to crawl three companies’ stock data derived from 2012 to 2020. Stock historical transaction data set includes Opening Price, Closing Prices, Highest Price, Lowest Price, and Trading Volume. Figure 5 plots the three companies’ stock price trends, which cover the various situations of stock movement.

Except for historical stock data, the movement of stock prices is also influenced by the trading willingness of investors. So it is significant to grasp the relevant information about stock market emotions. In this study, the textual data is crawled from the investing.com website, which is an online website that provides financial information. The financial textual content consisted of historical news and relevant articles on the US equities’ publicly traded information. The history data starts from 2012 to 2020, for different companies, the data size varies. Table 1 is the example of textual data after cleaning, which consists of five indicators.

Technical and sentiment indicators

In the data crawling stage, we collect the stock historical transaction data and financial news data from websites. The raw data is characterized as unstructured and has an amount of noise. To transform the raw data collected above into the predictor indicators, we need to use technical analysis and sentiment analysis technology to calculate the predictor indicators. In this section, we apply TTR package to calculate the 37 technical indicators based on 6 stock historical transaction indicators and obtain 4 sentiment indicators by the FinBERT model. The details of the data processing flow are shown in Fig. 6.

In Fig. 6, we can find that the data crawled from websites consists of historical and textual data. To access information that affects stock fluctuation, it is necessary to process the raw data and obtain predictor indicators. The process includes two stages, one stage is to calculate the sentiment scores from the textual content, and another stage is to calculate the indicators that reflect the changes patterns from the past stock prices.

Regarding technical indicators, scholars proposed that technical analysis could find the relationship between stock movement direction and capture the pattern of stock movement (Tsai & Hsiao, 2010). Murphy (1999) systematically studied the technical analysis of the stock market and proposed some technical indicators calculated formulas from the stock past prices. Recently, Yang, Hu & Jiang (2022) proposed group SCAD/MCP penalized logistic regression to predict up trends and down trends for stock prices with 24 technical indicators. To this end, we calculate 37 commonly used technical indicators by the Technical Trading Rules(TTR) package. The details are shown in Table 2.

TTR is a package in R software, which could calculate technical indicators according to technical trading rules based on stock historical transaction data set, details of the calculation formulas see the URL: https://github.com/joshuaulrich/TTR. The indicators that can be calculated by TTR include moving average trend indicators, oscillators, volatility indicators, and volume indicators.

After obtaining the 37 technical indicators, we hope to calculate sentiment indicators from textual content data. Baker & Stein (2004) started with market liquidity and established a model to explore the internal mechanisms of sentiment indicators and stock price fluctuations. In this article, raw textual data consists of a vast amount of articles and news. We extract the news of target companies and then split the content into sentences. Sentiment analysis aims to evaluate textual content based on sentences and calculate the scores for each emotion. The sentiment score is a continuous numeric value and applied to obtain sentiment probabilities for articles as follows: (12)${\displaystyle P_{pos}=\frac{N_{pos}}{N_{total}},P_{neg}=\frac{N_{neg}}{N_{total}},P_{neu}=\frac{N_{neu}}{N_{total}},P_{pos}+P_{neg}+P_{neu}=1,}$

where P_pos is the probability of the article with positive sentiment. N_pos denotes the number of sentences with positive sentiment, and N_t denotes the number of total sentences. Similarly, P_neg and P_neu represent the probability of the article with negative and neutral sentiments. In this article, we calculate the positive/negative/neutral emotional probabilities as the sentiment indicators.

In the sentiment analysis phase, first, we extract the textual content related to the target company, arranging them by date. Afterward, the daily text content is cleaned to remove punctuation and split into sentences. Finally, we input the sentences to the FinBERT model and obtain four sentiment indicators, namely Articles_Number, P_pos, P_neg and P_neu. The example of sentiment indicators after being processed is listed in Table 3, where the Articles_Number denotes the number of daily article news. After the above pre-processing of data, we transform the unstructured raw data into the 41 predictor variables.

Exploratory data analysis

Based on the fusion of 37 technical indicators and 4 sentiment indicators, we obtain the 41 predictor variables to forecasting. We perform an exploratory analysis of the predictor variables, calculate the correlation coefficients of the predictor variables, and visualize them. The details are shown in Fig. 7.

From Fig. 7, we can conclude that the most predictor variables have significant correlation, such as X₂₅, X₁₇,  and X₃₂ etc. The precise correlations or highly correlated relationships among predictor variables can produce multi-collinearity, which influences model stability or make the model parameters difficult to be estimated accurately. In this situation, the LASSO algorithm could eliminate the multi-collinearity by introducing the penalized function to compress some unimportant variables to 0. Due to this advantage, LASSO-LSTM is an effective method to process the stock data set that combines technical and sentiment indicators.

Model accuracy

To compare the prediction accuracy of the six models, we train LASSO-LSTM, LSTM, LSTMTI, LSTMSI, RF, and SVM based on three companies’ stock data. For consistency, the deep learning models use the Adam optimizer for the optimizer and the binary cross-entropy function for the loss function.

Since to the stochastic characteristics of deep learning models, the train models are different in each training phrase. We design the experiment: First, we train each model 30 times based on the stock data set; Then we obtain 30 trained models and evaluate the prediction accuracy on the test set; Finally, we select the trained model with the highest accuracy metrics as the prediction result. Such an experimental design avoids the interference of randomness in the model and makes the experimental results more reliable.

In the evaluation phase, we compare the prediction performance by five evaluation metrics. The prediction accuracy results are listed in Table 7.

In Table 7, it can be seen that the prediction accuracy of the proposed method has an average improvement of 8.533% compared with standard LSTM. We can notice that the proposed LASSO-LSTM with technical indicators and sentiment indicators is superior to LSTMTI or LSMSI, it proves that the combination of technical indicators and sentiment analysis is an effective method to improve prediction accuracy. According to FPR and TPR metrics, we visualised the forecast results, as shown in Fig. 9.

From Fig. 9, the LASSO-LSTM model has the larger Area Under Curve, which indicates the model performance outperforms the baselines. Based on the prediction results of three companies verified that the improved LASSO-LSTM outperforms the classical machine learning model such as LSTM, RF, and SVM. Compared with LSTMSI and LSTMTI, the proposed LASSO-LSTM has superior performance in terms of ROC curves or evaluation metrics. In addition, the LASSO-LSTM has better performance than the standard LSTM model, it can be concluded that the proposed model can grasp more key information by the fusion indicators and variable selection.

Model robustness evaluation

In addition to prediction accuracy, robustness is also an important metric in model evaluation. The model with better robustness has better prediction performance despite outliers, missing data, and white noise interference. The model with poor robustness has great fluctuations in prediction results under the interference of random factors, and their applicability is poor and difficult to generalize.

To evaluate the robustness of the model and eliminate the influence of random factors, we repeated the experiment 30 times for the six models and calculated the best accuracy, mean accuracy, and standard deviation of the accuracy. The model with a high standard deviation value indicates that the model is not stable and has poor robustness. The results are shown in the following Table 8.

From Table 8, we can conclude that: (1) The proposed LASSO-LSTM model outperforms other models in terms of Best Accuracy, Mean Accuracy, Sd of Accuracy, etc. Therefore, it can be concluded that the LASSO-LSTM model proposed in this article is more effective than the baseline model. (2) Traditional machine learning methods such as SVM and RF are less effective than LSTM and LASSO-LSTM models in terms of Mean Accuracy and SD accuracy, and it can be seen that for the multivariate time series prediction task in this article, using recurrent neural networks in the form of LSTM is more effective than traditional machine learning models.

Model statistics test

The Wilcoxon signed rank test determines whether there is a significant difference between two models by constructing a statistic for the difference between the predicted results of different models. In this article, we use the Wilcoxon signed rank test to compare whether there is a statistical difference between the proposed LASSO-LSTM model and baselines. We set the null hypothesis: the prediction results of two models have no statistical differences; the alternative hypothesis: the prediction results of two models has statistical differences.

Table 9 reports statistical test results for the proposed LASSO-LSTM model and five baselines. If P-value ≤0.05, the two models have a statistical difference, the positive statistic difference value indicates the proposed model outperforms the baselines. The statistical test results indicate that the performance of LASSO-LSTM model is superior to all other 5 models.