LSTM-based sentiment analysis for stock price forecast

LSTM neural networks

The LSTM neural network is one of the derivatives of RNN. It not only improves the lack of long-term memory of RNN, but also prevents the problem of gradient disappearance. The LSTM neural network can dynamically learn and determine whether a certain output should be the next recursive input. Based on this mechanism that can retain important information, it provides a good reference and application when constructing a predictive model for this study.

The LSTM neural network has a new structure called memory cell (Gao, Chai & Liu, 2017). The memory cell contains four main components: input gate, Forget gate, Output gate and Neurons, through these three gates, decide what information to store and when to allow reading, writing and forgetting. Figure 1 illustrates how data flows through the storage unit and is controlled by each gate.

Stock price forecast

In the past few decades, more and more people have conducted researches of stock prices forecast with the continuous expansion of the stock market. They try to analyze and predict fluctuations and price changes of the stock market. Stock prices are high dynamics, non-linearity, and high noise characteristics. The price of individual stock is affected by many factors, such as global economy, politics, government policies, natural or man-made disasters, investors’ behavior, etc. It is one of the challenging tasks in time series forecasting problems.

Li, Bu & Wu (2017) and Liu et al. (2017) described that some stock market studies based on Random Walk Theory and Efficient Market Hypothesis. Those past studies believed that stock price fluctuations were random, so it cannot be predicted. According to the efficient market hypothesis, stock prices are driven by news, rather than current or past prices. Since news is unpredictable and stock prices will follow a random walk pattern. It is hard to forecast the stock price with the accuracy more than 50%. However, more and more researches in recent years have shown that the price of the stock market is not random and could be forecasted to some extent.

Shah, Isah & Zulkernine (2019) and Selvin et al. (2017) analyzed the stock market and predicted stock prices. They utilized two commonly used methods that were fundamental and technical analysis. Fundamental analysis is an investment analysis that estimates the value of stocks by analyzing company profiles, industry prospects, political factors, economic factors, news and social media. The information used in fundamental analysis is usually unstructured. This method is most suitable for long-term forecasting. The method of technical analysis tries to use the historical prices of stocks to predict the future development trend of individual stock. By record the daily rise and fall changes of the stock price in the form of charts. And then determine the best buying or selling points by observing its changes. K-line, Moving Average, and Relative Strength Index are commonly used algorithms in technical analysis and are suitable for short-term prediction.

The literature review of stock prediction Shah, Isah & Zulkernine (2019); Bustos & Pomares-Quimbaya (2020) mentioned that technical analysis was one of the most commonly used methods to forecast the stock market and widely studied and used as a signal to indicate when to buy or sell stocks. However, some studies have found that returns obtained from the trading strategy are actually limited based on technical analysis. Fundamental analysis was rarely used in traditional researches because it is difficult to build models through relevant information. However, with the development of natural language processing and text analysis, some recent studies have analyzed non-structure stock related information to improve forecast accuracy. Those information could be official document, financial news, or posts on social networking sites. Due to the rapid progress of artificial intelligence, Bustos & Pomares-Quimbaya (2020) and Nelson, Pereira & de Oliveira (2017) have achieved good results by using SVM and Artificial Neural Networks for stock market forecasts.

SVM is a tool of machine learning that can be used to deal with classification and regression problems. It is also called Support Vector Regression (SVR) for regression. It could identify the hyperplane in a high-dimensional feature space to accurately predict the distribution of data. Xia, Liu & Chen (2013) used SVR to create a stock forecast method based on regression model with historical time series data, today’s opening price, highest price, lowest price, closing price, trading volume and adjusted closing price to predict opening price in the following day. The results have shown that the Mean Squared Error (MSE) is 0.0000253%.

RNN is often used among lots of ANN to deal with time series tasks, and it is also one of the most suitable techniques for dynamic time series forecasting. Liu et al. (2017) used RNN to predict stock volatility and found that the accuracy of the RNN model is better than MLP and SVM models. However, RNN will have the problem of gradient disappearance and explosion with multiple recursions. LSTM neural network has been developed to strengthen the operation of the RNN in the artificial intelligence fields. Gao, Chai & Liu (2017) collected the historical trading data of the Standard & Poor’s 500 (S&P 500) from the stock market in the past 20 days as input variables, they were opening price, closing price, highest price, lowest price, adjusted price and transaction volume. They used LSTM neural network as the prediction model, and then evaluated the performance by Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Mean Absolute Error Rate (MSE), and Mean Absolute Percentage Error (MAPE). Their method was better than other prediction ones. Lee & Soo (2017) combined LSTM neural network and CNN to predict the prices of Taiwanese stocks based on historical stock prices and financial news analysis, and found that it could reduce the error of prediction. Li, Bu & Wu (2017) used the LSTM neural network to predict the ups and downs of the China Securities Index 300 (CSI 300) by inputting the opening price, closing price and trading volume of the past ten days. The accuracy could reach 78.57%. Khare et al. (2017) also found that LSTM neural network can successfully predict the ups and downs of stock prices. Lu (2018) analyzed the positive and negative sentiment results only obtained from PTT posts and merged stock historical transaction information as the input vectors of LSTM neural network. Their proposed method can improve the accuracy of stock price prediction.

Text sentiment analysis

Emotions or sentiments are biological states associated with the nervous system and reactions to external stimuli. They could be happiness, anger, sadness, fear, etc. We can divide them into positive or negative emotions by dichotomy. With the rapid development of the Internet and the popularization of mobile devices, people like to express opinions and exchange information through online platforms. It generates a large amount of text data containing emotional information, such as blog articles, social media, and online forum posts, replies, and product reviews, etc. If the emotions related to stocks expressed by users can be effectively analyzed from these text, it can help us understand the online public opinions in time.

According to researches Li, Jin & Quan (2020), Kumar & Garg (2019) and Ren, Wu & Liu (2018) to text sentiment analysis (SA) are part of the field of natural language processing (NLP). A series of methods are proposed for online data like product review analysis, social network analysis, public opinion monitoring, movie box office forecasts, stock market fluctuation trend forecasts, etc (Ali et al., 2019; Ali et al., 2020; Basiri et al., 2020; Li et al., 2020).

Mäntylä, Graziotin & Kuutila (2018) expressed that the early Internet was not as developed and convenient as it is today. The amount of accumulated text data is not much, so there is less research on text sentiment analysis. Online comments, blogs, and social media (such as Facebook, Twitter) have emerged one after another after success of Web 2.0, and the number of text messages has increased rapidly. To process massive amounts of data, SA has become one of the most active research areas in NLP. Turney (2002) used an unsupervised algorithm to determine whether the review content of consumer review sites was recommended or not, including cars, banks. The average accuracy of the algorithm could reach 74%. Pang, Lee & Vaithyanathan (2002) adapted SVM and Na”ıve Bayes, and proposed Maximum Entropy Classification method to classify movie reviews into positive and negative emotional tendencies.

In recent years, many techniques have been proposed for sentiment analysis in different fields and tasks. Li, Jin & Quan (2020), Zainuddin, Selamat & Ibrahim (2018), and Soong et al. (2019) divide them into three categories. They are emotion calculation based on semantic dictionary, emotion classification method based on traditional machine learning and deep learning method.

The dictionary-based method does not require any training data. According to the open source emotional dictionary, count the emotional words that appear in the sentence. Each emotional word has an emotional score, and then calculate the emotional word score of the entire sentence. And it will output the emotional tendency of the sentence according to the score. Traditional machine learning methods use a large number of labelled corpora as training data to establish a classification model, and then apply this model to determine the emotional response of the target sentence, such as SVM, Naïve Bayes, and Decision Tree. In addition, since most emotional dictionaries have insufficient coverage of emotional words, lack of domain words, and ignoring context, and the performance of machine learning methods depends on the number of labeled samples, some studies have proposed a hybrid method combined with semantic dictionaries and machines. Learn to make up for each other’s shortcomings to enhance the effectiveness of emotion analysis.

The study of Bollen, Mao & Zeng (2011) found that the text content of Twitter is related to the fluctuation of the Dow Jones Industrial Average Index (DJIA). Liu et al. (2017) analyzed the sentiments of the posts from the China Stock Forum and converted sentiments as the input of RNN to predict the volatility of the Chinese stock market. They found that sentiment indicators can effectively improve The accuracy of the forecast.

Nofsinger (2001) found that investor’s sentiment is a key factor in the financial market. In some cases, investors tend to buy stocks after good news was released, and it will lead to rise stock prices. They sold stocks after negative news came out, so the price fell. Information in the Internet provides valuable resources to reflect investor sentiments. Many researchers now use SA and news analysis to forecast stock prices.

BERT

The structure of the BERT model is a multi-layer bidirectional transformer encoder. The transformer was a deep learning model using encoder and decoder for translation task. The BERT model took the advantages of the encoder part in transformer. Figure 2 is the BERT model diagram that takes E₁ E₂ ..., E_n as inputs. They could be words or special symbols. After computed through multi-layer bidirectional transformer encoder, E₁ E₂ ..., E_n are the output vectors. In the past researches, we used word2vec method try to map a word to a vector which was used the input of NLP. However, one word no matter in what context was always mapped to the same vector. The BERT model utilized multi-layer bidirectional transformer encoder, they will map a word into different vector according to different context around the input word. In other words, the BERT model is a model that will more precisely map a word to the word vector according the context. Unlike previous language representation models, BERT pre-training a deep bidirectional language representation model based on the upper and lower semantics of all layers.

After we received the word vector from the BERT model, we can combine other deep learning methods to solve NLP tasks utilized the benefits from word vectors that represents the word according to different context. Devlin et al. (2018) implemented BERT through two main steps, namely pre-training and fine-tuning. Use a large amount of unlabeled data to pre-train the model for different tasks, and then use the labeled related data to fine-tune on specific downstream tasks (such as text classification, sentence-to-sentence classification, and question answering systems). Figure 3 is the demonstration of BERT fine-tuning process, taking the input of Chinese text and linear classifier to generate the output as an example. [CLS] is a special symbol added before each input. And the output word embedding vector from BERT pre-training model will be send to a linear classifier which can be used for subsequent classification tasks.

System architecture

The stock market is affected by many factors. If you want to accurately predict the changes in the stock price of individual stocks, it is very important to effectively grasp the relevant information of the stock market. In this paper, we proposed a method that try to analyze sentiments in News and forum posts as the experience after fundamental analysis; and download the historical price of stocks as the technical analysis.

The data flow of our proposed method for stock price forecast is shown in Figs. 4 and 5. Figure 4 demonstrates the text sentiment analysis process for News and forum posts. The news was collected on the Internet and the forum posts were collected on the famous forum PTT stock board in Taiwan. Sentences extracted from articles or posts were sending into the BERT model to determine the positive or negative probabilities. Figure 5 shows that we combine four dimensions data of sentiments from news articles and forum posts along with five dimensions data of historical information, which are opening price, closing price, highest price, lowest price, and transaction volume in the past 20 days. The nine dimensions data are used as input features of the LSTM neural network prediction model to forecast the stock prices for individual stocks.

Data collection

In this paper, we try to forecast stock prices. According to our proposed method, three categories of data are needed. They are historical stock transaction information, news, and forum posts from PTT. We mainly uses Python modules Requests and Beautiful Soup to perform web crawling and HTML parsing to obtain experimental data from January 1, 2015 to March 31, 2020.

First, we collect the daily transaction information of individual stocks from the open data from Taiwan Stock Exchange Corporation, including opening price, closing price, highest price, lowest price, and transaction volume. And then, we collect financial, political, and international news content related to individual stocks from The Epoch Times (2021), China-Times (2021), Liberty Tines Net (2021), News in United Daily Network(UDN) (United Daily Network, 2021), Money Daily in UDN (Money Daily, 2021), TVBS Media (2021), and Yahoo (2021). Duplicated news is deleted.

PTT is one of the most famous and used online forums in Taiwan. A lot of users will posts or discuss stock market-related information on the stock board. We also collect posts related to individual stocks from PTT.

Data preprocessing

The raw data of news articles and forum posts collected form Internet by crawler can not directly send to deep learning models. We will describe how we preprocessing the raw data in detail in the following subsections.

Preprocessing of forum posts from PTT

The management of PTT stock board is rigorous, the board managers only allow posts within nine categories NEWS, EXPERIENCE, SUBJECT, QUESTION, INVESTMENT ADVICE, TALK, ANNOUNCE, QUESTIONNAIRES, and OTHERS. Figure 6 is the diagrammatic sketch of PTT forums’ stock board. The categories of INVESTMENT ADVICE, TALK, ANNOUNCE, QUESTIONNAIRES, and OTHERS are not the related categories to this research, so they were excluded for further analysis. Each category of posts has detailed specifications, so the content of the article will be processed using different methods.

We will first search for posts that match the names of individual stocks from the post titles. And then remove special characters and URLs in the post or replies. Posts of the SUBJECT category can be divided into four sub-categories that is set by board managers. They are Long for suggestion to buy, Short for suggestion to sell, Question for asking a question, and Experience for expressing the experience to the subject stock. If it is Long or Short, it has been revealed that the author’s sentiment towards this stock is positive or negative. The content of the article is only to explain his views and analysis, just label this post as positive or negative according to long or short. If the category is Question, it means that the author is only asking questions. The content of the post does not have emotions, you can omit it, just keep the replies. If it is Experience, the content of the article will be segmented by lines, because most authors end a sentence with a new line, and then sentences and replies are stored in the database.

The content of NEWS posts contains three items, which are the original link, the original content, and the replies of the posts. Because the original content is the same as the previously collected news, only the replies part is used for segmentation. Since the content of the EXPERIENCE and QUESTION category posts is similar to the sub-category of the SUBJECT category, the same processing method is adopted. In addition, since the content of replies has a limit number of words, the content that exceeds the number of words will be automatically displayed in multiple replies. If the consecutive replies are written by the same user, they could be combined into one sentence. After the above processing, the sentence is stored in the database, and then sentiment analysis is performed.

BERT model

The BERT code Devlin (2019) is open source provided by Google. We choose the traditional and simplified Chinese BERT pre-training model (Encoder with 12 layers of Transformer, 768 hidden units, 12 Self-attention heads, 110 million parameters). We apply the code on TensorFlow version 1.14.0 for text sentiment analysis.

9,600 sentences with manual labelled positive and negative sentiments are used as training data, 1,200 sentences are used as verification data, and 1,200 sentences are used as test data. A linear classification model after BERT is trained to perform classification. The fine-tuning parameters are set to the following values, the maximum length sequence is 300, the batch size is 16, the learning rate is 0.00002, and the epochs is 3.

Input the pre-processed sentences of news articles and PTT posts into our trained model, determine whether each sentence belongs to positive or negative sentiment. In order to avoid the result of sentiment classification affecting stock price prediction, only the sentence with probability equal or higher than 0.7 will be count as positive or negative. If the number of positive sentences in an article is more than negative, then this article will be regarded as a positive one, and vice versa. The calculation of Eqs. (1) and (2) will be used to obtain each positive and negative sentiment probabilities for news and PTT. Where P_pos is the probability of positive articles or posts (N_pos) in N_total, and P_neg is the probability of negative articles or posts (N_neg) in N_total. In this paper, we use dichotomy method for single article. There are a lot of articles related to one stock in a day, the values of P_pos and P_neg computed by Eqs. (1) and (2) can reduce the influence of error prediction in few articles. (1)${\displaystyle P_{pos}=\frac{N_{pos}}{N_{total}}}$ (2)${\displaystyle P_{neg}=\frac{N_{neg}}{N_{total}}}$

LSTM models for the stock price forecast

In order to recognize the impact of the sentimental information implicit in the news articles and PTT posts on the stock price changes, and to improve the accuracy of the forecast. We will select three stocks from top 50 volume-weighted stocks, and three stocks after top 50 from the open data of Taiwan Stock Exchange Corporation (TWSE, 2021). The selected stocks are based on the total number of news articles from online news sites (The Epoch Times, 2021; China-Times, 2021; Liberty Tines Net, 2021; United Daily Network, 2021; Money Daily, 2021; TVBS Media, 2021; Yahoo, 2021) and PTT posts from online forum system (PTT Forum, 2021). They are Formosa Plastics Corporation (FPC 1301), Hon Hai Precision Industry Co., Ltd. (Hon Hai 2317), Taiwan Semiconductor Manufacturing Co., Ltd. (TSMC 2330), HTC Corporation (HTC 2498), China Airlines (CAL 2610), Genius Electronic Optical (GSEO 3406). The weight stocks are selected as the target of the forecast is mainly due to the amount of training data, because the weighted stocks are the stocks with larger equity and higher market value in Taiwan. Those companies usually have more news and discussion. In addition, we also want to know whether the experimental model can also improve the forecast results for individual stocks that account for a small proportion of the market. Therefore, from the stocks ranked below 50, three stocks are also selected for this research. (3)${\displaystyle i_t={\sigma }_i\left(W_i{x}_t+U_i{c}_{t-1}+b_i\right)}$ (4)${\displaystyle f_t={\sigma }_f\left(W_f{x}_t+U_f{c}_{t-1}+b_f\right)}$ (5)${\displaystyle o_t={\sigma }_o\left(W_o{x}_t+U_o{c}_{t-1}+b_o\right)}$ (6)${\displaystyle c_t=f_t\cdot c_{t-1}+i_t\cdot {\sigma }_c\left(W_c{x}_t+b_c\right)}$ (7)${\displaystyle h_t=o_t\cdot {\sigma }_h\left(c_t\right)}$

Use the historical transaction information of individual stocks from January 1, 2015 to March 31, 2020 (1279 trading days in total) which is downloaded from Taiwan Stock Exchange Corporation, news and PTT content with positive and negative sentiments as input vectors, and LSTM neural network are used for each input time series vectors. The model is used to forecast the price of multiple stocks. Figure 1 illustrates how data flows through the storage unit and is controlled by each gate in LSTM model. Eqs. (3), (4)– (7) express how to compute the values where x_t is the input vector in time t, h_t is the output vector, c_t stores the state of the union, i_t is the vector of input gate, f_t is the vector of forgotten gate, o_t is the vector of output gate, W_i, W_f, W_o, W_c, U_i, U_f, U_o are weights, b_i, b_f, b_o, b_c are the shift vectors, and σ_i σ_f σ_o σ_c σ_h are activation functions.

Because the values range of historical trading data of the input data are not uniform, Min-Max Normalization will be performed before inputting into the LSTM neural network model. The value will be scaled to the interval from 0 to 1. As shown in Table 1, when forecasting is only through stock historical trading data, the RMSE value is mostly the smallest if the time steps is set to 20. This study makes forecasts based on stock prices in the past 20 days, as shown in Fig. 7. In the model, the neural network has three layers including an input layer, an LSTM layer and an output layer. Each nodes in the layer is connected to all nodes in the adjacent layer.