Deep learning in finance assessing twitter sentiment impact and prediction on stocks

Sentiment analysis

To begin, we gather a large number of tweets related to different companies that will serve as the fundamental basis for our sentiment analysis. We employ a pre-trained RoBERTa base model, which is known for its exceptional performance in natural language understanding tasks. This model is then further fine-tuned on a labeled Stock-Market Sentiment Dataset to tailor its predictions to the financial domain. During the fine-tuning process, the model’s weights are adjusted to better interpret the language and sentiments expressed in stock market-related conversations.

Fine-tuning is carried out using a supervised learning approach where the model is exposed to examples of tweets paired with sentiment labels. The loss function is then optimized to reduce the discrepancy between the predicted sentiment and the actual sentiment label. The result of this phase is a sentiment analysis model that is capable of identifying the underlying sentiment in tweets regarding companies, classifying them as positive, neutral, or negative.

Stock price prediction model

The approach we use involves two components. Firstly, we use a sentiment analysis model to determine the public sentiment towards a particular stock. Secondly, we use an RNN-based model to forecast stock price trends. RNNs are particularly suited for this task as they can maintain a memory of previous inputs, making them ideal for sequential data such as time series.

We train the RNN with a dataset comprising historical stock prices and the output sentiment from the sentiment analysis model. Through the training process, the RNN learns to recognize patterns and correlations between market sentiment and stock price movements. The objective of training is to minimize the error between the model’s predictions and the actual stock price movements. The RNN’s parameters are adjusted through backpropagation and optimization algorithms to improve its forecasting accuracy.

The integration of sentiment analysis output into the stock price prediction model is crucial. By incorporating sentiment data, the RNN model gains access to a broader context beyond mere price history, encompassing the public sentiment that can indicate future market behavior.

To predict future stock prices, the trained RNN model processes the latest sentiment analysis outputs along with recent stock price data. The model outputs predicted trends in stock prices, which are represented as a time series forecast.

Experimental setting

Datasets

In our study, we utilized three datasets, the first one depicted in Fig. 2, which was combined with binary sentiment scores. The dataset included a total of 5,791 text entries, each linked with a sentiment score indicating either positive (1) or negative (0) sentiment. The content of the ‘text’ variable encompassed market comments, opinions, and forecasts that reflected various themes related to the stock market. By using this data, we could utilize a pre-trained sentiment assessment model.

The second dataset that we have is focused on Twitter activity related to Tesla. This dataset consists of a total of 80,793 tweets that are associated with 25 different stocks. The purpose of this dataset is to investigate the correlation between public sentiment, discussion volume, and market behavior. Figure 2 shows the distribution of tweet lengths related to this company. This distribution captures the variation and concentration of character usage in public discourse over time. The tweets were collected to capture a wide range of sentiments, from brief mentions to more elaborate discussions about the company. Figure 3 shows a closer look at the length distribution can provide insights into the level of engagement and information depth that Twitter users contribute to Tesla. This dataset not only includes tweet length but also time-series data. The left panel of Fig. 4 presents the trend of tweet volumes alongside the stock price movements post-normalization. This provides a temporal viewpoint of social media’s influence on stock prices.

Table 1 displays the third dataset, which consists of tweet volumes for the top technology companies from 2015 to 2020. The dataset covers Apple, Amazon, Google Inc., Microsoft, and Tesla Inc., all of which are major players in their respective markets with significant public visibility. The tweet volumes act as a quantitative measure to assess public interest and sentiment towards these companies, forming a basis for comparative sentiment analysis across different entities within the tech sector.

Table 1:

Tweets volume about the top companies from 2015 to 2020.

Company	Tweet
Apple	1,425,013
Amazon	718,715
Google Inc.	720,138
Microsoft	375,711
Tesla Inc.	1,096,868

DOI: 10.7717/peerj-cs.2018/table-1

Each dataset has a specific purpose in our analysis. The distribution of tweet length helps us understand how Twitter users talk about Tesla, and we can correlate the length of the tweets with the intensity of sentiment or informational content. The time series and volume data give us a dynamic view of how social media activity correlates with stock prices over time. Finally, the cross-company Twitter volume dataset allows us to compare sentiment analysis and suggests that higher tweet volumes may indicate more significant market movements or changes in sentiment.

Integrating these datasets enables us to explore the influence of social media sentiment on stock market performance comprehensively. We can examine not only the breadth of discussion but also its depth and temporal alignment with stock price fluctuations.

Evaluation metrics

Sentiment analysis of stock-related tweets commonly uses precision, recall, and the F1 score as metrics. These quantitative measures help to compare the predictive capabilities of various models or the same model at different stages of training.

Precision is a measure of how accurate the model’s positive predictions are. It is defined as:

$$P=\frac{TP}{TP+FP}$$where TP represents the number of true positives, and FP represents the number of false positives.

Recall measures the model’s ability to identify all relevant instances correctly. The recall R is defined as:

$$R=\frac{TP}{TP+FN}$$where FN represents the number of false negatives.

The F1 score is the harmonic mean of precision and recall, providing a balance between the two metrics. It is particularly useful when the class distribution is uneven. The F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0. The $F1$ score is defined as:

$$F1=2\cdot \frac{P\cdot R}{P+R}$$

In the analysis of the fine-tuned sentiment analysis model, these metrics allow for a nuanced understanding of how well the model performs, particularly in the domain-specific context of stock market-related social media sentiment.

To calculate the value of the relationship between sentiment and stock price, we usually use the directional consistency percentage and the Pearson correlation coefficient.

$$\mathrm{\Delta }\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}(t)=\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}(t)-\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}(t-1)$$where sentiment(t) is the sentiment score on day $t$.

$$\mathrm{\Delta }\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{e}(t)=\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{e}(t)-\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{e}(t-1)$$where $\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{e}(\mathrm{t})$ is the price at time point $t$.

Next, determine the consistency of direction for each day.

$$\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{y}(t)=\{\begin{array}{cc}1& \mathrm{i}\mathrm{f}\left(\mathrm{\Delta }\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}(t)\times \mathrm{\Delta }\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{e}(t)>0\right)\\ 0& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}$$

$$\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{y}\mathrm{P}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{a}\mathrm{g}\mathrm{e}=\frac{{\sum }_{\mathrm{t}=1}^{\mathrm{n}}\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{y}(\mathrm{t})}{\mathrm{n}}\times 100\mathrm{\%}$$

To quantify the relationship between sentiment and stock price using the change rates and Pearson correlation coefficient we first need to calculate the change rate for both sentiment and price at each time point:

$$\mathrm{C}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{R}\mathrm{a}\mathrm{t}{\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}}(t)=\frac{\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}(t)-\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}(t-1)}{\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}(t-1)}$$

$$\mathrm{C}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{R}\mathrm{a}\mathrm{t}{\mathrm{e}}_{\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{e}}(t)=\frac{\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{e}(t)-\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{e}(t-1)}{\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{e}(t-1)}$$

Next, we compute the Pearson correlation coefficient $r$ between these change rates:

$$r=\frac{{\sum }_{t=1}^n(\mathrm{C}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{R}\mathrm{a}\mathrm{t}{\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}}(t)-\overline{\mathrm{C}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{R}\mathrm{a}\mathrm{t}{\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}}})(\mathrm{C}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{R}\mathrm{a}\mathrm{t}{\mathrm{e}}_{\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{e}}(t)-\overline{\mathrm{C}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{R}\mathrm{a}\mathrm{t}{\mathrm{e}}_{\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{e}}})}{\sqrt{{\sum }_{t=1}^n(\mathrm{C}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{R}\mathrm{a}\mathrm{t}{\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}}(t)-\overline{\mathrm{C}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{R}\mathrm{a}\mathrm{t}{\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}}}{)}^2}\sqrt{{\sum }_{t=1}^n(\mathrm{C}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{R}\mathrm{a}\mathrm{t}{\mathrm{e}}_{\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{e}}(t)-\overline{\mathrm{C}\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}\mathrm{R}\mathrm{a}\mathrm{t}{\mathrm{e}}_{\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{e}}}{)}^2}}$$where $n$ is the total number of observations. To assess the accuracy of the stock price prediction model, we use the mean square error. The mean squared error (MSE) is a common evaluation metric for regression models, measuring the average squared difference between the estimated values and the actual value. The MSE is defined as:

$$\mathrm{M}\mathrm{S}\mathrm{E}=\frac{1}{T}\sum _{i=1}^T{(P_i-{\hat{P}}_i)}^2$$where T denotes the total number of days in the forecast. $P_i$ represents the actual stock price on day i. $\widehat{P_i}$ represents the stock price predicted by the model on day i.

Fine-tuning

Throughout the 10 epochs of fine-tuning, we closely monitored the learning rate and loss metrics, which are illustrated in Fig. 5. The initial learning rate was set to 5e−5 and gradually decreased as the epochs progressed. At the same time, the loss metric consistently declined, indicating an improvement in the model’s ability to classify sentiment over time. To evaluate the efficacy of the fine-tuned model, we monitored performance metrics such as precision, recall, and F1 score, as shown in Fig. 6. These metrics were recorded regularly, providing valuable insights into the model’s stability and adaptability. We observed that the fine-tuned model consistently outperformed the non-fine-tuned baseline, demonstrating significant improvements. In particular, the fine-tuned model showed notable enhancements in recall and F1 score, suggesting a more balanced approach to precision and generalizability in sentiment classification. A comparative analysis between the fine-tuned and non-fine-tuned models revealed substantial post-tuning improvements. These enhancements confirm our hypothesis that appropriately adjusted pre-trained models can effectively capture domain-specific sentiment nuances essential for financial analysis applications.

Correlation between sentiment and stock

The word cloud shown in Fig. 7 is a visual representation of the frequency and importance of certain terms used in Twitter discussions related to TSLA. The keywords that appear with greater prominence, such as ‘stock’, ‘market’, ‘buy’, ‘earnings’, ‘Elon Musk’, and ‘TSLA’ suggest that these topics are highly discussed among Twitter users in the context of Tesla. The prominence of these terms indicates the public interest and sentiment surrounding Tesla as a company and as an investment option.

Figures 8 and 9 display the average positive sentiment trend of TSLA and APPLE stocks respectively, overlaid on their stock prices over a specific period. A notable observation is that whenever there is an increase in positive sentiment on Twitter, there appears to be a corresponding uptick in the stock price. This trend is particularly evident in scenarios where positive sentiment spikes, suggesting that investor optimism, as reflected in Twitter discourse, may have a concurrent or predictive relationship with stock performance.

Conversely, Fig. 10 juxtaposes the stock price against the average negative sentiment trend. Here, an inverse relationship appears to be evident, where peaks in negative sentiment align with dips in the stock price. This potentially indicates that negative public perception could contribute to or signal a forthcoming decrease in the stock’s value.

We have expanded our analysis by examining Pearson’s correlation coefficient (r) and Consistency Percentage as metrics to evaluate the relationship between positive sentiment on Twitter and the stock market performance across various stocks. The results have been summarized in Tables 2 and 3. These tables present a comprehensive overview of tweet volumes, consistency of positive sentiment, and their respective correlation coefficients with stock performance for multiple companies over different periods. The ‘tweet’ column represents the volume of tweets related to each stock, which indicates the level of social media engagement and discussion intensity about the company. The ‘Consistency Percentage’ reflects the proportion of consistent positive sentiment over time. Higher percentages suggest a more uniformly positive public opinion toward the company, while lower percentages indicate greater sentiment fluctuation. This metric could be particularly useful for investors seeking to understand the stability of public perception around a stock. The Pearson’s correlation coefficient (r) values range from positive to negative, suggesting varied strengths and directions of the linear relationship between stock performance and sentiment. A positive r value indicates that as positive sentiment increases, stock performance tends to increase, while a negative r value suggests that higher positive sentiment is associated with a decrease in stock performance. We can find that the r-value is also relatively high when the number of Twitter comments is sufficiently high. For example, ‘TSLA’ has the highest number of comments, and the sentiment of its comments is highly correlated with the stock price.

Price forecasts

In Fig. 11, the predicted data closely follows the actual data, suggesting that the model has a reasonable level of predictive power. The model appears to capture the general trend of the stock prices, although some deviations are present, indicating areas where the model could be refined. Notably, the model seems to have some limitations in capturing sharp spikes or drops in the stock price, which could be due to abrupt market movements that are not immediately reflected in sentiment scores.

In the case of Tesla as shown in Fig. 12, the predicted data appears to deviate less from the actual data when compared to AAPL, possibly due to Tesla’s stock being subject to more abrupt price changes influenced by factors beyond public sentiment, such as executive decisions, technological advancements, and regulatory news.

However, both figures suggest that while positive sentiment is a valuable predictor, it should be one of multiple factors considered in a comprehensive stock price prediction model. Additional variables that could be integrated into future models include market trends, economic indicators, and company-specific news.