A sequential recommendation method using contrastive learning and Wasserstein self-attention mechanism

Contrastive learning

Contrastive learning is a unique unsupervised learning technique that aims to derive data representations by maximizing the similarity between related samples while minimizing the similarity between unrelated samples. Therefore, the crucial aspect of contrastive learning lies in establishing rules for generating positive and negative samples with high flexibility and customization. The basic framework of contrastive learning is showed in Fig. 1.

The input comprises three parts: positive pare $I^{+}$, anchor $I^G$, and negative pare $I^{-}$. The encoder typically consists of two sections: the backbone and the head, the head is crucial in boosting the model’s performance, while the backbone is typically an encoder such as ResNet-50 or Transformer, utilized for feature extraction. The loss function can be computed as Eq. (1).

(1) $$L_{(q,k^{+},\{k^{-}\})}=-\log \left(\frac{\exp \left(\frac{q{k}^{+}}{\tau }\right)}{\exp \left(\frac{q{k}^{+}}{\tau }\right)+{\sum }_{k^{-}}\exp \left(\frac{q{k}^{-}}{\tau }\right)}\right).$$

Here q represents a query representation, $k^{+}$ denotes the representation of positive samples, and $\{k^{-}\}$ represents representation of the negative samples. $\tau$ represents a temperature hyperparameter.

BYOL is a self-supervised approach for image learning representations (Grill et al., 2020). It utilized online and target networks for interactive and collaborative learn. The online network is trained on augmented views of the image to predict the representation of the image seen by different augmentations in the target networks. InforNCE is a discriminative approach to contrastive learning that utilized contrast prediction coding (van den Oord, Li & Vinyals, 2018). It focused on extracting valuable representations by leveraging probabilistic contrast loss to encourage the capture of informative spatial data relevant for predicting future samples. Negative sampling was also utilized in this method to facilitate model training. SimCLR introduced a learnable transformation between representations and contrastive loss functions to enhance the quality of learned visual representations (Chen et al., 2020). The framework aimed to improve the effectiveness of learned representations through contrastive losses. SwAV leveraged contrastive methods for data clustering. It incorporated a swapped prediction mechanism where the code of one view was predicted based on the representation of another view (Caron et al., 2020). This approach enhanced representation learning through contrastive techniques. MoCo is a method designed for unsupervised visual representation learning. MoCo had demonstrated performance comparable to supervised representation learning in various visual tasks (He et al., 2020a). The approach incorporated momentum contrast to enhance the quality of learned visual representations in an unsupervised setting.

The recommendation system requires full use of the user’s behavior sequence to gain insight into the user’s behavior and potential interests. Transformer is used to fully explore the user’s behavior sequence and realize the modeling of the user’s behavior sequence. For example, models such as CBiT use multi-head attention in the Transformer layer to learn the correlation between the target item and the items in the user’s behavior sequence (Du et al., 2022). At the same time, the recommendation system needs to consider many attributes, and the graph representation can represent non-Euclidean data. The edges and nodes of the graph can represent rich attribute information, and the graph neural network can continuously aggregate and update features to obtain the information of high-order neighbors, thereby obtaining more accurate recommendation services. LightGCN learns user and item embeddings by linearly propagating them on the user-item interaction graph, and uses the weighted sum of the embeddings learned at all layers as the final embedding (He et al., 2020b). Although GNN can capture high-order correlation information, it also has problems such as over-smoothing.

Metric learning and distribution representations

Metric learning is a technique relies on a distance metric that aims to determine the similarity or dissimilarity between objects. Its goal is to bring similar objects closer together while pushing dissimilar objects further apart. Common examples of distances include the Euclidean distance, the Mahalanobis distance (McLachlan, 1999) and the graph distance (Gao et al., 2010). Distances and dot product are distinguished by their metric properties, with distances typically satisfying triangular inequalities. The triangular inequality serves as an inductive bias for distances and proves beneficial in dealing with data sparsity issues. Early works on metric learning in recommendation systems include CML (Hsieh et al., 2017), which utilized hinge loss to minimize the L2 distance between a user embedding and an interaction term. LRML expanded on CML by recognizing its geometric constraints and enhancing distance computation through the introduction of latent relations as translation vectors (Tay, Anh Tuan & Hui, 2018). TransRec drew inspiration from knowledge embedding and introduced translation vectors for sequential recommendations (He, Kang & McAuley, 2017). SML is a metric learning recommendation method that extended CML by incorporating additional item-centered metrics and adaptive margins (Li et al., 2020). Distributional representations for objects (e.g., words, nodes, items) had received extensive attention in research (Sun et al., 2018; Vilnis & McCallum, 2014). Distributional representations introduce uncertainty and offer more flexibility than fixed embeddings. DVNE employed Gaussian distributions as node embeddings and proposed a deep variational model for propagating high-order neighboring information (Zhu et al., 2018). TIGER represented words as Gaussian distributions and introduced Gaussian attention to improve the modeling of word entailment relationships (Vilnis & McCallum, 2014; Qian et al., 2021). DDN represented users and items using Gaussian distributions and learned mean and covariance embeddings via a neural network (Zheng et al., 2019). DT4SR used distributions to represent items and learned mean and covariance via separate Transformers (Fan et al., 2021). The Wasserstein distance measured the distance between discrete and continuous distributions, enabling continuous transformation from one distribution to another while preserving distributional features. Fan et al. (2022) designed a Wasserstein self-attention module to capture positional relationships between items within a sequence, incorporating uncertainty into model training effectively. During the sampling process, the Wasserstein reservoir operates on sessions with higher Wasserstein distance in recommendation results, leading to lower recommendation likelihood for sessions with higher Wasserstein distance.

Problem definition

Given a set of users U and items V along with their interactions, we can represent user interaction sequence as $S^u=[v_1^u,v_2^u,\dots ,v_{|S^u|}^u]$, where $v_i^u\in V$ denotes the $i-th$ item interaction sequence, and $S^u$ represents the user’s interaction sequence. The objective of SR is to recommend a list of top N items as the potential the next item in the sequence. Specifically, we aim to predict $p(v_{(|S^u|+1)}^{(u)}=v_{|}{S}^u)$, which calculates the probability that the user $u$ will interact with item $v$ at the next timestamp $|S^u|+1$.

Self-attention for recommendation

The core component of the proposed model is the self-attention mechanism as a sequence encoder. Firstly, when provided with the users’ sequence of operations $S^u$ and the maximum sequence length $n$, we truncate the sequence by removing the earliest item if $|S^u|>n$ or we pad it with zeros to create a sequence of fixed length $\mathbf{s}=(s_1,s_2,\dots ,s_n)$. Secondly, we define the item embedding matrix $\mathbf{M}\in {\mathbb{R}}^{|V|\times d}$, where $d$ represents the number of dimensions. Lastly, we incorporate the trainable position embedding $\mathbf{P}\in {\mathbb{R}}^{n\times d}$ into the sequence embedding matrix, as illustrated in Eq. (2):

(2) $${\hat{E}}_{S^u}=\left[m_{s_1}+p_{S_1},m_{s_2}+p_{S_2},\dots ,m_{s_n}+p_{s_n}\right].$$

Specifically, self-attention is used to compute the dot product between items in a sequence as a means of determining the correlation between them, as shown in Eq. (3):

(3) $$SA(Q,K,V)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{Q{K}^T}{\sqrt{d}}\right)V$$where $Q={\hat{E}}_{S^u}{W}^Q$, $K={\hat{E}}_{S^u}{W}^K$, and $V={\hat{E}}_{S^u}{W}^V$. Since Q and K use the same sequence of inputs, the scaled dot product component can learn potential correlations between items.

Stochastic embedding layer

We depict items as distributions and incorporate uncertainty into the item embedding. The various deterministic vectors symbolize the items envisioned as random distributions, enabling a broader coverage of space and inclusion of more collaborating neighbors. We utilize a multidimensional elliptical Gaussian distribution to portray the items, with the mean and covariance vectors governing the distribution’s elliptical shape. The covariance factor introduces potential uncertainty into the items being represented. We establish a mean embedding vector $M^{\mu }\in {\mathbb{R}}^{|V|\times d}$ and a covariance embedding vector $M^{\mathrm{\Sigma }}\in {\mathbb{R}}^{|V|\times d}$ for all items. Given that the mean and covariance convey distinct signals, we introduce positional embeddings $p^{\mu }\in {\mathbb{R}}^{n\times d}$ and $p^{\mathrm{\Sigma }}\in {\mathbb{R}}^{n\times d}$ for the mean and covariance, respectively. Consequently, for user $u$, the mean and covariance sequential embeddings can be represented as indicated in Eqs. (4) and (5).

(4) $${\hat{E}}_{s^u}^{\mu }=\left[{\hat{E}}_{s_1}^{\mu },{\hat{E}}_{s_2}^{\mu },\dots ,{\hat{E}}_{s_n}^{\mu }\right]=\left[m_{s_1}^{\mu }+P_{s_1}^{\mu },m_{s_2}^{\mu }+P_{s_2}^{\mu },\dots ,m_{s_n}^{\mu }+P_{s_n}^{\mu }\right]$$

(5) $${\hat{E}}_{s^u}^{\mathrm{\Sigma }}=\left[{\hat{E}}_{s_1}^{\mathrm{\Sigma }},{\hat{E}}_{s_2}^{\mathrm{\Sigma }},\dots ,{\hat{E}}_{s_n}^{\mathrm{\Sigma }}\right]=\left[m_{s_1}^{\mathrm{\Sigma }}+P_{s_1}^{\mathrm{\Sigma }},m_{s_2}^{\mathrm{\Sigma }}+P_{s_2}^{\mathrm{\Sigma }},\dots ,m_{s_n}^{\mathrm{\Sigma }}+P_{s_n}^{\mathrm{\Sigma }}\right].$$

As an illustration, for the sequence’s initial term $s_1$, its stochastic embedding is a Gaussian distribution of $d$-dimensional $N=({\mu }_{s_1},{\sum }_{s_1})$, where ${\mu }_{s_1}={\hat{E}}_{s_1}^{\mu }$ and ${\sum }_{s_1}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}({\hat{E}}_{s_1}^{\mathrm{\Sigma }})\in {\mathbb{R}}^{d\times d}$.

Wasserstein self-attention layer

Challenges persist in modeling the dynamics of sequence information via stochastic embeddings. One such challenge involves efficiently depicting items with distributions while adhering to triangular inequalities. Another obstacle pertains to aggregating sequence signals effectively to derive suitable sequence representations. To tackle these hurdles, CLWSR employes the Wasserstein distance to gauge the variability among items and introduces a novel Wasserstein self-attention layer. This layer characterizes attention weights as the Wasserstein distance between items and utilizes the linear summation trait of the Gaussian distribution to consolidate past items for acquiring a sequence representation.

Transformer layer

Each Transformer block includes a multi-head self-attention module and a feed-forward network. The multi-head self-attention module for efficient access to information in various subspaces at different positions. The calculation method is shown in Eq. (10).

(10) $$z_{s_t}^{\ast }=\frac{Q_k{K}_t^T}{\sqrt{d}}.$$where $Q=(z_{s_t}^{\ast }W{)}^Q$, $K=(z_{s_t}^{\ast }W{)}^K$, $V=(z_{s_t}^{\ast }W{)}^V$, and * can be either $\mu$ or $\mathrm{\Sigma }$.

As multi-head self-attention primarily relies on linear projections, incorporating a feed-forward network post the attention layer aids in capturing non-linear features. The calculation process is expressed as Eqs. (11) and (12).

(11) $$FF{N}^{\mu }(Z_{s_t}^{\mu })=\mathrm{E}\mathrm{L}\mathrm{U}(Z_{s_t}^{\mu }{W}_1^{\mu }+b_1^{\mu })W_2^{\mu }+b_2^{\mu }.$$ (12) $$FF{N}^{\mathrm{\Sigma }}(Z_{s_t}^{\mathrm{\Sigma }})=\mathrm{E}\mathrm{L}\mathrm{U}(Z_{s_t}^{\mathrm{\Sigma }}{W}_1^{\mathrm{\Sigma }}+b_1^{\mathrm{\Sigma }})W_2^{\mathrm{\Sigma }}+b_2^{\mathrm{\Sigma }}.$$where $W_1^{\ast }\in {\mathbb{R}}^{d\times d}$, $W_2^{\ast }\in {\mathbb{R}}^{d\times d}$, $b_1^{\ast }\in {\mathbb{R}}^d$, and $b_2^{\ast }\in {\mathbb{R}}^d$ are learnable parameters, and * can be $\mu$ or $\mathrm{\Sigma }$. Given the numerical stability benefit of the Exponential Linear Unit (ELU), we opt for ELU over Rectified Linear Unit (ReLU). Additionally, we incorporate other components like residual connections, normalization layer, and residual layers. The output of the layer can be expressed as:

(13) $$Z_{s_t}^{\mu }=z_{s_t}^{\mu }+\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}(FF{N}^{\mu }(\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}(z_{s_t}^{\mu })))$$ (14) $$Z_{s_t}^{\mathrm{\Sigma }}=\mathrm{E}\mathrm{L}\mathrm{U}(z_{s_t}^{\mathrm{\Sigma }}+\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}\left(FF{N}^{\mathrm{\Sigma }}\left(\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\left(z_{s_t}^{\mathrm{\Sigma }}\right)\right)\right))+1.$$

The proposed model utilizes the combination of ELU activation and covariance embedding to ensure the positive definite nature of the covariance.

Prediction layer

Considering the final output of any hidden representation $z_t$ at position $t$, where the mean embedding and covariance embedding at each position are summed to obtain $z_t$, we adopt a straightforward linear layer to transform $z_t$ into a probability distribution over candidate items:

(15) $$P(v)=w^p{z}_t+b^p.$$

In Eq. (15), $w^p\in {\mathbb{R}}^{|V|\times d}$ is the weight matrix, and $b^p\in {\mathbb{R}}^{|V|}$ is the bias term for the prediction layer.

Based on our practical observations, we do not use feed forward networks with item embedding matrices (Sun et al., 2019), but instead choose to use basic linear layers. We found that employing a prediction layer with a shared item embedding matrix can undermine the contrastive learning task, which depends on the shared item embedding matrix to compute item similarities. In addition to reducing computational overhead, the prediction layer without the item embedding matrix also decouples the dependency between the cloze task and the contrastive learning task, making the two tasks independent of each other.

Learning with the cloze task

The traditional BPR loss function is used to optimize the ranking of candidate items that users are interested in. It is trained by maximizing the rating difference between positive and negative samples, and is defined as shown in Eq. (16). However, this method is prone to overfitting or local optimization, resulting in uneven distribution of positive and negative samples, or too small interval between positive and negative samples.

(16) $$L=-\sum _{(u,i,j)\in D}\log \sigma ({\hat{r}}_{ui}-{\hat{r}}_{uj})$$where $(u,i,j)$ denotes a triad where $u$ is the user, $i$ is a positive sample (items that the user likes), and $j$ is a negative sample (items that the user does not like). ${\hat{r}}_{ui}$ is the predicted rating of item $i$ by user $u$. ${\hat{r}}_{uj}$ is the predicted rating of item $j$ by user $u$. $\sigma$ is the sigmoid function used to map the rating difference between $(0,1)$.

In this article, we generate multiple positive samples by introducing the cloze task, and in this way extend the sample generation method of traditional BPR. Unlike the traditional approach of considering only a pair of positive and negative samples for training, the new approach allows a user to learn by comparing and contrasting with multiple items, improving the model’s learning ability. In addition to this, dropout mask is a regularization technique commonly used in neural network training to avoid overfitting by randomly dropping some neurons. Here, dropout mask is not only used for neural network training but also used to generate multiple positive samples. This allows the model to generate multiple sample pairs for comparative learning during training, thus enhancing the model’s learning of potential preferences and features.

In summary, to train the Transformer, the cloze task is brought in. For each iteration step, given the sequence $s_u$, we use different random seeds to generate $m$ masking sequences $s_u^1,s_u^2,\dots ,s_u^m$. In each masking sequence $s_u^j$ ( $1\le j\le m$), the proportion $\rho$ of all items in the sequence $s_u$ is randomly replaced with a masking marker (mask), and the position index of the masked item is denoted as $I_u^j$. The model requires reconstruction of the masked items. Based on top of the BPR loss function formulation, the training target of the loss function with the cloze task is defined as Eq. (17):

(17) $$L_{\mathrm{m}\mathrm{a}\mathrm{i}\mathrm{n}}=-\sum _{j=1}^m\sum _{t\in I_u^j}\left[\log \sigma (P(v_t\mid s_u^j))+\sum _{v_t^{-}\notin s_u}\log 1-\sigma (P(v_t^{-}\mid s_u^j))\right].$$

Here, $\sigma$ represents the sigmoid function, and the probability $P(\cdot )$ is defined as Eq. (15). Each ground truth item $v_t$ is paired with a randomly sampled negative item $v_t^{-}$. Note that in computing the loss function for the cloze task, we only consider masked items.

Multi-pair contrastive learning

The purpose of contrastive learning is to bring positive samples closer to each other while separating negative samples from positive ones. Normally, given a batch of sequences $\{s_u{\}}_{u=1}^N$ with batch size N, a pair of hidden representations $Z_u^x$ and $Z_u^y$ stemming from the same original sequence $s_u$ are brought together as a pair of positive samples while the other $2(N-1)$ hidden representations from the same batch are considered negative samples (Chen et al., 2020). We define the contrastive learning loss for one pair based on InfoNCE (van den Oord, Li & Vinyals, 2018) as Eq. (18):

(18) $$l(Z_u^x,Z_u^y)=-\log \frac{e^{⟨Z_u^x,Z_u^y⟩/\tau }}{e^{⟨Z_u^x,Z_u^y⟩/\tau }+{\sum }_{k=1,k\ne u}^N{\sum }_{c\in \{x,y\}}{e}^{⟨Z_u^x,Z_k^c⟩/\tau }}.$$where $\tau$ is a temperature hyper-parameter, and $x$ and $y$ denote the index of two different mask sequences, with $1\le x,y\le m$. The cosine similarity function $⟨{\varphi }_1,{\varphi }_2⟩=\frac{{\varphi }_1^T\cdot {\varphi }_2}{||{\varphi }_1||||{\varphi }_2||}$ is used to calculate the similarity between two hidden representations.

Datasets

We conduct experiments on four benchmark datasets: Beauty, Toys, MovieLens-1M, and MovieLens-100M. The Amazon dataset (McAuley et al., 2015) contains reviews of products from various domains with relatively short sequence lengths. We choose Beauty and Toys as two distinct datasets derived from the Amazon dataset in our experiments. MovieLens-1M (ML-1M) and MovieLens-100M (ML-100M) comprise user ratings of movies with considerably long sequences (Harper & Konstan, 2015). All interactions are treated as implicit feedback. We eliminate duplicate interactions and compile user sequences by organizing each user’s interactions chronologically. Following the data filtering techniques outlined in previous studies (Liu et al., 2021; Qiu et al., 2022; Sun et al., 2019), we exclude users with fewer than five interactions and items associated with fewer than five users. Our approach employs a leave-one-out evaluation setup: the last item is reserved for testing, the penultimate item for validation, and the remaining items for training. The statistical details of the preprocessed datasets are presented in Table 1.

As can be seen from Table 1, these four datasets include different purposes, the data is extremely sparse, and the average length and data size are also significantly different, especially ML-100M, whose actions scale reaches 33,000,000, and the data volume is extremely large.

Metrics

For a fair comparison, we rank the predictions for the entire item set (Krichene & Rendle, 2020). We report top-K hit rate (HR@K) and NDCG@K as metrics.

(19) $$HR@K=\frac{1}{N}\sum _{i=1}^N\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}(i)$$where N is the total number of users. $\mathrm{h}\mathrm{i}\mathrm{t}\mathrm{s}(i)$ represents whether the value visited by the $i$-th user is in the recommendation list. It is 1 if yes, 0 otherwise.

(20) $$NDCG@K=\frac{1}{N}\sum _{j=1}^N\frac{1}{{\log }_2(p_i+1)}$$where $p_i$ is the real visit value of the $i$-th user in the position of the recommendation list. If the value does not exist in the recommendation list, then $p_i\to \mathrm{\infty }$.

Hyperparameter settings

The experimental setup utilizes the PyTorch framework, featuring two Transformer blocks with two attention heads, a hidden dimension, and a batch size of 256 each. The mask ratio $\rho$ is configured at 0.15, following the recommendation from BERT. We employ the Adam optimizer (Diederik, 2014) with a learning rate of 0.001, ${\beta }_1=0.9$ and ${\beta }_2=0.999$, while incorporating exponential decay of the learning rate after every 100 epochs (Loshchilov, 2017). In the process of hyperparameter tuning, we explore the dropout ratio from 0.1 to 0.9, the factor $\lambda$ from 1 to 9, $\tau$ from 0.1 to 6, the count of positive samples M from 2 to 8, and $\alpha$ values $\{0.0001,0.0005,0.001,0.05,0.1\}$. The model is trained for 250 epochs, and checkpoints demonstrating the best NDCG@10 on the validation set are selected for testing.

Experimental results

The presented baselines used for comparison are as follows:

- CoSeRec (Liu et al., 2021): Enhances CL4SRec by integrating data augmentation techniques.

- CBiT (Fan et al., 2022): Utilizes a bidirectional Transformer to enhance performance in sequential recommendation tasks through contrastive learning.

- HGN (Ma, Kang & Liu, 2019): Combines BPR to get both long-term and short-term user preferences.

- LightGCN (He et al., 2020b): Includes GCN-neighborhood aggregation, a pivotal component for collaborative filtering.

- TGT (Xia et al., 2022): Presents a temporal graph Transformer recommendation framework to capture dynamic short-term and long-term user-item interaction patterns jointly.

- HAM (Peng et al., 2021): Generates sequential recommendations by considering long-term preferences of users, recent activities, and item synergies. Item representation is achieved through simple pooling, while synergy modeling involves element-wise product operations.

- MADGA (Wang et al., 2024): MADGA dynamically transforms subsequences into graphs to capture the evolving interdependencies. Uniquely, this GA approach involves explicit alignment of both nodes and edges, employing Wasserstein distance for nodes and Gromov-Wasserstein distance for edges.

- SwiMDiff (Tian et al., 2024): SwiMDiff employs a scene-wide matching approach that effectively recalibrates labels to recognize data from the same scene as false negatives. This adjustment makes CL more applicable to the nuances of remote sensing.

CLWSR’s performance is evaluated against these baseline models, and the experimental results are summarized in Table 2.

From the analysis presented in Table 2, the CLWSR model performs relatively stably on all four datasets and outperforms other models overall, especially on the ML-1M and ML-100M dataset, where they perform better that other baseline models. This success can be attributed to the longer average sequence length of this database compared to other datasets. The Wasserstein self-attention mechanism implemented in the model effectively captures the dependencies between various positions in the extended session sequences. In addition, leveraging the cloze task for Transformer training contributes to improve the performance of the model.

However, it can be seen on the Beauty dataset that it slightly lags behind HAM at HR@5 and MADGA at NDCG@5 and NDCG@10, respectively. From the Toys dataset, it can be seen that it lags behind MADGA at NDCG@5 and SwiMDiff at NDCG@20, respectively. The main reason for the lagging behind is due to the fact that the three models have stronger adaptations on the shorter-length sequences, while our model is more adapted to longer length sequences. The performance enhancement observed in CLWSR model can be attributed to the synergistic influence of three key components: distributed embedding, Wasserstein self-attention, and the advantages derived from multi-pair comparison learning. The variation curves of NDCG@10 and HR@10 on four different datasets are shown in Figs. 4–7. We can see that the use of distributed embedding, Wasserstein self-attention and multi-pair comparison learning in CLWSR significantly improves performance on all four datasets.

Model complexity analysis

The computational complexity of CLWSR is dominated by the randomized mask operation, the Wasserstein self-attention layer, the feedforward network, Transformer layer, and contrastive learning module. The Wasserstein self-attention defined in Eq. (7) can be converted to using batch matrix multiplications. The second term in Eq. (7) can be transformed as a calculation of Euclidean norm as Eq. (21):

(21) $$\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}\left(\sum _{S_t}+\sum _{S_k}-2{\left(\sum _{S_k}^{1/2}\sum _{S_t}\sum _{S_k}^{1/2}\right)}^{1/2}\right)={\left||\sum _{S_t}^{1/2}-\sum _{S_k}^{1/2}|\right|}_F^2$$where $||\cdot |{|}_F^2$ is the Frobenius norm, which can be calculated by matrix multiplications. Also, since ${\sum }_{S_t}$ and ${\sum }_{S_k}$ are both diagonal matrices, we can further reduce the computational complexity to: $\frac{nd}{2}+\frac{n^{2}d}{2}+2n^2$, The Euclidean norm of the mean embeddings part in Eq. (7) can also be calculated by matrix multiplications with the same time complexity. Therefore, the overall time complexity of the Wasserstein self-attention is: $O(nd+n^{2}d+4n^2)$. The random cloze task is a mechanism for generating positive and negative samples by randomly masking part of the sequence. Assuming that each sample generates P negative samples, the overall complexity of the random cloze task is $O(B\cdot P\cdot n)$, where B is the batch size. In contrastive learning, the main computational complexity comes from the generation of positive and negative samples, the similarity calculation, and the loss function calculation. Therefore, the general complexity of contrastive learning is $O(B\cdot P\cdot n\cdot d+B\cdot P)$. By also considering the feed-forward networks, we obtain the final asymptotic computational complexity as $O(nd+n^{2}d+4n^2+\frac{n{d}^2}{2}+B\cdot P\cdot n\cdot (1+d)+B\cdot P)$. The computational complexity of traditional self-attention is $O(n^{2}d+n{d}^2)$. Note that both complexities are typically dominated by the $O(n^{2}d)$ term as $d$ is typically much larger than 4. This indicates that CLWSR has asymptotic time complexity similar to general conventional models.

We use the same GPU to measure the average running time per epoch. We compare the running time of the proposed model with several benchmarks. As shown in Table 3, CLWSR achieves the best results on Beauty, ML-1M, and ML-100M. The running time on the Toys datasets is moderate. Further analysis of the dataset in Table 2 shows that CLWSR performs better when operating on datasets with longer average sequence lengths, as Beauty, ML-1M, and ML-100M have relatively longer average sequences, so they effectively capture dependency relationships at various positions within extended sequences.

Ablation studies

We perform ablation studies of augmentation strategies that utilize stochastic embedding and Wasserstein self-attention mechanism to evaluate their effectiveness in this section.

We use a stochastic Gaussian distribution for each item, leveraging covariance to introduce sequence uncertainty into our model. The development of a novel Wasserstein self-attention model aids in defining item-item positional relationships within sequences, thereby effectively integrating uncertainty into the model training process. Moreover, the Wasserstein self-attention mechanism encourages collaborative and objective learning by adhering to the triangle inequality principle. Our model incorporates both stochastic embedding and Wasserstein self-attention mechanisms.

To evaluate these two strategies’ individual efficacy, we conducted ablation experiments on our approach, modifying only one of the mechanisms while maintaining optimal settings for the other hyperparameters. As depicted in Fig. 8, enabling contrastive learning surpasses a model without contrastive learning altogether. Notably, neither stochastic embedding nor the Wasserstein self-attention mechanism in isolation can achieve performance comparable to our combined hybrid strategy. This disparity can be traced to two key factors:

1. The amalgamation of these mechanisms introduces additional perturbations to the original sequence, yielding more challenging and higher-quality samples for comparative learning.

2. The collaborative and objective learning ingrained within CLWRS, where such principles aid cold items in identifying more collaborative neighbors, as popular items may exhibit correlations through triangular inequalities. This underlines the collaborative transferability signals’ requisite and superiority in the realm of sequential recommendation.

Discussion

The primary distinction between Wasserstein self-attention and the conventional attention mechanism lies in the calculation of attention weights. While the traditional attention mechanism employs dot product, Wasserstein self-attention utilizes negative 2-Wasserstein distances. As a result, the distribution of attention weights in Wasserstein self-attention tends to be more uniform compared to the traditional attention mechanism, which tends to focus on a select few items in the sequence. This difference can be attributed to the concept of collaborative transmissibility, which facilitates a closer association between neighboring items and introduces a wider range of collaborative neighbors in the modeling of item-item transitions.

Although the CLWSR model has achieved good performance on datasets of different sizes and its time cost is also low, the model still has some limitations, mainly manifested in the performance bottleneck on datasets with short sequence. For example, on the Beauty and Toys datasets, whose average sequence lengths are 8.9 and 8.6, respectively, the CLWSR model is not as good as the MADGA model in terms of NDCG. The reason is that the model we proposed combines the cloze task mask and dropout mask mechanisms to generate high-quality positive samples, but the sequence is short, and the cloze task mask and dropout mask mechanisms are difficult to play a role, which affects the performance of the model to a certain extent.