A social information sensitive model for conversational recommender systems

Encoding of conversation history

Encoding the conversation history C involves concatenating the utterances that constitute C in the same order in which they appear, following the approach used in previous works (Zhou et al., 2020a, 2022). Specifically, we merge the sequence of utterances $\{s_t{\}}_{t=1}^n$ into a single paragraph P. Next, we encode P with the help of a standard transformer encoder (Vaswani et al., 2017) as defined in Eq. (1) to obtain $h_c$ that is the contextual embedding of C.

(1) $$h_c=\mathrm{E}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}(P),$$

To weigh each word in $h_c$ according to its contribution, we apply self-attention (SA) given by Eq. (2) to get conversation history representation $e_c$.

(2) $$e_c=h_c\cdot \mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\mathbf{b}}^{\mathrm{\top }}\cdot \mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}(W_{sa}\cdot h_c)\right),$$where $W_{sa}$ and $b$ are the learnable parameters.

Encoding of user-user relationship graph

In the social relationship graph H, nodes correspond to users in the set U, and relationships between users are represented as edges in the form of $⟨u_i,u_j⟩$, where $u_i,u_j\in U$. To accurately capture and encode the significance of these relationships, we employ the GTO introduced in Shi et al. (2021) (see the ablation study section for further details). Specifically, we calculate a distinct vector for each user $u$ at layer ( $l+1$) as identified by Eqs. (3) and (4).

(3) $$n_u^{(l+1)}=W_1^{(l)}{n}_u^{(l)}+\sum _{u^{\prime }\in N(u)}{\alpha }_{u,u^{\prime }}^{(l)}{W}_2^{(l)}{n}_{u^{\prime }}^{(l)},$$

(4) $${\alpha }_{u,u^{\prime }}^{(l)}=\mathrm{softmax}\left(\frac{{\left(W_3^{(l)}{n}_u^{(l)}\right)}^{\mathrm{\top }}\left(W_4^{(l)}{n}_{u^{\prime }}^{(l)}\right)}{\sqrt{d}}\right),$$where ${\alpha }_{u,u\prime }^{(l)}$ denotes the attention coefficient. $\mathrm{softmax}$ is an activation function, $d$ represents the dimension of the feature vectors, $N(u)$ represents all neighbor nodes connected to the node $u$, the terms $W_1^{(l)}$, $W_2^{(l)}$, $W_3^{(l)}$, and $W_4^{(l)}$ represent learnable parameters, and $n_{u\prime }^{(l)}$ is the node $u^{\prime }$ representation from the preceding layer $l$.

Encoding of user-item interaction graph

In the user-item graph T, ratings provide important information about the strength of interaction between users and items. These interactions are denoted by triplets in the form $⟨u_i,r_k,i_j⟩$, where $u_i\in U$, $i_j\in I$, and $r\in R$, and $r$ represents a specific type of relationship drawn from the set $R=\{0,1,2,3,4,5\}$ that connects the nodes $u_i$ and $i_j$. To effectively represent the data in T, we utilize R-GCN (Schlichtkrull et al., 2018) (see the ablation study section for further details). Formally, the node $i$ within the graph at layer ( $l+1$) is represented as defined in Eq. (5).

(5) $$n_i^{(l+1)}=\sigma \left(\sum _{r\in R}\sum _{j\in N_r}\frac{1}{Z_i,r}{W}_r^{(l)}{n}_j^{(l)}+W^{(l)}{n}_i^{(l)}\right),$$where $\sigma$ denotes the activation function that introduces non-linearity into the model, $N_r$ represents all neighbor nodes connected via relation $r$, the normalization factor $\frac{1}{Z_i,r}$ adjusts the contributions from different nodes to ensure a balanced integration of information, the terms $W^{(l)}$ and $W_r^{(l)}$ represent learnable parameters, and $n_j^{(l)}$ is the representation of node $j$ at layer $l$.

Encoding of social information

To obtain the encoded representation of social information $e_s$, we aggregate the user-item interaction graph T and the user-user relationship graph H; the embeddings for all nodes are extracted from the topmost layers of these graphs. From graph T, we define X as the set of all node embeddings, where $M\in X$ represents the embeddings of user nodes. Similarly, let O be the set of all node embeddings in H.

To analyze the influence of social information on user preferences, we focus on each item $i_j$ in C. For each item, we identify the embeddings of all associated users in $B(j)$ as $M_u^j$ for T, where $M_u^j\in M$, and $O_u^j$ for H, where $O_u^j\in O$.

Formally, the user interaction profile ${\mathrm{P}}_{\mathrm{j}}$ for each $i_j$ is defined as shown in Eq. (6).

(6) $${\mathrm{P}}_{\mathrm{j}}=\mathrm{M}\mathrm{P}(M_u^j),$$where $\mathrm{M}\mathrm{P}$ refers to mean pooling. This operation ensures equal contribution from all users to capture the overall characteristics the involved embeddings.

Let the user social profile ${\mathrm{S}}_{\mathrm{j}}$ for each $i_j$ be defined as given by Eq. (7).

(7) $${\mathrm{S}}_{\mathrm{j}}=\mathrm{M}\mathrm{P}(O_u^j)$$

To obtain the enhanced user representation ${\mathrm{E}}_{\mathrm{j}}$ for each $i_j$ we apply Eq. (8) to stack ${\mathrm{P}}_{\mathrm{j}}$ and ${\mathrm{S}}_{\mathrm{j}}$.

(8) $${\mathrm{E}}_{\mathrm{j}}={\mathrm{P}}_{\mathrm{j}}\oplus {\mathrm{S}}_{\mathrm{j}}$$

For all items $I_c$ in C, to obtain the set $\mathrm{E}$ we aggregate the enhanced user representations as expressed in Eq. (9).

(9) $$\mathrm{E}=\{{\mathrm{E}}_{\mathrm{j}}\mid i_j\in I_c\},$$

Subsequently, we perform SA as described in Eq. (2), on the set $\mathrm{E}$ to ensure that each enhanced user representation within $\mathrm{E}$ is accurately weighted. By adopting this technique, we generate the social information encoded representation $e_s$.

Fine tuning for recommendation task

To enhance the recommendation, we leverage the pre-trained model’s user-item interaction graph T, user-user relationship graph H, and the conversation history encoder. Additionally, to ensure that the recommendations are highly relevant to the users’ interests involved in the conversation, we have considered the representation of the users, and the conversation history up to the conversation turn. This approach allows us to achieve a more personalized recommendation and ensure social contextual awareness. Let Y include all the item embeddings for T, where $Y\in X$. Let $Y_R$ and $Y_I$ include the items embeddings interacted with by the recommender and the seeker users, respectively, where $Y_R,Y_I\in Y$. Let $u_R$ and $u_I$ be the embeddings of the recommender and seeker users from H, respectively. The representation of the recommender user ${\mathrm{e}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{r}}$, is formally defined as shown in Eq. (13).

(13) $${\mathrm{e}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{r}}=\mathrm{M}\mathrm{P}(\mathrm{M}\mathrm{P}(Y_R)\oplus u_R),$$

Similarly, the seeker user representation ${\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{e}\mathrm{k}\mathrm{e}\mathrm{r}}$ defined as shown in Eq. (14)

(14) $${\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{e}\mathrm{k}\mathrm{e}\mathrm{r}}=\mathrm{M}\mathrm{P}(\mathrm{M}\mathrm{P}(Y_I)\oplus u_I)$$

The ${\mathrm{e}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{r}}$ is fed into an encoder with Multi-Head Attention (MHA) cross-attention (CA) sub-layers. These sub-layers incorporate ${\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{e}\mathrm{k}\mathrm{e}\mathrm{r}}$ and $e_c$ to the encoded representation through CA as described in Hammad, Moretti & Nojiri (2024). A similarly structured decoder is used, with an additional CA sub-layer to integrate the original ${\mathrm{e}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{r}}$ and other data representations into each decoder layer during the decoding. Following Zhou et al. (2022), MHA is formally defined as shown in Eq. (15).

(15) $$\mathrm{M}\mathrm{H}\mathrm{A}(Q,K,V)=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(\mathrm{h}\mathrm{e}\mathrm{a}{\mathrm{d}}_1,\dots ,{\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}}_{\mathit{h}}){\mathbf{W}}_{\mathit{O}},$$where each head is defined as expressed in Eq. (16).

(16) $${\mathrm{h}\mathrm{e}\mathrm{a}\mathrm{d}}_i=\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}({\mathbf{Q}\mathbf{W}}_i^Q,{\mathbf{K}\mathbf{W}}_i^K,{\mathbf{V}\mathbf{W}}_i^V),$$where $\mathbf{Q}$, $\mathbf{K}$ and $\mathbf{V}$ are the query, key and value respectively, and ${\mathbf{W}}_i^Q$, ${\mathbf{W}}_i^K$,and ${\mathbf{W}}_i^V$ are the learnable weights.

The ${\mathrm{e}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{r}}$ is processed by the encoder, as shown in Eqs. (17)–(21):

(17) $$A_0^n=\mathrm{M}\mathrm{H}\mathrm{A}(E^{n-1},E^{n-1},E^{n-1}),$$

(18) $$A_1^n=\mathrm{M}\mathrm{H}\mathrm{A}(A_0^n,{\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{e}\mathrm{k}\mathrm{e}\mathrm{r}},{\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{e}\mathrm{k}\mathrm{e}\mathrm{r}}),$$

(19) $$A_2^n=\mathrm{M}\mathrm{H}\mathrm{A}(A_1^n,e_c,e_c),$$

(20) $$E^n=\mathrm{F}\mathrm{F}\mathrm{N}(A_2^n),$$

(21) $$\mathrm{F}\mathrm{F}\mathrm{N}(x)=max(0,x{W}_1+b_1)W_2+b_2,$$where $E^{n-1}$ is the encoded representation of ${\mathrm{e}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{r}}$ at the ( $n-1$)-th layer, $A_0^n$ is the representation after applying SA with $E^{n-1}$, $A_1^n$ is the representation after applying CA with ${\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{e}\mathrm{k}\mathrm{e}\mathrm{r}}$, $A_2^n$ is the representation after applying CA with $\mathrm{F}\mathrm{F}\mathrm{N}$ is a feedforward neural network for dimension mapping, and $E^n$ is the output of the encoder at the topmost layer.

Subsequently, the output of the encoder is processed by the decoder as shown in Eqs. (22)–(26):

(22) $$B_0^n=\mathrm{M}\mathrm{H}\mathrm{A}(R^{n-1},R^{n-1},R^{n-1}),$$

(23) $$B_1^n=\mathrm{M}\mathrm{H}\mathrm{A}(B_0^n,{\mathrm{e}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{r}},{\mathrm{e}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{r}}),$$

(24) $$B_2^n=\mathrm{M}\mathrm{H}\mathrm{A}(B_1^n,{\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{e}\mathrm{k}\mathrm{e}\mathrm{r}},{\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{e}\mathrm{k}\mathrm{e}\mathrm{r}}),$$

(25) $$B_3^n=\mathrm{M}\mathrm{H}\mathrm{A}(B_2^n,e_c,e_c),$$

(26) $$R^n=\mathrm{F}\mathrm{F}\mathrm{N}(B_3^n),$$where $R^{n-1}$ is the decoded representation of the encoder output $E^n$ at the previous layer $n-1$, $B_0^n$ is the representation after applying SA with $R^{n-1}$, $B_1^n$ is the representation after applying CA with ${\mathrm{e}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{r}}$, $B_2^n$ is the representation after applying CA with ${\mathrm{e}}_{\mathrm{s}\mathrm{e}\mathrm{e}\mathrm{k}\mathrm{e}\mathrm{r}}$, $B_3^n$ is the representation after applying CA with $e_c$, and $R^n$ is the output of the decoder at the final layer $n$, which represents the probabilities of the recommended items after applying $\mathrm{softmax}$.

Let $P_{\mathrm{r}\mathrm{e}\mathrm{c}}(i)$ be the probability distribution for the recommended item at the index $i$ as defined in Eqs. (27) and (28).

(27) $$P_{\mathrm{r}\mathrm{e}\mathrm{c}}(i)=\mathrm{softmax}(R^n)[i],$$

(28) $$\sum P_{\mathrm{r}\mathrm{e}\mathrm{c}}(i)=1,$$

Lastly, to fine-tune the rich representations for the recommendation task, following Zhou et al. (2020a, 2022), we apply cross-entropy loss to the probabilities of the recommended items. In our case, we consider one recommended item per conversation as given by Eq. (29).

(29) $$\mathcal{L}\mathrm{r}\mathrm{e}\mathrm{c}=-\sum _{n=1}^N\sum _{i=1}^{IR}{y}_{ni}\cdot \log (P_{\mathrm{r}\mathrm{e}\mathrm{c}}^n(i)),$$where N is the number of conversations, IR represents the items being recommended, and $y_{ni}$ is the ground truth.

Fine tuning for conversational task

For the conversation task, all models within the pre-trained framework are utilized. The approach aligns with the methodology detailed in the previous section, with adjustments inspired by Zhou et al. (2020a, 2022) to fine-tune the pre-trained model for response generation. Specifically, an encoder, as described in Eq. (1) of a standard transformer (Vaswani et al., 2017), is used to encode the responses in training and inference. Additionally, in the decoder, different additional MHA sub-layers with CA sub-layers are used to incorporate the social information $e_s$, conversation history representation $e_c$, and the contextual embeddings $h_c$ from the pre-trained model during the decoding process as defined in Eqs. (30)–(34).

(30) $$C_0^n=\mathrm{M}\mathrm{H}\mathrm{A}(R^{n-1},R^{n-1},R^{n-1}),$$

(31) $$C_1^n=\mathrm{M}\mathrm{H}\mathrm{A}(C_0^n,e_s,e_s),$$

(32) $$C_2^n=\mathrm{M}\mathrm{H}\mathrm{A}(C_1^n,e_c,e_c),$$

(33) $$C_3^n=\mathrm{M}\mathrm{H}\mathrm{A}(C_2^n,h_c,h_c),$$

(34) $$R^n=\mathrm{F}\mathrm{F}\mathrm{N}(C_3^n),$$where $C_0^n$ is the representation after applying self-attention with the decoder output at the previous layer $R^{n-1}$, $C_1^n$ is the representation after applying cross-attention with $e_s$, $C_2^n$ is the representation after applying cross-attention with $e_c$, $C_3^n$ is the representation after applying cross-attention with $h_c$, and $R^n$ is the representation of the decoder output at the $n$-th layer.

The copying mechanism used in both Zhou et al. (2020a, 2022) is also adopted to copy useful information from user information through integrating enhanced user representation set $\mathrm{E}$ while generating responses to enhance performance and personalization. Formally, the probability of generating token $w_j$ given tokens $w_1$, …, $w_{j-1}$ is defined in Eq. (35).

(35) $$P(w_j|w_1,\dots ,w_{j-1})=P_{gen}(w_j|R^n)+P_{copy}(w_j|R^n,\mathrm{E}),$$where $P_{gen}$ is the generation probability over all the vocabulary given the decoder output at the final layer $R^n$, and $P_{copy}$ is the copy probability given $R^n$ and $\mathrm{E}$.

The loss function used for fine-tuning is the same as the one used in Zhou et al. (2022), where a weighted cross-entropy loss is used to improve the response generation as defined in Eqs. (36) and (37).

(36) $$\mathcal{L}\mathrm{g}\mathrm{e}\mathrm{n}=-\frac{1}{m}\sum _{j=1}^m\log ({\alpha }_{w_j}P(w_j|w_1,\dots ,w_{j-1})),$$

(37) $${\alpha }_{wj}=\{\begin{array}{cc}max(\gamma ,\frac{\beta }{f_{w_j}}),\hfill & \mathrm{i}\mathrm{f}{f}_{wj}\ge \beta \hfill \\ 1,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array},$$where $m$ is the total number of tokens in the generated response, ${\alpha }_{wj}$ is the weight of the token depending on its frequency, $f_{w_j}\in \{1,2,3,\dots \}$ is the word frequency, $\gamma$ is a random number in the range $[0,1]$ that balances the impact of word frequency during response generation, and $\beta \in \{1,2,3,\dots \}$ is the threshold of the word frequency.

Furthermore, we have used two decoding techniques. Greedy Search (Radford et al., 2019) is adopted to generate $w_j$ during training as defined in Eq. (38), while in the inference stage, we use Nucleus Sampling (Holtzman et al., 2020) to select $w_j$ as defined in Eqs. (39)–(42).

(38) $$w_j=\underset{w_j}{\mathrm{argmax}}{\alpha }_{w_j}P(w_j|w_1,\dots ,w_{j-1})),$$where $\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}{\mathrm{x}}_{w_j}$ is the argument of the maximum function.

(39) $$w_j\sim P^{\prime }(w_j\mid w_1,\dots ,w_{j-1}),$$

(40) $$P^{\prime }(w_j\mid w_1,\dots ,w_{j-1})=\{\begin{array}{cc}\frac{{\alpha }_{w_j}P(w_j\mid w_1,\dots ,w_{j-1})}{p\prime }\hfill & \mathrm{i}\mathrm{f}{w}_j\in V(p)\hfill \\ 0\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array},$$

(41) $$p^{\prime }=\sum _{w_j\in V(p)}{\alpha }_{w_j}P(w_j\mid w_1,\dots ,w_{j-1}),$$

(42) $$V(p)=\left\{w_j|\sum _{w_j\in V}{\alpha }_{w_j}P(w_j\mid w_1,\dots ,w_{j-1})\ge p\right\},$$where $p$ is a random number in the range $[0,1]$ that represents the cumulative probability threshold, V is the total vocabulary, $V(p)$ is the smallest set of tokens in which their cumulative probability is greater than or equal to $p$, $p^{\prime }$ is a rescaling factor that is the total probability of all tokens in $V(p)$ used to normalize the selected tokens’ probabilities, and $P^{\prime }(w_j\mid w_1,\dots ,w_{j-1})$ is the rescaled probability distribution used to sample $w_j$.

Datasets

The ReDial dataset (Li et al., 2018) and the INSPIRED dataset (Hayati et al., 2020). The ReDial dataset (Li et al., 2018) was created by Amazon Mechanical Turk (AMT) and serves as a conversational dataset for movie recommendations. It consists of 10,000 conversations with 182,150 utterances associated with 51,699 movies in the English language. The INSPIRED dataset (Hayati et al., 2020) is much smaller, consisting of 1,001 conversations with 35,811 utterances associated with 1,783 items. Both datasets are English CRS datasets. Each conversation involves a dialogue between two users, where one user acts as the recommender and the other acts as the seeker. The users might fluctuate their roles across conversations within the dataset, so mutual recommendations of movies take place. Table 1 shows the statistics of users for all datasets. Following the baselines, no sampling techniques are used for balancing user roles.

Our experiments require a user-item interaction graph and a user-user relationship graph for the datasets. However, the ReDial (Li et al., 2018) and the INSPIRED (Hayati et al., 2020) datasets only include conversational data. We leverage the available metadata in the ReDial dataset (Li et al., 2018) to infer reasonable interaction information. However, the INSPIRED dataset (Hayati et al., 2020) does not contain this metadata about each user’s mentioned movies and the details of each movie interaction, whether liked, suggested, or watched. To address this, we manually annotate each movie mentioned in conversations to add users’ movie interaction information to the INSPIRED dataset (Hayati et al., 2020). The metadata from both datasets are then used to generate ratings for both datasets separately. The process of annotating the dataset is described in detail in the next subsection.

We initiate the process by extracting an interaction matrix that records positive and negative interactions, specified by the liking or disliking of items mentioned by each user in the conversations. We form a social relationships matrix conditioned upon the criterion that conversing users demonstrate at least one shared interest, according to Hayati et al. (2020). In addition, as the recommender user influences the seekers who converse with them, we also establish friend relationships among the seekers who interacted with the same recommender under the same condition mentioned earlier. Therefore, we ensure that the connections formed are based on engagement and shared interests. Lastly, we utilize NGCF (Wang et al., 2019) to generate ratings for all items. The NGCF relies on the interaction information to predict ratings. These ratings contribute to creating the user-item interaction graph T.

We construct the social relationship graph H using GTO (Shi et al., 2021), where nodes represent users in the social relationships matrix. Similarly, we construct T using R-GCN (Schlichtkrull et al., 2018) by replacing the positive and negative interactions with the inferred ratings we obtained in the previous step, where nodes represent users and items, with directed edges from user nodes to item nodes.

Annotation process

Each dialogue in the ReDial dataset (Li et al., 2018) contains additional fields, such as a dictionary to map movie IDs to their names and dictionaries that associate movie IDs with labels indicating whether a movie is seen, liked, or suggested by the initiator or respondent. These labels include suggested (distinguishing who suggested the movie), seen (indicating whether the movie was watched), and liked (denoting if the user liked the movie). These labels are essential for the construction of the required graphs for SISSF. As the INSPIRED dataset (Hayati et al., 2020) initially does not contain these labels, we manually conduct the annotation process. Figure 4 shows the user interface we created, which is designed to display conversations with highlighted movie IDs coupled with a dynamic form that contains two dynamic sub-forms, where the annotators can add movies for the corresponding users.

Baselines

We evaluate our CRS based on recommendation and conversational tasks, using benchmark CRSs and stand-alone models as baselines.

• Popularity: The items in the corpus are ranked based on the frequency of recommendations they receive in the training set. This ranking is based on popularity.

• TextCNN (Kim, 2014): CCN extracts user preferences from text to recommend items.

• Transformer (Vaswani et al., 2017): Generates conversational responses using a transformer-based encoder-decoder model.

• KBRD (Chen et al., 2019): It uses the seq2seq model based on the transformer (Vaswani et al., 2017) and a general KG from DBpedia to improve the recommendation integration and the response generation components.

• KGSF (Zhou et al., 2020a): Enhances the CRS by adapting a way to enhance the semantic fusion between the external KGs and conversation history through mutual information maximization.

• ReDial (Li et al., 2018): Uses the seq2seq model based on HRED (Serban et al., 2016) for conversation generalization with a switch decoder (Gülçehre et al., 2016) to decide when to recommend or generate a token. The recommender is an autoencoder (Sedhain et al., 2015), and a sentiment analysis unit is used to check the user’s feedback. The model is proposed with the ReDial dataset (Li et al., 2018).

• INSPIRED (Hayati et al., 2020): Proposes the INSPIRED dataset (Hayati et al., 2020) for CRS encoded with sociable strategies to create a CRS that leverages social relationships.

• INSPIRED2 (Ahtsham & Dietmar, 2022): Proposes the INSPIRED2 dataset (Ahtsham & Dietmar, 2022) for CRS, which is an improved version of the INSPIRED dataset (Hayati et al., 2020).

• UniCRS (Wang et al., 2022): Utilize PL to guide PLMs for recommendation and conversation generations.

• C2-CRS (Zhou et al., 2022): Improves the semantic fusion by prompting a novel way to fuse many external data with the conversation history through CL (Jing et al., 2023).

• VRICR (Zhang et al., 2023): Uses variational Bayesian technique to infer the missing links among entities related to the subgraphs that are specific to a given conversation.

Popularity and TextCNN (Kim, 2014) serve as benchmarks exclusively for the recommendation task, while the Transformer (Vaswani et al., 2017) is only evaluated for the conversational task. The CRS models KBRD (Chen et al., 2019), KGSF (Zhou et al., 2020a), C2-CRS (Zhou et al., 2022), VRICR (Zhang et al., 2023), INSPIRED (Hayati et al., 2020), INSPIRED2 (Ahtsham & Dietmar, 2022), UniCRS (Wang et al., 2022), and ReDial (Li et al., 2018) are used to evaluate both recommendation and conversational tasks.

Since not all baselines are tested on every dataset, we grouped them by dataset for the SISSF evaluation. On the ReDial dataset (Li et al., 2018), we consider the following baselines: Popularity, TextCNN (Kim, 2014), Transformer (Vaswani et al., 2017), KBRD (Chen et al., 2019), KGSF (Zhou et al., 2020a), C2-CRS (Zhou et al., 2022), VRICR (Zhang et al., 2023), UniCRS (Wang et al., 2022), and ReDial (Li et al., 2018), while on the INSPIRED dataset (Hayati et al., 2020), the baselines include KBRD (Chen et al., 2019), KGSF (Zhou et al., 2020a), INSPIRED (Hayati et al., 2020), INSPIRED2 (Ahtsham & Dietmar, 2022), UniCRS (Wang et al., 2022), and ReDial (Li et al., 2018). Table 2 summarizes the details of various CRS models.

Evaluation metrics

We adopt different evaluation metrics tailored to the distinct nature of each task. For the recommendation task, according to the baseline methods chosen for this work, most of them primarily focus on Recall as the evaluation metric. To ensure consistency with these baselines, we opted to use Recall@K ( $k=1,10,50$). Similarly, for the conversational task, we apply human evaluation for the ReDial dataset (Li et al., 2018) and automatic evaluation for both datasets (Hayati et al., 2020; Li et al., 2018). During the automatic evaluation, we measure the diversity of generated utterances using Distinct $n$-gram ( $n=2,3,4$), while during the human evaluation, we used three annotators on a scale from 0 to 2 to score the generated responses in two aspects, namely Fluency and Informativeness. The average score is computed for all the annotators to get the final performance score.

Implementation details

We follow Algorithm 1 with PyTorch (https://pytorch.org) and WSDM2022-C2CRS (Zhou et al., 2022) (https://github.com/RUCAIBox/WSDM2022-C2CRS) to implement our approach. The embedding dimensionalities are set to 300 for token embeddings and 768 for the user dimensions. For the user-item interaction graph T and user-user relationship graph H, we set the layer number and the normalization constant to 1. We use the Adam optimizer (Kingma & Ba, 2015) with beta values of ( $0.9,0.999$), and batch size of 300. The learning rates for our model are carefully chosen for each model to ensure optimal performance. During the semantic fusion and fine-tuning of the recommendation task, we use a learning rate of 0.00001. For the conversational task, we start with 0.0001 and then resume learning with 0.00001 for the ReDial dataset (Li et al., 2018) and maintain a learning rate of 0.0001 for the INSPIRED dataset (Hayati et al., 2020). Regarding the CL, we set a temperature of 0.09 during semantic fusion and 0.13 during fine-tuning. For the transformer model, we set the number of layers and heads to 4 for both transformers used by both tasks. For the calculation of $\mathcal{L}\mathrm{g}\mathrm{e}\mathrm{n}$, we set $\gamma$ to 0.3 and $\beta$ to 100. We set the cumulative probability threshold $p$ to $0.95$ during the inference process. Following the section Applying CL to the Encoded Representations, the stage of aligning the data representation to minimize the loss is conducted during CL. In addition, during the fine-tuning stage, the section Fine Tuning the Rich Representation to CRS’s Tasks is considered to optimize the loss for each specific task. The project code and data are available at https://doi.org/10.5281/zenodo.15368530.

Automatic evaluation

Tables 5 and 6 present the results of the automatic evaluation of the conversational task on the ReDial (Li et al., 2018) and the INSPIRED (Hayati et al., 2020) datasets, respectively. For the ReDial (Li et al., 2018) dataset, the Transformer (Vaswani et al., 2017) performs poorly due to its slow dependency on the conversation history to generate results. ReDial (Li et al., 2018) performs better than the Transformer (Vaswani et al., 2017), which benefits from integrating the recommender component during response generation with the help of a switch mechanism. KBRD (Chen et al., 2019) outperforms ReDial (Li et al., 2018) on ReDial (Li et al., 2018) and INSPIRED (Hayati et al., 2020) datasets as it utilizes the KG to enrich the contextual information. KGSF (Zhou et al., 2020a) performs better than KBRD (Chen et al., 2019) due to semantic fusion to bridge the semantic gap between the KGs and conversational history, leading to richer contextual representation. UniCRS (Wang et al., 2022) surpasses KGSF (Zhou et al., 2020a) by incorporating fused information to prompt the PLMs. Zhou et al. (2020a) C2-CRS (Zhou et al., 2022) outperforms UniCRS (Wang et al., 2022) by adopting a new method of semantic fusion and involving more external data. VRICR (Zhang et al., 2023) achieves the highest results on all metrics among the baselines due to the more effective utilization of external data. Furthermore, our model outperforms all the baselines on all metrics on all datasets, primarily due to implementing an intuitive semantic fusion technique that integrates social information, providing additional context about user preferences and interests for more relevant responses. Additionally, utilizing the top-p sampling (Holtzman et al., 2020) technique enhances the diversity of response generation.

Human evaluation

The human evaluation results on conversational task on the ReDial dataset (Li et al., 2018) are presented in Table 7. First, VRICR (Zhang et al., 2023) and C2-CRS (Zhou et al., 2022) achieve the best results compared to all baselines. This is followed by UniCRS (Wang et al., 2022) that utilizes PL to integrate the fused KG with PLMs. Next, KGSF (Zhou et al., 2020a), where a KG-enhanced Transformer decoder injects the data from the rich representations generated by fusing conversational text and items via a knowledge graph. KBRD (Chen et al., 2019) achieved slightly better metrics results than ReDial (Li et al., 2018) by promoting low-frequency tokens by leveraging the knowledge graph. In contrast, ReDial (Li et al., 2018) used a pre-trained encoder, which exceeded the results of the Transformer (Vaswani et al., 2017). Finally, SISSF outperforms all the benchmarks on all metrics by leveraging CL to bridge the gap between conversational history and social information. Additionally, a Transformer decoder integrates the generated compact representation with a weighting mechanism to improve the informativeness of responses. This is enhanced by the top-p sampling technique (Holtzman et al., 2020), which has been proven to generate fluent, human-like responses, reflecting the significant improvement of our model’s fluency.

Hyperparameter analysis on recommendation task

The evaluation results of the hyperparameter analysis for all metrics of the recommendation task on the ReDial dataset (Li et al., 2018) are presented in Table 13. Generally, lower learning rates produce better results on all metrics for higher temperature settings. Specifically, a temperature of (0.13) gives the best recall, especially for lower learning rates. The performance decreases slightly for high learning rates (0.0001), with a more significant drop for even higher learning rates (0.001). In summary, lower learning rates combined with the highest temperature yield the best recall performance, while higher learning rates across all temperate configurations significantly decrease performance. Figure 5 contains heatmaps for each recall metric, which provides a comprehensive visual representation of the data in Table 13.

Hyperparameter analysis on conversational task

Table 14 shows the results of the hyperparameter analysis on the conversational task on the ReDial dataset (Li et al., 2018). Overall, higher learning rates tend to produce lower performance, which is improved gradually by decreasing the learning rate until the model reaches its best performance at (0.00001) for the range of values of temperature parameters. The lower temperatures also produce lower results than the highest value of (0.13). However, the values of all metrics are still comparatively high due to the usage of top-p sampling (Holtzman et al., 2020). Figure 6 provides heatmaps for each Distinct metric to visualize the data presented in Table 14.

Ablation study on recommendation task

The ablation study results on the recommendation task on the ReDial dataset (Li et al., 2018) are summarized in Table 15. The results show that the operation of semantic fusion by applying CL between conversational history and social information is essential to improve the model’s performance. Overall, the SISSF model outperforms SISSF w/o Semantic Fusion by a significant margin, which indicates the incorporation of semantic fusion leads to more accurate recommendations.

Ablation study on conversational task

The ablation study results on the recommendation task on the ReDial dataset (Li et al., 2018) are shown in Table 16, which proves that semantic fusion contributes to the diversity of the response generation and its informativeness. SISSF performs superior in both automatic and human evaluations compared to its variant. The automatic evaluation shows comparatively close results due to the use of top-p sampling (Holtzman et al., 2020). We also conducted a human evaluation to better demonstrate the performance differences between the model and its variant. Furthermore, Fig. 8 shows examples of the responses generated by both models; SISSF demonstrates its ability to recommend contextually relevant items compared to SISSF w/o Semantic Fusion. For example, SISSF recommends a scary movie, It (2017), which is relevant to the user’s query and a personalized recommendation, as this movie falls within the social circle of the seeker user. Conversely, SISSF without Semantic Fusion recommends a fiction movie that lacks personalization and contextual relevance.

Ablation study on the impact of semantic fusion on personalized recommendations

The primary purpose of this ablation study is to highlight the reliability of extracting social information from conversational history for personalized recommendations. To achieve this, first, we obtain the items that the users have not yet interacted with but are present within their social circle’s interests. Next, the recommendation task was evaluated on the ReDial dataset (Li et al., 2018) by testing whether the first item recommended by the model belongs to the set of those items for the seeker users. Table 17 indicates SISSF outperforms SISSF w/o Semantic Fusion, achieving a performance score of 0.1132 compared to 0.0154 in R@1. These findings demonstrate that extracting social information and applying semantic fusion to integrate it with conversational history is not merely a method of overfitting the dataset.

Ablation study on the impact of graph architectures and infer rating process on recommendation performance

In this ablation study, we investigated the use of different graph types to identify the best models for the user-user social graph H and the user-item interaction graph T on the ReDial dataset (Li et al., 2018). Table 18 shows that employing GTO (Shi et al., 2021) for H and R-GCN (Schlichtkrull et al., 2018) for T yields the best results. Moreover, using graph methodologies such as GTO (Shi et al., 2021), GCN (Kipf & Welling, 2017), or GAT (Veličković et al., 2018) for T did not contribute significantly to improving the outcome. This indicates leveraging NGCF (Wang et al., 2019) to infer ratings, which is used to weigh the relationships among nodes within R-GCN (Schlichtkrull et al., 2018), enhances the performance. These findings underscore the importance of incorporating inferred ratings with NGCF (Wang et al., 2019) to optimize the model’s performance. The reason behind the improvement of H with GTO (Shi et al., 2021) over other models is due to its ability to effectively weight relationships between nodes (Brody, Alon & Yahav, 2022).