Achieving optimal trade-off for student dropout prediction with multi-objective reinforcement learning

Markov decision process

Optimizing the accuracy-earliness trade-off issue in student dropout prediction task fundamentally presents an multi-objective optimization challenge (Jimenez et al., 2019). Reinforcement learning, commonly employed in such contexts, tackles this by enabling an agent to learn through trial and error within an unknown environment (Dulac-Arnold et al., 2020). This learning process is typically modeled as a Markov decision process (MDP) (Garcia & Rachelson, 2013), which is defined by the tuple $〈\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma 〉$, where $\mathcal{S}$, $\mathcal{A}$, $\mathcal{P}$, r, and γ represent state space, action space, transition distribution, reward and discount factor, respectively.

Scalar reward based RL

In researches addressing multi-objective trade-off issues with reinforcement learning, scalarization is a prevalent technique where multiple objectives are combined into a single one through static weights (Martinez et al., 2018; Martinez et al., 2020; Hartvigsen et al., 2019). This process can be formalized as transforming multiple objectives into a single optimization goal through a scalarization function (Zhang, Qi & Shi, 2023). The transformation facilitates the application of standard reinforcement learning (RL) algorithms for finding a policy aimed at maximizing an agent’s cumulative reward in a MDP. In this setting, the agent, defined by its policy π, chooses an action a in each state s as a = π(s), with the environment responding by providing a reward r = R(s, a) and transitioning to the next state $s^{\prime }=\mathcal{P}\left(s,a\right)$. The The interaction sequence $〈s,a,r,s^{\prime }〉$ continues to unfold, with the goal of maximizing the cumulative reward, guiding the agent toward an optimal policy or a terminal state.

The agent’s performance is quantified by the expected cumulative rewards, denoted by the action-value function, also known as the Q-function: (1)${\displaystyle Q_{\left(s,a\right)}={\mathbb{E}}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k}\mid s_t=s,a_t=a\right].}$

The Q-function indicates, for a given policy π, if selecting an action a in a particular state s is likely to have good repercussions in the following steps by getting large rewards or not.

The optimization process for standard value-based MDPs incorporates the utilization of Bellman’s optimality equation. This equation facilitates the decomposition of the action value into the immediate reward added to the discounted action value of the subsequent state. (2)${\displaystyle Q^{\ast }\left(s,a\right)=r\left(s,a\right)+\gamma {\mathbb{E}}_{s^{\prime }\sim \mathcal{P}\left(\cdot \mid s,a\right)}\underset{a^{\prime }\in \mathcal{A}}{max}{Q}^{\ast }\left(s^{\prime },a^{\prime }\right).}$

The loss function used for updating can be calculated by: (3)${\displaystyle L\left(\Theta \right)={\left(Q^{\ast }\left(s,a,{\Theta }^{-}\right)-Q\left(s,a,\Theta \right)\right)}^2={\left(r+\gamma \underset{a}{argmax}Q\left(s^{\prime },a,{\Theta }^{-}\right)-Q\left(s,a,\Theta \right)\right)}^2.}$

Learning the global optimal strategy

In our approach, we train a deep Q-network to learn the global optimal policy. Specifically, at each step of the time series, we observe an environment state s_t derived from the partial time series up to time t. This state is then combined with a preference vector $\omega ={\left({\omega }_0,{\omega }_1\right)}^{\top }$, uniformly sampled from the preference space $\Omega :{\sum }_{i=0}^L{\omega }_i=1$. The combination of s_t and ω creates a joint feature space, which serves as the input for our deep Q-network. The network is then trained on this composite input with the goal of discovering the global optimal policy. Conceptually, this training process can be visualized as exploring a series of rays extending radially in the first orthant of Fig. 1. The aim is to ascertain a Pareto set such that, corresponding to every valid ω_i, there exists a point d within the Pareto set for which the value of ω⋅F(d) achieves maximization. In other words, for each scalarization direction indicated by a specific ray ω, Our target is to identify points within the objective space that are situated at the maximum distance from the origin along the direction of that particular ray.

Moving from the conceptual framework to the specific implementation, the architecture of our deep Q-network is detailed in Fig. 3. This architecture, comprising convolution layers, flatten layers, and multi-layer perceptron (MLP) layers. Its output, ${\mathbf{Q}}_{\theta }\left(s_t,\omega ,a_t\right)$, manifests as a probability matrix of dimensions [1,  M × N], where M is the number of actions and N is the number of rewards. This output is then transformed into a matrix of dimensions [M, N]. Each row in this matrix corresponds to the action-value of each action, expressed as ${\mathbf{Q}}_{\mathbf{0}}\left(s_t,\omega ,a_t\right),{\mathbf{Q}}_{\mathbf{1}}\left(s_t,\omega ,a_t\right),{\mathbf{Q}}_{\mathbf{2}}\left(s_t,\omega ,a_t\right)$. The columns within this matrix represent the action-value for each objective, providing a detailed view of the potential outcomes for each possible action under different objectives.

Furthermore, to navigate the exploration-exploitation dilemma inherent in reinforcement learning, our global optimal strategy hinges on the ϵ-greedy algorithm (Sutton, 2020) for allowing the agent to alternate between exploring new actions and exploiting known high-value actions. To be specific, at every timestep, the agent chooses a random action from the action space $\mathcal{A}$ with a probability ϵ. Conversely, with probability 1 − ϵ, the agent opts for the action that maximizes the projected action-value ${max}_{a_t\in \mathcal{A}}{\omega }^{\top }\mathbf{Q}\left(s_t,\omega ,a_t;\theta \right)$. This process involves: (6)${\displaystyle a_t=\left\{\begin{array}{cc}\text{random action in}\mathcal{A},\hfill & \text{w.p.}ϵ\hfill \\ \underset{a_t\in \mathcal{A}}{max}{\omega }^{\top }\mathbf{Q}\left(s_t,\omega ,a_t;\theta \right),\hfill & \text{w.p.}1-ϵ\hfill \end{array}\right.}$ where $\mathbf{Q}\left(s_t,\omega ,a_t;\theta \right)={\left(Q_0\left(s_t,\omega ,a_t;\theta \right),Q_1\left(s_t,\omega ,a_t;\theta \right),Q_2\left(s_t,\omega ,a_t;\theta \right)\right)}^{\top }$.

This approach not only ensures a robust balance between gaining new knowledge (exploration) and leveraging existing information (exploitation), but also aligns the agent’s decisions with the multi-dimensional objectives in the SDP context, enhancing both the adaptability and effectiveness of the policy.

Parameter training procedure

Our training methodology for the deep Q-network is specifically tailored to the multi-objective challenges in student dropout prediction. Our approach significantly diverges from conventional reinforcement learning strategies which primarily rely on Q-learning algorithms. The Q-learning methods are underpinned by Bellman’s optimality equation (as indicated in Eq. (2)) and focus on maximizing expected rewards through the iterative refinement of the agent’s policy using temporal difference methods as Eq. (3). However, the standard Bellman’s equation becomes inadequate in our multi-objective setting due to the introduction of preference vectors. To tackle this challenge, we adopt envelope Q-learning (Yang, Sun & Narasimhan, 2019), a refined version of traditional Q-learning specifically engineered for multi-objective optimization. This critical adaptation enables us to extend the DDQN into its multi-objective counterpart, MODDQN. The MODDQN framework, detailed in Algorithm 1 and visually represented in Fig. 4, integrating the principles of multi-objective optimization with the complexities inherent in SDP, ensuring that the policy not only aligns with various preference vectors but also adapts our concept of optimality based on vectorized rewards.

The training process begins with the initialization of the replay buffer ${\mathcal{D}}_{\tau }$, which is essential for storing transitions during the learning process. Concurrently, we initialize the action-value function $\mathbf{Q}\left(s,\omega ,a;\theta \right)$ and its counterpart, the target action-value function ${\mathbf{Q}}^{\prime }\left(s,\omega ,a;{\theta }^{\prime }\right)$, which is essential for calculating the temporal difference in reinforcement learning.

For every episode, and at each timestep within the episode’s horizon, the model observes a state s_t, reflecting the current snapshot of a student’s learning behavior sequence up to time t. It is concatenated with sampled preference vectors, and executing actions based on the ϵ-greedy strategy as Eq. (6). The agent’s action leads to an environmental response, providing the next state s_t+1, a multi-objective reward vector r_t and the terminal indicator. Each transition, encapsulating the state, action, reward vector, next state and terminal indicator is stored in the replay buffer. When a mini-batch of the stored transitions is accumulated, they are utilized in the envelop Q-learning through the experience replay mechanism.

The envelope Q-learning, pivotal in our MODDQN framework, revolutionizes traditional Q-learning for multi-objective optimization. By employing vectorized value functions and updating them via the convex envelope of the solution frontier, this method deftly navigates the intricate landscape of multi-objective optimization. This is in stark contrast to scalarized Q-learning, which tends to optimize policies in isolation for each preference. Instead, envelope Q-learning concurrently optimizes a spectrum of policies across various preferences. This alignment with our multi-dimensional concept of optimality in SDP is concisely captured in the evaluation operator equation: (7)${\displaystyle Q^{\ast }\left(s,a,\omega \right)=\mathbf{r}\left(s,a\right)+\gamma {\mathbb{E}}_{s^{\prime }\sim \mathcal{P}\left(\cdot \mid s,a\right)}\underset{a^{\prime }\in \mathcal{A}}{max}{Q}^{\ast }\left(s^{\prime },a^{\prime },\omega \right).}$

By adopting this approach, we ensure that our policy is not just responsive but intricately aligned with the diverse and often conflicting objectives inherent in the realm of Student Dropout Prediction.

Furthermore, due to the imbalance in the agent’s memory—where classification actions are less frequent compared to delay actions—we adopt the Prioritized Experience Replay (PER) method. PER enhances learning efficiency from significant experiences and ensures more relevant transitions are frequently replayed. This method prioritizes transitions with higher learning potential, leading to a more focused learning process. To be specific, the calculation of the priority in PER for the lth transition is executed by: (8)${\displaystyle p_l=\left|{\delta }_l\right|+\chi }$ where δ_l denotes the temporal difference (TD) error of the lth transition, and χ > 0.

After calculating the priority of the lth transition, the transition is stored in memory buffer ${\mathcal{D}}_{\tau }$ as $\left(s_t,a_t,{\mathbf{r}}_{t+1},s_{t+1},p_l\right)$. When the total number of combined transitions exceeds the capacity of ${\mathcal{D}}_{\tau }$, the training of the model can proceed through the sampling of a mini-batch, guided by the probability distribution outlined below: (9)${\displaystyle P_l=\frac{p_l^{\alpha }}{\sum _m{p}_m^{\alpha }}}$ where P_l is proportional to the priority p_l of a transition, and α signifies how much prioritization being used.

Finally, the deep Q-network is updated using MODDQN with PER, and the loss function employed for updating the deep Q-network can be determined by: (10)${\displaystyle L_1\left(\theta \right)={\mathbb{E}}_{s,a,\omega }\left[{\parallel \mathbf{y}-\mathbf{Q}\left(s,a,\omega ;\theta \right)\parallel }_2^2\right]}$ where y is given by: (11)${\displaystyle \mathbf{y}={\mathbb{E}}_{s^{\prime }}\left[\mathbf{r}+\gamma {\mathbf{Q}}^{\prime }\left(s^{\prime },arg\underset{{\omega }^{\prime },a^{\prime }}{max}{\omega }^{\top }\mathbf{Q}\left(s^{\prime },{\omega }^{\prime },a^{\prime };\theta \right);{\theta }^{\prime }\right)\right].}$

Given optimizing L₁(θ) directly poses practical difficulties as the optimal frontier encompasses a vast array of discrete solutions, rendering the loss function’s landscape significantly non-smooth. To mitigate this, an auxiliary loss function L₂(θ) is employed: (12)${\displaystyle L_2\left(\theta \right)={\mathbb{E}}_{s,\omega ,a}\left[{\parallel {\omega }^{\top }\mathbf{y}-{\omega }^{\top }\mathbf{Q}\left(s,\omega ,a;\theta \right)\parallel }_2^2\right].}$

Ultimately, the target network Q′ undergoes updates at every T_update steps. The final loss function is formulated as follows: (13)${\displaystyle L\left(\theta \right)=L_1\left(\theta \right)+L_2\left(\theta \right).}$

Within the framework of Multi-Objective Reinforcement Learning algorithm, L₁ initially ensures that the Q-value is approximated to any real expected total reward, despite potential challenges in achieving optimality. Subsequently, L₂ exerts an auxiliary influence, nudging the current estimate towards a direction of enhanced utility. Following the learning phase, the agent is equipped to adapt to any given preference by merely inputting ω into the network, and our model could straightforwardly select the action with the highest Q-value as determined by the policy ${\Pi }_{\mathcal{L}}\left(\omega \right)$.

 
_______________________ 
Algorithm 1 Training Algorithm___________________________________________________________ 
  1:  Initialize replay buffer Dτ. 
  2:  Initialize action-value function Q(s,ω,a;θ). 
  3:  Initialize target action-value function Q′ (s,ω,a;θ′) by copying: θ′ ← θ. 
  4:  for episode = 1,...,M do 
 5:       Sample a training pair from the dataset { 
 ( 
  Xi,yi) 
      } 
       i=1...n 
  6:       while not terminal and t ≤ T do 
 7:            Obtain partially observable state st = Xi:t. 
  8:            Sample a preference ω ∼Dω and concatenate it with state st. 
  9:            Agent  receives  the  input  [st,ω]  and  picks  an  action  at  based  on 
     Eq (??). 
10:            Environment  steps  forward  according  to  at  and  gets  the  multi- 
     objective reward vector rt, the next state st+1, and the terminal state. 
11:            Store transition (st,at,rt,st+1,terminal) in Dτ. 
12:            if update then 
13:                  Sample  Nτ   transitions  (sj,aj,rj,sj+1)   ∼ Dτ   according  to 
     Eq (??). 
14:                  Sample Nω preferences W = {ωi ∼Dω}. 
15:                  Compute y according to Eq (??). 
16:                  Compute the loss function based on Eq (??) and Eq (??). 
17:                  Update Q-network by minimizing the loss function according to 
     Eq (??). 
18:            end if 
19:            if at = WAIT then 
20:                  Increment time t = t + 1. 
21:            else 
22:                  Predict and set terminal = True. 
23:            end if 
24:       end while 
25:  end for____________________________________________________________________________    

Experimental comparison between RL methods

In our comparative analysis, we first benchmarked our model against various reinforcement learning models, including EARLIEST, ECTS, and DDQN+PER. It is important to note the difference in training metrics: EARLIEST, being based on a deep learning approach, measures performance in epochs, indicating a complete cycle through the training dataset. In contrast, our model MORL, along with ECTS and DDQN+PER, are reinforcement learning models trained on episodes, each representing a full sequence of interactions in the environment.

To facilitate a consistent comparison across different training paradigms, we standardized our evaluation approach by focusing on iterations. Specifically, we assessed the models’ performance during each parameter update in the model training’s loss function process. This assessment was conducted on the testing set of both the KDD2015 and XuetangX datasets. The key metrics for evaluating model performance are Avg. Acc, APU and Avg. HM. We structured our assessment to provide outputs every 100 iterations, culminating in a total depth of 5,000 iterations. Furthermore, to ensure consistency in each code execution, we set a fixed random seed, stabilizing the initial random weights and data sequence, thereby maintaining uniformity in prediction results and loss. This method allowed for a detailed and consistent evaluation of models’ performance across different datasets, ensuring comparability and robustness in our findings.

The experimental outcomes are illustrated in Figs. 5 and 6. Our model exhibits robust and stable performance on both KDDCUP2015 and XuetangX datasets, showing an increasing predictive accuracy with fewer time series data over iterations, especially in maintaining excellent harmonic scores. This trend signifies a consistent and effective balance between prediction accuracy and earliness, highlighting the model’s proficiency in navigating the inherent trade-offs within these datasets. These findings highlight that our MORL model’s strategy, which focuses on optimizing the convex envelope of multi-objective Q-values, ensures an efficient alignment between preferences and the corresponding optimal policies. This approach effectively tackles the challenge of optimizing multiple objectives simultaneously.

While the ECTS model occasionally surpasses our MORL model in predictive accuracy on the KDD15 dataset but generally underperforms on the XuetangX dataset. However, its performance shows significant fluctuations, likely due to the imbalanced agent’s memory and the infrequency of classification actions, leading to uneven learning experiences.

Compared to ECTS, the DDQN+PER model demonstrates its stability, likely due to its implementation of prioritized sampling and prioritized storing. However, it still exhibits step-like fluctuations in predictive accuracy on both datasets and scores lower in harmonic metrics than our model. It may be attributed to the transformation of the multi-objective problem into a single-objective one, which cannot be tailored to optimize for certain preferences.

The experimental results for the EARLIEST model were somewhat unexpected. Although it demonstrates stability and gradual improvement in predictive accuracy across iterations, this approach requires more data in the later stage of the time series to maintain the prediction accuracy. This situation could result in compromises to prediction accuracy, leading to a decrease in its harmonic score. The experimental results suggest that, despite utilizing reinforcement learning for early stopping decisions, the EARLIEST model’s approach of using an additional loss term and preset hyper-parameters to balance prediction accuracy and earliness fails to achieve a satisfactory multi-objective balance in certain specific datasets.

Experimental comparison between Non-RL methods and RL methods

We conducted a comprehensive comparison of our MORL model against various Non-RL and RL methods. Our goal was to identify which method achieves the optimal trade-off between prediction accuracy and earliness on the KDD2015 and XuetangX testing sets.

For Non-RL methods, we began by evaluating the LSTM-Fix model, trained using the complete sequence of the training set. This model’s predictive accuracy was assessed at predefined static halting points. We noted the highest predictive accuracy and the corresponding proportion of the sequence utilized. The NonMyopic model, claiming to predict the optimal time of early warning for student dropout, was trained using the full sequence. Its predictive accuracy was then evaluated at the most appropriate early classification time on the testing set.

In the RL category, we selected optimal policies that performed best during training, i.e., among the most accurate policies, we selected the quickest one. We then evaluated this selected model on the testing set across five test trials. We reported the Avg. Acc and the APU on both MOOC datasets. These evaluation metrics were subsequently used to compute the Avg. HM, offering a balanced measure of accuracy and earliness. The results of these comparisons are summarized in Table 1.

The experimental results indicate that our model can achieve comparable prediction accuracy with the least amount of data on two datasets when compared to other models. This enables our model to obtain the highest harmonic mean, affirming its superior performance in achieving a favorable trade-off between prediction accuracy and earliness. It may attributed to the ability of MORL’s optimization over the space of all possible preferences could quickly align one preference with optimal rewards and produce the optimal policy for any user-specified preference.

LSTM-Fix and EARLIEST alternately achieved the first and second highest prediction accuracy on both datasets. However, their high accuracy was attained nearly using the whole sequence. This reliance on sacrificing earliness to gain prediction accuracy does not effectively balance the conflicting accuracy-earliness objectives in the SDP task. This demonstrates that both of these methods could only care about one objective and face challenges in balancing multiple conflicting objectives.

The NonMyopic model, which claimed could calculate optimal prediction timing for early classification, is on par with the predictive accuracy with MORL, but utilized significantly more data to reach the same performance. This may account for the static method can also achieve predictions as early as possible while ensuring prediction accuracy to a certain extent, but it may fail to dynamic adaptation to related tasks with different preferences.

The ECTS and DDQN+PER models underperform the MORL model across three evaluation metrics, indicating that the dependence on scalar reward to combine different objectives could only learn an sub-optimal policy over the space of preferences but cannot be tailored to be optimal for specific preferences. By comparison, the MORL model can effectively use the information of ${max}_{a}Q\left(s,a,{\omega }^{\prime }\right)$ to update the optimal solution aligned with a different preference ω in the multi-objective space. It is also noted that the ECTS model requires 30% more time series data than the DDQN+PER model to achieve comparable prediction accuracy, further confirming that the DDQN+PER model addresses the issue of memory imbalance in the ECTS model through the DDQN and PER methods. The techniques were also adopted in our MORL model.

Policy adaptation

During the adaptation phase, we assessed how our MORL agent responded to user preferences on the XuetangX dataset. The experimental setup included a dynamically adapting MORL agent and three control groups with fixed preference settings, where the weights of accuracy (Acc) and earliness were set at ratios of 0.3:0.7, 0.5:0.5, and 0.7:0.3, respectively. All models underwent training across 10,000 episodes, with their parameters saved for subsequent analysis.

During the testing phase, we varied the weight of the preference on prediction accuracy from 0.1 to 0.9. We assessed the model’s predictive Avg. Acc, APU and Avg. HM across different Accuracy ratios. Each data point was computed using three tests. We then plotted comparative curves of these three metrics (refer to Fig. 7). In these plots, the solid lines represent the mean of three independent runs under each configuration, whereas the light shadow denotes the standard deviations..

The results clearly demonstrate that our MORL agent consistently achieved robust predictive outcomes across a variety of preference ratios. This highlights the agent’s capability to adeptly adjust to the user’s preference. Furthermore, our deep MORL algorithm consistently outperformed other algorithms, particularly in scenarios where success was deemed more critical to the user.

In contrast, models trained with fixed preferences, although capable of achieving high predictive accuracy at higher Acc ratios, often failed to maintain this performance at lower Acc ratios. This limitation may stem from the fact that fixed preferences only find optimal solutions along predefined scalarized directions. When tested with varying preference ratios, the choice of weights might bias the model towards certain regions of the objective space. Consequently, this approach might overlook other potential optimal solutions not aligned with these predefined directions, leading to local, rather than global, Pareto optimality.