A review of visual sustained attention: neural mechanisms and computational models

Sustained attention and the frontal lobe

The frontal lobe is a section of the brain that covers the front part of the cerebral cortex. It is usually regarded as the executive control center of the brain (Luria, 1973). These executive functions consist of a number of individual capacities, such as inhibition, goal-directed behavior, and self-monitoring (Oliveira et al., 2012). These individual capacities control and regulate the process of sustained attention. The frontal lobe mainly comprises the premotor area, primary motor area, and prefrontal lobe. The prefrontal cortex is located in the anterior region of the frontal lobe. It is linked to higher-order processing abilities such as attention, working memory, language, and executive function (Raver & Blair, 2016). Frontal lobe system damage often affects sustained attention, resulting in lapses in sustained attention and attention-executive disorders (Esterman et al., 2013). Moreover, it was found that patients with injuries in some areas of the prefrontal cortex performed abnormally in the implementation of sustained attention tasks, whereas they performed normally in other cognitive ability tests (Sarter, Givens & Bruno, 2001; Langner & Eickhoff, 2013). The frontal cortex has rich functional connections with the posterior brain system as well as several subcortical systems, including the limbic system, midbrain reticular system, and thalamic structure.

Many neurophysiology studies have found increased activation in the frontal cortex during a vigilance task (see Table S1). In addition, Han, Lee & Choi (2019) proposed that theta and alpha-band (4–12 Hz) EEG activities in the frontal cortex were essential for sustained attention and goal-related behaviors. In particular, EEG activity of the CPT state shows the dominance of effective connections going from the prefrontal cortex toward the parietal lobe at 4 Hz (Francisco-Vicencio et al., 2022). Moreover, sustained attentional preparation can be indexed by the deployment of a centrally distributed event-related potential (ERP), named the contingent negative variation (CNV) (Segalowitz, Dywan & Unsal, 1997; Kropp et al., 2001). CNV and P3 within the frontal cortex appear to be good candidates to investigate different mechanisms supporting sustained attention and prediction abilities (Thillay et al., 2015).

Damage to the whole frontal cortex or attention-related brain region (such as, pMFC, mPFC) affects sustained attention function, manifesting as behavioral disturbances or functional abnormalities in an individual. For example, rs-fMRI study found that frontal functional disconnection may underlie the pathogenesis responsible for defective vigilance/sustained attention (Tu et al., 2020). In addition, increased functional connectivity in the right frontoparietal network might reflect excessive cognitive fatigue in patients with traumatic brain injury (TBI) (Shumskaya et al, 2012).

Deficits in sustained attention are the most common disorder caused by frontal cortex damage (Wilkins, Shallice & McCarthy, 1987). Many studies have shown that abnormal neuron development in the frontal lobe may cause sustained attention disorders or even hyperactivity disorders (Rubia et al., 2019). In addition to developmental disorders, stroke and brain tumors in the frontal cortex can also lead to sustained attention deficits (Torres et al., 2021). Closed head injuries caused by external impact or sudden violent exercise can also impair sustained attention (Parasuraman, Mutter & Molloy, 1991). Furthermore, sustained attention decreases with aging because of frontal lobe degeneration over time (Mitko et al., 2019). Patients with deficits in sustained attention often suffer from ADHD, epilepsy, depression, intellectual disability, and other complex neuropsychiatric problems as well (Malkovsky et al., 2012). Among these symptoms, ADHD is a representative disorder of deficits in sustained attention (Kass, Wallace & Vodanovich, 2003). It has been linked to inattention, impulsivity, and negative affect (Barkley, Knouse & Murphy, 2011). Patients with ADHD have difficulty maintaining focus and vigilance for extended periods of time, leading to poor academic performance, career mistakes, and even operator-related train/car accidents (Fortenbaugh, De Gutis & Esterman, 2017; Zeller, 2022).

Sustained attention and the cingulate cortex

The cingulate cortex consists of two distinct systems: (1) a posterior system that receives input from dorsal stream areas and projects to some cortical systems, the thalamus, and (2) an anterior system that receives signals from the thalamus and frontal-parietal lobes and projects to limbic structures (Jafari, Malayeri & Rostami, 2015). Some studies have revealed that the cingulate cortex participates in sustained attention tasks (see Table S2).

The anterior cingulate cortex (ACC) receives inputs from the lateral frontal cortex and the posterior parietal cortex. In addition, it compactly connects with the basal ganglia (BG). The anterior cingulate cortex, which is part of the limbic system, receives inputs from the thalamus and neocortex and has large projections to the nucleus accumbens and amygdala. Many experts have found that the anterior cingulate cortex plays a vital role in conflict monitoring (Jones et al., 2002). The ACC was found to be associated with cognitive impairment in rs-fMRI of sustained attention tasks (Loitfelder et al., 2012). Furthermore, some researchers have confirmed that the anterior cingulate cortex is constantly activated during tasks related to sustained attention (Fan et al., 2018). Subsequent studies using a series of Stroop tasks have suggested that strong reactions in the anterior cingulate cortex mediate attention and conflict resolution (Corlier et al., 2020) (see Table S2). Moreover, in healthy adults, better sustained attention was associated with more robust activation of the ACC during SART and gradCPT tasks (Esterman et al., 2013).

The posterior cingulate cortex (PCC) is considered to be a paralimbic cortical structure. It has rich projections to the frontal, parietal, and temporal cortex, as well as to subcortical systems such as the thalamic nucleus, pontine, and basal ganglia. Therefore, the PCC is thought to be involved in several cognitive activities, although its specific functions have not been clarified. The results of resting functional brain imaging revealed that the PCC is a critical node in the default mode network (DMN) and plays a vital role in attention regulation (Kral et al., 2019). It has been confirmed that the PCC is activated and has strong interactions with other parts of the DMN in both resting state and continuous working memory tasks (Lau, Leung & Zhang, 2020). Abnormalities of the DMN are frequently seen in neurological and psychiatric disorders such as ADHD, Alzheimer’s disease, schizophrenia, autism, and depression. Therefore, PCC has important clinical significance (Zhou et al., 2020).

The latest neuroimaging research found that there is a selective enhancement of oscillatory coupling between the ACC and the dorsal attention network (DAN) during attention tasks (Wong et al., 2022). Human single-neuron recordings during conflict tasks suggest that the dorsal ACC can be involved in attention-related performance monitoring (Fu et al., 2022). Resting-state functional connectivity within the DAN can predict individual performance in spatial attention tasks (Machner et al., 2022).

Sustained attention and subcortical structures

Subcortical structures are neural structures located deep in the brain that include the brainstem, midbrain, cerebellum, basal ganglia, thalamus, hypothalamus, and limbic nuclei. The hypothalamus and the reticular formation coordinate arousal through their vast array of projections to other brain regions (Rapoport et al., 1978). The basal ganglia and thalamic nucleus are responsible for processing gating information. They are closely related to attention and somatic movement (McAlonan, Cavanaugh & Wurtz, 2008). According to Fan et al. (2008), the basal ganglia appear to be central to executive regulation mechanisms, error monitoring, and sustained vigilance. Limbic nuclei include the amygdala, septal nucleus, and nucleus accumbens.

The anterior thalamic nuclei may serve as a site of integration between frontal areas and the hippocampus to regulate attentional processes (Nelson, 2021). Attention is also linked to the hippocampus, which is responsible for the storage, conversion, and orientation of long-term memory (Aly & Turk-Browne, 2016). Evidence from the reticular formation (Dietrich & Audiffren, 2011), thalamus (Rajab et al., 2014), and limbic structures (Wang et al., 2013b) suggests that exercise may help to facilitate attentional processes. The asymmetrical development of the right-lateralization of the frontal lobe and left-lateralization of the occipital lobe may affect ADHD severity (Chen et al., 2021). Furthermore, stereo-electroencephalography (SEEG) recordings provide direct evidence that the anterior nucleus of the thalamus modulates hippocampal gamma activity in attention and working memory tasks (Piper et al., 2022; Liu et al., 2021).

Based on the information above, attention is a byproduct of regulation from multiple brain regions rather than a strictly cortical phenomenon. Clinical studies have provided additional evidence that the nervous system contains the frontal lobe, cingulate cortex, and subcortical system, which play an appropriate role in sustained attention.

Neural pathways of visual sustained attention

Sustained visual attention is necessary for humans’ visual systems to have incredible perception and data processing capabilities. Many studies have been dedicated to exploring the neural pathways of sustained visual attention.

Dynamic causal modeling provides compelling evidence for the regulation of attention through the PFC ↔thalamic, ACC ↔thalamic, BG ↔thalamic, and PFC ↔BG pathways (Jagtap & Diwadkar, 2016). For example, the modulation of thalamic →PFC pathways is presumed to reflect ascending attention processes engaged by external sensory inputs of salient and novel stimuli. In comparison, modulation of frontal →thalamic pathways represents descending attention processes mediated by voluntary shifts of attention based on expectations of goals and rewards (Connor, Egeth & Yantis, 2004). Before reaching the cortex, visual information is filtered by the thalamus. The thalamus, the “gateway” to the cortex, comprises various subnuclei involved in attention gating (Brunia, 1993). The thalamus affects feedforward and feedback information transmission between the frontal, parietal and occipital cortex regions (Tokoro et al., 2015). Attention to stimuli suppresses the neuronal activity of the reticular nucleus over selected relay nuclei, and this disinhibition gates thalamocortical inputs (Conway, 2014). These functional effects appear to be mediated by anatomical connections between the thalamus (and specific thalamic nuclei) and regions of the frontal lobe, including the LPFC and the cingulate cortex. The PCC is closely linked to the thalamus (Leech & Sharp, 2014). It receives information from the visual cortex and sends it to the LPFC (Buschman & Kastner, 2015). Sustained attention responses also exist in the early visual cortex in the absence of visual stimuli (Silver, Ress & Heeger, 2007). Several elegant studies have found that the presupplementary motor area (pre-SMA) is the target of the BG (Akkal, Dum & Strick, 2007; Wiesendanger & Wiesendanger, 1985). The pre-SMA receives input from the cortex and delivers output to the thalamus. The BG organizes motivations that lead to the execution of goal-directed behaviors, for example, pushing a button. When the attention process in the ventral regions is goal-oriented, information from the visual cortex activates neural activity in the inferotemporal cortex (IT), which is followed by activation in the LPFC (Hommel et al., 2019). The anteromedial prefrontal cortex, ACC, anterior insula, and anterior thalamic nodes form the cingulo-opercular circuit, which is involved in distinguishing potential mismatches and conflicts (Williams, 2016).

In addition, the arousal model shows that the LC-norepinephrine system in the brainstem plays a critical role in the vigilance of sustained attention. Norepinephrine projections originating from the LC and ending in the thalamus mediate the attention process (Sarter, Givens & Bruno, 2001). Projections from the ACC to the LC-norepinephrine system indicate that the mPFC is involved in the regulation of arousal through low-frequency phase synchronization with the LPFC (Clayton, Yeung & Cohen Kadosh, 2015; Craigmyle, 2013). Based on the above literature, we propose possible pathways of sustained attention, as shown in Fig. 2.

In addition, functional brain networks play a crucial role in sustained attention. Recent research has posited that the visual processes of sustained attention emerge from an array of large-scale functional networks (Fortenbaugh et al., 2018). In largely independent lines of research, influential brain network models (Esterman et al., 2013) have suggested that optimal sustained attention requires cooperation among the task positive network (TPN), frontoparietal control network (FPN), ventral attention network (VAN), dorsal attention network (DAN), and DMN, as shown in Fig. 2. Intracranial electroencephalography (iEEG) in human subjects offers evidence that the DMN interacts negatively with both the DAN and salience network (SN) (Kucyi et al., 2020). Moreover, researchers who utilized BOLD of fMRI found that many attention-related brain networks, such as the DMN and DAN, were activated during the gradCPT task (Mitko et al., 2019). Better sustained attention is associated with stronger anticorrelations between the DAN and DMN (Chang et al., 2022). Furthermore, optimal sustained attention is less dependent on the DAN and more dependent on brain networks related to task automation, such as the DMN (Okabe, 2016). Therefore, characterizing both anatomic neural pathways and functional connectivity could allow for a more profound study and eventually provide a panorama of the neural mechanisms of sustained attention.

Biomathematical models

Early research on computational models was related to sleep loss and circadian rhythm. Researchers usually ask subjects to perform a vigilance task after sleep deprivation when sleep is insufficient or irregular. Then, the subject’s vigilance or sustained attention can be quantified by using a computational model that is based on task performance. A diverse set of vigilance tasks will cause a decline in performance as time-on-task increases (Davies & Parasuraman, 1982). These vigilance tasks proved that the reduction magnitude is influenced by various factors, including the event rate, signal probability, stimulus duration, stimulus modality, and others (Warm, Dember & Hancock, 1996; Greenlee & Hess, 2019). At present, sleep loss and fluctuations in circadian rhythms are used by researchers to explain the reasons for decreased vigilance (Walsh, 2014). There appear to be notable differences in the study of theories about sleep deprivation and vigilance reduction. However, studies have shown that insufficient sleep and reduced vigilance have the same effects on cognitive processing (Veksler & Gunzelmann, 2018). Kronauer, Forger & Jewett (1999) found that ambient light can affect vigilance by influencing the phase and amplitude of circadian pacemakers. Jewett & Kronauer (1999) proposed a circadian rhythm neurobehavioral performance and alertness (CNPA) model. Hursh et al. (2004) proposed a sleep activity, fatigue, and task effectiveness (SAFTE) model. These biomathematical models predict that an increase in sleep deprivation will lead to a continued decline in vigilance.

Therefore, early biomathematical models can only provide estimates of vigilance. They cannot predict potential changes in performance or cognitive processing, nor can they explain the mechanism of behavioral changes.

Integration models

Researchers integrated predictions of alertness levels, generated by biomathematical models, with information-processing methods in cognitive architecture model to produce precise predictions of sustained attention. Cognitive architecture model provides a unified information-processing framework that is based on decades of empirical evidence and psychological theories. Thus, the integration model not only helps in understanding the changes in human performance as vigilance declines but also explains the underlying neural mechanisms of sustained attention.

Anderson et al. (1998) proposed a cognitive architecture known as adaptive control of thought-rational (ACT-R). It has been used to provide a quantitative description of human performance in cognitive tasks. Gunzelmann et al. (2011) introduced the microlapse theory of fatigue (MTF), which integrated a biomathematical model with ACT-R to model different sustained attention tasks. MTF, as an instantiation of the computational model, describes the process of decreased vigilance. Jackson et al. (2013) improved Gunzelmann’s model by including a time-on-task component. Although Jackson’s model uses the same mechanism as the original model, it has an impact on sleep deprivation and circadian rhythm research. Gartenberg et al. (2018) proposed the microlapse theory of fatigue with replenishment (MTFR), a process model similar to MTF that supplements the mechanisms related to opportunistic rest periods and internal rewards.

These computational models, when combined with a specific cognitive framework, can achieve a strong fit between simulated and actual behavioral data to predict behavioral performance in sustained attention tasks.

Machine learning algorithms

With the advancement of modern science, neuroimaging technology is increasingly being used to evaluate sustained attention (Zhang et al., 2022). Previously, neuroimaging data were combined with classical statistical methods to construct a computational model of sustained attention. However, as research advances, neuroimaging technology will result in an explosive increase in data scale. Classical statistical methods cannot handle large quantities of neuroimaging data, and manually processing these complex neuroimaging data is time-consuming. Thus, computational models that can automatically and elegantly process massive quantities of neuroimaging data are urgently needed to meet the demands of state-of-the-art neuroimaging research. Machine learning, an advanced computing method, is uniquely suited to address these issues.

In traditional machine learning approaches, handcrafted features are commonly combined with support vector machines (SVMs), k-nearest neighbors (KNNs), and Bayesian networks. The classification accuracy is over 70% when solving a binary classification problem in most EEG-based studies (see Table 2).

SVM is a supervised learning model for classification and regression in machine learning. It performs well when recognizing small samples with high-dimensional data. By using the SVM model, some studies focused on assessing the attentional state in healthy people. Yeo, Shen & Wilder-Smith (2009) tested the usefulness of SVM in identifying or distinguishing between alert and drowsy EEG patterns. Cirett Galán & Beal (2012) used SVM to estimate sustained attention and cognitive workload while students were solving a series of math problems. Zhang et al. (2016) used SVM to detect sustained attention load based on passive brain-computer interface signals from functional near-infrared spectroscopy (fNIRS); Samaha, Sprague & Postle (2016) used SVM to assess whether the spatial selectivity of neural responses can be recovered from the topography of alpha-band oscillations during spatial attention. Batbat, Güven & Dolu (2019) achieved high accuracy by combining EEG data from visual, auditory, and auditory-visual tasks and using an SVM with a linear kernel to classify different attentional states. Moreover, SVM is regarded as a relatively fast classifier. It is practically suitable for cases where the number of features is greater than the number of instances. However, SVM is difficult to implement in large-scale samples and to address multilabel classification problems.

The KNN algorithm, unlike the SVM algorithm, is an instance-based learning method. The KNN classifier is very simple and intuitive. High accuracy was achieved in classifying clinical patients and nonclinical participants using a combination of features with a KNN classifier. There are studies that have distinguished between different attentional states. For example, Chen et al. (2010) classified sustained attention phases using electrocardiograms (ECG) with KNN. Borji, Sihite & Itti (2011) used KNN to predict human attention in the training set by implementing a nonlinear mapping from eye position to task state. Hu, Sun & Ratcliffe (2016) proposed a classification method that combines correlation-based feature selection (CFS) and a KNN algorithm to identify attentional states during the learning process. However, KNN depends heavily on training data. The complexity of KNN increases dramatically as the number of features increases.

The Bayesian model is more adaptable than the two other methods. It is a type of probabilistic graph model that uses Bayesian inference to compute the probability. The Bayesian model is fast and has been used for attentional classification problems by many researchers, such as Larue, Rakotonirainy & Pettitt (2010). In their study, the participants’ reaction time during the SART was used to detect vigilance decline in real time using a Bayesian model. They quantified the effect of monotony on overall performance. In addition, a Bayesian model can also be used to predict the likelihood of humans typically focusing on a scene (Pang et al., 2008). Luo et al. (2020) developed a hybrid incremental dynamic Bayesian network and constructed a visual focus detection method based on fusing drivers’ head and eye movement data. Borji, Sihite & Itti (2012a); Borji, Sihite & Itti (2012b) used Bayesian networks to estimate the attentional state of subjects while performing a task (for example, playing video games) and mapped the state to an eye position. In addition to the classifiers mentioned above, principal component analysis (Arruda et al., 2007) (PCA), artificial neural networks (ANNs) (Dowman & Ben-Avraham, 2008), linear discriminant analysis (LDA) (Ghassemi et al., 2009), K-Means (Gurudath & Bryan Riley, 2014), and other methods have been used to solve attention-related classification problems.

Classic machine learning algorithms require experts to design elegant, handcrafted features. However, deep learning can learn feature representations from datasets automatically. To interpret data, deep learning builds a deep neural network that mimics the neural mechanisms of the human brain (Zaharchuk et al., 2018). As data acquisition technology advances in scientific research, computer-aided data analysis based on deep learning will become more widely accepted. Therefore, we reviewed existing deep learning methods for attentional state classification (see Table 3), including convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

CNNs have been proven to be efficient in research areas such as image recognition and classification. In recent years, many researchers have used CNNs to measure attention and try to find the neural features that correspond to it. For example, Borhani et al. (2018) developed an EEG-based classifier that used CNN to investigate underlying subject-specific features related to early visual attention. Hosseini & Guo (2019) developed a channelwise deep CNN model to classify features extracted from EEG signals from a focusing state and mind wandering. Ho et al. (2019) proposed a framework for distinguishing mental workload by combining hemoglobin concentration features with CNN. Wang, Antonenko & Dawson (2020) proposed a multiscale convolutional neural network-dynamic graph convolutional network (AMCNN-DGCN) model that estimated driving fatigue using EEG data from the driving task.

However, CNNs are not well suited to processing sequential data. Therefore, researchers developed RNN models in which neural network nodes and connections form a directed graph along a temporal sequence. As a result, RNNs are suitable for tasks involving sequential data, such as online handwriting recognition (where features can be extracted from both the pen trajectory and the resulting image) (Wu et al., 2014) and speech recognition (Fayek, Lech & Cavedon, 2017). RNNs can also be used to assess the state of sustained attention over time. Moinnereau et al. (2018) presented a deep RNN architecture for learning robust features and predicting cognitive load levels from EEG recordings. Jeong & Jeong (2020) utilized RNNs to distinguish between possible attention states. In addition, some electrophysiological data with long recording times are also suitable for processing with RNNs. For example, Phan et al. (2018) proposed a feature learning approach for single-channel automatic sleep stage classification. This approach is based on a deep bidirectional RNN with an attention mechanism. Huve, Takahashi & Hashimoto (2018) collected fNIRS signals and then used both a deep neural network (DNN) and an RNN to evaluate the impact of a driver’s mental state under various environmental conditions.

Here, we reviewed several types of computational models proposed for attention identification and evaluation. With the advancement of modern neuroimaging technology, computational models may be able to reduce labor costs while also facilitating the assessment of sustained attention. Neuroimaging data from these studies could in turn improve the accuracy of computational models and help researchers find neuromarkers that can represent the sustained attention cognitive process.