An effective classification framework for brain-computer interface system design based on combining of fNIRS and EEG signals

EEG+NIRS single-trial classification dataset

“Open Access Dataset for EEG+NIRS Single-Trial Classification” is used to reveal the proposed framework’s performance in the study (Shin et al., 2017; Blankertz et al., 2010). This dataset consists of NIRS and EEG signals, including mental activity (MA) and motor imaginary (MI), two separate tasks. A total of 29 users (15 females, 14 males; 28 right hands, 1 left hand) participated in the study.

MI has two functions in itself, right hand and left hand. MA includes mental processing and resting-state tasks within itself. The experimental setup is designed with the instructions given to the subject sitting 1.6 m in front of the 50-inch screen. The paradigm of the experiment is given in Fig. 2. Both tasks started with one-minute rest before the experiment. Then, 2 s of visual information about the task, 10 s of task execution, and 15–17 s of rest after the task are given.

This process is repeated 20 times in each session. MI and MA tasks are recorded sequentially and in 3 sessions. As shown in Fig. 3A, fNIRS recordings are taken with 36 physiological channels produced using 14 sources and 16 detectors. The recording is performed with a sampling frequency of 12.5 Hz. The recordings are then downsampled at 10 Hz. Figure 3B shows that 30 EEG electrodes placed according to the international 10-5 system are given. The signals are collected with a 1,000 Hz sampling frequency and then downsampled at 200 Hz (Shin et al., 2017).

Preprocessing of the EEG and fNIRS Signals

Raw fNIRS and EEG signals obtained from the dataset are subjected to a series of processes before classification. The transactions performed are shown in Fig. 4. First, by applying the Modified Beer-Lambert law given in (1) to fNIRS signals, the concentration changes of oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) are calculated (Chiarelli et al., 2018; Trakoolwilaiwan et al., 2017; Shin & Jeong, 2014).

(1) $$\left[\begin{array}{c}\mathrm{\Delta }\left[HbO\right]\\ \mathrm{\Delta }\left[HbR\right]\end{array}\right]={\displaystyle \frac{1}{\rho }{\left[\begin{array}{cc}{\epsilon }_{O_{2}Hb}\left({\lambda }_1\right).DPF\left({\lambda }_1\right)& {\epsilon }_{HHb}\left({\lambda }_1\right).DPF\left({\lambda }_1\right)\\ {\epsilon }_{O_{2}Hb}\left({\lambda }_2\right).DPF\left({\lambda }_2\right)& {\epsilon }_{HHb}\left({\lambda }_2\right).DPF\left({\lambda }_2\right)\end{array}\right]}^{-1}\times \left[\begin{array}{c}O{D}_{{\lambda }_1}\\ O{D}_{{\lambda }_1}\end{array}\right]}$$

The HbO and HbR curves are filtered with a 3rd order 0.01–0.09 Hz Butterworth bandpass filter. Processes after this step are common for both fNIRS and EEG signals. Three sessions and 20 repetitions in each session are segmented from the moment stimulation began (0 s) to the moment it ended (10 s). The obtained signals are subjected to the baseline correction process with the average of the signal generated at the instruction stage before stimulation (−2 s to 0 s).

Feature extraction from the EEG and fNIRS Signals

The representation ability of hand-crafted features is known, especially in the analysis of complex signals. Mean, maximum, slope, variance, skewness, kurtosis, and median features frequently used in fNIRS signals are used in the literature (Aydin, 2020; Chiarelli et al., 2018). The feature combination of the proposed framework is as follows:

• (1) Mean: It is the average amplitude value of each epoch signal. It is calculated by Eq. (2). Here $\mu$ mean value, $N$ total data points, $X_k$ attribute of the signal to be calculated data point.

(2) $$\mu ={\displaystyle \frac{1}{N}\sum _{k=1}^N{X}_k}$$

• (2) Maximum: It is the highest amplitude value of each epoch signal.

• (3) Slope: It is the average of the slopes in a defined time window over the entire signal.

• (4) Variance: It is the value showing the distance of the distribution from the mean in the data. Where $V$ is variance, N total data point, $\mu$ arithmetic mean and $X_k$ attribute of the signal to be calculated data point.

(3) $$V={\displaystyle \frac{1}{N-1}\sum _{k=1}^N{\left|X_k-\mu \right|}^2}$$

• (5) Skewness: It is the value that gives the degree of non-symmetry of a distribution. X is data, $\mu$ is the mean, $\sigma$ is the standard deviation, and $\mathrm{E}$ represents the expected value.

(4) $$\mathrm{s}={\displaystyle \frac{\mathrm{E}{\left(\mathrm{x}-\mu \right)}^3}{{\sigma }^3}}$$

• (6) Kurtosis: It is the value that gives the sharpness or flatness of the curve. Where $\mathrm{x}$, $\mu$, $\sigma ,$ and $\mathrm{E}$ same value in skewness.

(5) $$\mathrm{s}={\displaystyle \frac{\mathrm{E}{\left(\mathrm{x}-\mu \right)}^4}{{\sigma }^4}}$$

• (7) Median: It is the value in the middle of the sorted data.

These seven features are calculated for each channel. Then, the feature space is created by adding each channel side by side. Six feature matrices are created for MI and MA after preprocessing. These matrices are HbO, HbR and Hbo+HbR obtained from fNIRS signals, EEG obtained from EEG signals, and their combination EEG+HbO and EEG+HbR.

As stated under the title of feature extraction in the study, seven features have been extracted for each channel. Since fNIRS signals are 36 channels, 36 * 7 = 252 features are obtained from HbO and HbR values obtained after Beer-Lambert transformation. The attribute matrix with tags has 253 columns. Similarly, 32 * 7 = 224 features are obtained from 32 channel EEG signals. Together with the tag column, an attribute matrix of 225 columns is obtained. In the hybrid studies, 476 features are obtained from 252 + 224 features for EEG+HbO and EEG+HbR data, while 252 + 252 = 504 features are obtained from HbO+HbR data. Tag vectors are then added to these. The number of observations made is; The data belonging to each user is divided into 10-s epochs from the start of the task. In this setup, 20 observations for each session and 60 observations in a total of 3 sessions are obtained. When 29 users are combined, 60 * 29 = 1740 observations are obtained. In summary;

1. HbO and HbR feature vectors (1740, 253),

2. HbO+HbR hybrid feature vectors (1740, 505),

3. EEG attribute vector (1740, 225),

4. The EEG+HbO and EEG+HbR hybrid feature vectors are of the size (1740, 476).

Feature (attribute) weighting algorithm

The main purpose of the feature weighting process is to transform nonlinearly separable data into a linearly separable form. In the study, two different k-means clustering methods are used for the weighting of the features. K-Means clustering methods pseudo-code is given in Table 1 (Polat & Durduran, 2012).

k-means clustering centers based attribute weighting method (KMCC-based)

The center of each feature set is found by k-means clustering (KMC), and then the ratio of the feature means to the cluster center is calculated. The pseudo-code for the method is given in Table 2 (Polat & Durduran, 2012). Where $i$ is class number, $j$ is features number, $c_i$ are feature matrixes for two-class, $z_i$ are cluster centers for two feature matrixes, ${\mu }_{i,j}$ are the mean value of features for two-class, $w_{i,j}$ are weight values of features for two class and $dat{a}_{weighted}$ is KMCC-based weighted data.

Means clustering centers difference based attribute weighting method (KMCCD-based)

In this method, the center of each feature set is found by KMC, and then the distance of each data point to the cluster center is calculated. The ratio of the mean of these distances to the cluster centers gives the weight value for each feature. The pseudo-code for the method is given in Table 3 (Polat, 2018). Where $i$ is class number, $j$ is features number, $c_i$ are feature matrixes for two-class, $z_i$ are cluster centers for two feature matrixes, $d_{i,j}$ are a distance of each data point to the cluster center, ${\mu }_{i,j}$ are the mean value of distances for two-class, $w_{i,j}$ are weight values of features for two class and $dat{a}_{weighted}$ is KMCC-based weighted data.

Classifier algorithms

In this section, LDA, SVM, and kNN classifiers are used to observe the effect of classifiers’ performance.

LDA searches for a vector that best separates data points. It creates a linear combination that gives the most significant mean differences according to the classes entered. In this classifier, a primary scoring function is defined, and the coefficients that will maximize this score are sought (Arican & Polat, 2019; Filho et al., 2014; Parah et al., 2020; Ohata et al., 2021).

SVM is a machine learning method recommended for classification problems in datasets where patterns between variables are unknown. SVM is based on statistical learning theory and structural risk minimization. For classification, it is possible to separate the two groups by drawing a boundary between two groups on a plane. The place where this border will be drawn should be the furthest from the members of both groups. Here SVM determines how this border will be drawn. SVMs are classifiers that do not take any parameters (nonparametric). There is no prior knowledge or assumption about the distribution. Inputs and outputs are matched in training sets. Decision functions that will classify the input variable in test sets and new data sets are obtained through the peers (Costantini et al., 2009; Parah et al., 2020; Ohata et al., 2021; Dourado et al., 2021).

kNN is one of the algorithms used for classification in supervised learning. It is considered to be the simplest machine learning algorithm. In model recognition, the nearest neighbor algorithm (kNN) is a nonparametric method used for classification. With kNN, basically, the closest points to the new point are searched. k represents the amount of the closest neighbors of the unknown point. The quantity k of the algorithm (k = 1 in this study) is chosen to predict the results (Şahan et al., 2007; Filho et al., 2014).

The obtained results with non-weighted features

The classification results of the MI dataset made without applying the weighting process for the kNN classifier are given in Table 4. Where EEG signal gave the highest result for the kNN classifier, it remained at 56.781%.

The classification results of the MI dataset made without applying the weighting process for the LDA classifier are given in Table 5. Similarly, the EEG signal gave the highest result for the LDA classifier; it remained 60.460%.

The classification results of the MI dataset made without applying the weighting process for the SVM classifier are given in Table 6. Again, the EEG signal gave the highest accuracy for the SVM classifier; it remained 60.402%.

The classification results of the MA dataset made without applying the weighting process are given in Table 7. Where EEG data gave the highest Accuracy rate for the kNN classifier, it remained at 62.701%.

The obtained classification results on the MA dataset without applying the weighting process are given in Table 8. HbO data gave the highest accuracy rate for the LDA classifier, it remained at 66.332%.

The classification results of the MA dataset made without applying the weighting process are given in Table 9. Where EEG+HbO data gave the highest Accuracy rate for the SVM classifier, it remained at 74.138%.

Figure 5 shows the classification results of the non-weighted MI and MA tasks for all three classifiers. Although the MA task gave higher accuracy than the MI task, it remained at fairly low levels.

The obtained results with weighted features

In Fig. 6, the data distribution for feature 1 and feature 2 for the EEG+HbO signal belonging to the randomly selected MI task is given. Figure 6A shows the distribution of the unweighted data, Fig. 6B the KMCC-based weighted data distribution, and Fig. 6C the KMCCD-based weighted data distribution. The separation of weighted data can be insight. Figure 7 shows the comparison of 1st and 2nd features for non-weighted and weighted data of MA tasks HbO features set.

k-means clustering centers based attribute weighting method (KMCC-based)

Table 10 shows the feature datasets’ results for the MI task for which KMCC based weighting algorithm is applied. The EEG+HBR data reaches an accuracy rate of 99.540%.

Table 11 shows the KMCC based weighted MI dataset for LDA classification results. The EEG data give the highest value for the LDA classifier, the same as non-weighted EEG data for the LDA classifier with 97.816%.

Table 12 shows the KMCC based weighted MI dataset for SVM classifier results. The fNIRS hybrid data give the highest value for the SVM classifier; it remained at 99.943%.

Table 13 shows the KMCC based weighted MA dataset for the kNN classifier results. The fNIRS hybrid data give the highest value for EEG+HbR hybrid data and it remained at 98.793%.

Table 14 shows the KMCC based weighted MA dataset for LDA classification results. The fNIRS hybrid data give the highest value for the LDA classifier; it remained at 97.356%.

Table 15 shows the KMCC based weighted MA dataset for SVM classification results. The fNIRS hybrid data give the highest value for EEG+HbR hybrid data, and it remained at 99.655%.

All classification results for MI and MA tasks are given comparatively in Fig. 8 for the KMCC-based weighted algorithm.

k-means clustering centers difference based attribute weighting method (KMCCD-based)

Table 16 shows the feature datasets’ results for the MI task for which KMCCD based weighting algorithm is applied for the kNN classifier. The kNN classifier, which has the lowest accuracy rates in the non-weighted classification process, reached an accuracy rate of 99.655% (for EEG+HbR features) as in KMCC.

Table 17 shows the KMCCD based weighted MI dataset for LDA classification results. The EEG data give the highest value for the LDA classifier, and it remained at 96.724%.

Table 18 shows the KMCCD based weighted MI dataset for SVM classification results. The EEG + HbO data give the highest value for the SVM classifier, and it remained at 99.080%.

Table 19 shows the KMCCD based weighted MA dataset for kNN classification results. The EEG+HbR data give the highest value for the kNN classifier, and it remained at 99.885%.

Table 20 shows the KMCCD based weighted MA dataset for LDA classification results. The EEG + HbR data give the highest value for the LDA classifier, which remained at 98.793%.

Table 21 shows the KMCCD based weighted MA dataset for SVM classification results. The EEG+HbR data give the highest value for the LDA classifier, and it remained at 99.943%.

All classification results for MI and MA tasks are given comparatively in Fig. 9 for the KMCCD-based weighted algorithm. Higher performances of EEG signal features and Hybrid features are seen.

The Classification Error-values are close to 0 here indicates that the rate of making an error in the label selected for externally entered data is low.