Gene selection based on adaptive neighborhood-preserving multi-objective particle swarm optimization

Multi-objective optimization problems

Several biomedical enhancement obstacles require synchronized optimization of distinct multiple objectives, known as multi-objective problems (MOPs) (Kong et al., 2023; Peng & Guo, 2023). MOPs have two main goals: the diversity and convergence of Pareto optimal solutions (POSs). The aim of diversity is to optimize a set of POSs as distinct as possible and convergence is to optimize a set of optimal solutions as close as possible to the Pareto front (Yang et al., 2020). Generally, the minimization problem for MOP is articulated as:

(1) $$MinimizeY=f\left(x\right)=f_1\left(x\right),f_2\left(x\right),\dots .,f_m\left(x\right)$$

$$Subjecttox\mathrm{ϵ}\mathrm{\Omega }$$where $\mathrm{\Omega }$ is the decision space and $Y$ is objective space. $y=\left(y_1,y_2,\dots ,y_m\right)\mathrm{ϵ}Y$ is a $m$-dimension objective vector and $x=\left(x_1,x_2,\dots ,x_D\right)\mathrm{ϵ}\mathrm{\Omega }$ is an $D$-dimension decision variable vector. Several definitions associated with multi-objective optimization are given as follows:

Definition 1 (Pareto optimal): A decision vector $x^{\ast }\mathrm{ϵ}X$ is said to be Pareto optimal if there is no other $x\mathrm{ϵ}X,$ such that $x\prec x^{\ast }$.

Definition 2 (Pareto optimal set): For a given MOPs, the Pareto optimal set is defined as:

(2) $$POS=\{x\in X|\nexists z\in X,z\prec x\}$$

Definition 3 (Pareto front): For a given MOPs, the Pareto front set is defined as:

(3) $$PF=\{F\left(X\right)|x\in PS\}$$

Further in multi-modal multi-objective (MMO) problems, a “global POS” is a group of solutions where none of them is outperformed by any other solutions in the entire possible range. On the other hand, a “local POS” is a group of solutions where none of them are dominated by their close neighbors. For a multi-objective optimization problem to be considered a MMO problem, it needs to meet one of following two conditions: either possess at least one local POS or have more than one global POS.

It can be realized from aforementioned that a global POS is distinct from a local. Consequently, if a problem includes a local POS, it necessarily includes at least one global POS. In rare situations, the local POS might consist of only one solution.

Particle swarm optimization

PSO is a single-objective computational procedure; encouraged by collective actions of the birdies that can be defined as a repeatedly budding system (Kumar, Pandey & Ahirwal, 2023; Li et al., 2023). It is a community-based speculative procedure that aims to optimize the ideal solution to a specified optimization problem. PSO has been used in various biomedical studies such as gene selection and cancer classification due to its efficacy of fast convergence and simplicity of employment (Chen et al., 2014; Jain, Jain & Jain, 2018).

PSO operates through two primary components: the exploration aspect, signified by the inertia weight $\left(w\right)$ and the cognitive coefficient $\left(c_1\right)$, dictating an individual’s peak position influence; and the exploitation facet, denoted by the social coefficient $(c_2)$, governing the effect of the swarm’s peak position. PSO implementation mainly includes: initialization of each individual, main loop that contains iterations for updating velocities of these individuals according to peak positions as well as global positions and determine the fitness function, and finally termination. Let $x_i$ and $v_i$ is the current position and current velocity of the particle and $r_1$ and $r_2$ are arbitrary numbers in the range [0, 1]. Then, particles travel conforming to the following equations throughout the exploration process.

(4) $$v_i=w{v}_x+c_1{r}_1\left({p^{peak}}_i-x_i\right)+c_2{r}_2\left({g^{peak}}_x-x_i\right)$$

(5) $$x_i=x_i+v_{i}1\le i\le n.$$

By iteratively updating the individual positions based on their velocities, PSO aims to converge towards the optimal solution by balancing exploration and exploitation. The algorithm’s success depends on carefully selecting the values for the parameters $\left(w,c_1,c_2\right)$ and the appropriate termination criterion for the specific problem at hand. Also, it can be noted that the PSO algorithm is versatile and can be adapted to different optimization problems by customizing the fitness evaluation function and problem-specific constraints.

Sobol sequence

Sobol sequence (SS) is a type of quasi-random sequence used in numerical computations, particularly in Monte Carlo simulations and optimization algorithms (Atanassov & Ivanovska, 2022). Unlike pseudo-random sequences generated by traditional methods, such as the linear congruential generator, SS offers superior properties in terms of low discrepancy and high-dimensional uniformity. Developed by the Russian mathematician Ilya M. Sobol, this sequence is designed to cover the entire domain space more evenly, leading to improved convergence rates and accuracy in numerical integration and optimization tasks (Hu et al., 2024; Zhang et al., 2024). Sobol Sequence finds applications in various fields, including finance, engineering, and computer graphics, where high-quality random number generation is crucial for achieving reliable and efficient results.

Firstly, the number of dimensions $d$ and the number of points to generate $n$ are chosen. Then, direction numbers are computed and stored for each dimension $j$, denoted as ${\left\{v_{j,i}\right\}}_{i=1}^{\mathrm{\infty }}$. For each dimension $j$, it generates the SS ${\left\{s_{j,i}\right\}}_{i=1}^n$ using the following formula:

(6) $$s_{j,i}={\displaystyle \frac{g\left(j\right)}{2^j}\oplus {\displaystyle \frac{g\left(j+1\right)}{2^{j+1}}\oplus \dots \oplus {\displaystyle \frac{g\left(j+m-1\right)}{2^{j+m-1}}}}}$$where $g\left(j\right)$is the integer representation of the Gray code of j, $\oplus$ represents the bitwise exclusive OR operation, $m$ is the number of bits required to represent $i$ in binary. The generated Sobol points ${\left\{s_{j,i}\right\}}_{i=1}^n$ are typically transformed to the unit interval $\left[0,1\right]$ to represent points in the search space. Finally, the Sobol points in the unit interval can be transformed to the actual search space based on the bounds and constraints of the optimization problem. This process ensures that the points in the SS are distributed evenly across the unit hypercube, leading to improved coverage and convergence properties compared to pseudo-random sequences.

Dimensionality reduction using WNPEE

To address the challenges of high-dimensional microarray data, we employ WNPEE as a robust dimensionality reduction technique. Unlike traditional methods that rely on a single graph to capture data relationships (Mehta, Zhan & Shen, 2019; Qin et al., 2024), WNPEE constructs an ensemble of $L$ adjacent graphs $G_1,G_2,\dots ,G_L$, each representing the local structure of the data with varying neighborhood sizes $k$. This ensemble approach enhances neighborhood preservation and improves robustness to parameter sensitivity. The graphs are constructed using the K-nearest neighbors (KNN) method, where a directed edge is created between nodes if one node lies within the neighborhood of the other. For each graph, edge weights are computed by minimizing:

(7) $$min\sum _i{\left||x_i-\sum _j\sum _k{\alpha }_k{W}_{k,ij}{x}_j|\right|}^2,s.t\sum _j{W}_{k,ij}=1,j=1,2,\dots ,N$$where $W_k$ is $N\times N$ weight matrix for each adjacent graph, with $W_{k,ij}$ representing the weight of the edge from node $i$ to $j$, and 0 if no edge exists. The reduced representation $Y$ is obtained by solving the eigenvalue problem:

(8) $$XM{X}^T\mathbf{a}=\lambda X{X}^T\mathbf{a}$$where $M={\left(I-\sum {\alpha }_k{W}_k\right)}^T\left(I-\sum {\alpha }_k{W}_k\right)$. The transformation matrix $A=[{\mathrm{a}}_0,{\mathrm{a}}_1,\dots ,{\mathrm{a}}_{\mathrm{d}-1}]$ maps the high-dimensional data $X$ into a reduced-dimensional space $Y=A^{T}X$, preserving local structures. The graph weights ${\alpha }_k$ are iteratively optimized to minimize reconstruction loss, ensuring that the ensemble accurately represents the data’s locality, as

(9) $${\mathrm{\alpha }}_k={\displaystyle \frac{\sqrt[r-1]{1/tr\left({\mathrm{\Sigma }}_k\right)}}{{\sum }_{j=1}^L\sqrt[r-1]{1/tr\left({\mathrm{\Sigma }}_j\right)}}}$$where $r>1$ is a control parameter for balancing multiple graphs. This iterative process continues until the loss function converges. The detailed pseudocode of WNPEE is presented in Algorithm 1. By incorporating WNPEE, the proposed method ensures that the optimization process operates on a compact and biologically meaningful representation of the data, improving classification performance and reducing computational overhead.

Initialization

Following pre-processing, the population of gene subsets is initialized using an SS as outlined in previous section. This approach provides a more efficient and structured initialization strategy compared to conventional random methods. Each individual in the population, denoted as $Y_i$, is generated as follows:

(10) $$X_i=X_{min}+s_{j,i}\times \left(X_{max}-X_{min}\right)$$where $X_{max}$ and $X_{min}$ are the upper and lower bounds, respectively. The parameter $s_{j,i}$ represents the $ith$ random number generated by the SS as calculated using Eq. (6) within the interval [0, 1]. From the generated initial population, the current optimal gene subset is selected by evaluating fitness values, ensuring that only high-quality solutions proceed. SS enhances the systematic exploration of the solution space, leading to faster convergence and improved overall performance of the MOPSO optimization algorithm. An overview of the initialization scheme in the proposed ANPMOPSO is shown in Fig. 2.

Fitness evaluation

Once the population is initialized, each particle’s solution is evaluated based on multiple objectives, including classification accuracy, gene count, and neighborhood preservation quality. These objectives provide a comprehensive assessment of gene subsets, ensuring that selected genes maintain predictive performance while capturing relevant biological insights. In ANPMOPSO, we calculate the classification accuracy $CA\left(X_i\right)$ as the ratio of correctly classified instances to the total number of instances in the validation set.

(11) $$CA\left(X_i\right)={\displaystyle \frac{Numberofcorrectlyclassifiedinstances}{Totalnumberofinstances}\times 100.}$$

The number of selected genes $NF\left(X_i\right)$, in a gene subset can be simply measured by counting the number of genes with a value of 1 in the binary representation of the subset. Let’s denote $X_i$ as a binary vector of length $L$, where $L$ is the total number of genes in the dataset. Each element $x_{ij}$ of $X_i$ represents whether gene $j$ is selected or not. $x_{ij}=1$ if gene $j$ is selected in subset $X_i$, and $x_{ij}=0$ otherwise. The number of selected genes $NF\left(X_i\right)$ in subset $X_i$ can be calculated as:

(12) $$NF\left(X_i\right)=\sum _{j=1}^L{x}_{ij}.$$

Let $NP\left(X_i\right)$ represent the neighborhood preservation quality of solution $X_i$ and we compute it as the average preservation score across all data points:

(13) $$NP\left(X_i\right)={\displaystyle \frac{1}{N}\sum _{i=1}^N{\displaystyle \frac{1}{k}\sum _{j=1}^k||x_{ij}-y_{ij}||}}$$where $N$ is the total number of data points, $k$ is the number of nearest neighbors considered, $x_{ij}$ and $y_{ij}$ are the $j^{th}$ nearest neighbor of data point $i$ in the original and embedded spaces, respectively, and $\parallel \cdot \parallel$ denotes Euclidean distance (ED), measuring the difference between corresponding neighbors. Therefore, this metric quantifies how well the local neighborhood structure is preserved. A lower value of $NP\left(X_i\right)$ indicates better preservation quality, meaning that the local structure is more faithfully retained in the embedded space. The overall fitness of each particle is computed based on the three defined objectives, with equal weights, as:

(14) $$Fitness{X}_i=\left\{CA\left(X_i\right),NF\left(X_i\right),NP\left(X_i\right)\right\}.$$

Optimization

The optimization loop of ANPMOPSO employs MOPSO with DE-mutation to iteratively refining gene subsets. Each particle in the population maintains a set of solutions representing the Pareto front, which is a set of non-dominated solutions in the objective space. During each iteration, particles dynamically adjust their velocity using DE-mutation inspired mechanisms, facilitating diversity and exploration of the solution space. Particle positions are then updated based on their personal best and the global best found by the entire swarm. The fitness of each particle is evaluated based on multiple objectives, including classification accuracy, gene count, and neighborhood preservation quality. By concurrently optimizing these competing objectives, ANPMOPSO aims to identify diverse, high-quality gene subsets, offering meaningful trade-offs between predictive performance, subset compactness, and local structure preservation. The update of particle velocities in ANPMOPSO is governed as:

(15) $$v_{i,d}\left(t+1\right)=w{v}_{i,d}\left(t\right)+c_1{r}_1\left(pbes{t}_{i,d}-x_{i,d}\left(t\right)\right)+c_2{r}_2\left(gbes{t}_d-x_{i,d}\left(t\right)\right)+DE\mathrm{\_}{v}_{i,d}$$where $v_{i,d}\left(t\right)$ represents the velocity of particle $i$ in dimension d at time t, $w$ is the inertia weight controlling the impact of the previous velocity, $c_1$ and $c_2$ are acceleration coefficients representing the cognitive and social components, respectively, $r_1$ and $r_2$ are random values sampled from a uniform distribution in the range [0, 1]. $pbes{t}_{i,d}$ is the personal best position of particle $i$ in dimension d, $gbes{t}_d$ is the global best position found by the swarm in dimension d, and $D{E}_{v_{i,d}}$ represents the velocity update obtained from the DE-mutation based adaptive mechanism.

The DE-mutation based adaptive velocity update is computed by randomly selecting three other particles $X_{r1},X_{r2},$ and $X_{r3}$ from the population for each particle $X_i$. A mutant vector $V_{mut}$ is generated by perturbing the quartet using DE-mutation strategy (DE/rand/1) as:

(16) $$V_{mut}=X_{r1}+F\times \left(X_{r2}-X_{r3}\right).$$

This mutant vector is then used to update the particle’s velocity adaptively:

(17) $$D{E}_{v_{i,d}}=K\times (V_{mut}-x_{i,d}\left(t\right)).$$

Here, $F$ is the differential weight factor, $K$ is an adaptive scaling factor determined based on the fitness of the quartet, and $x_{i,d}\left(t\right)$ is the current position of particle $i$ in dimension $d$ at time $t$. This adaptive adjustment ensures exploration in regions with potentially better solutions, preventing premature convergence and enhancing diversity. Once the adaptive velocity is obtained, the particle’s position is updated as:

(18) $$x_{i,d}\left(t+1\right)=x_{i,d}\left(t\right)+v_{i,d}\left(t+1\right).$$

Additionally, we ensure that position components are within minimum and maximum bounds $mi{n}_{pos}$ and $ma{x}_{pos}$, respectively. Further, we evaluate each particle’s fitness based on the defined objectives, maximizing classification accuracy, minimizing the number of selected genes, and maximizing neighborhood preservation in the embedded space. This approach allows for the capture of a diverse set of trade-off solutions representing various combinations of classification accuracy, gene count, and neighborhood preservation quality.

A novel selection strategy

After updating the particles’ velocity and position, the newly and previous generated particles are gathered into an external archive. However, given the limited archive size and the necessity to uphold particle quality, it is crucial to implement a suitable selection strategy for archive updates. This selection strategy, employed within the archive, directs the particles search towards the genuine Pareto front, guaranteeing that only the most promising solutions are preserved for further exploration.

Therefore, we propose a novel selection approach that considers both the proximity to the Pareto front and the quality of neighborhood preservation, contrasting with traditional MOPSO methods that focus solely on proximity. This approach evaluates each solution based on its harmonic mean distance (HMD) to the Pareto front, as opposed to ED used in existing techniques, and its neighborhood preservation quality. By integrating this selection criterion, our method ensures to prioritize solutions that not only achieve good trade-offs among multiple objectives but also preserve meaningful local structures within the microarray data. Therefore, to define a novel selection strategy, we aim to balance two aspects:

• Proximity to the Pareto front: In contrast to ED in traditional MOPSOs, which considers only the geometric distance between points in the objective space, the HMD accounts for both the distance to the Pareto front and the diversity of solutions across multiple objectives. By incorporating HMD, our approach effectively diversifies the selection process, ensuring that a broader set of trade-off solutions is explored. Let $HMD\left(X_i\right)$ represent the average HMD of solution $X_i$ to the Pareto front, it can be calculated as: (19) $$HMD\left(X_i\right)={\displaystyle \frac{n}{{\sum }_{j=1}^n{\displaystyle \frac{1}{d\left(X_i,X_j\right)}}}.}$$

• Neighborhood preservation quality: Solutions with better neighborhood preservation in the embedded space are preferred. We compute the neighborhood preservation quality $NP\left(X_i\right)$ for solution $X_i$ as the average preservation score across all data points using Eq. (15).

Finally, we define a combined score $S\left(X_i\right)$ for each solution $X_i$ as a weighted sum of its HMD to the Pareto front and its neighborhood preservation quality:

(20) $$S\left(X_i\right)=\beta \times HMD\left(X_i\right)+\left(1-\beta \right)\times NP\left(X_i\right)$$where $\beta$ controls the relative importance of $HMD\left(X_i\right)$ and $NP\left(X_i\right)$ in the optimization process, higher value of $\beta$ prioritizes $HMD\left(X_i\right)$, while a lower value prioritizes $NP\left(X_i\right)$. Solutions with lower $S\left(X_i\right)$ values are preferred, as they provide a better balance between proximity to the Pareto front and local structure preservation. By integrating this novel selection strategy, our method can effectively identify high-quality solutions that achieve a balance between the multiple objectives of the gene selection while capturing relevant biological insights encoded in the microarray data. Algorithm 2 presents the detailed pseudocode for proposed ANPMOPSO.

Dataset

To evaluate the performance of the proposed ANPMOPSO framework, we conducted experiments on six publicly available microarray datasets: Brain Cancer, Colon, Leukemia, Lung, Lymphoma, and Small-Round-Blue-Cell Tumor (SRBCT). These datasets were obtained from two primary sources: http://csse.szu.edu.cn/staff/zhuzx/Datasets.Html, and https://github.com/Pengeace/MGRFE-GaRFE. A detailed description of these datasets is presented in Table 2.

These datasets are commonly used for gene selection and classification tasks due to their high dimensionality and biological relevance. All datasets were preprocessed to remove missing values and normalized using min-max scaling to ensure consistency across experiments.

Hyperparameters

In our experiments, we set the key hyperparameters as:

• Swarm size: 200 particles (optimized for diversity and computational efficiency).

• Iterations: 50 generations (evaluated 30–100 generations; 50 iterations ensured convergence where error rate stabilization ±1% beyond 50).

• Sobol sequence initialization: Dimensions matched the gene count of each dataset, with quasi-random numbers generated in the range [0, 1].

• DE mutation rate (F): 0.5 (selected via grid search).

• Neighborhood size (k): 10 and parameter $\alpha$: 0.5 (optimized for local structure preservation in WNPEE).

• Inertia weight (w): 0.7298 (standard PSO configuration).

• Acceleration coefficients ( $c_1,c_2$): 1.49445 (empirically validated for balance).

• Parameter (β): 0.5 (via optimally balances HMD and NP for the lowest 1/PSP values).

Other parameters were configured according to corresponding references (Rostami et al., 2020; Shukla, Singh & Vardhan, 2020; Azadifar & Ahmadi, 2021; Madani, Engelbrecht & Ombuki-Berman, 2023). To ensure fair comparison and reliable evaluation, these experimental settings were applied consistently across all datasets.

The effectiveness of ANPMOPSO was benchmarked against state-of-the-art metaheuristic algorithms, including MaPSOGS (Azadifar & Ahmadi, 2021), Teaching–Learning-Based Optimization and Gravitational Search Algorithm for Feature Selection (TLBOGSA) (Shukla, Singh & Vardhan, 2020), CCMGPSO (Madani, Engelbrecht & Ombuki-Berman, 2023), and MPSONC (Rostami et al., 2020). All experiments were conducted on a desktop system equipped with an Intel Core i7 processor (2.4 GHz) and 16 GB RAM, utilizing MATLAB 2018a (The MathWorks, Natick, MA, USA) as the development environment.

Performance metrics

To comprehensively evaluate the performance of the proposed ANPMOPSO framework, the following metrics were used:

• 1/Hypervolume (1/HV): This metric is used to evaluate the convergence and diversity of the Pareto front generated by ANPMOPSO. The hypervolume (HV) is calculated as: (21) $$HV=vol\left({\cup }_{x\in PF}\left[x,r\right]\right)$$where PF is the set of non-dominated Pareto solutions and r is the reference point in objective space. The inverse is used for minimization. Lower values of 1/HV indicate better convergence and spread of the Pareto front.

• Classification accuracy (ACC): Classification accuracy is used to quantify the predictive performance of the selected gene subsets, defined in Eq. (11).

• Error rate: The error rate is the complement of accuracy and is used to monitor the algorithm’s convergence during optimization: (22) $$Errorrate=1-Acc$$

• Neighborhood preservation score (NP): NP evaluates the proportion of nearest neighbors in the original space that remain neighbors in the reduced space same as Eq. (13). A higher NP score indicates better local structure retention.

• 1/PSP (Pareto Sets Proximity): Pareto Sets Proximity (PSP) quantifies the proximity of obtained Pareto solutions to the true or ideal Pareto front. The PSP is calculated as: (23) $$PSP={\displaystyle \frac{1}{\left|P\right|}\sum _{x\in P}\underset{y\in P{F}^{\ast }}{min}\parallel x-y{\parallel }^2}$$where P is the set of obtained Pareto solutions, and $P{F}^{\ast }$ is the true Pareto front or its approximation. The lower the PSP, the closer the solutions are to the optimal front. We use 1/PSP to maintain consistency with our minimization objective.

• Average neighborhood overlap (ANO): ANO measures how well the local neighborhood of each point is retained after dimensionality reduction. (24) $$ANO={\displaystyle \frac{1}{N}\sum _{i=1}^N{\displaystyle \frac{\left|N_k^H\left(i\right)\cap N_k^L\left(i\right)\right|}{k}}}$$where: $N_k^H\left(i\right)$ and $N_k^L\left(i\right)$ are sets of k-nearest neighbors of point $i$ in the high-dimensional space and in the low-dimensional space, and $\left|N_k^H\left(i\right)\cap N_k^L\left(i\right)\right|$ is the number of overlapping neighbors between the original and reduced feature space. A higher ANO score suggests better neighborhood overlap, implying that dimensionality reduction minimally distorts the original local structure.

• Spearman’s rank correlation (SRC): SRC evaluates whether the relative ordering of pairwise distances between data points is preserved after dimensionality reduction. It is calculated as: (25) $$SRC=1-{\displaystyle \frac{6{\sum }_{i=1}^N{d}_i^2}{N\left(N^2-1\right)}}$$where $d_i$ is difference in rank for point $i$ before and after dimensionality reduction and $N$ is the total number of data points. A higher SRC (close to 1.0) suggests that the relative ranking of distances between points is well-preserved in the reduced space.

These metrics collectively assess not only classification performance but also solution quality, biological relevance, and structural preservation.

Comparative analysis of hypervolume values

Table 3 presents the mean ± standard deviation of 1/hypervolume (1/HV) values obtained by five multi-objective optimization algorithms: ANPMOPSO, Multi-objective Particle Swarm Optimization with Adaptive Strategies for Feature Selection (MOPSO-ASFS) (Han et al., 2021), Omni-optimizer (Deb & Tiwari, 2005), Decision Space Niching-based Non-dominated Sorting Genetic Algorithm II (DN-NSGAII) (Li, Wu & Tan, 2023), and Multi-objective Particle Swarm Optimization with Ring Topology for Solving Multimodal Multi-objective Problems (MO_Ring_PSO_SCD) (Yue, Qu & Liang, 2017), across eleven benchmark test functions. These test problems are derived from Yue, Qu & Liang (2017), and all algorithms are evaluated under identical experimental conditions. Each experiment is repeated 15 times for robustness, and the performance is assessed using the 1/HV performance indicator, where a lower value indicates superior performance. The population size is determined as $100\times N_{var}$, and the maximum fitness evaluations are set at $4,000\times N_{var}$. In the ANPMOPSO algorithm, the $F$ parameter in Eq. (16) is set to 0.5, and the crossover rate is also 0.5.

The results from Table 3 demonstrate that ANPMOPSO consistently outperforms other optimization algorithms across most benchmark functions, with the exception of MMF7 and SYM-PART simple. In particular, ANPMOPSO achieves the lowest 1/HV values for MMF1, MMF2, MMF4, MMF5, MMF6, and MMF8, highlighting its capability in solving complex multi-objective problems efficiently. Notably, on MMF8, it significantly outperforms other methods (2.3660 ± 0.05931), reinforcing its effectiveness in handling challenging problem landscapes. This consistent superiority suggests that ANPMOPSO is well-suited for diverse optimization scenarios, particularly those requiring robust exploration and exploitation strategies.

However, ANPMOPSO falls slightly behind on MMF7 and SYM-PART simple functions, where MOPSO-ASFS performs slightly better on MMF7 (1.1412 ± 0.9383), and MO_Ring_PSO_SCD marginally outperforms others on SYM-PART simple (0.75 ± 1.65e−03). The close results in SYM-PART functions indicate that these problems do not strongly differentiate algorithm performance. Nonetheless, ANPMOPSO’s dominance in most other functions demonstrates its ability to generate high-quality Pareto-optimal solutions and effectively handle multimodal and rotationally symmetric optimization problems. This underscores its robustness and adaptability in solving diverse multi-objective optimization challenges.

Comparison of true and obtained Pareto-optimal solutions

The true and obtained Pareto-optimal solutions (POSs) for the proposed ANPMOPSO algorithm across all eleven MMFs are shown in Figs. 3A–3K. The results illustrate how well the ANPMOPSO algorithm approximates the true Pareto fronts. In most cases, the obtained POSs (red circles) closely follow the true POSs (blue diamonds), indicating that ANPMOPSO effectively converges toward optimal solutions. This is particularly evident in MMF4, MMF5, and MMF6, where the alignment is nearly perfect, demonstrating the algorithm’s capability in solving multimodal and rotationally symmetric functions. However, some variations are observed in MMF1, MMF7, and MMF9, where minor deviations suggest challenges in fully capturing highly complex or oscillatory POSs.

Despite these minor discrepancies, the overall strong agreement between the true and obtained POSs confirms ANPMOPSO’s effectiveness in handling diverse multi-objective problems. The observed differences in certain MMFs suggest that additional refinements in exploration strategies or diversity maintenance could further enhance performance. Nonetheless, these results validate the robustness of ANPMOPSO in maintaining convergence and solution accuracy, making it a competitive approach for solving multi-objective optimization problems.

Comprehensive gene selection comparison

To evaluate the predictive performance of the selected gene subsets, the extreme learning machine (ELM) classifier was used across six microarray datasets. Each experiment was repeated 50 times, and the mean classification accuracies were recorded to ensure the statistical robustness of the results. As shown in Table 4, ANPMOPSO consistently achieved high classification accuracy, with notable performance in Leukemia and SRBCT datasets, where it attained 100% accuracy across all evaluation metrics. This demonstrates the algorithm’s capability in identifying the most discriminative genes, leading to highly accurate predictions with a minimal subset of features.

For other datasets such as Brain Cancer, Colon, Lung, and Lymphoma, the proposed method maintained competitive classification performance, with test accuracies frequently exceeding 90%. The five-fold cross validation (CV) and Leave-One-Out Cross-Validation (LOOCV) accuracies further confirm the generalization ability of ANPMOPSO, as they closely align with test accuracy values, minimizing concerns about overfitting. The variation in selected gene subsets across datasets highlights the adaptability of the algorithm in capturing dataset-specific patterns. While minor fluctuations in accuracy were observed in some cases, the overall performance indicates that ANPMOPSO is capable of effectively handling high-dimensional biological data while maintaining robust predictive accuracy. These findings validate the effectiveness of ANPMOPSO as a gene selection approach, demonstrating its ability to reduce dimensionality while preserving high classification performance.

Comprehensive biotic attributes analysis of selected genes set

The experiment was conducted 20 times, and the top 10 genes that were consistently selected across six microarray datasets are summarized in Table 5. Notably, the ANPMOPSO approach consistently identified several genes that are in line with methodologies proposed in previous studies (Han, Sun & Ling, 2014; Han et al., 2015, 2019; Xiong et al., 2019; Shah et al., 2020; Shukla, Singh & Vardhan, 2020; Lai & Huang, 2021; Aziz, 2022). Many of these genes were commonly selected by the existing gene selection methods, but there were specific genes, highlighted in bold, exclusively identified by ANPMOPSO, distinguishing it from traditional gene selection methods. These genes include N70358, N03128, L13852, D16471, 34012_at, 37899_at, 31314_at, X14046, D28151, M31572, N80914, U66559_at, X67098, and M90391, suggesting that ANPMOPSO can uncover novel biomarkers that may not be easily detected by other selection techniques.

For instance, in Leukemia data, an important gene, D16471, associated with MRNA, Xq terminal portion, has implications for various disease diagnosis. Another notable gene, N70358 is found that is associated with the growth hormone receptor (GHR). GHR plays a crucial role in regulating various physiological processes. Deregulation of GHR signalling has been linked to a range of diseases and chronic conditions, including acromegaly, cancer, aging, metabolic disorders, fibroses, inflammation, and autoimmunity. Also, 37899_at, thymidylate synthetase (TS) is an essential enzyme involved in DNA synthesis; serve as biomarkers for predicting the response to certain chemotherapy regimens. Additionally, TS polymorphisms have been studied for their potential role in modulating drug response and toxicity. Another significant discovery is the potential of anaplastic lymphoma kinase (Ki − 1) as a therapeutic target in cancer treatment. In cancers where Ki − 1 is aberrantly activated or overexpressed, targeted therapies such as Ki − 1 inhibitors have demonstrated efficacy in preventing tumour growth and enhancing patient outcomes. These findings highlight the potential of ANPMOPSO in identifying novel genes that can be valuable for prospective investigations in these microarray datasets.

Further, to assess the efficacy of our proposed method in gene selection, we showcase heat-maps illustrating the top ten frequently chosen genes across six datasets, as depicted in Fig. 4. The color intensity represents gene expression levels, where darker shades indicate lower expression, and lighter shades signify higher expression. Clear differences are observed between NORMAL vs. TUMOR (Colon), CURED vs. FATAL (Lymphoma), and AML vs. ALL (Leukemia), demonstrating the biological relevance of the selected genes. Distinct expression patterns across subtypes, such as EWS, BL, NB, and RMS in Brain Cancer and ADE, NOR, SQ, and PC in Lung Cancer, confirm that ANPMOPSO effectively identifies genes capable of distinguishing cancer subtypes. These findings highlight the robustness of the proposed method in selecting informative biomarkers for classification and potential therapeutic targets.

Gene selection results analysis

To assess the effectiveness of ANPMOPSO, we compared it with existing state-of-the-art gene selection techniques, including MaPSOGS (Azadifar & Ahmadi, 2021), TLBOGSA (Shukla, Singh & Vardhan, 2020), CCMGPSO (Madani, Engelbrecht & Ombuki-Berman, 2023), and MPSONC (Rostami et al., 2020), using the ELM classifier on six microarray datasets. As shown in Table 6, the evaluation was based on average testing accuracy, standard deviation, and the number of selected genes over 20 runs. ANPMOPSO consistently outperformed competing methods in five out of six datasets, achieving 100% accuracy on Leukemia and SRBCT with only three and five genes selected, respectively. This highlights ANPMOPSO’s efficiency in selecting minimal yet highly discriminative gene subsets for classification. While MPSONC achieved slightly higher accuracy (97.82%) on the Colon dataset, it required a larger gene subset, demonstrating a trade-off between accuracy and feature selection efficiency.

The results validate ANPMOPSO’s superior performance in gene selection, effectively balancing high classification accuracy with minimal gene usage. The ability to maintain competitive accuracy across diverse datasets while reducing dimensionality enhances its utility for biomedical applications, biomarker discovery, and precision medicine. By consistently identifying the most relevant genes, ANPMOPSO offers a computationally efficient and interpretable approach for high-dimensional cancer classification, further reinforcing its potential in personalized healthcare and clinical decision-making.

To further assess the effectiveness of ANPMOPSO, we analyzed the error rates of different gene selection methods over multiple iterations on six microarray datasets, as shown in Fig. 5. The results indicate that ANPMOPSO consistently achieves the lowest error rates for Leukemia, SRBCT, Lung, Lymphoma, and Brain Cancer datasets, demonstrating its ability to efficiently identify optimal gene subsets with minimal classification errors. Its fast convergence in early iterations highlights its capability to quickly select highly informative genes while avoiding redundancy. Although MPSONC performed best on the Colon dataset, ANPMOPSO exhibited superior performance in most cases, reinforcing its robust generalization capability across different datasets. The lower error rates and stable convergence suggest that ANPMOPSO effectively balances exploration and exploitation, ensuring compact yet highly discriminative gene selection. These findings validate its effectiveness in high-dimensional gene expression analysis, positioning it as a promising tool for cancer classification and biomarker discovery.

Local structure preservation analysis

To assess the effectiveness of WNPEE in ANPMOPSO for preserving local structures during dimensionality reduction (DR), we compared its performance with three commonly used DR techniques: neighborhood preserving embedding (NPE), principal component analysis (PCA), and locally linear embedding (LLE). The evaluation was conducted using the six microarray datasets previously used for gene selection and results are recorded in Table 7. We employed three key metrics: ANO (Eq. (23)), NP (Eq. (22)) and SRC (Eq. (24)) to quantify local structure preservation.

The results in Table 7 demonstrate that WNPEE consistently outperforms NPE, PCA, and LLE across all three-evaluation metrics ANO, NP Score, and SRC for all six microarray datasets. The higher ANO values obtained by WNPEE, such as 0.945 for SRBCT, 0.923 for Brain Cancer, and 0.91 for Colon, indicate that WNPEE retains local neighborhood structures more effectively than the competing methods. Similarly, higher NP values, such as 0.894 for SRBCT and 0.862 for Colon, further confirm its ability to minimize structural distortion during dimensionality reduction. Additionally, the higher SRC values, including 0.959 for Lymphoma and 0.948 for Colon, verify that WNPEE preserves pairwise distance rankings, maintaining meaningful biological relationships in the reduced feature space.

Among the competing methods, NPE performs relatively well but consistently lags behind WNPEE, with slightly lower scores across all metrics. For example, in Leukemia, NPE achieves 0.756 (ANO), 0.791 (NP), and 0.84 (SRC), while WNPEE achieves 0.887, 0.855, and 0.925, respectively, demonstrating WNPEE’s superior local structure retention. In contrast, PCA and LLE exhibit the lowest scores, which aligns with their known limitations; PCA focuses on global variance rather than local structure, while LLE struggles with high-dimensional data, leading to inconsistent neighborhood preservation. The most significant improvements with WNPEE are observed in Leukemia, SRBCT, and Brain Cancer datasets, where local structures are well-defined and effectively preserved. These findings establish WNPEE as the most effective dimensionality reduction technique, ensuring that key biological structures are maintained while significantly reducing computational complexity and enhancing classification performance.

Neighborhood size analysis

The ANPMOPSO algorithm employs the DE/rand/1 strategy, which relies on three key individuals for the velocity update. Unlike traditional methods that select these individuals from the entire population, ANPMOPSO selects them from a subset of the neighborhood, providing a more localized search mechanism. To determine the optimal neighborhood size, we evaluate neighborhood sizes ranging from eight to 18 in increments of two, ensuring a sufficiently diverse pool of neighbors for selecting the three individuals as shown in Fig. 6.

Two test functions: SYM-PART-rotated and MMF5 (Yue, Qu & Liang, 2017), which are known to be sensitive to neighborhood size, are used for the analysis. The performance metric Pareto sets proximity (PSP) is employed to evaluate the overlap ratio and distance between the true and obtained Pareto sets. The metric 1/PSP is utilized, where smaller values indicate better performance in the solution space. Figure 6 depicts the relationship between 1/PSP and neighborhood size for both test functions. Based on the results, the recommended neighborhood size is determined to be 10, which balances performance and neighborhood diversity effectively. This value is consistent with the settings used in the proposed study.

Analysis of the influence of parameter β

The parameter $\beta$ plays a crucial role in balancing the relative importance of $HMD\left(X_i\right)$ and $NP\left(X_i\right)$ in the optimization process of ANPMOPSO. A higher $\beta$ value places greater emphasis on $HMD\left(X_i\right)$, while a lower value prioritizes $NP\left(X_i\right)$. To evaluate its effect on performance, we tested $\beta$ values ranging from 0.1 to 1.0 in increments of 0.1, using the MMF5 test function.

The results, presented in Fig. 7, demonstrate that ANPMOPSO achieves optimal performance at $\beta =0.5$ and $\beta =0.7$, with the lowest 1/PSP values observed at $\beta =0.5$. This indicates that $\beta =0.5$provides the best trade-off between $HMD$ and $NP$, ensuring superior performance and stability. Based on these findings, we recommend setting $\beta =0.5$ in our study to achieve optimal 1/PSP values with high consistency.

Ablation study: analysis of the effectiveness of key operations in ANPMOPSO

To evaluate the impact of three key operations in the proposed ANPMOPSO algorithm, comparative experiments were conducted using different algorithm variants, each omitting one of these operations. The operations assessed include (1) WNPEE pre-processing with initial population generation using SS, (2) adaptive DE-based velocity update, and (3) a novel selection strategy. The experiment was performed using the MMF3 test function (Yue, Qu & Liang, 2017), and the results from 15 independent runs were analyzed. The obtained POSs were evaluated using the median 1/PSP metric, visualized in Fig. 8, while the mean 1/PSP values over 50 generations for each algorithm variant are presented in Fig. 9. Specifically, the original ANPMOPSO integrates all three operations, whereas the modified versions were tested as follows: ANPMOPSO1 excludes WNPEE and SS-based population initialization, ANPMOPSO2 omits the adaptive DE-based velocity update, and ANPMOPSO3 lacks the novel selection strategy.

The analysis of Figs. 8 and 9 reveals that ANPMOPSO exhibits degraded performance when any of these key operations are omitted. ANPMOPSO1 shows convergence but underperforms compared to ANPMOPSO, indicating that WNPEE and SS contribute to improved search space exploration. ANPMOPSO2 demonstrates divergence, as it neglects $NP\left(X_i\right)$, preventing individuals from dynamically adjusting their positions, resulting in incomplete Pareto sets, as observed in Fig. 8B. ANPMOPSO3 struggles to identify the optimal solutions for gene selection, causing deviations from the true POSs, as shown in Fig. 8C. Overall, ANPMOPSO consistently outperforms all three variants, confirming that the inclusion of all three key operations is essential for achieving optimal Pareto solutions and maintaining high classification performance.

Statistical analysis

Table 8 presents the p-values obtained from a one-tailed t-test, assessing the statistical significance of performance differences among the five comparative algorithms across six microarray datasets. A lower p-value indicates a more significant performance difference, suggesting that the corresponding algorithm demonstrates a distinct advantage over others. The results reveal that ANPMOPSO consistently achieves the lowest p-values across all datasets, with values as low as 3.08E−11 in Leukemia, 3.99E−12 in Brain Cancer, and 4.81E−12 in Lymphoma, confirming its statistical superiority in gene selection effectiveness.

Among the competing methods, MPSONC also exhibits strong performance, particularly in Brain Cancer (3.99E−10) and Colon (3.98E−10), where it achieves relatively low p-values, indicating a significant performance advantage over the other three algorithms. However, TLBOGSA presents the highest p-values, particularly in Leukemia (1.76E−02) and Brain Cancer (2.09E−02), suggesting that its performance is less statistically significant compared to other methods.

To further validate the robustness and reliability of ANPMOPSO’s classification performance, we computed the 95% confidence intervals (CI) for classification accuracy across all datasets, recorded in Table 9. The narrow confidence intervals, such as [99.5–100%] for SRBCT and [95.9–97.1%] for Colon, confirm the precision and stability of ANPMOPSO’s results. These intervals indicate a minimal variance in the accuracy, further highlighting the consistent effectiveness of ANPMOPSO compared to benchmark algorithms.

The integration of confidence intervals as well as p-values reinforces ANPMOPSO as the most statistically robust algorithm, demonstrating a clear and consistent advantage across all datasets, while MPSONC remains competitive in certain cases, and TLBOGSA emerges as the least effective method based on statistical significance.

Computational complexity analysis

The computational complexity of ANPMOPSO arises from four main components: initialization, fitness evaluation, velocity and position updates, and the selection strategy. During initialization, WNPEE reduces data dimensionality with a complexity of $O\left(N^2+L^2\right)$, and Sobol Sequence-based population initialization adds $O\left(P\cdot L\right)$, where $N$ is the number of genes, $L$ is the reduced dimension, and $P$ is the swarm size. The fitness evaluation, which includes classification accuracy, gene subset size, and neighborhood preservation quality, is the most computationally intensive component, with a complexity of $O(P.\left(N^2.L+k.L^2\right)$, mainly due to distance computations. Velocity and position updates add $O\left(P\cdot L\right)$, while the selection strategy, incorporating HMD calculations, contributes $O\left(P^2.n\right)$, where $n$ is the number of Pareto front solutions. Thus, the total complexity is approximately $O\left(P.N^2.L\right)+O\left(P^2.n\right)$. Regarding inference time, ANPMOPSO functions as an offline gene selection algorithm. Once the optimal gene subset is selected, downstream classification tasks operate on a substantially reduced feature space, enabling fast and efficient inference.

Furthermore, we performed a runtime experiment to record the average training time over 20 runs for ANPMOPSO and four comparative algorithms across six benchmark microarray datasets. The results, summarized in Table 10, reveal that ANPMOPSO consistently requires the highest training time among all methods, ranging from 3.5 min on the Colon dataset to 7.4 min on the Lung dataset. This increase is expected due to the integration of WNPEE, Sobol initialization, and DE-mutation-driven velocity updates, which collectively enhance search efficiency and accuracy but introduce additional computational overhead. In contrast, TLBOGSA demonstrates the lowest runtime across all datasets, reflecting its simpler structure and lack of neighborhood-aware components. Although MPSONC achieves lower execution times than ANPMOPSO, it falls short in classification performance and gene subset compactness. These results confirm that ANPMOPSO’s higher computational cost is a worthwhile trade-off for improved classification accuracy and biological relevance, making it suitable for large-scale gene selection despite limitations in time-sensitive or ultra-high-dimensional settings.

Discussions and limitations

The ANPMOPSO framework demonstrates significant advancements in gene selection by harmonizing multi-objective optimization with local structural preservation. Experimental results validate its superiority, achieving 100% classification accuracy on Leukemia and SRBCT datasets using only 3–5 genes (Table 6), outperforming state-of-the-art methods by 5–15%. This success stems from its novel components: WNPEE preserved neighborhood structures with 94.5% ANO on SRBCT (Table 7), ensuring biologically interpretable dimensionality reduction; Sobol initialization enhanced population diversity, accelerating convergence; and the DE-based adaptive velocity update balanced exploration-exploitation dynamics, yielding superior hypervolume values (e.g., 1.0617 ± 0.2225 on MMF1, Table 3). The Pareto-neighborhood ranking strategy further prioritized solutions with clinical relevance, identifying biomarkers like D16471 and 37899_at (Fig. 4), which align with known disease pathways. These innovations address longstanding gaps in MOPSO methods, such as neglect of local structures and reliance on random initialization, positioning ANPMOPSO as a robust tool for high-dimensional biomedical data.

Despite its strengths, ANPMOPSO’s computational complexity driven by WNPEE’s $O\left(N^2\right)$ cost and fitness evaluations results in 1.5-2-time longer runtimes than simpler methods (e.g., 7.4 min for Lung data vs. 4.8 min for TLBOGSA, Table 10). This limits real-time applicability in ultra-high-dimensional contexts (e.g., single-cell RNA-seq). Additionally, fixed hyperparameters $\left(\beta =0.5,F=0.5,k=10\right)$ may not generalize to noisy datasets, and Sobol Sequences’ deterministic nature risks suboptimal exploration in irregular spaces. Future work should integrate GPU acceleration for WNPEE to optimize inference speed, meta-learning for dynamic parameter tuning, and stochastic perturbations to enhance robustness. Exploring applications to medical imaging (e.g., MRI/CT feature selection) and scaling to multi-omics data, alongside experimental validation of prioritized genes (e.g., CRISPR assays), will further bridge computational innovation with clinical utility, solidifying ANPMOPSO’s role in precision oncology.