Multi-objective genetic programming strategies for topic-based search with a focus on diversity and global recall

Research questions

This work describes and evaluates a framework that addresses the problem of formulating topical queries when multiple objectives need to be balanced. The analysis of the proposed framework is guided by the following research questions:

• RQ1 How effective are the queries evolved by each of the proposed strategies in comparison to the initial queries that are formulated directly from the description of the topic?

• RQ2 Are the queries evolved by the proposed strategies effective when tested for the same topic but on a new corpus? In other words, are the proposed strategies able to avoid overfitting?

• RQ3 What are the trade-offs among different MOGP strategies for topical search when attempting to address conflicting objectives such as high precision and recall?

• RQ4 How effective are the queries evolved by the proposed strategies in comparison to queries that are built using MOEA strategies?

• RQ5 How effective are the queries evolved by the proposed strategies in comparison to queries that are built using state-of-the-art term-weighting schemes?

• RQ6 If there are significant improvements in terms of effectiveness, then we aim at finding out if the computational time required is justifiable based on the effectiveness achieved.

Topic-based search

Topic-based search is the process of seeking and suggesting material based on a topic of interest, which is usually defined as a topical context (Leake & Scherle, 2001; Maguitman, 2018). Topical contexts may be obtained from the user’s long-term or current interests and can be defined at different levels of granularity. For instance, a topical context may be defined as a collection of documents related to a particular subject of interest. Alternatively, a user may highlight a few words in a document with the purpose of requesting information related to the highlighted text. This gives rise to a more focused view of the topical context, which may include the sentence containing the highlighted text, the paragraph where the highlighted text resides, or the whole document the user is reading or editing. The topic of interest is initially captured as raw data and may include terms, stems, n-grams, or phrases.

The process of searching for online data can be guided by diverse objectives. There are essential differences between searching for information to fulfill a consultation need and seeking material to support the process of topic-based search of related material. Usually, the purpose of a consultation is to find specific answers to specific questions. In this case, it is crucial to attain high precision. On the other hand, when collecting resources to support topic-based search, rather than a specific question there is typically a topic of interest that can be used to initialize and refine the relevant search space (Barathi & Shanmugam, 2014). In this case, while precision is important, other criteria should be considered, such as high recall, novelty, and diversity.

With the advent of robust search interfaces, it has become feasible to create a plethora of applications focused on topic-based exploration. In many circumstances, as it is the case of the Deep Web and other dynamically generated content, the only access point to relevant material is through the submission of queries. As a result, acquiring the ability to automatically generate efficient topical queries has emerged as a crucial research challenge. The effectiveness of a query depends on what the user is trying to achieve. If the goal is to attain good coverage then diverse queries will be more effective than focused ones. In this case, the user will try to retrieve most of the relevant material, albeit diluted with irrelevant answers. On the other hand, if the goal is to attain good precision, specific queries will be preferred over broad and diverse ones.

Since a topic representation may contain numerous terms, selecting useful terms is crucial. Several studies have shown the benefits of having tools that provide assistance for topic-based query formulation (and reformulation) (Culpepper et al., 2021). Typically, these systems provide a browsing interface where the user intervention must be explicit. However, the burden implied by the need to explicitly formulate search requests can be alleviated if queries are produced automatically based on the topic representation (Budzik, Hammond & Birnbaum, 2001; Chang, Ounis & Kim, 2006; Cecchini et al., 2008; Lorenzetti & Maguitman, 2009; Cecchini et al., 2010, 2018). While not all the information contained in a topic representation can be summarized in a query, effective mechanisms can be designed to extract small sets of representative terms to construct queries. While the automatic generation of queries offers a convenient mechanism to reflect a topic in the search process, the use of a topic representation as a source of queries only offers a partial solution to the problem of topic-based search. If queries are built as a conjunction of terms, short queries will only partially represent the topic at hand, resulting in poor precision, while long queries will become too specific, resulting in poor recall. On the other hand, disjunctive queries will typically result in poor precision for both short and long queries. Learning to formulate queries with more complex syntaxes, such as general Boolean queries, may result in better specifications of the user requirements (Cordón, Herrera-Viedma & Luque, 2002).

While Boolean operators enable the creation of highly expressive queries, many search interfaces impose limitations on query length. Consequently, formulating shorter queries offers a more widely applicable solution compared to constructing lengthy ones. Furthermore, our objective involves identifying diverse suboptimal solutions that capture different aspects of the topic being analyzed, as opposed to relying on a single, optimal query that represents the target topic or several subtopics simultaneoulsy. Moreover, the availability of diverse queries within the population, rather than just a single query, makes the application of an evolutionary approach possible.

Multi-objective evolutionary algorithms

EAs (Holland, 1975; Eiben & Smith, 2015) are robust optimization techniques based on the principle of “natural selection and survival of the fittest”, postulated by Charles Darwin, a precursor of the scientific literature and the foundation of the theory of evolutionary biology. According to this principle “in each generation, the stronger individual survives and the weaker dies.” Therefore, each new generation would contain stronger (fitter) individuals than their ancestors.

To use EAs in optimization problems we need to define candidate solutions (also called individuals) by chromosomes consisting of genes and a fitness function to be minimized (or maximized). A population of individuals is maintained and the goal is to obtain better solutions after some generations. To produce a new generation, EAs typically use selection together with the genetic operators of crossover and mutation. Parents are selected to produce offspring, favoring those parents with the highest values of the fitness function. The crossover of population members takes place by exchanging subparts of the parent chromosomes (roughly mimicking a mating process), while mutation is the result of a random perturbation of the chromosome (e.g., replacing the value of a gene with another allele).

Although crossover and mutation can be implemented in many different ways, their fundamental purpose is to explore the search space of candidate solutions (also known as decision variable space), improving the population at each generation by adding better offspring and removing inferior ones. It is important to remark that this exploration is guided by the selection operator, which uses the fitness values associated with the candidate solutions. Therefore, these fitness values constitute the space where the selection works (also known as objective function space). For this reason, the fitness function defines a mapping between the decision variable space and the objective function space.

In MOOPs the quality of a solution is defined by its performance in relation to several, possibly conflicting, objectives. Traditional methods are very limited because, in general, they become too expensive as the size of the problem grows (Lin & He, 2005). In this context, MOEAs became suitable techniques for dealing with MOOPs (Deb, 2014; Eiben & Smith, 2015; Coello Coello, Lamont & Van Veldhuizen, 2007). There are many approaches to multi-objective optimization using MOEAs and, in general, they can be classified as Pareto or non-Pareto EAs. In the first case, the evaluation is made following the Pareto dominance concept (Pareto, 1896).

Dominance is a partial order that could be established among vectors defined over an $n$-dimensional space. By means of a multi-objective fitness function, it is possible to define a relation between vectors $x_i$ in the decision variable space and vectors $u_i$ in the objective function space. Therefore, a Pareto optimal solution is an individual, $x_j$, within the decision variable space, whose corresponding vector components in the objective function space, $u_j$, cannot be all simultaneously improved for any other $u_i$, with $i\ne j$. A non-dominated set of a feasible region in the objective function space defines a Pareto Front and the set of its associated vectors in the decision variable space is called the Pareto Optimal Set. The Pareto-based algorithms use the concept of domination for the selection mechanism to move a population toward the Pareto Front. The results discussed in this work are based on the application of NSGA-II (Non-dominated Sorting Genetic Algorithm-II) proposed by Deb et al. (2002), which is one of the most studied and efficient Pareto EAs and is based on non-domination sorting.

NSGA-II is one of the most studied and efficient MOEAs, consequently it was used in this work. The algorithm begins by creating a random parent population $P_0$ of size $n$. The population is sorted based on the non-domination concept. Each solution is assigned a rank equal to its non-dominated level (1 if it belongs to the first front, 2 for the second front, and so on). In this order, minimization of rank is assumed. After ranking the solutions, a population of $n$ offsprings, $Q_0$, is created using recombination, mutation and a diversity-preserving binary tournament selection operator based on crowding distances. According to this selection operator, known as crowded tournament selection, a solution $i$ wins a tournament with another solution $j$ if he following conditions are true:

• If solution $i$ has a better (smaller) rank than solution $j$,

• If the ranks of both solutions are the same, the solution located in the less crowded region is preferred.

The NSGA-II procedure for the $i$ th generation is outlined in the following steps:

• 1.

  Let $Q_i$ be the offspring population, which is created from the parent population $P_i$ using the crowded tournament selection operator, recombination and mutation.

• 2.

  A combined population $R_i=P_i\cup Q_i$ of size $2n$ is formed.

• 3.

  $R_i$ is ordered according to non-domination (i.e., each solution is assigned a rank). Since all previous and current population members are included in $R_i$, elitism is ensured. Solutions belonging to the best front, $F_1$, are the best solution in the combined population $R_i$.

• 4.

  If the size of $F_1$ is smaller than $n$, all members of the set $F_1$ are chosen for the new population $P_{i+1}$. The remaining members of the population $P_{i+1}$ are chosen from subsequent non-dominated fronts in the order of their ranking until no more sets can be accommodated. If $F_j$ is the last front from which individuals can be accommodated in the population, but not all the members can enter in the population, then a decision needs to be made to choose a subset of individuals from $F_j$. In order to decide which members of this front will win a place in the new population, NSGA-II uses once again the crowding distance strategy to favor solutions located in less crowded regions.

MOGP is a form of MOEA where individuals are encoded by means of rich structures (typically tree structures) (Koza, 1992). For instance, an individual can encode a computer program. In this work, each individual encodes a tree-structured Boolean query. We refer the reader to Deb et al. (2002) and Coello Coello, Lamont & Van Veldhuizen (2007) for a detailed explanation of NSGA-II, MOGP, and MOEAs in general.

Evolutionary algorithms for information retrieval

Several techniques based on EAs have provided useful solutions to personalize information access (Venturini, Carbó & Molina, 2008; Lin, Yeh & Liu, 2012; Zuo et al., 2015; Sadeghi & Asghari, 2017; Ibrahim & Landa-Silva, 2018; Mhawi et al., 2022). EAs have also been applied for query refinement, which includes learning query terms, query-term weights and query operators (Singh & Sharan, 2018; Sharma, Pamula & Chauhan, 2019). A seminal approach that uses EAs to automatically evolve a population of Boolean queries is presented by Smith & Smith (1997). In this work, the authors use the Cranfield Collection to generate an initial population of queries, which is then evolved by applying GP with an aggregative fitness function based on the classic precision and recall metrics. Larsen et al. (2000) propose to select terms based on a previous classification task made by a GA whose evolution depends on the user feedback opinions. Venturini, Carbó & Molina (2008) use a GA to infer users’ preferences based on previous users’ decisions. Another approach that addresses the problem of information retrieval using GP is presented in Kulunchakov & Strijov (2017) and proposes a solution to construct ranking models for information retrieval that depend on the document description. The main contribution is the definition of a new technique to avoid stagnation and to control the structural complexity of the generated models. The work presented by Singh & Sharan (2018) uses several single GAs to select the optimal candidate terms to expand the original query, and uses the expanded query to retrieve a final set of relevant documents. Sharma, Pamula & Chauhan (2019) use a hybrid evolutionary algorithm approach to automatically extract candidate keywords to expand the original query. The new keywords are selected based on the relevance of retrieved documents, whose ranking score is given by the Okapi BM25 ranking function. The Webnaut system (Nick & Themis, 2001) creates queries by combining keywords and logic operators generated by GAs. The best sets of keywords and logical operators are then used to construct high-dimensional concepts representing the user interests. Chen & Shahabi (2002) propose to improve the current user recommendation list by using a GA that starts from historic experts’ data and incorporates the new users’ navigation behaviors through the evolution process. The work presented by Malo, Siitari & Sinha (2013) shows the effectiveness of evolving queries based on Wikipedia concepts by means of a co-evolutionary GP scheme. The strategy starts from a set of relevant and irrelevant documents provided by the user and evolves the queries evaluating them in terms of F-measure. Baeza-Yates et al. (2019) propose a GP algorithm for ranking web documents called CombGenRank. This algorithm applies elitism to supporting web document ranking. The proposed algorithm is motivated by search services based on cloud computing. Another tool for topic-based recommendation is presented in Eckroth & Schoen (2019), where the authors describe a method based on GA to select news stories about artificial intelligence for AAAI’s weekly AI-Alert. Mhawi et al. (2022) propose a document indexing method that uses an EA-based retrieval system. This EA-System learns user-relevant queries, allowing retrieving the most relevant documents to the user’s interests.

Several proposals have demonstrated that Pareto-based multi-objective strategies offer a natural way to evolve populations of effective queries. Cordon, Herrera-Viedma & Luque (2006) combine the Strength Pareto Evolutionary Algorithm (SPEA) with GP to evolve a population of Boolean queries. Each query is codified as an expression tree where the inner nodes are Boolean operators and the terminal nodes are terms. In a subsequent work, López-Herrera, Herrera-Viedma & Herrera (2009) compare the performance of four well-known MOEAs (MOGA, SPEA, SPEA2 and NSGA-II) in the problem of automatically learning queries by IQBE. A GP-based algorithm was designed for each of the evaluated MOEA schemes, which allowed the authors to conclude that the NSGA-II scheme outperforms the others.

Our proposal shares insights and motivations with the studies presented in Cecchini et al. (2008, 2010, 2018). In the first work (Cecchini et al., 2008) the authors show the effectiveness of EAs to automatically learn topical queries to retrieve material from the web guided by the user’s thematic context. In particular, the work offers a mechanism for assessing query performance in the absence of relevance judgments. It also provides an analysis of the impact of different mutation rates on performance and proposes a technique to augment queries with new vocabulary extracted from the relevant documents that are incrementally retrieved. The queries, the user context and the retrieved results are treated as vectors of terms in order to calculate similarity measures, such as cosine similarity. The evaluation is based on the quality of the best queries at each generation of the GA, and the performance improvement is measured as the increase in the quality value as the generations pass.

The work presented by Cecchini et al. (2010) is related to our proposal in addressing the problem of topic-based search using a MOEA to account for multiple objectives. It also uses a topic description as a source for query generation and a topic ontology to train and test the framework. The authors treat queries as sequences of terms following the disjunctive semantics, which means that a single matching term is sufficient to retrieve a document. The objectives to be maximized are Precision at rank 10 and Recall and three strategies are compared: single objective EAs, a Pareto-based MOEA, and an aggregative MOEA. For each strategy the authors evaluate whether the queries at the initial generations are outperformed by those in subsequent generations, and then evaluate the evolved queries on a test set. The study also reports the results obtained by two baseline methods that are based on supervised and non-supervised generalizations of the Rocchio’s method (Amati & Van Rijsbergen, 2002). The authors discuss the limitations of the single-objective EA strategy and show that the Pareto-based and aggregative MOEAs have comparable performance.

In Cecchini et al. (2018) the authors propose a family of strategies for evolving topical queries that account for the problem of achieving high recall at the population level. In order to favor diversity, the authors propose new fitness functions, referred to as retrospective fitness functions, that penalize queries that retrieve the same results as other queries in the same population. The proposed strategies assume a total order over the queries, and the retrospective fitness functions of a query are calculated taking into account the results of the queries that precede it. The authors also show that the use of multiple populations favors diversity at the genotype level.

The present study differs from the previously mentioned ones in several respects: (1) we incorporate the use of Boolean operators to build the queries, which provides a more expressive syntax; (2) we propose the use of GP as the underlying evolutionary technique, which allows to encode the queries’ syntax trees; (3) we define new fitness functions specifically designed to attain high global recall and diversity; and (4) we evaluate seven different fitness function combinations based on classical GP settings.

Diversity maximization and diversity preservation in EAs

The problem of diversity maximization is concerned with selecting from a population a subset of items that are as diverse as possible. This problem is commonly known as the Maximum Diversity Problem and it is known to be NP-hard (Kuo, Glover & Dhir, 1993). Due to its intractability a number of heuristics have been proposed to obtain approximate solutions for this problem. Besides its time complexity, another challenge associated with this problem is the definition of a diversity measure on subsets of selected elements. A widely adopted measure is the Max-Sum Diversity measure (Ghosh, 1996), which is used to maximize the sum of distances among the selected elements. Alternatively, the Max-Min Diversity measure (Resende et al., 2010) is used to maximize the minimum distance among the selected elements. A natural approach to deal with this problem is by using evolutionary algorithms (EAs) that attempt to maximize the diversity among a selected subset of items (Katayama & Narihisa, 2005). Vera et al. (2017) proposed a Multi-Objective Evolutionary Algorithm (MOEA) to simultaneously optimize several diversity measures.

Closely related to the problem of diversity maximization is the problem of diversity preservation in EAs. The latter problem is typically associated with the problem of optimizing one or many objective functions while preserving diversity among individuals. Diversity can be based on either the individuals’ genotype representation (i.e., computed in the decision variable space), phenotype representation (i.e., computed in the expression trait space), or phenotype evaluation (i.e., computed in the objective function space). Our problem domain is suitable for computing similarity among individuals by any of these approach. For example, an approach based on genotype representation similarity could favor diversity by looking at the representations of the individual queries, computing the similarity among them and penalizing those queries that are similar to many other queries. On the other hand, an approach based on phenotype representation similarity could take into consideration information associated with the retrieval results of each query. A simple definition of phenotype similarity between queries could be based on counting the number of common results returned by the queries. Finally, the phenotype evaluation similarity could be assessed by computing a spread metric on the Pareto front.

In order to address issues related to diversity preservation several Niching methods have been developed. The best-known strategies aimed at niching are crowding (Mengshoel & Goldberg, 2008) and fitness sharing (Goldberg & Richardson, 1987). In both cases, the goal is to obtain solutions uniformly distributed over the Pareto Front. Crowding methods favor individuals from less crowded regions. On the other hand, fitness sharing is a mechanism for preserving diversity that modifies the landscape by reducing the payoff in densely populated regions. The main disadvantage of the fitness sharing method is that it is necessary to provide a niche size parameter, which defines a neighborhood of solutions in the objective function space.

The computational cost of estimating diversity among individuals is typically very high. A less expensive technique that favors the exploration of diverse solutions is local selection (Menczer, Degeratu & Street, 2000), an approach that is similar to, but more efficient than fitness sharing. It has the advantage of allowing parallel implementation for distributed tasks, an important pro in the information retrieval scenario. This is achieved by evaluating the fitness function by an external environment that provides appropriate data structures for maintaining shared resources. This technique is amenable to the information retrieval task, where each retrieved document can be marked so that the same document does not yield payoff multiple times.

Population fitness measures have been proposed to evaluate the diversity of a population as a whole. The concept of population fitness is present in biology (Markert et al., 2010) and is supported by the principle of community heritability (Shuster et al., 2006). In EAs, an example of population fitness known as global fitness is incorporated in the Parisian approach (Collet et al., 2000). In this algorithm, a global solution is built by the combination of the information provided by several members of the population, whose individual performances are evaluated by a local fitness function.

The algorithm proposed in Zhou et al. (2022), uses a diversity maintenance mechanism based on the global and local diversities by considering them collaboratively. The similarity between two individuals is defined as the cosine similarity between their objective vectors. Then, a balance strategy is used to adjust weights of convergence for global diversity and local diversity according to the population. Each solution reaches a score that represents its performance in diversity and convergence simultaneously.

Another strategy that favors diversity preservation is based on the concept of multiple populations (Gupta & Ghafir, 2012). The idea behind multi-population EAs consists in defining subpopulations which evolve independently; thus the unique features of each subpopulation can be effectively preserved, and the diversity of the entire population is benefited. In some algorithms, these subpopulations can exchange information, as it is the case in Multi-population Coevolutionary Algorithms (Shi et al., 2014), but in other cases a complete isolated evolution is executed over each subpopulation (Cochran, Horng & Fowler, 2003).

MOEAs deteriorate their search ability when solving many-objective optimization problems (MaOPs), i.e., problems with more than three objectives. Li et al. (2019) propose a dynamic convergence-diversity guided evolutionary algorithm for MaOPs by employing the decomposition technique where a set of uniformly distributed reference vectors is used to divide the objective space into K subspaces. Each subspace has its own subpopulation and evolves simultaneously with the others.

Diversity in information retrieval

The problem of diversification of search results has been extensively studied by the information retrieval community. Clustering offers an approach to maximizing diversity in information retrieval since a form of diversity can be attained by selecting items from different clusters (Yu et al., 2018). Another diversity maximization approach is based on nonnegative matrix factorization, which applies network dimensionality reduction and estimates relevance in a low dimension space (Meng et al., 2018). Other approaches attempt to identify diverse results for a query by accounting for different “aspects” associated with a query (Clarke et al., 2008; Agrawal et al., 2009; Bouchoucha, He & Nie, 2013). The approaches mentioned primarily concentrate on identifying various results linked to a particular query. In contrast, the methods presented in this article focus on creating diverse queries associated with a subject of interest. Carpineto, D’Amico & Romano (2012) take the notion of subtopic as a basis to optimize the relevance of results at the document set level by considering clustering and diversification of search results. In Ghosal et al. (2021), the authors explore the use of textual entailment models to identify document novelty at a semantic level. The approach is intended to automatically detect novel information in a given text by combining machine learning and natural language processing. Karakaya & Aytekin (2018) present a method to diversify the top-N items on a recommendation list. Their proposal tries to give the user a global view of possible user’s interests, rather than only suggesting similar items. The method reorganizes the results by looking for sub-topics. The performance is evaluated based on precision, recall, and user behavior, and showed that the user’s overall likeness goes beyond classical factors.

While the above-mentioned methods attempt to diversify search results, other methods aim at diversifying queries. This can be done by automatically generating diverse but semantically related queries. Some of these strategies rely on the use of query-URL clickthrough data (Ma, Lyu & King, 2010) while others are based on external semantic sources, such as Wikipedia (Hu et al., 2013; Baggio et al., 2019) or Wordnet (Zheng et al., 2014). In Baggio et al. (2019) we analyzed whether the use of Wikipedia concepts instead of terms positively affects the diversity of the search results. However, improvements were only obtained in terms of interpretability and readability of the queries for the user, not in terms of the results’ diversity. Patro, Niaz & Prasath (2023) propose an information retrieval framework that identifies different aspects of information and organizes clusters based on similarity to each aspect. In this way, the final retrieved results cover diverse information needs. Before the final response, the framework applies query expansion to refine the final set of clusters. Kim & Croft (2014) attain query diversification by taking a document from which multiple aspects (subtopics) can be identified. Query diversification methods have also been applied to develop query suggestion or auto-completion mechanisms (Kharitonov et al., 2013). Li, Schijvenaars & de Rijke (2017) investigate empty-result queries and discuss how to avoid them. A review of a large number of methods proposed in the literature for search result diversification is presented in Santos, Macdonald & Ounis (2015).

Data

To run our evaluations, the data were collected as previously described in Cecchini et al. (2010) using the evaluation framework introduced in Cecchini et al. (2011). To ensure the quality of our corpus, certain restrictions were placed on the selection process (for the dataset description and download (https://data.mendeley.com/datasets/9mpgz8z257/1), please refer to Lorenzetti, Maguitman & Baggio (2019). The corpus comprises over 350,000 webpages classified into 448 topics. To index these webpages and run our experiments we used the Python extension for the Lucene framework (http://lucene.apache.org/pylucene/). All terms underwent Porter stemming and stopwords were eliminated. The topics were divided into two parts, with two-thirds of the webpages used to create the training index, while the remaining one-third was allocated for building the testing index. The evaluations reported in this article were carried out using the topic descriptions extracted from 25 randomly chosen topics. Each topic has its own ODP description that typically consists of a few sentences briefly describing the content of the webpages indexed under that category.

Performance measures

The performance of a strategy can be measured by the collective effectiveness of the population of queries evolved by the strategy. For a given topic $t$, the following metrics provide a means to evaluate the effectiveness of a query population P:

• $\overline{Precision@10}$. The mean precision at rank 10 is calculated as the average of $Precision@10$ scores obtained for all queries within population P. For a given population P and a topic $t$, this metric is computed using the following formula:

  $$\overline{Precision@10}(P,t)=\frac{{\sum }_{q\in P}Precision@10(q,t)}{|P|}.$$

• $Global-Recall$. The global recall metric evaluates the performance of the entire query population in achieving high recall scores. The union of results returned by a query population P, denoted by $A(P)$, is defined as the collective set of results returned by all queries in P: $A(P)=\bigcup _{q_i\in P}Re{t}_{q_i},$ where $Re{t}_{q_i}$ is defined in the usual way as the set of results returned by $q_i$. To calculate $Global-Recall$ for a given population P and topic $t$, we use the following formula:

  $$Global-Recall(P,t)=\frac{|A(P)\cap Re{l}_t|}{|Re{l}_t|}.$$

• $\overline{Jaccard-Similarity}$. The mean Jaccard similarity metric results from averaging the values of $J-Sim-Index$ across all pairs of answer sets for queries in P restricted to relevant documents. It allows to assess the diversity of results returned by a population of queries. As mentioned before, let $Re{t}_{q_i}^{\star }=Re{t}_{q_i}\cap Re{l}_t$. Then, we define $\overline{Jaccard-Similarity}$ as follows:

  $$\overline{Jaccard-Similarity}(P,t)=\frac{{\sum }_{q_i,q_j\in P,i\ne j}J-Sim-Index(Re{t}_{q_i}^{\star },Re{t}_{q_j}^{\star })}{|P|\cdot (|P|-1)},$$where $J-Sim-Index(Re{t}_{q_i}^{\star },Re{t}_{q_j}^{\star })=|Re{t}_{q_i}^{\star }\cap Re{t}_{q_j}^{\star }|/|Re{t}_{q_i}^{\star }\cup Re{t}_{q_j}^{\star }|.$

MOGP strategies

To evaluate the proposed fitness functions, we experimented with seven MOGP strategies based on various combinations:

• Co1: $Precision@10$ and $Recall$. This is a baseline strategy that attempts to maximize the relevance of the top 10 retrieved results ( $Precision@10$) and the proportion of relevant documents that are successfully retrieved from the entire collection ( $Recall$) Balancing $Precision@10$ and $Recall$ is a challenge, as there is often a trade-off between the two.

• Co2: $Precision@10$ and $Entropic-Recall$. This strategy combines the classical $Precision@10$ metric with the novel $Entropic-Recall$. As in the case of Co1, the goal is to maximize the relevance of the 10 retrieved results and the proportion of relevant documents that are successfully retrieved from the entire collection. Different from $Recall$, $Entropic-Recall$ favors queries that retrieve results that other queries are unable to recover.

• Co3: $Entropic-Precision@10$ and $Entropic-Recall$. This strategy is similar to Co1 in that it attempts to simultaneously maximize the relevance of the top 10 retrieved results and the ratio of relevant documents effectively retrieved from the entire collection. However, it integrates both objectives with the goal of retrieving results that other queries fail to capture.

• Co4: $Precision@10$, $Recall$ and $Jaccard-Similarity$. This strategy complements Co1 by incorporating the objective of minimizing the similarity of the set of results retrieved by the query and the set of results retrieved by each of the other queries. This is carried out by minimizing the $Jaccard-Similarity$ fitness function.

• Co5: $Precision@10$, $Entropic-Recall$ and $Jaccard-Similarity$. This strategy is similar to Co4, except that it replaces the objective $Recall$ by $Entropic-Recall$, giving preference to queries that retrieve results not recovered by the other queries.

• Co6: $Precision@10$ and $Jaccard-Similarity$. This strategy attempts to maximize the relevance of the top 10 retrieved results, while also minimizing the similarity of the result set retrieved by the query and those obtained from each of the other queries. It does not incorporate the $Recall$ metric or any of its derivatives.

• Co7: $Precision@10$, $Jaccard-Similarity$ and $|Re{t}^{\star }|$. This strategy augments Co6 by also attempting to maximize the size of the retrieved set of relevant documents $Re{t}^{\star }$. Although it does not incorporate the $Recall$ metric or any of its variations, we expect to attain high coverage through the objective of maximizing $|Re{t}^{\star }|$.

In all the above combinations the objectives are maximized, except for the case of $Jaccard-Similarity$, which is minimized. For comparison purposes, combination Co1 was taken as a baseline, which is based on classical measures of $Precision@10$ and $Recall$.

The strategies were first assessed through a case study. Then a comparison in terms of $\overline{Precision@10}$, $Global-Recall$, and $\overline{Jaccard-Similarity}$ was completed on the training set by running each strategy 5 times on each of the 25 selected topics. Finally, the queries evolved by the strategies were compared on the testing set by means of the proposed performance metrics.

Table 1 summarizes the parameter configuration used during the training stage. These parameters were selected based on previous results and guidelines found in the GP literature (Cordon, Herrera-Viedma & Luque, 2006; Malo, Siitari & Sinha, 2013; Eiben & Smith, 2015) and our own previous work (Cecchini et al., 2008, 2010, 2011, 2018). Instead of focusing on the analysis of various parameter configurations, which can become complex due to their combinatorial nature, we prioritized the exploration of the proposed MOGP strategies. With this objective in mind, we evolved queries employing the MOGP strategies, resulting in the generation of more than 22.300.000 queries.

A common problem in GP is that queries experience a significant increase in size throughout the generations, therefore we selected the potential parents to perform the crossover and mutation using the selDoubleTournament operator, an alternative selection operator also used as a bloat control technique (Luke & Panait, 2002). This operator uses a special sample function to select candidates, which is another tournament based on the size of the individuals. The actual selection process is then carried out based on fitness. Query depth is limited to 17, which is the standard maximal depth limit for tree-based GP to prevent excessive growth.

For each topic description, the training index was used to evolve queries using five independent runs. For each run we considered 150 generations with a population size of 100 queries. The test set was used to assess the effectiveness of the evolved queries on a new corpus for a specific topic. It’s worth noting that while the train and test sets comprise the same topics (and hence the same topic descriptions), they contain different documents (webpages).

As a first step in the performance analysis, we looked at the evolution of the queries associated with the individual topics. For illustrative purposes, we report the analysis on the topic Solar System. The description available from the ODP distribution for this topic is “The Solar System currently consists of the Sun, nine planets, and the many asteroids and comets that also orbit the Sun.” The initial queries were generated by randomly selecting terms from this description. New vocabulary was incrementally learned based on the retrieved material that was relevant to the topic. Some of the learned terms, such as “mars”, “saturn” and “astrobiology”, were used to automatically build subsequent queries helping to retrieve novel but topic-related content.

Figure 4 shows the charts corresponding to the Precision@10 and Recall values of the queries evaluated on the topic Solar System based on a single run. For illustrative purposes, we only show the charts for the initial generation and the last generations of Co1 (taken as a baseline for comparison) and Co3 (taken as an example of strategy that incorporates a diversity preservation mechanism). The initial population is randomly generated and, as a consequence, it is highly diverse. However, as expected, the performance of these random queries is low in general, both in terms of Precision@10 and Recall. On the other extreme, the queries evolved by strategy Co1 have high Precision@10 and Recall, but there is an evident loss in phenotypic representation diversity. Note that the strategy that incorporates a mechanism for diversity preservation (Co3) preserves phenotypic representation diversity. Also, individuals in the Pareto front achieve Precision@10 and Recall comparable to Co1.

Evaluation of the MOGP strategies on the training set

A more rigorous analysis can be carried out by averaging the values of each performance metric over the five completed runs and the 25 analyzed topics. $\overline{\overline{Precision@10}}$, $\overline{Global-Recall}$ and $\overline{\overline{Jaccard-Similarity}}$ stand for the arithmetic mean of $\overline{Precision@10}$, $Global-Recall$ and $\overline{Jaccard-Similarity}$, respectively, computed across all the runs and topics. The evolution of these metrics during training for the analyzed strategies is illustrated in Fig. 5. It can be observed that Co3 achieves the highest $\overline{Global-Recall}$, followed by Co2, both of which are strategies with only two objectives, one of which is $Entropic-Recall$. Furthermore, it is interesting to note that when Co1 is extended to formulate Co4 by introducing $Jaccard-Similarity$ as a third objective, there is a substantial improvement in $\overline{Global-Recall}$, and it becomes comparable to Co2 and Co3.

Note that Co3 is the only strategy that does not attempt to maximize $Precision@10$ directly, but rather focuses on a reformulation of it, $Entropic-Precision@10$. However, it is noteworthy that it reaches an $\overline{\overline{Precision@10}}$ value close to 0.9, which represents an excellent performance. In addition, Co3 is highly effective in terms of $\overline{\overline{Jaccard-Similarity}}$ and has the highest $\overline{Global-Recall}$ among all the strategies. In general, we can observe that the strategies that incorporate $Entropic-Recall$ are more effective than those that use $Jaccard-Similarity$.

Strategy Co6 has only two objectives and it has been incorporated for completeness. Although it achieves the best $\overline{\overline{Jaccard-Similarity}}$, it is the worst in terms of $\overline{Global-Recall}$. This can be attributed to the fact that Co6 has $Jaccard-Similarity$ as an objective function, thus the material recovered might be diverse but consists of a reduced number of documents. Note that Co7 is a reformulation of Co6 that incorporates the number of relevant retrieved results ( $|Re{t}^{\star }|$) as a third objective to be maximized. This improvement notably enhances $\overline{Global-Recall}$ in comparison to the Co6 strategy. However, combination Co7 performs worse than Co2, Co3, and Co4 in terms of $\overline{\overline{Jaccard-Similarity}}$. Consequently, Co3 is the preferred choice among them.

It is worth mentioning that some strategies attain similar performance, as is the case of Co1 and Co5. For those cases, the computational time can be used as a criterion to prefer a strategy over another. Hence, Co1 would be preferred over Co5 because it involves only two objectives (instead of three) and it does not require the evaluation of global fitness functions ( $Entropic-Recall$ and $Jaccard-Similarity$), which are more expensive to compute than the local fitness functions.

Finally, for a more comprehensive analysis and to illustrate the variability of one of the most effective MOGP strategies examined across different topics, we present in Fig. 6 the means and confidence intervals for Co3 on each topic over the five runs. The small confidence intervals observed, especially for $\overline{Precision@10}$ and $Global-Recall$, serve as indicators of the method’s stability and robustness.

Evaluation of the MOGP strategies on the testing set

In this section, we evaluate the $\overline{\overline{Precision@10}}$, $\overline{Global-Recall}$ and $\overline{\overline{Jaccard-Similarity}}$ metrics achieved by the proposed MOGP strategies on the testing set. The questions addressed in the first place are: How effective are the queries evolved by each of the proposed strategies in comparison to the initial queries that are formulated directly from the description of the topic? (RQ1), Are the queries evolved by the proposed strategies effective when tested for the same topic but on a new corpus? In other words, are the proposed strategies able to avoid overfitting? (RQ2), and What are the trade-offs among different MOGP strategies for topical search when attempting to address conflicting objectives such as high precision and recall? (RQ3).

The comparison presented in Fig. 7 offers an answer to research question RQ1 by contrasting the results obtained by the first generation of queries and by the queries evolved by each of the evaluated strategies. Note that the first generation of queries is the same for all the assessed strategies, resulting in the same performance.

Firstly, the results suggest that, with the exception of Co6, all the other analyzed strategies demonstrate statistically significant improvements in $\overline{\overline{Precision@10}}$ when evaluating the queries from the final generation on the test set, in comparison to those queries generated directly from the topic description. A statistically significant improvement in $\overline{Global-Recall}$ is evident in Co3 when compared to Co1, Co5, Co6 and Co7. Additionally, Co3 achieves higher $\overline{Global-Recall}$ when compared to Co2 and Co4, although the difference is not statistically significant. A statistically significant loss in diversity (i.e., an increase in $\overline{\overline{Jaccard-Similarity}}$) is observed in the evolved queries for all the strategies when compared to the queries directly generated from the topic description. Co3 achieves statistically significant improvements in diversity when compared to Co1, Co5, Co6 and Co7. Improvements in diversity (although not statistically significant) can also be observed when comparing Co3 with Co2 and Co4.

Regarding research question RQ2, the results show evidence that the answer is positive, since for the majority of the cases the comparison among the strategies when evaluated on the test set provides results consistent with those observed on the training set. All the strategies, with the exception of Co6 (which appears to experience overfitting), exhibit similar performance in terms of $\overline{\overline{Precision@10}}$ on the test set. However, as anticipated, the performance is slightly lower than that achieved during the training stage. All the combinations, reach values above 0.8 for $\overline{\overline{Precision@10}}$ during training, while most of the combinations attained values over 0.5 for the same metric on the testing set.

Similar to what we observe on the training set, the best $\overline{Global-Recall}$ on the test set is obtained by Co3 followed by Co2 and Co4. Also, in the same way as we observe on the training set, Co2, Co3 and Co4 achieve very good performance in terms of $\overline{\overline{Jaccard-Similarity}}$. An interesting result is that Co1, Co5, Co6 and Co7 result in the worst performance in terms of $\overline{Global-Recall}$ and the same behavior is observed for $\overline{\overline{Jaccard-Similarity}}$. Our analysis reveals that the primary reason for Co6’s underperformance on the test set is its susceptibility to overfitting. An examination of the results produced by this strategy indicates that it tends to generate queries that are excessively specific, often returning only a single relevant document. Co7 significantly mitigates this issue by enhancing Co6 with the objective of maximizing $|Re{t}^{\star }|$.

Table 2 summarizes these comparisons, indicating the strategies with best and worst performance when evaluated on the training and testing sets. We include more than one strategy when these strategies result in a very similar performance. This trade-off analysis provides an answer to research question RQ3.

Comparison of the MOGP strategies against a MOEA strategy

To investigate research question RQ4 (How effective are the queries evolved by the proposed strategies in comparison to queries that are built using MOEA strategies?), we compare the performance of MOGP-Co3 (which showed the best balance of the performance metrics when compared to other MOGP strategies) against the most effective MOEA strategy reported in Cecchini et al. (2018), to which we refer to as MOEIR.

MOEIR is a MOEA for Information Retrieval designed for queries with a straightforward structure consisting of a disjunctive list of terms. The MOEIR strategy attempts to maximize two fitness functions aimed at penalizing queries that do not contribute to diversity. The inspiration for these fitness functions was drawn from biological principles, where a population of queries can be thought of as individuals competing for a finite resource, which, in this context, represents relevant documents. Consequently, the outcomes of these fitness functions for a specific query, denoted as $q_i$, depend not only on the query itself but also on other queries within the same population. To formally define these fitness functions it is necessary to assume a total order on the queries that constitute P. The first of these fitness functions, retrospective precision at rank 10, assesses the precision of a query’s top ten results while considering the resources (documents) consumed (retrieved) by preceding queries in a population. The second fitness functions, retrospective recall, quantifies how comprehensively a query retrieves relevant documents, considering the resources consumed by earlier queries in the population. Figure 8 compares the performance in terms of $\overline{\overline{Precision@10}}$, $\overline{Global-Recall}$ and $\overline{\overline{Jaccard-Similarity}}$ achieved by MOGP-Co3 and MOEIR on the test set during the first and last generations. We observe that MOGP-Co3 outperforms MOEIR in terms of $\overline{\overline{Precision@10}}$ and $\overline{Global-Recall}$, although it’s performance is slightly inferior for $\overline{\overline{Jaccard-Similarity}}$. Based on these results, we could conclude that, overall, MOGP-Co3 demonstrates better performance than MOEIR.

Comparison of the MOGP strategies against non-evolutionary term-selection strategies

To investigate research question RQ5 (How effective are the queries evolved by the proposed strategies in comparison to queries that are built using state-of-the-art term-weighting schemes?), we compare the MOGP-Co3 strategies against eleven traditional and state-of-the-art term-weighting schemes. The definitions of the term-weighting schemes used in our analysis are presented in Table 3 and are based on the following notation, adapted from Lan et al. (2005) and Domeniconi et al. (2015):

• A denotes the number of documents that belong to class $c_k$ and contain term $t_i$.

• B denotes the number of documents that belong to class $c_k$ but do not contain the term $t_i$.

• C denotes the number of documents that do not belong to class $c_k$ but contain the term $t_i$.

• D denotes the number of documents that do not belong to class $c_k$ and do not contain the term $t_i$.

• N denotes the total number of documents in the collection (i.e., N = A + B + C + D).

The eleven term-weighting schemes used for comparison include some traditional information retrieval models that apply unsupervised approaches to determine term importance (no topic labels are used to compute these values), as well as some term-weighting methods that take a supervised approach to assess the importance of a term in a topic or class. The code used for these experiments was adapted from Maisonnave et al. (2021) and is available to the research community (https://github.com/ceciliabaggio/term_weighting_methods_full_testing_dataset).

A simple unsupervised term-weighting scheme that we took as a baseline for comparison is based on counting how many documents in the corpus contain the given term. This scheme, which is known as term global frequency (TGF), aims to improve recall but typically results in poor precision. Alternatively, other unsupervised term-weighting schemes are based on the premise that common terms are poor discriminators. This is the case of the widely used global factor inverse document frequency (IDF) (Salton & Buckley, 1988), which penalizes terms based on the number of documents containing the term. To compare our approach with a basic global weighting scheme we also considered IDF as a baseline.

Other term-weighting methods that we used for comparison take advantage of class-label information, which gives rise to supervised term-weighting schemes. One of such methods is TGF*, which is a supervised variation of TGF that instead of counting the number of documents with the term in the corpus, counts the number of documents in a class that contain the term. A limitation of TFG* is that it favors those terms that occur frequently in a class but does not penalize terms that occur in many class. A supervised variant of IDF, known as inverse document frequency excluding category (IDFEC) (Domeniconi et al., 2015) penalizes frequent terms but avoids penalizing those terms occurring in several documents belonging to the relevant class. The combination of TGF* with IDFEC gives rise to a supervised term-weighing scheme known as TGF*-IDFEC, which simultaneously accounts for the descriptive and discriminating power of a term with respect to a relevant class. A traditional supervised term-weighting score used for comparison is given by the conditional probability of a term occurring given a class, which gives rise to the odds ratio (OR) (van Rijsbergen, Harper & Porter, 1981). We also took as baseline methods some supervised term-weighting scores derived from information theory (Shannon, 1948), including chi-squared ( $\chi {}^2$), information gain (IG) and gain ratio (GR) (Quinlan, 1986). Finally, we considered the FDD ${}_{\beta }$ score (Maisonnave et al., 2021), which is defined as follow:

$$FD{D}_{\beta }=(1+{\beta }^2)\frac{A/(A+C)\times A/(A+B)}{({\beta }^2\times A/(A+C))+A/(A+B)}.$$

The FDD ${}_{\beta }$ score is a supervised term-weighting scheme that uses a tunable parameter $\beta$ to favor different objectives in the information retrieval task. By using a $\beta$ value higher than 1 we can weight descriptive relevance higher than discriminative relevance while a $\beta$ smaller than 1 weights discriminative relevance higher than descriptive relevance. To carry out our comparative analysis we used FDD ${}_{0.5}$, FDD ${}_{1.0}$ and FDD ${}_{10}$.

To be able to compare the MOGP-Co3 strategy (which showed the best balance of the performance metrics when compared to other MOGP and MOEA strategies) with the schemes mentioned above, we run each method with each of the 25 topics from the dataset. Unlike the MOGP strategies, the query-building process (in the supervised approaches) does not require extra information besides the class-label or topic. This is in contrast with the approach adopted by MOGP, which requires a text snippet (topic description provided by the ODP dataset) to start formulating queries.

Each query built using a term-weighting method consists of the best term (unigram, bigram or trigram) weighted by the scheme at hand. Thus we obtained the first 100 terms ordered by a score, from greatest to smallest. The score represents the weight of the term according to the strategy. Here, the 100 queries are thought to constitute a “population”, similar to the ones in the evolutionary approaches. Queries were built based on the vocabulary extracted from the same set of documents used to train the MOGP methods (“training set”), which comprises 2/3 of the dataset. The resulting queries were used to retrieve results from the same dataset that was used to evaluate the MOGP methods (“testing set”), which contains 1/3 of the dataset. The set of matching documents was sorted using query-document TF-IDF similarity and used to compute the performance metrics following the same approach taken by the MOGP methods. For instance, the performance metric $\overline{\overline{Precision@10}}$ looks into the first 10 documents retrieved by each query.

The performance metrics measured on the training and testing sets comparing the term-weighting methods presented in Table 3 against MOGP-Co3 are reported in Figs. 9 and 10, respectively. Note that in these figures we refer to the average (avg) of the metrics. This means that the average in the case of MOGP-Co3 corresponds to the double over-lined averages mentioned before, while in the rest of the methods it represents the single average (single over-lined). The explanation behind this is that in the term-weighting schemes, there is no random seed to start the query generation process, thus we run each combination of method and topic only once, whereas, in the evolutionary schemes, we completed five runs for each combination.

In general, we observe that MOGP-Co3 has a better performance than the other methods in terms of $\overline{\overline{Precision@10}}$ and $\overline{\overline{Recall}}$ (thus in $\overline{\overline{F1-score}}$) both in the training and testing sets. Also, we observe that the $\overline{Global-Recall}$ score achieved by MOGP-Co3 is competitive, and the result achieved was only slightly worse when compared to the methods with the highest $\overline{Global-Recall}$. Hence, due to the superiority of MOGP-Co3 in terms of $\overline{\overline{Precision@10}}$ and $\overline{\overline{Recall}}$, we can claim that MOGP-Co3 has an overall advantage when simultaneously accounting for $\overline{\overline{Precision@10}}$, $\overline{\overline{Recall}}$ and $\overline{Global-Recall}$.

On the other hand, we observe that the MOGP-Co3 strategy achieves a higher $\overline{\overline{Jaccard-Similarity}}$ value than that attained by the term-weighting methods. However, the low $\overline{\overline{Jaccard-Similarity}}$ obtained by these methods is due to the fact that each of the built queries is necessarily different from the other queries in the same population, as they are constructed using different terms. In contrast to this, queries evolved by the MOGP-Co3 strategy will tend to share common terms, hence resulting in similar result sets.

Computational time of the MOGP strategies

The proposed MOGP strategies involve several processes that demand significant computational time. The initialization process involves constructing a population P of Boolean queries, with each query represented as a tree-like individual. The computational time required for building such initial population of queries is $O(|P|\times 2^d)$, with $d$ as the maximum tree depth (in our experiments, we set $d=17$). Throughout each generation, the computational time is mainly affected by the cost of submitting |P| queries to a search engine, retrieving a document collection for each query, computing the fitness functions associated with the respective MOGP strategy, executing the processes inherent in the NSGA-II algorithm, and managing an external archive of solutions discovered during the evolutionary process. The time required for submitting each query depends on the complexity of the query itself, the search algorithm used by the search interface, and the hardware and infrastructure, among other factors.

The computational time required to compute the fitness functions associated with each objective is highly dependent on each specific function. Global fitness functions are more expensive than local ones. In particular, for the Co3 strategy, the computation of $Entropic-Precision@10$ and $Entropic-Recall$ have a computational time of $O(|P|)$ and $O(|P|\times |Re{t}_q|)$, respectively, assuming constant-time relevant membership testing in the relevant set. On the other hand, the Co1 strategy requires computing $Precision@10$ and $Recall$, with a significantly lower computational time of $O(1)$ and $O(|Re{t}_q|)$, respectively. Finally, the time complexity of the classical NSGA-II algorithm at each generation is $O(m\times |P{|}^2)$, where $m$ is number of objectives (Deb et al., 2002). A more detailed and recent mathematical analysis of the time complexity of NSGA-II, taking into account enhancements obtained through crossover, is presented in Doerr & Qu (2023).

In our analysis, a typical run of 150 generations with a population of size $P=100$ requires building and evaluating 15,000 queries, with an average computational time of 2.5 s per query, including the overhead resulting from the various processes inherent to the experiments. For certain MOGP strategies and topics, the cumulative time required for evolving a query population exceeded 10 h based on a standard desktop PC with 16 GB of RAM, SSD, and Intel i7 processor. We anticipate that introducing various optimizations, such as effective caching strategies, can significantly reduce this computational time. However, this is outside the scope of the present work.

It is important to highlight that the process of evolving a population of queries through a MOGP strategy is typically a one-time task performed offline. The optimized queries generated through this process can subsequently be employed for the efficient real-time gathering of information for various purposes, including news and social media monitoring, without the necessity for retraining unless explicitly needed. Consequently, considering the superior performance of certain MOGP strategies in comparison to other approaches and their relevance in applications requiring offline training, we believe that the answer to research question RQ6 (Is the computational time required justifiable based on the effectiveness achieved?) is affirmative, even considering the time needed to evolve a population of topical queries.