Boolean logic algebra driven similarity measure for text based applications

Paper organization

The related works, including the methods compared, are covered in “Related WorK”. “The Proposed Method” introduces the proposed similarity measure. Experimental settings are concisely presented in “Experimental Setup”. The outcomes of experimental study are drawn in “Results”. The most important points out of this study are articulated in “Discussion”. Finally, “Conclusions and Future Work” concludes the paper and presents the avenues of future work.

The compared methods

In the following, all compared similarity measures and distance metrics are described. Having two documents document1 and document2 whose TF-IDF of their “n” terms (w.t) have been saved in both vectors: doc₁(w.t₁₁, w.t_12,…, w.t_1n) and doc₂(w.t₂₁, w.t_{22, …,} w.t_2n) respectively, the considered seven similarity measures or distance metrics [29–32] are listed as follows:

Euclidean distance (ED)

ED computes the distance between each point pair in N-dimensional space. It is define by the following equation:

(1) $$D_{Euc}(doc1,doc2)=\sum {sqrt(do{c}_{11-}do{c}_{12})}^2+{(do{c}_{21-}do{c}_{22})}^2+\dots (do{c}_{n1-}do{c}_{n2}{)}^2$$

Cosine measure

The pairwise similarity is found between each document pair using both the dot product as well as the magnitude of both vectors doc₁ and doc₂.

(2) $$Si{m}_{Cos}\left(doc1,doc2\right)=\frac{\sum _{i=1}^{n}do{c}_{i1}\ast do{c}_{i2}}{\sqrt{\sum _{i=1}^{n}do{c}_{i1}^2}\ast \sqrt{\sum _{i=1}^{n}do{c}_{i2}^2}}$$

Jaccard similarity measure

It is one of the most widely used similarity measures. However, the problem with Jaccard as we found experimentally is that Jaccard is highly reliant on the common values which are, unlike BoW representation in which Jaccard is working well, barely exist in TF-IDF representation. This reliance factor makes Jaccard a very poor option when used for TF-IDF-based document matching. This measure is defined by next equation.

(3) $$Si{m}_{jaccard}\left(doc1,doc2\right)=\frac{doc1\cap doc2}{doc1\cup doc2}$$

Extended Jaccard

The extended Jaccard (ex-jaccard, in short) can be considered as the inverse of Jaccard coefficient, and its equation is drawn as follows:

(4) $$Si{m}_{Ex-jaccard}\left(doc1,doc2\right)=\frac{doc1.doc2}{{\left|doc1\right|}^2+{\left|doc2\right|}^2-\left(doc1.doc2\right)}$$

Manhattan

This distance metric computes the sum of absolute differences between the coordinates of vectors of document pair. It is defined as follows:

(5) $$Manhattan-distance\left(doc1,doc2\right)=\sum _{i=1}^n\left|doc{1}_{i1}-doc{2}_{i2}\right|$$

Kullback leibler divergence (KLD)

KLD finds the difference between the probability distributions, and is defined as follow:

(6) $$Si{m}_{KL}\left(doc1,doc2\right)=\sum _{i=1}^{n}Pdoc1\left(do{c}_{i1}\right)\ast \log \left(\frac{Pdoc1\left(do{c}_{i1}\right)}{Pdoc2\left(do{c}_{i2}\right)}\right)$$

where Pdoc₁(doc_i1) is the value of i^th feature of doc₁ and Pdoc₂(doc_i2) is the value of i^th feature of doc₂.

SMTP

This similarity measure was presented in Lin, Jiang & Lee (2014) for text clustering and classification. It considers two key scenarios when the feature exists in both documents doc₁ and doc₂, and when the feature is absent in the pair. It was defined as follows:

(7) $$Sim\left(doc1,doc2\right)=\frac{F\left(doc1,doc2\right)+\lambda }{1+\lambda }$$

where

$$F\left(doc1,doc2\right)=\frac{N_{star}\left(doc1,doc2\right)}{N_{union}\left(doc1,doc2\right)}$$

$$N_{star}\left(doc1,doc2\right)=\{\begin{array}{cc}0.5(1+ex{p}^{-{\left(\frac{do{c}_{i1-}do{c}_{i2}}{var}\right)}^2})\hfill & ifdoci1.doci2>0\hfill \\ 0\hfill & ifdo{c}_{i1}=0,do{c}_{i2}=0\hfill \\ \lambda \hfill & \mathrm{O}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}$$

And

$$N_{union}\left(doc1,doc2\right)=\{\begin{array}{cc}0,\hfill & ifdoci1=0,doci2=0\hfill \\ 1,\hfill & Otherwise\hfill \end{array}$$

where λ is a constant that was fixed at values of (1) and (0.01), while conducting our experiments, for classification and clustering respectively. The variable var indicates the standard distribution of all non-zero features. The values of (1) and (0.01) were not randomly chosen for λ. They were selected based on the setting asserted and used in Lin, Jiang & Lee (2014) as the optimal option to compare SMTP with other measures for both classification and clustering.

Motivation

As mentioned earlier in the drawn-above sections, the text classification and clustering is tricky task chiefly when big data is set to be handled. Furthermore, even though the literature is full of effective similarity measures that have a good performance, the necessity for efficient and effective similarity measures is still incomplete. While there have been some efficient measures like Euclidean and Manhattan, these measures have been shown of poor performance when applied on texts of middle, big, or voluminous-sized datasets. On the other hand, there have been efficient and effective measures like cosine; however, cosine measure does not reach the desired performance. Therefore, several works in literature have been presenting new effective measures like PDSM, ISC, SMTP, and STB-SM. While PDSM was seen as a rather time-inefficient measure (Amer & Abdalla, 2020), SMTP has been seen time-inefficient as well in our work. On the other hand, these measures were seen much more effective than cosine; Yet not as efficient as cosine. The need to address the trade-off between efficiency and effectiveness motivates us mainly to enrich IR and ML with a new efficient and effective similarity measure (BLAB-SM). The newly-proposed measure has been proposed to shrink the already-mentioned trade-off. Experimentally, BLAB-SM is seen as effective as SMTP, and more efficient than cosine. It is also seen as efficient as Euclidean and Manhattan in most cases which is the ultimate objective of this work. Concisely, BLAB-SM comes to effectively fill the gap of efficiency problem from which some effective measures like SMTP suffers, while maintaining, if not outperforming in some cases, the effectiveness reached by SMTP.

BLAB-SM similarity measure

Three basic definitions for the Boolean logic algebra, which inspired us to propose the measure, are briefly highlighted before presenting our proposed measure, BLAB-SM. In doing so, BLAB-SM's explanation and analysis would be completely understood.

The logical gates

Strictly speaking, the digital systems are constructively defined by using the logic gates. In general, the basic gates are the AND, OR, NOT gates. The basic operations are described below along with their truth tables as follows:

The AND gate is an electronic circuit that yields an output (1) only and only if all its inputs (A and B, in Fig. 1, Table 1) are (1). This symbol (∧) is used to show the AND operation, i.e. A ∧ B. AND gate in our measure comprises only and only the shared features.

Table 1:

A AND B table truth.

A	B	A ∧ B
0	0	0
0	1	0
1	0	0
1	1	1

DOI: 10.7717/peerj-cs.641/table-1

Like AND gate, the OR gate is also an electronic circuit. However, the OR gate yields an output (1) if and only if one or more of its inputs (A and B, in Fig. 2, Table 2) are (1). This symbol (∨) is used to show the OR operation, i.e., A ∨ B. OR gate in our measure concerns both the shared and non-shared features at the same time.

Table 2:

A OR B table truth.

A	B	A ∨ B
0	0	0
0	1	1
1	0	1
1	1	1

DOI: 10.7717/peerj-cs.641/table-2

The ‘Exclusive-OR’ gate is a circuit that yields an output (1) if either one, but not both, of its two inputs are (1). The symbol with an encircled plus sign (⊕) is utilized to signal the Ex-OR operation, i.e., A ⊕ B (see A and B in Fig. 3 and Table 3). EXOR gate in our measure concerns only and only the non-shared features so as to non-shared features are given a chance to contribute in calculating similarity degree.

Table 3:

A EXOR B table truth.

A	B	A ⊕ B
0	0	0
0	1	1
1	0	1
1	1	0

DOI: 10.7717/peerj-cs.641/table-3

The Boolean logic algebra based similarity measure (BLAB-SM)

Based on the already drawn concepts, the Boolean algebra based similarity measure (BLAB-SM) is defined as follows:

(8) $$BLAB-SM\left(doc1,doc2\right)=\alpha \times X+\beta \times Y$$

Both α and β are parameters that have experimentally been tuned, as drawn in the results section, just to find their best values to detect the highest possible similarity. Although X is concerned with finding similarities between all non-shared features, it finds similarity between all shared features implicitly as well. It finds similarities based on logical sensing of the differences between both documents under consideration. All documents are logically treated and processed the same way logical gates work. On the other hand, Y emphasizes similarity through finding likeness between documents across all shared features only. Assuming having two documents which are document1 and document2 which are represented by vectors doc₁ and doc₂ in the vector space model using TF-IDF representation. Using doc₁ and doc₂ with “n” terms, X and Y are defined as follows:

$$X=\left(1-\frac{\sum _{I=1}^n\left(do{c}_{i1}\oplus do{c}_{i2}\right)}{\sum _{I=1}^n\left(do{c}_{i1}\vee do{c}_{i2}\right)}\right)$$

$$Y=\left(\frac{2\times \left(\sum _{I=1}^n\left(do{c}_{i1}\wedge do{c}_{i2}\right)\right)}{\left(\sum _{I=1}^{n}do{c}_{i1}\right)+\left(\sum _{I=1}^{n}do{c}_{i2}\right)}\right)$$

Meanwhile, doc₁ or doc₂=1, if doc₁ or doc₂ ≥1; Otherwise, 0.

BLAB-SM analysis

In this sub-section, we concisely, simply and informatively analyze the cases of the proposed measure. Assuming that X consists of X₁ and X₂, and Y consists of Y₁, Y₂, and Y₃. Accordingly, the X and Y equations are re-drawn as follows:

$$X=\left(1-\frac{X_1}{X_2}\right)$$

where:

$$X_1=\sum _{I=1}^n\left(do{c}_{i1}\oplus do{c}_{i2}\right)$$

$$X_2=\sum _{I=1}^n\left(do{c}_{i1}\vee do{c}_{i2}\right)$$

Similarly:

$$Y=\left(\frac{2\times Y1}{Y2+Y3}\right)$$

where:

$$Y_1=\sum _{I=1}^n\left(do{c}_{i1}\wedge do{c}_{i2}\right),$$

$$Y_2=\sum _{I=1}^{n}do{c}_{i1},and{Y}_3=\sum _{I=1}^{n}do{c}_{i2}$$

The perfect dissimilarity case:

This case happens when:

$$X_1=X_2\Rightarrow \mathrm{X}=\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o},\mathrm{b}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{X}\mathrm{w}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{d}\mathrm{e}\mathrm{q}\mathrm{u}\mathrm{a}\mathrm{l}\left(1-1=0\right)$$

$$\because X_1=X_2\Rightarrow Y_1=zero\left(itishereANDgate\right),X_1=1since{\displaystyle \frac{X_1}{X_2}=1}$$

$$\Rightarrow \mathrm{Y}=\frac{2\mathrm{x}\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o}}{Y_2+Y_3}=zero,regardlessofY2andY3values$$

$$\Rightarrow \mathrm{B}\mathrm{L}\mathrm{A}\mathrm{B}-\mathrm{S}\mathrm{M}\left(\mathrm{d}\mathrm{o}\mathrm{c}1,\mathrm{d}\mathrm{o}\mathrm{c}2\right)=\alpha \times X+\beta \times Y=\alpha \times zero+\beta \times zero=zero$$

Example (perfect dissimilarity case): assuming we have doc1 (3, 0, 1) and doc2 (0, 2, 0). By the applying the perfect dissimilarity scenario, we find that BLAB-SM=zero, for both documents (1, 0, 1) and (0, 1, 0), which is logically true since there is no shared feature exist.

The average similarity:

$$\begin{array}{c}{X}_2>X_1>zero\Rightarrow Y_1>\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o}\Rightarrow 1>\mathrm{X}>\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o},\mathrm{a}\mathrm{n}\mathrm{d}1>\mathrm{Y}>zero\Rightarrow 1>\\ \mathrm{B}\mathrm{L}\mathrm{A}\mathrm{B}-\mathrm{S}\mathrm{M}\left(\mathrm{d}\mathrm{o}\mathrm{c}1,\mathrm{d}\mathrm{o}\mathrm{c}2\right)>\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o}\end{array}$$

where (1) is the upper bound and (0) is the lower bound.

Example (Average similarity): assuming we have doc1 (3, 1, 1) and doc2 (6, 2, 0). By applying the average case scenario on both documents (1, 1, 1) and (1, 1, 0), BLAB-SM would have a value of roughly (0.73) which is bigger than zero and less than 1. It is worth indicating that 0.73 is logically reasonable than the cosine value which would reach 0.95. In fact, this is one novelty of our measure as similarity has never been exaggerated like what is done with the most state-of-art measure. Our measure allows non-zero non-shared features to have an implicit contribution to the similarity computation. Therefore, BLAB-SM takes the presence and absence of all features into consideration effectively.

The perfect similarity case:

$$X_1=zero\Rightarrow \mathrm{X}=1-\frac{\mathrm{z}\mathrm{e}\mathrm{r}\mathrm{o}}{X_2},$$

$\mathrm{X}=1,\mathrm{r}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{r}\mathrm{d}\mathrm{l}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}\mathrm{o}\mathrm{f}{X}_2$

$\because X_1=zero\Rightarrow Y_1=\overline{X_1}$

$\Rightarrow Y_1=1,since{Y}_1=complement\left(X_1\right)\Rightarrow \mathrm{Y}=\frac{2\mathrm{x}1}{Y_2+Y_3}$

$\because Y_1=1\Rightarrow Length\left(doc1\right)=Length\left(doc2\right)\Rightarrow Y_2=Y_3$

$\Rightarrow Y=\frac{2}{2x{Y}_2\left(or2x{Y}_3\right)}=\frac{2}{2}=1$

Example (perfect similarity Case): assuming we have doc1 (3, 3, 3) and doc2 (3, 3, 3), or doc1 (1, 1, 1) and doc2 (1, 1, 1). By applying the best case scenario, we find that BLAB-SM=1 which is logically true as both documents are equal and equivalent.

Similarity measure characteristics

Six characteristics should be defined on each similarity measure so this measure could be classified as good measure (Haroutunian, 2011; Amigó et al., 2020). These characteristics are given as follows:

Characteristic 1: The presence or non-presence of the targeted feature is more important than the difference between the values connected with the present feature.

Assuming we have doc₁ (3, 1, 1) and doc₂ (6, 2, 0). Then, the binary vectors, (1, 1, 1) and (1, 1, 0), of both documents are more important than the integer values linked to the values of features themselves in both documents. Let us take feature3 (f₃), we can say that feature3 has no link with doc₂ while it has a link with doc₁. In such case, both documents are dissimilar with respect to feature 3. So, we can conclude that feature3 has more weight in deciding similarity between doc₁ and doc₂ than f₁ and f₂ which are exist in both documents. In fact, BLAM-SM focus in this issue when similarity is being calculated as behavior of BLAM-SM goes in the same way the logic of gates mechanism works.

Characteristic 2: The similarity degree should increase when the difference between the non-zero features values decrease. For example, having feature1 and feature2 as two features (f₁ and f₂) of doc₁ and doc₂, similarity (doc₁, doc₂) while f₁=10 and f₂=5 is higher than the similarity between f₁ = 18 and f₂ = 4.

Characteristic 3: The similarity degree should be increased when the number of present features grows. For example, having three vectors of three documents; doc₁ (1, 1, 0), doc₂ (1,1,1), doc₃ (1, 0, 0), the similarity (doc₁, doc₂) is far higher than similarity (doc₁, doc₃) due to the difference in the number of present or non-present features between all documents.

Characteristic 4: The document Pairs are highly different to each other if there have not had almost equivalent zero-valued features versus non-zero-valued features. For example, having two vectors of two documents doc₁ (f₁, f₂) = (1,0) and doc₂ (f₃, f₄) = (1,1), doc₁.f₂ and doc₂.f₄ are the main reason for maximizing the difference between both documents as f₂ * f₄ = 0, and, f₂ + f₄ > 0.

Characteristic 5: the symmetric properties should be met by each similarity measure. For example, the similarity degree between doc₁ (0, 1, 1, 0) and doc₂ (1,1,1,1) must be the same when doc₂(1,1,1,1) and doc₁ (0, 1, 1, 0) are addressed.

Characteristic 6: The value of distribution should contribute to the similarity degree between each document pair.

KNN classifier

In general, KNN examines each test point based on its “K” closest “neighboring” points which are sorted and assumed to be the most similar points. Then, the test point is classified based on the voting technique. It takes the class label of the majority vote among all K neighboring training points in the feature space. The neighborhood is basically determined based on the used distance/similarity measure. Let us assume that d_i is the training set, and d_i^* is the testing set, c is the true class of training set, and c* is the predicted class for the testing set (c, c*= …, m) where m is the number of classes. In the training classification process, only the true class c of each training set is used, and during testing, class c* is predicted for each testing set. On the other hand, when using the one-nearest neighbor rule, the predicted class of the testing set d_i^* is set equal to the true class c of its nearest neighbor, where w_i is the nearest neighbor to d* at the distance:

(10) $$d\left(w_i,d_i^{\ast }\right)={}_j^{min}\left\{d\left(w_j,d_i^{\ast }\right)\right\}$$

For KNN, the predicted test sample class d_i^* is assigned to the most frequent true class among the K nearest training sets.

The experimental design flow chart, provided in Fig. 4, shows briefly how the KNN classifier was used to perform classification task successfully.

Machine description

This work has been written in python language, and run on Processor Intel Core i5-3320M (2.6 GHz), RAM 4GB with OS Windows 7 (64 bit).

Datasets description

Reuters-R8 (Table 4): it consists of 7674 documents with eight classes, and has 18308 features after it has been pre-processed.

Web-kb (Table 5): it consists of 4199 documents with four classes, and has 33,025 features after it has been processed. It is composed of web pages of computer science from universities: Cornell, Texas, Washington, and Wisconsin.

Both Tables 4 and 5 holds the description of both datasets used in this work. The data in each datasets was split into training and testing in ratio 2:1 (67%: 33%). It is worth indicating that these datasets are publicly available (https://github.com/aliamer/Boolean-Logic-Algebra-Driven-Similarity-Measure-for-Text-Based-Applications/blob/main/Reuters%20%2B%20WebKB%20datasets.rar).

Classification evaluation criterion

Clustering evaluation criterion

Classification results

Fixing α and β on values (0.5; 0.5), this work examines all similarity measures thoroughly using KNN classifier and K-means clustering algorithm. The K value in KNN was changed in each run from (1) to (120) as given in the Appendix. Furthermore, the number of features was falling in one of these values (10, 50, 100, 200, 350, 3000, 6000, NF) where NF is the whole number of features. After that, the results are averaged for each measure on all K values as given in Tables 6–14. On the other hand, albeit it has long been proven effective when dealing with the BoW model, the Jaccard measure has been excluded due to its being proven ineffective option when dealing with TF-IDF representation (see the Appendix). That is because of the fact that Jaccard is heavily based on the common values between each considered pair. However, the common values with TF-IDF is far less than it is with BoW. This fact justifies the poor behavior of Jaccard when dealing with TF-IDF-based document matching. The bolded values in the next Tables signify the best values. For readability, Accuracy, Precision, Recall, F-Measure, G-Measure, and Average Mean Precision are represented by ACC, PRE, REC, FM, GM, and AMP respectively.

In Table 6, for Reuters dataset, the BLAB-SM similarity measure, Manhattan followed by KLD achieved the highest accuracy. Ex-jaccard, although it was not among the best performers but it still outperformed Euclidean, SMTP, and cosine respectively with regard to accuracy. On the other hand, KLD followed by BLAB-SM and Manhattan had the best FM and AMP with BLAB is better with FM. However, with regard to the Web-KB dataset, Manhattan, followed by KLD and Euclidean, met the highest accuracy. For FM and AMP criterions, Manhattan followed by Euclidean and cosine, outweighed all other measures. Thus, the best measures were BLAB-SM, Manhattan, and KLD on Reuters, and Manhattan followed by KLD and Euclidean on Web-KB.

In Tables 7 and 8, it is obvious that, for Reuters dataset, BLAB-SM followed by SMTP and cosine, obtained the highest ACC. However, with FM and AMP, Manhattan, Euclidean, cosine and SMTP obtained the best values. On Web-KB dataset, SMTP, followed by BLAB-SM, and cosine, achieved the highest ACC, FM and AMP respectively. Thus, the top performer measures were SMTP, BLAB-SM, and cosine. From Table 9, on Reuters and Web-KB datasets, BLAB-SM, followed by SMTP and cosine, obtained the highest ACC, FM, and AMP with cosine being superior over BLAB-SM and SMTP in terms of FM and AMP. Thus, the top performers were BLAB-SM, SMTP, and cosine.

In Table 10, on Reuters dataset, Cosine, followed by BLAB-SM and SMTP, obtained the highest ACC. Interestingly, Cosine had been superior to BLAB-SM and SMTP with FM and AMP. However, on Web-KB dataset, like Table 9, BLAB-SM, SMTP followed by cosine obtained the highest ACC, FM, and AMP respectively. Thus, the top performers were BLAB-SM, SMTP, and cosine.

From Table 11, on Reuters dataset, SMTP, followed by BLAB-SM and cosine, obtained the highest ACC, FM, and AMP with cosine being superior over SMTP and BLAB-SM in terms of FM and AMP. However, on Web-KB dataset, BLAB-SM, followed by SMTP and cosine, obtained the highest ACC, FM and AMP respectively. Thus, the top performers were BLAB-SM, SMTP and cosine.

In Tables 12 and 13, on Reuters dataset, SMTP, followed by BLAB-SM and Ex-jaccard, obtained the highest ACC. Cosine, Euclidean, SMTP, and BLAB-SM were the best on FM and AMP. However, on Web-KB dataset, BLAB-SM, followed by SMTP and cosine, obtained the highest ACC, FM, and AMP with cosine being superior over BLAB-SM and SMTP in terms of FM only. Thus, the top performers were BLAB-SM, SMTP, Ex-Jaccard and cosine.

Finally, and most importantly in the averaged results case, in Table 14, BLAB-SM and SMTP followed by Cosine were the best measures on both datasets with BLAB-SM taking the lead over SMTP, and SMTP taking the lead over cosine. Surprisingly, cosine/Reuters was the top in terms of FM and AMP.

Clustering results

We fixed K on 5, 10, and the number of actual classes of each dataset. The K-means was conditioned to stop after (50) iterations, or the stability has been reached after two successive cycles. Centroids were randomly picked in each iteration. The best measures based on Table 15, BLAB-SM went side by side with Euclidean followed by SMTP and KLD. The bolded values in Tables 15 and 16 signify the best values.

On the other hand, based on results of Table 16, Manhattan, BLAB-SM, SMTP followed By Cosine have been the best measures.

Measures stability

Recalling the results of Tables 6–14, 17 holds the most stable measures. In Table 17, R and W refer to both Reuters and Web-KB respectively.

Table 17 shows that SMTP, BLAB-SM and cosine as the most stable measures with 40, 39, and 39 points respectively. It gives one plus (++1) for each measure when a measure has been superior in terms of any criterion (ACC, FM, and AMP). On Reuters, the competition was held between BLAB-SM, SMTP, cosine, and Ex-Jaccard. Moreover, it can be concluded, from Table 17, that BLAB-SM, SMTP, cosine can be used effectively for both low, middle and high dimensional datasets on each NF. Euclidean and Manhattan also performed well on low dimensional datasets (NF in [10–200] features). EX-Jaccard was observed to behave well on the middle and high dimensional datasets (NF in [350–N] features).

Performance analysis

In this section, using the accuracy, f-measure, and average mean precision, we analyze the measure’s performance.

Reuters

Figure 5 shows BLAB-SM, SMTP, Ex-Jaccard, and Cosine achieved the most stable performance. The Cosine and ex-Jaccard showed a punctuated accuracy as cosine was superior when NF was in the range [10–3,000]. Ex-Jaccard had the lead as NF grew, though. When NF = 6,000, competition restricted among Cosine, Ex-jaccard, SMTP, and BLAB-SM with SMTP, BLAB-SM, and Ex-Jaccard being the best. However, as NF exceeded 6,000 features, the competition was held between SMTP, BLAB-SM, and Cosine with SMTP and BLAB-SM being the top performers. On the other hand, Manhattan, Euclidean, and kullback Leibler had the worst performance chiefly as NF grew. Albeit that their performance is good when the number of features (NF) were between 10 and 350, performance deteriorated steadily as NF surpasses 350.

In general, based on Figures 6 and 7, it can be concluded that Cosine, BLAB-SM, and SMTP had an almost close performance with slight superiority given to Cosine. Euclidean was shown a competitor with cosine when all features were considered, though. As expected, KLD, Manhattan, Euclidean had poor performance compared to BLAB-SM, SMTP, and Cosine. The Ex-jaccard had a competitive performance on Reuters, though.

Web-Kb

Figures 8–10 illustrate the map of the criterion movements (results were averaged) for all measures over several NF values. On one hand, in Fig. 8, Cosine had been the middle ground between the first group of measures (KLD, Manhattan, Euclidean, and Ex-Jaccard) and those with the best performance (BLAB-SM and SMTP). BLAB-SM and SMTP had been involved in fierce competition with SMTP being superior as NF was in the range [10–100]. However, BLAB-SM had shown higher superiority as NF grew and surpassed 200 features. On the other hand, Fig. 8 reveals that Manhattan and kullback Leibler performed poorly as NF grew. Surprisingly, Ex-Jaccard had poor performance as NF was in the range [10–200]; yet, as NF grew, its performance grew and was competitive with Euclidean and cosine.

Figures 9 and 10, on the other hand, show almost the same conclusion deduced from Fig. 7, with BLAB-SM being better than SMTP in terms of FM and AMP. In general, like Reuters, both BLAB-SM and SMTP had a close performance trend with significant superiority given to BLAB-SM regarding ACC and AMP. Interestingly, cosine was highly competitive with BLAB-SM and SMTP in terms of FM and AMP. Euclidean was an equivalent to cosine when all features were considered, though. Finally, KLD, Manhattan, Euclidean, and ex-Jaccard had lower performance on Web-KB compared with BLAB-SM, SMTP, and Cosine. Similarly to Reuters, Euclidean was superior to Ex-Jaccard when NF was in the range [10–350]. However, Ex-Jaccard had the lead when NF grew except for one case.

Last but not least, Figs. 11 and 12 showcase the averaged performance of all measures on both datasets. Both figures stress that BLAB and SMTP have an almost equal performance trend as the top performers.

Finally, to provide the statistical evidence for the robustness of BLAB-SM’s performance, the results of BLAB-SM against its rival measures have been verified on both datasets using the standard test of the non-parametric Wilcoxon test and the paired t-test. The statistical details for BLAB-SM, against all seven similarity measures, have been made based on the Accuracy “ACC” metric on both datasets, and the results are drawn in Tables 18–23. Results show that BLAB-SM is highly effective on Web-KB, and competitively effective with SMTP and Ex-Jaccard on Reuters. The statistical analysis is made by setting the standard value of the significance level at 0.05 (95%), and results are analyzed regarding the MAE, MSE, Std Error, z-score, p-value, and t-score. The degrees of freedom (DF) for the paired t-test have been assigned the value of 59.0 as we have 60 K values (1–120, +2) for the KNN classifier (see Appendix). The negative sign of z-score and t-score implies the effectiveness and superiority of BLAB-SM comparing with its rival measures. In other words, both z-score and t-score indicate the significant difference between the BLAB-SM and its rivals.

Furthermore, for both datasets, the p-value of BLAB-SM is less than the value of the significance level (0.05). This is strong evidence for the performance robustness of BLAB-SM comparing with its rival measures. On one hand, based on the statistical results given in Tables 18–23, the negative sign of both z-score and t-score shows that the BLAB-SM measure is the top performer comparing with its rival measures. Moreover, z-score values of the BALB-SM measure are maximally significant in all cases as its p-values are less than (0.05) which is the level of significance. On the other hand, on Reuters only, in terms of t-score, all the results are significant except of SMTP and Ex-jaccard. Nevertheless, this slight insignificance of BLAB-SM could be compensated by the maximally-significant run time of BLAB-SM against both SMTP and Ex-Jaccard on Reuters.

Clustering analysis

Using the results of Tables 15 and 16, the analysis is made in Table 24. The points achieved by each similarity measure are accumulated on each corresponding metric. The whole number of points is 12 points. That is because of having two datasets and two metrics on three values of clustering variable (K = 5, K = 10, and K equal the number of actual classes).

Execution time analysis

The time of each measure was accumulated and averaged as given in Figs. 13 and 14. It is clear that all measures share one fact, in particular Ex-Jaccard and SMTP, the run time is increasing sharply as NF increases steadily. Figures 13 and 14 give the run time of classification on Reuters and web-kb respectively.

As it is shown in Fig. 13, BLAB-SM followed by Euclidean and Jaccard were the fastest similarity measures with exception when NF=3000 in which KLD was faster. Manhattan followed by cosine and KLD came the next batch of the fastest measures. On the other hand, the slowest measures were SMTP and Ex-Jaccard with Ex-Jaccard being the slowest. Considering Fig. 13, BLAB-SM has been the sole measure that meets the ultimate quest of this work in terms of both effectiveness and efficiency at the same time. It is also worth indicating that the BLAB-SM has been as effective as SMTP and much better than Cosine. However, it is maximally efficient comparing with SMTP and more efficient than Cosine. Considering being faster, BLAB-SM could replace SMTP for text classification and clustering effectively and efficiently.

Also, Fig. 14 shows that Jaccard, Euclidean, BLAB-SM, and Manhattan were the fastest similarity measures. Interestingly, like Reuters, BLAB-SM confirms it is being fast measure on Web-KB as well. In general, Cosine was faster than KLD yet slower than BLAB-SM. On the other extreme, Ex-Jaccard and SMTP were seen to be slower measures with Ex-Jaccard being the slowest. In conclusion, according to all results drawn in Tables 8–16, and considering Fig. 14, BLAB-SM has been the sole measure that meets the ultimate quest of this work in terms of effectiveness and efficiency comparing with SMTP in particular and all measures in general. BLAB-SM has been as effective as SMTP, but maximally efficient compared to SMTP, and it also could replace SMTP and Cosine for text classification effectively and efficiently.

Merits of BLAB-SM

In this subsection, it is worth referring briefly to the merits of our proposed measures over its competitors chiefly the SMTP. Concisely, the merits could be drawn as follows; (1) BLAB-SM has a simplistic design compared to SMTP. Moreover, BLAB-SM is bounded by upper and lower values in the same design with no complexity being added in the design, SMTP however needed an additional condition to restrict values of SMTP between zero and one (Kumar Nagwani, 2015). (2) BLAB-SM has been shown as effective as SMTP (and even better as it is the case on Web-KB) on the high dimensional dataset as all results asserted this claim when NF grew and exceeded 6,000 features on both datasets compared to SMTP. In other words, they both have an almost equal performance trend. Finally, (3) BLAB-SM has been observed to be impressively efficient compared to SMTP in particular and all state-of-art measures in general. Time Tables and graphs asserted this claim (see Table 25). When running measures on all features, the BLAB-SM has barely reached 29 min on Reuters and 23 min on web-KB compared to SMTP which needed roughly 804.933 min on Reuters and 336.567 min on Web-KB respectively. It is also noted that BLAB-SM is significantly faster than cosine which needed roughly 58.28 min on Reuters and 26.37 min on Web-KB respectively. With the BLAB-SM is being high-speed measure of a competitive performance, BLAB-SM could be nominated to replace Cosine and SMTP for the text-based applications, chiefly for the big “large-scale” datasets. The run time has been recorded when all features of both datasets were being considered. Table 25 summarizes the run time of all measures along with the BLAB-SM’s improvement Rate against these measures.