Self-optimizing neural network in the classification of real valued data

SONN formalism

The idea of Self-Optimizing Neural Networks (SONN) was introduced in Horzyk & Tadeusiewicz (2004); Horzyk (2012) In this subsection, we show a recapitulation of SONN formalism introduced by Horzyk.

Let $\mathcal{U}$ be a set of the form: (1)${\displaystyle \mathcal{U\; =}\left\{\left(u^n,c^n\right)\right\},}$ where ${\mathbf{u}}^n=\left[u_1^n,u_2^n,\dots ,u_F^n\right],$ $u_f^n$ is the value of the f-th feature for n-th pattern and $u_f^n\in \left\{-1,0,1\right\}$, n = 1, 2, …, N_P,  N_P is the total number of patterns, f = 1, 2, …, N_F, N_F is the number of features, cⁿ is the class for the n-th pattern and $c^n\in {\mathcal{N}}_{\mathcal{C}}$, where ${\mathcal{N}}_{\mathcal{C}}$ is the set of classes.

Let $P_f^c$ denote the number of patterns with values greater than 0 in f-th features in c-th class and $M_f^c$ denote the number of patterns features with values less than 0 as noted (2)${\displaystyle P_f^c=\sum _{u_f^i\in \left\{u_f^i>0,i=1,2,\dots ,Q^c\right\}}{u}_f^i,M_f^c=\sum _{u_f^i\in \left\{u_f^i<0,i=1,2,\dots ,Q^c\right\}}{u}_f^i,}$ where Q^c is the number of patterns in the given c class.

Having unknown values in a particular feature in the discrete value vector (marked with zero), we can estimate the probability of the occurrence of the feature with a value one or −1. We use the values of this feature in other patterns from the class under consideration. Then, a useful formula is the probabilistic determination of the value of this feature (the middle part of each formula (3)). (3)${\displaystyle x_{f^{+}}^n=\left\{\begin{array}{cc}1\hfill & u_f^n=1\hfill \\ \frac{P_f^c}{P_f^c+M_f^c}\hfill & u_f^n=0\hfill \\ 0\hfill & u_f^n=-1\hfill \end{array}\right.,x_{f^{-}}^n=\left\{\begin{array}{cc}1\hfill & u_f^n=-1\hfill \\ \frac{M_f^c}{P_f^c+M_f^c}\hfill & u_f^n=0\hfill \\ 0\hfill & u_f^n=1\hfill \end{array}\right.,}$ where $x_{f^{+}}^n$, $x_{f^{-}}^n$ are the approximate values of the indeterminate features. Then the patterns Eq. (2) are of the form (4)${\displaystyle {\hat{P}}_f^c=\sum _{u_f^i\in \left\{u_f^i>0,i=1,2,\dots ,Q^c\right\}}{x}_{f^{+}}^i,{\hat{M}}_f^c=\sum _{u_f^i\in \left\{u_f^i<0,i=1,2,\dots ,Q^c\right\}}{x}_{f^{-}}^i.}$

Even though SONN allows the use of indefinite values and the special masking procedure has been proposed, we do not deal with such data in our work, therefore (5)${\displaystyle {\hat{P}}_f^c=P_f^c{\hat{M}}_f^c=M_f^c}$

The described theoretical basis will allow us to understand the method of calculating the coefficient of discrimination (‘Fundamental coefficient of discrimination’).

Data preparation

Self-optimizing neural networks are adapted to operate on discrete values. Therefore, when working on a certain set of patterns (i.e., elements contained in a data set), we considered a vector of values representing them individually. This vector represents discrete features of a pattern. It has values from the set { − 1, 0, 1}, where −1 means that a feature does not appear in a given pattern, by 0 we denote a feature that is undefined, and 1 means that the pattern has values in a given feature. Regardless, SONN works on discrete feature space, we can use it for real valued datasets like: Banknote Authentication dataset, Iris dataset, and Combined Cycle Power Plant dataset (Dua & Graff, 2017). The real features existing in dataset (i.e., petal length for Iris) we call attributes. We use a special procedure of discretization such datasets. We adjusted each of these sets to use the self-optimizing neural network. For instance, the Iris dataset has four attributes represented as real numbers (x, y, z, t). Table 1 shows selected ten patterns from the Iris dataset (the whole dataset has 150 patterns).

Each of p-th patterns in a dataset is described by a real attributes vector (6)${\displaystyle {\mathbf{v}}^n=\left[v_1^n,v_2^n,\dots ,v_a^n\right],}$ where 1 ≤ n ≤ N_P, N_P is the total number of patterns and $v_a^n$ is the value of a-th attribute for n-th pattern, a = 1, 2, …, N_A, N_A is the total number of attributes.

In our article, we divide the range of variation of each N_A attribute into r uniform parts. Split by other division (e.g., clustering) methods are also possible. By uniformly dividing the range of each of the four attributes into three parts, we get three subranges (Table 2). In this way, we obtained a sequence of twelve discrete features representing the pattern (Table 3). Those features we call a discrete feature vector or discrete value vector. We distinguish attributes (real dataset features) from features (pattern discrete features). As was mentioned, if the attribute value of a given pattern is an inappropriate subrange, we put one in the corresponding place of the discrete feature vector, otherwise −1. If we have an attribute with an undefined value, we set the appropriate elements of the feature vector as 0. Described discrete feature vector we denote as ${\displaystyle {\mathbf{u}}^n=\left[u_1^n,u_2^n,\dots ,u_F^n\right],}$ where $u_f^n$ is the value of the f-th feature for n-th pattern (see 1).

The described discretization procedure can be applied to any real dataset.

Fundamental coefficient of discrimination

The discrimination coefficient (discriminant) concerns statistical differences in features quantity after their values in classes and the learning dataset. The bigger discriminant, the more important and distinguishing the feature is in view of classification (Horzyk & Tadeusiewicz, 2004).

Discriminants describe how well the feature f distinguishes class c from other classes, considering their patterns in the learning set.

Horzyk & Tadeusiewicz (2004) proposed the following formula for the discriminants (7)${\displaystyle d_f^{c^{+}}=\frac{{\hat{P}}_f^c}{\left(N_C-1\right)Q^c}\sum _{\begin{array}{c}h=1\hfill \\ h\ne c\hfill \end{array}}^{N_C}\left(1-\frac{{\hat{P}}_f^h}{Q^h}\right),d_f^{c^{-}}=\frac{{\hat{M}}_f^c}{\left(N_C-1\right)Q^c}\sum _{\begin{array}{c}h=1\hfill \\ h\ne c\hfill \end{array}}^{N_C}\left(1-\frac{{\hat{M}}_f^h}{Q^h}\right),}$ (8)${\displaystyle d_f=max\left\{d_f^{+},d_f^{-}\right\},}$ where $d_f^{c^{+}}$ and $d_f^{c^{-}}$ mean the coefficient of determination calculated for a given feature of a pattern in the appropriate class, taking into account the values of positive or negative features, respectively. Moreover c denotes the class, c = 1, 2, …, N_C and Q^c is the number of patterns in the given class c. The discrimination coefficients’ values can run across the whole set of real numbers.

Structure of the network and the weight factor

The structure of the network is related to the described discretization procedure (‘Data preparation’). It was mentioned that this method bases on division into subranges and then maps the real value attributes to the discrete values vector. (see example in Tables 1–3).

Having computed discriminants for each feature of each pattern, we have a matrix representing these values.

The first step in constructing the network was to rank the features of each pattern according to the chosen ordering (see 2.5). Horzyk proposed a certain method of network construction consisting of the iterative addition of successive layers of the network based on features that best distinguish patterns from a given class among all other patterns from other classes (Horzyk & Tadeusiewicz, 2004).

We replaced these concepts with a set of unique (on the training set scale) pairs consisting of the discrete values vector (features) and the corresponding class. Therefore, this pair is of the form: (9)${\displaystyle \left({\mathbf{u}}^{\mathbf{s}},c^s\right),}$ where u^s is the discrete vector and c^s is the corresponding class. From now on, we will call each such vector a path, and all paths will be a set denoted by $\mathcal{S}$. We will call a neuron each vector element used to create a path.

The resulting structure can be identified with a decision network (which shows similarities to neural networks) or a graph. There are situations where one path is associated with many classes and vice versa, that one class is related to several paths.

No matter how the structure was formed, the connections between neurons have a certain weight factor. The coefficient is calculated based on the properly organized network (in our case, identified by the set of path s $\mathcal{S}$).

The weight factor for a neuron j of a given path s is calculated as follows (10)${\displaystyle w_j^s=u_j\frac{d_j^s}{\sum _{\begin{array}{c}i<=j\hfill \end{array}}{d}_i^s},}$ where u_j is the value of the feature on the basis of which the neuron j is created, $d_j^s$ is the value of the discriminant on the basis of which this neuron is created. Moreover, $d_i^s$ specifies the value of the discriminant on the path under consideration $s\in \mathcal{S}$, for i <  = j neurons, i.e., all those neurons (with appropriate discriminants), which were considered before together with the current one. Note that according to the chosen ordering features with appropriate discriminants, there is a change to the denominator (see 10 and 2.5).

Network response

We tested some modifications of ordering features of each pattern during our work and calculate the weight factor. For further research, we chose two methods of calculating the response from the network for a given pattern.

Algorithm version v₂.

In the second approach, we used the decreasing ordering of discriminants. The response from the network for a given pattern is given by (12)${\displaystyle ou{t}_s=f\left(d_{max}^s\sum _{i=0}^{N_F-1}{w}_i^s{c}_i\right),}$ where $d_{max}^s$ is the highest value of the discriminant on path s,  $s\in \mathcal{S}$, $w_i^s$ is the weight factor for a given neuron i on the s path and f is the linear function. As we already know (see 2.3) coefficient of discrimination measures the relationship between features and path. High values of the discriminant mean that a given feature is important for a given path (perhaps only for this one) The use of $d_{max}^s$ may change the relative strength of the individual paths depending on the discriminants. The low value of the discriminants shows that there is little impact. The greater the value, we can assume that it becomes a certain component that allows strengthening the impact of the result for a given path.

The path introduced in Eq. (9) joins the feature vector with the corresponding class. We want to identify the vector u^s with a hypercube. The well-known method of presenting the multi-dimensional data is its placing in the N_A-dimensional space, where N_A is the number of attributes. In such an approach, the feature vector u^s identifies unambiguously one of the N_A-dimensional hypercubes, defined, in this space, by the real attributes division into subranges.

For an exemplary illustration, we show Fig. 1 and Table 4. In this figure, blue points are the learning patterns, whereas red ones are the test patterns we want to assign to a certain class. Furthermore, the shape of the blue markers indicates the distinction between classes. For simplicity, in this article, hypercubes in two dimensions are rectangles. The considered u^s is graphically represented as a rectangle in two-dimensional space. As can be noticed, in the rectangle (marked u^s), there are filled markers in two shapes. It means they have the same features vector u^s, i.e., they belong to the same hypercube. Moreover, there are two path with different classes.

When calculating the answer from the network for a certain test pattern, we use a defined neighborhood describing path s located at a certain distance from the test pattern and having an impact on the obtained result (Fig. 1). We used two different metrics: Chebyshev and Manhattan. Each of them is responsible for determining which paths contribute to the network’s response. (13)${\displaystyle ou{t}_s=ou{t}_s\ast 2^{-\mu \left({\mathbf{u}}^{\mathbf{i}}-{\mathbf{u}}^{\mathbf{s}}\right)},}$ where μ_Manhattan and μ_Chebyshev are the well known metrics to calculate distance between hypercubes, N_A is the number of real attributes, uⁱ and u^s are the feature vectors form the i pattern and the path s, respectively.

Table 4:

Two exemplary patterns are uⁱ and u^j, together with u^s they are the feature vectors representing the ith pattern, jth pattern and the path s.

Due to the fact that there are only two real features (x, y), the considered u^s is graphically represented as a rectangle in two-dimensional space.

	feature vector
	x			y
pattern	1	2	3	4	5	6
ui	−1	1	−1	−1	1	−1
uj	−1	−1	1	1	−1	−1
us	−1	−1	1	1	−1	−1

DOI: 10.7717/peerjcs.1020/table-4

Regardless of the used modification, the evaluation of the winning class, for each element in the test set, is as follows: (14)${\displaystyle ou{t}_{\sigma }=\underset{s\in \mathcal{S}}{max}\left\{ou{t}_s\right\}\text{for}\sigma \in \mathcal{S}.}$ The finding path σ gives the highest network response, and its class is the winning one.

The important information concerning the method is its computation complexity. It is ruled by Eq. (2), which assumes the analysis of all pattern vectors and all features inside these vectors. We can estimate the complexity of the learning phase as O(N_P∗N_F) where N_P is the number of patterns in the training set and, defined earlier, N_F - the number of features. For the classification process, Eq. (11) gives us direct estimation of single pattern recognition as O(N_F).

Data

As was mentioned before, we show how our procedure works for the data from the Machine Learning Repository (Dua & Graff, 2017). We choose the Iris dataset, Banknote Authentication dataset and Combined Cycle Power Plant dataset.

The Iris dataset (Dua & Graff, 2017; Fisher, 1936) is one of the most popular collections used for machine learning problems. It is used as the first and simplest dataset to verify clustering methods. This small dataset comprises 150 patterns representing three uniform classes (Iris Setosa, Iris Versicolor, and Iris Virginica). Each pattern is represented by a vector of four attributes (sepal length, sepal width, petal length, and petal width).

The next is the Banknote Authentication dataset (Dua & Graff, 2017; Gillich & Lohweg, 2010), which was prepared according to a special procedure. The banknotes images were analyzed using the wavelet transform, and several characteristics were collected from such transformed data. The set comprises 1,372 patterns belonging to two classes(binary classes). The values of five attributes describe patterns, and the statistical moments like variance, kurtosis or skewness, and also the entropy of the image. There are many references in the literature to the classification issue related to these data, i.e., Kumar & Nagamani (2018) and Shrivas & Gupta (2017).

The most complicated of investigated datasets is the Combined Cycle Power Plant dataset (CCPP) (Dua & Graff, 2017; Tufekci, 2014; Kaya, Tufekci & Gurgen, 2012). It contains 9,568 patterns gained from a combined-cycle power plant over six years (2006–2011). While collecting, the power plant was set to work with a full load. Patterns are described by attributes: hourly average ambient variables temperature (T), ambient pressure (AP), relative humidity (RH), and exhaust vacuum (V) to predict the net hourly electrical energy output (EP) of the plant. As we can see, all data have floating-point values. Contrary to the above datasets (CCPP), we do not have an explicit class division. The authors of this dataset do not define the number of classes, but we do it through our actions. Using the recursive k-means tool (Miniak-Gorecka, Podlaski & Gwizdalla, submitted), we divide the response set (EP) into three classes. There are many references to the classification issue related to these data in the literature i.e., (Saleel, 2021; Santarisi & Faouri, 2021; Alketbi et al., 2020; Siidiqui et al., 2021).

In Table 5, the datasets based on which we perform the research are listed briefly. In the visualization (see ‘Results’), we used the labeling from the first column. The first three rows contain the information about original datasets from Machine Learning Repository (Dua & Graff, 2017). The remaining CCPP1, CCPP2 were created from the original CCPP by reducing the number of elements and using specific selection procedures. The resulting datasets contain 450 elements selected as follows:

CCPP1

–elements are selected randomly from the entire CCPP,

CCPP2

–each of three classes is represented uniformly by selected 150 random instances.

We are aware that different selection methods may affect the results obtained; however, as we mentioned, we try to omit the ordering of the dataset.

The differences between the datasets used in experiments are presented in Fig. 2. We show the data projection onto a two-dimensional space with the t-SNE method (Hinton & Roweis, 2002). The t-distributed stochastic neighbor embedding (t-SNE) is based on the minimization of Kulback-Leibler divergence. The divergence is calculated between two distributions in the original and two-dimensional target space. As shown in this figure, the distances are usually calculated with the Euclidean metric, but arbitrary metrics can be used. We can observe that classes are well separated for Iris and Banknote datasets. It is impossible for the CCPP dataset to separate classes, especially patterns belonging to class 2, covering a whole range of working parameters. It suggests that for this dataset, the prediction can be most challenging.

Experiment setup

In the further descriptions of tests and the discussion of the obtained results, we use the terminology presented in Table 6. The differences between two approaches (algorithm versions) considered is clarified in ‘Network response’, where the self-optimizing neural network is described. The division specifies the number of {5, 6, …, 15} uniform subranges for each real attributes (Tables 1–3). When calculating the answer from the network for a certain test pattern, we use defined neighborhood pointing path s located at a certain distance from the test pattern that has an impact on the obtained result (‘Network response’ and Fig. 1). We use two different metrics: Chebyshev and Manhattan. They are responsible for determining which path s contribute to the network’s best response. The next term ratio specify what percentage of all patterns in dataset enters the training (80, 70, 60, 50) and test (20, 30, 40, 50) set, respectively.

All presented results of the recognition of test patterns are based only on those test patterns that do not have an equivalent among the training patterns. It means that in our approach, no u^s, $s\in \mathcal{S}$ from path s can efficiently classify a given test pattern (see (9) and Fig. 1). By the good answers we mean those answers that match the expected output (expected class). In addition, for each of the possible configurations of parameters (version, division, metric, ratio) we perform thirty tests. We denote as avg_SONN the average of good SONN answers for thirty of those tests for each given configuration of parameters. As avg_SVM we mark the average of good SVM answers for those thirty tests (applied parameters do not affect the operation of the SVM method). The scale of improvement determines how much our method is better (worse) than the considered classic SVM and it is calculated as follows

(avg_SONN − avg_SVM).

We also point out that the classic SVM works on real values (attributes), while our method is based on their discrete forms (in the form of discrete vectors) and our method has the possibility of setting a combination of parameter values (version, division, metric, ratio) to improve the network response.

All figures included have a uniform format for presenting data. The shapes of markers (diamonds and crosses) correspond to different metrics. The color (red and blue) indicates the version of approach. Filling the shape gives us a distinction in terms of ratio. For all pictures, we use the naming entered in the Table 6.

Results

In our experiments, we focused only on test patterns that do not have learning patterns in their hypercube (like uⁱ in Fig. 1). We establish that more than five of these test patterns must be in the set. For each configuration of parameters (version, division, metric, ratio) we perform thirty tests. The results presented in this section are the average results of the tests performed.

Figure 3 shows the dependence of the improvement on the division for all datasets. The ratio is set to twenty. In this case, the variable parameters are version and metric. Each of Figs. 3A–3E presents results for a separate dataset (Table 5). The positive value of the improvement indicates that compared method gives better results than the classic SVM. In our work, we use the python version of SVM(with the following parameters: kernel =’poly’, C = 1, gamma =’scale’, degree =3) (Pedregosa et al., 2011), and we provide the results of this tool so that we do not have to go into other research on this method. The higher the value of improvement, the more suitable our method is. Figures 3A and 3B show how the tested methods behave in the case of well-known and small datasets Iris and Banknote Authentication, respectively. We note that the SVM method works well for Iris. Currently, it is possible to find research carried out with the use of Iris dataset in various sources, in which, after numerous changes, even one hundred percent effectiveness was obtained. The SVM version is also specially modified for this dataset there. It should be noted that SVM uses more information than the proposed SONN method. The discretization procedure is responsible for this difference; the real-valued attributes have more information than the discrete value vector. With the increase of the division parameter, the values of the improvement become negative, and the difference between SVM and our results increases. SVM seems to be more suitable in that case. For the Banknote dataset, while increasing the partition density, we get the highest number of empty hypercubes. The average number of paths per cube decreases as the split is denser. For test patterns falling into empty hypercubes, we calculate the network response based on the responses of adjacent cubes. The influence of adjacent hypercubes increases when patterns are more sparsely distributed. For partition density above ten, our results and SVM results are practically the same. Although, for both so far discussed datasets, the results are unsatisfactory. We can observe the differences in the plots (Figs. 3A and 3B) presented. For one of the datasets (Iris) the performance of our method is visibly worse than SVM, but for Banknote we can notice the similarity of the results for both techniques. We want to test our method using more demanding data; that is why we also use the CCPP set containing almost ten thousand patterns. With the increase of the complexity of the studied data (Figs. 3C–3E), our method looks more efficient. All four Figs. 3C–3E show the results for CCPP. The results in Fig. 3C present the different behavior than the former ones (Figs. 3A and 3B). To better understand the behavior of average improvement, we also present its standard deviation (Table 7). We can see that with the growth of division parameter, it does not only increase the improvement but also its deviation decrease. It suggests that the proposed methods with both metrics give statistically better results than SVM. In order to assess the dependence of total datasets size, we create two subsets of CCPP dataset. These subsets preserve the complexity of the total CCPP dataset and are even more sparse distributed. For these subsets, we select the number of patterns settled between sizes of Iris and Banknote datasets. Figures 3D–3E are created, for the needs of research, excluding the ordering source data (Table 5). An interesting effect (especially Fig. 3E) is that for small sets we get a significant improvement. For some divisions, our method turns out to be much better than SVM used on the same data. Even for denser divisions (more than ten subranges), our method seems to be statistically better than SVM. As it is shown in Table 7 the average improvement exceeds the standard deviation for the higher value of the division. The Manhattan metric gives better results for small values of division, but when we go towards tighter divisions, it is surpassed by Chebyshev metric. Almost always, except for the crosses in Fig. 3E (symbolized Manhattan metric), our results are better. The amount of improvement is significant because it oscillates between 0.2 − 0.4, and it is more closely related to the metric than to the version.

In Fig. 3, we show the differences between the average of good responses from thirty tests for SONN and SVM, whereas in Fig. 4, we present absolute accuracy for both methods on selected datasets and configuration of parameters. These results allow us to determine whether the result is the effect of SONN method improvement or deterioration of both methods in the entire range of parameters and for all datasets. Figures 4A and 4B (related to relatively simple Iris and Banknote Authentication datasets) indicate that, regardless of the version and metric selection, SVM gives better results. Considering the more numerous, more complicated datasets, we indicate that our method gives significantly better results than the classic SVM approach for the datasets Figs. 4D–4E. Also, we show that the average number of good answers among tests achieves the best results for the set Fig. 4E).

Choosing only one from the attributes subranges (d10) from all presented in Figs. 3 and 4, in Fig. 5 we show the dependence the improvement of the ratio among all datasets for the set of parameters: version and metric. Presenting the test results in this form, we also observe an improvement for the aforementioned sets (Figs. 5D and 5E).

One of our goals is to show that the method we created can be successfully used for various types of datasets. Even though we examine several datasets (among them were the simple, standard ones), we want to focus on the more complicated ones, namely the Combined Cycle Power Plant Dataset. Figure 6C, for CCPP shows the dependence the comparison of the average response from thirty tests for SONN and for SVM approach. The parameters for SONN are: the whole range of ratio, division (only d6, d10, d15), version and metric. In Fig. 4C we only observed the test results for ratio 20, now we have a broader look at the behavior of CCPP. In the case of dividing the range of parameters variability into six uniform subranges (Fig. 6C1), in the case of two sets of parameters ((v1, μ1), (v2, μ1)) we observe a decline in the correctness of our method. For the remaining Figs. 6C2 and 7C3, the approach we propose gives much better results.

Figure 7 gives us a look at how the CCPP behaves in the case of different partitions of the parameters into uniform subranges. In more detail, it shows the dependence the improvement of the division for the set of parameters: ratio, version and metric. Comparing all Figs. 7C1–7C4 at the same time, it seems that for division d7 we get the best results, while the division d5 gives the worst effectiveness. However, on closer inspection, we can also indicate that the selection of the μ2 metric gives a satisfactory result for more than one subrange division.