A high-performance, hardware-based deep learning system for disease diagnosis

Basic ANN operation

Network inputs

The input values are first standardized in order to make them zero-centered. The process of standardization follows Eq. (2). In Eq. (1), X represents the input vector, µrepresents the average, and σ represents the standard deviation of data samples. The process of standardization is visually represented in Fig. 2. It is important to note that standardization is sometimes referred to as ‘normalization’ in literature, though normalization is, in reality, different from standardization.

(1)${\displaystyle X_{\text{standard}}=\frac{X_{\text{original}}-\mu }{\sigma }.}$

Proposed network topology and actuators

In this work, we have chosen the Sqish activation function for the hidden layer, and square logistic sigmoid (LogSQNL) for the output layer. The Sqish function is morphologically similar to swish, and LogSQNL is similar in behavior to the traditional sigmoid. The beauty of these functions is that both of these can be implemented in a single cycle using arithmetic and logic units (ALUs) only. As shown in Fig. 3, the proposed system can take 30 (or less) input features, has five hidden neurons, and two outputs.

The RTL schematic diagrams of the complete system and the predictor are shown in Figs. 4 and 5, respectively. The rectangular yellow and red boxes shown in this figure represent storage elements, and white rectangular/square boxes represent computational blocks. As shown in Fig. 5, the activation values corresponding to the two output neurons are given as input to the predictor. The predictor then calculates the maximum value and the class corresponding to this maximum value is the predicted class. The top level diagram of the proposed system is shown in Fig. 6. In Fig. 6, I_M represents Mth input. The multiplier units (MUs) are responsible for multiplying weights with incoming inputs and the accumulator is responsible for adding these products. The output of the multiplier-accumulator (MAC) unit is sent to the appropriate actuator for neuronal activation.

Mathematical setup

Now we provide details of the complete mathematical setup. Here, weights are denoted by W_i and inputs are denoted by X_i. Biases are represented by b_j, and the weighted sum is represented by Z_j. Finally, activation values are represented by A_j. Here, the subscript j represents the postsynaptic neuron and the subscript i represents the presynaptic neuron. The following equations represent the complete mathematical process. ${\displaystyle Z_1=\sum _i\left(W_i\cdot X_i\right)+b_1.}$

Since Sqish is used as the activation function for Layer 1 in the proposed scheme, A₁(Z₁) is given by the following equation: ${\displaystyle A_1=\left\{\begin{array}{cc}{Z}_1+\frac{Z_1^2}{32}\hfill & Z_1\ge 0\hfill \\ \hfill & \hfill \\ Z_1+\frac{Z_1^2}{2}\hfill & -2\le Z_1<0\hfill \\ \hfill & \hfill \\ 0\hfill & Z_1<-2.\hfill \end{array}\right.}$

The Layer 1 activation vector is then passed as input to Layer 2 in order to obtain the weighted sum Z₂, as shown in the following equation. ${\displaystyle Z_2=\sum _i\left(W_i\cdot A_1\right)+b_2.}$

The LogSQNL neurons in the output layer are activated according to the following rule: ${\displaystyle A_2=\left\{\begin{array}{cc}1\hfill & Z_2>2\hfill \\ \hfill & \hfill \\ \left(Z_2-\frac{Z_2^2}{4}\right)\frac{1}{2}+\frac{1}{2}\hfill & 0\le Z_2\le 2\hfill \\ \hfill & \hfill \\ \left(Z_2+\frac{Z_2^2}{4}\right)\frac{1}{2}+\frac{1}{2}\hfill & -2\le Z_2<0\hfill \\ \hfill & \hfill \\ 0\hfill & Z_2<-2.\hfill \end{array}\right.}$

The derivative of LogSQNL function is given by Eq. (2) and the dependence of the loss function on Layer 2 weight vector and bias vector is given by Eqs. (3) and (4). The derivative of the Sqish function is given by Eq. (5) and the dependence of the loss function on Layer 1 weight vector and bias vector is given by Eqs. (6) and (7). (2)${\displaystyle \frac{\partial A_2}{\partial Z_2}=\left\{\begin{array}{cc}\frac{2-Z_2}{4}\hfill & 0\le Z_2\ge 2\hfill \\ \hfill & \hfill \\ \frac{2+Z_2}{4}\hfill & -2\le Z_2<0\hfill \\ \hfill & \hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}\right.}$ (3)${\displaystyle \frac{\partial \mathit{L}}{\partial {\mathit{W}}_{\mathit{2}}}=\left\{\begin{array}{cc}\left(A_2-y\right)\cdot A_1\cdot \frac{2-Z_2}{4}\hfill & 0\le Z_2\ge 2\hfill \\ \hfill & \hfill \\ \left(A_2-y\right)\cdot A_1\cdot \frac{2+Z_2}{4}\hfill & -2\le Z_2<0\hfill \\ \hfill & \hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}\right.}$ (4)${\displaystyle \frac{\partial \mathit{L}}{\partial {\mathit{b}}_{\mathit{2}}}=\left\{\begin{array}{cc}\left(A_2-y\right)\cdot \frac{2-x}{4}\hfill & 0\le Z_2\ge 2\hfill \\ \hfill & \hfill \\ \left(A_2-y\right)\cdot \frac{2+x}{4}\hfill & -2\le Z_2<0\hfill \\ \hfill & \hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}\right.}$ (5)${\displaystyle \frac{\partial A_1}{\partial Z_1}=\left\{\begin{array}{cc}1+\frac{Z_1}{16}\hfill & Z_1\ge 0\hfill \\ \hfill & \hfill \\ 1+Z_1\hfill & -2\le Z_2<0\hfill \\ \hfill & \hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}\right.}$ (6)${\displaystyle \frac{\partial \mathit{L}}{\partial {\mathit{W}}_{\mathit{1}}}=\left\{\begin{array}{cc}\frac{\left(A_2-y\right)\cdot A_1\cdot \left(2-Z_2\right)\cdot W_2\cdot \left(16+Z_1\right)\cdot X_1}{64}\hfill & 0\le Z_1\le 2;0\le Z_2\le 2\hfill \\ \hfill & \hfill \\ \hfill \\ \frac{\left(A_2-y\right)\cdot A_1\cdot \left(2+Z_2\right)\cdot W_2\cdot \left(1+Z_1\right)\cdot X_1}{4}\hfill & -2\le Z_1<0;-2\le Z_2<0\hfill \\ \hfill & \hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}\right.}$ (7)${\displaystyle \frac{\partial \mathit{L}}{\partial {\mathit{b}}_{\mathit{1}}}=\left\{\begin{array}{cc}\frac{\left(A_2-y\right)\cdot A_1\cdot \left(2-Z_2\right)\cdot W_2\cdot \left(16+Z_1\right)}{64}\hfill & 0\le Z_1\le 2;0\le Z_2\le 2\hfill \\ \hfill & \hfill \\ \frac{\left(A_2-y\right)\cdot A_1\cdot \left(2+Z_2\right)\cdot W_2\cdot \left(1+Z_1\right)}{4}\hfill & -2\le Z_1<0;-2\le Z_2<0\hfill \\ \hfill & \hfill \\ 0\hfill & \text{otherwise}.\hfill \end{array}\right.}$

Proposed hardware system

As mentioned before, there are two layers in the proposed system: Layer 1 and Layer 2. Both these layers have their memories to store weights. It is pertinent to mention that all the weights are stored in the chip. The total number of weights in the system is 160 since there are 160 synapses. The on-chip weight memory consumes 1.1 kilobits. The required weights that are fetched from the corresponding memory are then multiplied by the respective layer inputs using the multiplier units (MUs) shown in Fig. 6. The multiplication is carried out using the built-in DSP48 elements. The resulting products are then summed up using an accumulator that contains an array of adders. This process is carried out for all the neurons in a layer. In the end, N weighted sums are obtained, where N represents the number of neurons in a layer. These N weighted sums are passed to their respective actuators for neuronal activation. The actuator then passes these calculated values onto the next layer and the same process is repeated. The complete structure of the proposed hardware system is shown in Fig. 6.

As mentioned earlier, Sqish neurons are used in Layer 1 and LogSQNL neurons are used in Layer 2. The structure of a Sqish neuron is shown in Fig. 7 and that of a LogSQNL neuron is shown in Fig. 8. A Sqish neuron can be implemented using two multiplexers (MUXes), two adders, two shifters and two multipliers. A LogSQNL neuron, on the other hand, consumes more resources than Sqish. A LogSQNL neuron can be implemented in hardware using three MUXes, four shifters, four adders, and two multipliers. However, since there are only two outputs in the proposed system for binary classification (one-hot encoding), the implementation of LogSQNL neurons is not a big deal.

Interestingly, the distribution of data in the dataset under consideration, i.e., Wisconsin Breast Cancer (WBC) (University of California, 2022) is highly non-uniform. The standard deviation of the data is extremely large. The final input values obtained after standardization require 11 bits, where four bits are reserved for the integer part and seven bits are reserved for the mantissa (fractional part). As per our observations and calculations, the classification accuracy is more sensitive to the fractional part than the integer part. Therefore, more bits are reserved for the mantissa.

Test conditions

Since the system has been developed for cancer diagnosis, the dataset used for experimentation is Wisconsin Breast Cancer (WBC) (University of California, 2022). This dataset has 30 features, 569 samples and two classes (benign and malignant). As per rules, about 80% samples have been used for training and 20% have been used for evaluation. To achieve class balance, some samples are picked up from the ‘benign’ class and others are picked up from ‘malignant’ class. We use Python for evaluation of the proposed scheme. The hardware is described in Verilog language at the register transfer level (RTL). The learning rate is kept equal to $\frac{1}{3}$. The momentum is equal to 0.9. The data is processed in batches to achieve high accuracy; the batch size used in the proposed system is 100. The network is trained for 4300 epochs. All these hyperparameter values have been found through empirical tuning using the so-called ‘grid search’ (Zheng, 2015). The specifications of the platform on which all the tests are carried out are given in Table 3.

Performance metrics

The metrics used for the evaluation of the proposed scheme are classification accuracy, precision, recall, hardware implementation cost, and system throughput. The classification accuracy is simply defined as the number of correctly-classified samples out of the total number of samples. In the context of disease diagnosis, accuracy is not a good measure of system performance. Therefore, we use precision and recall in order to properly quantify performance. The precision and recall are defined in Eqs. (8) and (9) respectively. Since these metrics are very common, we believe there is no need to discuss them in detail here. In Eqs. (8) and (9), TP stands for ‘true positive’, TN stands for ‘true negative’, FN stands for ‘false negative’, and FP stands for ‘false positive’. (8)${\displaystyle \mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$ (9)${\displaystyle \mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}.}$

To evaluate hardware efficiency, we use two metrics: the number of resources (number of slice registers, number of slice look-up tables, number of block memories, and DSP elements) consumed by the system, and system throughput. The throughput is defined in two ways: the number of multiply-and-accumulate (MAC) operations that can be performed in a second, and the number of input samples that can be processed by the system in a second.

Classification accuracy, precision, and recall

As per obtained results, the system can predict the type of cancer with 98.23% accuracy. Moreover, the average precision of the proposed system is 97.5% and recall is around 98.5%. The classification report and the confusion matrix for the proposed system are given in Tables 4 and 5, respectively. Moreover, the proposed system is compared with many other state-of-the-art systems in terms of classification accuracy in Table 6. The classification accuracy as a function of epochs is presented in Fig. 9A, and the confusion matrix is visually shown in Fig. 9B.

Implementation cost and throughput comparisons

The use of Sqish and Log_SQNL (Wuraola, Patel & Nguang, 2021) allows the processing of one sample in one clock cycle. In a single cycle, the system can perform all MAC operations and can activate all the neurons without using any divider or storage element. The multiplication operations can be performed using DSP48 multipliers that are abundantly available in an FPGA.

There are 160 synapses in the proposed neural system that operates at 63.487 MHz. The number of synaptic multiplications and additions to be performed are 160 and 153 respectively. Therefore, the system can perform 20 giga-operations in a second (GOPS). Since the system can classify one cancerous sample in one cycle (≈15.75 ns), the system can classify about 63.5 × 10⁶ (63.5 million) samples in a second. Since a sample contains 30 inputs, about 1.91 × 10⁹ 1-input samples can be classified by the proposed system in one second. The system is compared with other state-of-the-art systems in terms of implementation cost and throughput in Tables 7 and 8 respectively.