Generative adversarial network based adaptive data augmentation for handwritten Arabic text recognition

Adaptive data augmentation algorithm

Poor predictive performance can result from imbalanced data, especially for minority classes. Data augmentation is frequently used in a variety of deep learning approaches when only a small number of training samples are available. The original dataset’s class distribution is unaffected by rotation, position shifting, zooming, shearing, and other augmentation techniques. That is, if we have skewed data, it will remain skewed even after data augmentation. Any language or dataset can be affected by the problem of class imbalance based on differences in character frequencies. Observations like this are common when it comes to natural language texts (Wikipedia, 2021; Intellaren, 2021).

The adaptive data augmentation algorithm (Eltay, Zidouri & Ahmad, 2020) promotes more class-balance. This algorithm assigns a numerical weight to each word in the lexicon. The weight is calculated by taking the average probability of each class into account. Words containing less frequent characters will be augmented more than words containing characters appearing more frequently in texts. Table 2 shows the pseudo-code for our data-augmentation technique.

Generative adversarial networks (GANs)

Using successful generative modeling, a more domain-specific alternative to data augmentation can be found. Even though this is not well understood, data augmentation is a simpler version of generative modeling. Generative modeling offers a way to increase the number of training samples in challenging domains or domains with limited data. Deep reinforcement learning, for example, has seen a lot of success with GANs for Latin script (Arjovsky & Bottou, 2017).

When we generate new examples from an existing data set, we’re using generative modeling, which is an unsupervised machine learning task in which regularities or patterns in the input data are automatically discovered. When it comes to developing generational models, GANs shows promising results. They recognize the problem as one requiring supervised learning using two models: the generator and the discriminator. The generator model learns to generate new examples, while the discriminator model determines whether or not the new examples are real or fake (generated). GANs are an exciting and rapidly evolving field that fulfills the promise of generative models to provide proper examples in a wide range of problem areas, particularly image-to-image translation tasks (Goodfellow et al., 2020).

The GAN architecture was first described by Goodfellow et al. (2014). A basic GAN consists of two networks: a generator (G) and a discriminator(D). The G network generates data with a structure identical to training data while the D network attempts to classify the observations as “real” or “generated”. Each time a new sample is generated, the generator sends a new batch to the discriminator, together with actual examples from the domain. The discriminator is updated in the subsequent round to improve its ability to distinguish between real and fake samples, and the generator is updated according to the degree to which the generated samples deceived the discriminator. Figure 2 illustrates an example of a model architecture for a Generative Adversarial Network (GAN). The generator makes an attempt to convince the classifier that the samples it generates are authentic. Once the generator reaches convergence, the samples generated by it become indistinguishable from real data. Table 3 summarizes the most significant findings from the two GAN sub-networks.

Recently, Jha & Cecotti (2020) have presented a GAN architecture based on Convolutional Neural Network (CNN). This network is very similar to the one presented by Goodfellow et al. (2014) with a modification to the Generator G and Discriminator D networks to generate images of handwritten words particularly. The architecture of the generator and discriminator was rearranged to provide a significant increase in accuracy. It takes as an input image of size N_x × N_y × N_z, where N_x × N_y represents the dimensions of the image, and N_z represents the number of channels, i.e., the color space. In our case here N_z = 1 since we are dealing with grayscale images. At the output layer of the generator the sigmoid function was used and the loss function was chosen to be binary cross-entropy. The discriminator layers in Jha & Cecotti (2020) was arranged as follow:

• Layer.01: Input Layer (image).

• Layer.02: Hidden layer 1: Convolution Layer.

• Layer.03: Hidden layer 2: Convolution Layer.

• Layer.04: Hidden layer 3: Fully Connected Layer.

• Layer.05: Hidden layer 4: Convolution Layer.

• Layer.06: Output Layer: Fully Connected Layer.

The generator layers are based on a deconvolutional neural network with three deconvolutional layers along with interpolation and it is arranged as follow:

• Layer.01: Input Layer (Noise Vector Input).

• Layer.02: Hidden layer 1: Deconvolution Layer.

• Layer.03: Hidden layer 2: Deconvolution Layer.

• Layer.04: Hidden layer 3: Deconvolution Layer.

On the other hand, Alonso, Moysset & Messina (2019) proposed an adversarial architecture similar to Goodfellow et al. (2014), with the addition of bidirectional LSTM recurrent layers to encode the character sequence to be produced. In order to control the textual content of the generated images, they have also introduced an auxiliary network for text recognition. They obtained an embedding of the rendered word using bidirectional LSTM recurrent layers and then passed it to the generator network. The main idea was to embed each word into a 128-dimensional feature, which is then fed into a BigGAN (Brock, Donahue & Simonyan, 2018) network architecture. The textual content of the generated images was controlled using an auxiliary classifier (Odena, Olah & Shlens, 2017). The gradients stemming from R and D are balanced (to account for their different magnitude) and finally back-propagated to G. The full Architecture of the adopted GAN network in this paper can be described in Fig. 3. D, R, G, and φ are specifically trained to minimize the following objectives (Alonso, Moysset & Messina, 2019): (1)${\displaystyle L_D=-{\mathbb{E}}_{\left(x,s\right)\sim p_{\text{dat}}}\left[min\left(0,-1+D\left(\mathit{x}\right)\right)\right]-{\mathbb{E}}_{\mathbf{z}\sim p_{\mathit{z}}\mathit{s}\sim p_{\mathrm{w}}}\left[min\left(0,-1-D\left(G\left(\mathit{z},\phi \left(\mathit{s}\right)\right)\right)\right)\right]L_R=+{\mathbb{E}}_{\left(x,\mathit{s}\right)\sim p_{\text{dat}}}\left[\mathrm{CTC}\left(\mathit{s},R\left(\mathit{x}\right)\right)\right]L_{\left(G,\phi \right)}=-{\mathbb{E}}_{\mathbf{z}\sim p_z\mathit{s}\sim p_{\mathrm{w}}}\left[D\left(G\left(\mathit{z},\phi \left(\mathit{s}\right)\right)\right)\right]+{\mathbb{E}}_{\mathbf{z}\sim p_n,\mathit{s}\sim p_{\mathrm{u}}}\left[\mathrm{CTC}\left(s,R\left(G\left(\mathit{z},\phi \left(\mathit{s}\right)\right)\right)\right)\right]}$

The approach follows the GAN paradigm (Goodfellow et al., 2014), where the resulting image is evaluated by a text recognition network R in addition to the discriminator D. While D encourages the creation of realistic-looking handwriting, R encourages the creation of readable text that is true to the input text.

Rather than generating the image from an entire word representation, as in Goodfellow et al. (2014), Alonso, Moysset & Messina (2019), Jha & Cecotti (2020), each character is generated independently, utilizing CNN’s overlapping receptive field property to account for the influence of neighboring letters in ScrabbleGan which presented by Fogel et al. (2020). ScrabbleGAN builds on the architecture of the previous networks, but with enhancements to produce more realistic word images. This is because handwriting is a highly localized process in which each letter is influenced solely by the letters preceding and following it. This network follows the previous GAN paradigms, but with a change to the generator network, which is the method’s key technical novelty. As a result, G is a concatenation of identical class conditional generators (true/false classifiers). While the discriminator D encourages the use of realistic images, the recognizer R encourages the use of readable text by effectively distinguishing gibberish from real text. This architecture seeks to minimize the sum of the two networks’ loss terms l. Figure 4 shows the architecture overview for the case of generating the word . For each character, a filter f_∗ is selected from a filter-bank ς that is as large as the alphabet, for example ς = f_Alif, f_Baa, …, f_Yaa. Four-character filters are concatenated f₁, f₂, f₃, f₄ multiplied by the noise vector z and fed into the generator G. Both the discriminator (D) and the recognizer (R) use the resulting image, ensuring that the style and data fidelity are both upheld. (2)${\displaystyle l=l_D+\lambda .l_R}$

where, l_D and l_R are the loss term of D and R respectively. Also, it has been found by the authors Fogel et al. (2020) that better R does not lead to better general performance (generation). This is due to the LSTM layers in the RNN network learning an implicit language model and thus being able to identify the correct character even when it is not clearly written. As a result, the recurrent head is removed to force R to make decisions solely on visual features.

IFN/ENIT dataset

For handwritten Arabic text recognition research, the IFN/ENIT dataset is the most widely used and popular dataset published by Pechwitz et al. (2002). It contains over 27,000 handwritten Arabic names for Tunisian cities. The lexicon includes 937 place names. A Ground Truth (GT) file has been compiled for each word in the dataset. This file contains information about the word, such as its baseline position and the specific characters used. Figure 6 depicts image samples from the IFN/ENIT dataset.

AHDB dataset

The Arabic Handwritten dataset (AHDB), created by Al-Ma’adeed, Elliman & Higgins (2002), consists of approximately 10,081 handwritten Arabic words representing cheque numbers and quantities, as well as the most commonly used words in Arabic script The lexicon contains 96 words, 67 of which are handwritten words that directly relate to handwritten cheque numbers. The remaining 29 words were commonly used Arabic words. Figure 7 depicts several images from the AHDB dataset as an example.

Applying adaptive data augmentation algorithm

Using our adaptive data augmentation algorithm, Fig. 8 shows the frequency distribution of the IFN/ENIT characters compared to the standard algorithm. As we have seen, increasing the number of minorities helps us achieve class balance. We can see that there is a significant difference in the appearance of certain characters in the dataset prior to using our method. For example, the letter Alif is the most occurring class, compared to the other letters which are significantly few in numbers in the dataset.

The AHDB dataset, on the other hand, has about 10,000 handwritten Arabic words and 31,200 characters that can be utilized for the most frequent Arabic and handwritten cheques. Unlike the IFN/ENIT dataset, this one does not contain a significant number of Arabic characters because it was designed to identify the most often used Arabic words and terminology in cheque writing. In comparison to the original distribution of the characters, Fig. 9 shows the frequency distribution of the characters after our adaptive data augmentation approach is performed.

Handwritten arabic words synthesizer

We have used the GAN architecture for the adversarial generation of handwritten Arabic text images, which is forked from Fogel et al. (2020). The full implementation of this network is publicly available at GitHub (Krber, 2020). Each image was resized to a fixed height of 32 pixels while retaining the original image’s aspect ratio.

We have used the standard metrics for GAN performance evaluation, namely Fréchet Inception Distance (FID) (Heusel et al., 2017) and Geometry Score(GS) (Khrulkov & Oseledets, 2018). The FID calculates the distance between feature vectors calculated for real and generated images. The FID score is used to assess the quality of images generated by generative adversarial networks, and it has been demonstrated that lower scores are associated with higher-quality images. GS, on the other hand, is assessing the quality of the generated samples and detecting various levels of mode collapse.

We have trained the model shown in Fig. 5 twice on the IFN/ENIT and AHDB datasets, respectively. First, it was trained directly on the original datasets without using our adaptive augmentation algorithm. The datasets were then balanced in the second time using our adaptive data augmentation described previously, and the balance datasets were used to train the our word-synthesizer network.

The training loop starts with the generator receiving a set of random seeds and transcriptions from a random word list. For this part, we’ve chosen random words from the dataset and their transcriptions as input to produce a set of fake images. We changed the noise vector z that was fed into the network to generate different handwriting styles. Figure 10 shows some examples of synthesized words from the IFN/ENIT dataset generated in different handwriting styles. Every 10, 000 iterations, we computed the FID (with 10, 000 real and 10, 000 synthesized images) and the GS (with 5, 000 real and 5, 000 synthesized images, 100 repetitions, and default settings for the other parameters). Then we have selected the best FID and best GS among the various runs independently. We have to depend on visual inspection to review the textual content. We obtain the best FID of 38.79 and GS of 18.87 × 10⁻³ , while the synthesized image is both readable and realistic. Next, we have retrained the same network but this time on the AHDB dataset after it was balanced using our adaptive data augmentation algorithm. Figure 11 shows some examples of synthesized words from the AHDB dataset generated in different handwriting styles.

As before, we computed the FID (using 5,000 real and 5,000 synthesized images) and GS (using 2,000 real and 2,000 synthesized images, 100 repetitions, and default settings for the other parameters) every 2,500 iterations. Then, independently of one another, we determined the best FID and GS for each run. We have to depend on visual inspection to review the textual content. The best FID and GS are 27.70 and 11.40 × 10⁻³, respectively, with a readable and realistic image generated. Compared to the IFN/ENIT, the visual quality of the synthesized images is much better which are expected since the AHDB dataset is much simpler in term of the word content than the IFN/ENIT dataset.

Handwritten arabic word recognition

In this section, the value of the data generated for handwritten text recognition will be evaluated. The Recurrent Neural Network Library (RNNLIB) (Graves, 2016) was utilized in the development of the experimental models. There are many different deep learning architectures included in this library, including: (LSTM, BiLSTM, and RNN). Connectionist Temporal Classification (CTC) (Graves et al., 2006) is also included, allowing the system to transcribing sequence data that has not been segmented.

In each of the forward and backward layers, our network contains 200 extended LSTM memory blocks. There is one memory cell in each memory block, as well as an input gate, an output gate, a forget gate, and three peepholes. tanh is used for the cell input and output activation functions, while logistic sigmoid is used for the activation of the gate. There has been a decaying learning rate used in all CTC systems. Initially, the rate of learning was set at 0.001 and gradually decreased to 0.0001.