VIOLA jones algorithm with capsule graph network for deepfake detection

Removal of noise

Removing noise in the input forged image accurately detects the fake image. In this work, the Gabor filter is used. A two-dimensional Gabor filter is used, which modulates the Gaussian kernel function by using a sinusoidal plane wave in the spatial domain. The two-dimensional Gabor function is defined as: (1)${\displaystyle ga\left(a,b;\lambda ,\theta ,\psi ,\sigma ,\gamma \right)=exp\left(-\frac{{a^{\prime }}^2+{\gamma }^2{b^{\prime }}^2}{2{\sigma }^2}\right)exp\left(i\left(2\pi \frac{a^{\prime }}{\lambda }+\psi \right)\right).}$ The real part of the Gabor function is the following: (2)${\displaystyle ga\left(a,b;\lambda ,\theta ,\psi ,\sigma ,\gamma \right)=exp\left(-\frac{{a^{\prime }}^2+{\gamma }^2{b^{\prime }}^2}{2{\sigma }^2}\right)cos\left(2\pi \frac{a^{\prime }}{\lambda }+\psi \right).}$ The imaginary part of the Gabor function is: (3)${\displaystyle ga\left(a,b;\lambda ,\theta ,\psi ,\sigma ,\gamma \right)=exp\left(-\frac{{a^{\prime }}^2+{\gamma }^2{b^{\prime }}^2}{2{\sigma }^2}\right)sin\left(2\pi \frac{a^{\prime }}{\lambda }+\psi \right).}$ Here, (4)${\displaystyle a^{\prime }=acos\theta +bsin\theta }$ and (5)${\displaystyle b^{\prime }=-asin\theta +bcos\theta ,}$ where λ is a sinusoidal wavelength factor. In general, λ does not exceed 2, and the orientation of normal to parallel stripes in the Gabor function is defined as θ. The phase shift ψ represents the maximum offset in signal modulation, and the aspect ratio γ specifies the ellipticity of the Gabor function. σ is the standard deviation of the Gaussian envelope.

Data augmentation

This technique is used to increase the amount of data and generate new data points to reduce the overfitting problem. During the training process of the forged input image dataset, the augmentation processes are implemented as follows:

Zooming augmentation: The input image is clearly viewed using Zooming augmentation. View or monitor the input image in the zoom effect as larger with the value 0.4 in the range [0.4, 0.8]. Parameter values vary from 1 −value to 1 +value.

Horizontal flipping: Viewing the zoomed image horizontally flipped when its Boolean value is set with ‘true’,

Random rotation of almost about 30°.

Random contrast, brightness.

Coarse dropout with size 0.02.

Shear mapping

Each input image is converted to a vertical direction from the edge part. The parametric control values of the angle of deviation of the horizontal line and the displacement rate. The values of the shear range are 0.2.

A shear mapping is a linear transformation that distorts the shape of an object by skewing it along one or more axes. This transformation is accomplished by multiplying the coordinates of each point in the object by a matrix that includes non-zero off-diagonal entries. The amount of shear is determined by the values of these entries, which control the degree of displacement along each axis.

A shear mapping can be represented by a matrix equation. Suppose we have a 2D point with coordinates (x, y). To apply a shear mapping along the x-axis by a factor of k, we can use the following matrix equation: $\left[\begin{array}{c}\hfill x^{\prime }\hfill \\ \hfill y^{\prime }\hfill \\ \hfill \hfill \end{array}\right]$= $\left[\begin{array}{cc}\hfill 1\hfill & \hfill 0\hfill \\ \hfill k\hfill & \hfill 1\hfill \\ \hfill \hfill \end{array}\right]$.

Here, (x’, y’) are the coordinates of the transformed point, and the matrix on the right-hand side represents the shear mapping. To apply a shear mapping along the y-axis, we would use a similar matrix with k in the (1, 0) position.

Note that these equations assume that the origin of the coordinate system is fixed. If we want to apply a shear mapping with respect to a different point, we would need to first translate the coordinates so that the origin is at that point, apply the shear mapping, and then translate the coordinates back to their original position.

Rescaling image

The input image consists of RGB values from 0 to 255. The values are rescaled to the interval [0, 1] using the 1/255 scaling method to feed them into the proposed model.

The VIOLA-Jones algorithm

A. VIOLA-Jones algorithm is used to detect the face image. This algorithm has four modules; feature selection using the Haar function, integral function, training based on the Adaboost algorithm, and classifier of cascade operator.

A. Selection of Haar features

The selection of Haar features is based on the similarity properties of the face image. Some common similarities in face image properties are:

• The size and position of the mouth, eyes, and nose.

• Values of the intensity of pixels.

Figure 4 shows various types of features used in the Haar function.

The intensity value is calculated by the summation of pixel values in the colored area minus the summation of pixel values in the white area and it is defined as. (7)${\displaystyle intensityvalue=\sum \left(Pixelsincoloredarea\right)-\sum \left(PixelsinWhitearea\right).}$ In VIOLA-Jones, two-rectangular features are used in the detection of the face image. The differences are calculated between colored and white rectangles of the specific area (Lugstein et al., 2021; Zheng et al., 2021a; Yu et al., 2019). The image is to be divided as a sub-window and the features are related on a random basis to the location. Figure 4 shows the extraction of various features from the datasets.

B. Integral function

The input image is converted into an integral image by evaluating that every pixel value is equivalent to the cumulative sum of all pixels above and left to that particular input pixel. It can be calculated over input image by sub-window region. The integral of the image is defined below. (8)${\displaystyle p\left(x,y\right)=img\left(x,y\right)+p\left(x-1,y\right)+p\left(x,y-1\right)-p\left(x-1,y-1\right)}$ Figure 5 shows the summation of the particular area table in which Eqn. Eq. (8) is calculated.

Figure 6 shows the sample of the cumulative sum of the particular pixel value of the image.

C. AdaBoost training

E. It uses the image’s window size and computes the input image’s features. The image’s most relevant characteristics are selected from the collected features of the image. The features of the image are repeatedly executed for every sub-window size. Thus, it contains a vast set of features. By applying the Adaboost algorithm, it discards the irrelevant features from the featured dataset. These selected features are arranged in weighted order and classified with similar face image pixels. Therefore, these features are called weak classifiers (Chugh et al., 2020). In addition to this, the AdaBoost algorithm is implemented based on its threshold value. That is, it is based on the concept of the brute force method through which the trained weak classifier determines the best performer of the feature.

D. Classifier of the cascade operator

During the training phase using shear mapping, the cascaded classifier collects the robust classifier by verifying the particular sub-window area that is belonged to the real or the forged image. It is discarded if the sub-window region is classified as a forged image. It is passed on to the next phase in the cascade if it is an actual image. The process is repeated and will be cascaded to get a higher chance of classifying the face image as real or fake. The algorithmic procedure of Viol Jones is given below:

The algorithm divides the input forged image into smaller sub-windows. It looks for every feature specifically at all sub-windows and finds the face features. This algorithm checks the image in different positions and scales to identify many faces of different sizes. By implementing this Algorithm 1, the whole image features are scanned using a sub-window, and the face image is detected as real or forged using the capsule graph algorithm. To classify it accurately, the capsule graph network model is implemented. Because of using the V-J algorithm, the face image is detected and for fine-tuning the classification of a real or fake image, it needs to implement a capsule graph network model.

Capsule Graph Network (CGN)

The dual Capsule Graph Network consists of three capsules and for detecting the fake or the real image, two capsules are needed. At first, features are extracted using the k-means algorithm. Furthermore, these features are stored as a file and it endures for further detection in VIOLA-Jones Algorithm. This VIOLA Jones algorithm detects it as a face image and these feature values are fed as input to this CGN model. In the graph, each node represents the features of the face image or video. The dual graph (DGN) network consists of two CNN and the input data is represented as matrix Id ∈ ℝ^m×n with the data points $Dp=\left\{{dp}_1,{dp}_2,\dots ,{dp}_i,{dp}_{i+1},\dots {dp}_n\right\}$. The structure of the graph is given by the adjacency matrix A ∈ ℝ^n×n. Vector features and adjacency matrix A are the DGN model inputs. The output of local consistency for hidden layer i is declared in Eq. (6). (9)${\displaystyle {convo}_{LC}^i\left(P\right)=z^i=\sigma \left(P^{-\frac{1}{2}}{\overline{A}P}^{-\frac{1}{2}}{z}^{i-1}{r}^i\right)}$ where $\overline{A}=A+I_n$ is the self-loop adjacency matrix of the matrix A, I_n is the identity matrix, P is the normalized adjacency matrix, z represents the output of the (i − 1)th layer, r represents parametric value for the training, and σ is the activation function (ReLU). Figure 7 shows the workflow of the Capsule Graph Network.

Figure 7 contains three main capsules and produces two output capsules in terms of real fake images or videos. Features extracted from the k-means algorithm are fed as input to face detection using the VIOLA-Jones algorithm and its output is passed as input to the DGN model. Therefore, it is distributed into three main capsules. All three main capsules include statistical pooling which is mainly used for detecting the forgery image or videos. The outputs of three main capsules are dynamically routed to output capsules. Thus, the output capsules detect the image or video as real or fake. Algorithm 2 describes the capsule dual graph. The notations used in the Algorithm 2 are given in Table 2.

In Algorithm 2, the dynamic routing of three main output capsules out_j/i for it iterations are evaluated. To improve the efficiency of the algorithm, Gaussian noise is slightly added to the 3-D weight value of tensor Wet, and the squash is done in the following Eq. (10) before the process of routing through all iterations. (10)${\displaystyle squash\left(x\right)=\frac{{∥x∥}^2}{1+{∥x∥}^2}\cdot \frac{x}{∥x∥}.}$

Thus, the added Gaussian noise helps to reduce the over-fitting of the graph. The outputs of the main capsules are calculated as: (11)${\displaystyle L=-\left(xlog\left(\hat{x}\right)+\left(1-x\right)log\left(1-\hat{x}\right)\right)}$ where, x is the ground truth value of the label and $\hat{x}$ is the predicted value of the label calculated using Eq. (9). The output capsule o_j by using: (12)${\displaystyle \hat{x}=\frac{1}{m}\sum _{i}softmax\left(\left[\genfrac{}{}{0ex}{}{o_1^T}{o_2^T}\right]\right).}$ By using Eq. (12), the length of the output capsules separates the two output capsules, for example, detection of real image and fake image or video for all dimensions.