Deep fake detection using a sparse auto encoder with a graph capsule dual graph CNN

Preprocessing

The preprocessing of fake videos/images is shown in Fig. 2. Initially, the input fake videos are split into frames. Using multitask cascaded convolutional neural networks (MTCNNs) (Zhang et al., 2016), the face is detected and cropped from video frames. MTCNN is a Python face detection module with an accuracy of 95%. It can extract the face that focuses on computer vision transformation. This extracted face is pre-processed using the sequence of operations listed below to enhance image quality. The feature vector was generated by extracting the computer vision features from the preprocessed image using the graph LSTM method.

Feature extraction using the sparse autoencoder (SAE) with graph LSTM

The pre-processed image is fed into this section to extract the computer vision features using the proposed sparse autoencoder-based LSTM model. The traditional auto-encoder method has some problems, such as its inability to find features by copying memory into an implicit layer (Olshausen & Field, 1996). This problem is resolved with a sparsity approach with an autoencoder called a sparse autoencoder (SAE) (Hoq, Uddin & Park, 2021). It is an unsupervised deep learning method with a single hidden layer (Kang et al., 2017) used to encode the data for feature extraction. This will extract the most relevant features from the expressions of the hidden layer and estimate the error (Leng & Jiang, 2016). The regularization equation for the sparsity is defined in Eq. (1).

(1) $$Sparsity\mathrm{\_}regrsn=\sum _{i=1}^{u_2}KL(\vartheta ||{\hat{\vartheta }}_i),$$where the divergence implemented is the Kullback-Leibler divergence (KL), ${\hat{\vartheta }}_i$ is the activation function of the hidden node i^th, ϑ is the sparsity parameter. The KL-divergence is mathematically defined by Eq. (2)

(2) $$KL(\vartheta ||{\hat{\vartheta }}_i)=\vartheta \log {\displaystyle \frac{\vartheta }{{\hat{\vartheta }}_i}+\left(1-\vartheta \right)\log {\displaystyle \frac{1-\vartheta }{1-{\hat{\vartheta }}_i}.}}$$

The sparse AE is trained with the cost function declared in Eq. (3) which consists of the mean square error defined in Eq. (4). This will reconstruct the input vector X into the output vector $\hat{X}$ throughout the entire training dataset (Ng, 2011). The LASSO regression term is declared in Eq. (3) and the final term of the sparsity transformation is defined in Eq. (1). The importance of LASSO regression in SAE is to extract the most relevant features by assigning the feature coefficients as zero for features that are of little relevance, which will reduce the space of the parameter.

(3) $$Cost\mathrm{\_}function=MSE+\alpha \cdot Lasso\mathrm{\_}regrsn+\beta \cdot Sparsity\mathrm{\_}regrsn,$$

(4) $$MSE={\displaystyle \frac{1}{N}\sum _{i=1}^N(X_i-{\hat{X}}_i{)}^2,}$$

(5) $$Lasso\mathrm{\_}regrsn=\sum _{l=1}^{n_l-1}\sum _{i=1}^{u_l}\sum _{j=1}^{u_l+1}|w_{ij}^l|,$$where N is the total number of input data points, X represents the input vector, $\hat{X}$ represents the reconstructed output vector, α is the Lasso regression coefficient and β is the sparsity regularization coefficient, l indicates the l^th layer, n_l is the number of layers, u_l is the number of units in the layer l and w^l_ij is the weight value between the i^th node of the layer l and the j^th node of the layer l + 1. LASSO regression can add the magnitude of the absolute value as a penalty term. To improve the feature extraction process, this penalty coefficient is set to zero. The LSTM trains SAE to improve the feature extraction process. An LSTM is a backpropagation method for training the feature extraction model with three gates: input, forget, and output gates. The input gate is responsible for modifying the memory with the values decided by the sigmoid activation function. The forget gate is used to discard features from the previous state. The output gate is used to control the output features. Traditional LSTM is enhanced with a graph structure in which each node of the graph is represented as a single LSTM unit with forward and backward directions. In one direction, the node history is constructed and in another direction the response is characterized. Figure 3 shows the proposed SAE-GLSTM model to extract deepfake image features.

An SAE consists of an encoder, a compression/parameter space, and a decoder. The encoder encodes the given input to the parameter space through hidden layers, and the decoder is responsible for decoding the parameter space data to the output layer. Due to the autoencoder nature of dealing with negative values, a rectified linear unit (ReLU) is not suitable, and a sigmoid function is used as an activation function, which will reduce the training ability of the network. SAE can reduce the error between the input and the reconstructed data. The usage of hidden layers is reduced by the sparsity constraint. The regularization used here will avoid the overfitting issue and can be applied to larger datasets. The selected features are then passed on to the LSTM cell to enhance the feature selection process by extracting the relevant features. The LSTM graph model comprises six layers: the input layer, four hidden layers, and the output layer. The input layer of the LSTM graph consists of the features extracted from SAE. Each feature is represented as a neuron in the LSTM graph cell input layer graph, as shown in Fig. 4.

For the number of iterations t, SAE-GLSTM computes the hidden forward sequence as $\overleftarrow{s}$ and the hidden backward sequence as $\overrightarrow{s}$, and the output sequence is represented as Y. The input x_t has a hierarchical timing layer and the transition of the node state is declared a vector using the standard LSTM (Zhou, Xiang & Huang, 2020). Figure 5 shows the hierarchical forward and backward timing structure of Graph LSTM.

Let p(t) be considered the parent node of t and k(t) be the first child node. This hierarchical timing structure has predecessor p(t) and successor s(t) representing the forward and backward siblings, respectively. If there is no child, the values of k(t), p(t), and s(t) are set to null. For the forward GLSTM, the parameters, such as the input gate ig, temporal forget gate fg, hierarchical forget gate hg, cell c, and the output op, are updated and represented in Eqs. (6) to (10) with the vectors ig_t indicating the new information weight, fg_t indicating the memory data of siblings, hg_t indicating the memory data of the parents, and σ representing the sigmoid function. For the backward GLSTM, $\mu \left(t\right)$ is replaced by k(t), p(t) is replaced by s(t) and is represented in Eqs. (11) to (15) listed in Table 2.

The extracted features from SAE are enhanced with the LSTM graph by sending the SAE output as input to the LSTM unit. The optimal relevant feature subset has been selected as an output of this graph LSTM network. The extracted feature subset has numerical values of specific images in matrix form.

Deepfake detection using capsule dual graph CNN

The proposed detection system consists of three capsules. Two capsules are allotted for output to indicate fake and real images. CNN dual graph is represented in one capsule to perform detection. The features extracted from “Feature Extraction using the Sparse Autoencoder (SAE) with Graph LSTM” are given as input to this detection model. The dual graph neural network is the variance of the traditional neural network with a graph (Scarselli et al., 2008). Each node in the graph is a feature. A dual graph CNN consists of two CNNs and the input set of data points $\text{𝒳}=\left\{x_1,x_2,\dots ,x_l,x_{l+1},\dots ,x_n\right\}$, the set of labels $\text{𝒞}=\{1,2,\dots ,c\}$, the first l points have labels $\{y_1,y_2,\dots ,y_l\}\in \text{𝒞}$, and a graph structure. We assume that each point has at most k features; therefore, we denote the data set as a matrix $X\in {\mathbb{R}}^{n\times k}$ and represent the structure of the graph by the adjacency matrix $A\in {\mathbb{R}}^{n\times n}$. Using the input X, labels L, and A, our model aims to predict the labels of the unlabeled points.

The model is constructed with local consistency (LC) and is a type of feed-forward network that incorporates global consistency (GC) and a regularizer for the ensemble. The feature vector FV and the adjacency matrix A are the inputs of the DGCNN model. The local consistency output for the hidden layer i of the network Z⁽ⁱ⁾ is declared in Eq. (16) (Kipf & Welling, 2017)

(16) $$con{v}_{LC}^{(i)}\left(X\right)=Z^{(i)}=\sigma \left({\overline{D}}^{-\frac{1}{2}}\overline{A}{\overline{D}}^{-\frac{1}{2}}{Z}^{(i-1)}{W}^{(i)}\right),$$where $\overline{A}=A+I_n$ is the adjacency matrix A with self-loops, I_n is the identity matrix, and ${\overline{D}}_{i,i}={\sum }_j{\overline{A}}_{i,j}$. Therefore, ${\overline{D}}^{-\frac{1}{2}}\overline{A}{\overline{D}}^{-\frac{1}{2}}$ is the normalized adjacency matrix, Z⁽ⁱ⁻¹⁾ represents the output of the (i − 1)^th layer, Z⁽⁰⁾ = X W⁽ⁱ⁾ represents trainable parameters of the network, and σ indicates the activation function (ReLU). The output of the DGCNN can be visualized on the Karate club network, as shown in Fig. 6. The red color of this network indicates a labeled node and the green color indicates an unlabeled node. The local consistency network is optimized with PPMI (positive point-wise information) in the global consistency layer.

Global consistency is formed with PPMI to encode semantic information and is denoted as a matrix $P\in {\mathbb{R}}^{n\times n}$. Initially, the frequency matrix FM is calculated by random walk and on the basis of FM we calculate the matrix P. Furthermore, we define the convolution function Conv_GP based on P. We can calculate the matrix FM as follows: A random user can choose the random path. If the random user is at node x_i at time t, we define the state as s(t) = x_i. We set the probability of transition from node x_i to one of its neighbors x_j by

(17) $$p\left(s\left(t+1\right)=x_j|s\left(t\right)=x_i\right)=A_{i,j}/\sum _j{A}_{i,j}.$$

The frequency matrix is computed for all pairs of nodes and the path is calculated by random walk. The i^th vector of the frequency matrix is the i^th node and j^th node is the j^th column of the frequency matrix. This is called context c_j. This frequency matrix is used to calculate the PPMI matrix P, as shown in Eqs. (18) to (21).

(18) $$p_{i,j}={\displaystyle \frac{F{M}_{i,j}}{{\sum }_{i,j}F{M}_{i,j}},}$$

(19) $$p_{i,\ast }={\displaystyle \frac{{\sum }_{j}F{M}_{i,j}}{{\sum }_{i,j}F{M}_{i,j}},}$$

(20) $$p_{\ast ,j}={\displaystyle \frac{{\sum }_{i}F{M}_{i,j}}{{\sum }_{i,j}F{M}_{i,j}},}$$

(21) $$P_{i,j}=max\{pm{i}_{i,j}=\log {\displaystyle \frac{p_{i,j}}{p_{i,\ast }{p}_{\ast ,j}},0\},}$$where p_i,j is the estimated probability that node x_i occurs in context c_j, p_i,* is the estimated probability of node x_i, and p_*,j is the probability of context c_j, thus,

$$pm{i}_{i,j}=\{\begin{array}{cc}0,& \mathrm{i}\mathrm{f}{p}_{i,j}=p_{i,\ast }\cdot p_{\ast ,j}(x_i\mathrm{a}\mathrm{n}\mathrm{d}{c}_j\mathrm{a}\mathrm{r}\mathrm{e}\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{n}\mathrm{t});\\ >0,& \mathrm{i}\mathrm{f}{p}_{i,j}>p_{i,\ast }\cdot p_{\ast ,j}(\mathrm{b}\mathrm{e}\mathrm{t}\mathrm{w}\mathrm{e}\mathrm{e}\mathrm{n}{x}_i\mathrm{a}\mathrm{n}\mathrm{d}{c}_j\mathrm{i}\mathrm{s}\mathrm{a}\mathrm{s}\mathrm{e}\mathrm{m}\mathrm{a}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{c}\mathrm{r}\mathrm{e}\mathrm{l}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n});\\ <0,& \mathrm{i}\mathrm{f}{x}_i\mathrm{i}\mathrm{s}\mathrm{u}\mathrm{n}\mathrm{r}\mathrm{e}\mathrm{l}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{t}\mathrm{o}{c}_j.\end{array}$$

The PPMI matrix P increases the relationship between data points compared to the adjacency matrix A. Using the PPMI matrix, global consistency is calculated in Eq. (22)

(22) $$Con{v}_{GC}^{(i)}(X)=Z^{(i)}=\sigma (D^{-\frac{1}{2}}P{D}^{-\frac{1}{2}}{Z}^{(i-1)}{W}^{(i)}),$$where P represents the PPMI matrix and $D_{i,i}={\sum }_j{P}_{i,j}$ for normalization. To combine the local and global consistency convolution for the dual graph convolutional network, a regularizer was used. The loss function with this regularizer is represented in Eq. (23)

(23) $$Loss=L_0\left(Con{v}_{LC}\right)+\lambda (t)\cdot L_{reg}(Con{v}_{LC},Con{v}_{GC}),$$

(24) $$L_0\left(Con{v}_{LC}\right)=-{\displaystyle \frac{1}{|y_L|}\sum _{l\in y_L}\sum _{i=1}^c{Y}_{l,i}\log {\hat{Z}}_{l,i}^A,}$$where c is the number of different labels for prediction, $Z^A\in {\mathbb{R}}^{n\times c}$ is the output given by Conv_LC, ${\hat{Z}}^A\in {\mathbb{R}}^{n\times c}$ is the output of the softamax layer, y_L represents the set of data indices whose labels are observed for training and $Y\in {\mathbb{R}}^{n\times c}$ is the ground truth.

(25) $$L_{reg}\left(Con{v}_{LC},Con{v}_{GC}\right)={\displaystyle \frac{1}{n}\sum _{i=1}^n{\left|\left|{\hat{Z}}_{i\ast }^P-{\hat{Z}}_{i\ast }^A\right|\right|}^2,}$$where ${\hat{Z}}^P\in {\mathbb{R}}^{n\times c}$ is the output of applying the softmax activation function given by Conv_GC (the vectors ${\hat{Z}}_{i\ast }^A,{\hat{Z}}_{i\ast }^P\in {\mathbb{R}}^n$ are i^th columns of the matrices ${\hat{Z}}^A$, ${\hat{Z}}^P$, respectively). For the calculations of $L_0\left(Con{v}_{LC}\right)$ and $L_{reg}\left(Con{v}_{LC},Con{v}_{GC}\right)$, the activation function called ReLU was used. After applying the activation function, the output matrix is represented as $Z^A\in {\mathbb{R}}^{n\times c}$ and $Z^P\in {\mathbb{R}}^{n\times c}$. The structure of the CNN dual graph capsule is shown in Fig. 7. This proposed network consists of three main capsules. Each capsule consists of a dual graph CNN and two capsules to represent real and fake images or videos. The output of each CNN capsule called OC_j|i is directed through dynamic routing to produce the detected output O_j for r iterations, as mentioned in Algorithm 2.

Dataset description

Accuracy

(26) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}\cdot 100}$$

Sensitivity

(27) $$\mathrm{S}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{t}\mathrm{y}={\displaystyle \frac{TP}{TP+FN}\cdot 100}$$

Specificity

It is used to evaluate the rate between true negatives (TNs) and true positives (TPs)

(28) $$\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}={\displaystyle \frac{TN}{TN+FP}\cdot 100}$$

The comparison of the accuracy of the proposed SAE-GLSTM-C-DGCNN deepfake detection is shown in Table 3 for various datasets. With the baseline of various approaches such as VGG19, ResNet, and MobileNet, we experimented with the proposed efficient deepfake image/video feature extraction with CNN capsule dual graph CNN for various datasets. For the FFHQ datasets, existing and proposed systems obtained accuracies of 84.5%, 88.32%, 91.15%, and 96.92%. For 100K-Faces datasets, the approaches obtained the corresponding accuracies of 74.12%, 80.11%, 90.21%, and 97.15%. For the Cele-DF dataset, the accuracy values are 88.43%, 89.32%, 90.01%, and 98.12%, and for the WildDeepfake dataset, they are 89.25%, 86.52%, 96.75%, and 98.91%. The results showed that the proposed system achieved better percentage results compared to traditional deepfake detection systems.

The sensitivity, specificity, and ROC comparisons of various deepfake detection systems are evaluated using four different datasets, and the results are shown in Table 4 and Fig. 8.

Table 4 shows the sensitivity and specificity analyzes of the proposed SAE-GLSTM with the C-DGCNN system compared to the existing algorithms and various datasets from FFHQ, 100K-Faces, Celeb-DF, and WildDeepfake. The proposed SAE-GLSTM with C-DGCNN obtained a sensitivity score of 91.67% in the FFHQ dataset, 89.8% in the 100K-Faces dataset, 89.1% in the Celeb-DF dataset and 93.1% in the WildDeepfake dataset. Similarly, the specificity of the proposed SAE-GLSTM with the C-DGCNN achieved a score of 92.4% in the FFHQ dataset, 93.5% in the 100K-Faces dataset, 94.2% in the Celeb-DF dataset, and 95.2% in the WildDeepfake dataset. The proposed deepfake detection of fake video/images obtained improved sensitivity and specificity percentages compared to other existing approaches. Figure 8 shows the comparison of the ROC values of various deapfake detection systems with various datasets. From the analysis, the proposed system obtained an improved ROC value of 94.2% for the FFHQ dataset, 94.1% for the 100K-faces dataset, 96.12% for the Celeb-DF dataset, and 86.54% for the WildDeepfake dataset. On the contrary, the baseline VGG19 system obtained ROC values for the FFHQ, 100K-Faces, Celeb-DF, and WildDeepfake datasets of 83.1%, 85.1%, 95.53% and 61.12%, respectively, the baseline ResNet approach secured ROC values of 79.2%, 82.3%, 88.47% and 69.78%, respectively, and the baseline MobileNet approach obtained ROC values of 81.2%, 84.1%, 91.72% and 61.54%, respectively. From the comparison, the proposed deapfake detection system secured an improved ROC value compared to traditional baseline systems.

The error detection rate comparison of the proposed vs existing approaches is shown in Table 4 evaluated on the FFHQ, 100K-Faces, Celeb-DF, and WildDeepfake datasets. These datasets are used in both the training and testing processes by using the deepfake detection classifier baseline methods namely, VGG19, ResNet, and MobileNet, with our proposed work of SAE-GLSTM with the C-DGCNN model. Table 5 shows that the proposed approach obtained a minimum error rate of 5.1 for the WildDeepfake dataset, 7.12 for Celeb-DF, 6.01 for 100K-Faces and 5.91 for FFHQ datasets. The error rate is minimal compared to the baseline deepfake detection methods such as VGG19, ResNet, and MobileNet.

Figure 9 illustrates the equal error rate (EER) of various approaches with respect to various datasets. The proposed deapfake detection system secured a minimum EER of 1.9 for the WildDeepfake dataset, 1.67 for the Celeb-DF dataset, 2.1 for the 100K-Faces dataset and 1.32 for the FFHQ dataset. These EERs are minimal compared to traditional baseline systems such as VGG19, ResNet and MobileNet.

Figure 10 shows the comparison of computational time. The proposed system secured 8.1 ms for the WildDeepfake dataset, 10.3 ms for Celeb-DF, 12.1 ms for 100K-Faces and 21.2 ms for the FFHQ dataset. This is minimal compared to other traditional baseline deepfake detection systems, where VGG19 secured computational times for the datasets of 24.2, 34.12, 44.24 and 54.23 ms. ResNet obtained 14.22, 23.45, 34.5, and 36.81 ms, MobileNet secured 30.2, 35.1, 40.2, and 44.4 ms. Therefore, all evaluation results have shown that the proposed SAE-GLSTM with CNN capsule dual graph improved sensitivity, specificity, accuracy, and minimum error rate, EER, and computation time. Ultimately, these findings have proven that the proposed system efficiently detects deepfake images/videos with improved accuracy and minimum error.