Deep fake detection using cascaded deep sparse auto-encoder for effective feature selection

Feature extraction using proposed cascaded deep sparse auto encoder trained by TCNN

The normalized dataset is transformed into a reduced dimension using the Auto Encoder (AE) approach. This paper used traditional AE to reduce the dimensionality of the dataset and the version of the Cascaded Deep Sparse AE to select the computer vision features from the reduced dataset. AE is a deep learning-based neural network learned by an identity function. It is an unsupervised learning method widely used for dimensionality reduction (Hinton & Salakhutdinov, 2006). The feed-forward AE has been used, consisting of one layer with many nodes acting as a bottleneck. AE used backpropagation after the training process. In a bottleneck, the layer above is an encoder, and the layer below is called a decoder, as shown in Fig. 2. The input vector is transformed into a low-dimensional space in the encoder and reconstructed using the decoder. The hidden layer is denoted as H, X is the input, and Y is the reconstructed input. AE uses a linear activation function for a single hidden layer and a non-linear activation function for more than one hidden layer.

Deep AE provides better performance than traditional AE with an equal number of parameters (Lu et al., 2013; Maheswaran et al., 2015). The Deep AE is composed of many hidden layers, and the error is minimized using backpropagation compared to traditional AE which consists of one hidden layer with its input and output layer. Network performance is generalized by adding sparsity, which reduces network links. The sparse representation of classification has been studied by various researchers for face recognition (Gui et al., 2015; Maheswaran et al., 2017b), object categorization (Gui et al., 2014b), classification (Gui et al., 2014a) and regression oriented tasks (Bradski, 2000). Deep sparse AE is defined by two steps where the first step is to perform unsupervised layer wise greedy training as pre state. This step is used to extract the computer vision features that are more relevant with the help of unlabeled data in the form of encoder and decoder format. During this pre-training, the labeled data are not needed to reproduce the input to output. Let X_p be the input of the encoder which is encoded in the form of a function indicated in Eq. (1). The decoder decodes the encoded input as in Eq. (2) to reconstruct the real data input. (1)${\displaystyle E_p=func\left(X_p\right),}$ (2)${\displaystyle D_p=func\left(E_p\right).}$

During pre-training, AE can reproduce the input to its respective output, which will lead to an overfitting issue. To handle overfitting, the sparsity term is added to the loss value function, which can generalize the training phase. The loss value function or the cost function is denoted in Eq. (3) (3)${\displaystyle Loss=MSE+\alpha .{\tau }_{wt}+\beta \cdot {\tau }_{sty},}$ where α = coefficient of regression weight to prevent over-fitting,

β = parameter of sparsity regularization and penalty is set to the sparsity term

where MSE means square error between the decoded and real input defined as (4)${\displaystyle MSE=\frac{1}{S}\sum _{i=1}^I\sum _{j=1}^J{\left({X_p}_{ij}-{D_p}_{ij}\right)}^2,}$ H is the number of hidden layers, S = number of input samples τ_wt is weight regularization with coefficient α, τ_sty is the sparsity regularization term with coefficient β: (5)${\displaystyle {\tau }_{wt}=\frac{1}{2}\sum _{l=1}^L\sum _{i=1}^I\sum _{j=1}^J{\left(W_{ij}^l\right)}^2,}$ (6)${\displaystyle {\tau }_{ty}=\sum _{k=1}^{S^{\prime }}{PL}_X\left(\frac{\eta }{{\eta }_k}\right)=\sum _{k=1}^{S^{\prime }}\left(\eta \cdot log\frac{\eta }{{\eta }_k}+\left(1-\eta \right)log\frac{1-\eta }{1-{\eta }_k}\right),}$ where I is the number of instances, J is the number of variables, W are weights that control the weights of the network. η_k is an activation value of the neuron k, η is a desired activation value, PL sparsity proportion, L total number of layer and s total number of neurons.

For a selected neuron, if the desired and average activation function value is the same, then the sparsity regularization is zero. When the difference between η and η_k increases, then the sparsity regularization also increases. In DSAE, if the decoder is removed, then it becomes a deep network. The proposed computer vision-based feature extraction process using the Cascaded DSAE. The proposed architecture shown in Fig. 3 consists of the Cascaded DSAE each representing each frame. The output of multiple DSAEs is passed through the flatten layer, and the output is merged. Maximum voting is used to choose the best output from the merged results. The best features extracted through DSAE are trained by Temporal CNN to enhance the feature extraction process.

(a) CDSAE-T-CNN training and optimization

The proposed CDSAE is optimized and trained by T-CNN which consists of two one-dimensional convolution layers, two interconnected layers, and one softmax layer to perform the softmax function as shown in Fig. 4. For the overfitting issue, the maximum pooling, normalization, and dropout layers are used. The training and T-CNN based optimization is executed as follows;

1. One-dimensional convolution layer: it is used to comprise the feature input vectors and apply 64 various filters with a filter scale of 3.

2. Second-dimensional convolution layer: this works with 64 filters on four dimensions and learns the advanced functions until the pooling layer starts.

3. Maximum pooling: it chooses the filter that has secured the maximum value to avoid overfitting.

4. Batch Norm layer: it normalizes the data that are received from the previous layer.

5. Dense layer (fully connected): it has 140 intermediate nodes and 30% of dropout values.

6. Softmax layer: it generates two group; one for deepfake detection and the other for normal.

The fully connected layers at the beginning are responsible for estimating the output probability, as stated in Eq. (7) where D is the irregular event. This gate is changed to reflect the output as defined in Eq. (8) (7)${\displaystyle P\left(l_n:v_{1:n}\right)=p\left(D_{l_n}\left(H\right)\right),}$ (8)${\displaystyle I_t=\frac{1}{e_t}\sigma \left(w_I{X}_t+V_i{H}_{t-1}+b_I\right),}$ where l_n is the number of nodes in each layer, e_t is the type of event (in this study it is set to 2) and σ is the usual sigmoid activation function of a feature vector. The output of TCNN is generated as in Eq. (9) (9)${\displaystyle Y_t=\sigma \left(w_Y{X}_t+w_Y{H}_{t-1}+b_Y\right),}$ where the parameters are defined as the previous sections X_t be the encoder input, w is weight, H is the number of hidden layers and b is the bias. Based on the number of layers, epochs, number of features, and nodes in each layer, the computational complexity of the proposed system is defined. Due to the implementation of the proposed deep learning model, the complexity of the proposed system is reduced. The extracted computer vision features are presented in Table 2.

Table 2:

Extracted computer vision features using proposed CDSAE-TCNN.

No. of feature	Feature name	Description
1	MSE	Mean square error is the average variance between actual and estimated values.
2	PSNR	Peak signal to noise ratio is the ratio between maximum signal power and corrupted noise.
3	SSIM	Structural similarity index measure is the quality of cinematic and television pictures.
4	RGB	The percentage of image red, green and blue color value.
5	HSV	The percentage of image hue, saturation and value.
6	Histogram	Based on image brightness, it plots the no. of pixels in the image or frames.
7	Luminance	Total image brightness mean value.
8	Variance	Variance of image.
9	Edge-Density	Ration between edge pixel and total pixel of image.
10	DCT	Discrete Cosine transform: Image DCT bias value.

DOI: 10.7717/peerjcs.1040/table-2

MSE is calculated using the intensity of two differences in the image pixels. PSNR aims to find the numerical difference rather than the visual variance of the human, since if MSE is zero, PSNR is also set as 0. SSIM has computed the temporal difference between luminance, contrast, and structure. R, G, B, and H, S, and v represent the color of the image. The distribution of the hue in the image is represented as a histogram. Average image brightness is represented as luminance. The brightness of the image variance is represented as a variance. Edge density is the ratio of the edge component of the pixel, and DCT is the sharpness of the image. Because the Deep Fake image synthesizes the target picture of each frame, it may lead to unnatural modifications in the computer vision features. When creating the deepfake, the target image should have limited resolution, and the size of the image has also changed. Therefore, inferior sharpness, blurring, and distortion occur in the deepfake, and the selection of these computer vision features should enhance the detection process.

Detection using DNN

From the selected features, the variance is calculated based on the rate of change and is used for DNN learning. The dependent variable denotes the data as deepfake or real. The final output is computed as in Eq. (10) (10)${\displaystyle D_k=\frac{1}{n}\sum _{i=1}^n{\left({data}_{k,i}-\overline{dat{a}_k}\right)}^2,}$ D_k is the value of the kth feature for the ith sample which is used for the training of DNN, data_k,i represents the ith data from prepossessing of the kth feature and $\overline{dat{a}_k}$ is the average of all data from feature extraction of the kth feature.

Datasets used

The evaluation is performed using deepfake datasets such as Face2Face, FaceSwap developed by FaceForensics++ (Deepfake Detection Challenge, 2022) and DFDC(Deepfake Detection challenge) dataset from Kaggle (Zagoruyko & Komodakis, 2016). This dataset is comprised of 1000 videos and more. DFDC consists of more than 470GB of data. Additionally, the details of the datasets are presented in Table 3. The information in the dataset is related to genders, various races, and shooting circumstances. This proposed study has utilized 205 videos of Face2Face, 211 videos of FaceSwap, and 175 videos from DFDC datasets for experiment. Among the videos, there are 310 frames that are extracted from each video and the face of the frames that are extracted using MTCNN with the setting of 150 ×150 pixels. To extract the computer vision features from the frames, OpenCV has been used.

Hyper-parameter settings

The DNN hyperparameter and DNN have been fixed by choosing more than one optimizer and evaluating the proposed system. Table 4 shows the accuracy of various optimizers and the various numbers of hidden layers to fix the DNN hyperparameter. From the observation, the Adam optimizer has secured a lower loss and improved accuracy. Among the setting of hidden layers, five layers reach the improved accuracy. Therefore, the proposed model has been implemented with the setting of five hidden layers and Adam optimizer, which is used to handle the loss function.

Evaluation

The proposed model is evaluated using evaluation metrics using the datasets and compared with ResNet (Sandler et al., 2018) and MobileNet (Schuldt, Laptev & Caputo, 2004; Maheswaran et al., 2021b) and SVM (Maheswaran et al., 2020b). Table 5 shows the comparative results on the accuracy of detection.

All methods perform well on Face2Face and FaceSwap datasets with a precision of 90% and more, except SVM. But for the DFDC dataset the performance methods have been reduced, and the minimum accuracy percentage has been obtained. Compared to the evaluation methods, the proposed feature selection approach has secured an improved accuracy of 98.7%, 98.5% and 97.63% for the Face2Face dataset, FaceSwap dataset, and DFDC datasets, respectively. The proposed model comparison based on content sharing performance is shown in Table 6. The comparison has revealed that the proposed system is superior in terms of structure, integrity, security, transparency, storage, and data sustainability.

The sensitivity and specificity of the comparison for the proposed system is shown in Figs. 5 and 6. In addition to that, the graphical illustrations show that the improved percentage of sensitivity and specificity for all the datasets is obtained by the proposed model compared to the other similar deepfake detection systems. In addition to the proposed model, ResNet has performed well.

The evaluation of the proposed system in terms of computation time, AUC, and error rate comparison are shown in Figs. 7, 8 and 9 using the deepfake datasets. From Fig. 7, for Face2Face dataset, the proposed model has utilized 19.2 s of time which is minimum than the other approaches such as ResNet, MobileNet and SVM which have secured 41.3 s, 38.6 s and 53.4 s respectively. For FaceSwap dataset, the proposed model has secured 10.2 s of time, which is minimal compared to other methods such as ResNet, MobileNet and SVM that have secured 39.4 s, 33.6 s, and 41.12 s sequentially. For the DFDC data set, the proposed model has secured 7.1 s, ResNet has secured 32.1 s, MobileNet has obtained 31.8 s and SVM has obtained 33.01 s. Compared to a similar deepfake detection system, the proposed model has obtained less computational time.

The evaluation of the proposed system in terms of AUC Fig. 8 using the deepfake datasets. For Face2Face dataset, the proposed model utilized 0.988 of time, which is improved than other approaches such as ResNet, MobileNet and SVM which secured 0.943, 0.963, and 0.714, respectively. For FaceSwap dataset, the proposed model secured 0.973 of time which is improved compared to the methods such as ResNet, MobileNet and SVM secured 0.912, 0.934 and 0.767 sequentially. For the DFDC dataset, the proposed model secured 0.982, ResNet secured 0.921, MobileNet obtained 0.945, and SVM obtained 0.784. Compared with similar deepfake detection systems, our proposed model obtained an improved AUC rate.

The proposed system evaluation in terms of the comparison of Error rate is shown in Fig. 9 using the deepfake datasets. From Fig. 9 the observations are as follows; for the Face2Face dataset, the proposed model has utilized 5.3 s of time, which is minimal compared to other approaches such as ResNet, MobileNet and SVM which have secured 15.9, 14.2 and 17.51 respectively. For FaceSwap dataset, the proposed model has secured 4.2 of time, which is minimum compared to the methods such as ResNet, MobileNet and SVM that have secured 12.3, 10.76 and 18.5 sequentially. For the DFDC dataset, the proposed model has secured 6.01, ResNet has secured 10.6, MobileNet has obtained 9.8 and SVM has obtained 14.02 respectively. Compared to similar deepfake detection systems, the proposed model has a lower error detection rate. Hence in the kind of evaluation, the proposed computer vision feature extraction based deepfake detection system has secured improved accuracy and efficiency on the detection of deepfake from normal video/image.

The fake detection that is performed by the proposed technique has a lower error rate compared to the existing techniques. Of the three datasets, the proposed CDSAE-DNN has a lower error in detecting fake faces in the input images.