Evaluation of deepfake detection using YOLO with local binary pattern histogram

Pre-processing

The pre-processing of fake videos is divided into frames using Multi-Task Cascaded Convolutional Neural Networks (MTCNN) (Zhang et al., 2016). The fake video detects the face image from the video frames using the MTCNN model. To enhance the image quality and the accurate detection of the fake image, the alignment of the face image is done. Here, faces are extracted from the video in 224 × 224 RGB format. After detecting the face image in the video, the MTCNN model is used to extract the face. This will lead to a reduction in the amount of misrelated information in the dataset. The MTCNN model is used for detecting the human face, which includes three stages; P-Net, R-Net, and O-Net. Figure 2 shows the workflow of the MTCNN model. The first figure is resized, and P-NET, R-NET, O-NET bound box regression is computed in sequential order.

The use of the MTCNN model reduces the computation time. The cascade structure of the MTCNN model has the following stages:

• Stage 1: P-Net detects the different regions of the face using a shallow convolutional network.

• Stage 2: R-Net is used to remove the non-face regions from the detected image.

• Stage 3: O-Net is used to produce fine output using a complex network architecture.

Face feature extraction

The feature extraction is considered as an essential step in extracting the facial features. To handle the larger size of videos, face detection is a challenging task in the aspect of quick access and accuracy. In this work, EfficientNet-B5 (Tan & Le, 2019) is used to extract facial features. To obtain a better and more robust face feature extraction, EfficientNet-B5 with additive loss of angular margin (also known as ArcFace) is used (Deng et al., 2019). The classification loss function (softmax loss) is defined as follows.

(1) $$L_1=-{\displaystyle \frac{1}{M}\sum _{i=1}^M\log {\displaystyle \frac{e^{W_{y_i}^T{x}_i+b_{y_i}}}{\sum _{j=1}^n{e}^{W_j^T{x}_i+b_j}}},}$$where $x_i\in {\mathbb{R}}^d$ is the deep feature of the $i^{th}$ sample that belongs to the $y_i^{th}$ class (usually d is set to 512). $W_j\in {\mathbb{R}}^d$ is the $j^{th}$ column of the weight $W\in {\mathbb{R}}^{d\times n}$ and $b_j$ is the bias term. The batch size and the class number are M and n, respectively. ArcFace is based on the concept of classification cross-entropy loss. The main aim of ArcFace is to reduce the intra-class features and enhance the inter-class features of the face image. In the training process, the steps needed to implement ArcFace are provided below:

In Algorithm 1, the bias value $b_j=0$, then logit is transformed by $W_j^T{x}_i=\Vert W_j\Vert \Vert x_i\Vert \cos {\theta }_j$, where ${\theta }_j$ is the angle between the weight $W_j$ and the feature $x_i$, where ||⋅|| is the usual vector norm ${\ell }^2$. Furthermore, we fixed $||W_j||=1$ and the norm ${\ell }^2$ of the feature $x_i$ we rescaled the value as s (the normalization step on the features and the weights makes the predictions depend only on the angle between the features and the weights). The learned embedding features are distributed on a hypersphere with radius s.

(2) $$L_2=-{\displaystyle \frac{1}{M}\sum _{i=1}^M\log {\displaystyle \frac{e^{s\cos {\theta }_{y_i}}}{e^{s\cos {\theta }_{y_i}}+\sum _{j=1,j\ne y_i}^n{e}^{s\cos {\theta }_j}}}}$$

In Step 3, the angular margin penalty m is added with the ground truth value. m is equal to the geodesic distance margin penalty in the normalized hypersphere, and the final loss is represented as

(3) $$L_3=-{\displaystyle \frac{1}{M}\sum _{i=1}^M\log {\displaystyle \frac{e^{s\cos ({\theta }_{y_i}+m)}}{e^{s\cos ({\theta }_{y_i}+m)}+\sum _{j=1,j\ne y_i}^n{e}^{s\cos {\theta }_j}}}}$$

Proposed face detection using YOLO-LBPH

The input video image file is layered as frame by frame; whole frame of the video image is used to extract the features. it is considered as a tedious process. Therefore, the features are extracted only from the face region using a YOLO-LBPH face detector. You Only Look Once (YOLO) with Linear Binary Pattern Histogram (LBPH) is a face detector that detects faces from the frames of the video image. The main advantage of using hybrid technique of YOLO-LBPH is a faster detection of real-time object, and it also recognizes the image with less computational time complexity. The various frames of the videos helps to detects the head posture and angle diffrences in each frame for fake image identification. Read the input video image and divide the image into img1 × img1. Here, img1 is the grid size and is selected in random basis. Each cell of the grid represents multiple bounding boxes. This boxes used to predict certain aspects, such as class probabilities, an object, and confidence scores. The Non-Maximal Suppression (NMS) method is used for predicting the boxes. finally the overlapping boxes are remove. The every object is detected by NMS at once and false object is removed. Here, confidence score for each boundary box is predicted. This score is represented as probability of each class and detects only one object in a box. The boundary boxes are created by clustering the ground truth boxes from its dataset. Figure 3 shows the framework of YOLO for detecting the object.

The YOLO works based on Convolution Neural Network principle. It consists of a total of 24 convolutional layers followed by two fully connected layers. The first 20 convolutional layers are followed by an average pooling layer with fully connected layer, and it is a pre-trained data set of images with a resolution of 224 × 224 × 3. In addition to that, the training process of detecting the object needs the last four convolutional layers followed by two fully connected layers. The object detection is made comfortable with increasing resolution of the images in the dataset to 448 × 448. In the last layer, it predicts the probability of class and bounding boxes, and linear activation function is executed here, and the remaining layers implement the ReLU activation function.

LBPH is used to recognize facial images in the database. the binary operator is used to Extract facial features from image, it recognizes the image with less computational time complexity. Algorithm 2 describes the LBPH.

In Algorithm 2, LBPH is the fusion of Local Binary Patterns (LBP) technique with the Histograms of Oriented Gradients (HOG) descriptor. It is a simple and powerful method to extract features and label pixels in the face image. LBPH is easy and time saving methodology.

Data collection

This proposed YOLO-LBPH is implemented by using the deepfake forensics (DF) dataset Celeb-DF-FaceForencics++(c23) which is a combination of Face Forencies++(c23) and Celeb-DF. For training the dataset, it randomly splits the dataset into a training sub-dataset and a validation sub-dataset. Pixels in the image are normalized to the range (−1, 1). The Face Forencies++ dataset and the Celeb-DF dataset are collected. The Celeb-DF dataset was originally divided into 5,299/712 to train the data and 340/178 to test the data as fake and real videos. Additionally, data from DFFD and CASIA-WebFace are collected.

Performance metric measures

The evaluation of algorithm is done using the metrics like measuring mean square error and accuracy. To evaluate the performance of the proposed algorithm, it is compared with other existing approaches such as YOLO EfficientNet-B0 + [Bi-LSTM], YOLO + [XceptionNet] + [Bi-LSTM], YOLO + [XceptionNet] + [LSTM] (Ismail et al., 2021; Masi et al., 2020; Rossler et al., 2019; Güera & Delp, 2018). To evaluate the performance of the metric measures, such as the accuracy, sensitivity, specificity, and error rate of the Root Mean Square Error (RMSE), Signal-to-Noise-Ratio (SNR), Peak Signal-to-Noise-Ratio (PSNR), and Mean Absolute Error (MAE) are used. One of the evaluation metrics, the Area Under the Receiver Operating Characteristic (AUROC) curve, is used to evaluate the real and fake image using the proposed YOLO-LBPH.

(7) $$TPR={\displaystyle \frac{TP}{TP+FN},}$$

(8) $$FPR={\displaystyle \frac{FP}{FP+TN}.}$$

Accuracy

(9) $$Aaccuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}\cdot 100,}$$

(10) $$Recall={\displaystyle \frac{TP}{TP+FN},}$$

(11) $$Precision={\displaystyle \frac{TP}{TP+FP},}$$

(12) $$F-Measure=2{\displaystyle \frac{Precision\cdot Recall}{Precision+Recall},}$$

The error rate is given by

(13) $$PSNR=20{\log }_{10}\left({\displaystyle \frac{{255}^2}{MAE}}\right).$$

(14) $$MAE={\displaystyle \frac{1}{MN}\sum _{i=1}^M\sum _{j=1}^N\left|X\left(i,j\right)-Y\left(i,j\right)\right|,}$$

(15) $$RMSE=\sqrt{{\displaystyle \frac{1}{N}\sum _{i=1}^N{\left(X_i-\widehat{X_i}\right)}^2}},$$

(16) $$SNR(db)=20\log \left({\displaystyle \frac{V_{RMS(Signal)}}{V_{RMS(Noise)}}}\right).$$

Table 1 shows the precision of various algorithms with the proposed work. The publicly available datasets are implemented such as FaceForencies++, Celeb-DF, DFFD, CASIA-Web Face. The accuracy rate of the proposed work YOLO-LBPH was 98.12% in the CASIA-Web Face image dataset. The next position in terms of the accuracy rate is 97.56% for the DFFD dataset.

Table 2 shows the error detection rate for the proposed work for various dataset.

Table 2 shows the mentioned algorithms error rate in different datasets. The proposed YOLO-LBPH had a minimum error rate equal to 6.16 for the CASIA-Web Face dataset. Fig. 4 shows the AUROC curve for the deepfake forensics (DF) datasets Celeb-DF-FaceForencics++(c23), DFFD and CASIA-WebFace, YOLO EfficientNet-B0 + [Bi-LSTM] has produced a better result.

Figure 4 shows that for the deepfake forensics (DF) dataset Celeb-DF-FaceForencics++(c23), the proposed work YOLO-LBPH has produced a better result compared to the other existing algorithms. In the DFFD dataset, YOLO + [XceptionNet] + [LSTM] has produced a better result in the CASIA-Web Face dataset. Figure 5 shows the error rate value calculated based on its accuracy by using Eqs. (13)–(16).

In Fig. 5, it is shown that the proposed algorithm YOLO-LBPH requires less computation time compared to the other existing algorithms. Various frames analyses of same image makes the prediction very fast. The analysis of training and testing of the data set in fake face video images detection and validation of face input data in terms of accuracy and loss metric information is done for the data set of deepfake face images. Figure 6 shows the computation time for various algorithms with various datasets such as Celeb-DF-FaceForencics++(c23), DFFD, CASIA-Web Face dataset.

In Fig. 6, it is shown that the proposed algorithm YOLO-LBPH needs less computation time compared to the other existing algorithms. The analysis of training and testing of the dataset in the detection of fake face video image is presented. The result produces the validation face input data in terms of accuracy and loss metric information for the deepfake face image dataset.

Figure 7 shows the precision metric measures of different datasets with different algorithms. The proposed work provides better performance compared to the existing algorithms in various datasets such as Celeb-DF-FaceForencics++(c23), DFFD, and CASIA-Web Face. Here, the proposed YOLO-LBPH has a precision score of 86.88% in the Celeb-DF-FaceForencics++(c23) dataset, 88.9% in the DFFD dataset, and 91.35% in the CASIA-Web Face dataset.

Figure 8 shows the recall metric measures of different datasets with different algorithms. The proposed work has provided better performance compared to the existing algorithms. The various datasets of Celeb-DF-FaceForencics++(c23), DFFD, CASIA-Web Face dataset are used in evaluation. We can see that the proposed YOLO-LBPH has a recall score of 92.45% in the Celeb-DF-FaceForencics++(c23) dataset, 93.76% in the DFFD dataset and 94.35% in the CASIA-Web Face dataset.

In Table 3, the F1-Measure of different algorithms was analyzed. The proposed method provides better performance compared to the existing algorithms and various datasets of Celeb-DF-FaceForencics++(c23), DFFD, and CASIA-Web Face. Here, YOLO-LBPH has the F1-measure as 91.45% in the Celeb-DF-FaceForencics++(c23) dataset, 92.88% in the DFFD dataset, and 95.35% in the CASIA-Web Face dataset.