Identifying defective solar cells in electroluminescence images using deep feature representations

Data acquisition

We used a public collection of monocrystalline and polycrystalline PV modules with high-resolution EL images. At a scale of (300 × 300) pixels, the collection contains 2,624 solar cell images taken from 44 different PV modules, of which 18 are monocrystalline and the other 26 are polycrystalline. The experts initially classified the dataset into four types according to the severity of the problem (Buerhop-Lutz et al., 2018). Classified as the ratio of none defected is 0%, ratio of possibly normal is 33%, the ratio of possibly defective is 66%, and the ratio of defected is 100%. There are 1,508, 295, 106, and 715 images in these categories. Occurrence in monocrystalline and polycrystalline solar modules is inherent and extrinsic faults in the solar cells. Microcracks and cells with electrically isolated and damaged sections, short-circuited cells, open interconnects, and soldering faults are all included in the collection in this manner.

Data augmentation

To train DNNs, a large dataset is required to achieve high performance in mapping an input to the desired output effectively. Due to the very small size of the training set in the ELPV dataset, different data augmentation techniques can be applied to expand its size. Here, the orientation of the cracks is not important if it is at positive or negative angles. Furthermore, the appearance of cracks at the upper or lower edge does not matter. Thus, different data augmentation techniques can be successfully applied. In this study, several data augmentations have been applied on the training set to enhance the generalization capacity of the adopted deep learning models and avoid overfitting issues. First, all original images have been rotated to 90°, 180° and 270° and added to the original training set to have a different diversity of defect patterns. Second, we flip original images along the x-axis and y-axis. To get more images with a diversity of defect patterns, the Gaussian blur reduce the effect of the dark regions in the EL images and produce new images with different and useful information. Finally, the image contrast technique was also applied to the original EL images to have a wide variety of EL images with different crack appearances.

Learning deep features representations

Two deep learning networks, Inception-V3 and ResNet50, have been used to extract two set of characteristics from the EL images. The features are then combined to create more accurate data representations. Due to the potential of deep learning feature descriptors to automatically learn the fundamental features in an EL image, their use can significantly reduce the need for handcrafted feature extraction. This is the first work to come up with a way to tell the difference between EL images of working solar cells and those that are malfunctioning. In the end, these feature representations are given to the classifier layer, which assigns them to a specific defect type.

Feature extraction using ResNet50 model

ResNet is one of the most advanced deep models for image categorization. When the deep network cannot be trained, ResNet overcome the challenge to ameliorate classification accuracy while simultaneously reducing the number of parameters. Therefore, we use ResNet50 as a deep convolutional feature extractor. ResNet adds to the network’s depth without affecting classification performance based on residual learning. The remainder modules have two paths: one executes two or three convolutions on the input feature to produce the residual of that feature, and the other is a direct path from the input feature. The output of the residual module is the sum of the outputs of these two pathways. As shown in Fig. 3, ResNet50 was used to extract features. The weights that have been pre-trained on EL datasets are used to initialize the network. Deeper layers’ learnt weights tend to be more class-specific, such as ResNet50’s completely connected layer. We were interested to know how well the output vectors of the previous convolutional layers could be classified. Networks with deep convolutional layers are significant features when applied correctly. We used the last residual unit outputs from convolutional layers 3, 4, and 5 as feature vectors. The third layer features have a smaller dimension than the fifth layer features. Table 2 shows that we used ResNet50 in our investigation.

Table 2:

Layers and parameters applied by resnet50 model for defective solar cell detection.

Layer name	Output size	50-layer
Conv1	112 × 112	7 × 7, 64, stride 2
Conv2-x	56 × 56	3 × 3 max pool, stride 2
		$\left[\begin{array}{c}1\times 1,64\\ 3\times 3,64\\ 1\times 1,256\end{array}\right]$×3
Conv3-x	28 × 28	$\left[\begin{array}{c}1\times 1,128\\ 3\times 3,128\\ 1\times 1,512\end{array}\right]$×4
Conv4-x	14 × 14	$\left[\begin{array}{c}1\times 1,256\\ 3\times 3,256\\ 1\times 1,1,024\end{array}\right]$×6
Conv5-x	7 × 7	$\left[\begin{array}{c}1\times 1,512\\ 3\times 3,512\\ 1\times 1,2,048\end{array}\right]$×3
	1 × 1	Average pool, 1,000-d fc, softmax
FLOPs		1.8 × 109

DOI: 10.7717/peerj-cs.992/table-2

To improve the classification accuracy, a more robust feature vector is obtained using feature fusion process. The most recent research shows that the fusion process enhances overall performance, however the major drawback is that it requires a lot of time to compute. Although our primary goal is to enhance classification accuracy. New Parallel Maximum Covariance (PMC) features fusion method is used to achieve this goal. The lengths of the two feature extraction vectors must be equalized in this method. We then calculate the single-matrix fusion’s greatest covariance. The proposed feature fusion approach is depicted in Fig. 4.

Consider ${\phi }^{\left(\mathbf{k}1\right)}$ and ${\phi }^{\left(\mathbf{k}2\right)}$ are two different feature vectors based deep learning of two different dimensions (n × m) and (n × q). The number of images is represented by n, m is the Inception-V3 deep learning feature vector of (n × 2,048) dimensions, and the ResNet50 feature vector of (n × 2,048) dimensions represents by q. To make equalization between lengths of feature vectors, we need to know what the maximum length of a certain vector after that calculate the padding of average value. The average feature is computed from vector with a higher vector. Assume n is a random column m unit vector displaying some pattern in the $\phi 1$ field. The pattern in the field of $\phi 2$ is represented by a random unit of column vector which indicated by b. The projections of time series on row vectors are formulated in the following manner:

(6) $$x_1={\phi }_1^T{\phi }^{\left(k1\right)}$$

(7) $$x_2={\phi }_2^T{\phi }^{\left(k2\right)}$$

For optimal solution $\phi 1$ and $\phi 2$ maximize their covariance as follows:

(8) $$c^{\sim }=Cov\left[x_1,x_2\right]$$

(9) $$c^{\sim }=Cov\left[{\phi }_1^T{\phi }^{\left(k1\right)},{\phi }_2^T{\phi }^{\left(k2\right)}\right]$$

(10) $$c^{\sim }={\displaystyle \frac{1}{n-1}\left[{\phi }_1^T{\phi }^{\left(k1\right)}\left({\phi }_2^T{\phi }^{\left(k2\right)}\right)\right]}$$

(11) $$c^{\sim }=\phi 1\left(C_{\phi 1\phi 2}\right)\phi 2$$

(12) $$C_{\phi 1\phi 2}={\displaystyle \frac{1}{n-1}\left[{\phi }^{\left(k1\right)}{\phi }^{{\left(k2\right)}^T}\right]}$$

${\mathit{C}}_{\phi 1\phi 2}$is the covariance value between $\phi 1$ and $\phi 1$ whose i^th and j^th features are $\phi \mathbf{i}\left(\mathbf{t}\right)$ and $\phi \mathbf{j}\left(\mathbf{t}\right)$.

The last fused vector saves the feature pair i and j with the highest covariance ${\mathit{C}}_{\phi 1\phi 2}$. Once all pairs have been compared, the procedure repeats again. Finally, a fused vector is acquired, represented by ${\phi }^{\left(\mathbf{f}\mathbf{u}\right)}$ of dimensions $\left(\mathit{N}\times \mathit{K}\right),$ where $\mathit{K}$ specifies the feature length, which changes depending on the features picked. (N × 4,096) is the feature length in this study. Dropout is a method for avoiding overfitting neural networks by removing units at random. We set a dropout of 0.5 for the fully connected layer in our investigation.

The proposed training strategy

This work pre-trained the weight configurations of ResNet50 and Inception-V3 architectures. Initializing entire weights with the weight configurations has been performed of the pre-trained architectures in the learning strategy. Then, Adam optimizer is employed to optimize them through the current task. To detect most relevant and significant features the pre-trained models based on their weight configurations have been learned. Training time could be reduced, the ability of model generalization could be enhanced, and over-fitting can be prevented. In the first classification scenario we have performed binary classification to classify defective solar cell into functional and defective categories. However, multi classification scenario has been performed to classify defective solar cells into four different categories namely, severe, moderate, functional, and mild. Both previous scenarios are performed based on unaugmented and augmented dataset. During the process of learning, the proposed deep feature representation model has been trained based on distributing dataset into 80% as training data and 20% as testing data. The selection dataset process was done randomly. Finally, the performance of training process as well as the final performance based on using proposed deep hybrid feature have been reported.

Image classification stage

Combining the findings of several models, the ensemble of deep CNNs may prove to be an effective strategy for improving outcomes. The procedure of classification involves the organized and methodical assignment of every structure to one and just one category in a system of classes that are mutually exclusive and do not overlap categories. In this study, binary and multi-classes classification are carried out using sigmoid and Softmax activation layers, respectively. The fault probability class of a solar cell was determined utilizing binary class classification technique in this work. Extracted features based deep network models can be used to identify functional and faulty cells in the EL images of this data set. The class 0 (functional) samples make up 65% of the EL dataset, while class 1 samples make up 35% (defective). For binary classification of EL images, sigmoid activation is employed. In this work, the parameters of the adopted deep learning models are trained using the Adam optimizer algorithm for 300 epochs with a mini-batch size of 16, a learning rate set to be 0.01, a weight decay value of 0.0005, and a momentum value of 0.95. In classification of EL images based, the first dense layer has 64 neurons instead of 128.

Impact of data augmentation on results

Train the deep learning models on small datasets is a significant challenge due to the overfitting problem that became as a significant issue in this situation. Data augmentation procedures are implemented on original images to obtain more information that can help to train the deep learning model. For data augmentation, we have performed rotation, flipping, and cropping operations. It was necessary to carefully examine the newly obtained images to generate new set of images with the same labels, to identify variations from the original images, and to extract important information from them. Data augmentation procedure results in an improvement in model accuracy, leading to the conclusion that the data augmentation operations that were chosen are successfully implemented. They help to reduce the difference between images and aid in the model’s learning of more representative feature subsets, which results in a significant improvement in recognition accuracy. Experiments were carried out with Inception-V3, ResNet50, and proposed models with various data augmentation settings to determine the effect of data augmentation. All features from Inception-V3, ResNet50m and the proposed model are extracted for the purposes of these experiments, and the dataset is separated into two different parts: 80% and 20% as training and testing, respectively. From results obtained in Tables 4, 5, 6, and 7, the highest accuracy is achieved with data augmentation techniques.

Image classification results

Two experiments have been implemented to evaluate the outcome of the proposed model. First, the PV cells divided into two classes (normal and defective) using the ground truth labels to train and test the proposed hybrid model. Second, the same dataset divided into four classes (normal, mild, moderate, and severe) and using them to train a new model that can give more specific classification for the testing data. As a result, this study presents two-class classification results, one for normal classes and another for the defective classes. EL images of normal cells show that their surfaces are homogeneous and that the faults in those cells are distinct from the background in the image. Cells in these classes have a wide range of textures, and defects in those cells are frequently similar in appearance to the background in an EL image, making it difficult to distinguish between them. Both classifications were evaluated for their performance and the results of this evaluation are presented in the following subsection.

Tables 5 and 6 demonstrate results of evaluation of performance of test before applying data augmentation as well as after applying data augmentation. features extracted from Inception-V3, ResNet50, and integrated features are presented in Tables 5 and 6. Accuracy rates were obtained as 94.44% for Inception-V3, 95.91 for ResNet50 and 98.15% for combined features. features extracted from Inception-V3 network achieved worst result between m. proposed model (combined features) provided better results than or models when used independently. Confusion matrices are given in Fig. 5 for binary classification task before and after data augmentation.

Performance evaluation is demonstrated using ROC curves, which represent the performance of binary classification based on the sigmoid activation function at different false positive rates. ROC curve in Figs. 6A and 6B presents the performance result of the Inception-V3 and ResNet50 models, respectively. It shows that the two models achieved ROC curve performance nearly similar by obtaining 94.44% for Inception-V3 model and 95.91% for ResNet50 model. Figure 6C shows the ROC performance for the proposed hybrid model by obtaining 98.15%. ROC of hybrid Inception-V3 and ResNet50 performs results outperformed of each model independently.

Multi-classes classification task

A four-class classification performance experiments utilizing the Softmax activation function are presented as shown in Tables 7 and 8. The integrated feature vectors in four class classification tasks have achieved the best results. The accuracy rates of 48.39% and 87.67% for Inception-V3, 52.68% and 91.60% for ResNet50, finally 98.15% and 95.35% for combined features were achieved for each model before and after applying data augmentation, respectively.

Table 8:

Comparative performance (%) for binary defective solar cells classification approaches.

Approaches	Feature dimension	Image size	Accuracy	Recall	Precision	F1-Score	Specificity
L-CNN (3)	2,000	100 × 100	89.33	95.42	90.44	92.86	–
DFB-SVM (3)	2,000	100 × 100	94.52	97.36	94.79	96.06	–
Deep Siamese CNN (12)	4,096	100 × 100	93.02	93	93	92.49	–
VGG + Linear SVM (5)	4,096	300 × 300	82.44	–	–	82.52	–
CNN (5)	4,096	300 × 300	88.42	–	–	88.39	–
Light CNN (7)	4,096	100 × 100	93.02	93	93	92.49	–
MVGG19 (29)	2,500	300 × 300	97.88	98.57	96.49	97.5	–
Inception-V3	2,048	300 × 300	94.44	94.81	94.12	94.46	94.07
ResNet50	2,048	300 × 300	95.91	96.3	95.59	95.94	95.52
Hybrid Model	4,096	300 × 300	98.15	98.51	97.78	98.14	97.79

DOI: 10.7717/peerj-cs.992/table-8

Figure 7 shows the confusion matrices of all the methods before and after applying data augmentation. For the two class confusion matrices, the proposed model obviously outperformed both Inception-V3 and ResNet50 for all classes. Mild and Severe cell images with defects such as the background have a high likelihood of being misclassified. Cell images with minor defects are incorrectly classified in the case of the Normal and Moderate images. Misclassification of two-class classification tasks occurs due such EL images of the cells are at the boundary between obviously differentiable normal and defective cells, resulting in incorrect classification.

Comparison study

Using the same dataset that we used in our study, we compare the results of recently developed approaches with our own findings. Table 8 shows the results of the study compared to previous studies. Compared to other techniques, our proposed model is superior with 98.15% accuracy, 98.51% recall, 97.78% precision, and 98.14% F1 score.

Performance of the proposed system was also compared with most current state-of-the-art existing works in term of multi-classification task, as shown in Table 9. Our method achieved 95.35% accuracy, 95.5% recall, 95.53% precision, and 95.25% F-Score. The proposed approach outperformed all prior ones in terms of accuracy and other metrics, as can be shown in the results.

The classification of solar cell faults in EL images is a challenging process because of the intrinsic silicon structure that creates crystal grain boundaries in solar cells. This makes the classification of solar cell defects particularly difficult. It becomes more difficult to distinguish between defective and normal areas because there is a lack of a large enough collection of both normal and defective cell images, as well as normal, mild, moderate, and severe cell images, is another important consideration. At a certain level, these issues are addressed by data augmentation.

Research limitations should be highlighted. First, the EL images suffer from a specific type of noise. Further investigation is required to understand and reduce the noise and keep the important texture information. Second, the proposed hybrid model needs to be trained and tested on more data from different environments to have more challenges with different data. Finally, the similarity between the texture of the normal cells and defected cells is a hard challenge. It still needs more investigation to find more effective descriptor that can reduce overlap between classes.