Identifying optical microscope images of CVD-grown two-dimensional MoS₂ by convolutional neural networks and transfer learning

Transfer learning

While machine learning methods exhibit remarkable capabilities in solving complex problems, their effectiveness hinges on the availability of sufficient amount of training data. Challenges in data acquisition may arise due to factors such as high costs, time constraints, and confidentiality concerns. In cases where acquiring specific data is challenging, data obtained from any analogous system, device, or process, related to the existing problem in need of a solution can be used in deep learning as a preliminary training process before training with data specific to the problem (Tan et al., 2018). For instance, addressing an issue with a spacecraft’s engine might require more data than is currently available. In such cases, data from aircraft engines, which share similarities, can be employed in the pre-learning task. Similarly, the characterization of 2D materials faces a parallel challenge, and obtaining a sufficient number of images is a demanding process. Additionally, analyzing enlarged MoS₂ flakes for classification demands significant time from researchers due to the high pixel count even in a single image. However, finding analogous images for pre-training in the transfer learning method may not be feasible due to the uncommon nature of the problem in this study. On the other hand, leveraging Fresnel equations to artificially generate images offers a more precise and reliable means to mitigate this dependency. Data generation can be swiftly accomplished within seconds with the aid of an algorithm capable of generating realistic 2D material images using mathematical expressions. Subsequently, these artificially generated data serve as the foundation for pre-training of the CNN algorithm. Then, the limited real optical microscope images can be used to fine-tune the parameters in the CNN. In this study, the Fresnel equations, elaborated upon in the subsequent section, were initially employed to generate artificial images based on the reflection’s light intensity values from various regions within an image, which were then utilized in the pre-training phase.

Fresnel equations

In a brightfield optical microscope image, structures become visible due to the contrast in reflected light from various regions on the examined samples. When examining 2D materials with this type of microscope, they become visible due to the contrast (Eq. (1)) between the material and the substrate on which they are grown (Fig. 1), enabling their analysis.

As shown in Fig. 1, there are two distinct regions, with region A containing an additional MoS₂ layer on SiO₂ and Si layers. For the sake of readability, the layers of air, MoS₂, SiO₂ and Si are indexed as 0, 1, 2 and 3, respectively. A SiO₂ layer of 300 nm and Si are commonly chosen as a substrate due the high contrast observed with the 2D material (region A) compared to the absence of the material (region B) (Ciregan, Meier & Schmidhuber, 2012). The contrast (C) for these regions is as follows: (1)${\displaystyle C=\frac{I_A-I_B}{I_A+I_B}}$

where I_A and I_B are the light intensities in region A and region B, respectively. The intensities can be expressed as in Eqs. (2) and (3).

(2)${\displaystyle I_A=\left|{\overline{r}}_A{r}_A\right|}$ (3)${\displaystyle I_B=\left|{\overline{r}}_B{r}_B\right|}$

where, r_A and r_B are the reflection Fresnel coefficients for the related regions and can be calculated using the refractive indices and phase shifts (Blake et al., 2007; Zhang et al., 2021), as given in Eqs. (4) and (5).

(4)${\displaystyle r_A=\frac{r_{01}{e}^{i\alpha }+r_{12}{e}^{-i\beta }+r_{23}{e}^{-i\alpha }+r_{01}{r}_{12}{r}_{23}{e}^{i\beta }}{e^{i\alpha }+r_{01}{r}_{12}{e}^{-i\beta }+r_{01}{r}_{23}{e}^{-i\alpha }+r_{12}{r}_{23}{e}^{i\beta }}}$ (5)${\displaystyle r_B=\frac{r_{01}-r_{12}{e}^{-i2{\Phi }_1}}{1+r_{01}{r}_{12}{e}^{-i2{\Phi }_1}}}$

where α = Φ₁ + Φ₂ and β = Φ₁ − Φ₂.

Fresnel coefficients (r_ij) are functions of refractive indices (n_i and n_j), where layer i is above layer j, as given in Eq. (6). Specifically, there are three coefficients: r₀₁, r₁₂, and r₂₃. These coefficients represent the relative refractive indices between air and MoS₂, MoS₂ and SiO₂, and SiO₂ and Si, respectively. (6)${\displaystyle r_{ij}=\frac{\left(n_i-n_j\right)}{\left(n_i+n_j\right)}}$

Due to the varying thickness of different layers, a phase shift Φ_i occurs in each layer during the propagation of light in each medium. This phase shift is defined by Eq. (7): (7)${\displaystyle {\Phi }_i=\frac{2\pi n_i{d}_i}{\lambda }}$

where n_i, d_i and λ represent the refractive index, the thickness of the medium i, and the wavelength of the light, respectively.

As a result, when a stack of layers made of different materials is observed under brightfield optical microscope lighting, varying intensities will be obtained, as illustrated in Fig. 1. The upper layers of region A and B consist of 2D material (MoS₂) and SiO₂, respectively. The thickness of SiO₂ is chosen to be 300 nm, providing high contrast, and used as the standard thickness in the growth experiments (Blake et al., 2007). The light propagates through different materials, taking paths of different lengths, which results in a phase shift. Consequently, regions A and B will exhibit different total reflection intensity values as defined in Eqs. (4) and (5). The red, green, and blue channels of an RGB image are obtained by using the wavelength corresponding to these colors. These RGB intensity values can then be used to construct the artificial images of a sample, analogous to real images produced experimentally.

In optical microscopy, tungsten-filament light bulbs, white LEDs, xenon, and mercury lamps can be used as white light sources for illuminating the specimen. These artificial sources combine appropriate amounts of red, blue, and green light to generate white light. While the spectral distribution curves vary for each of these sources, a common characteristic among them is the broad spectral range, particularly in the green and red parts (Tawfik, Tonnellier & Sansom, 2018). Therefore, to achieve this effect and align with the typical spectra of various white light sources, the intensities of red, green, and blue are adjusted accordingly. Various color filters can be used to make images more visible in microscopes. During the pre-training phase, generating images by manipulating intensity across different wavelengths, as opposed to images produced within a specific spectrum of light, contributes to increased diversity in the dataset. This, in turn, mitigates overfitting during the pre-learning phase by preventing algorithms from memorizing encountered instances, thereby addressing overfitting challenges in AI algorithms.

Convolutional neural networks

The convolutional neural network is a specialized type of artificial neural network designed specifically for computer vision tasks, such as image classification and object detection. Early studies on CNNs originated from the discovery of simple, complex, and hyper-complex cells in the visual cortex of animals, which play a role in recognizing objects through their eyes (Hubel & Wiesel, 1959; Hubel & Wiesel, 1968). Based on these studies, the neocognitron was introduced, describing the layers of a CNN to imitate such complex and simple cells (Fukushima, 1980). To identify patterns, CNN employs mathematical operations from linear algebra, particularly matrix multiplications, demanding substantial computational power. This computational complexity initially hindered the immediate adoption of CNNs and other neural network techniques following their invention. However, advancements in computer technology have paved the way for CNNs to address various challenges through diverse architectures in numerous fields (Alzubaidi et al., 2021), including image processing (Schmidhuber, Meier & Ciresan, 2012), video analysis (Ji et al., 2012), medical image analysis (Tajbakhsh et al., 2016), natural language processing (Collobert & Weston, 2008), and time series analysis (Tsantekidis et al., 2017). Compared to other neural network techniques, CNNs involve fewer computations, as they utilize a shared set of weights (filters or kernels) for different regions within an input matrix, such as images. These filters slide over the input matrix, performing convolution operations to extract features.

Essentially, a CNN architecture comprises a convolution layer, a pooling layer, and a fully connected layer, in addition to the input and output layers (Fig. 2). The distinctive properties of a CNN, distinguishing it from other types of neural networks, are the inclusion of a convolution layer responsible for feature extraction from an image and a pooling layer that reduces the size of the input following the convolution operation. These layers can be applied iteratively, capturing low-level features in earlier layers and high-level features in subsequent ones, with the information extracted at each level progressing through the layers.

A CNN is trained using a labeled dataset. In each iteration, forward operations are conducted through the layers, resulting in an output vector. Subsequently, an error is calculated using the desired outputs provided with the dataset. Optimization techniques are then applied to adjust the parameters (filter weights, biases) of the network through backward operations. This process aims to reduce the total error at the end of the subsequent forward operations. The training concludes when the error falls below a predetermined level. At this point, the network possesses optimal parameters and can be tested on unseen data to evaluate its success. The functions of typical layers of a CNN are explained in the following.

Input layer

The typical input for a CNN is an image, where the values of this image are organized into a matrix, creating the input layer. An RGB image, for instance, can be represented as a matrix with dimensions M × N × 3, where M is the height, N is the width of the image, and 3 represents the depth for the red, green, and blue channels. Normalizing the elements of the input matrix at the beginning of the algorithm facilitates the rapid discovery of optimized parameters during training.

Convolution layer

The key layer in a CNN structure is the convolution layer, responsible for performing the convolution operation on the input matrix using a set number of filters. Each filter is designed to detect a distinct feature, mapping it to the next layer. This process entails sliding the filter across the image and applying it to different areas, essentially performing a dot product of the receptive field and the filter at each step. The step size (stride) can be any positive integer. For an M × N × d size image, an m ×n × d size filter, and a stride of 1, the convolution layer’s output will generate a new matrix with dimensions (M−m +1) × (N−n +1). The algorithm may involve the application of multiple filters, and if k number of filters are applied to the image, the output of this convolution layer will be a tensor of size (M −m +1) ×(N −n +1) ×k. The convolution equation is provided below: (8)${\displaystyle C_{i,j}^k=\sum _{c=1}^d\sum _{b=1}^n\sum _{a=1}^m{I}_{\left(i+a-1\right),\left(j+b-1\right),c}{f}_{a,b,c}^k}$

where C denotes the feature map, which is the outcome of the convolution between I and f. I is the input matrix, f is the filter matrix, m is the height and n is the width of the filter. d is the depth of both the input matrix and filter. As an example, the outcome of convolving an image with a Sobel filter, designed to detect horizontal edges, is illustrated in Fig. 3. As mentioned earlier, a CNN incorporates numerous filters, each responsible for extracting distinct low-level or high-level features.

Pooling layer

The result of a convolution operation is still a matrix, with the data number being the product of their height and width. The pooling layer aims to reduce the feature map’s size, a process also known as sub-sampling. This not only enhances the algorithm’s speed by reducing the number of parameters but also offers advantages in preventing overfitting and promoting network generalization. Pooling is executed by filters that operate differently from convolution filters, depending on their types. Various pooling operations exist, such as max pooling, average pooling, and global pooling. Maximum and average pooling are the most common, where the maximum or average value of a defined sub-matrix is transferred to the next layer, respectively. Importantly, it should be noted that the depth of the input matrix remains unchanged after pooling operations. An image before and after the max pooling operation is illustrated in Fig. 4.

Fully connected layer

The subsequent layer following multiple convolution and pooling layers in a CNN architecture is the fully connected layer. Essentially, it constitutes a multi-layer neural network (Rumelhart, Hinton & Williams, 1985) that receives input from the last pooling layer. However, the output of the pooling layer is not a vector and must be flattened (vectorized) before reaching this layer. While the flattening operation is sometimes considered an additional layer, it essentially involves rearranging the matrix into a vector. The number of hidden layers can vary in a fully connected layer. The transformation of information from the flattened input to the output follows the same principles as a multi-layer network. The output of each layer is computed using the equations below:

(9)${\displaystyle h_l^{in}=W_l{h}_{l-1}^{out}+B_l}$ (10)${\displaystyle y_l=\sigma \left(h_l^{in}\right)}$

where $h_{l-1}^{out}$ is the output vector of the previous hidden layer or the flattened vector, W_l is the weight matrix between the current and previous layer, B_l is the bias vector related with each neuron on the current layer, $h_l^{in}$ is the activation potential and y_l is the output vector of the current layer. If the next layer is not the output layer, y_lwill serve as the $h_l^{out}$ for the next layer. σ represents the activation function for each neuron, introducing nonlinearity to the network.

Output layer

This layer is responsible for classification or decision-making and positioned at the end of the structure. In the case of a multi-class or multi-label classification problem, the number of neurons in the output layer corresponds to the number of classes/labels involved. Typically, for mutually exclusive classes, a softmax transfer function serves as the activation. However, in scenarios involving only two classes, such as in this study, a sigmoid transfer function is preferred. This function produces output values between 0 and 1 for each output, allowing for the selection of a threshold value to convert the outputs into either 0 (negative) or 1 (positive).