3D model retrieval based on interactive attention CNN and multiple features

Introduction

3D modeling is coming into our daily life. It is convenient to obtain sketch, and sketch can describe a 3D model’s shape. It is in line with people’s habits to retrieve 3D models based on a sketch. However, a sketch contains little shape information, and there is a big gap between sketch and 3D model. It causes low accuracy of 3D model retrieval.

3D model retrieval includes text-based retrieval algorithm, content-based retrieval algorithm and sketch-based retrieval algorithm. In text-based retrieval algorithm, 3D model is manually described according to its shape and topological structure. According to user’s description, 3D model which meets design intent is retrieved. In content-based algorithm, 3D model’s information is used for 3D model retrieval. Tangelder & Veltkamp (2008) surveyed the literature on methods for content-based 3D model retrieval, taking into account the applicability to surface models as well as to volume models. It can be divided into three methods including view-based algorithm, voxel-based algorithm and point cloud-based algorithm. In view-based algorithm, 3D model is converted into a series of 2D views. Then, features of 2D view are calculated and compared for 3D model retrieval. In voxel-based algorithm, 3D model is expressed as the distribution of 3D voxel grid. Features of voxel data are extracted for 3D model retrieval. In point cloud-based algorithm, feature descriptors are directly extracted from original point cloud to retrieve 3D models. In sketch-based algorithm, 3D models are usually converted into 2D images from which features are extracted to realize 3D model retrieval. Main contributions of this article are summarized as follows:

• (1) Interactive attention module is embedded into CNN, in which attention weights of two adjacent convolutional layers are merged.

• (2) D1, D2 and D3 are used to describe shape of sketch and 2D view, and gist feature is used to describe spatial structure of sketch and 2D view.

• (3) Semantic feature, shape distribution features and gist feature are combined adaptively to find 3D model similar to sketch.

Materials and Methods

Methods

Retrieve 3D model based on sketch

When 3D model is retrieved based on sketch from model library, it is necessary to extract features from sketch and 3D models. The dimension of sketch is 2, and the dimension of 3D model is 3. The fixed projection algorithm is used in this article. 3D model is fixed to the center of virtual sphere, where virtual camera is placed above it. The model is rotated one circle to render a set of 2D views at 60° every step. In order to reduce texture difference between sketch and 2D views, edge detection algorithm is used to extract contours from sketch and 2D views.

CNN is adopted to extract semantic feature of view and sketch. Meanwhile, gist feature and shape distribution features of view and sketch are also used as global features. Euclid distance is used to calculate the similarity between sketch and 3D model’s view. These three features are combined to compute the similarity between sketch and 3D model. Feature of good discriminative ability is assigned with high weight. Otherwise, it is assigned with low weight. The weighted sum of these three features is used to compute the similarity between sketch and 3D model. Then, similar model is retrieved from 3D model library.

Feature extraction of 3D model and sketch

1. Semantic feature

In this article, CNN is adopted to extract semantic features from 3D model’s views and sketch. The process of extracting semantic features based on CNN is shown in Fig. 1. The above CNN is composed of five convolutional layers, four pooling layers, an attention layer, a fusion layer, a dropout layer, a fully connected layer and an output layer. In order to reduce the influence of image texture on 3D model retrieval, 2D view is rendered as contour. Input layer of convolutional neural network can accept sketch and contour of 2D views. Since fully connected layer requires a fixed input dimension, the size of sketch should be the same with that of 2D view. Output of convolutional layer is shown in formula (1).

(1) $$a^l=f\left(a^{l-1}\ast K^l+b^l\right)$$where, l is the number of layers, K^l is convolution kernel matrix, b^l denotes bias, as asterisk (*) represents convolution operation. Here, f is Relu function. Output of pooling layer is shown in formula (2).

(2) $$a^l=max\left(a^{l-1}\right)$$

Output of fully connected layer is shown in formula (3).

(3) $a^l=f(W^l\ast a^{l-1}+b^l)$where, W^l represents weight matrix in fully connected layer, b^l denotes bias. Interactive attention module is introduced into the framework as shown in Fig. 1. Output from Conv1 is processed by attention to get its attention weight. Output from Conv2 is processed by attention to get its attention weight. These two weights are averaged. The averaged weight is fused with output from Conv2. The result is summed with the max pooling of output from Conv1, which is output to the second max pooling layer. Output from Conv2 is processed with attention weights of output from Conv1 and Conv2. The purpose is to fuse more convolutional layers to improve the quality of semantic features. Interaction attention module is shown in Fig. 2. In order to calculate information quantity in each channel of the first and second convolutional layer, pooling operation with global receptive field is used to compress spatial information. Then, feature map conv_nk is converted into a real number as shown in formula (4). All real numbers are used to construct vector Z.

(4) $$Z_k=F_{sq}\left(con{v_{\mathrm{n}}}_k\right)={\displaystyle \frac{1}{H_n\ast W_n}\sum _{i=1}^{H_n}\sum _{j=1}^{W_n}con{v_{\mathrm{n}}}_k\left(i,j\right)}$$where, conv_nk represents the kth feature map in output of the nth convolutional layer, and its size is W_n * H_n. In order to tune attention weight of each channel adaptively, two fully connected layers are connected after global pooling layer. The first one A_C₁ has K/r neurons. Here, r is scaling ratio. The second one A_FC₂ has K neurons. Attention weight S_kn is computed as shown in formula (5).

(5) $$S_{kn}=F_{ex}\left(Z,W\right)=\sigma \left(g\left(Z,W\right)\right)=\sigma \left(W_2\delta \left(W_{1}Z\right)\right)$$where, δ is Relu function and σ is sigmoid function. S_kn is weight vector of convolutional layer Conn and its dimension is k. W₁, W₂ are respectively weights of A_FC₁ and A_FC₂. The algorithm of computing interaction attention is shown as follows:

Input: feature map con_v1 = (conv₁₁, …, conv_1k) of the first convolutional layer Con₁, feature map conv₂ = (conv₂₁, …, conv_2k) of the second convolutional layer Con₂, feature map a_p of the first pooling layer.

Output: feature map a₂ of the second convolutional layer Con₂.

• ① Calculate vector Z_k of conv_1i (i = 1, 2, …, k) in the first convolutional layer Con₁ according to formula (4).

• ② Calculate attention weight S_k1 of convolutional layer Con₁ according to formula (5).

• ③ Calculate vector Z_k of conv_2i (i = 1, 2, …, k) in the second convolutional layer Con₂ according to formula (4).

• ④ Calculate attention weight S_k2 of convolutional layer Con₂ according to formula (5).

• ⑤ Fuse S_k1 and S_k2 according to formula (6).

(6) $$S_k=Average\left(S_{k1},S_{k2}\right)$$where, Average (.) is average function.

• ⑥ Attention weight S_k is fused with conv₂ and a_p as shown in formula (7).

(7) $a_2=con{v}_2\otimes s_k+a_p$

2. Gist feature

Gist feature is global feature that contains similar shape and spatial structure. The process of extracting gist feature is shown as follows:

(1) Divide sketch or 2D view with size m * n into 4 * 4 blocks. The size of each block is a * b, where a = m/4, b = n/4.

(2) Each block is processed by 32 Gabor filters with four scales and eight directions. Processed features are combined to obtain gist feature as shown in formula (8).

(8) $$G\left(x,y\right)=\underset{32}{\underbrace{cat}}\left(I\left(x,y\right)\ast {g_i}_j\left(x,y\right)\right)$$where, i = 4, j = 8. G(x, y) is gist feature of 32 Gabor filters, and cat() represents concatenation operation. Here, x and y are positions of pixel, I(x, y) denotes the block. At the same time, g_ij(x, y) is filter with the ith scale and the jth direction. Here, * represents convolution operation.

(3) Feature G(x, y) is averaged after each filtering. All average block features are combined into a row vector to obtain final gist block feature.

(4) Features of 16 blocks are combined to form gist feature of 2D view.

3. Shape distribution features

Osada proposes shape distribution for 3D model’s representation which is used to retrieve and classify 3D models (Osada, Funkhouser & Chazelle, 2001). We modify 3D model’s shape descriptors and apply them to 2D views. Points on the boundary of sketch or 2D view are randomly sampled equidistantly, which are collected into Points = {(x₁, y₁), …, (x_i, y_i), …, (x_n, y_n)}. Here, (x_i, y_i) is point coordinates. Extract points from Points and collect them into PD1 = {ai₁, …, ai_k, …, ai_N}. The set of D1 shape distribution feature is {D1_v₁, …, D1_v_i, …, D1_v_Bins}. Here, D1_v_i is statistics in interval (BinsSize * (i−1), BinsSize * i), Bins represents the number of intervals, and BinsSize is interval length. D1_v_i is computed as shown in formula (9).

(9) $$D1\mathrm{\_}{v}_i=\left|\left\{P|dist\left(P,O\right)\in \left(BinSize\ast \left(i-1\right),BinSize\ast i\right),P\in PD1\right\}\right|$$where, BinsSize = max({dist(P, O)|P∈PD1})/N, dist() is Euclidean distance between two points. O is the centroid of sketch or 2D view. Extract point pairs from Points and collect them into PD2 = {(ai₁, bi₁), (ai₂, bi₂), …, (ai_N, bi_N)}. The set of D2 shape distribution feature is {D2_v₁, …, D2_v_i, …, D2_v_Bins}. Here, D2_v_i represents statistics in interval (BinSize * (i−1), BinsSize * i). D2_v_i is calculated as shown in formula (10).

(10) $$D2\mathrm{\_}{v}_i=\left|\left\{P|dist\left(P\right)\in \left(BinSize\ast \left(i-1\right),BinSize\ast i\right),P\in PD2\right\}\right|$$where, BinsSize = max({dist(P)| P∈PD2})/N. Extract point triples from Points and collect them into PD3 = {(ai₁, bi₁, ci₁), (ai₂, bi₂, ci₂), …, (ai_n, bi_n, ci_n)}. The set of D3 shape distribution feature is {D3_v₁, …, D3_v_i, …, D3_v_Bins}. Here, D3_v_i represents statistics in interval (BinSize * (i−1), BinsSize * i). D3_v_i is computed as shown in formula (11).

(11) $$D3\mathrm{\_}{v}_i=\left|\left\{P|\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{s}\mathrm{o}\mathrm{n}\left(P\right)\in \left(BinSize\ast \left(i-1\right),BinSize\ast i\right),P\in PD3\right\}\right|$$where, $BinSize=max(\{\sqrt{herson(P)}|(P)\in PD3\})/N$. Here, herson() represents Helen formula, which is used to compute area of triangle P =(P₁, P₂, P₃) as shown in formulas (12) and (13).

(12) $$herson(P)=herson\left(P_1,P_2,P_3\right)=\sqrt{s(s-a)(s-b)(s-c)}$$

(13) $$s={\displaystyle \frac{1}{2}\ast (a+b+c)}$$where, a = dist (P₁, P₂), b = dist (P₁, P₃), c = dist (P₂, P₃). D1_v_i, D2_v_i and D3_v_i are connected to form shape distribution feature, i = 1, 2, …, Bins.

Similarity fusion and model retrieval

The distance between features is used to measure the similarity between sketch and 2D view. Feature of sketch is denoted as X_s = (x_s1, x_s2, …, x_sn). 2D view’s feature is represented as X_v = (x_v1, x_v2, …, x_vn). Euclidean distance between X_s and X_v is computed as shown in formula (14).

(14) $$D_{Euc}\left(X_s,X_v\right)=|X_s,X_v|$$

Algorithm of calculating the similarity between sketch and 2D view is shown as follows:

Input: sketch A and 2D view v

Output: the similarity between A and v

1. Use the improved CNN to extract X_s1 from A and normalize X_s1.

2. Use the improved CNN to extract X_v1 from v and normalize X_v1.

3. Calculate sim_se (X_s1, X_v1) = 1/D_Euc (X_s1, X_v1) by formula (14).

4. Extract gist feature from A to get X_s2 and normalize X_s2.

5. Extract gist feature from v to get X_v2 and normalize X_v2.

6. Calculate sim_gi (X_s2, X_v2) = 1/D_Euc (X_s2, X_v2) by formula (14).

7. Extract shape distribution feature from A to get X_s3 and normalize X_s3.

8. Extract shape distribution feature from v to get X_v3 and normalize X_v3.

9. Calculate sim_sh (X_s3, X_v3) = 1/D_Euc (X_s3, X_v3) by formula (14).

10. Fuse similarities by formula (15).

(15) $$Sim\left(A,v\right)=w_1\times si{m}_{se}\left(X_{s1},X_{v1}\right)+w_2\times si{m}_{gi}\left(X_{s2},X_{v2}\right)+w_3\times si{m}_{sh}\left(X_{s3},X_{v3}\right)$$where, 0 < w_i < 1, i = 1, 2, 3, w₁ + w₂ + w₃ = 1.

Experimental results and discussion

The proposed method can extract semantic feature, gist feature and shape distribution features from 2D views. If we get 2D views of 3D model, we can use the proposed method to retrieve 3D model according to sketch. This article uses NN, FT, ST, E(F), DCG to evaluate the performance of 3D model retrieval.

The performance of CNN is determined by convolutional layer number and pooling layer number. Convolutional layer number and pooling layer number are respectively set to 3, 4, 5, 6. When convolutional layer number is 3 and pooling layer number is 3, each convolutional layer is followed by a pooling layer. When convolutional layer number is 4 and pooling layer number is 3, each of the first three convolutional layers is followed by a pooling layer. CNN is optimized and tested as shown in Table 2.

From Table 2, we can find that CNN achieves the best and accuracy is 0.964 when convolutional layer number is 5 and pooling layer number is 4. So, the proposed network contains five convolutional layers and four pooling layers.

Here, accuracy of test data is used to evaluate the quality of feature extraction. At the same time, accuracy of test data is also adopted to determine position and way of adding channel attention into CNN.

AT-CNN1 represents classifier where AT (attention module) is only added into the first convolutional layer. AT-CNN2 is classifier where attention module is added between the first convolutional layer and the second one. AT-ALL-CNN denotes classifier where attention module is added into all convolutional layers. ICA-12 represents classifier where interaction weight attention is added between the first convolutional layer and the second one. ICA-23 is classifier where interaction weight channel attention is added between the second convolutional layer and the third one. In order to prevent the overfitting problem, we add dropout into CNN and set different dropout coefficients. Table 3 shows accuracy under different dropout coefficients and different ways of adding channel attention into CNN.

From Table 3, it can be seen that when more attention modules are added into CNN, the lower is accuracy. Channel attention mechanism can focus on some channels. The reason may be that there are few lines in sketch and 2D views, and accuracy will be reduced if information of some channels is ignored. Interaction weight channel attention can reduce information loss to a certain extent and improve the performance of 3D model retrieval. This is because that channel attention is added between two convolutional layers and its weight is considered in max pooling layer. Therefore, interaction weight channel attention CNN is selected to extract semantic features in high level from sketches and 2D views.

Regularization coefficient is set to 0.04. Initial learning rate is set to 1E−4. The number of training iterations is set to 20. A total of 80% of training data are selected as training set and the rest is used as validation set. Figure 3 shows the loss of training set and validation set during the training process. Figure 4 shows accuracy of training set and validation set during the training process. It can be seen that the loss steadily decreases and convergence speed is fast under this learning rate. At the same time, accuracy achieves the best.

Alexnet algorithm uses CNN to extract features. CA algorithm adds channel attention into each convolutional layer to extract semantic features. ICA algorithm adds interactive attention module between the first convolutional layer and the second one to extract features. Hog algorithm extracts histograms of oriented gradients (Lei et al., 2014). LBP algorithm extracts local features (Singh & Agrawal, 2016). Gist algorithm uses gist feature. Shape distribution feature is only adopted in SD algorithm. Through sharing Vgg16 weights, siamese learning network is established. Joint feature mapping of sketches and views is realized. Based on joint feature distribution, the similarity is computed between sketch and 3D model to realize 3D model retrieval based on sketch. MFFR (multiple feature fusion retrieval) algorithm combines features extracted by ICA algorithm, Gist algorithm and SD algorithm.

Gist feature and shape distribution features describe global information of sketch and view. Semantic feature denotes local details of sketch and view. These three features have their own advantages and disadvantages. We fuse semantic feature, gist feature and shape distribution features to describe sketch and 3D model adequately. The purpose is to make them complement each other for getting the best performance of 3D model retrieval. Discriminative ability of feature can be evaluated with NN, FT, ST, E(F) and DCG when it is applied to the retrieval process.

We need determine optimal values of w₁, w₂, w₃ in formula (15) through experiments to make MFFR algorithm achieve the best. We set different values to parameters w₁, w₂, w₃, where the sum of w₁, w₂, w₃ is 1. Under different values of parameters w₁, w₂, w₃, MFFR algorithm is applied to search 3D models from test data which are the most similar with these nine models. NN, FT, ST, E(F) and DCG are used to evaluate the performance of MFFR algorithm as shown in Table 4.

From Table 4, MFFR algorithm all achieves the best at NN, ST, E(F) and DCG when w₁ = 0.6, w₂ = 0.3, w₃ = 0.1. FT of MFFR algorithm is 0.560, which is only 0.04 lower than optimal one. So, we set 0.6, 0.3, 0.1 respectively to w₁, w₂, w₃ in formula (15). We can find that the weight of semantic feature is 0.6, the weight of gist feature is 0.3, and the weight of shape distribution feature is 0.1. The weight of semantic feature is the biggest. This shows that semantic feature has more influence on the performance of 3D model retrieval than gist feature and shape distribution feature.

MFFR algorithm is compared with Alexnet algorithm, CA algorithm, ICA algorithm, Gist algorithm, HOG algorithm, LBP algorithm, and SD algorithm, Vgg16+Siamese. Experimental results are shown in Table 5.

Experimental results show that when one retrieval result is returned, MFFR algorithm achieves the highest precision and precision is 0.889, followed by Vgg16+Siamese, Alexnet algorithm and CA algorithm. Precision of Vgg16+Siamese is 0.778. Precisions of Alexnet and CA algorithm are all 0.667. ICA algorithm ranks the fourth, and precision is 0.556. When evaluation criteria are FT and ST, ICA algorithm has the highest recalls, which are 0.577 and 0.766 respectively. The second is MFFR algorithm, which are 0.544 and 0.756 respectively. When evaluation criteria is E(F), MFFR algorithm and ICA algorithm achieve the highest precision and precisions are both 0.533. HOG algorithm, LBP algorithm, Gist algorithm and SD algorithm have low performance. 3D model retrieval method based on neural network is better than traditional algorithms. ICA is better than CA at FT, ST and E(F). CA is only better than ICA at NN. When DCG is used as evaluation criteria, MFFR achieves the best and its DCG is 3.939, followed by Vgg16+Siamese, Alexnet algorithm and ICA algorithm. CA algorithm ranks the fourth. It shows that interactive attention model can focus on more useful information in process of extracting features and reduce the loss of information. The effect of feature extraction is good. However, the improved CNN still has certain limitations due to the lack of training data. The retrieval effect does not meet the expected standard when one result is only returned. MFFR combines semantic feature, shape distribution features and gist feature to retrieve 3D model, and its performance at NN criterion is improved. Compared with ICA, the performance of MFFR reduces a little under FT and ST criterion.

SD algorithm uses shape distribution features to describe sketch and views. Gist algorithm uses gist feature to describe sketch and view. Gist feature and shape distribution feature describe global information of sketch and view. Alexnet algorithm uses CNN to extract features from sketch and view. CA algorithm adds channel attention into each convolutional layer to extract semantic features from sketch and view. ICA algorithm adds interactive attention between the first convolutional layer and the second one to extract features from sketch and view. They can get local information of sketch and view. Vgg16+Siamese uses siamese network to share the weights of two Vgg16. Joint feature mapping of sketches and views is built for retrieving 3D model based on sketch. From Table 5, we can find that the retrieval method based on local information achieves better than that based on global information. This is because that local information denotes details of sketch and view, which can distinguish sketch or view from others better. But, gist feature and shape distribution feature have global information. When global and local information are applied to 3D model retrieval, the performance is improved. So, MFFR achieves the best.

Figure 5 describes precisions of nine algorithms under different recall values.

In these nine algorithms, precision decreases gradually as recall increases. MFFR has the highest precision when recall is between 0 and 0.2. When recall is 0.1, Vgg16+Siamese has higher precision, which is better than Alexnet, ICA and CA. The reason is that Siamese network is used to share weights of two Vgg16 for building joint feature mapping of sketches and views better. SD algorithm is higher than HOG algorithm and LBP algorithm. This is because that shape distribution feature can give more accurate description of sketch and view than HOG and LBP features. When recall is 0.2, ICA is better than Alexnet and CA. When recall is between 0.3 and 1, ICA has the highest precision and is followed by MFFR. SD algorithm, HOG algorithm, LBP algorithm and Gist algorithm have lower precision. Experimental results show that deep learning algorithms are better than traditional ones at precision. When semantic feature, shape distribution features and gist feature are fused, information of sketch and view can be described accurately.

Figure 6 shows the performance of these nine algorithms at E(F) for four categories of 3D models. It can be seen that the performance of these nine algorithms is significantly different under different categories. For some categories such as chair, ICA is better than other algorithms. For some categories such as person, the performance of ICA does not reach the expected results. But, Gist algorithm has the best performance. For some complex categories, neural network is not trained sufficiently and does not perform well in process of extracting features. Traditional method of feature extraction does not require training process and has relatively good effect. Therefore, it is further proved that MFFR algorithm can improve the performance of 3D model to a certain extent.

It can be found that Alexnet algorithm, CA algorithm, ICA algorithm, Vgg16+Siamese and MFFR algorithm achieve better than Gist algorithm and SD algorithm. This is because that deep learning algorithm has a good effect in process of extracting 3D model’s features. It can be found that ICA algorithm and MFFR algorithm achieve better than Alexnet algorithm and CA algorithm. This is because that attention mechanism is added into CNN and uses adaptive weight to reduce the loss of information. It greatly improves the performance of retrieving some kinds of models. MFFR algorithm fuses multiple features and the performance of 3D model is improved at criterion NN.

3D model can be denoted as its 2D views. The number of 2D views has influence on the expression of 3D model. More views can describe details of 3D model. But, it will introduce noises into the process of training the proposed network. At the same time, time and space complexity will increase greatly. 450 3D models are used to testify accuracy of the proposed network under different view number. We set 3, 4, 6, 8, 9, 12, 15, 18 to view number respectively. 2D views of these 450 3D models are extracted under different view number. Then, they are used to train and test the proposed network. Accuracies of the proposed network under different view number are shown in Table 6.

From Table 6, it can be seen that accuracy of the proposed network increases with the increase of view number. When view number is 6, the classification accuracy achieves 0.9832. When we use 18 views to express 3D model, accuracy is 0.9910. accuracy is only increased by 0.79%, but the number of views increases three times. The complexity of training the network will increase greatly. When four views and eight views are adopted to express 3D model, accuracy is 0.8864 and 0.9774 respectively. So, we use six views to express 3D model in this article.

OpenSketch dataset (Gryaditskaya et al., 2019) is divided into training data and test data. There are nine categories including bumps, flange, hairdryer, house, mixer, mouse, potato_chip, shampoo_bottle, tubes, vacuum_cleaner, waffle_iron, wobble_surface. Training data is used to optimize MFFR algorithm. Test data is adopted to testify its performance and experimental results are shown in Table 7. A total of 11 categories are selected from MCB dataset (Kim et al., 2020) including Bushes, Clamps, Fan, Lever, Lockwashers, Pulleys, Split_pins, Springs, Switch, Threaded_rods, Wheel. Training data is used to optimize MFFR algorithm. Test data is adopted to testify its performance and experimental results are shown in Table 7. Nine categories are selected from CADNET dataset (Manda, Bhaskare & Muthuganapathy, 2021) including Bearing_Blocks, Contact_Switches, Curved_Housings, Gear_like_Parts, Handles, Intersecting_Pipes, Oil_Pans, Posts, Rocker_Arms. Training data is used to optimize MFFR algorithm. Test data is adopted to testify its performance and experimental results are shown in Table 7.

From Table 7, we can find that the performance of MFFR algorithm on OpenSketch dataset, CADNET dataset, MCB dataset is all good at NN criterion. MFFR algorithm achieves the best on CADNET dataset at NN, FT, ST, E(F) and DCG.

Conclusions

This article proposes a method of 3D model retrieval based on interactive attention CNN and multiple features. 3D model is projected into a set of 2D views, and edge detection algorithm is used to extract line graph of 2D view. CNN with interactive attention is used to extract semantic features from sketches and 2D views. 3D shape distribution descriptors are improved to be suitable for 2D views. Then, shape distribution features and gist feature are extracted from 2D views. Semantic features, shape distribution features and gist feature are merged to retrieve 3D model similar to sketch. Compared with the retrieval methods based on single features, the proposed one can improve the performance of 3D model retrieval.

The 3D model is rotated one circle to extract a set of 2D views. We represent a 3D model with its 2D views. In fact, contours of 2D views have important influence on shape of 3D model. Internal part of 2D view has little influence on shape expression of 3D model. So, we select 2D views’ contours to train the proposed network. At the same time, contours are combined with gist feature and shape distribution features to express 3D model further.

The novelty of the proposed method is that interactive attention CNN is adopted to extract features from 2D views of 3D model. Gist algorithm and SD algorithm are used to extract global features. Euclidean distance is adopted to calculate the similarity of semantic feature, the similarity of gist feature and the similarity of shape distribution features between sketch and 2D view. Then, the weighted sum of three similarities is used to compute the similarity between sketch and 2D views for retrieving 3D model. It can combine global and local information for retrieving 3D model.

Supplemental Information