CrossAlignNet: a self-supervised feature learning framework for 3D point cloud understanding

Global semantic alignment

In the global semantic alignment task, a decoder $Decode{r}_{pc}^f$ is used to extract semantic information from the point cloud features obtained by the encoder. Then the global semantic features of the visible point cloud patches are generated and mapped to the semantic space shared with the image branch by the feature prediction head $F_{pc}$. For the image branch, we only use the image projection head $Pr{o}_{img}$ to extract the global semantic feature of the image. In this case, the decoder $Decode{r}_{pc}^f$ for the feature prediction task is structured similarly to the encoder, but with fewer Tranformer blocks. The feature prediction header $F_{pc}$ has the same structure as the image projection header $Pr{o}_{img}$, both consisting of two fully connected layers, and we use batch normalization and nonlinear activation only after the first layer.

Specifically, the visible point cloud tokens $E_{pc}^{vis}$ and the learnable point cloud mask tokens $E_{pc}^{mask}$ are concatenated and fed into $Decode{r}_{pc}^f$. The output is then passed through the feature prediction head to obtain the predicted feature vector $F_{pc}$. Simultaneously, the image tokens are mapped to the image feature vector $F_{img}$ via the projection head $Pr{o}_{img}$. This process can be formally expressed as:

(5) $$F_{pre}=Pr{e}_f(Decode{r}_{pc}^f(E_{pc}^{vis},E_{pc}^{mask}))$$

(6) $$F_{img}=Pr{o}_{img}(concat(E_{img}^{vis},E_{img}^{mask})).$$

According to the idea of contrastive learning, point cloud and image data under the same viewpoint can be treated as positive sample pairs, and point cloud and image data under different viewpoints can be treated as negative sample pairs. For each sample pair ( $F_{pc}^i$, $F_{img}^i$), a contrastive learning task can be set up, and its loss function is calculated as follows:

(7) $$l(i,F_{pre},F_{img})=-\log \frac{e^{⟨F_{pre}^i,F_{img}^i⟩}/\tau }{{\sum }_{k\ne i}^N{e}^{⟨F_{pre}^i,F_{img}^K⟩}/\tau +{\sum }_i^N{e}^{⟨F_{pre}^i,F_{img}^K⟩}/\tau }$$

(8) $$Los{s}_f=\frac{1}{2N}\sum _{i=1}^{N}l(i,F_{pre},F_{img})+l(i,F_{img},F_{pre}).$$

Local mask reconstruction

This task achieves fine local geometric reconstruction through a cross-modal feature fusion mechanism. The decoder $Decode{r}_{pc}^{rec}$ is composed of cross-attention and self-attention layers, while the point cloud prediction head $Pr{e}_{pc}$ consists of a simple fully connected layer.

Specifically, the visible point cloud tokens $E_{pc}^{vis}$, the masked point cloud tokens $E_{pc}^{mask}$, and the visible image tokens $E_{img}^{vis}$ are jointly fed into $Decode{r}_{pc}^{rec}$. After processing by the decoder, the intermediate feature $D_{pre}$ is obtained. This feature is then passed through the prediction head and undergoes dimension reshaping to generate the reconstructed point cloud patches $P_{pre}$. The process can be formally expressed as:

(9) $$D_{pre}=Decode{r}_{pc}^{rec}(concat(E_{pc}^{vis},E_{pc}^{mask}),E_{img}^{vis})$$

(10) $$P_{pre}=Reshape(Pr{e}_{pc}(D_{pre})).$$

Reshape( $\cdot$) operation reconstructs the vector into a $K3$ dimensional coordinate matrix (K denotes the number of points per patch). The reconstruction objective minimizes coordinate errors in masked regions through an enhanced Chamfer Distance metric between predicted patches $P_{pre}\in {\mathbb{R}}^{K\times 3}$ and ground truth patches $P_{gt}\in {\mathbb{R}}^{K\times 3}$:

(11) $$Los{s}_{pc}=\frac{1}{|P_{pre}|}\sum _{x\in P_{pre}}{min}_{y\in P_{gt}}\Vert x-y{\Vert }_2^2+\frac{1}{|P_{gt}|}\sum _{x\in P_{gt}}{min}_{y\in P_{pre}}\Vert x-y{\Vert }_2^2.$$

Synthetic 3D object classification

ModelNet40 (Yi et al., 2016) is a meticulously annotated 3D CAD model dataset comprising 40 categories with a total of $12\text{,}311$ models. The primary challenges of the ModelNet40 dataset stem from significant intra-class shape variations and diverse object poses and orientations. Derived from its synthetic data nature—characterized by absence of noise, well-defined shapes, and pristine backgrounds—this dataset emphasizes evaluating models’ capability to capture precise local geometric and global semantic representations.

Following the standard protocol, we utilize $9\text{,}843$ models for training and $2\text{,}468$ models for testing. To ensure a fair comparison, our method uses only 1,024 points sampled from each model, containing solely coordinate information, as input. For reporting results, we employ the standard voting method (Liu et al., 2019) for final evaluation.

As shown in Table 1, our method achieves $93.2$ accuracy, outperforming all comparable self-supervised methods, narrowing the gap with the state-of-the-art supervised method PointTransformer to merely $0.5$, while significantly surpassing classical supervised models such as PointNet++ and Dynamic Graph Convolutional Neural Network (DGCNN).

Experimental results demonstrate that our pre-training approach effectively learns precise local geometric and global semantic representations. Nevertheless, a performance gap persists when compared to top-tier supervised methods: Pyramid Vision Transformer (PVT) ( $93.6$) and PointTransformer ( $93.7$). We believe this is primarily because both PVT and PointTransformer utilize modified Transformer models, whereas our approach employs the standard Transformer model, and a performance gap still exists.

Real-world 3D object classification

The ScanObjectNN dataset (Qi et al., 2017a) comprises 15 categories of objects scanned from real-world scenes. Its samples inherently present challenges such as object occlusions, background noise, and sensor artifacts. Consequently, models operating on this dataset must demonstrate the ability to accurately extract local geometric features amidst noise and occlusion disturbances, while also performing effective reasoning about the global semantics of complex scenes.

Following Wang et al. (2021), we evaluate three variants: OBJ-BG preserves complete objects with injected background noise; OBJ-ONLY maintains full object clouds with artificial background removal; PB-T50-RS combines background interference with $50$ random object occlusion while applying random spatial transformations.

Table 2 demonstrates that our method achieves state-of-the-art performance on the OBJ-ONLY variant, while attaining competitive results on OBJ-BG ( $0.8$ lower than Inter-MAE) and PB-T50-RS ( $0.6$ below Point-M2AE). These findings indicate that our approach effectively learns both local geometric and global semantic representations, though direct comparisons with Inter-MAE and Point-M2AE reveal context-dependent strengths and limitations.

Compared to Inter-MAE: despite trailing by $0.8$ on OBJ-BG, our method surpasses Inter-MAE by $1.2$ on the most challenging PB-T50-RS variant. Crucially, the performance drop when switching from OBJ-BG to PB-T50-RS is $6.0$ for Inter-MAE but only $4.0$ for our method. This demonstrates that the learned local and global representations from our pre-training approach exhibit superior robustness against spatial transformations and object occlusions/partiality present in PB-T50-RS.

Compared to Point-M2AE: while our method demonstrates marginal improvements on the OBJ-BG ( $+0.1$) and OBJ-ONLY ( $+0.6$) variants, it lags behind by $0.6$ on the most challenging PB-T50-RS variant. This performance gap primarily stems from Point-M2AE’s hierarchical encoder architecture. Its multi-scale feature fusion mechanism excels at capturing local geometric details across scales, a capability critically important for handling complex scenarios characterized by object occlusion and partiality. In contrast, our approach relies on a standard Transformer encoder, which inherently struggles with extracting multi-scale local features essential for this difficult setting.

Few-shot classification

To validate the generalization and transferability of the pretrained representations, we perform fine-tuning with extremely limited labeled samples, forcing the model to exclusively rely on the prior knowledge acquired during pre-training. Following the evaluation protocol in Xu et al. (2022), we construct four few-shot learning tasks (covering different category-sample combinations) on the ModelNet40 dataset (Wu et al., 2015). For each task configuration, we conduct 10 independent experimental trials and report the mean classification accuracy along with its standard deviation.

As shown in Table 3, our method achieves higher average accuracy than all methods under the 5-way 10-shot, 10-way 10-shot, and 10-way 20-shot settings, while demonstrating the smallest variance among all self-supervised methods. In the 5-way 10-shot scenario, our average accuracy is second only to Point-MAE. These experimental results indicate that our pre-training approach learns more generic representations, delivering stronger performance in data-limited scenarios.

Part segmentation

ShapeNetPart (Cheng et al., 2023) is the core benchmark dataset for 3D object part segmentation. Its primary task is to assign a part semantic label to each point within a point cloud data. This dataset encapsulates several key challenges: imbalanced part scales, cross-category semantic ambiguity (geometrically identical structures possessing different semantics across object categories), and blurred part boundaries. These challenges demand robust model capabilities encompassing multi-scale feature capture, context-aware global semantic understanding, and discrimination of local geometric features across diverse objects.

In our experiments, following Xue et al. (2023), we report three key metrics: Intersection over Union (IoU) per category $mIo{U}_{avg}$, mean IoU per instance $mIo{U}_{ins}$, and mean IoU per category.

As shown in Table 4, our method achieves comparable performance to Point-M2AE in $mIo{U}_{avg}$ while outperforming all other comparative methods. For the $mIo{U}_{ins}$, our approach ranks second to Point-M2AE, trailing by a marginal $0.3$ gap. Experimental results demonstrate that our method delivers competitive performance in part segmentation tasks, though there remains explicit room for refinement compared to Point-M2AE.

Compared to Inter-MAE, our approach achieves higher scores on both $mIo{U}_{ins}$ and $mIo{U}_{avg}$. Further analysis of categories exhibiting IoU differences >1% reveals: for e-phone and bag classes, our approach achieves $+6.0$ and $+2.7$ advantages over Inter-MAE respectively, validating its superior capability in discriminating local geometric structures—particularly evident in resolving boundary ambiguity in non-rigid soft-body deformation (bag) and adapting to microscopic components (e-phone). However, a performance gap of $2.4$ is observed in the rocket category, indicating Inter-MAE’s enhanced contour modeling capability for large-scale continuous surfaces through its fusion of unmasked image information. Qualitative results (Fig. 4) corroborate that within regions annotated with red boxes, our method demonstrates significantly higher precision in parsing local geometric structures.

3D object detection

To further evaluate the applicability of our method in real-world scenarios, we conducted experiments on indoor 3D object detection tasks. Following the PiMAE (Chen et al., 2023) approach, we first performed pre-training on the SUN RGB-D (Song, Lichtenberg & Xiao, 2015) dataset and subsequently fine-tuned on the ScanNetV2 (Dai et al., 2017) dataset. When processing SUN RGB-D dataset scenes containing 20,000 points, we employ scaled-down PointNet (Qi et al., 2017a) to extract 2,048 key points.

We replace the point cloud encoder with a three-layer standard Transformer module from 3D End-to-end Transformer Detection (3DETR) (Misra, Girdhar & Joulin, 2021), where each layer features a hidden dimension of 256 and a 4-head multi-head attention mechanism. To maintain dimensional consistency, we reduce the number of attention heads in the decoder from six to four while keeping all other architectural components unchanged. During the fine-tuning phase, we strictly adhered to 3DETR’s (Misra, Girdhar & Joulin, 2021) training protocol.

As shown in Table 5, our approach outperforms PiMAE by 0.7% in $A{P}_{25}$ and 2.3% in $A{P}_{50}$. This improvement, particularly the more significant gain (+2.3%) on the challenging $A{P}_{50}$ metric, clearly demonstrates the superiority of our pre-training strategy. By combining two self-supervised tasks, our approach effectively overcomes the issue of insufficient encoding of global semantic information in PiMAE due to its reliance on a single masked reconstruction task.

Percentage of masks

To investigate the impact of masking ratios on classification accuracy,we conducted experiments with different masking proportions. As indicated in Table 6, our method achieves optimal performance at a 60% masking ratio with $84.5$ classification accuracy. Experimental results demonstrate that both excessively high and low masking ratios lead to performance degradation.

Excessively high masking ratios critically impair point cloud understanding by obliterating essential local geometric structures and disrupting object continuity. This dual degradation prevents models from effectively reasoning about global semantics and reconstructing masked regions. In our method, high masking ratios applied to point clouds necessitate proportionally aggressive masking of images. This inevitably compromises global semantic extraction from the visual modality. Under such conditions, enforcing cross-modal semantic alignment tasks introduces disruptive noise rather than meaningful learning signals.

Conversely, excessively low masking ratios create information overload, fostering lazy learning behaviors where models exploit superficial patterns instead of learning robust representations.

Masking strategy

In this experiment, a fixed random masking ratio of $60$ was applied to the point cloud, while three distinct masking strategies were compared for the image modality: (1) random masking; (2) correspondence masking (the masked image regions correspond to the masked point cloud regions); (3) complementary masking (the masked image regions correspond to the visible point cloud regions).

As shown in Table 7, the complementary masking strategy achieved the best performance with an accuracy of $84.5$, outperforming corresponding masking by $+0.3$ and random masking by $+0.4$. The superiority of complementary masking primarily stems from its synergistic enhancement of the two self-supervised tasks:

(1) Optimization of Local Mask Reconstruction: when local geometric information in point clouds is masked, complementary masking retains 2D features (e.g., textures, edges) in the image corresponding to the masked point cloud regions. This provides critical auxiliary information for the point cloud reconstruction task.

(2) Enhancement of Global Semantic Alignment: under complementary masking, the globally encoded semantic information in the image forms a correspondence with the semantic information of the masked regions in the point cloud. This design forces the model to infer the global semantics of masked regions based solely on visible areas of the point cloud, thereby reinforcing its representational capacity for global semantic features.

Selection of self-supervised tasks

To evaluate the impact of global semantic alignment and local mask reconstruction as self-supervised tasks on representation learning, we compared different task combinations and varying $\lambda$ values on classification accuracy by adjusting the weighting coefficient in Eq. (10). During pre-training, we observed a significant magnitude discrepancy between the local mask reconstruction loss ( $Los{s}_{pc}$) and the global semantic alignment loss ( $Los{s}_f$), with $Los{s}_f$ being approximately 1,000 times larger than $Los{s}_{pc}$. Therefore, setting $\lambda =0.001$ ensured comparable contributions from $los{s}_{pc}$ and $\lambda \ast los{s}_f$ to the total loss magnitude. Experimental results are summarized in Table 8.

Experimental results demonstrate that models using only local mask reconstruction task significantly outperform those relying solely on global semantic alignment task, while combining both tasks yields clear advantages over either single-task configuration. This not only confirms the critical importance of locally learned geometric features for classification accuracy but also reveals complementary benefits in representation learning between the two tasks.

When $\lambda =0.001$, the contributions of the two losses to the total loss reach equilibrium; when $\lambda$ decreases to $0.0001$, the contribution of the global semantic alignment task reduces by a factor of ten, yet the classification accuracy increases by $0.3$ percentage points. This suggests an intrinsic primary-secondary relationship between the two tasks (with local mask reconstruction as primary and global semantic alignment as secondary). By selecting an appropriate $\lambda$ to regulate the contributions of both tasks, the model’s representation learning capability can be effectively enhanced.