point2vec

PCA projection of the learned representation of the 3D point cloud.

Abstract

We present point2vec, a self-supervised method for learning point cloud representations through masked prediction of a teacher network's latent features. Building on data2vec, we investigate a limitation of applying this approach to point clouds: positional information associated with masked patches reveals the overall object shape to the student, weakening the learning task even under heavy masking. point2vec addresses this issue by passing only visible patches to the student encoder and introducing a shallow decoder to predict representations for masked patches. The teacher processes the complete point cloud to provide contextualized training targets. These targets capture global context through the teacher's self-attention, providing supervision beyond the local geometry of individual patches. This design conceals masked-patch positions from the student and reduces its computational cost during pre-training. Our ablations show that the decoder alone provides only part of the improvement; concealing positional information is key. After pre-training on ShapeNet, point2vec outperforms the evaluated self-supervised baselines on shape classification on ModelNet40 and ScanObjectNN and few-shot classification on ModelNet40, while achieving competitive part segmentation results on ShapeNetPart.

Visualization of Learned Representations

PCA projections of point cloud representations from random initialization, data2vec–pc, and point2vec. — We use PCA to project the learned representations into RGB space. Both a random initialization and data2vec–pc pre-training show a fairly strong positional bias, whereas point2vec exhibits a stronger semantic grouping without being trained on downstream dense prediction tasks.

BibTeX

point2vec.bib

@inproceedings{knaebel2023point2vec,
    title     = {{point2vec} for Self-Supervised Representation Learning on Point Clouds},
    author    = {Knaebel, Karim and Schult, Jonas and Hermans, Alexander and Leibe, Bastian},
    year      = 2023,
    booktitle = {German Conference on Pattern Recognition (GCPR)},
}