3D Object Recognition

2D image recognition answers one question: what is in this picture? 3D object recognition answers three: what the object is, where it sits in physical space (x, y, z in meters), and how it is oriented. The output is no longer a label or a flat rectangle. It is a labeled 3D box, a labeled set of points, or a full shape.

That extra geometry is why self-driving cars, warehouse robots, AR headsets and drones depend on it. A robot arm cannot grasp a mug from a 2D box. It needs the mug's distance, size and handle direction.

Images have one natural format: a grid of pixels. 3D has several, and the choice of representation drives the choice of model. Point clouds are the closest to raw sensor data and convert easily into the other forms, which is why most modern work starts there.

Every 3D model inherits the strengths and blind spots of its sensor. Knowing the sensor tells you what errors to expect before you train anything.

A point cloud is a set. The same chair can arrive as points in any order, and the answer must not change. A normal neural network, which reads inputs in a fixed order, breaks this rule. Before 2017 researchers dodged the issue by converting points to voxels or images, losing detail and burning memory.

PointNet (Qi et al., CVPR 2017) solved it directly. It runs the same small network (a shared MLP) on every point independently, then combines all point features with a symmetric function: max pooling. Max does not care about order, so the global feature is identical however the points are shuffled. A small learned transform (T-Net) also aligns the input to reduce sensitivity to rotation.

PointNet++ added local neighborhoods: sample centers, group nearby points, apply PointNet to each group, repeat. That gives the hierarchy of local-to-global features CNNs get from stacked convolutions.

Detection finds every object in a whole scene, which can hold over 100,000 LiDAR points. Three families dominate:

Predictions are scored by Intersection over Union (IoU): the overlap volume divided by the combined volume. On KITTI, a predicted car only counts if its 3D IoU with the ground truth is at least 0.7. Small errors in height or position punish 3D IoU much more than 2D IoU.

Self-attention is naturally permutation-invariant, so transformers fit point sets well. Point Transformer (V1 to V3) became the leading backbone for 3D segmentation. Point Transformer V3 (CVPR 2024 oral) traded complex neighbor search for serialization: it orders points along space-filling curves so attention runs fast on large scenes.

The 2025 shift is self-supervised pretraining, the same move DINO made for images. Meta's Sonata (CVPR 2025 Highlight) pretrains PTv3 on about 140,000 unlabeled scenes. Its features are strong enough that a simple linear probe reaches 72.5% mIoU on ScanNet segmentation. Follow-up work such as Concerto distills 2D image foundation models into the 3D encoder.

A parallel trend is open-vocabulary 3D: lifting CLIP-style language features onto points so you can query a scan for "fire extinguisher" without training a class for it.

Sometimes there is no LiDAR, only photos. Two techniques turn photos into 3D scenes that recognition can then run on.

NeRF (Neural Radiance Fields) trains a network to return color and density for any 3D point and viewing direction. It renders by ray marching, which is high quality but slow.

3D Gaussian Splatting (Kerbl et al., 2023) represents the scene as millions of small 3D Gaussian blobs with position, shape, opacity and view-dependent color. They are rasterized, not ray-traced, so scenes render in real time. Recognition research now attaches semantic features to each Gaussian, giving segmentable, queryable scenes.

3D scans capture more than objects. Home and office scans record floor plans, belongings and sometimes people. LiDAR street data can reveal license plates through camera fusion and identify individuals by gait or body shape. Treat scan data as personal data where GDPR or similar laws apply.