A map should know what it does not know

Proposed experiment

A concrete pipeline to test

Begin with timestamped LiDAR, IMU, and camera streams. Validate clock alignment, rigid transforms, lens calibration, and motion compensation. Feed the LiDAR–IMU frontend into local submaps. Keep a separate pose-graph process for candidate loop closures, with geometric verification and a robust rejection rule before optimization.

For semantics, project only visible, geometrically consistent image observations into a submap. Accumulate evidence over viewpoints rather than stamping a label from one frame onto every projected point. Store the observation time, source image, class scores, visibility estimate, and pose uncertainty with each update.

Use a local vocabulary suited to the site: pillar, fence, pathway, equipment, person, and unknown. Similarity to a text label is not a calibrated probability. Select acceptance thresholds using a separate validation set; reserve an unknown state for ambiguous or out-of-domain observations. Human detections should be handled as dynamic observations rather than permanently fused into a static asset map.

Our proposed interface passes label suggestions to the viewer while retaining geometric evidence independently. A user can inspect a point or asset and see why it received a label, when it was last observed, and whether the label is tentative. The viewer should not render low-confidence labels with the same certainty as repeatedly supported ones.

Proposed experiment

The benchmark matrix

Build evaluation sequences that vary one difficulty at a time before combining them: low texture, repetitive structure, changing illumination, moving people, fast rotation, and weak geometric constraints. A corridor with repeated pillars is a different challenge from an open field. Train/validation/test splits should separate physical sites or capture sessions, not adjacent frames.

For geometry, measure trajectory error under a declared alignment convention, relative drift, map completeness against a reference, loop-closure precision, and failure-to-relocalize rate. For semantics, measure class-level precision and recall, unknown detection, temporal stability, and asset instance consistency. Small labeled evaluation sets are still needed even if deployment labeling is automatic.

Log end-to-end latency by stage, dropped frames, peak memory, and measured power on the intended compute. A faster semantic model is not a system improvement if synchronization or map fusion becomes the bottleneck. Publish distributions and difficult sequences, not only an attractive map from a successful run.