at-a-glance TL;DR · every block expanded elsewhere (links inline) · surface-coverage recall@K=1 · 5 seeds · n_cand=20 · hybrid top_k=12 · mean ± SE across seeds
compute_mep_recall divided captured instances by the instances visible in the partial cloud, using the partial scan as its own ground truth. So observing a handful of instances scored near 1.0 and oracle inflated 5-50×. We re-anchored the denominator to the scene-full GT (scene_instance_class): oracle drops to 0.4-0.5 on test_locked and 0.13-0.18 on held_out. A survey of the NBV literature found the field standard already normalizes against complete-scene GT, so we make no claim that prior work is affected. This was a bug in our own pipeline.
surface-coverage recall@K=1, hybrid policy · mean ± SE, 5 seeds · best non-oracle per column in green · oracle = myopic Δ ceiling
| policy | test_locked 3 residential |
held_out 7 GNI / heatpump |
ifc-bench OOD 10 scenes, 3 buildings |
|---|---|---|---|
| greedy_coverage classical | 0.563 ±.004 | 0.062 ±.002 | 0.037 ±.007 |
| octomap_ig Bircher 2016 · the floor | 0.500 ±.006 | 0.054 ±.002 | 0.026 ±.004 |
| greedy_lookahead_1 2-step, true extractor | 0.494 ±.017 | 0.129 ±.010 | 0.070 ±.006 |
| v4 mep_recall · SHIPPED | 0.541 ±.006 | 0.147 ±.018 | 0.052 ±.005 |
| v6 pure pairwise rank | 0.507 ±.016 | 0.129 ±.009 | 0.059 ±.008 |
| v7_attn cross-attention | 0.496 ±.012 | 0.088 ±.008 | 0.067 ±.011 |
| v7_realsynth realism-fixed synth | 0.486 ±.019 | 0.094 ±.009 | 0.071 ±.003 |
| v7_k12 K=2 combined · failed | 0.474 ±.016 | 0.086 ±.011 | 0.054 ±.005 |
| oracle myopic Δ ceiling | 0.572 ±.003 | 0.185 ±.017 | 0.093 ±.006 |
Best non-oracle in each column where directly comparable: v6 on test_locked (the v4 0.541 row is the shipped hybrid, scored separately), v7_realsynth on held_out and OOD. The v4 hybrid is the shipped operating point: 0.541 test_locked, 0.147 held_out, 0.052 OOD. Oracle is the myopic Δ ceiling, greyed.
paired per-seed differences (scene-mean A − scene-mean B), then averaged · SE has 4 dof · surface coverage
| comparison | test_locked | held_out | ifc-bench OOD |
|---|---|---|---|
| hybrid − octomap_ig the headline | +0.041 (+5.4σ) | +0.094 (+5.0σ) | +0.026 (+5.6σ) |
| hybrid − greedy_lookahead_1 | +0.047 (+2.0σ) | +0.018 (+1.1σ) | −0.018 (−3.6σ) |
| learned − greedy_coverage ranker alone | −0.075 (−7.5σ) | +0.028 (+2.5σ) | −0.016 |
| v7_realsynth − v4 (hybrid) | −0.001 (tied) | −0.001 (tied) | +0.009 (+2.6σ) |
| v7_k12 − v7_realsynth (hybrid) | −0.002 (tied) | −0.003 (tied) | −0.018 (−3.0σ) |
peak ρ between learned score and true Δ-recall target on the train set · v7_realsynth leads · all weak (the target is hard)
Three target-engineering bets died. v5 (surface-coverage target) regressed below v4. v7_attn (a 2.6M-param cross-attention head, ~10× the parameters) bought no gain (ρ tied v6 at 0.13, in-distribution hybrid 0.496 ≤ v6's 0.507). v7_k12 hit a mathematically unavoidable K=2 degeneracy: the pair-coverage target g_star_k2 has exactly zero top-2 gap, since the pair (i,j)=(j,i) symmetry ties the two members and the gradient vanishes. The combined K2+0.1·K1 fix broke ties (10/10 groups) but eval still regressed: ρ 0.10, ifc-bench hybrid 0.054, losing to v4's 0.052 and v7_realsynth's 0.071. Topology-following also died. The most natural MEP-specific prior, that the system is a connected graph of pipe and duct runs you can follow, does not help: a perfect ground-truth-graph frontier policy scored below greedy_coverage and below random on both surface coverage and MEP-Recall (frontier_oracle − greedy_coverage = −5.66σ on MEP-Recall@K=4), and committing harder to graph adjacency made it strictly worse (the deployable version beat the GT-graph version). Run-following is a wrong objective, not a noisy signal, which is exactly why generic coverage-greedy is so hard to beat here. Earlier negatives also documented: DAgger, Sonata encoder, SuperDec primitive head, edge-recall objective, 3-seed ensemble. The throughline: the model already extracts what it can from the scene geometry it is given; mismatched targets and even the obvious structural prior do not help. → full ledger with dates, motivation, and what broke
Eval protocol held fixed across the study: surface-coverage recall@K=1 · 5 seeds · n_cand=20 · hybrid top_k=12 · mean ± SE across seeds. Splits: test_locked (3 residential) · held_out (7 GNI / heatpump) · ifc-bench OOD (10 scenes across duplex_mep, wbdg_office_mep, west_riverside_hospital_fire). Hybrid = LearnedPlusLookaheadBaseline (model scores M candidates, takes top-K=12, runs the lookahead). Classical baseline: OctoMap-IG (Bircher et al., ICRA 2016). Corpus: 297 manifest scenes plus IFC-Bench v2 (sylvainHellin/ifc-bench, CC-BY, 93 sub-scenes). Every cell regenerable from runs/v4_validate/headline.json and runs/v5_validate/v5_heldout_*.