[Hakandede]

What

When {cd:1,_id:1} is missing, the run builds it during mapping. On large sources that wait can take hours to days, and the dashboard showed nothing: the run progress bar stays hidden until chunks exist, and index progress was only polled after clicking "Build missing indexes".

The Overview tab now shows an index bar while the run waits on an index:

Building {cd:1,_id:1} index on drill_events_xxx — collection 1 of 2
step 1 of 4: scanning documents · 54% (1,613,851 / 3,000,000) · waiting 7s · chunk copying starts once the index is ready

When the index is ready the bar disappears and the normal run progress bar takes over. Guide step 2 lists the run's own builds too, not only button builds.

Behaviour

• Progress comes from $currentOp (the builder thread's progress {done,total} and msg); the client op waiting on the build has no counters and only shows when no builder is visible.
• MongoDB restarts the percentage per step, so the step travels with it: 1 scanning documents, 2 sorting index keys (the builder keeps saying "scanning collection" without counters for 30–45% of the build after the scan ends; seen in every recording), 3 writing index keys, 4 applying writes made during the build.
• Sharded clusters (through mongos): every shard reports its own builder with per-shard counters. The bar shows the slowest shard and how many shards are still building, since the run waits for all of them.
• Secondaries report the builder without command.$db; matched through ns.
• The "waiting on index" state is set around ensureIndex itself, not the hasRequiredIndex check: an in-progress build is listed as present but still blocks.
• The dry run's builds show too (it maps on its own orchestrator).

Fixes found on the way

• Server-side progress never worked under the default config. The source client defaults to readConcern: majority, and $currentOp refuses anything but local; the error was swallowed. The query now pins local, and any failure is returned as serverOpsError and shown in the UI instead of "no builds".
• "Build missing indexes" said "all indexed" while a build was running. It now joins builds already in flight.

CI

• The integration job now runs on the mongo:8.0 service image (renamed to Integration (MongoDB 8.0 + ClickHouse 26.8)), the MongoDB line deployments run. No ruleset requires the job by name.
• The unit job also runs tests/unit (it only ran tests/differential).

Blast radius

• mapCollection: only records state around the existing ensureIndex call (try/finally); no change to what it reads or writes.
• /api/index-progress: adds auto, phases, serverOpsError; serverOps entries gain phase, phaseLabel, done, total, shard, shards. Only the dashboard consumes it.
• /control/build-indexes: response gains inProgress.
• No change to chunking, copying, verification or the ledger.

Tests

Tested against MongoDB 8.0.34 and ClickHouse 26.8.34.13.

• Unit (41) — run in CI's unit job (added tests/unit there; it only ran tests/differential):
  • parser against real $currentOp recordings from MongoDB 8.0.34: replica-set primary, replica-set secondary, 2-shard cluster via mongos;
  • whole-build replays: one entry per sample, steps never go backwards, percentage never drops within a step, sorting only follows the scan;
  • orchestrator logic against a fake MongoDB: readConcern pin and projection, surfaced errors, joining in-flight builds, failed builds, state cleared when ensureIndex fails.
• Integration (3): the mapping-time state per collection under the default readConcern: majority, the listed-but-building case, and the button path.
• Mutation check: 8 deliberate bugs (readConcern pin removed, errors swallowed, fastest shard reported, no ns fallback, state not cleared, in-flight builds ignored, step not reported, no sorting step) each make tests fail.
• Full suite on the final head against MongoDB 8.0.34 + ClickHouse 26.8.34.13: 218 passed, 1 skipped (pre-existing), 0 failed; typecheck clean.

Run for real

Real migrator and dashboard on a dev VM, CPU-throttled MongoDB so builds are watchable:

• MongoDB 8.0.34 single server, 3M docs, 2 collections: step 1 (4→89%) → step 2 sorting → step 3 (10→99%) → bar gone, copying started; no serverOpsError at any sample.
• MongoDB 8.0.34, 2 shards via mongos, 1M docs: slowest shard reported throughout ("slowest of 2 shards still building: s1"), then copying started.

Not covered

• Atlas-managed clusters.
• Users without the inprog privilege: unit-tested only (the bar shows the reason instead of a percentage).