Repository navigation
fix(ingest): preserve scan progress across worker claims - #472
Merged
Merged
Conversation
beinan
force-pushed
the
codex/wal-preserve-worker-scan-progress
branch
from
October 22, 2026 16:49
e71fbf1 to
0145aba
Compare
beinan
changed the base branch from
codex/wal-distributed-preparation-public
to
main
October 22, 2026 16:49
beinan
marked this pull request as ready for review
October 22, 2026 16:49
beinan
added a commit
that referenced
this pull request
Oct 22, 2026
Queueing behind replication work and reading or validating an accepted record currently contribute to the same fetch latency. Expose separate cumulative queue and service timings in each group's local `/status` observation, alongside found, missing, failed, and returned-byte counts. Counters use fixed storage and saturating arithmetic. They cover serviced requests, including callers that disconnect; queued work discarded on shutdown is excluded. Queue timing starts after runtime admission, and service timing excludes reply transmission. These local wall-time observations do not establish quorum freshness or change durable acknowledgements. Refs #378. Built on merged #471 and #472. ## Testing - Real blocked and failed storage-sync regression with both replication modes: existing and missing-record fetches complete and are counted before the unrelated sync, while the new append remains unacknowledged. - Actual WAL service and Lance recovery fixture with full-column readback and retained process identities. - WAL all-feature/all-target Clippy and tests passed. - Complete local CI-equivalent script passed: workspace formatting, Clippy and all-target tests; Python tests (229 passed, 2 skipped, 1 expected failure), Ruff formatting/lint and Pyright.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A bounded source scan can encounter only records from other tables. The worker previously released its claim and discarded the scan cursor, then started from the same durable checkpoint on its next claim. A table behind more than one scan budget of retained foreign records could remain unpublished indefinitely.
Retain the empty-page scan hints per fixed worker assignment and move them through claim jobs. Hints are checked against the complete table binding, contain only quorum-observed ranges without that table's records, and never advance durable checkpoints. Scan limits, claim rotation and publisher fencing remain unchanged. Restart may rescan those ranges safely.
Refs #378. Built on merged #466, #468, #469, #470 and #471.
Validation:
The first worker-suite run encountered an existing lease-revoke timeout with slow local etcd synchronization. The unchanged suite passed on a fresh run with the same storage and deadlines; the original failure was retained. This change fixes publication progress and does not claim a throughput improvement.