Repository navigation
feat(wal): overlap bounded replication with local persistence - #469
Merged
Merged
Conversation
This was referenced Oct 22, 2026
beinan
marked this pull request as ready for review
October 22, 2026 11:36
beinan
force-pushed
the
codex/wal-early-peer-dispatch
branch
from
October 22, 2026 11:37
e37a9b5 to
d7b5094
Compare
beinan
added a commit
that referenced
this pull request
Oct 22, 2026
When an uncompressed engine block misses the read cache, the decoder currently copies the newly read block into another allocation before parsing entries. Reuse the original allocation after verifying the complete block checksum, removing one block-sized allocation and copy. Compressed decoding, cache isolation, entry parsing, durable ACKs, and file formats retain their existing behavior. Refs #378. Built on merged #469. Validation: actual-file cache and rewrite regressions; compressed/uncompressed multi-entry reopen; corruption rejection and recovery after restoring the original byte; log/checkpoint/snapshot tests; all-feature WAL checks; full local `ci-pr-helper` checks. The candidate release also passes nine-process WAL crash/reopen and complete Arrow/Lance recovery with a missing preparer and lost preparation cache. Local allocation diagnostics confirm the removed allocation; no end-to-end throughput improvement is claimed.
beinan
added a commit
that referenced
this pull request
Oct 22, 2026
A bounded source scan can encounter only records from other tables. The worker previously released its claim and discarded the scan cursor, then started from the same durable checkpoint on its next claim. A table behind more than one scan budget of retained foreign records could remain unpublished indefinitely. Retain the empty-page scan hints per fixed worker assignment and move them through claim jobs. Hints are checked against the complete table binding, contain only quorum-observed ranges without that table's records, and never advance durable checkpoints. Scan limits, claim rotation and publisher fencing remain unchanged. Restart may rescan those ranges safely. Refs #378. Built on merged #466, #468, #469, #470 and #471. Validation: - Real three-replica WAL, etcd and Lance regression: a target behind 129 retained foreign records fails with the old worker and publishes its exact payload with the fix, while another table remains unpublished. - Materializer, worker, service and service/http-client feature Clippy checks; ingestion and etcd-backed worker recovery suites. - Full repository CI helper, including Rust/Python tests, formatting and lint/type checks. The first worker-suite run encountered an existing lease-revoke timeout with slow local etcd synchronization. The unchanged suite passed on a fresh run with the same storage and deadlines; the original failure was retained. This change fixes publication progress and does not claim a throughput improvement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The runtime currently waits for the leader's complete synchronous storage poll before sending replication packets, delaying the start of follower persistence. Add the default-off
runtime.pipeline_replicationoption to dispatch ordinary, non-snapshot RaftReady.messagesthrough the existing bounded transport before local persistence.Follower persistence-required responses, all existing sync barriers, client completions, quorum reads, snapshots, and public synchronous
Node::poll()semantics retain their ordering. Packets move shared payload buffers and retain node-wide byte/request credits; a full queue drops for Raft retransmission without blocking the owner. An early packet followed by failed leader persistence remains an uncertain attempt and never a successful ACK from that poll.This is an experimental option, with no established throughput improvement. It does not make storage asynchronous or allow the Raft owner to tick during a blocking filesystem operation.
Part of #378.
Testing
A cloud comparison using identical binaries and changing only the node option is queued; no result is claimed here. Final-head CI must pass before merge.