[YuanYuYuan]

Problem

With an unpatched 1.10.x peer, SHM negotiation can end with one peer on SHM and the other plain. The SHM peer sends descriptors. The plain peer delivers them as payloads, so the application receives corrupt data. Addresses #2863.

The cause is the SHM handoff change in #2646, released in 1.10.0. Before it, 1.9.0 enabled SHM from a confirmed flag. Since it, each peer enables transport SHM when it opened the segment of the other peer. One failed open on one side leaves one peer on SHM and the other plain.

The diagrams show the same case: the acceptor cannot validate the initiator segment and declines.

Before #2646 (1.9.0): SHM needs a confirmation

sequenceDiagram
    participant I as Initiator
    participant A as Acceptor
    I->>A: InitSyn (SHM offer)
    A->>I: InitAck (acceptor segment)
    I->>A: OpenSyn (challenge)
    Note over A: challenge mismatch: decline
    A->>I: OpenAck (SHM not confirmed)
    Note over I,A: SHM needs the confirmation on both ends: both run plain

Loading

Since #2646 (1.10.x): SHM follows the open

sequenceDiagram
    participant I as Initiator
    participant A as Acceptor
    I->>A: InitSyn (SHM offer)
    A->>I: InitAck (acceptor segment)
    Note over I: opened the acceptor segment: SHM on
    I->>A: OpenSyn (challenge)
    Note over A: challenge mismatch: decline, run plain
    A->>I: OpenAck (no SHM extension)
    I->>A: put (SHM descriptor)
    Note over A: plain: delivers the descriptor bytes as the payload

Loading

In a two-process test, this case produced 695 to 697 corrupt samples per run.

With this PR: a decline fails the handshake

sequenceDiagram
    participant I as Initiator
    participant A as Acceptor
    I->>A: InitSyn (SHM offer)
    A->>I: InitAck (acceptor segment)
    I->>A: OpenSyn (challenge)
    Note over A: challenge mismatch: decline, fail the handshake
    A->>I: Close
    Note over I,A: no link, so no SHM descriptor reaches a plain peer

Loading

Fix: before and after

This PR makes a decline fail the handshake, so both peers agree or the acceptor refuses the link.

Case	Before	With this PR
The acceptor declines the SHM proposal	The acceptor runs plain and an unpatched initiator runs SHM	The handshake fails and the link is refused
The acceptor sent no SHM offer but puts an extension in its OpenAck, as 1.10.x does	The initiator never gets a transport	The initiator ignores the extension and the link comes up plain
Both peers complete the negotiation	SHM on both sides	Unchanged

Commits

#	Commit	Purpose
1	Require completed SHM negotiation before enabling transport SHM	The handshake commit from #2873, unchanged. Both peers enable SHM only after the negotiation completes
2	fix(shm): ignore OpenAck extension without offer	An initiator that got no SHM offer ignores an OpenAck extension and runs plain
3	fix(shm): fail handshake on declined proposal	An acceptor that declines fails the handshake
4 to 6	test(shm): ...	Tests for the rejection causes, the TX counters after a decline, and a 1.9.0 initiator

Each fix has a test that fails without it.

Relation to #2873

This PR carries the handshake half of #2873. The lifecycle commit of #2873 merges on its own. The two commits touch different files. If #2873 lands first with its handshake commit, commit 1 drops out on rebase.

Breaking changes

None to the wire format or the public API.

What changes	Who is affected	Before	After	Action
An acceptor that declines an SHM proposal fails the handshake	A 1.10.x initiator whose segment the acceptor cannot validate, for example across /dev/shm namespaces	The acceptor accepted the link plain, while the initiator could run SHM	The acceptor refuses the link. Its log shows "refusing the SHM proposal"	Make the segment reachable, or disable SHM on the initiator

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.99%. Comparing base (766b455) to head (484922d).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2883      +/-   ##
==========================================
+ Coverage   74.95%   74.99%   +0.03%     
==========================================
  Files         419      419              
  Lines       64143    64170      +27     
==========================================
+ Hits        48080    48123      +43     
+ Misses      16063    16047      -16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.