Skip to content

fix(hoymiles): protect inverter list with a shared_mutex, harden radio queue against concurrent removal - #3259

Open
mathieucarbou wants to merge 1 commit into
tbnobody:masterfrom
mathieucarbou:fix/hoymiles_shared_mutex
Open

mathieucarbou wants to merge 1 commit into
tbnobody:masterfrom
mathieucarbou:fix/hoymiles_shared_mutex

Conversation

@mathieucarbou

Copy link
Copy Markdown
Contributor

HoymilesClass::_inverters was mutated (addInverter/removeInverterBySerial, invoked from Web API handlers on the async_tcp task) while other tasks iterated the very same vector via getInverterByPos(), getInverterBySerial(), getInverterByFragment() and getNumInverters(), all without locking. A push_back() reallocation or an erase() during such an iteration could leave tasks dereferencing freed vector storage (use-after-free). removeInverterBySerial() also only locked after its unsynchronized search loop had already inspected the vector.

The inverter list has a classic reader/writer profile: writes are rare (user adding/removing an inverter) while reads happen constantly (poll loop, MQTT publishing, WebSockets, Prometheus endpoint), from multiple tasks. Switch the single std::mutex to a std::shared_mutex:

  • addInverter()/removeInverterBySerial() take the unique lock; removeInverterBySerial() now holds it for the whole operation.
  • getInverterByPos()/BySerial()/ByFragment()/getNumInverters() take a shared lock, so concurrent readers no longer serialize.
  • Hoymiles.loop() drops its whole-body exclusive lock. It was over-broad anyway, and with a non-recursive shared_mutex it would self-deadlock as the radio layers call back into the public accessors from inside handleReceivedPackage(). All its uses go through the locked accessors; the daily-task section iterates over a locked snapshot copy.

Relatedly, dropping loop()'s whole-body lock means the radio's command queue can now be emptied concurrently while a transaction is in flight (an inverter being deleted purges its pending commands). The radio code called _commandQueue.front() unconditionally, which is undefined behavior on an empty deque. Harden this by replacing ThreadSafeQueue::front() with tryFront(), which checks emptiness and fetches the element atomically and returns an empty optional when the queue was emptied. All call sites in HoymilesRadio handle that case by aborting the transaction instead of dereferencing an empty queue.

Also add missing null checks after getInverterByPos() in MqttHandleInverter::loop(), MqttHandleHass::publishConfig(), WebApiPower::onPowerStatus(), WebApiLimit::onLimitStatus() and WebApiPrometheus::onPrometheusMetricsGet(), which could dereference a dangling inverter when it was removed concurrently.

@mathieucarbou

mathieucarbou commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Note: I have included this fix in my dev branch and runs with it today.

@mathieucarbou

Copy link
Copy Markdown
Contributor Author

@tbnobody : I am pretty confident about this pr also. Running with it yesterday and today.

with the early exit mechanism and dtu polling and regulation interval reduced, now the radio queue can be used a lot more but also drained a lot faster, which increased the likelihood of somebody runnning into the race that is fixed here, i.e. if someone is updating the inverter list while a request is in progress.

but I never played with add or removal to specifically trigger the race so only reads are happening in my production DTU.

If you are considering merging, a way to test is to flash in a setup with random HMS serials and at startup do some add / remove from the UI while inverters are being polled (and requests retried).

@mathieucarbou

Copy link
Copy Markdown
Contributor Author

@tbnobody Will this fix be included in next release ?

…o queue against concurrent removal

HoymilesClass::_inverters was mutated (addInverter/removeInverterBySerial,
invoked from Web API handlers on the async_tcp task) while other tasks
iterated the very same vector via getInverterByPos(), getInverterBySerial(),
getInverterByFragment() and getNumInverters(), all without locking.
A push_back() reallocation or an erase() during such an iteration could
leave tasks dereferencing freed vector storage (use-after-free).
removeInverterBySerial() also only locked after its unsynchronized search
loop had already inspected the vector.

The inverter list has a classic reader/writer profile: writes are rare
(user adding/removing an inverter) while reads happen constantly
(poll loop, MQTT publishing, WebSockets, Prometheus endpoint), from
multiple tasks. Switch the single std::mutex to a std::shared_mutex:

- addInverter()/removeInverterBySerial() take the unique lock;
  removeInverterBySerial() now holds it for the whole operation.
- getInverterByPos()/BySerial()/ByFragment()/getNumInverters() take a
  shared lock, so concurrent readers no longer serialize.
- Hoymiles.loop() drops its whole-body exclusive lock. It was over-broad
  anyway, and with a non-recursive shared_mutex it would self-deadlock
  as the radio layers call back into the public accessors from inside
  handleReceivedPackage(). All its uses go through the locked accessors;
  the daily-task section iterates over a locked snapshot copy.

Relatedly, dropping loop()'s whole-body lock means the radio's command
queue can now be emptied concurrently while a transaction is in flight
(an inverter being deleted purges its pending commands). The radio code
called _commandQueue.front() unconditionally, which is undefined behavior
on an empty deque. Harden this by replacing ThreadSafeQueue::front() with
tryFront(), which checks emptiness and fetches the element atomically and
returns an empty optional when the queue was emptied. All call sites in
HoymilesRadio handle that case by aborting the transaction instead of
dereferencing an empty queue.

Also add missing null checks after getInverterByPos() in
MqttHandleInverter::loop(), MqttHandleHass::publishConfig(),
WebApiPower::onPowerStatus(), WebApiLimit::onLimitStatus() and
WebApiPrometheus::onPrometheusMetricsGet(), which could dereference a
dangling inverter when it was removed concurrently.
@mathieucarbou
mathieucarbou force-pushed the fix/hoymiles_shared_mutex branch from e43155d to 067e4b1 Compare November 11, 2026 16:51
@mathieucarbou

Copy link
Copy Markdown
Contributor Author

Note: commit added in latest OpenDTU Zero-Export release v26.11.11-ze1 rebased on latest OpenDTU version v26.11.11

I am running with this fix without issue since Sept 30.

@tbnobody

Copy link
Copy Markdown
Owner

I've just merged it to my local dev branch. will be included in the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants