Repository navigation
fix(hoymiles): protect inverter list with a shared_mutex, harden radio queue against concurrent removal - #3259
mathieucarbou wants to merge 1 commit into
Conversation
|
Note: I have included this fix in my dev branch and runs with it today. |
|
@tbnobody : I am pretty confident about this pr also. Running with it yesterday and today. with the early exit mechanism and dtu polling and regulation interval reduced, now the radio queue can be used a lot more but also drained a lot faster, which increased the likelihood of somebody runnning into the race that is fixed here, i.e. if someone is updating the inverter list while a request is in progress. but I never played with add or removal to specifically trigger the race so only reads are happening in my production DTU. If you are considering merging, a way to test is to flash in a setup with random HMS serials and at startup do some add / remove from the UI while inverters are being polled (and requests retried). |
|
@tbnobody Will this fix be included in next release ? |
…o queue against concurrent removal HoymilesClass::_inverters was mutated (addInverter/removeInverterBySerial, invoked from Web API handlers on the async_tcp task) while other tasks iterated the very same vector via getInverterByPos(), getInverterBySerial(), getInverterByFragment() and getNumInverters(), all without locking. A push_back() reallocation or an erase() during such an iteration could leave tasks dereferencing freed vector storage (use-after-free). removeInverterBySerial() also only locked after its unsynchronized search loop had already inspected the vector. The inverter list has a classic reader/writer profile: writes are rare (user adding/removing an inverter) while reads happen constantly (poll loop, MQTT publishing, WebSockets, Prometheus endpoint), from multiple tasks. Switch the single std::mutex to a std::shared_mutex: - addInverter()/removeInverterBySerial() take the unique lock; removeInverterBySerial() now holds it for the whole operation. - getInverterByPos()/BySerial()/ByFragment()/getNumInverters() take a shared lock, so concurrent readers no longer serialize. - Hoymiles.loop() drops its whole-body exclusive lock. It was over-broad anyway, and with a non-recursive shared_mutex it would self-deadlock as the radio layers call back into the public accessors from inside handleReceivedPackage(). All its uses go through the locked accessors; the daily-task section iterates over a locked snapshot copy. Relatedly, dropping loop()'s whole-body lock means the radio's command queue can now be emptied concurrently while a transaction is in flight (an inverter being deleted purges its pending commands). The radio code called _commandQueue.front() unconditionally, which is undefined behavior on an empty deque. Harden this by replacing ThreadSafeQueue::front() with tryFront(), which checks emptiness and fetches the element atomically and returns an empty optional when the queue was emptied. All call sites in HoymilesRadio handle that case by aborting the transaction instead of dereferencing an empty queue. Also add missing null checks after getInverterByPos() in MqttHandleInverter::loop(), MqttHandleHass::publishConfig(), WebApiPower::onPowerStatus(), WebApiLimit::onLimitStatus() and WebApiPrometheus::onPrometheusMetricsGet(), which could dereference a dangling inverter when it was removed concurrently.
e43155d to
067e4b1
Compare
|
Note: commit added in latest OpenDTU Zero-Export release v26.11.11-ze1 rebased on latest OpenDTU version v26.11.11 I am running with this fix without issue since Sept 30. |
|
I've just merged it to my local dev branch. will be included in the next release. |
HoymilesClass::_inverters was mutated (addInverter/removeInverterBySerial, invoked from Web API handlers on the async_tcp task) while other tasks iterated the very same vector via getInverterByPos(), getInverterBySerial(), getInverterByFragment() and getNumInverters(), all without locking. A push_back() reallocation or an erase() during such an iteration could leave tasks dereferencing freed vector storage (use-after-free). removeInverterBySerial() also only locked after its unsynchronized search loop had already inspected the vector.
The inverter list has a classic reader/writer profile: writes are rare (user adding/removing an inverter) while reads happen constantly (poll loop, MQTT publishing, WebSockets, Prometheus endpoint), from multiple tasks. Switch the single std::mutex to a std::shared_mutex:
Relatedly, dropping loop()'s whole-body lock means the radio's command queue can now be emptied concurrently while a transaction is in flight (an inverter being deleted purges its pending commands). The radio code called _commandQueue.front() unconditionally, which is undefined behavior on an empty deque. Harden this by replacing ThreadSafeQueue::front() with tryFront(), which checks emptiness and fetches the element atomically and returns an empty optional when the queue was emptied. All call sites in HoymilesRadio handle that case by aborting the transaction instead of dereferencing an empty queue.
Also add missing null checks after getInverterByPos() in MqttHandleInverter::loop(), MqttHandleHass::publishConfig(), WebApiPower::onPowerStatus(), WebApiLimit::onLimitStatus() and WebApiPrometheus::onPrometheusMetricsGet(), which could dereference a dangling inverter when it was removed concurrently.