Repository navigation
feat: OpenDTU Zero-Export (supports Shelly LNM or MQTT) - #3227
mathieucarbou wants to merge 1 commit into
Conversation
6840d97 to
40bd9ea
Compare
37e9f14 to
6a0d8e4
Compare
80a4bf5 to
427cb5e
Compare
427cb5e to
e22aec2
Compare
|
Thanks @mathieucarbou, I'm currently testing your branch (from your repo including your 3 PRs) with MQTT measurement source. My second HMS is a bit picky about timings and signal, but you branch is working fine. I wonder if it would be possible to implement a dynamic limit relative to current production: Use case:
We can also imagine to have steps (20, 40, 60, 80, 100 %) and set production limit to the next step above current production. I tweaked the hoymileszeroexport from reserve85 this way ... |
Thanks! Do not forget that the regulation interval has to be configured depending on how fast the power control commands are dequeued. This can be seen in the logs. If you have an inverter relying on timeout and not early exit, then its command power control command could take 2 second to be dequeued. In my case my HMS-1000-@r work nicely and are all applying power control within 2 sec.
It cannot be done easily because if you set an upper limit, it means you are allowing more excess. It is equivalent as setting a negative setpoint to increase the limit. The problem is that you never know if this allowance will lead to some production or not. If yes, then what do you do ? You step back ? You allow more excess ? So if you play this game with trying to increase the limit always to "see" if it lead to production or not, just in order to place your limit above, you end up creating production and consumption spikes. That's not a good way to regulate. The current algorithm as it is should be quite immediate because it does not work by reading inverter states. It works by:
So if you have a sudden production spike or decrease, the algorithm should adjust that in about 1-3 rounds max, so in my case 30 sec since I set a regulation interval of 10 sec. |
77210b4 to
ac57261
Compare
|
This is the exact same algorithm I'm using.
Inverters will increase their production (if possible), next round will adjust again if not enough to reach the setpoint. The only idea is to avoid to have a limit way too high above what inverters are currently producing. This will not change at all the global system behavior. I'm trying to avoid such situation:
|
Oh! I think I did not understood what you meant by +20%. If I am correct, you are stepping the increases and decreases to avoid sudden increases or decreases (spikes upon sudden changes) right ? If so, this is equivalent to introducing a PID fo regulation. It is definitely of course possible to add a PID implementation. I did one for YaSolR (solar router - https://yasolr.carbou.me/manual/#pid) specially for MCU, that is quite tested and robust and even support a noise reduction feature, and it is in a lightweight lib that OpenDTU could definitely use (https://mathieu.carbou.me/MycilaUtilities/pid). It even supports proportional on measurements, which helps limit overshoot (that's what you are talking about I think - avoiding overshoots). But I doubt this would help. here is my regulation today at ~ -200W since lunch time:
And a zoom on a spikes:
If you were working with a stepping PID, the grid power would not remain flat but would oscillate more because the sudden change would not be accounted for and would be progressive. So you end up having more excess or consumption. With the changes did in OpenDTU with others PRs to fasten the com, you are not anymore subject to a slow regulation. Changes are applies immediately in less than 2 seconds. Shelly LNM allows a 1 sec interval data: this is even more precise than what HA gets from MQTT because Shelly's only do MQTT publications when the % change in power is above a threshold. With Shelly LNM this is different: every state is received each second. And the regulation is fast: each 10 sec in my case but could even be lowered. So with such second-level scale, there is no need to step the increase or decrease I think... This would need more testing of course, I can be wrong, but let's see if these cases happen first before introducing a PID. For example we see in the graph such spike where the grid power bumped to +1000W then -1000W. But these spikes lasted what ? about 10 seconds. Compared to if you were using a PID which would adjust in 2-3 rounds, you would probably consume or inject more maybe ? |
ac57261 to
76d2380
Compare
|
I would be very happy to test and experiment with a tuned regulation. Shelly Gen1 behave better than new gen regarding MQTT updates. Initially my HMS inverters were both working fine. They are installed the same way, antenna on the top. When it works well, it always has lowest RX success (around 66-75%), the other is 100% Edit: |
is that better now for the graphs ? I am currently doing an update to keep track of the last update intervals in an exponential moving average in order to determine the average update interval time for the metrics and consequently provide a limit update that is in function of that. For example, if the calculated average delay is 10 sec to get a new grid power update and the regulation interval is 5 second, then when applying a new power control, half of the value will be applied (something similar). This is a source-adaptive correction which is automatically spreading the correction to apply across the time. The advantage of that: user does not need to know that the interval of the regulation has to be larger than the delays at which the grid power is coming. |
|
Example of scenario were home consumption if higher than solar production (+setpoint) Stable starting point, stable home consumption, setpoint -800W Starting a 1200W device. Regulation increased limit by nearly 1200W : OK But there is not enough solar power to reach the limit Once again regulation increased limit, reaching the max I stopped the 1200W load. |
|
The algorithm is OK as soon as solar is producing enough. |
76d2380 to
6807edb
Compare
|
@elmr91 : you can install v26.3.30-mathieu8 ;-) I did also and will keep it running tomorrow also to see how it behaves with bigger load spikes. Works fine with a source-adaptive correction:
and this is on my poor MQTT setup ;-) |
0c6d036 to
923cbf5
Compare
|
Today:
Conclusion Everything is working pretty well! v26.3.30-mathieu14 was releases which brings support for Shelly 3EM both in 3-phase and single-phase use cases. |
3774936 to
a8ff7a2
Compare
|
New release: v26.3.30-mathieu15 Bugfix for MQTT and clear separation between the persisted config used at startup and the live config (enable and setpoint) used at runtime that can be changed through Home Asisstant
|
7af97d8 to
a36caf3
Compare
|
You last build (18) is running fine for the whole week-end. |
Yes I am also quite happy with the result it is running fine on my side also! I've looked at the PR and its state and it would need some cleanup, rebase and extracting a fix out of it. I am also irritated by that, and now do not use these sensors anymore. They give wrong values in case some inverters are unreachable, besides creating stat jumps yes. The way the PR solves that by making the sensor unavailable is good. It just needs to be cleaned up though. |
a36caf3 to
72f4522
Compare
|
Hi @mathieucarbou , I noticed some non symmetric exchanges with inverters: I notice the polling rate of both are not identical.
I'm not familiar with Hoymiles radio protocol, but it seems strange. I recorder console logs during these 10 minutes. Edit: There are 2 kinds of expected message sent to the inverters:
But there are lots of AlarmData messages sent to the second inverter only. |
|
@elmr91 : you are running with the new firmware -19 ? |
I noticed only today you have released beta19 with rebase. Edit: beta19 is behaving exactly the same way regarding AlarmData |
Do you know if this is new or was there with last release ? Maybe worth opening an issue or discussion also ? |
Just checked it here. definetely not the case. with yesterdays release (Just have a 4 channel HM and 4 Channel HM available for testing) |
|
@tbnobody @elmr91 : fyi checked my logs in debug and I don't see any, whether zero-export is enabled at 10 sec interval or not... |
I have the exact same settings. I will try to understand how it is supposed to work between DTU and inverters regarding alarm buffer. |
Did you check its firmware version and grid profile to see if there is a difference with the other one ? |
|
@mathieucarbou when did you rebase to the latest master? Was beta18 still running with the source of 26.3.30? |
Yesterday after all your merges i rebased and created -19. -18 was based on master before all your merges. but @elmr91 seemed to had the issue in -18 as he told above also so that’s why I asked him above if the behavior was also there with last release. I agree that finding when the issue started would be nice but for that he needs first to reflash the last release and check. |
|
@tbnobody : Just to let you know: I am moving this zero-export feature from draft to ready in a way to mean that users can now pull it and use it. The feature was written in a way to easily add more grid sources, although I would IMO not add any grid source relying on polling like calling a REST API or TCP Modbus in order to not add extra libraries or memory usage in this project. The current grid source implementations do not add any big overhead since they are using UDP and MQTT which are already used in OpenDTU. A "quick and simple" Zero-Export feature in OpenDTU is a nice addition I think, especially that it integrates well with Home Assistant and allows a Zero-Export control of inverters without requiring any external Home Assistant for the control loop. I'm a big fan of simple/efficient/isolated features like that which solves probably 80% of the use cases for people who need Z-E because in France people having a solar installation are more and more required or advised to activate Z-E. So this feature mimics what's available from Hoymiles (DTU Pro + DDSU). It plays well will the current electrical constraints in Europe where electricity prices are falling into negative values and Zero-Export is becoming more and more common / required. If one day you think about accepting it in the project and need help / rebase / changes, please let me know 👍 Note: if the feature is accepted I will create a PR in https://github.com/tbnobody/OpenDTU-docs/tree/master/docs for the doc and internal working algorithm (how production limit caps, source-adaptive algorithm and failsafe are working) |
72f4522 to
4804197
Compare
|
v26.3.30-mathieu21 was released which also includes a bunch of fixes not yet merged. |
4804197 to
a1fb947
Compare
|
Latest OpenDTU Zero-Export release v26.11.11-ze1 rebased on latest OpenDTU version v26.11.11 |
…d sources Add a Zero-Export module that regulates Hoymiles inverter power limits so that the grid power stays around a configurable setpoint (default: 0 W, i.e. neither export to nor import from the grid), based on the grid power received from a configurable grid source. Two sources are currently available: - Shelly LNM (Local Network Messaging): Shelly meters stream LNM status messages through UDP multicast, avoiding periodic HTTP polling. OpenDTU listens with WiFiUDP and processes incoming datagrams from the scheduler without blocking inverter communication. The wire format is a 12-byte binary header starting with the "SL" magic followed by a JSON payload (em1:0 / em:0 / etc.) whose act_power field carries the grid power. - MQTT: any power meter publishing a plain numeric value or a JSON object with an act_power / total_act_power field (Shelly EM / 3EM style) can be used by subscribing to a configurable "Grid Power Topic" on the existing OpenDTU MQTT client. Retained messages are ignored so stale retained values never drive the regulation. Grid sources are modelled as a small ZeroExportGridSource interface that tracks measurement freshness, estimates the source update period as an EWMA of the intervals between accepted samples, and enforces a per-source fail-safe timeout. Adding a new source means implementing the interface and adding a switch case. The regulation loop reads the grid power from the active source and distributes the needed correction proportionally to the inverters' rated power, never commanding an inverter below a configurable minimal production (default: 40 W per inverter). Inverters that are temporarily unreachable keep their last commanded limit while the others compensate. Upward corrections are damped for slow sources, using the measured source period plus the estimated inverter response latency, to avoid overshoot; downward corrections remain immediate to reduce export. The tracked state is the set of limits this controller has requested, not the values read back from the inverters (the one-time startup adoption below is the only exception). Safety behavior: - Limits are only ever sent to reachable, command-enabled inverters. - On disable, a RelativNonPersistent 100 % release is sent and retried each regulation tick until every command-enabled inverter has received it. - After a DTU restart, limits left by a previous session are reverted by the inverters' own non-persistent-limit timeout; nothing is commanded until regulation is established. On startup each inverter adopts the limit currently active in the inverter once it becomes known through the regular SystemConfigPara poll, instead of guessing one, so a producing inverter is never yanked to another value. Re-enabling after a release starts at nominal power. - When a grid measurement existed but has expired (measurement outage), a minimum-production fail-safe is commanded once per inverter and retried until it could be sent. It is deliberately not kept alive: if the inverter's own non-persistent-limit timeout reverts it to full power, the regulation loop readjusts as soon as fresh grid values arrive again. - A source that has not delivered a single value yet (module just enabled, source just switched, network just reconnected) is a startup condition, not an outage: no RF traffic is generated and the inverters' limits are left untouched. - Requested limits are capped above the currently measured production (with headroom), so they cannot drift far above what the inverters actually produce. The cap's base never drops below the enforced per-inverter floor sum, so a low-light start can always ramp up again. The default regulation interval is 30 seconds (allowed range: 5-3600 seconds). With the current radio layer, each command occupies the radio for its full RX timeout and the command queue also carries the regular poll requests, so a 30 s interval leaves room for a full poll cycle and for the grid source to measure the effect of a correction. Once tbnobody#3225 ("feat(hoymiles): early RX exit for faster inverter control") is merged, a limit command completes in its actual round-trip time (~100-150 ms instead of the full 2000 ms timeout) and a regulation cycle across 8 HMS inverters takes about 2 seconds instead of ~16, which would allow reducing the regulation interval to the 5-second minimum. The settings page exposes: - Enable/disable switch - Grid power setpoint - Minimum production power per inverter (minimum 40 W) - Regulation interval (5-3600 seconds) - Grid source dropdown (Shelly LNM / MQTT) - Shelly LNM multicast group, port and fail-safe timeout (defaults: 239.255.55.55:5555, 10 s) - MQTT grid power topic and fail-safe timeout (default: 30 s) - Live status with a 5-second refresh: grid power, measured grid update interval, and the currently applied production limit The status API additionally reports per-inverter state (commanded limit, pending startup / release / fail-safe flags, reachability, command enablement). Zero-Export state is published under dtu/zeroexport/status (enabled, setpoint, production_limit, grid_update_period), and the module can be controlled through dtu/zeroexport/cmd (enabled, setpoint). Changes received through MQTT are applied live without persisting the configuration to avoid flash wear. Home Assistant integration (when MQTT auto-discovery is enabled) publishes a switch (enabled), a setpoint number, a production-limit sensor and a grid-update-interval diagnostic sensor; the switch and the setpoint can be changed from Home Assistant.
a1fb947 to
d246f2a
Compare
|
Latest OpenDTU Zero-Export release v26.11.11-ze2 rebased on latest OpenDTU version v26.11.11 |






















feat: OpenDTU Zero-Export (supports Shelly LNM or MQTT)
Summary
This PR adds a native Zero-Export controller to OpenDTU.
It regulates Hoymiles inverter power limits directly from the grid power measurement, removing the need for an external Home Assistant automation loop. The grid power is kept around a configurable setpoint (default: 0 W, i.e. neither export to nor import from the grid).
The regulation follows the same feedback model previously used in Home Assistant:
The correction is distributed proportionally to the inverters' rated power and constrained by:
The regulation feedback is only the grid error: the controller tracks the last absolute limit it requested per inverter and uses those values as the starting point of the next step. The inverters' reported limits and production telemetry are deliberately not used as feedback because those values may be delayed or stale. (The one-time startup adoption below is the only exception.)
Two additional safeguards shape the correction:
How try this firmware ?
You can find complete released firmware to download to test that from my fork:
=> https://github.com/mathieucarbou/tbnobody-OpenDTU/releases/
Grid Sources
Grid sources are implemented through the
ZeroExportGridSourceinterface, which also tracks measurement freshness and estimates the source update period. This keeps the regulation algorithm independent from the transport and makes additional sources easy to add later.Currently supported sources:
Future sources such as Victron Modbus TCP or Fronius Modbus TCP can be added without changing the regulation logic.
Shelly LNM
Shelly Local Network Messaging is Shelly's local UDP multicast protocol. Shelly devices publish measurements such as active power, voltage, current, power factor, and frequency without requiring HTTP polling.
OpenDTU:
WiFiUDP"SL"magic) and its JSON payloadem1:0/em:0):act_power(Shelly EM),total_act_power(Shelly 3EM), or the per-phasea_act_power/b_act_power/c_act_powerDefault configuration:
239.255.55.555555act_powerMQTT
The MQTT source uses OpenDTU's existing MQTT client and subscribes to a configurable grid-power topic.
Supported payloads (configurable "MQTT Data Type"):
-123.4act_power,total_act_power,a_act_power,b_act_power, orc_act_power(Shelly EM / 3EM style payloads, as used in YaSolR)Retained MQTT messages are ignored because they may contain an old grid value that must not drive regulation.
The MQTT failsafe timeout defaults to 300 seconds (configurable, 5-300 s), allowing slower MQTT publishers than Shelly LNM.
Inverter Power Limits
Zero-Export only sends non-persistent limits:
The persistent inverter limit is never modified. The controller tracks the last absolute limit it requested per inverter; that value — not the value read back from the inverter — is the starting point of each regulation step.
Startup and Reboot
After a DTU reboot, the previous session's non-persistent limits are no longer known locally; they are reverted by the inverters' own non-persistent-limit timeout.
When Zero-Export starts regulating, each inverter therefore:
SystemConfigParapoll, so regulation starts from reality instead of guessing a value (which could dump a producing inverter's output or leave it unrestrained)Startup commands are additionally deferred until the grid source has delivered a value, so an unavailable source never generates RF traffic.
Unreachable Inverters
When an inverter becomes temporarily unreachable:
Release commands follow the same queue-safety rule: a pending release or fail-safe is retried on each regulation tick until the inverter is reachable and command-enabled.
Grid Source Failure
The controller distinguishes two situations when no fresh grid value is available:
Startup (no value received yet — module just enabled, source just switched, network just reconnected): this is not an outage. No regulation and no fail-safe happen, and the inverters' current limits are left untouched, so enabling Zero-Export never yanks producing inverters down to the minimum.
Measurement outage (a value existed but has expired): normal regulation is paused, the previous grid value is not reused, and a fail-safe limit is commanded to every reachable, command-enabled inverter. Instead of dropping every inverter to its bare floor, the fail-safe distributes the configured Home Minimal Consumption (default: 0 W) as a total production target proportionally across the inverters pending the command, falling back to the per-inverter minimum production floor (default: 40 W) when it is 0:
The status API exposes the grid power, its age, the effective failsafe threshold, the measured source update period and the active fail-safe state, so a dashboard does not keep displaying a stale value.
MQTT Integration
Zero-Export state is published under
dtu/zeroexport/status(enabled,setpoint,production_limit,grid_power_age,grid_update_period,failsafe), and the module can be controlled throughdtu/zeroexport/cmd(enabled,setpoint).Changes received through MQTT are applied live without persisting the configuration, to avoid flash wear.
Regulation Interval
The default regulation interval is 30 seconds (minimum: 5 seconds).
PR #3225 ("feat(hoymiles): early RX exit for faster inverter control") has been merged into the base of this branch: a limit command now completes in its actual round-trip time (~100-150 ms instead of the full 2000 ms RX timeout), and a regulation cycle across 8 HMS inverters takes about 2 seconds instead of ~16. The regulation interval can therefore be reduced down to the 5-second minimum.
Configuration
The Zero-Export configuration includes:
The configuration is included in the existing
config.jsonexport and import flow. The config version was bumped, and older configuration files remain compatible through defaults and migration handling.Web API
Added endpoints:
The status endpoint exposes:
nullwhen no fresh value is available)age > failsafecan be evaluated without a config round-trip)Home Assistant
When MQTT Home Assistant discovery is enabled, OpenDTU publishes:
The enable switch and setpoint can be controlled directly from Home Assistant; the sensors are tied to Zero-Export's availability.
Web Interface
Added a Zero-Export settings page under:
The page provides:
Status values refresh automatically every 5 seconds.
The page is available in English, German, and French.
Observability
The controller logs its decisions for on-device diagnosis:
For command-queue observability (unique command IDs,
enqueue -> TX -> RX -> success/failurelifecycle, queue inspection), see companion PR #3226.Validation
The following checks pass:
generic_esp32s3_usbgit diff --checkReferences
Use Case
Automatic activation of Zero-Export or Full Export with 3ERL in France.
You can have a look at my Home Assistant Dashboard and how I now use OpenDTU + Shelly LNM to regulate my inverters:
https://gist.github.com/mathieucarbou/382556f1279d612962e03232544692d1
I simply need to switch the on/off flag of OpenDTU
And set an appropriate setpoint depending if my EV is plugged for charging or not.
Screenshots
Shelly LNM Setup
OpenDTU Zero-Export Configuration
Shelly LNM:
MQTT
Home Assistant Integration
Regulation Example