Skip to content

fix(config): save config.toml atomically and serialise its updates - #562

Open
vitorcamillo wants to merge 8 commits into
games-on-whales:betafrom
vitorcamillo:fix/config-atomic-persistence
Open

vitorcamillo wants to merge 8 commits into
games-on-whales:betafrom
vitorcamillo:fix/config-atomic-persistence

Conversation

@vitorcamillo

@vitorcamillo vitorcamillo commented Nov 6, 2026 •

Copy link
Copy Markdown

Problem

Every runtime change to the configuration (pair, unpair, update_client_settings, update_profiles, and the
v≤6 → v7 migration) rewrites config.toml in place: rfl::toml::save() / std::ofstream truncate the file and then
write it. Two consequences:

  1. Truncated config. A crash, SIGKILL, power loss or full disk between the truncate and the end of the write
    leaves an empty or partial config.toml; Wolf then fails to load it on the next start and paired clients /
    profiles have to be recovered by hand. Any concurrent reader (Wolf itself while handling another API call,
    wolf-ui, a backup job) can also observe the partial file.
  2. Lost updates. Those functions do load → modify → save of the whole file with no synchronisation. Two API
    calls running concurrently (e.g. clients/settings and profiles/..., or a pairing while the UI saves profiles)
    each start from the same old file and the last writer silently drops the other change.

How to reproduce

  • (1) Loop update_profiles() while another thread reads config.toml: the reader sees content different from what
    is being written (in our run on beta: 1748 bad reads out of 100 writes).
  • (2) Call update_client_settings() and update_profiles() from two threads: the final file keeps only one of the
    two changes (app_state_folder reverted to "some/folder").

Both are covered by the new tests.

Fix

  • save_file_atomically(): write the new content to <config>.tmp next to the file and rename() it over
    config.toml. There is no non-atomic fallback: if the write or the rename fails it throws and the original file is
    left as it was. This assumes the config lives in a directory (the documented /etc/wolf volume), not a single
    bind-mounted file.
  • Config gets a std::shared_ptr<std::mutex> config_file_mutex (shared by every copy of the Config), held by the
    four read-modify-write functions.
  • Those functions now save the file first and only then update the in-memory immer::atoms, so a failed save leaves
    both the file and memory unchanged.

Tests

New LocalState config is saved atomically test case in tests/testMoonlight.cpp (works on a copy of
config.test.toml in a temp dir):

  • Content is saved and no temporary file is left behind.
  • Readers never see a partially written file: 100 update_profiles() while a thread keeps reading the file; every
    read must equal the expected content. Fails on beta (1748 bad reads).
  • Concurrent updates to different parts of the config are not lost: update_client_settings() and
    update_profiles() (on a copy of the Config) from two threads, 30 times each; both changes must be in the file.
    Fails without the mutex.
  • A failed save leaves file and memory untouched: the save is forced to fail; update_profiles() throws, the file
    and the in-memory profiles are unchanged.

Built in the devbase container (GCC, Release, C++20): ~[DOCKER] suite All tests passed (617 assertions in 38 test cases), also with MALLOC_PERTURB_=165; clang-format 18.1.8 clean.

🤖 Generated with Claude Code

vitorcamillo and others added 2 commits November 6, 2026 13:49
pair(), unpair(), update_client_settings(), update_profiles() and the
config migration rewrite config.toml in place (truncate + write). A
crash, a full disk or a concurrent reader in between sees an empty or
partially written file; after a crash Wolf then fails to load it on
the next start and paired clients/profiles have to be recovered by
hand.

Write the new content to a temporary file in the same directory, fsync
it, copy the original mode and owner, rename it over the destination
and fsync the directory. Symlinks are followed so the link target is
updated. When the file can't be replaced atomically (directory not
writable, ownership we can't reproduce, or config.toml bind mounted as
a single file: rename() fails with EBUSY) fall back to the previous
in-place write with a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pair(), unpair(), update_client_settings() and update_profiles() load
the whole config.toml, change their part and save it back without any
synchronisation, so two concurrent API calls start from the same file
and the last writer silently drops the other change.

Hold a mutex, shared by all the copies of the Config, for the duration
of the file round-trip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JBailes

JBailes commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

I think there's value in fixing the concurrent updates and potential config corruption, but I'm not convinced this implementation is the right approach.

We're adding quite a bit of low-level filesystem handling for what should be a relatively simple operation, and the non-atomic fallback effectively reintroduces the original problem. We also still update in-memory state before persistence succeeds, so failures can leave the two inconsistent.

I'd prefer we keep the synchronization fix and simplify the atomic write implementation around Wolf's actual deployment requirements. The underlying issues are worth fixing, but I don't think the additional complexity is justified as it stands.

…mory

Address review feedback on games-on-whales#562:
- write config.toml to `<file>.tmp` and rename it over the original, with
  no non-atomic fallback; a failed save now throws instead of truncating
- drop the permission/ownership/symlink handling and the fsync calls
- pair/unpair/update_client_settings/update_profiles save the file first
  and only then update the in-memory state, so a failed save leaves both
  untouched (new test case)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vitorcamillo

vitorcamillo commented Nov 6, 2026 •

Copy link
Copy Markdown
Author

Thanks, that makes sense. I've pushed a simpler version (2796376):

  • the save is now just "write <config>.tmp, then rename() it over config.toml", with no fallback. If the write or the rename fails it throws and leaves the old file alone, so the original problem can't come back through a side path;
  • dropped the permission/ownership/symlink handling and the fsyncs, since Wolf keeps its config in the /etc/wolf volume;
  • kept the mutex, and pair/unpair/update_client_settings/update_profiles now save the file first and only then update the in-memory state, so a failed save leaves both consistent. There's a new test case for that.

The source change against beta went from +162 to +78 lines (most of what is left is the reordering in the four functions). Happy to trim further if something still feels like too much.

Resolve the conflict in tests/testMoonlight.cpp by keeping both new test
cases: the atomic config save one from this branch and the unknown-issuer
cert one from beta.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vitorcamillo

Copy link
Copy Markdown
Author

I merged beta into this branch to fix the conflict in tests/testMoonlight.cpp (kept both new test cases). Could you approve the workflow run when you have a moment?

@andrewdunndev

Copy link
Copy Markdown
Contributor

Tested on an NVIDIA host (GA102): beta ac2461f against ac2461f with this branch merged (the config files are byte-identical to ba9625f). Both were built the same way, run in the sha-ac2461f image, and pointed at a copy of a real config.toml. A script sends each round's API writes at once, from threads released together on the unix socket. It then checks that config.toml parses and holds every write, in the file and in memory. The pairing rows use a real moonlight-qt pairing while apps/add and apps/delete churn.

Round (concurrent writes) beta PR
8 apps/add, then 8 apps/delete (200 / 150 rounds) 1 lost, 1 crash 0
clients/settings on 3 different clients (150 / 100) 150 lost a write in the file 0
clients/settings run_uid + run_gid on one client (150 / 100) 77 lost, 10 files corrupt 100 lost
4 profiles/add (150 / 100) 82 lost 100 lost
2 apps/add + profiles/add + 3 clients/settings (200 / 150) 101 crash, 99 lost 150 lost the profile
pairing during churn (24 / 19) 1 pairing missing from the file 0

On beta, two concurrent ofstream saves each truncate at open and write from their own offset, so a shorter write leaves the tail of a longer one behind. All 10 corrupt files were a valid prefix plus a leftover tail, the same signature #492 describes. The next writer's rfl::toml::load(...).value() then throws, nothing catches it, and Wolf exits (112 times here). With the tail on disk, Wolf will not start again. A reader parsing the file during the apps rounds failed 788 of 93.5k reads. The atomic rename and the mutex remove all of this: 0 corrupt files, 0 crashes, and 0 of 63k reads failed in the apps rounds (2 of 525k in the profiles rounds, caused by our own non-atomic test copy, not by Wolf).

Still open: writes to the same section are lost, more often than on beta. The profile endpoints call profiles->load() outside the lock and pass the whole new list, and clients/settings merges from get_client_by_id before the lock. update_* then stores whatever value it gets, and now only after the file write, which widens the window. Proposed on top of ba9625f: pass a function of the current state, applied under config_file_mutex. AddApp, RemoveApp and RemoveProfile change the same way as AddProfile (excerpt below, verbatim from the diff), and clients/settings gets an update_client_settings overload that takes the merge, plus an update_profiles overload that takes the function:

-    state::update_profiles(
-        state_->app_state->config,
-        profiles.push_back(rfl::Reflector<events::Profile>::to(p, this->state_->app_state->event_bus)));
+    state::update_profiles(state_->app_state->config, [&](const state::ProfilesList &profiles) {
+      return profiles.push_back(rfl::Reflector<events::Profile>::to(p, this->state_->app_state->event_bus));
+    });

With it, every row above is 0 lost (n=100, pairing n=15). A new unit test (4 threads x 10 profile adds, plus run_uid and run_gid on one client) fails 12 == 42 when update_fn runs outside the lock. Two smaller points: with config.toml bind-mounted as a single file, Wolf exits on the first write (beta keeps working). The error reads "Unable to replace ...: Success", because fs::remove(tmp_file, ec) overwrites ec before it is formatted, which hides the errno. EBUSY is the likely cause (rename onto a mount point), but that is from Linux semantics, not observed. The full diff with the test (clang-format 18 applied) is ready to push wherever suits you. #492 leaves the endpoints as they are, so the same-section loss applies there too (from reading the code, not run).

We have a test bench (one AMD host with a discrete and an integrated GPU, one NVIDIA host, scripted Moonlight clients) and are glad to take on A/B runs or side quests. GPU age is the one limit worth naming.

vitorcamillo and others added 4 commits November 21, 2026 19:24
…ted config.toml

- update_profiles/update_client_settings take a function of the current
  state, applied while holding config_file_mutex, so concurrent writes to
  the same section are no longer lost; the API endpoints use it
- fall back to an in-place write when config.toml is a bind-mount point
  (rename fails with EBUSY/EXDEV)
- keep the rename error code when formatting the error message

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keeps the previous file content in memory and the complete new content in
config.toml.tmp until the in-place write succeeds, so a failed write on a
bind-mounted config.toml never leaves a truncated file without a copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vitorcamillo

Copy link
Copy Markdown
Author

Thanks a lot for the thorough test report and the bench, @andrewdunndev, it was really helpful. I've pushed fixes for the three points:

  1. Lost updates under concurrency: update_profiles and update_client_settings now take a callback that is applied to the current state while holding the config lock, so concurrent API calls (AddApp/RemoveApp/AddProfile/RemoveProfile/UpdateClientSettings) can no longer overwrite each other. UpdateClientSettings now returns 404 when the client is gone.
  2. Single-file bind mount (config.toml mounted directly): when the rename fails with EBUSY/EXDEV, Wolf logs a warning and falls back to writing the file in place. The previous content is read first (if it can't be read, the file is left untouched), and it is restored if the in-place write fails; the full new content is kept in config.toml.tmp until the write succeeds.
  3. Error message: the rename failure now reports the real rename error instead of the cleanup one.

Tests added:

  • concurrent profile/uid/gid updates (checked both in memory and on disk);
  • missing client in update_client_settings;
  • the fallback via injectable filesystem ops: EBUSY, EXDEV, unreadable source, vanished source, partial write with restore, failed restore, and non-fallback rename errors.

CI is green on my fork (build + tests on g++/clang, shared/static, x86_64/aarch64, and clang-format).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants