Skip to content

Make e2e workflows faster and more reliable - #4413

Merged
felipeelia merged 5 commits into
developfrom
fix/e2e-playwright-install-resilience
Nov 11, 2026
Merged

felipeelia merged 5 commits into
developfrom
fix/e2e-playwright-install-resilience

Conversation

@felipeelia

@felipeelia felipeelia commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Description of the Change

Every e2e job in run 43038219702 (the push of the 5.3.6 release merge to develop) failed before any test ran. This PR fixes that and adds a few other improvements to the e2e workflows. Unless noted, each change applies to both playwright.yml and playwright-privileged.yml.

No more apt installs, and a cached browser

The step that failed was npx playwright install --with-deps chromium ffmpeg: --with-deps runs apt-get, and the runners could not resolve azure.archive.ubuntu.com. apt retried for about ten minutes and then gave up.

The log also shows what --with-deps was installing: 9 font packages (fonts-freefont-ttf, fonts-ipafont-gothic, fonts-tlwg-loma-otf, fonts-unifont, fonts-wqy-zenhei, xfonts-cyrillic, xfonts-encodings, xfonts-scalable, xfonts-utils). No shared libraries were needed, because ubuntu-latest ships Google Chrome and already has all of them. The suite has no visual comparisons, so the fonts don't matter.

  • Drop --with-deps. npx playwright install chromium ffmpeg only downloads the browser from Playwright's CDN, so apt mirror outages can no longer slow down or fail the jobs.
  • Cache the browser. ~/.cache/ms-playwright is cached, keyed on the Playwright version. playwright.yml saves the cache; playwright-privileged.yml only restores it, the same as the npm and composer caches.

If a future runner image or Playwright release needs a library the image doesn't have, Chromium will fail to launch with "Host system is missing dependencies" and a list of the packages to install.

Cancel superseded runs on pull requests (playwright.yml)

A new push to a pull request cancels the e2e cells still running for the previous push. The rule is set per job, so each matrix cell decides for itself:

  • EP.io cells are never cancelled mid-run. They create indices on the shared EP.io host, and only the end of the job deletes them. A newer EP.io cell waits for the older one to finish instead.
  • Pushes to develop and trunk get a group per run, so nothing is cancelled or queued there.

Cleanup steps

  • Test plugin uninstall (added in Require the utils file in the uninstaller #3407 as a regression check for uninstall.php) used if: always(). It still runs when the tests fail, but no longer when the job was cancelled or WordPress never started. That is why it showed up as a second failure in the run above.
  • Its deactivate commands go back to || true. The ! prefix was a way to ignore errors (one of the two commands always fails, depending on how the tests leave the plugin activated), but it read like an assertion.
  • Delete Elasticsearch indices keeps always(), so EP.io indices are deleted even on cancellation. It is now skipped only when the EP.io environment never started, since no indices exist then.

Traces and videos only on retries (playwright.config.ts)

trace and video were retain-on-failure, which records every test and throws the recording away when it passes. On CI they are now on-first-retry, so a test that fails every attempt still leaves a trace and a video. Locally there are no retries, so they stay retain-on-failure.

How to test the Change

  • The e2e checks on this PR pass, and no job runs apt-get.
  • On a re-run, the browser cache is restored and "Install Playwright browser" finishes in seconds.
  • Pushing a new commit to this PR cancels the local Elasticsearch cells of the previous run, while its EP.io cells finish and delete their indices.

Changelog Entry

Developer - Faster and more reliable e2e tests.

Credits

Props @felipeelia

Checklist:

  • I agree to follow this project's Code of Conduct.
  • I have updated the documentation accordingly.
  • I have added tests to cover my change.
  • All new and existing tests pass.

This was done by Claude under my supervision.

…flows

npx playwright install --with-deps failed every e2e job when the Ubuntu
apt mirrors stopped resolving on the runners. Cache ~/.cache/ms-playwright
keyed on the Playwright version, download the browser without --with-deps,
and install the system dependencies in a separate step that only warns on
failure, since ubuntu-latest already ships nearly all of them.
@felipeelia felipeelia added this to the 5.3.6 milestone Nov 11, 2026
ubuntu-latest already has every library Chromium needs; on the current
image apt only added fonts the tests don't use. Without the step, an apt
mirror outage can no longer slow down or fail the e2e jobs.
@felipeelia felipeelia changed the title Cache Playwright browsers and tolerate apt mirror outages in e2e workflows Cache Playwright browsers and skip apt installs in e2e workflows Nov 11, 2026
On pull requests, a new push cancels the e2e cells still running for the
previous one. EP.io cells are never cancelled mid-run, since only the end
of the job deletes their indices on the shared host; a newer EP.io cell
waits instead. Pushes to develop/trunk are unaffected.

Test plugin uninstall no longer runs when the job was cancelled or
WordPress never started, and Delete Elasticsearch indices is skipped when
the EP.io environment never started. The deactivate commands go back to
|| true, which is what the ! prefix was doing.
Recording slows down every test, even when the recording is thrown away
because the test passed. Locally there are no retries, so keep recording
failed tests.
@felipeelia felipeelia changed the title Cache Playwright browsers and skip apt installs in e2e workflows Make e2e workflows faster and more reliable Nov 11, 2026
@felipeelia
felipeelia merged commit 609c968 into develop Nov 11, 2026
68 of 69 checks passed
@felipeelia
felipeelia deleted the fix/e2e-playwright-install-resilience branch November 11, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant