Repository navigation
Make e2e workflows faster and more reliable - #4413
Merged
Merged
Conversation
…flows npx playwright install --with-deps failed every e2e job when the Ubuntu apt mirrors stopped resolving on the runners. Cache ~/.cache/ms-playwright keyed on the Playwright version, download the browser without --with-deps, and install the system dependencies in a separate step that only warns on failure, since ubuntu-latest already ships nearly all of them.
ubuntu-latest already has every library Chromium needs; on the current image apt only added fonts the tests don't use. Without the step, an apt mirror outage can no longer slow down or fail the e2e jobs.
On pull requests, a new push cancels the e2e cells still running for the previous one. EP.io cells are never cancelled mid-run, since only the end of the job deletes their indices on the shared host; a newer EP.io cell waits instead. Pushes to develop/trunk are unaffected. Test plugin uninstall no longer runs when the job was cancelled or WordPress never started, and Delete Elasticsearch indices is skipped when the EP.io environment never started. The deactivate commands go back to || true, which is what the ! prefix was doing.
Recording slows down every test, even when the recording is thrown away because the test passed. Locally there are no retries, so keep recording failed tests.
1 of 4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the Change
Every e2e job in run 43038219702 (the push of the 5.3.6 release merge to
develop) failed before any test ran. This PR fixes that and adds a few other improvements to the e2e workflows. Unless noted, each change applies to bothplaywright.ymlandplaywright-privileged.yml.No more apt installs, and a cached browser
The step that failed was
npx playwright install --with-deps chromium ffmpeg:--with-depsrunsapt-get, and the runners could not resolveazure.archive.ubuntu.com. apt retried for about ten minutes and then gave up.The log also shows what
--with-depswas installing: 9 font packages (fonts-freefont-ttf,fonts-ipafont-gothic,fonts-tlwg-loma-otf,fonts-unifont,fonts-wqy-zenhei,xfonts-cyrillic,xfonts-encodings,xfonts-scalable,xfonts-utils). No shared libraries were needed, becauseubuntu-latestships Google Chrome and already has all of them. The suite has no visual comparisons, so the fonts don't matter.--with-deps.npx playwright install chromium ffmpegonly downloads the browser from Playwright's CDN, so apt mirror outages can no longer slow down or fail the jobs.~/.cache/ms-playwrightis cached, keyed on the Playwright version.playwright.ymlsaves the cache;playwright-privileged.ymlonly restores it, the same as the npm and composer caches.If a future runner image or Playwright release needs a library the image doesn't have, Chromium will fail to launch with "Host system is missing dependencies" and a list of the packages to install.
Cancel superseded runs on pull requests (
playwright.yml)A new push to a pull request cancels the e2e cells still running for the previous push. The rule is set per job, so each matrix cell decides for itself:
developandtrunkget a group per run, so nothing is cancelled or queued there.Cleanup steps
uninstall.php) usedif: always(). It still runs when the tests fail, but no longer when the job was cancelled or WordPress never started. That is why it showed up as a second failure in the run above.|| true. The!prefix was a way to ignore errors (one of the two commands always fails, depending on how the tests leave the plugin activated), but it read like an assertion.always(), so EP.io indices are deleted even on cancellation. It is now skipped only when the EP.io environment never started, since no indices exist then.Traces and videos only on retries (
playwright.config.ts)traceandvideowereretain-on-failure, which records every test and throws the recording away when it passes. On CI they are nowon-first-retry, so a test that fails every attempt still leaves a trace and a video. Locally there are no retries, so they stayretain-on-failure.How to test the Change
apt-get.Changelog Entry
Credits
Props @felipeelia
Checklist:
This was done by Claude under my supervision.