Skip to content

feat: plan 033 — privacy lookup index, cart lock on series start, Skipped runs for refused cart claims - #40

Merged
jack9x merged 20 commits into
developfrom
jack/feature/033-privacy-index
Nov 12, 2026
Merged

jack9x merged 20 commits into
developfrom
jack/feature/033-privacy-index

Conversation

@jack9x

@jack9x jack9x commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR must ship before the WordPress.org submission.

  • Privacy lookups use an index. Export and erase used to scan every run and every step's JSON on each page. Now they read a new run_emails table plus runs.subject. At 200k runs a page takes about 4 ms instead of 5.5 s.
  • A cart is locked only when a series starts. An abandon that no active workflow takes leaves the cart free. The same cart can then start the Checkout reminder when the shopper leaves at checkout later.
  • Refused cart claims show up. A cart claim refused because the cart already has a series, the cart was ordered, the shopper unsubscribed, or the email had a series in the last 7 days now records a Skipped run with that reason. It shows under the existing Skipped tab.

Base: develop 9b31ca7. Schema change: DB 7 → 8 (new run_emails table, KEY subject (subject, status) on runs). No backfill: runs written before 033 aren't indexed, so export and erase can't find them. Staging: after deploy, run ./run.sh reset-data --staging, then seed, then ./run.sh probe-mysql --staging. Tester sites: no reset; test only with emails never used on that site.

One commit per checkpoint; CP3 and CP4 share one.

Checkpoints

  • CP1 Schema (4da6801)
    • run_emails (run_id, email; primary (run_id, email), index (email, run_id)) and KEY subject on runs.
    • Uninstaller::TABLES and reset-data list the new table.
  • CP2 Indexing at every write (b98b1e2)
    • RunStore::index_emails() runs in create, save and cancel_waiting, and in RunStepStore start, finish, close_open and cancel_waiting.
    • JSON values count when they are a whole normalized email or a claim key. Error text counts every address replace_text() would replace, with the same boundaries.
    • One INSERT IGNORE … SELECT per write, selecting from runs or run_steps, so a missing run gets no row. A failure is logged and the write stands.
  • CP3 + CP4 Lookups and sync (ad1b581)
    • about() is a UNION of run_emails (email + claim key) and runs.subject. for_export keeps OFFSET.
    • anonymize() removes the person's rows, except for a run whose scrub failed or changed nothing.
    • delete_with_steps() deletes steps, then runs, then index rows.
  • CP5 Proof (aa9dff7)
    • ./run.sh bench-privacy (wp-env only) and ./run.sh check-privacy-index (in check-all).
    • The old scan lives only in dev/scripts/lib/privacy-old-scan.php.
  • CP6 Cart lock on series start (7ccb72c)
    • The sweep fires yaymatik_cart_abandoned( $cart_id, $kind ) on every abandon of a cart with no series yet.
    • CartStore::start_series() sets series_at, abandoned_at and abandon_trigger in one UPDATE on series_at IS NULL.
  • CP7 Skipped runs (a476dfb)
    • New Trigger::skip_reason(), null by default. On a refused claim with a reason, the Dispatcher records one finished skipped run per active workflow of that trigger.
    • The reasons reuse the existing "placed order #N" and "unsubscribed" texts; two new strings (cart already in a series, 7 days).
    • A Skipped run "placed order #N" links to the order, like a cancelled one.
  • CP8 .pot, docs, PR text (eacce9d)
  • Review fixes:
    • 3143d5b: a new run first drops any index rows left for its id.
    • 6152b40: a cart already in a series, abandoned again, records a Skipped run "This cart already got a reminder series."
    • e0df0db, e2e harness only: through Docker the specs' WP-CLI steps land a few seconds late. The journeys re-run the queue and check again until each expected run status or mail appears (up to 30 s), each try making the same assertion. The unsubscribe journey is marked slow. The health-check test reloads before each check and stamps the hourly self-check after removing the job.
    • languages/yaymatik.pot rebuilt.
    • Decisions: PI (privacy index), M8 updated.

Bench (wp-env, MariaDB, PHP 8.1)

The measured shopper has 125 runs at 50k and 500 at 200k.

Lookup 50k old ms 50k new ms 200k old ms 200k new ms
for_export p1 1393 1.6 5510 3.6
for_export p2 1507 1.4 5598 4.3
ids_mentioning 1353 1.3 5675 3.6
in_progress_mentioning 0.7 0.9 2.2 2.1
waiting_ids_for_subjects 45 1.1 255 3.1
  • No plan scans runs or run_steps in full.
  • Write cost of a run with 5 steps whose values all hold an email: 8.9 ms with indexing vs 5.6 ms with nothing to index.

Side effects

  • yaymatik_cart_abandoned fires on every abandon, a cart already in a series included, not once per cycle, and gains $kind. It is experimental, like the other hooks.
  • A cart in a series that is abandoned again records a Skipped run "This cart already got a reminder series." each time, in the workflows of that abandon's kind.
  • Skipped runs count in the overview's Runs (30 days) and Last run, and toward the 1000-runs-per-minute ceiling. A sweep is at most 200 carts, so skips alone can't pause a workflow.

Tests

  • pnpm test:php (PHPUnit), pnpm lint:php.
  • ./run.sh check-all --wp-env and --wp-env-min (check-privacy-index added).
  • pnpm e2e in mcr.microsoft.com/playwright:v1.64.1-noble against wp-env: 11/11. develop with only this PR's e2e harness change: 11/11 too.
  • ./run.sh probe-mysql --staging. Its erase-REGEXP part is replaced by the index SQL: the derived-table SELECT now, and the three lookups once staging is at DB 8. Run it again after the deploy and reset-data.
  • ./run.sh bench-privacy --wp-env.

WordPress.org review notes

  • RunStore::index_emails() takes $runs, a fixed SELECT … FROM %i WHERE id = %d-style fragment written in RunStore / RunStepStore, with its values passed separately. about() returns a fixed derived-table fragment the same way. As with $filter and Schema::in(), no value from a request or the database is written into the SQL text.

Browser check for Jack (UI only)

0 Before you start

  • A cart gets at most one reminder series until it is ordered. An email gets at most one series in 7 days.
  • The abandon that starts the series decides its kind (Cart or Checkout). An abandon that no workflow picks up does not lock the cart.
  • A refused abandon (the cart already has a series, it was ordered, the email unsubscribed, or the email already had a series in 7 days) shows a Skipped row with its reason in Workflow runs.
  • Timing: a cart counts as abandoned after 15 minutes idle, and the site checks carts every 5 minutes. So after leaving, wait about 20–25 minutes and reload the Workflows page now and then. Both templates then wait 1 hour before emailing: the run shows "Waiting" first, and the email arrives about 1 hour later.

1 Setup

  • Plugins › Add New Plugin › Upload Plugin: upload the 033 zip and choose "Replace current with uploaded". Then YayCommerce › YayMatik › Settings › Health check › Check again.
  • Use only customer emails never used on this site. Runs written before 033 aren't indexed, so export and erase won't find them.
  • Settings › Cart tracking: turn on "Track carts"; set "Count a cart as abandoned after" to 15 minutes; Save changes.
  • Workflows › Browse templates: create "Cart reminder" and "Checkout reminder". Leave both off for now.
  • Three customer accounts with different emails (A, B, C). Each customer uses a separate browser or private window, logged in there.
  • An inbox to read the emails (or the site's mail log, if there is one).

2 Steps

B1. Cart only

  • Goal: a cart left on the cart page starts Cart reminder.
  • Do: turn on "Cart reminder". Customer A adds 1 product to the cart, does not go to Checkout, and closes the tab. Wait about 20–25 minutes and reload Workflows › Cart reminder a few times.
  • Pass when: Cart reminder's Workflow runs has 1 "Waiting" run for A's cart. About 1 hour later, A gets the email "You left something in your cart".
  • Why: an active workflow picks up this abandon and starts the series.

B2. Checkout only

  • Goal: an abandon before Checkout does not use up the later Checkout reminder.
  • Do: turn off "Cart reminder", turn on "Checkout reminder". Customer B adds a product, does not go to Checkout, and closes the tab. Wait about 20–25 minutes and reload Checkout reminder. Then B opens the site again, goes to the Checkout page, and closes the tab. Wait about 20–25 minutes and reload a few times.
  • Pass when: after the first leave, Checkout reminder has no runs. After the second, it has exactly 1 "Waiting" run for B's cart. About 1 hour later, B gets the checkout reminder email.
  • Why: no workflow picked up the first abandon, so the cart was not locked.

B3. Both on

  • Goal: a cart never gets two reminder series.
  • Do: turn both on. Customer C adds a product, does not go to Checkout, and closes the tab. Wait about 20–25 minutes and reload: Cart reminder has 1 run for C's cart. Then C opens the site again, goes to the Checkout page, and closes the tab. Wait about 20–25 minutes and reload a few times.
  • Pass when: Cart reminder still has only 1 real run for C's cart. Checkout reminder has no real runs, only 1 row in the Skipped tab: "This cart already got a reminder series.". C gets only the Cart email.
  • Why: the cart already has a series, so the later abandon does not start a new one; it only records the reason.

B4. Same cart again

  • Goal: a repeat attempt shows up instead of passing silently.
  • Do: keep "Cart reminder" on. Customer A (the cart from B1, not ordered) opens the site again, goes to the cart page, does not go to Checkout, and closes the tab. Wait about 20–25 minutes and reload Cart reminder a few times.
  • Pass when: Cart reminder's Workflow runs › Skipped tab has 1 new row; opening it shows "This cart already got a reminder series.". A gets no email beyond the one from B1, and Cart reminder still has only 1 real run for this cart.
  • Why: a cart gets one reminder series. Before 033 this attempt left no trace, so testers thought the workflow was broken.

B5. Privacy: export and erase

  • Goal: export finds the customer's runs, Skipped ones included; erase removes the email from those runs and keeps the runs.
  • Do: Tools › Export Personal Data, enter A's email (it has the Cart run from B1 and the Skipped row from B4), click Send Request, then click "Download personal data" on the request row. Then Tools › Erase Personal Data, enter the same email, click "Erase personal data". Finally, export again.
  • Pass when: the first export file has a YayMatik runs section with the B1 run and the B4 Skipped row. Erase reports done. The runs are still in Cart reminder's Workflow runs: the Subject column shows "Cart #N" (no email), and opening the run shows "Sent to [deleted]" on the Send email step. The second export has no YayMatik runs (only the unsubscribe record, if any).
  • Not a bug: after erase, WooCommerce may report that some data was not erased (download permissions). That is WooCommerce's data, not YayMatik's.
  • Why: export and erase now find runs through the index table instead of scanning everything.

3 Results

Step Pass when Result Notes
B1 Cart only 1 Cart run (Waiting), email after ~1 hour
B2 Checkout only First leave: no run; second: exactly 1 Checkout run
B3 Both on 1 Cart run; Checkout only 1 Skipped row "already got a reminder series"; Cart email only
B4 Same cart again Skipped row "This cart already got a reminder series.", no new email
B5 Privacy Export has the run + Skipped row; after erase the runs stay, Subject "Cart #N", Send email "Sent to [deleted]"; second export has none

…s; erase and run deletes keep the index in sync
… series; each abandon without one fires with its kind
@jack9x
jack9x merged commit 6be688a into develop Nov 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant